...

One of the tables in vo.astron.nl is called ivoa.obscore. This table is a special table which follows the ObsCore standard defined by the IVOA and it contains key observational information describing all the data products (images, cubes, etc) in our VO service. Currently, there is not service to directly query this table, but it can be done using ADQL.

...

XXX

Selecting a data collection allows the user to perform a cone search through a webform (Fig. 40). The result is either a source or Gaussian list, or a table of data products of that given class overlapping a given pointing. The size of the continuum images as well as the cubes extend beyond the 10% primary beam level for cleaning the secondary lobes of bright offset sources. To ensure that the search is done in the area of maximum sensitivity the search is performed on a maxim radius of 0.75 degrees from the center (this represents the average value of where the sensitivity drops). This value can be modified using the Max distance from center. A different output with respect to the default can be customized using More output fields selection button.

Info
icon false
Image Removed Fig. 40 Query search form for continuum images.

The result is a table in the requested output format in which every row corresponds to a data product (Fig. 41).

In each row there is a column, Product key, which is a link that allows the user to download the fits file of the image of interest. The column titles should generally be self-descriptive. However, the long human-readable description of the content of each column is a tooltip that will appear when hovering over the column name.

The selected target and the position of the individual pointings can be visualized using the Quick plot button at the top of the window of the results of the search query (Fig. 41).

In the column Related products another link connects to a page containing a list of links to additional related data that can be useful to interpret or reanalyze that given product, for which a preview is provided (Fig. 42).

Info
icon false
Image Removed Fig. 41 Result of an image search query. Click for a bigger image.

Info
icon false
Image Removed Fig. 42 Links to data products related to the target of interest. The top two items represent the primary data product (ie the one that can be directly downloaded from the table view) and a link to the thumbnail of that data product. The other products are the anciliary data. Click for a bigger image. Note that the number of related products is too long to fit readably in a single screen shot.

Some data products are directly accessible, in which case the link on this table will initiate a direct download. However, the larger (and rawer) data products are stored on tape (in which case the text 'on tape' appears in the description). The link will then bring you to the entry of the corresponding pointing in the SURF Data repository (Fig. 43/1). If data is on tape, the status is either "online", in which case it can be downloaded by pressing the "Download" button, or "offline" in which case it can be requested to be put online by pressing the "Request" button. Please note that you need to be logged in to perform the request. If a request has been correctly performed, the status of the file will change to "staging" unill it becomes "online". 

Info
icon false
Image Removed Fig. 43/1 Surf Data Repository view of the data from a pointing. Click for a bigger image.

The source and Gaussian cone search forms each return a table with source positions and properties. As before, the long descriptions are available using tool tips. The columns "Mosaic_URL" links to the anciliary data product page of the mosaic from where the Gaussian or source was extracted (like e.g. Fig 42).

The columns shown in Figure 41 are the most informative for the astronomers (e.g. position, observing frequency, observing date, quality assessment, format etc), please note that more columns are available but not displayed here. The complete set of columns can be visualized via topcat as described below or using More output fields selection button in the search query. Querying the released data is also possible using e.g. TOPCAT using TAP. Via the TAP protocol, it is possible to query the registry in a more flexible way using an enriched SQL syntax called ADQL. An example is given in Fig. 43 : click the link indicated with the red ellipse on the left panel Try ADQL and place your ADQL query on the query form.

Info
icon false
Image Removed Fig. 43/2 ADQL query form.

The table names to use in the query form of Fig. 43/2, are summarized in Table 10. The URL for the query is then: https://vo.astron.nl/lotss_dr2/q/{Table name}/form (e.g. https://vo.astron.nl/lotss_dr2/q/src_cone/form}.

It is possible to query all the available dataproducts at once by using the table ivoa.obscore and by appending to the ADQL statement “where obs_collection=” it is possible to limit the search to the apertif_dr1 only.

VO-Apertif DR1 Processed Data Products

Table names to be used in the ADQL query.

...

Table name

...

obscore type

...

obscore subtype

...

gauss_cone

...

table (not an obscore type)

...

src_cone

...

table (not an obscore type)

...

lotss_dr2_mosaic

...

image

...

total intensity map

Data access through DACHS interface

Selecting a data collection allows the user to perform a cone search through a webform (Fig. 40). The result is either a source or Gaussian list, or a table of data products of that given class overlapping a given pointing. The size of the continuum images as well as the cubes extend beyond the 10% primary beam level for cleaning the secondary lobes of bright offset sources. To ensure that the search is done in the area of maximum sensitivity the search is performed on a maxim radius of 0.75 degrees from the center (this represents the average value of where the sensitivity drops). This value can be modified using the Max distance from center. A different output with respect to the default can be customized using More output fields selection button.

Info
icon false
Image Added Fig. 40 Query search form for continuum images.

The result is a table in the requested output format in which every row corresponds to a data product (Fig. 41).

In each row there is a column, Product key, which is a link that allows the user to download the fits file of the image of interest. The column titles should generally be self-descriptive. However, the long human-readable description of the content of each column is a tooltip that will appear when hovering over the column name.

The selected target and the position of the individual pointings can be visualized using the Quick plot button at the top of the window of the results of the search query (Fig. 41).

In the column Related products another link connects to a page containing a list of links to additional related data that can be useful to interpret or reanalyze that given product, for which a preview is provided (Fig. 42).

Info
icon false
Image Added Fig. 41 Result of an image search query. Click for a bigger image.

Info
icon false
Image Added Fig. 42 Links to data products related to the target of interest. The top two items represent the primary data product (ie the one that can be directly downloaded from the table view) and a link to the thumbnail of that data product. The other products are the anciliary data. Click for a bigger image. Note that the number of related products is too long to fit readably in a single screen shot.

The source and Gaussian cone search forms each return a table with source positions and properties. As before, the long descriptions are available using tool tips. The columns "Mosaic_URL" links to the anciliary data product page of the mosaic from where the Gaussian or source was extracted (like e.g. Fig 42).

The columns shown in Figure 41 are the most informative for the astronomers (e.g. position, observing frequency, observing date, quality assessment, format etc), please note that more columns are available but not displayed here. The complete set of columns can be visualized via topcat as described below or using More output fields selection button in the search query. Querying the released data is also possible using e.g. TOPCAT using TAP. Via the TAP protocol, it is possible to query the registry in a more flexible way using an enriched SQL syntax called ADQL. An example is given in Fig. 43 : click the link indicated with the red ellipse on the left panel Try ADQL and place your ADQL query on the query form.

Info
icon false
Image Added Fig. 43/2 ADQL query form.

The table names to use in the query form of Fig. 43/2, are summarized in Table 10. The URL for the query is then: https://vo.astron.nl/lotss_dr2/q/{Table name}/form (e.g. https://vo.astron.nl/lotss_dr2/q/src_cone/form}.

It is possible to query all the available dataproducts at once by using the table ivoa.obscore and by appending to the ADQL statement “where obs_collection=” it is possible to limit the search to the apertif_dr1 only.

Access via TOPCAT

The Apertif DR1 data collection tables can be accessed using TOPCAT, an interactive graphical viewer and editor for tabular data. The data can be sent from vo.astron.nl to TOPCAT using one of the

Access via TOPCAT

The Apertif DR1 data collection tables can be accessed using TOPCAT, an interactive graphical viewer and editor for tabular data. The data can be sent from vo.astron.nl to TOPCAT using one of the two protocols: SAMP or TAP. The two subsections below provide a description on how to access the tabular data using either SAMP(link to Send via SAMP subsection) or TAP(link to VO Table Access Protocol (TAP) subsection).

...

Info
icon false
Fig. 48 How to transfer the TOPCAT query results to ALADIN. Click for a bigger image.

Access via ALADIN

Catalogues

The Apertif DR1 VO data collection can also be discovered directly via ALADIN either via simple image access protocol (SIAP) or tabular access protocol (TAP). The examples shown here require the desktop version of ALADIN.

...

Info
icon false
Fig. 54 Example of image selected from the Apertif DR1 displayed in ALADIN. Click for a bigger image.

ADQL

Anchor
sec-adql sec-adql

Python access

The data collection and the table content can be accessed directly via python using the pyvo tool. Working directly in python the tables and the data products can be simply queried and outputs can be customized according to the user’s needs, without the involvement of TOPCAT or ALADIN.

An example of a TAP query and image download can be found in the python script below (it has been tested for python 3.7). The result of the query can also be plotted using python.

Simple examples

Export machine readable table

There are multiple ways to export a catalog of the various data products of the data release. On the vo.astron.nl pages, the results of a query can be exported to a csv file or fits table; running an empty query with a table limit of 5000 or more will return all entries.

TOPCAT and the pyvo interface demonstrated above also provide functionality for exporting machine-readable files.

The ADQL form is another option, and below we provide an example query that also provides information about the calibrators used for each beam. This query is specific to the continuum_images data product but can be adapted to other (beam-based, processed) data products by replacing the table name, e.g., for polarization cubes/images use pol_cubes (see Table 10 for a full list of the available tables).

select data.*,
flux_cal.obsid as flux_calibrator_obs_id,
pol_cal.obsid as pol_calibrator_obs_id  from apertif_dr1.continuum_images data
join  apertif_dr1.flux_cal_visibilities flux_cal on data.obsid=flux_cal.used_for and data.beam_number=flux_cal.beam
join apertif_dr1.pol_cal_visibilities pol_cal on data.obsid=pol_cal.used_for and data.beam_number=pol_cal.beam
order by obsid

Python access

The data collection and the table content can be accessed directly via python using the pyvo tool. Working directly in python the tables and the data products can be simply queried and outputs can be customized according to the user’s needs, without the involvement of TOPCAT or ALADIN.

An example of a TAP query and image download can be found in the python script below (it has been tested for python 3.7). The result of the query can also be plotted using python.

#To start you have to import the library pyvo (it is also possible to use astroquery if you want)
import pyvo

## To perform a TAP query you have to connect to the service first
tap_service = pyvo.dal.TAPService('https://vo.astron.nl/__system__/tap/run/tap')

# This works also for
form pyvo.registry.regtap import ivoid2service
vo_tap_service = ivoid2service('ivo://astron.nl/tap')[0]

# The TAPService object provides some introspection that allow you to check the various tables and their
# description for example to print the available tables you can execute
print('Tables present on http://vo.astron.nl')
for table in tap_service.tables:
 print(table.name)
print('-' * 10 + '\n' * 3)

# or get the column names
print('Available columns in apertif_dr1.continuum_images')
print(tap_service.tables['apertif_dr1.continuum_images'].columns

#To start you have to import the library pyvo (it is also possible to use astroquery if you want)
import pyvo

## To perform a TAP query you have to connect to the service first
tap_service = pyvo.dal.TAPService('https://vo.astron.nl/__system__/tap/run/tap')

# This works also for
form pyvo.registry.regtap import ivoid2service
vo_tap_service = ivoid2service('ivo://astron.nl/tap')[0]

# The TAPService object provides some introspection that allow you to check the various tables and their
# description for example to print the available tables you can execute
print('Tables present on http://vo.astron.nl')
for table in tap_service.tables:
 print(table.name)
print('-' * 10 + '\n' * 3)

# or get the column names
print('Available columns in apertif_dr1.continuum_images')
print(tap_service.tables['apertif_dr1.continuum_images'].columns)
print('-' * 10 + '\n' * 3)

## You can obviously perform tap queries accross the whole tap service as an example a cone search
print('Performing TAP query')
result = tap_service.search(
 "SELECT TOP 5 target, beam_number, accref, centeralpha, centerdelta, obsid, DISTANCE(" \
     "POINT('ICRS', centeralpha, centerdelta),"\
     "POINT('ICRS', 208.36, 52.36)) AS dist"\
 " FROM apertif_dr1.continuum_images"  \
 " WHERE 1=CONTAINS("
 "    POINT('ICRS', centeralpha, centerdelta),"\
 "    CIRCLE('ICRS', 208.36, 52.36, 0.08333333)) "\
 " ORDER BY dist ASC"
 )
print(result)

# The result can also be obtained as an astropy table
astropy_table = result.to_table()
print('-' * 10 + '\n' * 3)

## You can alsoobviously perform downloadtap andqueries plotaccross the image
import astropy.io.fitswhole tap service as fits
from astropy.wcs import WCS
import matplotlib.pyplot as plt
import requests, os
import numpy as np

# DOWNLOAD only the first result
#
print('Downloading only the first result')
file_name = '{}_{}_{}.fits'.format(
 result[0]['obsid'].decode(),
 result[0]['target'].decode(),
 result[0]['beam_number'])
path = os.path.join(os.getcwd(), file_name)
http_result = requests.get(result[0]['accref'].decode())
print('Downloading file in', path)
with open(file_name, 'wb') as fout:
 for content in http_result.iter_content():
     fout.write(content)
hdu = fits.open(file_name)[0]
wcs = WCS(hdu.header)
# dropping unnecessary axes
wcs = wcs.dropaxis(2).dropaxis(2)
plt.subplot(projection=wcs)
plt.imshow(hdu.data[0, 0, :, :], vmax=0.0005)
plt.xlabel('RA')
plt.ylabel('DEC')
plt.show()

Export machine readable table

There are multiple ways to export a catalog of the various data products of the data release. On the vo.astron.nl pages, the results of a query can be exported to a csv file or fits table; running an empty query with a table limit of 5000 or more will return all entries.

TOPCAT and the pyvo interface demonstrated above also provide functionality for exporting machine-readable files.

The ADQL form is another option, and below we provide an example query that also provides information about the calibrators used for each beam. This query is specific to the continuum_images data product but can be adapted to other (beam-based, processed) data products by replacing the table name, e.g., for polarization cubes/images use pol_cubes (see Table 10 for a full list of the available tables).

select data.*,
flux_cal.obsid as flux_calibrator_obs_id,
pol_cal.obsid as pol_calibrator_obs_id  from apertif_dr1.continuum_images data
join  apertif_dr1.flux_cal_visibilities flux_cal on data.obsid=flux_cal.used_for and data.beam_number=flux_cal.beam
join apertif_dr1.pol_cal_visibilities pol_cal on data.obsid=pol_cal.used_for and data.beam_number=pol_cal.beam
order by obsidan example a cone search
print('Performing TAP query')
result = tap_service.search(
 "SELECT TOP 5 target, beam_number, accref, centeralpha, centerdelta, obsid, DISTANCE(" \
     "POINT('ICRS', centeralpha, centerdelta),"\
     "POINT('ICRS', 208.36, 52.36)) AS dist"\
 " FROM apertif_dr1.continuum_images"  \
 " WHERE 1=CONTAINS("
 "    POINT('ICRS', centeralpha, centerdelta),"\
 "    CIRCLE('ICRS', 208.36, 52.36, 0.08333333)) "\
 " ORDER BY dist ASC"
 )
print(result)

# The result can also be obtained as an astropy table
astropy_table = result.to_table()
print('-' * 10 + '\n' * 3)

## You can also download and plot the image
import astropy.io.fits as fits
from astropy.wcs import WCS
import matplotlib.pyplot as plt
import requests, os
import numpy as np

# DOWNLOAD only the first result
#
print('Downloading only the first result')
file_name = '{}_{}_{}.fits'.format(
 result[0]['obsid'].decode(),
 result[0]['target'].decode(),
 result[0]['beam_number'])
path = os.path.join(os.getcwd(), file_name)
http_result = requests.get(result[0]['accref'].decode())
print('Downloading file in', path)
with open(file_name, 'wb') as fout:
 for content in http_result.iter_content():
     fout.write(content)
hdu = fits.open(file_name)[0]
wcs = WCS(hdu.header)
# dropping unnecessary axes
wcs = wcs.dropaxis(2).dropaxis(2)
plt.subplot(projection=wcs)
plt.imshow(hdu.data[0, 0, :, :], vmax=0.0005)
plt.xlabel('RA')
plt.ylabel('DEC')
plt.show()