• in query form say we use the default but more can be found using the service info and topcat/pyvo.

Introduction

This page desribes access to the data in ASTRON's data holdings using the Virtual Observatory standards. Those standards support access to data like catalogs and images. We introduce the most important standards on this page and give some usage exaples taylored to the ASTRON data holdings. If you want more advanced information on how to use the VO standards, you can start with the IVOA website for astronomers and the IVOA wiki page on educational resources. For our data holdings we use the DAta Center Helper Suite (DACHS), which offers a web interface of its own. We will link to several places in DACHS in the documentation but in principle all information showed by DACHS can also be accessed using any VO tools.

Data access protocols

The VO consists of several different protocols. In this section we describe the import ant protocols, and provide the acronyms that are generally used in the VO tooling. Most data in vo.astron.nl consist of tables, that can be accessed using the Tabular Access Protocol (TAP). Images are offered using the Simple Image Access Protocol (SIA) which in essence is a table with a field containing a link to an image. A few services offer cutout fuctionality (using SODA). A cutout is essentially a service that returns an image (or more generally a data product) that only contains the data inside the cone that has been requested by the user.

In some cases, the table describing the data will contain a so-called DataLink document that links the primary data product (often an image or cube) to related data products (e.g. raw visibilities, calibration solutions).

Another relevant VO standard to mention here is the Simple Application Messaging Protocol (SAMP). This protocol makes it possible for applications to share information. This allows users to for example query the VO with one application, visuaklise the results with another and do analysis with a third one.

HiPS

The Hierarchical Progressive Survey (HiPS) protocol defines a way to visualise sky surveys by breaking them up in different hierarchical views which represents different zoom levels. This makes it possible to scroll through a survey in a way that is comparable to using Google Maps (or other online mapping/satellite tools). ASTRONs HiPS collections are all available in Aladin and the most important ones are available through other tools, like ESASky. However users can also have direct access to all our HiPS collections through hips.astron.nl. Typically, HiPS projections are used to visually look through the data from a survey, and combine the coordinates with table-based services to obtain source information or data. The following table shows an overview of the ASTRON HiPS server contains the following data collections.

The ObsCore table

One of the tables in vo.astron.nl is called ivoa.obscore . This table is a special table which follows the ObsCore standard defined by the IVOA and it contains key observational information describing all the data products (images, cubes, etc) in our VO service. Currently, there is not service to directly query this table, but it can be done using ADQL.

List of vo.astron.nl services

The data published in the VO can also be accessed using a web browser at https://vo.astron.nl. The web interface can be used to perform simple queries. The power of the VO lies in the fact that several applications exist that can interact with the standards we offer. In the architecture, there is a distinction between services and tables. A service is an entity that takes user input and provides a table as output. The tables themselves are the entities that actually hold the data. Tables can also be queried directly, for instance using ADQL. The colomns of the table are:

This web interface provides a page on which all the collections present in the registry are list

Data access through DaCHS interface

The DaCHS system comes with a web view on the services offered. Even though the interface is somewhat rudimentary, it does offer the relevant functionality to query data in the ASTRON VO. The main page, listing all services (as listed in the table above) can be fount at https://vo.astron.nl . Specific services are linked to that (either by clicking on the name of the collection, or the Q icon next to it.

Selecting a data collection allows the user to perform a cone search through a webform (Fig. 1). The result is either a catalogue (list of sources, Gaussians or visibilities), or a table of data products (images, cubes) of that given class overlapping a given pointing. In some cases, the number query parameters can differ, in which case the field description explains what the data is that can be submitted to them.

Fig. 1 Query search form for continuum images.

The result is a table in the requested output format in which every row corresponds to a data product (Fig. 2). In each row there is a column, Product key, which is a link that allows the user to download the fits file of the image of interest. The column titles should generally be self-descriptive. However, the long human-readable description of the content of each column is a tooltip that will appear when hovering over the column name. The result is a table (like Fig. 2) which consists of a link to the data product, a thumbnail image  that appears when hovering it (both only if querying for a data product) and relevant metadata describing the results. The Quick Plot button on top of the results page can be used to quicikly plot the numerical results (e.g. making an RA/DEC overview of the images or sources). The Send via SAMP button can be used to send the result set to an application using the SAMP protocol. This essentially means that if you have another application open which supports the SAMP protocol (as do both TOPCAT and Aladin), clicking this button will make the table appear in those applications.

In some cases, auxiliary data is available for the primary image/cube results (e.g. raw visibilities, calibration solutions, etc). Those files are connected to the primary result using a DataLink document. The link in the table is generally shown with the name dlmeta. This is essentially a table with links and predefined descriptions. In the first column the link is shown, together with an (in some cases estimated) file size. The second column consists of a human-readable description and the last column shows the relation between the linked dataset and the primary product, using the vocabulary defined for this, making the result machine-readable. An example of a DataLink document is shown in Fig. 3.

One special service is the ADQL query service, shown on Fig 4.. This contains a single box that can be used to provide custom queries using the Astronomical Data Query Language (ADQL), which in essence is a dialect of SQL. Some more information on, and examples of ADQL can be found further below.

Fig. 4 ADQL query form.

Using the DaCHS interface can be useful for users but the advantage of using a standard is that generic tooling can be used to query and access the data. Two very well-known tools are TOPCAT and Aladin. We will give an overview 

Access via TOPCAT

The Apertif DR1 data collection tables can be accessed using TOPCAT, an interactive graphical viewer and editor for tabular data. Since the ASTRON VO is regsitered to the so-called registry of registries, TOPCAT can find the relevant services in its query menus. In the VO menu of TOPCAT you can pick the SIA query menu item, which will bring up a form as shown in Fig. 5. When selecting one of the data collections the SIA URL will be automatically filel in. One can then either type coordinates, or an object name and click the "Resolve" button which uses Simbad to obtain coordinates for that obiect. When clicking OK, a table of the data products found will appead in the main TOPCAT window.

Querying a table through TAP

The data can be sent from vo.astron.nl to TOPCAT using one of the two protocols: SAMP or TAP. The two subsections below provide a description on how to access the tabular data using either SAMP(link to Send via SAMP subsection) or TAP(link to VO Table Access Protocol (TAP) subsection).

Send via SAMP

With TOPCAT opened, and once you are satisfied with the output of the cone search in the Astron VO webform, click the grey button “Send via SAMP” as shown on the top of the output list of Figure 3. Authorize the connection and wait until the download is completed.

Once completed, the catalogue will be visible in the left panel of TOPCAT (Table List). Click on the new entry as shown by the arrow in Fig. 44. At this point the table browser will open showing the content of the DACHS results (PLACE HOLDER use DR1 in selection). From here any TOPCAT tool can be used for further inspection and analysis of the results. Alternatively the table can be saved in various formats and used locally with other programs (e.g. python scripts etc).

Fig. 44 TOPCAT table browser view of the Apertif DR1 data collection tables. Click for a bigger image.

VO Table Acess Protocol (TAP)

From the TOPCAT menu bar, select VO and in the drop down, select Table Access Protocol (TAP) as shown by the red arrow in Fig. 45.

This will open the Table Access Query window where the ASTRON VO TAP server is listed. Select it and click on Use Service at the bottom of the window (Fig. 46). Another tab will open showing the Apertif DR1 data collection. Select one, e.g. continuum image, and enter a query command in the bottom panel, an example of which is indicated by the red arrow in Fig. 47. Submit the query using Run Query. This will show the resulting table in the Table list view shown before in Fig. 44. From here, any TOPCAT tool can again be used. As mentioned in the previous section, the query result in vo.astron.nl will display a subset of the columns of the Apertif DR1 table (e.g. position, observing frequency, observing date, quality assessment, format etc). The complete set of columns (e.g. pipeline version, wcs references etc) belonging to each data collection of the Apertif DR1 can be explored using the option described in this section.

The position of the targets can be visualized using the option skyplot in TOPCAT, once the search query has been sent via SAMP or TAP as described in this section.

Fig. 45 TOPCAT menu bar for VO services. Click for a bigger image.

Fig. 46 TOPCAT TAP service query form. Click for a bigger image.

Fig. 47 TOPCAT menu bar for VO services. Click for a bigger image.

Another useful way to inspect the Apertif DR1 data collection, but also other data collections exposed via the ASTRON-VO, is the ivoa-obscore table. The same selection as before can be used (Fig. 47) but instead of selecting Apertif_dr1 tables, the table ivoa.obscore is to be selected. In this way it is possible to glance over all the data collections exposed via the ASTRON-VO. This might be useful for instance to enable multi-wavelength science exploring LOTSS and Apertif DR1 data or, as mentioned in the case of the ADQL query, to visualize multiple data collections at once.

Having ALADIN open, and once you are satisfied with the resulting table, it can be sent to ALADIN following the instructions of Fig. 48.

Fig. 48 How to transfer the TOPCAT query results to ALADIN. Click for a bigger image.

Access via ALADIN

Catalogues

The Apertif DR1 VO data collection can also be discovered directly via ALADIN either via simple image access protocol (SIAP) or tabular access protocol (TAP). The examples shown here require the desktop version of ALADIN.

Open ALADIN and on the left panel for SIAP: select Others > SIA2 > astron.nl > The VO @ASTRON SIAP Version 2. Alternatively for TAP select Others > TAP > astron.nl > The VO @ASTRON TAP service (Fig. 49). A pop-up window will open. Click load, and enter a query using the Server selector (Fig. 50) or TAP access with astron.nl/tap (Fig. 51) to select the target of interest for SIAP and TAP respectively.

Fig. 49 ALADIN display panel. Click for a bigger image.

Fig. 50 ALADIN server selector panel for SIAP. Click for a bigger image.

Fig. 51 ALADIN TAP access panel. Click for a bigger image.

After loading, the data collection catalogues can be plotted on the main panel by selecting them first on the right panel (e.g. highlighted in blue in Figures 14 and 15) and then by selecting the regions of interest on the bottom panel as shown in Figures 14 and 15. From here the usual functionality of ALADIN can be used.

HiPS

Fig. 52 Example of data collection selection via SIAP in ALADIN. Click for a bigger image.

Fig. 53 Example of data collection selection via TAP in ALADIN. Click for a bigger image.

Images

Downloading images or cubes in ALADIN is also possible (see Fig. 54). The user will need to click on the url-link in the column access_url of the bottom panel. Then, once the image is loaded, click on the right panel as shown in Fig. 54. From here the usual functionality of ALADIN can be used.

Fig. 54 Example of image selected from the Apertif DR1 displayed in ALADIN. Click for a bigger image.

ADQL

Simple examples

Export machine readable table

There are multiple ways to export a catalog of the various data products of the data release. On the vo.astron.nl pages, the results of a query can be exported to a csv file or fits table; running an empty query with a table limit of 5000 or more will return all entries.

TOPCAT and the pyvo interface demonstrated above also provide functionality for exporting machine-readable files.

The ADQL form is another option, and below we provide an example query that also provides information about the calibrators used for each beam. This query is specific to the continuum_images data product but can be adapted to other (beam-based, processed) data products by replacing the table name, e.g., for polarization cubes/images use pol_cubes (see Table 10 for a full list of the available tables).

select data.*,
flux_cal.obsid as flux_calibrator_obs_id,
pol_cal.obsid as pol_calibrator_obs_id  from apertif_dr1.continuum_images data
join  apertif_dr1.flux_cal_visibilities flux_cal on data.obsid=flux_cal.used_for and data.beam_number=flux_cal.beam
join apertif_dr1.pol_cal_visibilities pol_cal on data.obsid=pol_cal.used_for and data.beam_number=pol_cal.beam
order by obsid

Python access

The data collection and the table content can be accessed directly via python using the pyvo tool. Working directly in python the tables and the data products can be simply queried and outputs can be customized according to the user’s needs, without the involvement of TOPCAT or ALADIN.

An example of a TAP query and image download can be found in the python script below (it has been tested for python 3.7). The result of the query can also be plotted using python.

#To start you have to import the library pyvo (it is also possible to use astroquery if you want)
import pyvo

## To perform a TAP query you have to connect to the service first
tap_service = pyvo.dal.TAPService('https://vo.astron.nl/__system__/tap/run/tap')

# This works also for
form pyvo.registry.regtap import ivoid2service
vo_tap_service = ivoid2service('ivo://astron.nl/tap')[0]

# The TAPService object provides some introspection that allow you to check the various tables and their
# description for example to print the available tables you can execute
print('Tables present on http://vo.astron.nl')
for table in tap_service.tables:
 print(table.name)
print('-' * 10 + '\n' * 3)

# or get the column names
print('Available columns in apertif_dr1.continuum_images')
print(tap_service.tables['apertif_dr1.continuum_images'].columns)
print('-' * 10 + '\n' * 3)

## You can obviously perform tap queries accross the whole tap service as an example a cone search
print('Performing TAP query')
result = tap_service.search(
 "SELECT TOP 5 target, beam_number, accref, centeralpha, centerdelta, obsid, DISTANCE(" \
     "POINT('ICRS', centeralpha, centerdelta),"\
     "POINT('ICRS', 208.36, 52.36)) AS dist"\
 " FROM apertif_dr1.continuum_images"  \
 " WHERE 1=CONTAINS("
 "    POINT('ICRS', centeralpha, centerdelta),"\
 "    CIRCLE('ICRS', 208.36, 52.36, 0.08333333)) "\
 " ORDER BY dist ASC"
 )
print(result)

# The result can also be obtained as an astropy table
astropy_table = result.to_table()
print('-' * 10 + '\n' * 3)

## You can also download and plot the image
import astropy.io.fits as fits
from astropy.wcs import WCS
import matplotlib.pyplot as plt
import requests, os
import numpy as np

# DOWNLOAD only the first result
#
print('Downloading only the first result')
file_name = '{}_{}_{}.fits'.format(
 result[0]['obsid'].decode(),
 result[0]['target'].decode(),
 result[0]['beam_number'])
path = os.path.join(os.getcwd(), file_name)
http_result = requests.get(result[0]['accref'].decode())
print('Downloading file in', path)
with open(file_name, 'wb') as fout:
 for content in http_result.iter_content():
     fout.write(content)
hdu = fits.open(file_name)[0]
wcs = WCS(hdu.header)
# dropping unnecessary axes
wcs = wcs.dropaxis(2).dropaxis(2)
plt.subplot(projection=wcs)
plt.imshow(hdu.data[0, 0, :, :], vmax=0.0005)
plt.xlabel('RA')
plt.ylabel('DEC')
plt.show()