• in query form say we use the default but more can be found using the service info and topcat/pyvo.

Introduction

This page desribes access to the data in ASTRON's data holdings using the Virtual Observatory standards. Those standards support access to data like catalogs and images. We introduce the most important standards on this page and give some usage exaples taylored to the ASTRON data holdings. If you want more advanced information on how to use the VO standards, you can start with the IVOA website for astronomers and the IVOA wiki page on educational resources. For our data holdings we use the DAta Center Helper Suite (DACHS), which offers a web interface of its own. We will link to several places in DACHS in the documentation but in principle all information showed by DACHS can also be accessed using any VO tools.

Data access protocols

The VO consists of several different protocols. In this section we describe the import ant protocols, and provide the acronyms that are generally used in the VO tooling. Most data in vo.astron.nl consist of tables, that can be accessed using the Tabular Access Protocol (TAP). Images are offered using the Simple Image Access Protocol (SIA) which in essence is a table with a field containing a link to an image. A few services offer cutout fuctionality (using SODA). A cutout is essentially a service that returns an image (or more generally a data product) that only contains the data inside the cone that has been requested by the user.

In some cases, the table describing the data will contain a so-called DataLink document that links the primary data product (often an image or cube) to related data products (e.g. raw visibilities, calibration solutions).

Another relevant VO standard to mention here is the Simple Application Messaging Protocol (SAMP). This protocol makes it possible for applications to share information. This allows users to for example query the VO with one application, visuaklise the results with another and do analysis with a third one.

HiPS

The Hierarchical Progressive Survey (HiPS) protocol defines a way to visualise sky surveys by breaking them up in different hierarchical views which represents different zoom levels. This makes it possible to scroll through a survey in a way that is comparable to using Google Maps (or other online mapping/satellite tools). ASTRONs HiPS collections are all available in Aladin and the most important ones are available through other tools, like ESASky. However users can also have direct access to all our HiPS collections through hips.astron.nl. Typically, HiPS projections are used to visually look through the data from a survey, and combine the coordinates with table-based services to obtain source information or data. The following table shows an overview of the ASTRON HiPS server contains the following data collections.

The ObsCore table

One of the tables in vo.astron.nl is called ivoa.obscore . This table is a special table which follows the ObsCore standard defined by the IVOA and it contains key observational information describing all the data products (images, cubes, etc) in our VO service. Currently, there is not service to directly query this table, but it can be done using ADQL.

List of vo.astron.nl services

The data published in the VO can also be accessed using a web browser at https://vo.astron.nl. The web interface can be used to perform simple queries. The power of the VO lies in the fact that several applications exist that can interact with the standards we offer. In the architecture, there is a distinction between services and tables. A service is an entity that takes user input and provides a table as output. The tables themselves are the entities that actually hold the data. Tables can also be queried directly, for instance using ADQL. The colomns of the table are:

This web interface provides a page on which all the collections present in the registry are list

Data access through DaCHS interface

The DaCHS system comes with a web view on the services offered. Even though the interface is somewhat rudimentary, it does offer the relevant functionality to query data in the ASTRON VO. The main page, listing all services (as listed in the table above) can be fount at https://vo.astron.nl . Specific services are linked to that (either by clicking on the name of the collection, or the Q icon next to it.

Selecting a data collection allows the user to perform a cone search through a webform (Fig. 1). The result is either a catalogue (list of sources, Gaussians or visibilities), or a table of data products (images, cubes) of that given class overlapping a given pointing. In some cases, the number query parameters can differ, in which case the field description explains what the data is that can be submitted to them.

Fig. 1 Query search form for continuum images. Click for a bigger image.

The result is a table in the requested output format in which every row corresponds to a data product (Fig. 2). In each row there is a column, Product key, which is a link that allows the user to download the fits file of the image of interest. The column titles should generally be self-descriptive. However, the long human-readable description of the content of each column is a tooltip that will appear when hovering over the column name. The result is a table (like Fig. 2) which consists of a link to the data product, a thumbnail image  that appears when hovering it (both only if querying for a data product) and relevant metadata describing the results. The Quick Plot button on top of the results page can be used to quicikly plot the numerical results (e.g. making an RA/DEC overview of the images or sources). The Send via SAMP button can be used to send the result set to an application using the SAMP protocol. This essentially means that if you have another application open which supports the SAMP protocol (as do both TOPCAT and Aladin), clicking this button will make the table appear in those applications.

In some cases, auxiliary data is available for the primary image/cube results (e.g. raw visibilities, calibration solutions, etc). Those files are connected to the primary result using a DataLink document. The link in the table is generally shown with the name dlmeta. This is essentially a table with links and predefined descriptions. In the first column the link is shown, together with an (in some cases estimated) file size. The second column consists of a human-readable description and the last column shows the relation between the linked dataset and the primary product, using the vocabulary defined for this, making the result machine-readable. An example of a DataLink document is shown in Fig. 3.

One special service is the ADQL query service, shown on Fig 4.. This contains a single box that can be used to provide custom queries using the Astronomical Data Query Language (ADQL), which in essence is a dialect of SQL. Some more information on, and examples of ADQL can be found further below.

Fig. 4 ADQL query form. Click for a bigger image.

Using the DaCHS interface can be useful for users but the advantage of using a standard is that generic tooling can be used to query and access the data. Two very well-known tools are TOPCAT and Aladin. We will give an overview 

Access via TOPCAT

The VO data collections can easily beThe Apertif DR1 data collection tables can be accessed using TOPCAT, an interactive graphical viewer and editor for tabular data. Since the ASTRON VO is regsitered to the so-called registry of registries, TOPCAT can find the relevant services in its query menus. In the VO menu of TOPCAT you can pick the SIA query menu item, which will bring up a form as shown in Fig. 5. When selecting one of the data collections the SIA URL will be automatically filel in. One can then either type coordinates, or an object name and click the "Resolve" button which uses Simbad to obtain coordinates for that obiect. When clicking OK, a table of the data products found will appeared in the main TOPCAT window. Alternatively the table can be saved in various formats and used locally with other programs (e.g. python scripts etc).

Access to the tables (and the observant reader may realise here that the image data is also tabulated) in the ASTRON VO can be done using the TAP standard. The way to do this is very similar to SIA. First select TAP in the VO menu. Then you will see a list of registered TAP services. You can query those by keyword. The most obvious keyword to use to find the ASTRON VO is ASTRON. Click the Find Services and the URL will be filled in the TAP URL box. Clicking the Use Service button brings you to a window that shows a list of all the tables in the data centre and a box in which you can type ADQL. Clicking the Examples button gives you several example queries you can use to find data. In the example in Fig. 6 we are querying for the first 1000 items from tne ivoa.obscore table (which is the table that contains information on all data products in the ASTRON VO).  Running the query will, again, result in a table in the the TOPCAT main window.

TOPCAT and SAMP

If you have found data in the DACHS web overview, and you have TOPCAT open, you can press the Send via SAMP on the results page (see Fig. 2). When you do this, the table will automatically appear in TOPCAT. Click on the new entry as shown by the arrow in Fig. 7. At this point the table browser will open showing the content of the DACHS results). From here any TOPCAT tool can be used for further inspection and analysis of the results.

When making selections in TOPCAT, you can broadcast the resulting table using SAMP. For example Aladin will then read the table and load it in the main interface. To do this, start Aladin and push the button with the antenna indicated in Fig 8. You will see the table appear as one of the layers in Aladin.

Access via Aladin

One of the most popular tools to access data through the VO is Aladin Sky Atlas. There are both a browser and standalone desktop edition. When refering to Aladin in the following, this will apply to the desktop edition. Aladin is a tool that makes it possible to visualise tables and images and perform some analysis on them.

HiPS

Because the ASTRON VO has been registered with a central registry, all HiPS data collections can be visualised through Aladin. However, some of our collections have been "promoted" by the Aladin admins, and they show up at a different place in the menu structure. The promoted collections are shown in the HiPS overview at the top of this page. To bring up a HiPS, either select the Image → Radio entry in the menu (yellow directory icons) where you can see the promoted collections, or the Others → HiPS → astron.nl (purple directory icons). Alternatively you can use the select box to (partially) type the name of the HiPS you want in the background. Fig. 9 shows the HiPS of LoTSS DR1 image, and the list of ASTRON HiPSes and where to find them (please note that for presentational purposes only the ASTRON data is displayed in this screen shot).

Images

Catalogues

Catalogues

The Apertif DR1 VO data collection can also be discovered directly via ALADIN either via simple image access protocol (SIAP) or tabular access protocol (TAP). The examples shown here require the desktop version of ALADIN.

Open ALADIN and on the left panel for SIAP: select Others > SIA2 > astron.nl > The VO @ASTRON SIAP Version 2. Alternatively for TAP select Others > TAP > astron.nl > The VO @ASTRON TAP service (Fig. 49). A pop-up window will open. Click load, and enter a query using the Server selector (Fig. 50) or TAP access with astron.nl/tap (Fig. 51) to select the target of interest for SIAP and TAP respectively.

Fig. 49 ALADIN display panel. Click for a bigger image.

Fig. 50 ALADIN server selector panel for SIAP. Click for a bigger image.

Fig. 51 ALADIN TAP access panel. Click for a bigger image.

After loading, the data collection catalogues can be plotted on the main panel by selecting them first on the right panel (e.g. highlighted in blue in Figures 14 and 15) and then by selecting the regions of interest on the bottom panel as shown in Figures 14 and 15. From here the usual functionality of ALADIN can be used.

HiPS

Fig. 52 Example of data collection selection via SIAP in ALADIN. Click for a bigger image.

Fig. 53 Example of data collection selection via TAP in ALADIN. Click for a bigger image.

Images

Downloading images or cubes in ALADIN is also possible (see Fig. 54). The user will need to click on the url-link in the column access_url of the bottom panel. Then, once the image is loaded, click on the right panel as shown in Fig. 54. From here the usual functionality of ALADIN can be used.

Fig. 54 Example of image selected from the Apertif DR1 displayed in ALADIN. Click for a bigger image.

ADQL

Simple examples

Export machine readable table

There are multiple ways to export a catalog of the various data products of the data release. On the vo.astron.nl pages, the results of a query can be exported to a csv file or fits table; running an empty query with a table limit of 5000 or more will return all entries.

TOPCAT and the pyvo interface demonstrated above also provide functionality for exporting machine-readable files.

The ADQL form is another option, and below we provide an example query that also provides information about the calibrators used for each beam. This query is specific to the continuum_images data product but can be adapted to other (beam-based, processed) data products by replacing the table name, e.g., for polarization cubes/images use pol_cubes (see Table 10 for a full list of the available tables).

select data.*,
flux_cal.obsid as flux_calibrator_obs_id,
pol_cal.obsid as pol_calibrator_obs_id  from apertif_dr1.continuum_images data
join  apertif_dr1.flux_cal_visibilities flux_cal on data.obsid=flux_cal.used_for and data.beam_number=flux_cal.beam
join apertif_dr1.pol_cal_visibilities pol_cal on data.obsid=pol_cal.used_for and data.beam_number=pol_cal.beam
order by obsid

Python access

The data collection and the table content can be accessed directly via python using the pyvo tool. Working directly in python the tables and the data products can be simply queried and outputs can be customized according to the user’s needs, without the involvement of TOPCAT or ALADIN.

An example of a TAP query and image download can be found in the python script below (it has been tested for python 3.7). The result of the query can also be plotted using python.

#To start you have to import the library pyvo (it is also possible to use astroquery if you want)
import pyvo

## To perform a TAP query you have to connect to the service first
tap_service = pyvo.dal.TAPService('https://vo.astron.nl/__system__/tap/run/tap')

# This works also for
form pyvo.registry.regtap import ivoid2service
vo_tap_service = ivoid2service('ivo://astron.nl/tap')[0]

# The TAPService object provides some introspection that allow you to check the various tables and their
# description for example to print the available tables you can execute
print('Tables present on http://vo.astron.nl')
for table in tap_service.tables:
 print(table.name)
print('-' * 10 + '\n' * 3)

# or get the column names
print('Available columns in apertif_dr1.continuum_images')
print(tap_service.tables['apertif_dr1.continuum_images'].columns)
print('-' * 10 + '\n' * 3)

## You can obviously perform tap queries accross the whole tap service as an example a cone search
print('Performing TAP query')
result = tap_service.search(
 "SELECT TOP 5 target, beam_number, accref, centeralpha, centerdelta, obsid, DISTANCE(" \
     "POINT('ICRS', centeralpha, centerdelta),"\
     "POINT('ICRS', 208.36, 52.36)) AS dist"\
 " FROM apertif_dr1.continuum_images"  \
 " WHERE 1=CONTAINS("
 "    POINT('ICRS', centeralpha, centerdelta),"\
 "    CIRCLE('ICRS', 208.36, 52.36, 0.08333333)) "\
 " ORDER BY dist ASC"
 )
print(result)

# The result can also be obtained as an astropy table
astropy_table = result.to_table()
print('-' * 10 + '\n' * 3)

## You can also download and plot the image
import astropy.io.fits as fits
from astropy.wcs import WCS
import matplotlib.pyplot as plt
import requests, os
import numpy as np

# DOWNLOAD only the first result
#
print('Downloading only the first result')
file_name = '{}_{}_{}.fits'.format(
 result[0]['obsid'].decode(),
 result[0]['target'].decode(),
 result[0]['beam_number'])
path = os.path.join(os.getcwd(), file_name)
http_result = requests.get(result[0]['accref'].decode())
print('Downloading file in', path)
with open(file_name, 'wb') as fout:
 for content in http_result.iter_content():
     fout.write(content)
hdu = fits.open(file_name)[0]
wcs = WCS(hdu.header)
# dropping unnecessary axes
wcs = wcs.dropaxis(2).dropaxis(2)
plt.subplot(projection=wcs)
plt.imshow(hdu.data[0, 0, :, :], vmax=0.0005)
plt.xlabel('RA')
plt.ylabel('DEC')
plt.show()