Astroquery: CADC

[keflavich]

From Felix:
Adding CADC to astroquery probably would be very efficient:
1) they have loads of collections (including HST, by the way)
2) these collections are all homogenized
3) they already have VO services for them (query and download)

If you implement the use of VO services while doing CADC,

http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/

then you will be able to use many other archives using the exact same
services. Probably you would want to start with ObsCore.

http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/sync?REQUEST=doQuery&LANG=ADQL&QUERY=select+*+from+ivoa.ObsCore+where+INTERSECTS%28s_region%2C+CIRCLE%28%27ICRS%27%2C+180%2C+5%2C+2%29%29+%3D+1

[juandesant]

I endorse Felix (I'd guess I know the surname of this particular Felix 😉) proposal, and specially the support for TAP services, and the ObsCore schema for those that have it. Queries to VO registry can give back TAP services with support for ObsCore.

[keflavich]

We can probably do this with http://pyvo.readthedocs.org/en/latest/pyvo/data_access.html

[caseyjlaw]

I'm interested in playing with VLASS data via the CADC service. I am told that there is progress on integrating CADC with astroquery. Is there anything to report there? Can I help?

[keflavich]

@andamian Do you have anything to add here?

See also https://github.com/astropy/astroquery/pull/913, which should have been linked to this issue originally.

[caseyjlaw]

I can see the basic idea in the cadc-tap library, but it does not look ready enough for me to jump in. If there are any suggestions for pieces I can test or develop, please let me know.

[keflavich]

@caseyjlaw I'd really like to have access to this service, and I see you've gotten a start at a query module. Do you think you could start piecing this into an astroquery module based on the template?

When are you next in Socorro? We could try to hack it together then.

[caseyjlaw]

@keflavich I am told that the CADC integration has been started in this fork: https://github.com/brianmajor/astroquery. And this is distinct from the effort back in 2017.
However, this change is coupled to some TAP changes being accepted by GAIA people. Not sure what that process is like, so it is hard to say whether this is imminent or languishing.

[andamian]

@keflavich, your assesment is correct. This is very much distinct from the 2017. During the development, we found a few issues with the GAIA TAP library which we tried to correct but we were told to hold because the GAIA TAP library is going through a major re-work effort. Hence the forking.
The fork works with the current code so feel free to use and contribute, but please bare in mind that it might need refactoring when ready to merge back into astroquery. We'd like to proceed as soon as possible.

[keflavich]

@caseyjlaw @andamian thanks both for the help.

I would very much like to have a working CADC query tool merged into astroquery. I'd prefer to push this ahead of the GAIA TAP update, since you have something that works now. Of course, this means we'll have our work cut out for us if we have to support two different versions of TAP within astroquery... but I lean toward getting more functionality in and fixing that sort of infrastructure later. Likely we'll need some face-to-face or dedicated remote worktime to get this all fixed...

[ijiraq]

I would also prefer to move ahead with this integration now rather than waiting for the next release of TAP modules. @andamian can the CADC fork merge into the current astroquery in a non disruptive way (i guess) and then we can do this again once the astroquery TAP module is updated?

[keflavich]

for future reference, it would be very helpful to the maintainers (me and @bsipocz) if you could do this development in an open pull request rather than on a branch that we don't get notified about. It's much easier to review pull requests and help push along development if we get to be somewhat involved from the start. It also helps us plan out when we're going to have time to review things... we are horribly backlogged right now.

[andamian]

I didn't directly write that code (just reviewed it), but I was told that patches to GAIA's TAP were not accepted due to the imminent update of that library, so there were a few not-so-elegant workarounds that we had to come up with (e.g overrides of the TAP code). Bottom line, this version is not that nice but it's working and it's better to have something working now. Cleanup will need to occur after GAIA's work. I'll prepare a pull request.

[andamian]

@caseyjlaw - an update: we've realized that the cadc module only included code to search for data. We want to include an easy 'data download' functionality with it as well and a bit of development is required. I estimate a week or two for that. Thanks for your understanding.

[keflavich]

@andamian Could you please update us on the status of CADC module development? Are you working on it? If so, where is the pull request? We very strongly recommend that you submit a work-in-progress pull request prior to its completion if you have one.

[andamian]

@keflavich, created the initial pull request: https://github.com/astropy/astroquery/pull/1354. Not quite ready as I'm dealing with a few other unexpected things. How do I ping you when PR ready? Assign you as a code reviewer?

[keflavich]

Yes, that works, and/or just write a message saying 'ready for review'. Thanks!

[keflavich]

Closed by #1354