Astroquery: Alma.query_region can't find some projects

[mjastro]

Hi,

I guess I am currently using up-to-date the astroquery version (0.4.2.dev0), and I find a little difficulty in finding specific ALMA project that observed a target of interest using astroquery. I could find a project from the latest ALMA archive on the web, with the following searching information of the position and searching radius:
356.6203971 12.7926505, 0.3
which returns me one ALMA project: 2013.1.00208.S

But, if use Alma.query_region :

pos = coordinates.SkyCoord('356.6203971  12.7926505',frame='fk5',unit=(u.deg,u.deg))
Alma.query_region(pos,radius=0.3*u.arcmin,public=True)

which gives a huge list of observations, which does not include the project '2013.1.00208.S'.

Can you please help and teach me whether I am using an improper searching method, and if so, what should I change?

Thank you in advance for your response and help!

[keflavich]

Thanks for the report @mjastro. ALMA just changed their archives dramatically last week, and we pushed out a new version to meet their new interface, but there do appear to be bugs.

@andamian can you check this out too?

[andamian]

Looking into it now.

[privong]

I've also been having ALMA archive query issues. But in contrast to @mjastro, issuing the same query (or an analogous one for my source coordinates) fails completely (below I show the same test as @mjastro):

The first invocation returns:

ValueError                                Traceback (most recent call last)
<ipython-input-6-44b3e54079e0> in <module>
      1 pos = coordinates.SkyCoord('356.6203971  12.7926505',frame='fk5',unit=(u.deg,u.deg))
----> 2 Alma.query_region(pos,radius=0.3*u.arcmin,public=True)

/usr/lib/python3.8/site-packages/astroquery/utils/class_or_instance.py in f(*args, **kwds)
     23         def f(*args, **kwds):
     24             if obj is not None:
---> 25                 return self.fn(obj, *args, **kwds)
     26             else:
     27                 return self.fn(cls, *args, **kwds)

/usr/lib/python3.8/site-packages/astroquery/utils/process_asyncs.py in newmethod(self, *args, **kwargs)
     24             verbose = kwargs.pop('verbose', False)
     25 
---> 26             response = getattr(self, async_method_name)(*args, **kwargs)
     27             if kwargs.get('get_query_payload') or kwargs.get('field_help'):
     28                 return response

/usr/lib/python3.8/site-packages/astroquery/alma/core.py in query_region_async(self, coordinate, radius, cache, public, science, payload, **kwargs)
     99         payload.update({'ra_dec': rdc})
    100 
--> 101         return self.query_async(payload, cache=cache, public=public,
    102                                 science=science, **kwargs)
    103 

/usr/lib/python3.8/site-packages/astroquery/alma/core.py in query_async(self, payload, cache, public, science, max_retries, get_html_version, get_query_payload, **kwargs)
    139             payload['science_observations'] = '=%TARGET%'
    140 
--> 141         self.validate_query(payload)
    142 
    143         if get_query_payload:

/usr/lib/python3.8/site-packages/astroquery/alma/core.py in validate_query(self, payload, cache)
    196 
    197         # Check that the keywords specified are allowed
--> 198         self._validate_payload(payload)
    199 
    200         vurl = self._get_dataarchive_url() + '/aq/validate'

/usr/lib/python3.8/site-packages/astroquery/alma/core.py in _validate_payload(self, payload)
    959                                   for row in section]
    960             if len(self._valid_params) == 0:
--> 961                 raise ValueError("The query validation failed for unknown "
    962                                  "reasons.  Try again?")
    963             # These parameters are entirely hidden, but Felix says they are

ValueError: The query validation failed for unknown reasons.  Try again?

And repeating the query gives:

InvalidQueryError                         Traceback (most recent call last)
<ipython-input-7-44b3e54079e0> in <module>
      1 pos = coordinates.SkyCoord('356.6203971  12.7926505',frame='fk5',unit=(u.deg,u.deg))
----> 2 Alma.query_region(pos,radius=0.3*u.arcmin,public=True)

/usr/lib/python3.8/site-packages/astroquery/utils/class_or_instance.py in f(*args, **kwds)
     23         def f(*args, **kwds):
     24             if obj is not None:
---> 25                 return self.fn(obj, *args, **kwds)
     26             else:
     27                 return self.fn(cls, *args, **kwds)

/usr/lib/python3.8/site-packages/astroquery/utils/process_asyncs.py in newmethod(self, *args, **kwargs)
     24             verbose = kwargs.pop('verbose', False)
     25 
---> 26             response = getattr(self, async_method_name)(*args, **kwargs)
     27             if kwargs.get('get_query_payload') or kwargs.get('field_help'):
     28                 return response

/usr/lib/python3.8/site-packages/astroquery/alma/core.py in query_region_async(self, coordinate, radius, cache, public, science, payload, **kwargs)
     99         payload.update({'ra_dec': rdc})
    100 
--> 101         return self.query_async(payload, cache=cache, public=public,
    102                                 science=science, **kwargs)
    103 

/usr/lib/python3.8/site-packages/astroquery/alma/core.py in query_async(self, payload, cache, public, science, max_retries, get_html_version, get_query_payload, **kwargs)
    139             payload['science_observations'] = '=%TARGET%'
    140 
--> 141         self.validate_query(payload)
    142 
    143         if get_query_payload:

/usr/lib/python3.8/site-packages/astroquery/alma/core.py in validate_query(self, payload, cache)
    196 
    197         # Check that the keywords specified are allowed
--> 198         self._validate_payload(payload)
    199 
    200         vurl = self._get_dataarchive_url() + '/aq/validate'

/usr/lib/python3.8/site-packages/astroquery/alma/core.py in _validate_payload(self, payload)
    968         invalid_params = [k for k in payload if k not in self._valid_params]
    969         if len(invalid_params) > 0:
--> 970             raise InvalidQueryError("The following parameters are not accepted"
    971                                     " by the ALMA query service:"
    972                                     " {0}".format(invalid_params))

InvalidQueryError: The following parameters are not accepted by the ALMA query service: ['ra_dec', 'result_view', 'format', 'download', 'public_data', 'science_observations']

I installed astroquery with: pip install astroquery==0.4.2.dev6278

[keflavich]

@privong you're two versions out of date; you should have 0.4.2.dev6280 . I'm pretty sure that's the cause of your issue.

[privong]

@keflavich okay. not sure how i got two versions old since I just updated it before posting. But i upgraded to dev6280 and am now getting the same issue as @mjastro reported initially (many results in excess of what the web reports).

[andamian]

astroquery.alma is using the ALMA TAP service underneath and I can confirm there's a problem with the spatial querying on that system.

While we are fixing it, you could get access to the data by issuing a direct TAP query (using query_tap method). The query that works is: select * from ivoa.obscore WHERE s_ra < 356.62 + 0.0005 and s_ra > 356.62-0.0005 AND s_dec < 12.7927 +0.0005 AND s_dec > 12.7927 - 0.0005 AND calib_level>1 AND data_rights='Public'.

[andamian]

@mjastro & @privong - I've informed ALMA about the problem and I'm waiting to hear from them on the resolution.

[andamian]

@mjastro & @privong - the issue is still open with ALMA TAP services. We've logged a ticket with them from the beginning. Since this is not a problem with astroquery.alma I'm going to close the ticket but I suggest you open your own tickets with the ALMA help desk (https://help.almascience.org). The problem is that their TAP service does not work properly with spatial constraints. This can be easily reproduced with the following command:
curl -L -d "LANG=ADQL&QUERY=select s_region from ivoa.obscore WHERE (INTERSECTS(CIRCLE('ICRS',356.6203971,12.7926505,0.005), s_region) = 1) AND calib_level>1 AND data_rights='Public'" https://almascience.nrao.edu/tap/sync

[privong]

@andamian Earlier today I had put some comments/questions here but deleted them to avoid confusion to anyone reading this later. I'm at the NAASC and have followed up on this with local (Charlottesville) folks.

[andamian]

@privong - sorry for the delay, I just returned from vacation. The ticket in question is https://ictjira.alma.cl/browse/ICT-17281.

[privong]

@andamian No worries. Thanks for following up. I'm "watching" the ticket in JIRA, but not sure there's much I can do.

[andamian]

@mjastro & @privong : ALMA has fixed their TAP services. Now this works:

>>> pos = coordinates.SkyCoord('356.6203971  12.7926505',frame='fk5',unit=(u.deg,u.deg))
>>> result = Alma.query_region(pos,radius=0.3*u.arcmin,public=True)
>>> result['proposal_id']
<MaskedColumn name='proposal_id' dtype='bytes64' description='Identifier of proposal to which NO observation belongs.' length=4>
2013.1.00208.S
2013.1.00208.S
2013.1.00208.S
2013.1.00208.S

Sorry for the delay and please let me know if this does not solve the problem to your satisfaction.