Astroquery: Executing multiple alma.stage_data() commands sometimes fails with HTTPError: 500

[alipnick]

When attempting to execute multiple alma.stage_data() commands, instances after the first call often fail with an HTTPError: 500 for an unknown (or not obvious to me) reason (see ipython session below).

In [1]: from astroquery.alma import Alma as alma

In [2]: alma.login('alipnick')
WARNING: No password was found in the keychain for the provided username. [astroquery.query]
alipnick, enter your password:

INFO: Authenticating alipnick on asa.alma.cl ... [astroquery.alma.core]
INFO: Authentication successful! [astroquery.alma.core]

In [3]: link_list = alma.stage_data('uid://A001/X131b/X57')
INFO: Staging files... [astroquery.alma.core]
.
In [4]: link_list = alma.stage_data('uid://A001/X131b/X57')
INFO: Staging files... [astroquery.alma.core]
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
<ipython-input-5-44c9337454d6> in <module>()
----> 1 link_list = alma.stage_data('uid://A001/X131b/X57')

/home/da_data/lava/lib/python2.7/site-packages/astroquery/alma/core.pyc in stage_data(self, uids)
    331         staging_submission = self._request('GET', submission_url, cache=True)
    332         self._staging_log['staging_submission'] = staging_submission
--> 333         staging_submission.raise_for_status()
    334 
    335         data_page_url = staging_submission.url

/home/da_data/lava/lib/python2.7/site-packages/requests/models.pyc in raise_for_status(self)
    937 
    938         if http_error_msg:
--> 939             raise HTTPError(http_error_msg, response=self)
    940 
    941     def close(self):

HTTPError: 500 Server Error: 500 for url: https://almascience.nrao.edu/rh/error?errorMessage=java.lang.IllegalArgumentException%3A+Invalid+UUID+string%3A+alipnick

This seems to happen intermittently, as sometimes I have had this not cause a failure which brings up another issue with alma.stage_data() where changes to the alma.archive_url variable are ignored by alma.stage_data() (I'll open a different ticket for that).

[pllim]

Error code 500 usually indicate server side problem, which I don't think is something astroquery itself can fix. Did you try to do a time.sleep(1) or something, so you don't appear as a spammer to the server?

[alipnick]

I just attempted it again with my currently open session (it's been about 5 minutes) and it has still failed with HTTPError 500.

[alipnick]

More weirdness, even though the command fails, I still receive an email from the archive alerting me of a successful query.

[alipnick]

Is #434 / #435 a related issue?

[teuben]

I confirm the same error, for me this happens always on a 2nd staging for private data. It doesn't happen with public data, viz.

    data = alma.query(payload={'project_code':pid},public=False,science=True)
    uids = np.unique(data['Member ous id'])
    r1=re.compile('.*_of_.*tar$')
    for uid in uids:
          link_list = alma.stage_data(uid)                # <--- dies here 2nd time
           url = link_list['URL']
           url1 = [x for x in url if r1.match(x)]
           alma.download_files(url1)

[alipnick]

In fact, another user reported this same error to the ALMA helpdesk a couple months ago. The work-around is to simply supply all the MOUS uids to alma.stage_data() so that you only run a single alma.stage_data() command. However, I don't see why it shouldn't work the way @teuben shows.

[teuben]

I had to do the one by one, because staging the whole list would not work for private data.
When I tried
link_list = alma.stage_data(uids)
instead of one by one, it would sit there forever. I will try be more patient, and define what "forever" is.

[keflavich]

Have you tried staging MOUSes from multiple different (public) programs? I don't think I've ever tested that case, and I'm not convinced it will necessarily work.

Note that you generally do need an independent Session for each stage_data call, i.e., you should do:

alma = Alma()
alma.stage_data(...)

each time you call stage_data. Failure to do this often results in error 504s, if I recall correctly.

[teuben]

The looping example I gave earlier works fine for public data, you can try to run my example
https://github.com/teuben/study7/blob/master/test_alma2
without arguments and it will work. It should download 6 smallish tar files. But the test_alma3 (you may need to edit some True/False and use your own PID/USR example) will only work one tar file at the time in my experience.

Of course I could have mis-interpreted the QA examples

[keflavich]

Interesting. I have definitely had problems trying to stage data from multiple different observations.

However... I believe if you give two MOUSes from the same observation, there's only one server-side operation: you get the whole observation whether you want it or not. So if you're looping over multiple requests for the same parent observation, it might work because it doesn't have to do anything.

[teuben]

but my test_alma2 and test_alma3 are identical in their processing of the UID's, except for dealing with public (2) and non-public (3) data, yet #2 works and #3 fails on the 2nd UID.

Do you mean 'two MOUSes from the same observation' to be same as "two UID's from the same project code' ?

[keflavich]

Do you mean 'two MOUSes from the same observation' to be same as "two UID's from the same project code' ?

yes

[alipnick]

I'd just like to add that I use astroquery every day and use the stage_data command with a list of non-public MOUSs from anywhere between 1 - 40 different projects and have never had an issue [with stage_data grabbing the correct data].

[mtazzari]

I am trying to stage data but I am getting error 500 for, apparently, maintenance server-side.
I paste below commands and full traceback.
Has this been happening only recently or is it a well known issue?
Any way to find when/whether the functionality will be restored?
Thanks!

In [38]: uids
Out[38]:
['uid://A001/X87d/X5db',
 'uid://A001/X87d/X5e3',
 'uid://A001/X87d/X5d7',
 'uid://A001/X87d/X5df',
 'uid://A001/X87d/X5e7']

In [39]: link_stage = Alma.stage_data(uids)
INFO: Staging files... [astroquery.alma.core]
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
<ipython-input-39-23f910a41d94> in <module>
----> 1 link_stage = Alma.stage_data(uids)

~/anaconda/envs/py37/lib/python3.7/site-packages/astroquery/alma/core.py in stage_data(self, uids)
    331         staging_submission = self._request('GET', submission_url, cache=True)
    332         self._staging_log['staging_submission'] = staging_submission
--> 333         staging_submission.raise_for_status()
    334
    335         data_page_url = staging_submission.url

~/anaconda/envs/py37/lib/python3.7/site-packages/requests/models.py in raise_for_status(self)
    938
    939         if http_error_msg:
--> 940             raise HTTPError(http_error_msg, response=self)
    941
    942     def close(self):

HTTPError: 500 Server Error: 500 for url: https://almascience.eso.org/rh/error?errorMessage=ca.nrc.cadc.requesthandler.exceptions.ApplicationException%3A+Request+Submission+is+temporarily+disabled+for+reasons+of+maintenance

[keflavich]

This is news to me - we may have to ask someone in the archive department at ALMA for more information.

[bsipocz]

What happens if you try the other, non eso servers?

[mtazzari]

Thanks Adam for the quick response.
Funnily enough, I get the same error if I try to download the data from the
new ALMA archive!
Ouch...

On Thu, 23 Jan 2020 at 22:00, Adam Ginsburg [e-mail]
wrote:

This is news to me - we may have to ask someone in the archive department
at ALMA for more information.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/astropy/astroquery/issues/1319?email_source=notifications&email_token=ABN5ZATXNMMYYC5BZDXNJUTQ7IHOXA5CNFSM4GL3NPI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJZAHGI#issuecomment-577897369,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABN5ZAVIHZVYPXCTUBWJU7TQ7IHOXANCNFSM4GL3NPIQ
.

[mtazzari]

@bsipocz How can I choose the server?

[bsipocz]

They may redirect you anyway based on your geo, but here is the list: https://astroquery.readthedocs.io/en/latest/_modules/astroquery/alma.html#Conf

[bsipocz]

Funnily enough, I get the same error if I try to download the data from the
new ALMA archive!

yes, that's expected.

[teuben]

Odd, this works for me. I also happen to update to 4.0 a few days ago, and ran my test. Was fine.

So, in my case the NRAO version worked, the ESO version is under maintenance.Just tried that out.

data = alma.query(payload={'project_code':pid},science=True,public=True)
alma.cache_location = pwd
alma.dataarchive_url = 'https://almascience.nrao.edu.org'
alma.dataarchive_url = 'https://almascience.eso.org'

[bsipocz]

Thanks @teuben for confirming the suspicion.

[mtazzari]

Thanks for the insight, I've tried right now with the same commands I used
before and it works just fine.
It is downloading from https://almascience.eso.org/ so the ESO archive must
be working fine again.
Perhaps I've caught the archive just at the wrong time...

On Thu, 23 Jan 2020 at 23:29, Brigitta Sipőcz [e-mail]
wrote:

Thanks @teuben https://github.com/teuben for confirming the suspicion.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/astropy/astroquery/issues/1319?email_source=notifications&email_token=ABN5ZATDZZXRIW67S32NBT3Q7IR4XA5CNFSM4GL3NPI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJZHE5Q#issuecomment-577925750,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABN5ZAUTQY4F3OHPW4YCTKLQ7IR4XANCNFSM4GL3NPIQ
.