Astroquery: (403) Forbidden Returned from astroquery.mast.Observations

[exowanderer]

Using the AWS tutorial on https://mast-labs.stsci.io/ -- but for the entire WFC3/IR Image database -- I ran into hundreds of "(403) Forbidden" errors when accessing

s3_urls = Observations.get_hst_s3_uris(filtered), which is just a loop over s3_urls = Observations.get_hst_s3_uri(filtered)

When I looked through the code, I found that the error was generated in

$HOME/anaconda3/lib/python3.6/site-packages/astroquery-0.3.9.dev4981-py3.6.egg/astroquery/mast/core.py

then inside get_hst_s3_uri(..), on line 1372 the code reads
s3_client.head_object(Bucket=self._hst_bucket, Key=path, RequestPayer='requester')

This is where the error "(403) Forbidden" is being generated.

It only generated the error 2213 out of 60945 image requests; but I am not sure this is a reasonable error or a typo somewhere along the chain.

The code that I used is below (it takes about 2 hours to process):

---

from astroquery.mast import Observations
from glob import glob
from pyql.file_system.make_fits_file_dict import make_fits_file_dict
from pyql.ingest.make_jpeg import make_jpeg
from tqdm import tqdm

import boto3
import numpy as np
import os

WFC3IR_Filters = ['F105W', 'F110W', 'F125W', 'F140W', 'F160W', 'F098M', 'F127M', 'F139M', 'F153M', 
                    'F126N', 'F128N', 'F130N', 'F132N', 'F164N', 'F167N']

# Enable 'S3 mode' for module which will return S3-like URLs for FITs files
Observations.enable_s3_hst_dataset()

s3 = boto3.resource('s3')

# Create an authenticated S3 session. Note, download within US-East is free
# e.g. to a node on EC2.
s3_client = boto3.client('s3',
                         aws_access_key_id=os.environ['AWS_ACCESS_KEY_ID'],
                         aws_secret_access_key=os.environ['AWS_SECRET_ACCESS_KEY'])

bucket = s3.Bucket('stpubdata')

obsTable_dict = {ir_filt:Observations.query_criteria(obs_collection='HST', 
                                                        instrument_name='WFC3/IR', 
                                                        filters=ir_filt) 
                                                        for ir_filt in tqdm(WFC3IR_Filters)}

# Need to chunk the input data because the `Observations.get_product_list` command crashes 
#   with an error related to either json or DataFrames -- neither are diagnosable.
chunk_size = 1000
product_dict_full = {}
for ir_filt in tqdm(WFC3IR_Filters):
    n_obs = len(obsTable_dict[ir_filt])
    n_chunks = n_obs // chunk_size +1
    product_dict_full[ir_filt] = []
    for k in tqdm(range(n_chunks)):
        try:
            product_dict_full[ir_filt].append(Observations.get_product_list(obsTable_dict[ir_filt][k*chunk_size:(k+1)*chunk_size]))
        except Exception as e:
            print(str(e))

# Select only FLT files
# mrp = minimum recommended products
filtered_dict = {}
for ir_filt in tqdm(WFC3IR_Filters):
    filtered_dict[ir_filt] = []
    for product_tbl in product_dict_full[ir_filt]:
        filtered_dict[ir_filt].append(Observations.filter_products(product_tbl, 
                                        mrp_only=False, productSubGroupDescription='FLT', 
                                        dataproduct_type='image'))

# Grab the S3 URLs for each of the observations
s3_urls_dict = {}
for ir_filt in tqdm(WFC3IR_Filters):
    s3_urls_dict[ir_filt] = []
    for kf, filtered_tbl in tqdm(enumerate(filtered_dict[ir_filt]), total=len(filtered_dict[ir_filt])):
        try:
            s3_urls_dict[ir_filt].append(Observations.get_hst_s3_uris(filtered_tbl))
        except Exception as e:
            s3_urls_dict[ir_filt].append([kf, str(e)])

[bsipocz]

I'm cc-ing @ceb8, but honestly a forbidden response seems to be beyond astroquery and being a server side issue instead. Also, if possible please try provide a minimal examples, needing to run tests for 2 hours just to try to replicate a possible problem is not ideal.

[bsipocz]

something related: @ceb8 do you think it's feasible to introduce a get_query_payload like argument that provides a query text that can be submitted on the webform? Don't worry about it if there is no one-on-one webform equivalent, it's just sometime a good debugging functionality.

[ceb8]

Cc-ing @cam72cam as this looks like it has to do with the AWS connection and he is more familiar with that part.

[mfox22]

Can you provide specific examples of the image requests that failed from the set of 2213? Do these fail continually, or is the 403 intermittent?

[exowanderer]

I talked with @ivastar about this in person and she mentioned that the astroquery.mast would also have access to the proprietary MAST holdings; but that they are not yet up on the AWS stpubdata s3. Because there are ~2000 images out of ~60000 queries, this makes a lot of sense. We are pretty sure that the 2213 403 error failures are actually the proprietary images that have yet to be uploaded to MAST.

[exowanderer]

I should say that I will confirm that directly by selecting the exact name of the image from one of the astroquery 403 errored results and then look it up on MAST to see if its proprietary.

[cam72cam]

@exowanderer any updates? The way you described the issue it seems that the 403 was coming from AWS calls. This should not happen as far as I am aware. Could you post a entry that gave a 403 so we can debug it?

As far as proprietary goes, we do not currently have any plans on putting proprietary data in stpubdata.

Would you mind re-opening the ticket?

[exowanderer]

Hello. I had to loop over the whole data set in order to find one of the "broken" files (403 Error).

Below is a succinct script that will reproduce the error.

I checked for this program (14680) at https://archive.sts sure ci.edu/proposal_search.php?mission=hst&id=14680

I think that these files are the yellow flagged entries (id6l03) are propriety, which is what @ivastar suggested early on.

It's highly likely that the files I cannot access are all proprietary (2200 out of 61000 images).

[Note: In case someone finds this Issue page a year from now, it's highly likely that the currently proprietary data will become public later. So we would need to re-loop over the products_list for an entry that breaks; i.e. 357 will not flag an error soon or later]

---

from astroquery.mast import Observations
from termcolor import colored

import boto3
import os

# Enable 'S3 mode' for module which will return S3-like URLs for FITs files
Observations.enable_s3_hst_dataset()

s3 = boto3.resource('s3')

# Create an authenticated S3 session. Note, download within US-East is free
# e.g. to a node on EC2.
s3_client = boto3.client('s3',
                         aws_access_key_id=os.environ['AWS_ACCESS_KEY_ID'],
                         aws_secret_access_key=os.environ['AWS_SECRET_ACCESS_KEY'])

bucket = s3.Bucket('stpubdata')

obsTable = Observations.query_criteria(obs_collection='HST', 
                                                        instrument_name='WFC3/IR', 
                                                        filters='F167N')

product_list = Observations.get_product_list(obsTable)

filtered_product = Observations.filter_products(product_list, 
                                                mrp_only=False, 
                                                productSubGroupDescription='FLT', 
                                                dataproduct_type='image')
'''
# This loop will break somewhere, the last used `k_broken` should be stored as `known_2b_broken` later
for k_broken, filt_prod in enumerate(filtered_product): Observations.get_hst_s3_uri(filtered_product[k_broken])
'''

''' I looped over the entire filtered_product list and saved the index per entry
        and stored when it flagged the error: 

            An error occurred (403) when calling the HeadObject operation: Forbidden

    That happened at index 356, which is stored below
'''

known_2b_broken = 357 # k_broken from loop above
print('\x1b[32m\n-------------------------------\nThis is the product list entry:\x1b[0m\n')
for colname in product_list[known_2b_broken].columns: 
    print('\x1b[34m{:27}\x1b[31m: {}\x1b[0m'.format(colname, product_list[known_2b_broken][colname]))

print('\x1b[32m\n-----------------------------------\nThis is the filtered product table:\x1b[0m\n')
for colname in filtered_product[known_2b_broken].columns: 
    print('\x1b[34m{:27}\x1b[31m: {}\x1b[0m'.format(colname, filtered_product[known_2b_broken][colname]))

print('\x1b[32m\n-----------------------------\nThis will generate the error:\x1b[0m')

try:
    Observations.get_hst_s3_uri(filtered_product[known_2b_broken])
except Exception as e:
    print('\n\x1b[1m{}'.format(e))

[pllim]

If you are forbidden to access them, then I guess "Forbidden" is indeed the appropriate error message?

[cam72cam]

It looks like a permission issue in our s3 bucket may be causing some issues. @mfox22 is looking at our roles and rules.

[exowanderer]

Interesting. I am still under the impression that these are just proprietary objects; such that (quoting @ivastar) astroquery can access the existence of this data; but that it is not yet publicly available on AWS.

@cam72cam: does this solve the question? The data exists in astroquery, but not in the s3 bucket.

[mfox22]

I am not looking at any bucket permission issues related this this ticket. As far as I can tell from this ticket @exowanderer discovered that the failed requests are for proprietary data. If that is not the case then please list some products that are failing that you believe should not be. Thanks.

[exowanderer]

This is correct. From my knowledge, the answer to the original question is that astroquery is accessing proprietary root names that do not exist on the AWS HST Public Data bucket (thanks to @ivastar). So, to me, the issue is solved.

[ivastar]

What does astroquery.mast.Observations.get_hst_s3_uri return for proprietary data? Does it still construct the expected path? What should it return? None/False? @cam72cam

[cam72cam]

The get_hst_s3_uri should raise a custom exception when a file is not able to be looked up. We have to spider parts of S3 to try to find a file because our filenames are not always consistent with our directory names.

I still don't see how a HEAD request could be returning a 403 in that case.

Code:
https://github.com/astropy/astroquery/blob/master/astroquery/mast/core.py#L1326-L1382