Astroquery: bytes/str issue for heasarc.query_region() in python3.6
I'm using latest builds on python 3.6:
> print(astropy.__version__, astroquery.__version__)
4.1.dev27307 0.4
And when querying a heasarc catalog in a region with no survey data, I get this:
>>> from astropy.coordinates import SkyCoord
>>> import astropy.units as u
>>> from astroquery import heasarc
>>> he = heasarc.Heasarc()
>>> coo = SkyCoord(ra=0*u.deg, dec=-90*u.deg)
>>> he.query_region(coo, 'first', radius='1 degree')
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
~/anaconda3/envs/py36/lib/python3.6/site-packages/astroquery/heasarc/core.py in _parse_result(self, response, verbose)
223 data = BytesIO(response.content)
--> 224 table = Table.read(data, hdu=1)
225 return table
~/anaconda3/envs/py36/lib/python3.6/site-packages/astropy-4.1.dev27307-py3.6-macosx-10.7-x86_64.egg/astropy/table/connect.py in __call__(self, *args, **kwargs)
58
---> 59 out = registry.read(cls, *args, **kwargs)
60
~/anaconda3/envs/py36/lib/python3.6/site-packages/astropy-4.1.dev27307-py3.6-macosx-10.7-x86_64.egg/astropy/io/registry.py in read(cls, format, *args, **kwargs)
522 reader = get_reader(format, cls)
--> 523 data = reader(*args, **kwargs)
524
~/anaconda3/envs/py36/lib/python3.6/site-packages/astropy-4.1.dev27307-py3.6-macosx-10.7-x86_64.egg/astropy/io/fits/connect.py in read_table_fits(input, hdu, astropy_native, memmap, character_as_bytes)
215 return read_table_fits(hdulist, hdu=hdu,
--> 216 astropy_native=astropy_native)
217 finally:
~/anaconda3/envs/py36/lib/python3.6/site-packages/astropy-4.1.dev27307-py3.6-macosx-10.7-x86_64.egg/astropy/io/fits/connect.py in read_table_fits(input, hdu, astropy_native, memmap, character_as_bytes)
202 else:
--> 203 raise ValueError("No table found")
204
ValueError: No table found
During handling of the above exception, another exception occurred:
TypeError Traceback (most recent call last)
<ipython-input-10-7906b1189339> in <module>
----> 1 tab = he.query_region(coo, 'first', radius='1 degree')
~/anaconda3/envs/py36/lib/python3.6/site-packages/astroquery/utils/class_or_instance.py in f(*args, **kwds)
23 def f(*args, **kwds):
24 if obj is not None:
---> 25 return self.fn(obj, *args, **kwds)
26 else:
27 return self.fn(cls, *args, **kwds)
~/anaconda3/envs/py36/lib/python3.6/site-packages/astroquery/utils/process_asyncs.py in newmethod(self, *args, **kwargs)
27 if kwargs.get('get_query_payload') or kwargs.get('field_help'):
28 return response
---> 29 result = self._parse_result(response, verbose=verbose)
30 self.table = result
31 return result
~/anaconda3/envs/py36/lib/python3.6/site-packages/astroquery/heasarc/core.py in _parse_result(self, response, verbose)
225 return table
226 except ValueError:
--> 227 return self._fallback(response.content)
228
229 def _args_to_payload(self, **kwargs):
~/anaconda3/envs/py36/lib/python3.6/site-packages/astroquery/heasarc/core.py in _fallback(self, content)
190 new_table = []
191
--> 192 old_table = content.split("END")[-1].strip()
193 for line in old_table.split("\n"):
194 newline = []
TypeError: a bytes-like object is required, not 'str'
Setting the coordinate to a region that know has data returns data as expected.
caseyjlaw
All 12 comments
This might be fixed by astropy/astropy#9505
pllim
on 5 Mar 2020
I still see this error, and arguably I'm on top of astropy master. Also, edited the code snippet so it's a full copy paste-able MFE.
bsipocz
on 5 Mar 2020
Okay, I don't think this is related to the astropy PR. The offending line is
It needs to be
old_table = content.split(b"END")[-1].strip()
because content is bytes. The code got here in _fallback because this failed
Maybe I am just seeing the symptoms. I wouldn't know which is the real bug to fix.
pllim
on 6 Mar 2020
I'd like to give this a bump, as the bug renders HEASARC Queries useless. Most of the examples given on https://astroquery.readthedocs.io/en/latest/heasarc/heasarc.html no longer work. For example, running:
from astroquery.heasarc import Heasarc
heasarc = Heasarc()
mission = 'rospublic'
object_name = '3c273'
table = heasarc.query_object(object_name, mission=mission)
gives me:
Traceback (most recent call last):
File "test.py", line 5, in <module>
table = heasarc.query_object(object_name, mission=mission)
File "/Users/efrembraun/anaconda3/lib/python3.7/site-packages/astroquery/utils/class_or_instance.py", line 25, in f
return self.fn(obj, *args, **kwds)
File "/Users/efrembraun/anaconda3/lib/python3.7/site-packages/astroquery/utils/process_asyncs.py", line 29, in newmethod
result = self._parse_result(response, verbose=verbose)
File "/Users/efrembraun/anaconda3/lib/python3.7/site-packages/astroquery/heasarc/core.py", line 224, in _parse_result
table = Table.read(data, hdu=1)
File "/Users/efrembraun/anaconda3/lib/python3.7/site-packages/astropy/table/connect.py", line 52, in __call__
out = registry.read(cls, *args, **kwargs)
File "/Users/efrembraun/anaconda3/lib/python3.7/site-packages/astropy/io/registry.py", line 520, in read
'read', cls, path, fileobj, args, kwargs)
File "/Users/efrembraun/anaconda3/lib/python3.7/site-packages/astropy/io/registry.py", line 603, in _get_valid_format
"{}".format(format_table_str))
astropy.io.registry.IORegistryError: Format could not be identified based on the file name or contents, please provide a 'format' argument.
The available formats are:
Format Read Write Auto-identify Deprecated
--------------------------- ---- ----- ------------- ----------
ascii Yes Yes No
ascii.aastex Yes Yes No
ascii.basic Yes Yes No
ascii.cds Yes No No
ascii.commented_header Yes Yes No
ascii.csv Yes Yes Yes
ascii.daophot Yes No No
ascii.ecsv Yes Yes Yes
ascii.fast_basic Yes Yes No
ascii.fast_commented_header Yes Yes No
ascii.fast_csv Yes Yes No
ascii.fast_no_header Yes Yes No
ascii.fast_rdb Yes Yes No
ascii.fast_tab Yes Yes No
ascii.fixed_width Yes Yes No
ascii.fixed_width_no_header Yes Yes No
ascii.fixed_width_two_line Yes Yes No
ascii.html Yes Yes Yes
ascii.ipac Yes Yes No
ascii.latex Yes Yes Yes
ascii.no_header Yes Yes No
ascii.rdb Yes Yes Yes
ascii.rst Yes Yes No
ascii.sextractor Yes No No
ascii.tab Yes Yes No
asdf Yes Yes Yes
fits Yes Yes Yes
hdf5 Yes Yes Yes
pandas.csv Yes Yes No
pandas.fwf Yes No No
pandas.html Yes Yes No
pandas.json Yes Yes No
votable Yes Yes Yes
aastex Yes Yes No Yes
cds Yes No No Yes
csv Yes Yes No Yes
daophot Yes No No Yes
html Yes Yes No Yes
ipac Yes Yes No Yes
latex Yes Yes No Yes
rdb Yes Yes No Yes
I'm using Python 3.7.6, AstroPy 4.0.1.post1, and AstroQuery 0.4.1.
EfremBraun
on 25 Sep 2020
I have exactly the same problem with @EfremBraun
rhuang-astro
on 26 Sep 2020
This problem is deeper than identified by @pllim, unfortunately. It looks like the upstream API has changed; these queries are returning things like:
HEASARC_FIRST : NO MATCHING ROWS
so either the queries were blank and the error message is misleading, or the query was poorly specified. I'm investigating further.
keflavich
on 27 Sep 2020
@EfremBraun your problem is that rospublic no longer exists. I'm not sure what it was originally pointing to, but maybe rospspc. However, correcting this problem led to another: the returned catalog appears malformatted. Errors like this:
ValueError: Invalid format for column 'COUNT_RATE': could not display values in this column using this format (Unable to parse format string "{:10.4f}" for entry " 4.5660 " in column "COUNT_RATE")
This looks like a real error in the returned data.
keflavich
on 27 Sep 2020
@caseyjlaw your error is that there is no FIRST data at dec=-90, which I suspect you knew but you were using it as a MWE? This query works for me:
>>> from astropy.coordinates import SkyCoord
>>> import astropy.units as u
>>> from astroquery import heasarc
>>> he = heasarc.Heasarc()
>>> coo = SkyCoord(ra=12*u.h, dec=50*u.deg)
>>> result = he.query_region(coo, 'first', radius='1 degree')
but I'll add a better error message
keflavich
on 27 Sep 2020
@EfremBraun: The HEASARC catalog rospublic has been deprecated for 16 years. :) We (the HEASARC) recently switched to a new database system, and it's not in the new database. You really should be using the HEASARC catalog rosmaster instead.
esabol
on 3 Oct 2020
Maybe someone should modify the examples given on https://astroquery.readthedocs.io/en/latest/heasarc/heasarc.html.
Most of them are rospublic.
rhuang-astro
on 3 Oct 2020
@EfremBraun: The HEASARC catalog
rospublichas been deprecated for 16 years. :) We (the HEASARC) recently switched to a new database system, and it's not in the new database. You really should be using the HEASARC catalogrosmasterinstead.
That indeed fixes it. I'll fix the examples in the docs now.
EfremBraun
on 5 Oct 2020
I believe all the issues here have been addressed and fixed therefore I'm closing the issue. If you find it otherwise please reopen.
bsipocz
on 5 Oct 2020
Related issues
StuartLittlefair
路
9Comments
Venustiano
路
10Comments
lupinix
路
5Comments
esabol
路
6Comments
prajwel
路
4Comments