Astroquery: Occasional raw HTML codes in NIST results break FixedWidth Table parsing

[DBerke]

Atomic transition searches from NIST occasionally contain footnotes which display information when hovered over. The addition of such symbols to the HTML code returned from a query can break the parsing of the results into a useable Astropy Table object.

Specifically, the results for vanadium I contain a lot of entries where either the upper or lower energy level is marked with a dagger symbol (†), HTML: †. (A specific example is the V I line with a vacuum wavelength of 433.3330 nm, but I found 77 instances over the wavelength range from ~430–625 nm.) The addition of the HTML to the returned text breaks the parsing of the results, as shown below:

Here's the row of the table as it's returned from Nist.query:

Observed Transition       Ei           Ek                   Lower level                         Upper level           
-------- ---------- --------------------------- ------------------------------------ ---------------------------------
 433.333   23076.94 20606.467   -   43683.397&d gger; | 3d3.(4F).4s.4p.(3P*) | z 4D* 1/2  | 3d3.4s.(5F).4d        | 4D

The dagger is being returned in HTML as "†" but it appears that the table is breaking at the wrong point (on the 'a' of 'dagger') due to the use of a FixedWidth reader in the _parse_results method of NistClass. This then throws off the parsing for every column after the third one (the 'Ei Ek' one); the fourth column should be 3d3.(4F).4s.4p.(3P*) | z 4D* 1/2 but is instead gger; | 3d3.(4F).4s.4p.(3P*) | z 4D*.

This problem is particularly insidious because there's no indication to the user that anything has happened until they try to use the data and something breaks, as the parsing produces an output without any warning that something is off.

On an optimistic note, I _think_ it should be possible to fix the issue by using a regular expression to find things in the form of an HTML code (something like &.*;) then stripping out all matches from the raw Response text before parsing it into a Table. I'll try to see if can fix it on my own and submit a pull request.

[bsipocz]

@DBerke - what astroquery version do you use?

[DBerke]

Sorry, forgot to mention that:
astroquery: 0.3.10.dev5283
astropy: 3.1.2
Should be the latest, as I pulled from upstream/upgraded both just prior to writing this to see if the behaviour persisted.

[DBerke]

Ok, I've come up with a possible fix, but would like some advice:

The problem is that the dagger character, which takes up a single space's width in the formatted table, breaks the formatting in its raw HTML multi-character encoding form. The following code works to remove it, while leaving the fixed-width formatting of the table intact by replacing it with a space:

html_re = re.compile('&.*;')
table = html_re.sub(' ', table)

However, I'm now thinking it would be better to replace it with the actual Unicode † symbol again (changing the regex to "†"), and leave it for the user to find/be aware of, rather than silently remove it.

While testing this fix I found there are also occasionally question marks in the results as well, but since they render as a single-width character in HTML they don't break the formatting. In both cases they're there for a purpose, to inform the user of something related to the dubiousness of the attached quantity, so it seems like it might be better to leave them in and let the user deal with them. It's pretty trivial to strip them from the returned results if the user wants to (and they'll need to be doing some parsing of the results anyway to use them, most likely), it's the breaking of the table formatting that's impossible to handle externally. So, anyone have thoughts on this?

[keflavich]

I'm loosely in favor of a regex to replace the HTML characters with their unicode counterparts. This sort of hack is needed for things like NIST tables which are pseudo-fixed-width but don't have any truly fixed standard, afaik. Is there a library for converting HTML character codes to unicode?

[DBerke]

That's an excellent point. I checked, and there's the html module in the standard Python library with the unescape function that converts named and numeric character references in a given string to the corresponding Unicode characters. I added table = html.unescape(table) to the parsing in place of what I had and it seems to work (in that it converted the known cases of † to †). That should theoretically handle any other Unicode characters that might pop up anywhere in the table as well. If that seems like a good solution I can submit a pull request with what I've got.

[keflavich]

Yes, please submit a PR! That sounds like a great solution.

[DBerke]

Okay, I submitted a PR.