Electricitymap-contrib: Namibia is offline

[alixunderplatz]

In the last days, no data for Namibia was on the map.

Looks like the timestamp on the nampower site is from the future (+1 hour ahead of actual local time, although it's clearly live data).
Can you check if this is the reason? The last modification was due to timestamp issues, too.

Maybe the following part can be expanded to incude data that is up to 3600 seconds ahead of time:

    if diff.seconds > 7200:
        raise ValueError('Namibia scada data is too old to use, data is {} hours old.'.format(diff.seconds/3600))

[jarek]

I'm betting this is summer/daylight savings time thing again, as seen in https://github.com/tmrowco/electricitymap-contrib/pull/1344

But to be honest https://www.nampower.com.na/Scada.aspx doesn't load for me because the SSL certificate is invalid. That's also the error returned by the parser for me:

requests.exceptions.SSLError: HTTPSConnectionPool(host='www.nampower.com.na', port=443): Max retries exceeded with url: /gis/images/Gx.png (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)'),))

We could try telling it to ignore SSL errors but that's kinda messy.

edit: yep, https://en.wikipedia.org/wiki/Time_in_Namibia#1994%E2%80%932017 "Winter time began on the first Sunday in April at 03:00, and lasted until the first Sunday in September, 02:00 hours." - that matches date of https://github.com/tmrowco/electricitymap-contrib/pull/1344 and date seen just now, so they're just using old logic. Maybe they'll "fix" the hour offset next week when people look at it while working. The issue with the SSL cert I noted earlier this week, so it's unrelated.

[jarek]

Ok, adding verify=False to https://github.com/tmrowco/electricitymap-contrib/blob/44024d5c00903fc6a5db4c5cc3676d510a8d2011/parsers/NA.py#L40 then gives ValueError: Namibia scada data is too old to use, data is 23.03472222222222 hours old. This is because diff is something like -1 day, 23:01:04.836388.

Using diff.total_seconds() in the comparison "works", and on my machine returns:

(venv3)jarek@x1:~/projects/electricitymap/parsers$ python NA.py 
fetch_production() ->
/home/jarek/projects/electricitymap/venv3/lib/python3.4/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
/home/jarek/projects/electricitymap/venv3/lib/python3.4/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
{'production': {'oil': 0.0, 'wind': 4.88, 'coal': 0.0, 'solar': 35.52, 'hydro': 141.15}, 'datetime': datetime.datetime(2018, 9, 2, 14, 31, 52, 952389, tzinfo=tzfile('/usr/share/zoneinfo/Africa/Windhoek')), 'zoneKey': 'NA', 'source': 'nampower.com.na', 'storage': {}}
fetch_exchange(NA, ZA)
/home/jarek/projects/electricitymap/venv3/lib/python3.4/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
/home/jarek/projects/electricitymap/venv3/lib/python3.4/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
{'netFlow': -205.1, 'sortedZoneKeys': 'NA->ZA', 'source': 'nampower.com.na', 'datetime': datetime.datetime(2018, 9, 2, 14, 31, 56, 529752, tzinfo=tzfile('/usr/share/zoneinfo/Africa/Windhoek'))}
fetch_exchange(NA, ZM)
/home/jarek/projects/electricitymap/venv3/lib/python3.4/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
/home/jarek/projects/electricitymap/venv3/lib/python3.4/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
{'netFlow': -6.14, 'sortedZoneKeys': 'NA->ZM', 'source': 'nampower.com.na', 'datetime': datetime.datetime(2018, 9, 2, 14, 32, 0, 109046, tzinfo=tzfile('/usr/share/zoneinfo/Africa/Windhoek'))}

diff.total_seconds() is something like -3504.518509 (slightly less than -1 hour), which is obviously less than 7200. The datapoint datetime is then the current time (which is what is coded in the parser, to allow for OCR failures).

So, my takeaways:

• in any case, https://github.com/tmrowco/electricitymap-contrib/blob/44024d5c00903fc6a5db4c5cc3676d510a8d2011/parsers/NA.py#L80 should be modified to use diff.total_seconds()
• we might want to add verify=False to ignore SSL errors, since we're not doing anything security critical in the parser (unless an exploit is found in Image.open() or pytesseract...)
• for data in the future, abs(diff.total_seconds()) is an option, adding a comment that says we also accept data with timestamps 2 hours in the future. This feels increasingly hacky, but I suppose the data source is hacky to begin with.

[jarek]

Removing bug label as it's not a bug in our code

[systemcatch]

Does anyone else find that the scada page just loads forever but nothing appears?

[jarek]

Right now, yes, the images don't load. Maybe they're in the process of fixing the timezone issue? :D It used to work (after ignoring SSL warning) earlier.

[alixunderplatz]

The image is refreshing and showing the recent timestamp and date correctly.

It is still grey on the map, though 😢 What is the reason?

[systemcatch]

SSL verification is failing for the page.
HTTPSConnectionPool(host='www.nampower.com.na', port=443): Max retries exceeded with url: /gis/images/Gx.png (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:841)'),))

edit: yup setting verify=False fixes it like jarek said.

[jarek]

Namibia is now back follow systemcatch's PR