Electricitymap-contrib: Offline zones and parser fixes: GB-ORK, GT, US-HI/Oahu, AM, CR and others

[alixunderplatz]

• [x] GB Orkney Islands / GB-ORK.py
  possible SSL-Error:
  HTTPSConnectionPool(host='www.ssen.co.uk', port=443): Max retries exceeded with url: ... (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))
  fixed in #2504
• [x] Guatemala / GT.py
  maybe it's timing out? (too many requests?)
  Traceback (most recent call last): File "/home/src/electricitymap/feeder/lib/fetch_data.py", line 176, in launch_parsers objs = res.get(timeout) ...

• [x] US Hawaii Oahu / US-HI.py
  Arrow package issue? Or is it just the same issue as observed for other US-zones after implementing the new geometries?
  File "/usr/local/lib/python3.6/site-packages/arrow/parser.py", line 224, in parse "Failed to match '{}' when parsing '{}'".format(fmt, datetime_string) arrow.parser.ParserMatchError: Failed to match 'YYYY-MM-DDTHH:mm:ss' when parsing '2020-05-03T13:30:00.000Z'
  Fixed in #2463

• [ ] Armenia / AM.py
  See https://github.com/tmrowco/electricitymap-contrib/issues/2611

• [x] Croatia / HOPS.py
  solar = float(solar.replace(",", ".")) AttributeError: 'numpy.float64' object has no attribute 'replace'
  fixed in #2525

• [x] Cyprus / CY.py
  see #2376

• [x] Russia (both zones) / RU.py
  see #2461

• [ ] Reunion (temporary) / FR-O.py
  setting negative values for oil to "None" or "0" should fix this:
  invalid point: {'zoneKey': 'RE', 'datetime': datetime.datetime(2020, 5, 3, 14, 10, tzinfo=tzoffset(None, 14400)), 'source': 'opendata-reunion.edf.fr', 'production': {'nuclear': 0, 'biomass': 0.0, 'coal': 186.15, 'gas': 2.10047, 'hydro': 55.941, 'oil': -0.21995, 'solar': 56.498999999999995, 'wind': 0.91, 'geothermal': 0, 'unknown': 0}, 'schemaVersion': 3}, reason:RE: key oil has negative value -0.21995
  It's okay if the value is negative, it just means "self consumption" while the type is not producing.

Some Indian zones haven't responded for many months, so they could be deactivated:

• [ ] IN-GJ.py
  HTTPConnectionPool(host='www.sldcguj.com', port=80): Max retries exceeded with url: /RealTimeData/PrintPage.php?page=realtimedemand.php (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f46c4203f28>: Failed to establish a new connection: [Errno 110] Connection timed out',))
• [ ] IN-UT.py
  Failed to match 'DD-MM-YYYY HH:mm:SS' when parsing 'Last Updated:'
  But the actuall issue is that data on the website just shows "0 MW" everywhere.

[systemcatch]

@alixunderplatz I can fix the Hawaii parser, looks like the format string given is wrong.

[q--]

https://www.ssllabs.com/ssltest/analyze.html?d=www.ssen.co.uk reports that the SSEN site has an incomplete certificate chain (missing an intermediate certificate), which is probably the reason for the SSL errors.

(The reason that we don't get an error when visiting in the browser is that all or at least most browsers cache intermediate certificates the first time they encounter them, so the browser already has the intermediate certificate cached from visiting another web site.)

The site's SSL configuration also has a bunch of other problems, chief among them being vulnerable to the POODLE attack, so it might be worth sending them an e-mail to see if they can fix it.

Alternatively, manually downloading the missing certificate and telling Python to trust it would also work, but I'm not sure if Python offers a way to do this and if so, how.

[Fred-3000]

The Australian parsers seem to be lacking most of the solar production lately.

[Varmin123]

For Armenia, I have looked at http://epso.am/poweren.htm
It seems that they broke their html code page, but the source continues to have every required information (in the form var XXXX = "yy"\n ). This breaks the soup parser that assumes a valid html syntax.
It should be better for the parser to go and extract the lines of code that match the mappings present in the parser file. https://raw.githubusercontent.com/tmrowco/electricitymap-contrib/master/parsers/AM.py

[q--]

Guatemala has been working correctly for some time now, and Russia is back after ElectricityMap got unblocked a few weeks ago

[q--]

• Cyprus is back
• Armenia broke again
• Russia has been back for some time now

[FelixDQ]

• Cyprus is back
• Armenia broke again
• Russia has been back for some time now

Thanks. I updated the main post

[corradio]

Hi folks,

Any news here? If not, I suggest breaking things down as individual issues to make it more actionable?

[q--]

I've looked into the AM parser a week or so ago and the entire website http://epso.am is gone.

I've looked at a few other related Armenian websites like Armenian Energy Agency, Electric Networks of Armenia, HIGH VOLTAGE ELECTRIC NETWORKS and http://www.minenergy.am/en/page/539 but none of them have real-time data.

So the AM parser is unlikely to start working again.