Electricitymap-contrib: Countries offline: AG, PE, RU-1+RU-2, AU-WA

[alixunderplatz]

Several countries are / were offline for a while now or have reoccurring issues.
Creating a checklist below so everyone can see what the issue could be and which parser had been fixed already. Kibana-logger error output below.
Should you work on fixing one of the parsers, please comment below and then tag this issue in the PR. :)

• [x] Argentina (AR.py)
  Could fail due to a html-format issue:
  'NoneType' object has no attribute 'getText'
  fixed by #2139

• [x] Peru (PE.py)
  Could fail due to a faulty/outdated date-time format:
  Failed to match '(?P\d{1,2})/(?P\d{1,2})/(?P\d{4}) (?P\d{1,2}):(?P\d{2}):(?P\d{2}) (?P(am|pm|AM|PM))' when parsing '2020/01/03 01:00:00'
  partly fixed in #2135

Quoting Alex

Data for 3rd of January up until last hour. So we should rather find a way to check for
start "today-1" and end "today"
and also
start "today" and end "today + 1" and merge the two datasets when both are returned.

This way we will get the most recent stuff. This could be why the parser failed in the past for a couple of hours each day, from what I can remember.

• [x] Russia West + Russia East (RU.py) + Exchanges of Russia with neighbouring states
  Maybe there was a change in the source data format? But running it locally works fine for me. Maybe too many requests?
  Expecting value: line 1 column 1 (char 0)

• [x] Western Australia (AU_WA.py or /lib/AU_solar.py)
  Maybe there was a change in the source data format?
  Expecting value: line 1 column 1 (char 0)
  fixed by #2140

[systemcatch]

Hey @alixunderplatz, happy new year to you! :tada:

I ran the AR parser locally and got this back. Initially I thought the service was down (very common as it's a big pile of spaghetti) but it looks like the HTMl has changed.

(EM) chris@ThinkPad:~/electricitymap$ python parsers/AR.py 
fetch_production() ->
Traceback (most recent call last):
  File "parsers/AR.py", line 964, in <module>
    print(fetch_production())
  File "parsers/AR.py", line 832, in fetch_production
    gdt = get_datetime(session=None)
  File "parsers/AR.py", line 640, in get_datetime
    find_hour = timesoup.find("option", selected="selected", value="1").getText()
AttributeError: 'NoneType' object has no attribute 'getText'

I'll send a PR for this.

[alixunderplatz]

Hey @systemcatch , thanks, happy new year to you, too!!!
One thing I could recently find for Argentina is the breakdown of the renewables in one diagram:
https://despachorenovables.cammesa.com/renovables/
No need for a mapping for these. I feel like "hidraulico" might only include small hydro which needs to be added to large hydro.

[systemcatch]

@alixunderplatz that's useful, I can take a look at adding that to the parser.

[con-cat]

Hi, I've worked on fixing Western Australia, just putting up a PR now.

[systemcatch]

@robertahunt any chance you could check the EM server to see why the RU parser is failing? I remember when I ran it locally everything was fine.

[robertahunt]

Hey @systemcatch! I am having no luck with tracking down the error on the server-side on my own. It looks like the data is fine and the feeder processes it properly in my tests. I will catch up with oli on this tomorrow.

[robertahunt]

@systemcatch , Unfortunately our server is getting a 404 response from the data source, which is why our parser is failing. @corradio suspects this is because our server's IP address has been blocked by the Russion TSO due to a high volume of requests. I have it on my to-do list to ask them to whitelist us, but I am not sure what else to do. Any other suggestions?

[systemcatch]

@robertahunt hmmm that's annoying, as to what to do asking them to whitelist us is a good idea. If they refuse maybe using a proxy is possible ( if Tomorrow is comfortable with that).

Maybe the EM needs to have a review of how often they are requesting data in general, some of the sources aren't really designed to be used by outsiders and too many requests might result in being blocked.

[robertahunt]

Just to keep this up to date, I sent a mail to our contact at the russian tso. I will let you know when he gets back to me.

[systemcatch]

RU parsers look to be working again!

[con-cat]

I noticed that the Western Australia parser was offline again with a different issue - I've put up another PR.