Electricitymap-contrib: RU-parser not working

[alixunderplatz]

Data for the Russian zones hasn't been on the map for a while now.
Maybe we should revert to not using the API of the data source for RU, but scrape from the website instead, if possible? br.so-ups.ru
You can find the small bar chart on the page, indicating production from the sources for the recently displayed hour.
@denplis can you help?

[steinwinde]

You saw this? https://br.so-ups.ru/BR/GenConsum

[q--]

When running the parser directly on my system it seems to work fine. Perhaps ElectricityMap's server got blocked or something?

[corradio]

The logs at https://kibana.electricitymap.org/app/kibana#/discover/10af54f0-0c4a-11e9-85c1-1d63df8c862c?_g=(refreshInterval:('$$hashKey':'object:232',display:'5%20minutes',pause:!f,section:2,value:300000),time:(from:now-24h,mode:quick,to:now))&_a=(columns:!(message,extra.key,level),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!t,index:'96f67170-0c49-11e9-85c1-1d63df8c862c',key:level,negate:!f,params:(query:ERROR,type:phrase),type:phrase,value:ERROR),query:(match:(level:(query:ERROR,type:phrase)))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'96f67170-0c49-11e9-85c1-1d63df8c862c',key:extra.key,negate:!f,params:(query:RU-1,type:phrase),type:phrase,value:RU-1),query:(match:(extra.key:(query:RU-1,type:phrase))))),index:'96f67170-0c49-11e9-85c1-1d63df8c862c',interval:auto,query:(language:lucene,query:''),sort:!('@timestamp',desc)) show:

Traceback (most recent call last):
File "/home/src/electricitymap/feeder/lib/fetch_data.py", line 175, in launch_parsers
objs = res.get(timeout)
File "/usr/local/lib/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
File "/usr/local/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(args, *kwds))
File "/home/src/electricitymap/feeder/lib/fetch_data.py", line 173, in
**parser_kwargs))
File "/home/src/electricitymap/contrib/parsers/RU.py", line 97, in fetch_production
json_content = json.loads(response.text)
File "/usr/local/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[FelixDQ]

The server is indeed blocked.

Trying to connect from the server results in error 404 and an empty response:

Python 2.7.17 (default, Apr 15 2020, 17:20:14) 
[GCC 7.5.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.get("http://br.so-ups.ru/webapi/api/CommonInfo/PowerGeneration?priceZone[]=1&startDate=2020.07.02&endDate=2020.07.02").text
u''
>>> requests.get("http://br.so-ups.ru/webapi/api/CommonInfo/PowerGeneration?priceZone[]=1&startDate=2020.07.02&endDate=2020.07.02").status_code
404

Doing the same thing locally works fine.

Is there someone that we can reach out to?

[FelixDQ]

I have reached out to the data provider.
I update here when I get a response.

[FelixDQ]

Our server has been whitelisted again 🎉
We might need to make changes to the parser to prevent this from happening again. I will create a new issue with details if that is necessary.