Electricitymap-contrib: Uruguay unreliable data

[jarek]

This seems unlikely to be accurate:

It is not a matter of exports either, or interchange data is inaccurate:

[systemcatch]

This is the history graph at the moment.

Frequent error in the logs here

Data page can be accessed in browser.

Running manually;

(EM-env) chris@ThinkPad:~/electricitymap$ python parsers/UY.py 
fetch_production() ->
Traceback (most recent call last):
  File "parsers/UY.py", line 153, in <module>
    print(fetch_production())
  File "parsers/UY.py", line 102, in fetch_production
    obj = parse_page(session)
  File "parsers/UY.py", line 92, in parse_page
    salto_grande = get_salto_grande(session)
  File "parsers/UY.py", line 43, in get_salto_grande
    generation = float(tie.text)
AttributeError: 'NoneType' object has no attribute 'text'

[systemcatch]

This is the url that the Salto Grande function is trying to access, http://www.cammesa.com/uflujpot.nsf/FlujoW?OpenAgent&Tensiones y Flujos de Potencia&15/10/2018 10:00

It returns an error for now as well as past dates. I've noticed that all of the Argentinian exchanges and parser are offline as well. I'll need to investigate this further.

ref #1334

[systemcatch]

I don't see an error with the AR production parser in the logger so not sure why it's not displayed, @corradio ?

All the AR exchanges seem to have moved, none of the links work anymore. Hopefully we can fix them, it's very annoying that the Salto Grande function is stopping UY working.

[corradio]

I think we have a problem with our logger that is not directing output to
Kibana. We're looking into it (cc @maxbellec)

Olivier

On Tue, Oct 16, 2018 at 1:30 PM Chris [e-mail] wrote:

I don't see an error with the AR production parser in the logger so not
sure why it's not displayed, @corradio https://github.com/corradio ?

All the AR exchanges seem to have moved, none of the links work anymore.
Hopefully we can fix them, it's very annoying that the Salto Grande
function is stopping UY working.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tmrowco/electricitymap-contrib/issues/1633#issuecomment-430202276,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABlEKOrt6QvbxQHxfmMAK0lnBejH64anks5ulcNCgaJpZM4XVNT0
.

[systemcatch]

Could we use _compras a salto grande_ instead? cc @alixunderplatz

[maxbellec]

@systemcatch the errors are logged in kibana. For errors in AR in the last 7 days, see below link.

https://kibana.electricitymap.org/app/kibana#/discover/1710fdd0-2460-11e8-a779-9d01de8d7a71?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-7d,mode:quick,to:now))&_a=(columns:!(level,extra.key,message),filters:!(('$state':(store:appState),exists:(field:level),meta:(alias:!n,disabled:!f,index:'93e631f0-245f-11e8-a779-9d01de8d7a71',key:level,negate:!f,type:exists,value:exists)),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'93e631f0-245f-11e8-a779-9d01de8d7a71',key:extra.key,negate:!f,params:(query:AR,type:phrase),type:phrase,value:AR),query:(match:(extra.key:(query:AR,type:phrase)))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'93e631f0-245f-11e8-a779-9d01de8d7a71',key:level,negate:!f,params:(query:ERROR,type:phrase),type:phrase,value:ERROR),query:(match:(level:(query:ERROR,type:phrase))))),index:'93e631f0-245f-11e8-a779-9d01de8d7a71',interval:auto,query:(language:lucene,query:''),sort:!('@timestamp',desc))

[systemcatch]

Looks like the AR exchanges (including Salto Grande) are working again!

@maxbellec for the production parser I don't see errors anymore.

(EM-env) chris@ThinkPad:~/electricitymap$ python parsers/AR.py 
fetch_production() ->
{'production': {'biomass': 2.0, 'nuclear': 361.0, 'gas': 9296.7, 'solar': None, 'coal': 0, 'unknown': 183.79999999999927, 'wind': None, 'geothermal': 0.0, 'hydro': 6357.0, 'oil': 246.5}, 'zoneKey': 'AR', 'source': 'portalweb.cammesa.com', 'datetime': datetime.datetime(2018, 10, 17, 11, 0, tzinfo=tzstr('UTC-3')), 'storage': {'hydro': None}}
fetch_price() ->
{'zoneKey': 'AR', 'price': 240.0, 'source': 'portalweb.cammesa.com', 'datetime': datetime.datetime(2018, 10, 17, 12, 0, tzinfo=tzstr('UTC-3')), 'currency': 'ARS'}
fetch_exchange(AR, PY) ->
{'sortedZoneKeys': 'AR->PY', 'netFlow': 26.0, 'datetime': datetime.datetime(2018, 10, 17, 12, 5, 4, 611477, tzinfo=tzstr('UTC-3')), 'source': 'cammesa.com'}
fetch_exchange(AR, UY) ->
{'sortedZoneKeys': 'AR->UY', 'netFlow': 48.0, 'datetime': datetime.datetime(2018, 10, 17, 12, 5, 12, 735280, tzinfo=tzstr('UTC-3')), 'source': 'cammesa.com'}
fetch_exchange(AR, CL-SING) ->
{'sortedZoneKeys': 'AR->CL-SING', 'netFlow': -0.0, 'datetime': datetime.datetime(2018, 10, 17, 12, 5, 17, 281674, tzinfo=tzstr('UTC-3')), 'source': 'cammesa.com'}

[jarek]

I would _guess_ that it was mostly correct up to around noon UTC today (screenshot timezone is UTC+2), then started breaking. Can we see any reason why?

[systemcatch]

Big chunk of hydro missing. Checking the logger it looks like the salto grande function was failing again (uses cammesa.com which is a bit unreliable). We might be able to fill in with past data.

[systemcatch]

Looking better at the moment.

[alixunderplatz]

I think I just observed the reason for the hydro data issues:

SCADA data from 10:04
There is a negative hydro generation. It consists of the sum of "Terra", "Baygorra", "Constitucion" plus "Compras a Salto Grande" (aka traded energy from Salto Grande Uruguay to Argentina). Since the traded amount sometimes is higher than the generation of the other three, there is a negative hydro generation.
Hidraulica = 135 + 97 + 315 + (-845) = -298 MW

SCADA data from 10:08
Compras a Salto Grande is 0 MW

The "compras a Salto Grande" value is constantly changing. Mostly it was 0 MW in the last minutes, but for a few minutes it is in the range of -8xx MW, which causes the trouble :)

---

This is where the error in the parser originates (I assume):

MAP_GENERATION = {
    'Hidráulica': 'hydro',
    'Eólica': 'wind',
    'Fotovoltaica': 'solar',
    'Biomasa': 'biomass',
    'Térmica': 'oil'
}

and

    salto_grande = get_salto_grande(session)
    obj['Hidráulica'] = obj.get('Hidráulica', 0.0) + salto_grande

Here we take the entire "hidraulica" category, including what is traded from Salto Grande to Argentina. So sometimes we also account for the traded amount, then the output of the UY part of Salto Grande is added on top, and we get the typical "hydro gaps" due to partially negative data (reducing total hydro output).
We should only extract these three from the second table on the website:

(((Another option is using the hourly data (TOTAL) from http://www.ute.com.uy/SgePublico/ConsPotenciaBalanceCentral.aspx, but we'd lose the high granularity.)))

Next we add the SG generation from the cammesa website as usual.

There is an option to extract the data directly from UTE without the cammesa data for Salto Grande. Selecting the table button next to "Compras a Salto Grande", you get a new table below with the real-time SCADA output of the Salto Grande units:

I'm not sure if there is a way to trigger this table to be shown to extract the per-unit-data. This would solve all the hydro generation issues. Units 1-7 belong to Argentina, units 8-14 to Uruguay, each unit with a max output of 135 MW.

[alixunderplatz]

There is something that still worries me a bit is the whole Salto Grande / Argentina / Uruguay thing in terms of double-accounting for the production in UY and AR.
SG has 14x135 MW, so 1890 MW total. Each country, to my understanding, owns 7x135, so 945 MW.
UY sells a lot of the output to AR.

Uruguay's UTE live SCADA data says 1725 MW output (about 45min later than the Cammesa data).

The output of SGDEHIAR (which is interpreted as "Salto Grande Hidro Argentina") from cammesa website says 1628 MW, but this cannot happen entirely on AR's share of the dam, but has to include some of the UY generators. Not sure about my first statement that "UTE HI CO" is for the constitucion dam due to the figure behind it (installed capacity 300something MW), but it is more likely standing for "compras", so maybe it's traded energy...

Máquina | Costo Específico Medio | Potencia Generada | Potencia Disponible | Reserva Rotante | Acumulado para Bajar
-- | -- | -- | -- | -- | --
SGDEHIAR | 0 | 1628 | 1628 | 0 | 4546
UTEIHICO | 0 | 839 | 839 | 0 | 5385
UTEIHISG | 0 | 770 | 770 | 0 | 6155

There must be an exact relation between all of these figures we need to investigate. I'll try to observe it this weekend.

[uy-rrodriguez]

Hi there, I'm new to this community. Any updates on these UY-AR issues?
I'd like to participate in the project and this may be a good starting point, not because of my knowledge in Electricity Engineering, but because I'm from Uruguay :)

By the way, I've found this project in the Hacktoberfest challenge website, happening in London and linked to UCL. You may get more students/staff involved in issues if you include the label "hacktoberfest" to self-contained issues.

Best,
Ricardo

[corradio]

Closing due to inactivity. Thank you all for your inputs!
(Data looks good though)