Electricitymap-contrib: Add real-time data to Chile

[robertahunt]

The parser for Chile appears to be failing for real time. (see kibana) and also for historic data (gives arrow timestamp formatting error)

I am also a bit confused about Chile in general. It looks like in zones.json the only parser we have set up is CL-SING-SIC, but this parser seems to then point to CL.py, which appears to (attempt to) fetch production for all of Chile. (link to data source).

I can fix the hiccup the parser is having, but I want to first understand if it is fetching data for all of chile or only CL-SING-SIC.

[alixunderplatz]

@robertahunt the separated systems SING (Northern Chile) and SIC (central Chile) were merged through an interconnector a while ago, creating one common zone which was named SING-SIC here. The data from the link is for that region SING+SIC only.
Like in many Spanish-speaking countries, it seems that zone is called "SEN" now (sistema electrico nacional), so we could switch to that terminology while the parser is being fixed.
Fetching real-time data for Chile wasn't really possible in the past, since the data was partially delayed and incomplete, and complete generation per type data mostly was like 7-8 hours behind real-time.
At the moment, it seems like "Mix Generación real por tecnología" tab isn't showing any data on https://www.coordinador.cl/operacion/graficos/operacion-real/generacion-real-del-sistema .
Console shows a server error 500 on coordinador's side.
In the past, I could see the some incomplete data up to real-time after selecting today's date. How's the parser requesting dates? I've never seen data on the map in the 24hour graph although there could have been some of it with a delay.

[robertahunt]

@alixunderplatz That makes sense, thanks for explaining. Yeah, does it makes sense to rename the parser as well, to be CL-SEN.py or CL-SING-SIC.py?

I can download the 'tsv' from the site and it shows data up to 2020-02-01 03:00 at the moment, which is quite far back, but still reasonable for getting historic data. Just comparing the csvs from Jan and Feb, the February one also looks like it is missing a lot of datapoints, so maybe more recent data is incomplete which would be unfortunate.

[systemcatch]

If I remember rightly Chile was 4 separate systems, SING and SIC were combined and make up the majority of usage. The 2 systems to the south (SEA & SEM) are isolated and generate very little power.

The zones look right on the map.

655 provides more context for the current situation.

[robertahunt]

Just to update here:
It looks like they have put most of their data behind an API with token authorization (https://www2.coordinador.cl/sistema-informacion-publica/api-publica/). I have requested an API key (although in english), and I hope they will get back to me.

[corradio]

I've renamed this issue so we can focus on adding real-time data (hopefully through the API that @robertahunt asked access to).

[robertahunt]

Just to update, I still have not received a response to my API request. We could consider scraping the website directly, although that is not as robust.

[peterhughgriffin]

Is it worth requesting for an API key again, but in Spanish?
Afraid I'm not offering, but I imagine we have a Spanish speaker somewhere...

[robertahunt]

I agree that would be best. I don't know any spanish speakers in the community off the top of my head either. I'll send out a quick ask on slack. fingers crossed!

[robertahunt]

While looking at chile again, I also found this site, which looks promising, although it says it is in beta, and I was able to automatically obtain an api key, but I am still not certain if it has the data we need http://energiaabierta.cl/about-the-open-data-platform/?lang=en

[alixunderplatz]

hey @robertahunt , have you had a look at the "coordinador" app already? there is solar and wind near-live data and "other generation without solar and wind". I think @systemcatch and I were discussing a way to extract data from the apps datasources


if you take that solar+wind+other for the "live" view and fill up once the "previous day" data is available, that should be a solution for CL-SEN

[alixunderplatz]

@robertahunt Links to the app data: https://github.com/tmrowco/electricitymap-contrib/issues/1683#issuecomment-441279533 :)

[ericmassip]

I sent a message requesting for access to the Public API in Spanish, I'll let you know if they respond ;)

[robertahunt]

Hey! @alixunderplatz I hadn't seen that! Nice!! Would also be a good way to go. This isn't on my immediate to do list yet, in case anyone else wants to take a stab at a parser. Otherwise I'll have a more in depth look when I find time.

[peterhughgriffin]

I'd be keen to work on this, but it will be my first stab at such a problem, so will need some guidance and will probably be quite slow!

I'll look round things a bit more, then probably drop some clarifying questions.

[robertahunt]

Sounds good, and no worries. Feel free to ask here or ping me on slack with any questions. Also, if you see anything that is lacking in our documentation, please feel free to make a pr to update it!

[systemcatch]

@peterhughgriffin feel free to ping me if you need advice on implementing this.

[peterhughgriffin]

So, having looked through things in more detail, I want to clarify what we think the best bet is. We have four possible data sources:

1. Coordinador API (This is what the current parser uses, but the most recent data is incomplete): https://sipub.coordinador.cl/api/v1/recursos/generacion_centrales_tecnologia_horario?
2. Coordinador API 2 (This requires a key that we have asked for, but don't (yet) have): https://www2.coordinador.cl/sistema-informacion-publica/api-publica/
3. Coordinador app (Data looks great, but will require scraping)
4. Energia Abierta API (Is in beta and we are not sure about the data source): http://energiaabierta.cl/about-the-open-data-platform/?lang=en

My understanding is that we are suggesting to pursue 3, because we know the data is there and is good, whereas the others have access questions and may also lack the real-time data.

Is that correct?

I investigated the API in 4, but after making an account there was no indication of how I can obtain my API key?! My only hope is that it will be emailed to me in due course. It does claim to give hourly data though: https://desarrolladores.energiaabierta.cl/9/generacion-bruta

In terms of scraping data from an app, this is an exciting challenge for me! I have only ever grabbed data from APIs before. Is there another parser and or some key commands/methods that I can get started with?

[systemcatch]

The app is fed by json using the links I mentioned in https://github.com/tmrowco/electricitymap-contrib/issues/1683#issuecomment-441279533

It should be a case of using requests to fetch the data (it has a built in json() method) then sorting it into the appropiate categories, the Brazil parser could be a useful implementation reference.

[robertahunt]

@peterhughgriffin - you can get the API key from #4 by logging in, clicking on your name, and going to 'Administrador de Aplicaciones', then clicking on 'crear aplicaciones'. That should create it. Then in the main screen, you can scroll down to the endpoints and click on one, for example 'generacion bruta', and you can then click 'Pruebe este método', to try the api through their website, and you can choose which newly created api token it should use -

The site is annoying to navigate, and I am still unsure if the data is useful to us or not, so up to you. I think #3 is a sure method that will get us somewhere!

[peterhughgriffin]

Submitted PR #2454