Electricitymap-contrib: Brazil 🇧🇷

[corradio]

Unfortunately we only have daily values for now:
http://www.ons.org.br/resultados_operacao/SDRO/Diario/index.htm

[alexanmtz]

@corradio the link is broken for me. It raises a 404. It needs a registered account to access the data?

I saw these ones very interesting:

• http://www.ons.org.br/pt/paginas/energia-agora/carga-e-geracao
• http://www.ons.org.br/pt/paginas/energia-agora/balanco-de-energia

But I don't know how the data is retrieve it, I can investigate.

[corradio]

They must have changed it :(
Wow that's perfect. It's exactly the data we need!
Can you try to see if they have an API?

[alexanmtz]

I'm looking for it, I could not found an API, but I will at least try to know where the requests came from to fill those graphics. I will try to create an account to see if that is possible have access to those data as well!

[corradio]

A quick look in the Chrome inspector shows that this is used to get the
live data:
http://tr.ons.org.br/Content/GetBalancoEnergetico/null

On Mon, Sep 11, 2017 at 10:56 AM, Alexandre Magno [e-mail]
wrote:

I'm looking for it, I could not found an API, but I will at least try to
know where the requests came from to fill those graphics. I will try to
create an account to see if that is possible have access to those data as
well!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tmrowco/electricitymap/issues/695#issuecomment-328464669,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABlEKBm-FN3ePF5pv-YCQ2IRr4txeh7Pks5shPW1gaJpZM4O4gFT
.

[systemcatch]

Very nice @alexanmtz !

Looking further both pages seem to update every 2-3mins. Data returned is json and looks very usable.

Generation Page
http://tr.ons.org.br/Content/Get/Carga_SIN
http://tr.ons.org.br/Content/GetBalancoEnergetico/364
http://tr.ons.org.br/Content/GetBalancoEnergetico/null
http://tr.ons.org.br/Content/GetBalancoEnergeticoConsolidado/364

Balance Page
http://tr.ons.org.br/Content/GetBalancoEnergetico/null

[alexanmtz]

So actually it's public, so that's nice, we can use it!

I've tried to create a account there and I'm waiting to approve from an admin to see what kind of access I could have.

[corradio]

I think in the meantime we can go ahead and just writer a parser for the
public endpoints we've found.
If they see it on the map it might already strengthen the case.

Olivier

On Mon, Sep 11, 2017 at 1:33 PM, Alexandre Magno [e-mail]
wrote:

So actually it's public, so that's nice, we can use it!

I've tried to create a account there and I'm waiting to approve from an
admin to see what kind of access I could have.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tmrowco/electricitymap/issues/695#issuecomment-328502245,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABlEKCodAwkyfFeUkWYdhvbgk1-eBzO2ks5shRqEgaJpZM4O4gFT
.

[systemcatch]

I will start working on this today. Should a relatively simple parser to write (famous last words!).

[corradio]

Awesome!!

On Mon, Sep 11, 2017 at 1:38 PM, systemcatch [e-mail]
wrote:

I will start working on this today. Should a relatively simple parser to
write (famous last words!).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tmrowco/electricitymap/issues/695#issuecomment-328503323,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABlEKG99UaaDaE4cVkWNcoQTLFVDG5V7ks5shRvGgaJpZM4O4gFT
.

[alexanmtz]

I was about to work on this too @systemcatch, I can help you if needed reviewing or testing.

[systemcatch]

@alexanmtz The production parser is up and running so you should be able to implement the exchanges now. Let me know if you need help with the structure or testing.

[leorjorge]

That's interesting! I had looked up this project some months ago when it was europe only, great to see it is expanding, just two days ago I looked it up and Brazil wasn't included yet. I see it has just been included, and things are likely to be still work in progress, but I wonder if it would be possible to separate the four subsystems, as per the info here: http://www.ons.org.br/pt/paginas/energia-agora/balanco-de-energia
That would yield a more detailed picture, similar to what is available to Australia and Canada.

[alexanmtz]

Yeah @leorjorge , it's possible to do that because we already have the data and exchange between regions.

[systemcatch]

Hey @leorjorge yes we've been able to add quite a few countries in South America recently. It's good to see the map continue to expand. As to Brazil yes we do want to split it down into regions. Would you be interested in helping with that?

[leorjorge]

Hey @systemcatch and @alexanmtz, thanks for the very fast responses! I don't have a solid background in coding, so not sure how I could help... I tried looking at the file for the parser and things look straightforward though (I know no python but grasped the general idea).
What is not clear to me is how a given map region is mapped into the parser file. I guess that would be the challenging part, as I think shapefiles for each region would have to be added somewhere, right? I can't even find in the current parser how that works...
If the mapping simple, it would just be a matter of having four different parsers, one for each region, changing the code to get only the data from a single region and adding the fluxes, right? Each regional parser would be even simpler than the current one, given no sums will need to be made.
If I can help somehow, let me know...

[corradio]

Hi @leorjorge ,

The code that downloads and generates the topologies is here:
https://github.com/tmrowco/electricitymap/blob/master/web/topogen.sh

The mapping is done here:
https://github.com/tmrowco/electricitymap/blob/master/web/app/countrytopos.js#L78

And yes, then we can just write one parser per zone.

[leorjorge]

Ok, so if I understand it, for each region I'll have a line in https://github.com/tmrowco/electricitymap/blob/master/web/app/countrytopos.js, defining the set of states that are in a given region. Any idea if I can use a set of states instead of just one?
And in https://github.com/tmrowco/electricitymap/blob/master/web/topogen.sh#L9 I need to add a 'BRA' entry along with the other subdivided countries. Is that right? Is there someplace where I can consult the list of states' codes, just to make sure they are the same as the commonly used acronyms?

[corradio]

You need to run the topogen.sh script and look at the output to investigate if the data contains subunits or states.
In the countrytopos file you will find examples of zones where subunits are merged (i.e. GB)

[leorjorge]

Do I need to download any additional files besides the repository, or install some compiler/module? I'm getting the following errors when I try to run the script:
./topogen.sh: line 29: node_modules/.bin/shp2json: No such file or directory
./topogen.sh: line 30: node_modules/.bin/ndjson-map: No such file or directory

[corradio]

You need to run npm install to install the dev dependencies from
package.json

On Wed, Sep 13, 2017 at 2:26 PM, Leonardo R. Jorge <[email protected]

wrote:

Do I need to download any additional files besides the repository, or
install some compiler/module? I'm getting the following errors when I try
to run the script:
./topogen.sh: line 29: node_modules/.bin/shp2json: No such file or
directory
./topogen.sh: line 30: node_modules/.bin/ndjson-map: No such file or
directory

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tmrowco/electricitymap/issues/695#issuecomment-329151570,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABlEKHwuWcCNCPmZyjkIrSHuxa0v43Dtks5sh8oMgaJpZM4O4gFT
.

[jarek]

Hint: to preview what kind of subdivisions are available, download the ZIP files from Natural Earth (build the URL by following along with what topogen.sh does), then open it with http://mapshaper.org/

So download http://naciscdn.org/naturalearth/10m/cultural/ne_10m_admin_1_states_provinces_lakes.zip then select the zip file on mapshaper and click "import" - it should show it within a few seconds

[leorjorge]

Ok, I managed to find the matadata file in the zip shapefile, and that has all the codes. Brazilian state codes are available at one of the columns (code_hasc), can I use it? There is the issue that two of the northern states have no code (just BR.) but given they are in the same region it shouldn't be an issue. Can I just create new entries in countrytopos.js and add BRA to the topogen.sh script? @systemcatch do you think you can adapt the parser to grab values separately? It looks simpler than the one you wrote...

[systemcatch]

@leorjorge I'll modify the parser to fetch each region separately. The fetch_production function can be adapted to work with regions (BR_N, BR_NE, BR_CS, BR_S) easily.

Things that need to be done.

• [x] Adapt production parser to fetch data by region instead of country.
• [x] Find capacities by region
• [x] Exchange with Argentina (BR-S).
• [x] Exchange with Paraguay (BR-S).
• [x] Exchange with Uruguay (BR-S).
• [x] Regional interconnections.
• [x] Research energy storage in Brazil.

[corradio]

We have two PRs ready:

• geo: https://github.com/tmrowco/electricitymap/pull/749
• production parsers: https://github.com/tmrowco/electricitymap/pull/759

Would be nice to add the exchanges and then I'll merge all at once!

[corradio]

It's now merged and live!
We still need cross-border exchanges, and to add arrows:

Also, can someone double check all capacities sources are added to the README?

Note: The previous exchange parser with UY is not used:
https://github.com/tmrowco/electricitymap/blob/master/parsers/BR.py#L120
It should be updated to point to the correct region, and we should reference it in exchanges.json.

[systemcatch]

Capacities for the regions are from the same source as the country totals, so we are good there.

@alexanmtz do you want to modify the exchange function to include Argentina/Paraguay and update Uruguay or shall I do it?

[corradio]

PR here: https://github.com/tmrowco/electricitymap/pull/768#pullrequestreview-64232616
Still will require AR and PY exchanges.

[alexanmtz]

@corradio should we close this?

[corradio]

Yep!