Electricitymap-contrib: add Namibia

[jarek]

http://www.nampower.com.na/Scada.aspx (linked from Nampower home page) has live generation and exchange data, but it's outputted as text within a PNG image. Optimally we would contact the utility and ask if they could publish the same data as text. Failing this, perhaps we could use some API to do
optical character recognition (OCR), e.g. as in https://gist.github.com/dannguyen/a0b69c84ebc00c54c94d, but OCR is generally not trivial.

Mapping of the plant names to fuel at http://www.nampower.com.na/Page.aspx?p=182

It looks like most of Namibian electricity is imported from Eskom, the South African utility. ZESCO is the Zambian utility.

[jarek]

There is also http://www.sappmarket.com/ which as far as I can tell provides a live or semi-live view onto the Southern African Power Pool. It seems like it should include the current generation as well, but I don't know the terms/lingo ("Day-Ahead Market", "Forward Physical Market", "APF-IDM", etc), so I don't know if I'm missing it or if it's not updated.

[systemcatch]

A few notes on this. I've run some preliminary tests using tesseract-ocr on generation images downloaded from Nampower's Scada dashboard. Initial results are ok but not good enough to parse the data. The images will have to be cleaned up before applying OCR on them. It's likely a case of trial and error until the right method is found.

[systemcatch]

Based on discussions in #817 with @alixunderplatz I can now get OCR to work reliably on the nampower scada images. The next step is to automate all of this into a parser.

@corradio
There are some pros and cons with building this;

Pro

• First African country on the map (may lead to more sources/collaboration).
• OCR techniques can be reused in other parsers, allowing several issues to be resolved.
• The challenge is fun!

Con

• More python dependencies, PIL & pytesseract at a minimum.
• Have to install tesseract & image magick on the EM server.

Overall I think this is worth doing.

[corradio]

how long does the parser take?

On Fri, Feb 16, 2018 at 1:08 PM, Chris [e-mail] wrote:

Based on discussions in #817
https://github.com/tmrowco/electricitymap/issues/817 with
@alixunderplatz https://github.com/alixunderplatz I can now get OCR to
work reliably on the nampower scada images. The next step is to automate
all of this into a parser.

@corradio https://github.com/corradio
There are some pros and cons with building this;

Pro

• First African country on the map (may lead to more
  sources/collaboration).
• OCR techniques can be reused in other parsers, allowing several
  issues to be resolved.
• The challenge is fun!

Con

• More python dependencies, PIL & pytesseract at a minimum.
• Have to install tesseract & image magick on the EM server.

Overall I think this is worth doing.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tmrowco/electricitymap/issues/606#issuecomment-366219283,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABlEKLg8j3Z4YbxS4XEz1H7Z6ZZ5njv9ks5tVW-2gaJpZM4OFs88
.

[systemcatch]

It's not written yet but based on my tests the limiting factor will be the request to http://www.nampower.com.na/Scada.aspx . I expect this parser will be similar in speed to most of the current ones, maybe even a little faster than some.

[jarek]

If getting tesseract on the EM parser server is a problem, I can try hosting a OCR proxy service for this and Singapore solar on my VPN

[systemcatch]

Some good news here. I think PIL should be enough on it's own for this job, meaning we won't have an additional dependency in Image Magick.

[jarek]

Namibia is now on map! Data is imperfect, because imports are assumed too optimistically... but without South African data, the server can't do any better (see #563). As it doesn't appear we can do more from Namibian data, I'll close this :heart:

[corradio]

Good job man!

On Fri 9 Mar 2018 at 20:56, jarek [e-mail] wrote:

Closed #606 https://github.com/tmrowco/electricitymap/issues/606.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/tmrowco/electricitymap/issues/606#event-1514018200,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABlEKC7r0gB3-2D109ZSHyXab6vIIxkhks5tct5TgaJpZM4OFs88
.