Electricitymap-contrib: Change data source for UK-GB (5-min data)

[alixunderplatz]

I've seen this happen frequently in the last weeks: the wind energy production on ENTSO-E for the UK would drop and leave a very spiky and obviously wrong graph for the UK-GB zone.

This issue got much worse in the last three days. Here is what it looked like for 1st of May on ENTSO-E:

Offshore generation would almost completely disappear and onshore wind would drop by 30-40%.

Other sources, like these dashboards don't seem to be affected by that data quality issue.:
http://electricinsights.co.uk/#/dashboard?&_k=dxtgvl
http://gridwatch.co.uk/
http://grid.iamkate.com/

So maybe it's time to switch to data from ELEXON, which have a super short reporting interval of 5 minutes for both generation and all exchanges. Seems like there is a delay of 1 hour:

https://www.bmreports.com/bmrs/?q=generation/fueltype

Check out the XML and CSV links on the site that'll lead you to the data.
Historical data available, too!

[lorrieq]

ELEXON is great! I have dealt with this source in my day job. Let me know where I can help out.

[brunolajoie]

Yeah, true, the closer to the source is always the best...! and GB is soon going to able to be broken down by DSO's regions in National Grid / Elexon systems apparently. Is there also a 1h delay is their API?

[systemcatch]

We should collect this source in addition to entsoe, there may be some interesting differences. The xml looks like a better choice as the csv data doesn't have column headers?

[corradio]

CSV will be easier to parse though, so I'd suggest going the CSV way.

[lorrieq]

I'm going to code this up this evening, if nobody else has started.
I far prefer parsing CSVs. The only thing to note is you are relying on the fuel column order outlined in the documentation being and staying correct as well as applying small edits when new fuels are added. Not sure if this will happen again but I recall Biomass was split out of one of the other fuel types around November '17.

[systemcatch]

@lorrieq we could check the number of columns in the csv and log a warning if it's changed from the current number.

[lorrieq]

Hey all, I'm almost ready to submit an initial PR for this but wanted to check a few things.

1) To confirm, the datetimes in the parsers output should be in UTC right?

2) The datetime for each observation. Should this be the published time minus 5 minutes? In the example, the first observation for SP 28 which is 12:30GMT has a published time of 12:35GMT. My interpretation is the parser give this observation a datetime of 12:30GMT, the observation with 12:40GMT publish time should have a datetime of 12:35GMT and so on.

3) The exchanges. I don't think there is any way to tell the netFlow, there is no reference to the direction of flow, just a total MW value which probably is an aggregation of import/export?

4) How are we dealing with multiple parsers per zone? I don't see any examples of this to date. Maybe allow the parsers dictionary values map to a list (as shown in image) rather than single string at the moment? I presume this would need some change to the backend code.

[systemcatch]

1. datetime object with a timezone is fine.

2. How exactly is the settlement period used?

3. The exchanges on the generation by fuel page seem to ignore negative values, but on https://www.bmreports.com/bmrs/?q=generation/avghalfhourIC they don't.

4. I think that's the way to go but @corradio is the one who will make the changes to the backend.

[corradio]

1. The datetime should be the time of the data point. If the measured
   average production between H and H+1 is measured at time H+1, the timestamp
   should be set at H (i.e. the datapoint is valid from H until a new
   datapoint arrives).
2. During an interval, and for a pair of zones, the parser should return
   the average flow during that interval (netFlow). Positive between a and b
   means from a -> b, negative means from b to a.
3. We can't have multiple parsers atm. We will need to switch from ENTSOE
   to ELEXON. The switch happens in config/zones.json where we simply replace
   ENTSOE by ELEXON in the parsers dictionary values.

On Sun, May 6, 2018 at 5:39 PM Chris [e-mail] wrote:

>

1.

datetime object with a timezone is fine.
2.

How exactly is the settlement period used?
3.

The exchanges on the generation by fuel page seem to ignore negative
values, but on
https://www.bmreports.com/bmrs/?q=generation/avghalfhourIC they don't.
4.

I think that's the way to go but @corradio
https://github.com/corradio is the one who will make the changes to
the backend.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tmrowco/electricitymap/issues/1376#issuecomment-386888457,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABlEKE_K_c6Yxf-kn38236MLqKL8B6hZks5tvxlHgaJpZM4TvVR_
.

[alixunderplatz]

@lorrieq @systemcatch @corradio Hey guys, I am sorry I just noticed this now, but there seems to be no solar data on the table on the page that I linked above, is that correct?

The wind and biomass data seems to be too low as well, compared with recent data on the map.

The fossil and hydro stuff looks good and precise, though. Seems like the 5-min-data is just usable for these categories. Not for solar, wind and bio :(

This page below has solar, wind and biomass figures which look complete and valid, but just for the "old" 30-min-intervals :(
https://www.bmreports.com/bmrs/?q=actgenration/actualaggregated

I don't understand why they would publish it in such a strange way.

ref PR #1390

[lorrieq]

Oh no, the best laid plans! I can't understand how I didn't realise there was no solar there either.

Well, if it's worth investigating I can contact them about their seemingly low reporting of wind and biomass data and if they have any plans to include solar generation there in the near future.

[brunolajoie]

Don't they have an API at Elexon/BMRS that could have potentially cleaner data?

[systemcatch]

Posting here what I said in the PR.

Most websites use https://www.solar.sheffield.ac.uk/pvlive/ for solar data. It's a fairly reliable estimate provided by the University of Sheffield at 30min intervals.

[lorrieq]

5 minute data:
Valid for fossil fuels and hydro here.

30 minute data:
Valid wind and biomass here as @alixunderplatz pointed out.
It seems solar generation can be found here which we didn't spot yet, but I haven't looked much into the accuracy of the data.

So acceptable data seems to be there for all, however in differing granularities.
The 5 minute wind generation values seem to be more representative of onshore wind than total wind? Maybe this is the reason for the under reporting? But then again, the values don't completely line up either.

[systemcatch]

Currently (9am) Sheffield Solar shows 5.24GW of generation while https://www.bmreports.com/bmrs/?q=actgenration/actualorestimated shows 3.67GW.