Electricitymap-contrib: Add India

[corradio]

Possible data source:

• http://vidyutpravah.in/ (lacks production mix breakdown)

[hectorespert]

I found the demand and production of the State of Punjab, but to implement the parser we require split India in states.

I tried it but I don't found the India subunits.

• Punjab Production
• Punjab Demand

[corradio]

That's a pretty cool milestone if we can start adding parts of India.
If you want to look at states (and not subunits), you should try to add IND to https://github.com/tmrowco/electricitymap/blob/master/web/topogen.sh#L9, and then re-run that script. It will re-generate world.json, and another file, world_states_props.json (https://github.com/tmrowco/electricitymap/blob/master/web/topogen.sh#L35) that you can inspect in order to find the proper state.
Just ping me if you have troubles, but this might help get you started.

[jarek]

Excellent catch @blackleg! It hasn't occured to me to check for data by state.

Looks like Gujarat also has data:
https://www.sldcguj.com/RealTimeData/RealTimeDemand.php

Maharashtra as well, but I only found the image format so far: http://mahasldc.in/wp-content/themes/mahasldc/report2.html

Google searches for "(state_name) electricity generation" have found these for me and might find subsequent ones

[systemcatch]

Generation data I've found so far.

Andhra Pradesh
http://www.core.ap.gov.in/CMDashBoard/UserInterface/Power/PowerReport.aspx

Capital Territory of Delhi
http://www.delhisldc.org/Redirect.aspx?Loc=0804
Unsure if there is a breakdown by type.

Karnataka
http://kptclsldc.com/StateGen.aspx
The plants will need to be mapped to type.

[jarek]

I wonder how much power goes over interchanges... Karnataka http://kptclsldc.com/Snapshot.aspx looks to be about 10%?

We could really use someone well-versed in India electricity situation

[systemcatch]

Wikipedia has some interesting information. The states seem very well connected.
https://en.wikipedia.org/wiki/Electricity_sector_in_India
https://en.wikipedia.org/wiki/National_Grid_(India)

[hectorespert]

India installed capacity: http://www.cea.nic.in/reports/monthly/installedcapacity/2016/installed_capacity-03.pdf

[prson]

I am slightly aware of Indian power scenario and have been following this amazing initiative since sometime. For India, there is a lack of data sources and a lot of manual parsing has to be done.
@blackleg the link you mentioned (http://www.cea.nic.in/reports/monthly/installedcapacity/2016/installed_capacity-03.pdf) is the best source for capacity. They release at the end of each month. More data is available as pdfs mostly.
@systemcatch Unfortunately, wikipedia is mostly outdated.

[acomets]

Hi, I just found out about your initiative, great job!
If I may chip in on a little research I did:

Capital Territory of Delhi
I saw most of the power stations in the parser (https://github.com/tmrowco/electricitymap/blob/master/parsers/IN_DL.py) were missing fuel type
delhisdldc provides us with a list of generation companies codes (https://www.delhisldc.org/psgencocodes.aspx) from which here is what I could get

• DMSWSL (Delhi Municipal Solid Waste Solutions Limited) is a waste management facility operated by Ramky, so a biomass power plant
• EDWPL (East Delhi Waste Processing Company Private Limited) is a waste management facility operated by IL&FS, so a biomass power plant
• GT (IPGCL Gas Turbine Power Station) which comprises
  
  
  
  • 6 gas turbine units of 30 MW each
  
  
  • 3 steam turbine units of 34 MW each
  
  
• TOWMP (Timarpur Okhla Waste Management Company Private Limited) is a waste management facility, so a biomass power plant

Karnataka
https://github.com/tmrowco/electricitymap/blob/master/parsers/IN_KA.py
As for what CGS stands for, my guess would be Conventional/Coal Generating Stations, wdyt?

Keep up the good work!

[systemcatch]

@acomets thanks for doing research into those power stations, I think we should be able to improve our breakdown by type there. As for CGS I tend to think it means conventional generation systems while NCEP means non conventional electricity production.

[anirudh-ramesh]

1. State Load Dispatch Centre of Gujarat (GJ) - the power plants indicated on the dashboard have a corresponding description in the following file (CSV):
   gujarat.txt
2. State Load Dispatch Centre of Uttar Pradesh (UP) - this doesn't have a scrapable webpage, but the data could possibly be retrieved through an API.
3. Western Regional Load Dispatch Centre - doesn't provide a breakdown of plants, so this can't be used to determine carbon emissions, but it provides a breakdown of states within the region, so the fuels for the generation can currently be marked as "unknown".

[anirudh-ramesh]

Folks, "CGS" stands for "Central Government Share". The logic is that the power plants are owned by one or more of three parties: The state government, the central (identical to federal) government, and/or private enterprises. Any amount of energy flowing through a transmission line operated and audited by the load dispatch centre (LDC) that was generated from one of the plants owned by the central government is labelled as CGS.
@systemcatch @acomets @blackleg @jarek @corradio

[systemcatch]

@anirudh-ramesh Thank you for digging up these new sources. It's useful to know that CGS refers to ownership rather than generation type.

Gujarat should be ready to go based on what I see, do you want to have a go at writing the parser?

Uttar Pradesh cannot be scraped directly but watching the developer console I see POST requests that appear to be updating the data.

[anirudh-ramesh]

Yes, I've been moulding the GJ parser. It doesn't look very professional, so I'll need help refactoring after I have a fully functional version, @systemcatch.

[corradio]

@anirudh-ramesh the GJ parser has an issue:
The date doesn't have a timezone, so it falls back to UTC, and therefore doesn't get shown in the UI
Also, it would be nice to print out to the console all the units that do not match your list of units (so we can add them later on)

[acomets]

On a more general note, I was wondering if there was any way to account for off-grid/non-utility generation. Indeed, I found the following:

• On the Electricity sector in India Wikipedia page, they mention 50 GW installed capacity Captive power but also 1.5 GW installed capacity Renewable energy
• Also on the Renewable energy in India Wikipedia page they mention this same 1.5 GW installed capacity Off-grid renewable energy, and in Solar power they mention

Much of the country does not have an electrical grid, so one of the first applications of solar power has been for water pumping, to begin replacing India's four to five million diesel powered water pumps, each consuming about 3.5 kilowatts, and off-grid lighting.

which represents 4-5 million x 3.5 kW ≈ 15 GW of capacity to be replaced.

[anirudh-ramesh]

@corradio, credits for the current version of the GJ parser to @sahitya-pavurala. I've replied to #1079 with details on IN-GJ.

[anirudh-ramesh]

@acomets, good point!

1. Off-grid: Most of these small-scale off-grid deployments don't have a meter, so it's practically impossible to retrieve data from these.
2. Non-utility: The parsers we have right now retrieve data from government-owned transmission line operators - there are private lines as well, and they all don't offer a public real-time data access interface - that's another problem to accurately quantify CO2 footprint. I'll look for these sources of information and the worst case is retrieving data from periodic reports.
3. Solar PV in specific: Several hundred MW have been installed off-grid or on-grid, but several have been idle for reasons such as grid failure, no PPAs, poor maintenance, frequency mismatch etc (I've been studying these of late). I have access to info (hard copies) on who installed what, when and where - should that be of use.

[brunolajoie]

Yeah, offgrid is a very important topic indeed...There are two aims to the electricity map:

1/ Inform citizen/policymakers on their country's power carbon footprint, in which case offgrid is important to account for (but we don't have data to account for it)

2/ Inform electricity consumer of where the power he consumes comes from. Either he is offgrid and he knows it. Else he's connected to the power grid and off-grid does not impact its carbon footprint!

So, all in all, for the moment, let's continue adding on-grid values only until we find good, reliable data for off-grid

[alixunderplatz]

@brunolajoie For my master thesis, I have simulated power generation of PV systems based only on measured global horizontal irradiation (GHI) and ambient temperature using the package solaR for R, which can seperate GHI in direct and diffuse shares and does all the other related yield calculaltion, too. Don't know if something similar exists for python/JSON/... . The live solar potential layer (equal to GHI) is available, so there is a way to calculate estimated generation using date and time, geographic coordinates, and GHI. This could be used as indicator "Wgen/Winstalled" or yield factor in %, so users will see how much may be produced from their off-grid PV in that hour in their area.

[brunolajoie]

Wow nice! In the short term, estimations to fill live data gaps is not on the plan of electricity map, but lets keep that in mind. That could be added later on as adds on indeed

[tmslaine]

Some additional resources:

• electricity bidding zones and regions and states on a map: https://www.iexindia.com/bidareas.aspx?id=31&mid=2
• prices for each bidding zone: https://www.iexindia.com/marketdata/areaprice.aspx
• real-time flows between regions: http://wbs.nldc.in:82/Web_TTC_ATC.aspx

In addition Western Region Load Dispatch Centre has generator level real-time data: http://www.wrldc.in/onlinegenvsact.aspx
Most of these are covered by existing parsers (once OCR for Maharasthra is done), however two of these generators are in Madhya Pradesh: Vindhyachal NTPC (coal 4760MW) and SASAN (coal 3960MW). The generation data for Western Region states on http://www.wrldc.in/onlinestate.aspx is inconsistent with generator level data. The last time I checked both of these coal power plants exceeded the generation listed for Madhya Pradesh.

[alixunderplatz]

Very good news! I found two wonderful maps with capacity data for all the Indian federal states :)

1. This one is as of Nov 2017 (IP as a link ... )
   http://172.104.62.103:8080/CAPACITY/capacity.html
   The breakdown is not the best, as it covers hydro, nuclear, thermal and renewables.
   

2. This dashboard has 10 GW more, so probably more recent data.
   https://npp.gov.in/dashBoard/cp-map-dashboard
   The breakdown also includes wind, biomass and solar :) VERY PROMISING!!! Select the colored larger areas first, then you can select the smaller states:
   

EDIT: I had overlooked there is a thermal capacity breakdown in the orange stripe above the map :)))))
https://npp.gov.in/dashBoard/cp-map-dashboard

[alixunderplatz]

The website of the national power portal https://npp.gov.in/ has some interesting links and apps.

Let's hope one day things like live generation will be centralized here in another dashboard! 🙏

[alixunderplatz]

Market prices for all areas hooray!

The displayed market prices are for the recent quarter hourly block (Don't get confused by the weird 30 minute offset!!! GMT+5:30)
http://www.vidyutpravah.in/

edit: I just saw Oli added this link in his very first comment in this issue 👍

[alixunderplatz]

To keep the ball rolling:

On the website for marginal costs of generation per state
http://meritindia.in/

I may have found a very exciting thing: Live generation for India as a whole:

At least I hope it is:

Unfortunately, the renewables section has no breakdown to wind, bio and solar :/ but the other numbers look very good!
That thermal stuff is mainly coal, luckily gas is a seperate category.

The aggregated data is there, which means it's got to be somewhere else, too! Struggling to find it finally!

plz hlp! 😸

[systemcatch]

When you need something found, put @alixunderplatz on the case!!!!

[alixunderplatz]

For all the interconnectors, some grid maps of the Central Energy Authority of India may be useful:
http://www.cea.nic.in/powermaps_hindi.html

[alixunderplatz]

I think it's worth writing a little script to collect data for some days or a week from this website
http://meritindia.in/
to gather some info on the data presented in the table "ALL INDIA POWER POSITION (MW) [CURRENT]"

• What's the interval of data updates?
• What does the generation profile look like?
• What does the "renewables" profile look like?

Although a higher spatial granularity for the data is desirable in the long run, I have some concerns this will be possible in the near future for all the states in India. The states that are currently featured are all lacking electricity import and export data (some could be added using the maps to determine interconnectors). And the complexity of the Indian power system is extremely high in general.

So I suggest checking this source out and possibly switching to an India-wide parser.
The "Renewable Generation" field is the only con that I see, lacking a breakdown for wind, solar and biomass. "Thermal generation" is almost 100% coal, so no issues with that.

After collecting some data, we can see if assigning the "mixed renewables" to "unknown" or even minimizing the "unknown" by keeping solar, wind and biomass data from the other parsers is an option.

[alixunderplatz]

Here are the results of some manual tracking I did on Friday for http://meritindia.in.

I watched the website in irregular intervals and copied the figures once I had noticed some changes. This does not necessarily represent the interval the numbers are updated. I may have skipped a few datapoints, especially when not tracking it for longer intervals.

In general, the data quality is very good and precise! The update-intervall itself is a bit irregular and goes from around 2 min to maybe 5 to 10 min, but that's not an issue.
I seperated the types in the graphs because of the great gap in the dimensions of "coal and demand" vs. rest.

As previously mentioned, everything that is considered "thermal" can be assigned to coal, which covers around 80% of the demand. Demand can vary by a few GW in a short interval of 2-4 min. During the period of tracking, it was somewhere between 140 to 160 GW. Time on the x-axis is middle european summer time, so add 3:30 h to get local time.

The green line for renewables (RES) is declining in the graph, because of solar component getting smaller and smaller towards the night as expected. Nuclear and gas were on constant levels. Hydro was peaking to cover the rising evening demand peak. Time on the x-axis is middle european summer time, so add 3:30 h to get local time.

Combined view of all types of generation:

And everything stacked (note: the time interval is a mess):

[jarek]

A couple of thoughts:

• with this data we would probably want to introduce "other renewable" category with intensity of something like 50-100 gco2eq/kwh (depending on how we can estimate biomass). or maybe just use "unknown" and give it that intensity - but then it's not included in "% renewable" metric
• compared to individual states as we have now: it would be nice to have data for a larger area, and it gets rid of uncertainty regarding imports/exports, but it'd be a bit of a shame to lose more details on regions with more renewables, e.g. Karnataka had hit 42% renewable in own generation today

[alixunderplatz]

@jarek good points!

1. to avoid a new category and still getting the "% renewable" right, what about making that "other renewables" type a "hidden export zone" with carbon intensity x and renewable share 100%. Like for the country specific export carbon intensity. It would be stacked on top in a bright green and could be labeled "Wind, Solar & Biomass" or "Renewables" (and get a special tooltip):

1.1 Or is there a way to collect all the wind/solar/biomass data as of now in a seperate parser and then apply it to the reduce the "other renewable" category?

1. I agree, it's a pity loosing details for some regions. When I compare Karnataka's production with the "renewables" generation for India as a whole, around 1/5 of it is coming from there right now. I guess subtracting the output of a single state from the entire mix will be too complex and increase the risk to lose data due to errors?

2. On the other hand, unless there will be a reliable dashboard by state showing imports, exports and all the other relevant info, there is no chance of getting every state totally right because of the lack of info for other states. Exchanges are not considered for any of the recently covered states in India. The state of Delhi generates about 900 MW, while the demand is beyond 4 000 MW, but we cannot really determine where that energy is coming from at how it is composed.

3. Power plants belong either to "state sector", to "central sector" or to "IPPs". They are located in a certain state, but other states own and report some generation shares, too. There is a really good map to determine the location of every single power plant, although it's plenty of work. But it's very hard to find relevant interconnector data on state levels and to try to assign and combine all that info without double or triple accounting for something.

4. If the Western/Northern/Southern/Eastern/North Eastern regional dispatch centres had reliable generation and interconnector real-time data for their areas, this was probably the "safest" granularity to go for at the moment. At least the real-time power flows between them are published here: http://wbs.nldc.in:82/Web_TTC_ATC.aspx , as was pointed out above (it's stuck at the moment^^).

[alixunderplatz]

I had a little script running in the last days which was collecting around 36 hours of data. Scrape-interval was 30 seconds. Mostly, the data would change each 2-3 minutes for all categories.
Here are the results:
In the area with dotted lines, I lost my internet connection at night. The peak on the right is an occasional data issue (coal + hydro).

Stacked view:

Categories on their own:

You can clearly see the green renewables curve peaking due to PV (between 6 and 7 UTC = 12:30-13:30 IST), and some RES generation increasing during night from wind power. There were low wind speeds recently, so not much was generated from it at night. As of March 2018, India has 21 GW solar, 34 GW of wind power and 8.7 GW of biomass capacity installed. I'm curious to see what it looks like when it's really windy.

Renewables

120 GW of coal running constantly ... that's

• 1 640 t CO2/min or
• 98 400 t CO2/hour or
• 2 361 600 t CO2/day

OMG 😭 😞 🤢 🌡 🌏 📈

[alixunderplatz]

I just finished collecting data from meritindia.in, and here are the final results for the period that I observed in the last last 3 days. There were some more value peaks due to minor data quality issues yesterday, but nothing to really worry about. This happens everywhere from time to time and could be fixed by setting some limits to skip them. Other than that, the data is unbelievably smoooooooth and updates are very frequent.

Stacked:

Seperated:

@corradio and @brunolajoie What do you guys think about it? Any comments on the suggestions and thoughts from above?
(These data cover about 1/6 of the recent world's population ;-) )

[corradio]

@alixunderplatz it seems like you found A LOT of things. Sorry for not having replied before. The website also seems to have breakdowns by regions also! I would favor continuying by region.
Could you make a short summary of what needs to be done so we can more easily try to get people who want to contribute on the case?

[alixunderplatz]

@corradio What would you do with the "renewable" category (most likely consisting of wind & solar, maybe biomass)?

---

Unfortunately, the regional data on http://meritindia.in for India has no breakdown of types.
It just gives "demand met", "own generation" as a total, and "import".
When you go through all of the states, all show "import" because of the difference of "demand met" and "own generation". Would be nice to have export numbers included to calculate the balance, but that figure is totally missing everywhere. No idea if everything is included in "own generation" or if it is just a part (state owned, IPP, central generation).

---

This site has current demand and current price per state
http://www.vidyutpravah.in/
http://vidyutpravah.in/state-data/gujarat

---

I'd first create a simple parser which collects the breakdown for India as a whole from http://meritindia.in:

No idea on how to continue or apply this to the other states, though.
Is it possible to apply stripes with a color/transparent/color/transparent layer to the states are covered already?

[corradio]

Maybe the plantwise energy summary can be used to derive breakdown, but I realise it's going to be a lot of work:

If we can't split wind/solar from biomass, then unfortunately the only solution is to put it in "unknown" with a decent carbon intensity based on the average yearly mix of that category.

[alixunderplatz]

@corradio the plantwise stuff looks like it's including plants that are outside of the respective states, too.

Check Dehli and expand the details below.
http://meritindia.in/state-data/delhi
For instance, some of the hydro power is located in Kashmir in the NW of India. See "capacity allocated to state" column for all where this is the case. Yes, assigning all units to the respective states and accounting for the correct amount of energy is muuuuch work.
In addition, these numbers summarize the generated energy on a daily level for the previous day :-/

[corradio]

You are completely right. I've been looking at this too quickly.

On Fri 4 May 2018 at 18:20, Alex [e-mail] wrote:

@corradio https://github.com/corradio the plantwise stuff looks like
it's including plants that are outside of the respective states, too.

Check Dehli and expand the details below.
http://meritindia.in/state-data/delhi
For instance, some of the hydro power is located in Kashmir in the NW of
India. See "capacity allocated to state" column for all where this is the
case. Yes, assigning all units to the respective states and accounting for
the correct amount of energy is muuuuch work.
In addition, these numbers summarize the generated energy on a daily
level for the previous day :-/

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tmrowco/electricitymap/issues/304#issuecomment-386652225,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABlEKLKGQ2XkeRd28y0YhJKwlPU_DcH8ks5tvH_BgaJpZM4LpACJ
.

[alixunderplatz]

Live state-wide generation for West Bengal, no fuel type specified:
http://www.wbsldc.in/

Live generation of WBPDCL, which owns [five coal power plants with 4.865 MW]:(https://en.wikipedia.org/wiki/List_of_power_stations_in_West_Bengal#Coal):
https://www.wbpdcl.co.in/cgis/

No data on hydro or renewables,

[systemcatch]

@alixunderplatz maybe we could build a parser that just splits everything into coal and unknown?

[anirudh-ramesh]

@systemcatch and @alixunderplatz, for any region in India, we need to implement a merit-order-of-dispatch algorithm.

In Delhi, for example, we consider all plants located within Delhi and the ones elsewhere that have allocated a portion of power to the Delhi market. There is a legislation that power from renewable energy sources must not be curtailed when available i.e. substituted by a fossil fuel source regardless of cost. Next, we keep adding power from other fuel sources e.g. coal thermal or nuclear or hydroelectric power plants until the real-time demand is met.

Unless we have this algorithm in place, the models calculating carbon emissions intensity will be inaccurate. Right now, the Delhi parser only accounts for <700 MW generation of the >5000 MW production since we are not considering imports from power plants located elsewhere.

[vidya666]

Data on the generation of power from thermal stations in Tamil Nadu is present in https://www.tangedco.gov.in/thermal-stations.html
Data on generation of power from renewable sources like hydro stations and gas-turbines :
https://www.tangedco.gov.in/hydro-stations.html
https://www.tangedco.gov.in/gas-turbine-stations.html

[q--]

https://www.nerldc.org/ has, besides for its own area, also a demand figure for all of India (which is not rounded to GW as it is on http://vidyutpravah.in/)

Day-ahead electricity prices URL has moved to https://www.iexindia.com/maps.aspx

[DineshCyanam]

The base map for India is incorrect. Andhra Pradesh was split into two states in 2014. One being Andhra Pradesh (IN-AP) and the other is Telangana (IN-TG). The data we have here for IN-AP is for the new state of Andhra Pradesh.

IN-AP: http://vidyutpravah.in/state-data/andhra-pradesh
IN-TG: http://vidyutpravah.in/state-data/telangana

[jarek]

See also #1097 and #234 regarding Telangana. Thanks for the confirmation that the data is only for the new borders of IN-AP!

[willbeaufoy]

Hi there, I'd like to contribute a parser. You do not yet have one for Himachal Pradesh, and I have found real time data. It does not list generation by source, but does list it by power plant (GENERATION OF HP(Z) table), and from googling them they are all hydropower (though I'm not sure yet what the last three in the list are). Is this an acceptable data source? If so I can build the parser.

[FelixDQ]

Hi guys. @willbeaufoy made a PR with data from Himachal Pradesh. It would be awesome if some of you have any input for the interpretation on the data. Check it out here: https://github.com/tmrowco/electricitymap-contrib/pull/2654