Covid_19: Adapt LU-Scraper to overwrite incomplete data

Created on 29 Apr 2020  路  6Comments  路  Source: openZH/covid_19

Lucerne publishes different figures at different times. So when the scraper picks up the values at first not all numbers are already available. It would make sense to adapt the scraper to pick up the data again again and overwrite the incomplete entry for a given date when all values have been reported, in order that the missing values must not be added manually. Additionally, the timestamp should be removed (left empty).
scraper
https://github.com/openZH/covid_19/blob/master/scrapers/scrape_lu.py
workflow-file
https://github.com/openZH/covid_19/blob/master/.github/workflows/run_lu_scraper.yml

Ideally, it should be possible to (de-)activate the overwrite process in the workflow file, by setting SCRAPER_OVERWRITE=yes, as already possible for other Cantons:

https://github.com/openZH/covid_19/blob/master/.github/workflows/run_fr_scraper.yml#L32

scraper

Most helpful comment

@jb3-2 @simgraworldwide timestamp removed with ; https://github.com/openZH/covid_19/commit/7266107ca881e836bbdaece0b54f9a3d79da2c47 . Let's observe if it works well.

All 6 comments

@tlorusso Do you know if Lucerne publishes somewhere a full data list with historical data points ?

@dominikgehl unfortunately they don't. And its unlikely that this will change in the short term.

@dominikgehl i guess we should only invest time in adapting the Scraper if someone finds an easy and quick way of handling all the problems that arise from scraping the numbers as they are published now. Otherwise (partial) manual updating migh be more efficient.

Also, I suggest to remove the timestamp (since they publish their data with two different ones) completely.

I have set the scraper to overwrite historical data with this commit: https://github.com/openZH/covid_19/commit/7b7391c9c309217b577f344c6fa9433d6dae0da6. Let's see if that helps.

@jb3-2 @simgraworldwide timestamp removed with ; https://github.com/openZH/covid_19/commit/7266107ca881e836bbdaece0b54f9a3d79da2c47 . Let's observe if it works well.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

simgraworldwide picture simgraworldwide  路  6Comments

Pseudomanifold picture Pseudomanifold  路  6Comments

baryluk picture baryluk  路  3Comments

MeMent0L picture MeMent0L  路  8Comments

herbertvonkaramalz picture herbertvonkaramalz  路  7Comments