Covid_19: add CSV option for meta_scrape.sh to create CSV file.

[zdavatz]

@baryluk how do you create the CSV file? Would be nice if there was an option:

meta_scrape.sh --csv to be able to do meta_scrape.sh --csv > current.csv

I can see this commit: https://github.com/baryluk/covid_19/commit/cd1934e961183759aeb294a9ce3e2581e1d1aeb0 but I do not see how you create the CSV output file. Would be great if that is documented somewhere.

[fabian]

The CSV is created with sqlite3 here:

https://github.com/openZH/covid_19/blob/40d7d7f2cd03ba34a0b80694c20103ed33fff4b7/scrapers/run_scraper.sh#L31-L32

[zdavatz]

Thank you @fabian

~/.software/covid_19/scrapers> ./run_scraper.sh
SCRAPER_KEY and SCRAPER_SOURCE env variables must be set

this will not run. Would be great to have some documentation how to use it.

[fabian]

@zdavatz as per error message, you must set the environment variables, e.g.

SCRAPER_KEY=TI SCRAPER_SOURCE=ti.ch scrapers/run_scraper.sh

[zdavatz]

yes, ok thank you @fabian, but that will not give a CSV file with all the entries.

I opened the issue, because something like this would be nicer and more useful:

meta_scrape.sh --csv > current.csv

[baryluk]

@zdavatz The https://www.functor.xyz/covid_19/scrapers/outputs/latest.csv is generated from https://www.functor.xyz/covid_19/scrapers/outputs/latest.txt by a simple conversion script, but it is not published or merged into this repo. It is not using run_scaper.sh or add_db_entry.py, but it does use currently sqlite too. I want to rewrite it to not need sqlite at all, and convert to CSV on the fly.

It is not documented, because https://www.functor.xyz/covid_19/scrapers/outputs/latest.csv is by no means official. :) This repo is and CSV files in it are.

The problem with run_scraper.sh is that it only run on scraper at a time, where meta_scrape.sh and my to csv converter, runs all of them, and populates sources automatically.

So I think it is useful to be merged. I just didn't have time to clean it up (like not use absolute paths that only work on my computer).

[baryluk]

yes, ok thank you @fabian, but that will not give a CSV file with all the entries.

I opened the issue, because something like this would be nicer and more useful:

meta_scrape.sh --csv > current.csv

Yep. That is the plan, more like meta_scarpe.sh | convert_to_csv.py > current.csv. But yes.

[zdavatz]

Yep. That is the plan, more like meta_scarpe.sh | convert_to_csv.py > current.csv. But yes.

Cool, looking forward to that! Thank you @baryluk

[zdavatz]

works great, thank you!

[baryluk]

Extras (hospitalized, ICU, Ventilated, Released) should also work now:

$ curl https://www.functor.xyz/covid_19/scrapers/outputs/latest.csv
date,time,abbreviation_canton_and_fl,ncumul_tested,ncumul_conf,ncumul_hosp,ncumul_ICU,ncumul_vent,ncumul_released,ncumul_deceased,source
2020-03-28,18:00,AG,,364,50,12,9,,3,Scraper for AG at 2020-03-28T21:05:21+01:00 using https://www.ag.ch/de/themen_1/coronavirus_2/alle_ereignisse/alle_ereignisse_1.jsp
2020-03-27,18:00,AI,,12,,,,,,Scraper for AI at 2020-03-28T21:05:24+01:00 using https://www.ai.ch/themen/gesundheit-alter-und-soziales/gesundheitsfoerderung-und-praevention/uebertragbare-krankheiten/coronavirus
2020-03-28,08:00,AR,,45,,,,,2,Scraper for AR at 2020-03-28T21:05:30+01:00 using https://www.ar.ch/verwaltung/departement-gesundheit-und-soziales/amt-fuer-gesundheit/informationsseite-coronavirus/
2020-03-28,,BE,,767,,,,,9,Scraper for BE at 2020-03-28T21:05:41+01:00 using https://www.besondere-lage.sites.be.ch/besondere-lage_sites/de/index/corona/index.html
2020-03-28,,BL,,502,,,,,5,Scraper for BL at 2020-03-28T21:05:43+01:00 using https://www.statistik.bl.ch/files/sites/Grafiken/COVID19/Grafik_COVID19_BL_Linie.htm
2020-03-28,10:00,BS,,573,,,,,,"Scraper for BS at 2020-03-28T21:05:50+01:00 using https://www.gd.bs.ch/, https://www.gd.bs.ch//nm/2020-tagesbulletin-coronavirus-573-bestaetigte-faelle-im-kanton-basel-stadt-gd.html"
2020-03-28,,FR,,421,67,,,,15,Scraper for FR at 2020-03-28T21:05:54+01:00 using https://www.fr.ch/covid19/sante/covid-19/coronavirus-statistiques-evolution-de-la-situation-dans-le-canton
2020-03-27,12:00,GE,,1924,313,54,,,23,Scraper for GE at 2020-03-28T21:06:06+01:00 using https://www.ge.ch/document/point-coronavirus-maladie-covid-19/telecharger
2020-03-28,13:30,GL,,47,3,,,,,Scraper for GL at 2020-03-28T21:06:12+01:00 using https://www.gl.ch/verwaltung/finanzen-und-gesundheit/gesundheit/coronavirus.html/4817
2020-03-27,,GR,,409,52,,,,9,"Scraper for GR at 2020-03-28T21:06:25+01:00 using https://www.gr.ch/DE/institutionen/verwaltung/djsg/ga/coronavirus/info/Seiten/Start.aspx, https://www.gr.ch/DE/institutionen/verwaltung/djsg/ga/coronavirus/_layouts/15/GenericDataFeed/feed.aspx?PageID=26&ID=g_1175d522_e609_4287_93af_d14c9efd5218&FORMAT=JSONRAW"
2020-03-28,16:00,JU,,118,23,5,,,,"Scraper for JU at 2020-03-28T21:06:33+01:00 using https://www.jura.ch/fr/Autorites/Coronavirus/Accueil/Coronavirus-Informations-officielles-a-la-population-jurassienne.html, https://www.jura.ch/fr/Autorites/Coronavirus/Chiffres-H-JU/Evolution-des-cas-COVID-19-dans-le-Jura.html"
2020-03-28,11:00,LU,,317,,,,,4,Scraper for LU at 2020-03-28T21:06:41+01:00 using https://gesundheit.lu.ch/themen/Humanmedizin/Infektionskrankheiten/Coronavirus
2020-03-28,14:00,NE,,311,,,,,6,Scraper for NE at 2020-03-28T21:06:45+01:00 using https://www.ne.ch/autorites/DFS/SCSP/medecin-cantonal/maladies-vaccinations/Pages/Coronavirus.aspx
2020-03-28,15:30,NW,,55,,,,,0,Scraper for NW at 2020-03-28T21:06:52+01:00 using https://www.nw.ch/gesundheitsamtdienste/6044
2020-03-27,,OW,,37,,,,,,Scraper for OW at 2020-03-28T21:06:56+01:00 using https://www.ow.ch/de/verwaltung/dienstleistungen/?dienst_id=5962
2020-03-28,,SG,,339,,,,,5,Scraper for SG at 2020-03-28T21:07:01+01:00 using https://www.sg.ch/tools/informationen-coronavirus.html
2020-03-28,07:30,SH,,37,,,,,,Scraper for SH at 2020-03-28T21:07:06+01:00 using https://sh.ch/CMS/content.jsp?contentid=3209198&language=DE&_=1584807070095
2020-03-28,00:00,SO,,173,,,,,1,Scraper for SO at 2020-03-28T21:07:09+01:00 using https://corona.so.ch/
2020-03-28,,SZ,,122,,,,33,,Scraper for SZ at 2020-03-28T21:07:14+01:00 using https://www.sz.ch/behoerden/information-medien/medienmitteilungen/coronavirus.html/72-416-412-1379-6948
2020-03-28,,TG,,134,,,,,,Scraper for TG at 2020-03-28T21:07:17+01:00 using https://www.tg.ch/news/fachdossier-coronavirus.html/10552
2020-03-28,08:00,TI,,1727,385,69,60,,87,"Scraper for TI at 2020-03-28T21:07:32+01:00 using https://www4.ti.ch/dss/dsp/covid19/home/, https://www4.ti.ch/area-media/comunicati/, https://www4.ti.ch/area-media/comunicati/dettaglio-comunicato/?NEWS_ID=187538&tx_tichareamedia_comunicazioni%5Baction%5D=show&tx_tichareamedia_comunicazioni%5Bcontroller%5D=Comunicazioni&cHash=736a7f1a780ed9bdb1d8e45f8a0326ff"
2020-03-27,12:00,UR,,48,7,,,3,0,Scraper for UR at 2020-03-28T21:07:35+01:00 using https://www.ur.ch/themen/2962
2020-03-27,,VD,,2745,370,,,174,48,Scraper for VD at 2020-03-28T21:07:35+01:00 using https://api.datawrapper.de/v3/charts/tr5bJ/data
2020-03-28,,VS,,902,,,,,21,Scraper for VS at 2020-03-28T21:07:40+01:00 using https://www.vs.ch/de/web/coronavirus
2020-03-27,18:00,ZG,,101,,,,18,1,Scraper for ZG at 2020-03-28T21:07:44+01:00 using https://www.zg.ch/behoerden/gesundheitsdirektion/amt-fuer-gesundheit/corona
2020-03-28,14:00,ZH,,1720,187,,42,,15,Scraper for ZH at 2020-03-28T21:07:46+01:00 using https://gd.zh.ch/internet/gesundheitsdirektion/de/themen/coronavirus.html

[zdavatz]

Double cool. We are building this map now from the CSV: https://github.com/zdavatz/covid2019_ch_map/ - data is also used on an hourly basis here: http://covid19.ddrobotec.com/ with new doubling time being calculated. Great work @baryluk and everybody!

[zdavatz]

this is the map being created: https://twitter.com/zdavatz/status/1244016259924189186?s=20 - if all is read, no scrapers have failed ;). yay! ;)