Covid_19: Port scrapers from bash to Python

[baryluk]

Bash is nice and it was quick to add many scraper, but we do see many failures, and it leads to some issues.

1. Not many people are familiar with bash, grep, sed, awk, to help with fix contribution.
2. Currently we use set -e (without set -x), which means if the scraper fails, it leaves no useful error messages.
3. Due to nature of sed, it often needs to be protected by extraneous and somehow redundant grep or head -1 around it. This is because if the sed doesn't match anything on the line, it will output the line as is, malforming the output.

Use of Python (we can port one scraper at a time) will help:

1. More familiar.
2. Easier use of functions, logging, and exception messages, which should help with debugging.
3. Instead of using grep+sed+tricks, or sed with s/^.*....().....*$/\1/ trickery, we can just use something like re.search("...()...", d)[1], or a helper function (shared by scrapers) to make it ever easier. This should actually make shorter and cleaner script (no redundant grep+sed), that is also a bit more robust against parsing of multiple lines (no need for head -1, and eliminate some other steps that we do in bash right now.
4. Python has raw strings, which helps with regexp writing, due to reduction of backslash escaping needed.

[zdavatz]

Ok! Sounds like a plan although awk is great, as Linus says ;).

[metaodi]

The scraper for SG that is run by GitHub actions is already written in Python: https://github.com/openZH/covid_19/blob/master/scrapers/scraper_canton_sg.py

And i still have some floating around from morph.io, so I can simply move them here.

[baryluk]

The script you have is not just a scraper (it does few other things), and is 5 times longer than needs to be. The same can be achieved in just few lines of code.

I will port most of he bash scrapers to Python today, it is a very easy process. Nothing fancy.

[metaodi]

@baryluk I agree it's longer than needed (the scraper part being from L35-L52). But IMHO the sqlite part really makes sense to manage input and output:

• it defines a clear schema
• I can add constraints (like a UNIQUE) and therefore make sure to only generate one entry per date + time
• make sure the data in the CSV file is always used as an input (and if it's invalid it breaks)

Feel free to rewrite it shorter, but please be aware that we need more than "just a scraper", the result must be added to the existing file without losing information (e.g. if someone manually added data to a column the scraper does not provide).

[baryluk]

@metaodi Yes, it is just a job of other scripts to do that.

The scraper output can easily be consumed by others (after paring by parser).

Here is the prototype for ZH: https://github.com/baryluk/covid_19/commit/72de5616652280d6ba9b306d3ff68a776572a725 / https://github.com/baryluk/covid_19/blob/master/scrapers/scrape_zh.sh

[metaodi]

Oh okay. This is really just a 1:1 port to python without using any library or idioms.

But it's an okayish first step, let's iterate if needed :wink:

[baryluk]

I would say it is very idiomatic. KISS principle. :wink:

[baryluk]

PR for the library and ZH scraper sent: https://github.com/openZH/covid_19/pull/158

I also have AI scraper done using same library. It is really short and readable.

[zdavatz]

This output should only contain the successful data, not the errors or the error output: https://www.functor.xyz/covid_19/scrapers/outputs/latest.txt

If the script fails, then just add a timestamp when it failed, same as with the successful one.

See also: https://github.com/zdavatz/covid19_ch/issues/17

Our Dashboard is now updating every hour and it make no sense to write a parser for the parser. Also see the left side of our Dashboard. We now publish the sources of all the data that OpenZH have in their files.

[ebeusch]

If it's worth anything: https://github.com/ebeusch/covid-19_scraper_LU
here's my python attempt for LU from the past weekend. maybe you can use one line or two ^^'
Note: my script does not account for deceased individuals as there had been none at that point in time. At the moment LU follows German language rules of writing numbers<10 as words. A scraper should potentially account for them switching to number later on. (But I'd rather they never have that many deaths ofc.)

[metaodi]

I close this issue. The ZH scraper was successfully migrated from shell to python. And AI has been implemented in the same manner.

Finally with SG we have yet another python scraper with a different approach. Feel free to migrate or implement other scrapers as well.

[baryluk]

@metaodi Could you please leave it open for a moment. I have almost all existing scrapers ported (the only ones not ported are the ones that do use extra pdf conversion, which is the case for just few parsers at the moment), just didn't make a PR for it, because of some other PRs in progress, and issues with merges.

[baryluk]

@ebeusch I will take a look. It might be useful to have parser for some number as words. For example https://www.ur.ch/themen/2920 for UR, states number of deaths as a word.

[baryluk]

PDFs are also now supported in Python3 scraper library.

Porting status (not merged yet into this repo):

scrape_ag.sh:#!/usr/bin/env python3
scrape_ai.sh:#!/usr/bin/env python3
scrape_ar.sh:#!/usr/bin/env python3
scrape_be.sh:#!/usr/bin/env python3
scrape_bl.sh:#!/usr/bin/env python3
scrape_bs.sh:#!/usr/bin/env python3
scrape_ge.sh:#!/usr/bin/env python3  (pdf)
scrape_gl.sh:#!/usr/bin/env python3
scrape_gr.sh:#!/usr/bin/env python3
scrape_ju.sh:#!/usr/bin/env python3
scrape_lu.sh:#!/usr/bin/env python3
scrape_ne.sh:#!/usr/bin/env python3
scrape_nw.sh:#!/usr/bin/env python3
scrape_ow.sh:#!/usr/bin/env python3
scrape_sg.sh:#!/usr/bin/env python3
scrape_sh.sh:#!/usr/bin/env python3
scrape_so.sh:#!/usr/bin/env python3
scrape_sz.sh:#!/usr/bin/env python3
scrape_tg.sh:#!/usr/bin/env python3
scrape_ti.sh:#!/usr/bin/env python3
scrape_ur.sh:#!/usr/bin/env python3
scrape_vd.sh:#!/bin/sh
scrape_vs.sh:#!/usr/bin/env python3
scrape_zg.sh:#!/usr/bin/env python3
scrape_zh.sh:#!/usr/bin/env python3

[zdavatz]

great work, @baryluk - just tested your pythonize branch. Output looks nice with the URLs. Please put a timestime when the scraper fails, same as with the other entries.

~/.software/covid_19/scrapers> ./meta_scrape.sh 
AG 2020-03-25T15:15     319       2 OK 2020-03-26T09:28:32+01:00 https://www.ag.ch/de/themen_1/coronavirus_2/alle_ereignisse/alle_ereignisse_1.jsp
AI 2020-03-28T17:00       9       - OK 2020-03-26T09:28:33+01:00 https://www.ai.ch/themen/gesundheit-alter-und-soziales/gesundheitsfoerderung-und-praevention/uebertragbare-krankheiten/coronavirus
AR 2020-03-26T09:00      40       2 OK 2020-03-26T09:28:33+01:00 https://www.ar.ch/verwaltung/departement-gesundheit-und-soziales/amt-fuer-gesundheit/informationsseite-coronavirus/
BE 2020-03-26T          660       7 OK 2020-03-26T09:28:33+01:00 https://www.besondere-lage.sites.be.ch/besondere-lage_sites/de/index/corona/index.html
BL 2020-03-25T          341       5 OK 2020-03-26T09:28:33+01:00 https://www.statistik.bl.ch/files/sites/Grafiken/COVID19/Grafik_COVID19_BL_Linie.htm
BS 2020-03-25T10:00     466       - OK 2020-03-26T09:28:34+01:00 https://www.gd.bs.ch/, https://www.gd.bs.ch//nm/2020-tagesbulletin-coronavirus-466-bestaetigte-faelle-im-kanton-basel-stadt-gd.html
GE 2020-03-25T12:00    1604      15 OK 2020-03-26T09:28:34+01:00 
GL 2020-03-25T13:30      40       - OK 2020-03-26T09:28:34+01:00 https://www.gl.ch/verwaltung/finanzen-und-gesundheit/gesundheit/coronavirus.html/4817
GR 2020-03-25T          322       6 OK 2020-03-26T09:28:35+01:00 https://www.gr.ch/DE/institutionen/verwaltung/djsg/ga/coronavirus/info/Seiten/Start.aspx
JU 2020-03-25T18:00      78       - OK 2020-03-26T09:28:36+01:00 https://www.jura.ch/fr/Autorites/Coronavirus/Accueil/Coronavirus-Informations-officielles-a-la-population-jurassienne.html
LU 2020-03-25T11:00     228       - OK 2020-03-26T09:28:36+01:00 https://gesundheit.lu.ch/themen/Humanmedizin/Infektionskrankheiten/Coronavirus
NE 2020-03-25T14:00     256       2 OK 2020-03-26T09:28:36+01:00 https://www.ne.ch/autorites/DFS/SCSP/medecin-cantonal/maladies-vaccinations/Pages/Coronavirus.aspx
Error: invalid literal for int() with base 10: 'None'
Traceback (most recent call last):
  File "./parse_scrape_output.py", line 180, in <module>
    cases = int(v)
ValueError: invalid literal for int() with base 10: 'None'

NW - - - FAILED
OW 2020-03-25T           27       - OK 2020-03-26T09:28:37+01:00 https://www.ow.ch/de/verwaltung/dienstleistungen/?dienst_id=5962
SG 2020-03-26T          280       1 OK 2020-03-26T09:28:37+01:00 https://www.sg.ch/tools/informationen-coronavirus.html
SH 2020-03-26T07:00      35       - OK 2020-03-26T09:28:37+01:00 https://sh.ch/CMS/content.jsp?contentid=3209198&language=DE&_=1584807070095
SO 2020-03-26T00:00     141       1 OK 2020-03-26T09:28:38+01:00 https://corona.so.ch/
SZ 2020-03-25T           99       - OK 2020-03-26T09:28:38+01:00 https://www.sz.ch/behoerden/information-medien/medienmitteilungen/coronavirus.html/72-416-412-1379-6948
TG 2020-03-25T           96       - OK 2020-03-26T09:28:38+01:00 https://www.tg.ch/news/fachdossier-coronavirus.html/10552
TI 2020-03-25T08:00    1354      60 OK 2020-03-26T09:28:39+01:00 https://www4.ti.ch/dss/dsp/covid19/home/
UR 2020-03-24T12:00      25       - OK 2020-03-26T09:28:39+01:00 https://www.ur.ch/themen/2920
Error: Unknown date/time format: Date\tHospitalisations
Traceback (most recent call last):
  File "./parse_scrape_output.py", line 177, in <module>
    date = parse_date(v)
  File "./parse_scrape_output.py", line 143, in parse_date
    assert False, f"Unknown date/time format: {d}"
AssertionError: Unknown date/time format: Date\tHospitalisations

VD - - - FAILED
VS 2020-03-25T          651      14 OK 2020-03-26T09:28:40+01:00 https://www.vs.ch/de/web/coronavirus
ZG 2020-03-26T08:00      87       0 OK 2020-03-26T09:28:40+01:00 https://www.zg.ch/behoerden/gesundheitsdirektion/amt-fuer-gesundheit/corona
ZH 2020-03-25T09:30    1363       7 OK 2020-03-26T09:28:40+01:00 https://gd.zh.ch/internet/gesundheitsdirektion/de/themen/coronavirus.html

[baryluk]

Ack. Good. Thanks for testing.

VD works for me.

What kind of shell are you using? (ls -l /bin/sh)

I already sent fix for NW (both bash and python version, and it is in the pythonize branch). Their page was updated ~2 hours ago.

I also just updated the GE to also show URL. It was not showing because of slightly different logic in PDF downloader. Now it should be same as others.

I will address timestamp for failures next, but in a separate issue.

[zdavatz]

ls -l /bin/sh

/bin/sh -> bash

[baryluk]

ls -l /bin/sh

/bin/sh -> bash

@zdavatz Ok. Filled a bug about it. https://github.com/openZH/covid_19/issues/205 It can be fixed to work in bash too probably. Or we can make it work in Python too. It is the last one not implemented in Python.

[zdavatz]

Request for failed timestamp issue is here: https://github.com/openZH/covid_19/issues/206