Covid_19: track timestamps for failures

Created on 26 Mar 2020  路  7Comments  路  Source: openZH/covid_19

As requested, please track timestamps for failures, see

Use Case: Please always print a timestamp, it does not matter if the scraper fails or not. Just always print a timestamp.

~/.software/covid_19/scrapers> ./meta_scrape.sh 
AG 2020-03-25T15:15     319       2 OK 2020-03-26T10:10:21+01:00 https://www.ag.ch/de/themen_1/coronavirus_2/alle_ereignisse/alle_ereignisse_1.jsp
AI 2020-03-28T17:00       9       - OK 2020-03-26T10:10:21+01:00 https://www.ai.ch/themen/gesundheit-alter-und-soziales/gesundheitsfoerderung-und-praevention/uebertragbare-krankheiten/coronavirus
AR 2020-03-26T09:00      40       2 OK 2020-03-26T10:10:21+01:00 https://www.ar.ch/verwaltung/departement-gesundheit-und-soziales/amt-fuer-gesundheit/informationsseite-coronavirus/
BE 2020-03-26T          660       7 OK 2020-03-26T10:10:22+01:00 https://www.besondere-lage.sites.be.ch/besondere-lage_sites/de/index/corona/index.html
BL 2020-03-25T          341       5 OK 2020-03-26T10:10:22+01:00 https://www.statistik.bl.ch/files/sites/Grafiken/COVID19/Grafik_COVID19_BL_Linie.htm
BS 2020-03-25T10:00     466       - OK 2020-03-26T10:10:22+01:00 https://www.gd.bs.ch/, https://www.gd.bs.ch//nm/2020-tagesbulletin-coronavirus-466-bestaetigte-faelle-im-kanton-basel-stadt-gd.html
GE 2020-03-25T12:00    1604      15 OK 2020-03-26T10:10:23+01:00 https://www.ge.ch/document/point-coronavirus-maladie-covid-19/telecharger
GL 2020-03-25T13:30      40       - OK 2020-03-26T10:10:23+01:00 https://www.gl.ch/verwaltung/finanzen-und-gesundheit/gesundheit/coronavirus.html/4817
GR 2020-03-25T          322       6 OK 2020-03-26T10:10:24+01:00 https://www.gr.ch/DE/institutionen/verwaltung/djsg/ga/coronavirus/info/Seiten/Start.aspx
JU 2020-03-25T18:00      78       - OK 2020-03-26T10:10:25+01:00 https://www.jura.ch/fr/Autorites/Coronavirus/Accueil/Coronavirus-Informations-officielles-a-la-population-jurassienne.html
LU 2020-03-25T11:00     228       - OK 2020-03-26T10:10:25+01:00 https://gesundheit.lu.ch/themen/Humanmedizin/Infektionskrankheiten/Coronavirus
NE 2020-03-25T14:00     256       2 OK 2020-03-26T10:10:25+01:00 https://www.ne.ch/autorites/DFS/SCSP/medecin-cantonal/maladies-vaccinations/Pages/Coronavirus.aspx
NW 2020-03-25T15:30      44       0 OK 2020-03-26T10:10:25+01:00 https://www.nw.ch/gesundheitsamtdienste/6044
OW 2020-03-25T           27       - OK 2020-03-26T10:10:26+01:00 https://www.ow.ch/de/verwaltung/dienstleistungen/?dienst_id=5962
SG 2020-03-26T          280       1 OK 2020-03-26T10:10:27+01:00 https://www.sg.ch/tools/informationen-coronavirus.html
SH 2020-03-26T07:00      35       - OK 2020-03-26T10:10:27+01:00 https://sh.ch/CMS/content.jsp?contentid=3209198&language=DE&_=1584807070095
SO 2020-03-26T00:00     141       1 OK 2020-03-26T10:10:27+01:00 https://corona.so.ch/
SZ 2020-03-25T           99       - OK 2020-03-26T10:10:27+01:00 https://www.sz.ch/behoerden/information-medien/medienmitteilungen/coronavirus.html/72-416-412-1379-6948
TG 2020-03-25T           96       - OK 2020-03-26T10:10:28+01:00 https://www.tg.ch/news/fachdossier-coronavirus.html/10552
TI 2020-03-25T08:00    1354      60 OK 2020-03-26T10:10:28+01:00 https://www4.ti.ch/dss/dsp/covid19/home/
UR 2020-03-24T12:00      25       - OK 2020-03-26T10:10:28+01:00 https://www.ur.ch/themen/2920
Error: Unknown date/time format: Date\tHospitalisations
Traceback (most recent call last):
  File "./parse_scrape_output.py", line 177, in <module>
    date = parse_date(v)
  File "./parse_scrape_output.py", line 143, in parse_date
    assert False, f"Unknown date/time format: {d}"
AssertionError: Unknown date/time format: Date\tHospitalisations

VD - - - FAILED
VS 2020-03-25T          651      14 OK 2020-03-26T10:10:29+01:00 https://www.vs.ch/de/web/coronavirus
ZG 2020-03-26T08:00      87       0 OK 2020-03-26T10:10:29+01:00 https://www.zg.ch/behoerden/gesundheitsdirektion/amt-fuer-gesundheit/corona
ZH 2020-03-25T09:30    1363       7 OK 2020-03-26T10:10:29+01:00 https://gd.zh.ch/internet/gesundheitsdirektion/de/themen/coronavirus.html
picture zdavatz

Most helpful comment

Can't you just print "date" if the output is "Failed"?

Can be done. Can you send PR request with something like this:

--- a/scrapers/meta_scrape.sh
+++ b/scrapers/meta_scrape.sh
@@ -30,6 +30,6 @@ for s in ./scrape_*.sh;
 do
   if ! ./$s | ./parse_scrape_output.py 2>/dev/null; then
     a=$(echo "$s" | sed -E -e 's/^.*scrape_(..)\..*$/\1/' | tr a-z A-Z) # ' # To make my editor happy.
-    echo "$a" - - - FAILED
+    echo "$a" - - - FAILED "$(date --iso-8601=seconds)"
   fi
 done

I have minor issues with git at the moment.

picture baryluk on 26 Mar 2020
👀1 ❤1 🎉1 😄1 👍1

All 7 comments

Ack. I agree it is useful to have.

The issue is, it is not technically easy ;) FAILED can be due to multitude of reasons, and is printed by the top level meta_scrape.sh, which is different what happens for timestamps when it prints OK (it is done by scraper and parser).

Can be done, but it can take a day or two before we fix it. We need to get some other PRs merged first to make it work.

baryluk picture baryluk on 26 Mar 2020

Can't you just print "date" if the output is "Failed"?

zdavatz picture zdavatz on 26 Mar 2020

Can't you just print "date" if the output is "Failed"?

Can be done. Can you send PR request with something like this:

--- a/scrapers/meta_scrape.sh
+++ b/scrapers/meta_scrape.sh
@@ -30,6 +30,6 @@ for s in ./scrape_*.sh;
 do
   if ! ./$s | ./parse_scrape_output.py 2>/dev/null; then
     a=$(echo "$s" | sed -E -e 's/^.*scrape_(..)\..*$/\1/' | tr a-z A-Z) # ' # To make my editor happy.
-    echo "$a" - - - FAILED
+    echo "$a" - - - FAILED "$(date --iso-8601=seconds)"
   fi
 done

I have minor issues with git at the moment.

baryluk picture baryluk on 26 Mar 2020
👀1 ❤1 🎉1 😄1 👍1

Can you pull from here: https://github.com/zdavatz/covid_19/blob/master/scrapers/meta_scrape.sh? I applied your patch.

zdavatz picture zdavatz on 26 Mar 2020

patch works:

~/.software/covid_19/scrapers> ./meta_scrape.sh 
AG 2020-03-25T15:15     319       2 OK 2020-03-26T10:44:23+01:00 https://www.ag.ch/de/themen_1/coronavirus_2/alle_ereignisse/alle_ereignisse_1.jsp
AI 2020-03-28T17:00       9       - OK 2020-03-26T10:44:23+01:00 https://www.ai.ch/themen/gesundheit-alter-und-soziales/gesundheitsfoerderung-und-praevention/uebertragbare-krankheiten/coronavirus
AR 2020-03-26T09:00      40       2 OK 2020-03-26T10:44:23+01:00 https://www.ar.ch/verwaltung/departement-gesundheit-und-soziales/amt-fuer-gesundheit/informationsseite-coronavirus/
BE 2020-03-26T          660       7 OK 2020-03-26T10:44:24+01:00 https://www.besondere-lage.sites.be.ch/besondere-lage_sites/de/index/corona/index.html
BL 2020-03-25T          341       5 OK 2020-03-26T10:44:24+01:00 https://www.statistik.bl.ch/files/sites/Grafiken/COVID19/Grafik_COVID19_BL_Linie.htm
BS 2020-03-25T10:00     466       - OK 2020-03-26T10:44:24+01:00 https://www.gd.bs.ch/, https://www.gd.bs.ch//nm/2020-tagesbulletin-coronavirus-466-bestaetigte-faelle-im-kanton-basel-stadt-gd.html
GE 2020-03-25T12:00    1604      15 OK 2020-03-26T10:44:25+01:00 https://www.ge.ch/document/point-coronavirus-maladie-covid-19/telecharger
GL 2020-03-25T13:30      40       - OK 2020-03-26T10:44:25+01:00 https://www.gl.ch/verwaltung/finanzen-und-gesundheit/gesundheit/coronavirus.html/4817
GR 2020-03-26T          373       9 OK 2020-03-26T10:44:26+01:00 https://www.gr.ch/DE/institutionen/verwaltung/djsg/ga/coronavirus/info/Seiten/Start.aspx
JU 2020-03-25T18:00      78       - OK 2020-03-26T10:44:26+01:00 https://www.jura.ch/fr/Autorites/Coronavirus/Accueil/Coronavirus-Informations-officielles-a-la-population-jurassienne.html
LU 2020-03-25T11:00     228       - OK 2020-03-26T10:44:26+01:00 https://gesundheit.lu.ch/themen/Humanmedizin/Infektionskrankheiten/Coronavirus
NE 2020-03-25T14:00     256       2 OK 2020-03-26T10:44:27+01:00 https://www.ne.ch/autorites/DFS/SCSP/medecin-cantonal/maladies-vaccinations/Pages/Coronavirus.aspx
NW 2020-03-25T15:30      44       0 OK 2020-03-26T10:44:27+01:00 https://www.nw.ch/gesundheitsamtdienste/6044
OW 2020-03-25T           27       - OK 2020-03-26T10:44:28+01:00 https://www.ow.ch/de/verwaltung/dienstleistungen/?dienst_id=5962
SG 2020-03-26T          280       1 OK 2020-03-26T10:44:28+01:00 https://www.sg.ch/tools/informationen-coronavirus.html
SH 2020-03-26T07:00      35       - OK 2020-03-26T10:44:28+01:00 https://sh.ch/CMS/content.jsp?contentid=3209198&language=DE&_=1584807070095
SO 2020-03-26T00:00     141       1 OK 2020-03-26T10:44:28+01:00 https://corona.so.ch/
SZ 2020-03-25T           99       - OK 2020-03-26T10:44:29+01:00 https://www.sz.ch/behoerden/information-medien/medienmitteilungen/coronavirus.html/72-416-412-1379-6948
TG 2020-03-25T           96       - OK 2020-03-26T10:44:29+01:00 https://www.tg.ch/news/fachdossier-coronavirus.html/10552
TI 2020-03-26T08:00    1401      67 OK 2020-03-26T10:44:29+01:00 https://www4.ti.ch/dss/dsp/covid19/home/
UR 2020-03-24T12:00      25       - OK 2020-03-26T10:44:30+01:00 https://www.ur.ch/themen/2920
Error: Unknown date/time format: Date\tHospitalisations
Traceback (most recent call last):
  File "./parse_scrape_output.py", line 177, in <module>
    date = parse_date(v)
  File "./parse_scrape_output.py", line 143, in parse_date
    assert False, f"Unknown date/time format: {d}"
AssertionError: Unknown date/time format: Date\tHospitalisations

VD - - - FAILED 2020-03-26T10:44:30+01:00
VS 2020-03-25T          651      14 OK 2020-03-26T10:44:30+01:00 https://www.vs.ch/de/web/coronavirus
ZG 2020-03-26T08:00      87       0 OK 2020-03-26T10:44:30+01:00 https://www.zg.ch/behoerden/gesundheitsdirektion/amt-fuer-gesundheit/corona
ZH 2020-03-25T09:30    1363       7 OK 2020-03-26T10:44:30+01:00 https://gd.zh.ch/internet/gesundheitsdirektion/de/themen/coronavirus.html
zdavatz picture zdavatz on 26 Mar 2020

@zdavatz I know it works. Did you send a PR?

baryluk picture baryluk on 26 Mar 2020

@baryluk no sorry, my branch is way of with lots of other stuff in it. How can I send you a PR just for that file?

zdavatz picture zdavatz on 26 Mar 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

andreasamsler picture andreasamsler  路  7Comments
jeanbrouille picture jeanbrouille  路  6Comments
herbertvonkaramalz picture herbertvonkaramalz  路  7Comments
jb3-2 picture jb3-2  路  6Comments
baryluk picture baryluk  路  3Comments