Covid_19: Primitive scraper for AG

[baryluk]

echo AG; URL=$(curl --silent 'https://www.ag.ch/de/themen_1/coronavirus_2/lagebulletins/lagebulletins_1.jsp' | sed -E -e 's/<li>/\n<li>/g' | grep Bulletin | grep pdf | grep href | awk -F '"' '{print $6;}' | head -1); d=$(curl --silent "https://www.ag.ch/${URL}" | pdftotext - - | egrep -A 2 "(Aarau, .+Uhr|Stand [A-Za-z]*, [0-9]+)"); echo "Scraped at $(date --iso-8601=seconds)"; echo -n "Date and time: "; echo "$d" | grep Aarau, | sed -E -e 's/.*, (.+)/\1/'; echo -n "Confirmed cases: "; echo "$d" | egrep '^[0-9]+$'

Scraped at 2020-03-21T17:36:17+00:00
Date and time: 20. März 2020 15.00 Uhr
Confirmed cases: 168

[zdavatz]

I love it! Real hacking! A beauty!

[baryluk]

I love it! Real hacking! A beauty!

It is hacky, but actually it is actually a bit more robust than parsing HTML / XML directly, and navigating the DOM tree from the top.

I am just finishing the meta-scrapper to run all the scripts, report any errors on one that fail, and present a table of new data.

[zdavatz]

It is hacky, but actually it is actually a bit more robust than parsing HTML / XML directly, and navigating the DOM tree from the top.

It is the best way you are doing it!

I am just finishing the meta-scrapper to run all the scripts, report any errors on one that fail, and present a table of new data.

Great! If possible please write a report file after every run.