Covid_19: Scraper for VD

[MeMent0L]

Here is a python scraper for Canton VD.

Hopefully it will do the job ongoing, but it's not perfect as I've had to extract the data provided in text format as I didn't find out how to download the data from datawrapper (javascript).
Also, the data provided there doesn't include data prior to 10/03/2020.

Anyways, hope this helps.

# -*- coding: utf-8 -*-
from selenium import webdriver
from bs4 import BeautifulSoup
import numpy as np
import pandas as pd

### Watch-out: installing Selenium requires Gekko and it may be easier to configure it with Chrome
geckk=r'C:\Program Files (x86)\Mozilla Firefox\firefox.exe'

'''Documentation & resources to help set-up & use selenium:  
    https://www.tutorialspoint.com/python_web_scraping/python_web_scraping_dynamic_websites.htm
    https://www.selenium.dev/documentation/en/webdriver/web_element/
    https://realpython.com/modern-web-automation-with-python-and-selenium/
    https://stackoverflow.com/questions/7861775/python-selenium-accessing-html-source
    https://stackoverflow.com/questions/51273995/selenium-python-dynamic-table
'''

### Set options for Selenium 
options = webdriver.FirefoxOptions()
options.headless = True
options.add_argument("disable-gpu")
options.add_argument("headless")
options.add_argument("no-default-browser-check")
options.add_argument("no-first-run")
options.add_argument("no-sandbox")
options.add_argument("marionette=True")
options.add_argument("--test-type")
options.set_preference('browser.download.manager.showWhenStarting', False)
options.set_preference('browser.helperApps.neverAsk.saveToDisk', 'text/csv')
options.set_preference("browser.download.folderList",2)
options.set_preference("browser.download.manager.showWhenStarting",False)
options.set_preference("browser.download.dir","c:\\downloads")

profile = webdriver.FirefoxProfile()
profile.accept_untrusted_certs = True

driver = webdriver.Firefox(firefox_binary=geckk,options=options, firefox_profile=profile)

### Download
driver.get("https://datawrapper.dwcdn.net/tr5bJ/16/")
soup=BeautifulSoup(driver.page_source, 'html.parser')

### Get the data required (didn't manage to do differently than browsing across text)
data_cursor_start=soup.text.find('chartData: "')
data_cursor_stop=data_cursor_start+soup.text[data_cursor_start:].find('",')
zoom=str(soup.text)[data_cursor_start:data_cursor_stop]

### Create DataFrame
table_lines=zoom.split('\\n')
line_array=[]
for each_line in table_lines[1:]:
    line=each_line.split('\\t')
    line_array += line
line_matrix= [line_array[x:x+5] for x in range(0, len(line_array),5)]

df=pd.DataFrame(line_matrix, columns=['date', 'ncumul_hosp','ncumul_released','ncumul_deceased','ncumul_conf'])
df['date']=pd.to_datetime(df['date'],yearfirst=True)
df['abbreviation_canton_and_fl']='VD'
df['time']=np.NaN
df['ncumul_tested']=np.NaN
df['ncumul_ICU']=np.NaN
df['ncumul_vent']=np.NaN
df['source']='https://datawrapper.dwcdn.net/tr5bJ/16/'

### Format & Save CSV
new_order=['date','time','abbreviation_canton_and_fl','ncumul_tested','ncumul_conf',\
           'ncumul_hosp','ncumul_ICU','ncumul_vent','ncumul_released','ncumul_deceased','source']
df=df.T.reindex(new_order).T.set_index('date')

df.to_csv('COVID19_Fallzahlen_Kanton_VD_total.csv')

[zdavatz]

Thank you, did you have a look at the other scraper by @baryluk ? If possible we should stick to that format.

[tlorusso]

Thanks @MeMent0L , looks useful! We're evaluating how to organize and monitor the different scrapers, so we'll let you know shortly :)

[MeMent0L]

@zdavatz the table format is the one i found on the open_ZH page, but I believe you use a slightly different format in your repo?
I didn't check what @baryluk had done, scraping in bash is 5 levels above my head, however :)
Pandas dataframes are useful to get and wrangle the data. If the code i shared here is redundant, just scrap it, or use the pieces that are helpful @tlorusso.

[zdavatz]

ok.

[MeMent0L]

By the way, the website https://www.corona-data.ch/ from @daenuprobst is real good!
I couldn't find in their Covid-19 tracking repo if the input data is scrapted or manually updated; maybe the above can help as well.

[zdavatz]

@MeMent0L yes, but he collects the data manually AFAICT.

[baryluk]

Implemented in https://github.com/openZH/covid_19/pull/108

[metaodi]

Thank you @MeMent0L for your scraper. Since we already have a different one, I will close this issue for now.