NE - - - FAILED 2020-03-31T20:00:07+02:00 # URLs: https://www.ne.ch/autorites/DFS/SCSP/medecin-cantonal/maladies-vaccinations/Pages/Coronavirus.aspx # Errors: Missing date. Missing cases. Date and time is None. Confirmed cases is None. Deaths is None
cc @baryluk
zdavatz
All 26 comments
I don't see any numbers any more on https://www.ne.ch/autorites/DFS/SCSP/medecin-cantonal/maladies-vaccinations/Pages/Coronavirus.aspx
I send them an email.
Could you look around yourself, maybe I missed some page where this data is available?
If not send them an email too please.
baryluk
on 31 Mar 2020
I see some data in https://www.ne.ch/autorites/DFS/SCSP/medecin-cantonal/maladies-vaccinations/Documents/Covid-19-Statistiques/COVID19_PublicationInternet.pdf
Scrape possible, but still non trivial.
baryluk
on 31 Mar 2020
I looked into pdftotext for this PDF, and I would say, it is essentially impossible to parse reliably, due to nature of the graph, rotated text, invisible line breaks, and other factors (like there is no guarantees of the data order, and correspondence of data and number of cases).
baryluk
on 31 Mar 2020
I checked archive.org, it seems as if they just removed the block with the numbers, that was previously in the section titled "Quelle est la situation actuelle ?"
metaodi
on 31 Mar 2020
@baryluk I think you should be able to grab the number here with "pdfgrep". This is on Linux. See:
If you know what the last number is ;). Just always take the last number in the row. This is actually Text in the file ;).
zdavatz
on 31 Mar 2020
@zdavatz That is not usable.
baryluk
on 31 Mar 2020
@metaodi Yes,
http://web.archive.org/web/20200329210634/https://www.ne.ch/autorites/DFS/SCSP/medecin-cantonal/maladies-vaccinations/Pages/Coronavirus.aspx
The numbers for Switzerland and Neuch芒tel are now simply not there.
baryluk
on 31 Mar 2020
@zdavatz , @metaodi Send them an email of some form to complain. Thanks.
baryluk
on 31 Mar 2020
@jb3-2 Could you nudge them a bit too. Thanks.
baryluk
on 1 Apr 2020
@zdavatz , @metaodi Send them an email of some form to complain. Thanks.
zdavatz
on 1 Apr 2020
scraper didn't pick up on the number of cases from yesterday's data
claudia013
on 2 Apr 2020
I did receive response for my email 3 hours ago, claiming it is now fixed, but it is not. The webpage still doesn't have numbers.
I respond, back to check it again.
Maybe they think about the PDF, which is unusable for our purposes. Not sure.
@claudia013 Yes, it is expected scraper is still broken. We can't do anything about it, other than to complain to Neuch芒tel Canton people about it.
baryluk
on 3 Apr 2020
I will disable the scraper and note that NE is updated manually on the README
metaodi
on 4 Apr 2020
Hi, I'm working on a small script to parse that PDF into a csv. Would that be of use? If so, what header should I use for it?
maekke97
on 6 Apr 2020
anyone you like, as long as you get the numbers ;)
zdavatz
on 6 Apr 2020
@maekke97 Yes, if you can make it work that would be really nice. Any output format would be fine for start. Could you make it available somewhere to test?
It appears there is a table at the end of PDF now (it wasn't there few days ago), that is in reasonable form, and can be scraped.
Data from graphs are much harder to scrape, possible, but not very reliable.
With pdftotext -layout I got this for part of the table:
...
27mars2020 17 316 36 2 7 45 2 7 8 17 1 12
28mars2020 21 337 39 3 8 50 3 8 6 17 2 14
29mars2020 9 346 44 10 6 60 10 6 0 16 3 17
30mars2020 32 378 45 6 6 57 6 6 3 15 2 19
31mars2020 24 402 45 6 8 59 6 8 3 17 2 21
1avr
il2020 18 420 52 6 8 66 6 8 4 18 2 23
2avr
il2020 10 430 49 2 10 61 2 10 4 16 2 25
3avr
il2020 14 444 50 2 9 61 2 9 3 14 25
4avr
il2020 5 449 49 1 9 59 1 9 4 14 1 26
5avr
il2020 10 459 46 4 8 58 4 8 5 17 26
6avr
il2020 49 2 8 59 2 8 4 14 1 27
Which looks decent, but still not trivial to scrape. Probably the best approach is to try to parse it into arrays, and then have checks in the scraper for the other known dates, that columns match properly.
baryluk
on 6 Apr 2020
tesseract is also doing very good job:
$ convert -density 300 'COVID19_PublicationInternet.2020-04-06.pdf[4]' covid19_ne_table.png
$ tesseract -l fra --psm 1 covid19_ne_table.png stdout
...
1 mars 2020 1 1
2 mars 2020 0 1 1 1
3 mars 2020 0 1 1 1
4 mars 2020 7 8 4 4
5 mars 2020 1 9 4 4
6 mars 2020 4 13 2 2
7 mars 2020 5 18 2 2
8 mars 2020 6 24 3 3
9 mars 2020 3 27 4 4
10 mars 2020 4 31 3 1 4 1 1
11 mars 2020 6 37 5 1 6 1 1
12 mars 2020 9 46 5 1 6 1 1
13 mars 2020 13 59 8 1 9 1 1
14 mars 2020 9 68 9 1 10 1 1
15 mars 2020 6 74 8 1 9 1 1
16 mars 2020 19 93 11 2 13 2 2
17 mars 2020 21 114 12 4 16 4 4 1 1
18 mars 2020 32 146 15 3 18 3 3 1 2
19 mars 2020 29 175 16 3 19 3 3 1 3
20 mars 2020 13 188 17 4 21 4 2 6 3
21 mars 2020 12 200 15 5 20 5 1 6 1 4
22 mars 2020 16 216 22 6 28 6 1 7 4
23 mars 2020 31 247 22 5 6 33 5 6 0 11 1 5
24 mars 2020 18 265 24 2 6 32 2 6 3 11 1 6
25 mars 2020 15 280 31 3 7 41 3 7 2 12 3 9
26 mars 2020 19 299 33 2 7 42 2 7 1 10 2 11
27 mars 2020 17 316 36 2 7 45 2 7 8 17 1 12
28 mars 2020 21 337 39 3 8 50 3 8 6 17 2 14
29 mars 2020 9 346 44 10 6 60 10 6 0 16 3 17
30 mars 2020 32 378 45 6 6 57 6 6 3 15 2 19
31 mars 2020 24 402 45 6 8 59 6 8 3 17 2 21
1 avril 2020 18 420 52 6 8 66 6 8 4 18 2 23
2 avril 2020 10 430 49 2 10 61 2 10 4 16 2 25
3 avril 2020 14 444 50 2 9 61 2 9 3 14 25
4 avril 2020 5 449 49 1 9 59 1 9 4 14 1 26
5 avril 2020 10 459 46 4 8 58 4 8 5) 17 26
6 avril 2020 49 2 8 59 2 8 4 14 1 27
$
Unfortunately, due to the holes in the table, it is still not reliable.
It is possible to extra more layout info, using tesseract -l fra --psm 1 covid19_ne_table.png stdout -c tessedit_create_hocr=1, and then do some analysis of bounding boxes:
<span class='ocr_line' id='line_1_52' title="bbox 199 2262 3181 2287; baseline 0 -1; x_size 31.216118; x_descenders 5.7807016; x_ascenders 7.6750002">
<span class='ocrx_word' id='word_1_596' title='bbox 199 2263 215 2287; x_wconf 96'>6</span>
<span class='ocrx_word' id='word_1_597' title='bbox 225 2262 287 2287; x_wconf 96'>avril</span>
<span class='ocrx_word' id='word_1_598' title='bbox 299 2263 372 2287; x_wconf 96'>2020</span>
<span class='ocrx_word' id='word_1_599' title='bbox 986 2263 1022 2287; x_wconf 95'>49</span>
<span class='ocrx_word' id='word_1_600' title='bbox 1398 2263 1414 2286; x_wconf 96'>2</span>
but that looks tedious and hard.
baryluk
on 6 Apr 2020
I took pdftotext and then went through decreasing separators to catch all the lines. Works for the current pdf, I'm not sure if it is 100% stable...
import pdftotext
import re
COLUMNS = [
'date',
'ncases_per_day',
'ncumul_cases',
'n_hospital_non_icu',
'n_hospital_icu',
'n_hospital_icu_non_tube',
'n_hospital_icu_tube',
'n_hospital_total',
'n_icu_covid',
'n_icu_non_tube',
'n_icu_tube',
'n_icu_non_covid',
'n_icu_total',
'n_deceased',
'ncumul_deceased'
]
def pdf_ne_to_csv(input_filename: str, output_filename: str):
with open(input_filename, 'rb') as ifh:
pdf_content = pdftotext.PDF(ifh)
pdf_content_string = '\n'.join(pdf_content)
# magic column fix (don't know if this is stable)
pdf_content_string = pdf_content_string.replace('avr\n il', 'avril')
# pdf_content_string = pdf_content_string.replace('avr\n il', 'avril')
# find the start of the table on page 5
res = re.search('1mars2020', pdf_content_string)
data_str = pdf_content_string[res.span()[0]:]
data_str = data_str.replace('2020 ', '2020:')
data_str = data_str.replace(' ', ':')
data_str = data_str.replace(' ', ':')
data = [[dat.strip() for dat in line.split(':')] for line in data_str.split('\n')]
with open(output_filename, 'w', encoding='utf-8') as ofh:
ofh.write(','.join(COLUMNS) + '\n')
for line in data:
ofh.write(','.join(line) + '\n')
@maekke97 Could you edit your comment and use a triple backtick to include the source?
Like this:
def xyz(a): pass
baryluk
on 6 Apr 2020
@baryluk Thanks for the hint.
maekke97
on 6 Apr 2020
@maekke97 Thanks. Nice script. It looks like the python module pdftotext does internally default to layout_mode = poppler::page::physical_layout;, which is exactly what pdftotext -layout also does.
It should be easy to incorporate this into our existing scraper and infrastructure.
I will try to write a prototype based on it with minor tweaks. I think it should be relatively stable.
But your script would be also useful to getting historical tables if needed.
baryluk
on 6 Apr 2020
I just made a fork and added some minor improvements like parsing the french date string
https://github.com/maekke97/covid_19/blob/scrap_ne_pdf/scrapers/scrap_NE_pdf.py
Should I open a pull request or do you want to take it from there?
maekke97
on 6 Apr 2020
@maekke97 I borrowed some code from you, and it is close to functional:
$ ./scrape_ne.py
NE
Downloading: https://www.ne.ch/autorites/DFS/SCSP/medecin-cantonal/maladies-vaccinations/Pages/Coronavirus.aspx
Scraped at: 2020-04-06T22:34:08+02:00
Date and time: None
Confirmed cases: None
Deaths: None
Downloading: https://www.ne.ch/autorites/DFS/SCSP/medecin-cantonal/maladies-vaccinations/Documents/Covid-19-Statistiques/COVID19_PublicationInternet.pdf
Scraped at: 2020-04-06T22:34:08+02:00
Date and time: 6avril2020
Confirmed cases:
Deaths: 27
Hospitalized: 59
ICU: 14
Vent: 8
It is unclear how to use the intubated numbers. There are hospitalized and ICU values for intubated, but for last 15 days, they are the same, so I assume one is inclusive of the other, and I should just use max, or the ones from hospitalized column. (Not add them).
The code still requires a bit of massaging to get right, as the last row doesn't necessary have confirmed cases available, like today.
baryluk
on 6 Apr 2020
@maekke97 Take a look here: https://github.com/baryluk/covid_19/blob/new_ne_pdf_scraper/scrapers/scrape_ne.py line 26 to the end of file. (Before line 26 there is code for the old scraper).
You can see details of date parsing here: https://github.com/baryluk/covid_19/commit/98644f4b87314ecf589834c09f59fe74ef2e7864 it was trivial change.
baryluk
on 6 Apr 2020
$ ./scrape_ne.py | ./parse_scrape_output.py
NE 2020-04-05T 459 26 OK 2020-04-06T23:01:21+02:00 # Extras: ncumul_hosp=58,ncumul_ICU=17,ncumul_vent=8 # URLs: https://www.ne.ch/autorites/DFS/SCSP/medecin-cantonal/maladies-vaccinations/Pages/Coronavirus.aspx, https://www.ne.ch/autorites/DFS/SCSP/medecin-cantonal/maladies-vaccinations/Documents/Covid-19-Statistiques/COVID19_PublicationInternet.pdf
Looks good. This is for yesterday, because a row for yesterday, doesn't have confirmed cases published yet. (it does some other numbers, like deceased reported, +1 compared to yesterday).
baryluk
on 6 Apr 2020
Fixed in #470
metaodi
on 10 Apr 2020
Related issues
baryluk
路
3Comments
zdavatz
路
7Comments
zdavatz
路
4Comments
herbertvonkaramalz
路
7Comments
Pseudomanifold
路
6Comments