Farm: Usecols do not match columns?

Created on 9 Oct 2020  路  1Comment  路  Source: deepset-ai/FARM

Question
Why does FARM not recognize my columns?

Additional context
I'm a newbie to FARM and tried to reproduce an online tutorial on colab but wanted to use "my own" dataset.
This one: https://github.com/tblock/10kGNAD

train and text are have no header so I did it on my own (dirty):

trainupdf = pd.read_csv("10kGNAD/train.csv",usecols=[0, 1], header=None, delimiter=";")
trainupdf.to_csv("train_up.csv", header=['label', 'text'], index=False, sep=";")
yy = pd.read_csv("train_up.csv", delimiter=";")
yy.head()

Which results in:

label | text
-- | --
Sport | 21-J盲hriger f盲llt wohl bis Saisonende aus. Wie...
Kultur | 'Erfundene Bilder zu Filmen, die als verloren ...
Web | Der frischgek眉rte CEO Sundar Pichai setzt auf ...
Wirtschaft | Putin: "Einigung, dass wir Menge auf Niveau vo...
Inland | Estland sieht den k眉nftigen 枚sterreichischen P..


Same with the test.csv file.

Now if I try to use the TextClassificationProcessor and the DataSilo, FARM tells me that it does not find the columns text and label.

Here is the code I use:

label_list = ['Etat', 'Inland', 'International', 'Kultur', 'Panorama', 'Sport', 'Web', 'Wirtschaft', 'Wissenschaft'] #labels in our data set
metric = "f1_macro" # desired metric for evaluation
processor = TextClassificationProcessor(tokenizer=tokenizer,
                                            max_seq_len=512, # BERT can only handle sequence lengths of up to 512
                                            data_dir='./', 
                                            label_list=label_list,
                                            label_column_name='label', # our labels are located in the "label" column
                                            metric=metric,
                                            quote_char='"',
                                            multilabel=True,
                                            train_filename="train_up.csv",
                                            dev_filename=None,
                                            test_filename="test_up.csv",
                                            dev_split=0.1 # this will extract 10% of the train set to create a dev set
                                            )
data_silo = DataSilo(
    processor=processor,
    batch_size=batch_size)

This is the error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-61-88a4c7ba532d> in <module>()
      1 data_silo = DataSilo(
      2     processor=processor,
----> 3     batch_size=batch_size)

10 frames
/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py in _validate_usecols_names(usecols, names)
   1291     if len(missing) > 0:
   1292         raise ValueError(
-> 1293             f"Usecols do not match columns, columns expected but not found: {missing}"
   1294         )
   1295 

ValueError: Usecols do not match columns, columns expected but not found: ['text', 'label']





What am I doing wrong here?

question

Most helpful comment

Nevermind, I have to set the seperator to "t".

trainupdf = pd.read_csv("10kGNAD/train.csv",usecols=[0, 1], header=None, delimiter=";")
trainupdf.to_csv("train_up.csv", header=['label', 'text'], index=False, sep="\t")
yy = pd.read_csv("train_up.csv", delimiter="\t")
yy.head()

>All comments

Nevermind, I have to set the seperator to "t".

trainupdf = pd.read_csv("10kGNAD/train.csv",usecols=[0, 1], header=None, delimiter=";")
trainupdf.to_csv("train_up.csv", header=['label', 'text'], index=False, sep="\t")
yy = pd.read_csv("train_up.csv", delimiter="\t")
yy.head()
Was this page helpful?
0 / 5 - 0 ratings