Question
Why does FARM not recognize my columns?
Additional context
I'm a newbie to FARM and tried to reproduce an online tutorial on colab but wanted to use "my own" dataset.
This one: https://github.com/tblock/10kGNAD
train and text are have no header so I did it on my own (dirty):
trainupdf = pd.read_csv("10kGNAD/train.csv",usecols=[0, 1], header=None, delimiter=";")
trainupdf.to_csv("train_up.csv", header=['label', 'text'], index=False, sep=";")
yy = pd.read_csv("train_up.csv", delimiter=";")
yy.head()
Which results in:
label | text
-- | --
Sport | 21-J盲hriger f盲llt wohl bis Saisonende aus. Wie...
Kultur | 'Erfundene Bilder zu Filmen, die als verloren ...
Web | Der frischgek眉rte CEO Sundar Pichai setzt auf ...
Wirtschaft | Putin: "Einigung, dass wir Menge auf Niveau vo...
Inland | Estland sieht den k眉nftigen 枚sterreichischen P..
Same with the test.csv file.
Now if I try to use the TextClassificationProcessor and the DataSilo, FARM tells me that it does not find the columns text and label.
Here is the code I use:
label_list = ['Etat', 'Inland', 'International', 'Kultur', 'Panorama', 'Sport', 'Web', 'Wirtschaft', 'Wissenschaft'] #labels in our data set
metric = "f1_macro" # desired metric for evaluation
processor = TextClassificationProcessor(tokenizer=tokenizer,
max_seq_len=512, # BERT can only handle sequence lengths of up to 512
data_dir='./',
label_list=label_list,
label_column_name='label', # our labels are located in the "label" column
metric=metric,
quote_char='"',
multilabel=True,
train_filename="train_up.csv",
dev_filename=None,
test_filename="test_up.csv",
dev_split=0.1 # this will extract 10% of the train set to create a dev set
)
data_silo = DataSilo(
processor=processor,
batch_size=batch_size)
This is the error:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-61-88a4c7ba532d> in <module>()
1 data_silo = DataSilo(
2 processor=processor,
----> 3 batch_size=batch_size)
10 frames
/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py in _validate_usecols_names(usecols, names)
1291 if len(missing) > 0:
1292 raise ValueError(
-> 1293 f"Usecols do not match columns, columns expected but not found: {missing}"
1294 )
1295
ValueError: Usecols do not match columns, columns expected but not found: ['text', 'label']
What am I doing wrong here?
Nevermind, I have to set the seperator to "t".
trainupdf = pd.read_csv("10kGNAD/train.csv",usecols=[0, 1], header=None, delimiter=";")
trainupdf.to_csv("train_up.csv", header=['label', 'text'], index=False, sep="\t")
yy = pd.read_csv("train_up.csv", delimiter="\t")
yy.head()
Most helpful comment
Nevermind, I have to set the seperator to "t".