Ludwig: pandas.errors.ParserError: Error tokenizing data. C error: Expected 83 fields in line 40, saw 92

[IzzyHibbert]

Describe the bug
Hi,
I am getting this error message:
pandas.errors.ParserError: Error tokenizing data. C error: Expected 83 fields in line 40, saw 92

I made all the possible in order to use a good CSV, but I guess something is not fine.
As a suggestion I believe it could be good to find a way to handle bad lines (skip) and present a value (total) of ignored lines.

Back to my settings.
I have UTF-8 for the saved csv file. My columns contain mostly text and some got ',' which I turned into '\t' as founded in the documentation.
Also, I made the following basic test: I have re-imported the csv in Excel and made sure all the lines and columns (24 in total in my case) are in the right place. That was the case: all fine inside of MS Excel view.

The YAML has been validated correctly. I exclude it is that one my case.

_Here below the lines from my terminal._

ludwig_version: '0.1.0'
command: ('/Library/Frameworks/Python.framework/Versions/3.6/bin/ludwig train '
'--data_csv /Users/my_mac/Projects/ML/LUDWIG/Wine/red_dataset.csv '
'--model_definition_file '
'/Users/my_mac/Projects/ML/LUDWIG/Wine/dataset.yaml')
dataset_type: '/Users/my_mac/Projects/ML/LUDWIG/Wine/red_dataset.csv'
model_definition: { 'combiner': {'type': 'concat'},
'input_features': [ { 'encoder': 'parallel_cnn',
'level': 'word',
'name': 'Order',
'tied_weights': None,
'type': 'text'},
{ 'encoder': 'parallel_cnn',
'level': 'word',
'name': 'Produttore',
'tied_weights': None,
'type': 'text'},
{ 'encoder': 'parallel_cnn',
'level': 'word',
'name': 'Tipo',
'tied_weights': None,
'type': 'text'},
{ 'encoder': 'parallel_cnn',
'level': 'word',
'name': 'Descrizione',
'tied_weights': None,
'type': 'text'},
{ 'encoder': 'parallel_cnn',
'level': 'word',
'name': 'Vitigni',
'tied_weights': None,
'type': 'text'},
{ 'encoder': 'parallel_cnn',
'level': 'word',
'name': 'Vigneti',
'tied_weights': None,
'type': 'text'},
{ 'encoder': 'parallel_cnn',
'level': 'word',
'name': 'Vinificazione',
'tied_weights': None,
'type': 'text'},
{ 'encoder': 'parallel_cnn',
'level': 'word',
'name': 'Affinamento',
'tied_weights': None,
'type': 'text'},
{ 'encoder': 'parallel_cnn',
'level': 'word',
'name': 'Filosofia',
'tied_weights': None,
'type': 'text'},
{ 'encoder': 'parallel_cnn',
'level': 'word',
'name': 'Temperatura',
'tied_weights': None,
'type': 'text'},
{ 'encoder': 'parallel_cnn',
'level': 'word',
'name': 'Quando_aprire',
'tied_weights': None,
'type': 'text'},
{ 'encoder': 'parallel_cnn',
'level': 'word',
'name': 'Ideale',
'tied_weights': None,
'type': 'text'},
{ 'encoder': 'parallel_cnn',
'level': 'word',
'name': 'Quando_bere',
'tied_weights': None,
'type': 'text'},
{ 'encoder': 'parallel_cnn',
'level': 'word',
'name': 'Descrizione_long',
'tied_weights': None,
'type': 'text'},
{ 'encoder': 'parallel_cnn',
'level': 'word',
'name': 'Colore',
'tied_weights': None,
'type': 'text'},
{ 'encoder': 'parallel_cnn',
'level': 'word',
'name': 'Profumo',
'tied_weights': None,
'type': 'text'},
{ 'encoder': 'parallel_cnn',
'level': 'word',
'name': 'Gusto',
'tied_weights': None,
'type': 'text'},
{ 'in_memory': True,
'name': 'Immagine-src',
'should_resize': False,
'tied_weights': None,
'type': 'image'}],
'output_features': [ { 'dependencies': [],
'loss': { 'type': 'mean_squared_error',
'weight': 1},
'name': 'Prezzo',
'reduce_dependencies': 'sum',
'reduce_input': 'sum',
'type': 'numerical',
'weight': 1},
{ 'decoder': 'generator',
'dependencies': [],
'level': 'char',
'loss': { 'class_distance_temperature': 0,
'class_weights': 1,
'type': 'softmax_cross_entropy',
'weight': 1},
'name': 'Denominazione',
'reduce_dependencies': 'sum',
'reduce_input': 'sum',
'type': 'text',
'weight': 1},
{ 'decoder': 'generator',
'dependencies': [],
'level': 'char',
'loss': { 'class_distance_temperature': 0,
'class_weights': 1,
'type': 'softmax_cross_entropy',
'weight': 1},
'name': 'Regione',
'reduce_dependencies': 'sum',
'reduce_input': 'sum',
'type': 'text',
'weight': 1},
{ 'decoder': 'generator',
'dependencies': [],
'level': 'char',
'loss': { 'class_distance_temperature': 0,
'class_weights': 1,
'type': 'softmax_cross_entropy',
'weight': 1},
'name': 'Gradazione',
'reduce_dependencies': 'sum',
'reduce_input': 'sum',
'type': 'text',
'weight': 1}],
'preprocessing': { 'bag': { 'fill_value': '',
'format': 'space',
'lowercase': 10000,
'missing_value_strategy': 'fill_with_const',
'most_common': False},
'binary': { 'fill_value': 0,
'missing_value_strategy': 'fill_with_const'},
'category': { 'fill_value': '',
'lowercase': False,
'missing_value_strategy': 'fill_with_const',
'most_common': 10000},
'force_split': False,
'image': {'missing_value_strategy': 'backfill'},
'numerical': { 'fill_value': 0,
'missing_value_strategy': 'fill_with_const'},
'sequence': { 'fill_value': '',
'format': 'space',
'lowercase': False,
'missing_value_strategy': 'fill_with_const',
'most_common': 20000,
'padding': 'right',
'padding_symbol': '',
'sequence_length_limit': 256,
'unknown_symbol': ''},
'set': { 'fill_value': '',
'format': 'space',
'lowercase': False,
'missing_value_strategy': 'fill_with_const',
'most_common': 10000},
'split_probabilities': (0.7, 0.1, 0.2),
'stratify': None,
'text': { 'char_format': 'characters',
'char_most_common': 70,
'char_sequence_length_limit': 1024,
'fill_value': '',
'lowercase': True,
'missing_value_strategy': 'fill_with_const',
'padding': 'right',
'padding_symbol': '',
'unknown_symbol': '',
'word_format': 'space_punct',
'word_most_common': 20000,
'word_sequence_length_limit': 256},
'timeseries': { 'fill_value': '',
'format': 'space',
'missing_value_strategy': 'fill_with_const',
'padding': 'right',
'padding_value': 0,
'timeseries_length_limit': 256}},
'training': { 'batch_size': 128,
'bucketing_field': None,
'decay': False,
'decay_rate': 0.96,
'decay_steps': 10000,
'dropout_rate': 0.0,
'early_stop': 3,
'epochs': 10,
'gradient_clipping': None,
'increase_batch_size_on_plateau': 0,
'increase_batch_size_on_plateau_max': 512,
'increase_batch_size_on_plateau_patience': 5,
'increase_batch_size_on_plateau_rate': 2,
'learning_rate': 0.001,
'learning_rate_warmup_epochs': 5,
'optimizer': { 'beta1': 0.9,
'beta2': 0.999,
'epsilon': 1e-08,
'type': 'adam'},
'reduce_learning_rate_on_plateau': 0,
'reduce_learning_rate_on_plateau_patience': 5,
'reduce_learning_rate_on_plateau_rate': 0.5,
'regularization_lambda': 0,
'regularizer': 'l2',
'staircase': False,
'validation_field': 'combined',
'validation_measure': 'loss'}}

Using full raw csv, no hdf5 and json file with the same name have been found
Building dataset (it may take a while)
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ludwig/utils/data_utils.py", line 46, in read_csv
df = pd.read_csv(data_fp, header=header)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/io/parsers.py", line 678, in parser_f
return _read(filepath_or_buffer, kwds)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/io/parsers.py", line 446, in _read
data = parser.read(nrows)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/io/parsers.py", line 1036, in read
ret = self._engine.read(nrows)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/io/parsers.py", line 1848, in read
data = self._reader.read(nrows)
File "pandas/_libs/parsers.pyx", line 876, in pandas._libs.parsers.TextReader.read
File "pandas/_libs/parsers.pyx", line 891, in pandas._libs.parsers.TextReader._read_low_memory
File "pandas/_libs/parsers.pyx", line 945, in pandas._libs.parsers.TextReader._read_rows
File "pandas/_libs/parsers.pyx", line 932, in pandas._libs.parsers.TextReader._tokenize_rows
File "pandas/_libs/parsers.pyx", line 2112, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: Expected 83 fields in line 40, saw 92

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/bin/ludwig", line 11, in
sys.exit(main())
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ludwig/cli.py", line 86, in main
CLI()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ludwig/cli.py", line 64, in __init__
getattr(self, args.command)()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ludwig/cli.py", line 70, in train
train.cli(sys.argv[2:])
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ludwig/train.py", line 663, in cli
full_train(**vars(args))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ludwig/train.py", line 224, in full_train
random_seed=random_seed
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ludwig/data/preprocessing.py", line 457, in preprocess_for_training
random_seed=random_seed
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ludwig/data/preprocessing.py", line 54, in build_dataset
dataset_df = read_csv(dataset_csv)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/ludwig/utils/data_utils.py", line 48, in read_csv
logging.WARNING('Failed to parse the CSV with pandas default way,'
TypeError: 'int' object is not callable

[w4nderlust]

@IzzyHibbert thanks for posting this. In the docs we suggest to escape the commas within the text with \\,, so first thing I would try to do that.
Let me know if this solves your problem.

Personally I prefer to be a bit more strict on the data side rather than letting things pass or being filtered out, because those could become problems down the line.

[IzzyHibbert]

Apologies for the typo in my original message.
I already used \\, as escape for the TEXT columns. (so suggestion does not work, issue is elsewhere).

Also, the error message tells about row 40. Unless I am understanding it wrongly, the first 39 lines did run smooth (I have commas and other punctuation there also).
I am using Line breaks Unix (LF) for my csv.

My line 40 is :

ID-39,San Giovenale,'Habemus - Etichetta Bianca' San Giovenale 2016,L’Habemus di San Giovenale è un vino rosso nobile\, intenso e potente\, nato nel viterbese da varietà internazionali coltivate ad alberello\, vinificato in acciaio e affinato per 20 mesi in barrique nuove. Si esprime con ricchezza\, eleganza e intensità\, concedendo ampi sentori di frutta in confettura\, spezie dolci\, fiori rossi\, grafite\, pepe e liquirizia. Vino fatto come una volta con metodi artigianali,,52.5,Lazio IGT,Grenache 40%\, Syrah 30%\, Carignano 20%\, Tempranillo 10%,Lazio,15.0,Viti ad alberello su terreno argilloso a 300-400 metri di altitudine\, con esposizione a sud. Agricoltura biologica,Pigiodiraspatura e fermentazione con lieviti indigeni in acciaio per 15 giorni,20 mesi in barrique nuove e almeno 6 mesi in bottiglia,Artigianali fatti come una volta\, Lieviti indigeni\,,18-20,Per gustarlo al meglio\, ti consigliamo di versare il vino in decanter 1 o 2 ore prima di servirlo,Ideale per i grandi vini rossi robusti\, corposi e intensi\, che necessitano di un'intensa ossigenazione per dischiudere tutti i loro profumi profondi e complessi ed esaltarne l’evoluzione nel calice,Vino ottimo da bere adesso\, ma che può avere una bella evoluzione se lasciato riposare un po’ di tempo in cantina,L'Habemus Etichetta Bianca è un vino potente e ricchissimo\, dotato di tattilità vellutata e carezzevole. La mano di Marco Casolanetti per ciò che concerne la produzione di vino è una delle più riconoscibili\, figlia di un'idea chiara che abbraccia il lavoro in vigna e in cantina. Marco\, già proprietario di Oasi degli Angeli\, opta per rese bassissime in pianta in modo da concentrare aromi e estratti in pochi frutti\, raccolti a piena maturazione. Nel caso dell'Habemus i vitigni scelti sono Grenache\, Syrah\, Carignano e Tempranillo\, tutti in grado di fornire un apporto aromatico generoso e ben delineato. Liquidi concentrati\, profondi\, impattanti\, e di sicuro fascino e coinvolgimento\, per la loro suadenza e piacevolezza di beva. Inutile dirlo\, gli estimatori sono tanti e le bottiglie meno\, pertanto conviene fare spazio in cantina! Il rosso Habemus Etichetta Bianca è ottenuto da un uvaggio costituito da Grenache\, Syrah\, Carignano e Tempranillo. Le uve\, che vengono raccolte manualmente\, sono frutto di una selezione accuratissima di vigne ad alberello piantate a 300-400 metri di altitudine su terreno argilloso con esposizione sud. La filosofia in vigna di Marco è da sempre volta al rispetto delle varietà e del sistema\, pertanto le piante non vengono trattate con alcuna sostanza chimica o di sintesi. In cantina si prosegue con fermentazione spontanea in contenitori d'acciaio e affinamento per 20 mesi in barrique nuove\, seguito da almeno 6 mesi in bottiglia. Il vino Habemus Etichetta Bianca si palesa nel calice con colore rosso rubino estremamente cupo ed impenetrabile. Naso ampissimo\, su cui un degustatore particolarmente in forma potrebbe soffermarsi ore: cacao\, tabacco\, prugna\, cannella\, pepe nero tratteggiano quello che è un vero e proprio tripudio aromatico. L'assaggio non è da meno\, ed è da vero cavallo di razza. Potenza\, ricchezza e concentrazione\, in un equilbrio mirabile tra parte estrattiva e freschezza. Lunghissimo il finale\, per quello che è un liquido quasi da meditazione\, o da bere in occasioni importanti. Garanzia! Colore Rosso rubino concentrato Profumo Ampio\, nobile e complesso\, con sentori di frutti di bosco in confettura\, fiori rossi\, spezie dolci\, grafite\, pepe e liquirizia Gusto Potente\, corposo\, ricco e ben equilibrato tra struttura\, morbidezza e freschezza\, di lunga persistenza,Rosso rubino concentrato,Ampio\, nobile e complesso\, con sentori di frutti di bosco in confettura\, fiori rossi\, spezie dolci\, grafite\, pepe e liquirizia,Potente\, corposo\, ricco e ben equilibrato tra struttura\, morbidezza e freschezza\, di lunga persistenza,http://data.myglass.com/i-2019_13441.jpg

While the 1st line:
Order,Produttore,Tipo,Descrizione,,Prezzo,Denominazione,Vitigni,Regione,Gradazione,Vigneti,Vinificazione,Affinamento,Filosofia,Temperatura,Quando_aprire,Ideale,Quando_bere,Descrizione_long,Colore,Profumo,Gusto,Immagine-src

[w4nderlust]

looking at the text you posted it seems you used \, not \\,.

[msaisumanth]

@IzzyHibbert your escape characters are fine. I put your header and line 40 in a csv and I was able to read it using pandas (the way Ludwig reads it)

df = pd.read_csv('temp.csv', header=0, escapechar='\\')

Could you please try to read your entire dataset with the above command and see what's happening? Maybe you should check line 41 in the file (counting for the header). This is a common error when using Pandas.

[IzzyHibbert]

@msaisumanth Thanks for your reply.

Now it's getting interesting, as ..

I made the _same test_ you did: keeping line 40 (same one pasted above) and the header in a csv and try to read it (your suggested Pandas lines) . Result is OK.

_Second test_: I take lines 39,40,41 only (in this order) and the result is KO. Still complaining about line 40 (my third line here, as I keep header all the times).

_Third test_: When I remix the line orders, creating a csv with 40,39,41 it works .. OK

CSV is UTF8 with \\, as columns delimiter.

I have no ideas. Apparently, because of the kind of tests I made, I am no longer convinced that there is a character that needs to be escaped. What else could be ..?

the csv is available here :
https://we.tl/t-0MgomW6ySY

[msaisumanth]

@IzzyHibbert I looked at the csv as well. I'm not sure what's going on. One worst case option is to delete the erroneous lines (if it's not too many lines)

[msaisumanth]

one way to do that is to add error_bad_lines = False in pd.read_csv
You can also add it in your local ludwig code.

[w4nderlust]

I'm guessing pandas does some inference on the number of columns by looking at the lines. And potentially there are other characters that convince pandas of the presence of additional columns.
error_bad_lines = False Is a workaround, but I would be curious to figure out what actually is happening with pandas so that similar situations in the future may be solved easily.
Also i tried to open the csv with libreoffice and the result is really messy, which suggests that probably your best shot is fixing the dataset first, as this doesn't seem to be anything specific to Ludwig.
I'm not closing so that if you find a solution you can post it it if other people encounter the same problem.

[IzzyHibbert]

Hi guys,

this is the response I got from Pandas community .. :

"Your file has two literal backslashes but an escapechar can only be one character. The fact that you send two backslashes as an argument is required only to escape it as a single character.

I don't think this is a bug as much as a mismatch in expectations. Since escapechar only supports one character you'll most likely have to choose a different escape char for your csv file, or cleanse it before reading into pandas"

See it here :
https://github.com/pandas-dev/pandas/issues/25724

I am disagreeing with the no-answering to the real question.
What is your opinion ?
( remove lines can't be a good practice, as I can't find a REAL problem with the csv I am processing).

I am interested to hear your opinion..

[w4nderlust]

I'm not sure what the problem with the CSV is really, unfortunately. We are going to add TSV support soon, so hopefully that may solve your issues, stay tuned.

[ashwan1]

Can we have dictionary of options that will be passed to pandas. That will provide lot of flexibility. Users can use any delimiter they want, can use either python or C engine, and more.
May be a method in api and option in cli to set pandas options.

[w4nderlust]

I have mixed feeling about this request, on one hand it will provide flexibility, for sure, on the other hand it would be clunky and error prone, in particular for the cli. For the api, I think it's not really a problem as you can provide data both as a dataframe and as a csv, if you have really custom requirements you can just load the data yourself. For the cli it will add an additional input parameter that would have to parse something like a json string, and it will have to be escaped so it would look like --pandas_params "{\"something\": some_numerical_value, \"something_else\": "some_string_value", ...}" which I don't really like much. An alternative would be to request a yaml string, that would be cleaner, but still pretty clunky. users can also always load the dataframe and then save it in a utf-8 comma separated way, that takes 2 lines fo python code, not really a lot to ask probably.

[raybg]

Hi,
I too have a same case where i am trying to convert a .txt file into .csv. It has | pipe, @ at symbol, ; semi colon. So while doing a read_csv, it is giving me this error.

ParserError: Error tokenizing data. C error: Expected 2 fields in line 274, saw 3

I do have multiple data in a particular row and that is not uniform, but thats how i would want it ..

Can anyone give any suggestion ??

[msaisumanth]

Can you give a sample of the data (A few example rows) ?

You need to make sure it is possible to split each row into columns appropriately based on an identifier (',' or '\t' for example).

[mangleddata]

Hi,
I too have a same case where i am trying to convert a .txt file into .csv. It has | pipe, @ at symbol, ; semi colon. So while doing a read_csv, it is giving me this error.

ParserError: Error tokenizing data. C error: Expected 2 fields in line 274, saw 3

I do have multiple data in a particular row and that is not uniform, but thats how i would want it ..

Can anyone give any suggestion ??

@raybg - I suspect you have issues with escaping certain cells incorrectly. Your error is a parsing error. See example

import sys
if sys.version_info[0] < 3: 
    from StringIO import StringIO
else:
    from io import StringIO
import pandas as pd

works = StringIO("""text,val
ludwig, 1
""")

donotwork = StringIO("""text,val
ludwig",", 1
""")

df = pd.read_csv(works, sep=",")
print(df.shape)

# this will throw an error
df = pd.read_csv(donotwork, sep=",")

[jazoom]

Thank you for the deep link to the docs section on CSV formatting at the very bottom of the "user guide". I didn't find that in my searching earlier today since there was no hint of it in any of the menu items and I was working through "getting started". I'm still not sure that info helps me, though. I'm also having problems with a CSV file.

CSV Format¶ Ludwig uses Pandas under the hood to read the UTF-8 encoded CSV files. Pandas tries to automatically identify the separator (generally ',') from the data. The default escape character is '\'. For example, if ',' is the column separator and one of your data columns has a ',' in it, Pandas would fail to load the data properly. To handle such cases, we expect the values in the columns to be escaped with backslashes (replace ',' in the data with '\,').

That says "default escape character", which suggests it can be changed? Or are you saying it is pandas' default and you're not surfacing a way to change it when using Ludwig? I'd love to change it to see if I can get the CSV working.

For some reason it's choking on a 1-row CSV file due to an apostrophe. I have no idea why this should make pandas crash: fish 'n' chips. Why would an apostrophe need to be escaped?

Also, how do we use tab-separation? You said that is supported. Do we just have to rely on autodetection?

I have mixed feeling about this request, on one hand it will provide flexibility

I vote yes allow passing through options. It would make this escaping/separating madness easier to debug and work around.

You mention here using python code to prepare the data first, but that goes against the very first promise of this library. Getting a CSV working is the most fundamental step, and at least in my case, I haven't even completed the "getting started" section yet because the very first CSV file I tried didn't work and I spent my whole afternoon trying to work out why the errors.

Tomorrow I'll try again now that I know the default escape character is different to the CSV export library I used. We'll see if that helps.

[jazoom]

I've just had another crack at this. Changing the output delimiter to \\ has not helped.

This two-line CSV file fails to train:

category,name
fashion,Tradie Scratch 'n' Sniff Trunk

Quoting the name doesn't make it work. Escaping the single quotes does. Do you have any idea why they need to be escaped? There are many single quotes throughout the 11 million lines of the training CSV file and most of them don't seem to bother the CSV parser.

Curiously, this does successfully train (TSV):

category  name
fashion Tradie Scratch 'n' Sniff Trunk

I guess Ludwig or pandas does auto-detect TSV. I'll re-export the data using TSV. Hopefully that helps. I still don't understand what the problem is with the single quotes when the separator is a comma.

[w4nderlust]

Thank you for the deep link to the docs section on CSV formatting at the very bottom of the "user guide". I didn't find that in my searching earlier today since there was no hint of it in any of the menu items and I was working through "getting started". I'm still not sure that info helps me, though. I'm also having problems with a CSV file.

There's a search bar on top ;)

CSV Format¶ Ludwig uses Pandas under the hood to read the UTF-8 encoded CSV files. Pandas tries to automatically identify the separator (generally ',') from the data. The default escape character is ''. For example, if ',' is the column separator and one of your data columns has a ',' in it, Pandas would fail to load the data properly. To handle such cases, we expect the values in the columns to be escaped with backslashes (replace ',' in the data with '\,').

That says "default escape character", which suggests it can be changed? Or are you saying it is pandas' default and you're not surfacing a way to change it when using Ludwig? I'd love to change it to see if I can get the CSV working.

Let me point you to the relevant piece of code, as in the meantime htere has a been a contributed PR with code that tries to figure out the separator automatically: https://github.com/uber/ludwig/blob/master/ludwig/utils/data_utils.py#L50-L80
We are also refactoring the prepocessing to allow for a more flexible process. This will likely allow you to provide additional parameters to the preprocessing pupeline withing the preprocessing section of the model definition, but it's not ready yet: https://github.com/uber/ludwig/tree/preprocessing_strategy

For some reason it's choking on a 1-row CSV file due to an apostrophe. I have no idea why this should make pandas crash: fish 'n' chips. Why would an apostrophe need to be escaped?

Tried this little python code:

>>> import pandas as pd
>>> data = {"text": ["fish 'n' chips"]}
>>> df1 = pd.DataFrame(data)
>>> df1
             text
0  fish 'n' chips
>>> df1.to_csv("my.csv", index=False)

The content of the my.csv is:

fish 'n' chips

Let's load it:

>>> df2 = pd.read_csv("my.csv")
>>> df2
             text
0  fish 'n' chips

no errors.
This suggests there's probably some other issue with your csv.

Also, how do we use tab-separation? You said that is supported. Do we just have to rely on autodetection?

Yes at the moment if should be autodetected (look at the code linked before).

I have mixed feeling about this request, on one hand it will provide flexibility

I vote yes allow passing through options. It would make this escaping/separating madness easier to debug and work around.

You mention here using python code to prepare the data first, but that goes against the very first promise of this library. Getting a CSV working is the most fundamental step, and at least in my case, I haven't even completed the "getting started" section yet because the very first CSV file I tried didn't work and I spent my whole afternoon trying to work out why the errors.

Providing well formatted CSV doesn't seem like a big ask imho and does not defeat the purpose of the tool at all. You could use any tool for that, not only code. It's unfortunate that you couldn't make it work, but again, like any ML or even any software in general with garbage input you obtain garbage outputs (or in this case no output at all).
You'll have a way to specify the separator or something like that in the future, because of the refactoring, but if pandas doesn't read your CSV, you have a more fundamental problem to solve first than training an ML model on it.

Tomorrow I'll try again now that I know the default escape character is different to the CSV export library I used. We'll see if that helps.

The escape of strings is adding "" around them and \ is for escaping commas in general. Both are the standard way pandas writes and reads strings:

>>> import pandas as pd
>>> data = {"text": ["Hello, world!"]}
>>> df1
            text
0  Hello, wolrd!
>>> df1.to_csv("my.csv", index=False)

The content of the my.csv is:

"Hello, wolrd!"

Let's load it:

>>> df2 = pd.read_csv("my.csv")
>>> df2
            text
0  Hello, wolrd!

[jazoom]

But that doesn't explain the problem with the single quotes. I just double-checked by manually typing out a CSV file to ensure no invisible characters are in there, and it still chokes on parsing.

category,name
fashion,Men 'S Shoes Fashion Martin Boots High Boots
fashion,Great Outdoors Dive In Print Boys' Recycolor T-Shirt 992

To add more ridiculousness to my story before, this CSV file:

category  name
fashion Tradie Scratch 'n' Sniff Trunk

did indeed SEEM to train, but, when I changed the extension to .tsv it no longer parses. Changing it back to CSV (while keeping the tab separation) makes it parse again. So probably it just thought the name was empty because it didn't find a separator, or something.

BUT, that is just that line. Other lines with single quotes still fail to parse (e.g. the first example I gave in this comment that I manually typed out).

What am I missing here? So is there a problem with that text right there? It's just ASCII characters in CSV format. Does the line ending matter? Yesterday I did try with windows-style line endings and it didn't make the CSV file parse.

[jazoom]

I can confirm that changing to Windows-style line endings doesn't make a difference for this CSV file:

category,name
fashion,Men 'S Shoes Fashion Martin Boots High Boots
fashion,Great Outdoors Dive In Print Boys' Recycolor T-Shirt 992

The trace:

Using full raw csv, no hdf5 and json file with the same name have been found
Building dataset (it may take a while)
Failed to parse the CSV with pandas default way, trying \ as escape character.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/ludwig-0.2.1-py3.6.egg/ludwig/utils/data_utils.py", line 73, in read_csv
  File "/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py", line 676, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py", line 454, in _read
    data = parser.read(nrows)
  File "/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py", line 1133, in read
    ret = self._engine.read(nrows)
  File "/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py", line 2037, in read
    data = self._reader.read(nrows)
  File "pandas/_libs/parsers.pyx", line 859, in pandas._libs.parsers.TextReader.read
  File "pandas/_libs/parsers.pyx", line 874, in pandas._libs.parsers.TextReader._read_low_memory
  File "pandas/_libs/parsers.pyx", line 928, in pandas._libs.parsers.TextReader._read_rows
  File "pandas/_libs/parsers.pyx", line 915, in pandas._libs.parsers.TextReader._tokenize_rows
  File "pandas/_libs/parsers.pyx", line 2070, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: Expected 8 fields in line 3, saw 9


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/ludwig", line 11, in <module>
    load_entry_point('ludwig==0.2.1', 'console_scripts', 'ludwig')()
  File "/usr/local/lib/python3.6/dist-packages/ludwig-0.2.1-py3.6.egg/ludwig/cli.py", line 108, in main
  File "/usr/local/lib/python3.6/dist-packages/ludwig-0.2.1-py3.6.egg/ludwig/cli.py", line 64, in __init__
  File "/usr/local/lib/python3.6/dist-packages/ludwig-0.2.1-py3.6.egg/ludwig/cli.py", line 74, in train
  File "/usr/local/lib/python3.6/dist-packages/ludwig-0.2.1-py3.6.egg/ludwig/train.py", line 804, in cli
  File "/usr/local/lib/python3.6/dist-packages/ludwig-0.2.1-py3.6.egg/ludwig/train.py", line 301, in full_train
  File "/usr/local/lib/python3.6/dist-packages/ludwig-0.2.1-py3.6.egg/ludwig/data/preprocessing.py", line 339, in preprocess_for_training
  File "/usr/local/lib/python3.6/dist-packages/ludwig-0.2.1-py3.6.egg/ludwig/data/preprocessing.py", line 485, in preprocess_for_training_by_type
  File "/usr/local/lib/python3.6/dist-packages/ludwig-0.2.1-py3.6.egg/ludwig/data/preprocessing.py", line 614, in _preprocess_csv_for_training
  File "/usr/local/lib/python3.6/dist-packages/ludwig-0.2.1-py3.6.egg/ludwig/data/preprocessing.py", line 60, in build_dataset
  File "/usr/local/lib/python3.6/dist-packages/ludwig-0.2.1-py3.6.egg/ludwig/utils/data_utils.py", line 78, in read_csv
  File "/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py", line 676, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py", line 454, in _read
    data = parser.read(nrows)
  File "/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py", line 1133, in read
    ret = self._engine.read(nrows)
  File "/usr/local/lib/python3.6/dist-packages/pandas/io/parsers.py", line 2037, in read
    data = self._reader.read(nrows)
  File "pandas/_libs/parsers.pyx", line 859, in pandas._libs.parsers.TextReader.read
  File "pandas/_libs/parsers.pyx", line 874, in pandas._libs.parsers.TextReader._read_low_memory
  File "pandas/_libs/parsers.pyx", line 928, in pandas._libs.parsers.TextReader._read_rows
  File "pandas/_libs/parsers.pyx", line 915, in pandas._libs.parsers.TextReader._tokenize_rows
  File "pandas/_libs/parsers.pyx", line 2070, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: Expected 8 fields in line 3, saw 9

The command:

ludwig train --skip_save_processed_input --data_csv out.csv --model_definition "{input_features: [{name: name, type: text}], output_features: [{name: category, type: category}]}"

[jazoom]

And you should be able to reproduce this exactly, because I used a build of your Dockerfile to run all this:

docker run -it --rm --entrypoint=/bin/sh -v /path:/data local/ludwig

[w4nderlust]

Given the CSV:

category,name
fashion,Tradie Scratch 'n' Sniff Trunk

It can be loaded by Ludwig without problems:

>>> from ludwig.utils import data_utils
>>> df = data_utils.read_csv("my.csv")
>>> df
                                 category,name
fashion,Tradie Scratch 'n' Sniff         Trunk

But given the CSV:

category,name
fashion,Men 'S Shoes Fashion Martin Boots High Boots
fashion,Great Outdoors Dive In Print Boys' Recycolor T-Shirt 992

I got a similar error.
The reason was that whitespace was among the possible separators, and the function that determines automatically the delimiter was returning whitespace as the tokenization character instead of comma.

Did a minor fix and now it works:

>>> from ludwig.utils import data_utils
>>> df = data_utils.read_csv("my.csv")
>>> df
  category                                               name
0  fashion       Men 'S Shoes Fashion Martin Boots High Boots
1  fashion  Great Outdoors Dive In Print Boys' Recycolor T...

Pushing it. Please install Ludwig from master pip install git+http://github.com/uber/ludwig.git and confirm that now it works.

[jazoom]

Thanks @w4nderlust. I can confirm that has fixed all the problems.

And now that I have successfully trained something I can also confirm that Ludwig is bloody awesome. The user experience is much better than any other machine learning tool I've used. Sensible defaults plus configuration works well. Thank you for creating this and making it open source.

[w4nderlust]

You're very welcome!