Ludwig: Can't split set features by comma

[anakin87]

Describe the bug
In python, while defining set feature (with format='comma'), can't split it by comma.

To Reproduce

import pandas as pd
df=pd.DataFrame({'feat': ['di maio, salvini', 'conte, della loggia'], 'target': [0, 1]})

import ludwig
from ludwig.api import LudwigModel
model_definition={'input_features':[ {'name':'feat','type':'set', 'format':'comma'}],
              'output_features':[{'name':'target','type':'binary'}],
              'training':{'learning_rate':0.01,
                              'batch_size': 10,
                              'epochs': 1000,
                              'early_stop': 5,
                          'validation_measure':'loss','validation_field':'target'}}


model = LudwigModel(model_definition)

train_stats = model.train(df)

Obtained training_set_metadata.json is the following:

{
    "feat": {
        "idx2str": [
            "<PAD>",
            "<UNK>",
            "di",
            "maio,",
            "salvini",
            "conte,",
            "della",
            "loggia"
        ],
        "max_set_size": 3,
        "preprocessing": {
            "computed_fill_value": "<UNK>",
            "fill_value": "<UNK>",
            "lowercase": false,
            "missing_value_strategy": "fill_with_const",
            "most_common": 10000,
            "tokenizer": "space"
        },
        "str2freq": {
            "<PAD>": 0,
            "<UNK>": 0,
            "conte,": 1,
            "della": 1,
            "di": 1,
            "loggia": 1,
            "maio,": 1,
            "salvini": 1
        },
        "str2idx": {
            "<PAD>": 0,
            "<UNK>": 1,
            "conte,": 5,
            "della": 6,
            "di": 2,
            "loggia": 7,
            "maio,": 3,
            "salvini": 4
        },
        "vocab_size": 8
    },
    "target": {
        "preprocessing": {
            "computed_fill_value": 0,
            "fill_value": 0,
            "missing_value_strategy": "fill_with_const"
        }
    }
}

Set feature is splitted by space, not by comma.

Expected behavior
Set feature should be splitted by comma, when specified with format parameter.

Environment

• OS: Google Colab
• Python version: 3.6.9
• ludwig version: 0.3.1
• pandas version: 1.1.4

I tried to understand why this happen...
COMMA_SPLITTER works properly...

[tgaddair]

Hey @anakin87, the format param was changed to tokenizer in v0.3. Apologies that this hasn't been updated in the official docs yet.

Try changing your input features as follows:

'input_features':[ {'name':'feat','type':'set', 'tokenizer':'comma'}]

[w4nderlust]

Updated the docs website to reflect this.