Datasets: Issue to read a local dataset

Created on 14 Apr 2020  ·  5Comments  ·  Source: huggingface/datasets

Hello,

As proposed by @thomwolf, I open an issue to explain what I'm trying to do without success. What I want to do is to create and load a local dataset, the script I have done is the following:

import os
import csv

import nlp


class BbcConfig(nlp.BuilderConfig):
    def __init__(self, **kwargs):
        super(BbcConfig, self).__init__(**kwargs)


class Bbc(nlp.GeneratorBasedBuilder):
    _DIR = "./data"
    _DEV_FILE = "test.csv"
    _TRAINING_FILE = "train.csv"

    BUILDER_CONFIGS = [BbcConfig(name="bbc", version=nlp.Version("1.0.0"))]

    def _info(self):
        return nlp.DatasetInfo(builder=self, features=nlp.features.FeaturesDict({"id": nlp.string, "text": nlp.string, "label": nlp.string}))

    def _split_generators(self, dl_manager):
        files = {"train": os.path.join(self._DIR, self._TRAINING_FILE), "dev": os.path.join(self._DIR, self._DEV_FILE)}

        return [nlp.SplitGenerator(name=nlp.Split.TRAIN, gen_kwargs={"filepath": files["train"]}),
                nlp.SplitGenerator(name=nlp.Split.VALIDATION, gen_kwargs={"filepath": files["dev"]})]

    def _generate_examples(self, filepath):
        with open(filepath) as f:
            reader = csv.reader(f, delimiter=',', quotechar="\"")
            lines = list(reader)[1:]

            for idx, line in enumerate(lines):
                yield idx, {"idx": idx, "text": line[1], "label": line[0]}

The dataset is attached to this issue as well:
data.zip

Now the steps to reproduce what I would like to do:

  1. unzip data locally (I know the nlp lib can detect and extract archives but I want to reduce and facilitate the reproduction as much as possible)
  2. create the bbc.py script as above at the same location than the unziped data folder.

Now I try to load the dataset in three different ways and none works, the first one with the name of the dataset like I would do with TFDS:

import nlp
from bbc import Bbc
dataset = nlp.load("bbc")

I get:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/anaconda3/envs/transformers/lib/python3.7/site-packages/nlp/load.py", line 280, in load
    dbuilder: DatasetBuilder = builder(path, name, data_dir=data_dir, **builder_kwargs)
  File "/opt/anaconda3/envs/transformers/lib/python3.7/site-packages/nlp/load.py", line 166, in builder
    builder_cls = load_dataset(path, name=name, **builder_kwargs)
  File "/opt/anaconda3/envs/transformers/lib/python3.7/site-packages/nlp/load.py", line 88, in load_dataset
    local_files_only=local_files_only,
  File "/opt/anaconda3/envs/transformers/lib/python3.7/site-packages/nlp/utils/file_utils.py", line 214, in cached_path
    if not is_zipfile(output_path) and not tarfile.is_tarfile(output_path):
  File "/opt/anaconda3/envs/transformers/lib/python3.7/zipfile.py", line 203, in is_zipfile
    with open(filename, "rb") as fp:
TypeError: expected str, bytes or os.PathLike object, not NoneType

But @thomwolf told me that no need to import the script, just put the path of it, then I tried three different way to do:

import nlp
dataset = nlp.load("bbc.py")

And

import nlp
dataset = nlp.load("./bbc.py")

And

import nlp
dataset = nlp.load("/absolute/path/to/bbc.py")

These three ways gives me:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/anaconda3/envs/transformers/lib/python3.7/site-packages/nlp/load.py", line 280, in load
    dbuilder: DatasetBuilder = builder(path, name, data_dir=data_dir, **builder_kwargs)
  File "/opt/anaconda3/envs/transformers/lib/python3.7/site-packages/nlp/load.py", line 166, in builder
    builder_cls = load_dataset(path, name=name, **builder_kwargs)
  File "/opt/anaconda3/envs/transformers/lib/python3.7/site-packages/nlp/load.py", line 124, in load_dataset
    dataset_module = importlib.import_module(module_path)
  File "/opt/anaconda3/envs/transformers/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'nlp.datasets.2fd72627d92c328b3e9c4a3bf7ec932c48083caca09230cebe4c618da6e93688.bbc'

Any idea of what I'm missing? or I might have spot a bug :)

picture jplu
👍1

Most helpful comment

Ok, there are some news, most good than bad :laughing:

The dataset script now became:

import csv

import nlp


class Bbc(nlp.GeneratorBasedBuilder):
    VERSION = nlp.Version("1.0.0")

    def __init__(self, **config):
        self.train = config.pop("train", None)
        self.validation = config.pop("validation", None)
        super(Bbc, self).__init__(**config)

    def _info(self):
        return nlp.DatasetInfo(builder=self, description="bla", features=nlp.features.FeaturesDict({"id": nlp.int32, "text": nlp.string, "label": nlp.string}))

    def _split_generators(self, dl_manager):
        return [nlp.SplitGenerator(name=nlp.Split.TRAIN, gen_kwargs={"filepath": self.train}),
                nlp.SplitGenerator(name=nlp.Split.VALIDATION, gen_kwargs={"filepath": self.validation})]

    def _generate_examples(self, filepath):
        with open(filepath) as f:
            reader = csv.reader(f, delimiter=',', quotechar="\"")
            lines = list(reader)[1:]

            for idx, line in enumerate(lines):
                yield idx, {"id": idx, "text": line[1], "label": line[0]}

And the dataset folder becomes:

.
├── bbc
│   ├── bbc.py
│   └── data
│       ├── test.csv
│       └── train.csv

I can load the dataset by using the keywords arguments like this:

import nlp
dataset = nlp.load("bbc", builder_kwargs={"train": "bbc/data/train.csv", "validation": "bbc/data/test.csv"})

That was the good part ^^ Because it took me some time to understand that the script itself is put in cache in datasets/src/nlp/datasets/some-hash/bbc.py which is very difficult to discover without checking the source code. It means that doesn't matter the changes you do to your original script it is taken into account. I think instead of doing a hash on the name (I suppose it is the name), a hash on the content of the script itself should be a better solution.

Then by diving a bit in the code I found the force_reload parameter here but the call of this load_dataset method is done with the builder_kwargs as seen here which is ok until the call to the builder is done as the builder do not have this force_reload parameter. To show as example, the previous load becomes:

import nlp
dataset = nlp.load("bbc", builder_kwargs={"train": "bbc/data/train.csv", "validation": "bbc/data/test.csv", "force_reload": True})

Raises

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jplu/dev/jplu/datasets/src/nlp/load.py", line 283, in load
    dbuilder: DatasetBuilder = builder(path, name, data_dir=data_dir, **builder_kwargs)
  File "/home/jplu/dev/jplu/datasets/src/nlp/load.py", line 170, in builder
    builder_instance = builder_cls(**builder_kwargs)
  File "/home/jplu/dev/jplu/datasets/src/nlp/datasets/84d638d2a8ca919d1021a554e741766f50679dc6553d5a0612b6094311babd39/bbc.py", line 12, in __init__
    super(Bbc, self).__init__(**config)
TypeError: __init__() got an unexpected keyword argument 'force_reload'

So yes the cache is refreshed with the new script but then raises this error.

picture jplu on 15 Apr 2020
❤1 👍1

All 5 comments

My first bug report ❤️
Looking into this right now!

thomwolf picture thomwolf on 14 Apr 2020
👍2

Ok, there are some news, most good than bad :laughing:

The dataset script now became:

import csv

import nlp


class Bbc(nlp.GeneratorBasedBuilder):
    VERSION = nlp.Version("1.0.0")

    def __init__(self, **config):
        self.train = config.pop("train", None)
        self.validation = config.pop("validation", None)
        super(Bbc, self).__init__(**config)

    def _info(self):
        return nlp.DatasetInfo(builder=self, description="bla", features=nlp.features.FeaturesDict({"id": nlp.int32, "text": nlp.string, "label": nlp.string}))

    def _split_generators(self, dl_manager):
        return [nlp.SplitGenerator(name=nlp.Split.TRAIN, gen_kwargs={"filepath": self.train}),
                nlp.SplitGenerator(name=nlp.Split.VALIDATION, gen_kwargs={"filepath": self.validation})]

    def _generate_examples(self, filepath):
        with open(filepath) as f:
            reader = csv.reader(f, delimiter=',', quotechar="\"")
            lines = list(reader)[1:]

            for idx, line in enumerate(lines):
                yield idx, {"id": idx, "text": line[1], "label": line[0]}

And the dataset folder becomes:

.
├── bbc
│   ├── bbc.py
│   └── data
│       ├── test.csv
│       └── train.csv

I can load the dataset by using the keywords arguments like this:

import nlp
dataset = nlp.load("bbc", builder_kwargs={"train": "bbc/data/train.csv", "validation": "bbc/data/test.csv"})

That was the good part ^^ Because it took me some time to understand that the script itself is put in cache in datasets/src/nlp/datasets/some-hash/bbc.py which is very difficult to discover without checking the source code. It means that doesn't matter the changes you do to your original script it is taken into account. I think instead of doing a hash on the name (I suppose it is the name), a hash on the content of the script itself should be a better solution.

Then by diving a bit in the code I found the force_reload parameter here but the call of this load_dataset method is done with the builder_kwargs as seen here which is ok until the call to the builder is done as the builder do not have this force_reload parameter. To show as example, the previous load becomes:

import nlp
dataset = nlp.load("bbc", builder_kwargs={"train": "bbc/data/train.csv", "validation": "bbc/data/test.csv", "force_reload": True})

Raises

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jplu/dev/jplu/datasets/src/nlp/load.py", line 283, in load
    dbuilder: DatasetBuilder = builder(path, name, data_dir=data_dir, **builder_kwargs)
  File "/home/jplu/dev/jplu/datasets/src/nlp/load.py", line 170, in builder
    builder_instance = builder_cls(**builder_kwargs)
  File "/home/jplu/dev/jplu/datasets/src/nlp/datasets/84d638d2a8ca919d1021a554e741766f50679dc6553d5a0612b6094311babd39/bbc.py", line 12, in __init__
    super(Bbc, self).__init__(**config)
TypeError: __init__() got an unexpected keyword argument 'force_reload'

So yes the cache is refreshed with the new script but then raises this error.

jplu picture jplu on 15 Apr 2020
❤1 👍1

Ok great, so as discussed today, let's:

  • have a main dataset directory inside the lib with sub-directories hashed by the content of the file
  • keep a cache for downloading the scripts from S3 for now
  • later: add methods to list and clean the local versions of the datasets (and the distant versions on S3 as well)

Side question: do you often use builder_kwargs for other things than supplying file paths? I was thinking about having a more easy to read and remember data_files argument maybe.

thomwolf picture thomwolf on 15 Apr 2020

Good plan!

Yes I do use builder_kwargs for other things such as:

  • dataset name
  • properties to know how to properly read a CSV file: do I have to skip the first line in a CSV, which delimiter is used, and the columns ids to use.
  • properties to know how to properly read a JSON file: which properties in a JSON object to read
jplu picture jplu on 15 Apr 2020
👍1

Done!

jplu picture jplu on 11 May 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

david-waterworth picture david-waterworth  ·  3Comments
astariul picture astariul  ·  6Comments
ratthachat picture ratthachat  ·  6Comments
NielsRogge picture NielsRogge  ·  3Comments
carlos-aguayo picture carlos-aguayo  ·  6Comments