Datasets: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors

[BramVanroy]

I am trying out the library and want to load in pickled data with from_dict. In that dict, one column text should be tokenized and the other (an embedding vector) should be retained. All other columns should be removed. When I eventually try to set the format for the columns with set_format I am getting this strange Userwarning without a stack trace:

Set __getitem__(key) output type to torch for ['input_ids', 'sembedding'] columns (when key is int or slice) and don't output other (un-formatted) columns.
C:\Users\bramv.virtualenvs\dutch-simplification-nbNdqK9u\lib\site-packages\datasets\arrow_dataset.py:835: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at ..\torch\csrc\utils\tensor_numpy.cpp:141.)
return torch.tensor(x, **format_kwargs)

The first one might not be related to the warning, but it is odd that it is shown, too. It is unclear whether that is something that I should do or something that that the program is doing at that moment.

Snippet:

    dataset = Dataset.from_dict(torch.load("data/dummy.pt.pt"))
    print(dataset)
    tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
    keys_to_retain = {"input_ids", "sembedding"}
    dataset = dataset.map(lambda example: tokenizer(example["text"], padding='max_length'), batched=True)
    dataset.remove_columns_(set(dataset.column_names) - keys_to_retain)

    dataset.set_format(type="torch", columns=["input_ids", "sembedding"])
    dataloader = torch.utils.data.DataLoader(dataset, batch_size=2)

    print(next(iter(dataloader)))

PS: the input type for remove_columns_ should probably be an Iterable rather than just a List.

[sheffier]

I have the same issue

[ClaartjeBarkhof]

Same issue here when Trying to load a dataset from disk.

[samuelstevens]

I am also experiencing this issue, and don't know if it's affecting my training.

[kyoungrok0517]

Same here. I hope the dataset is not being modified in-place.

[thomwolf]

I think the only way to avoid this warning would be to do a copy of the numpy array before providing it.

This would slow down a bit the iteration over the dataset but maybe it would be safer. We could disable the copy with a flag on the set_format command.

In most typical cases of training a NLP model, PyTorch shouldn't modify the input so it's ok to have a non-writable array but I can understand the warning is a bit scary so maybe we could choose the side of non-warning/slower by default and have an option to speedup.

What do you think @lhoestq ?

[AndreasMadsen]

@thomwolf Would it be possible to have the array look writeable, but raise an error if it is actually written to?

I would like to keep my code free of warning, but I also wouldn't like to slow down the program because of unnecessary copy operations.

[thomwolf]

@AndreasMadsen probably not I would guess (no free lunch hahah)

[AndreasMadsen]

@thomwolf Why not? Writable is checked with arr.flags.writeable, and writing is done via magic methods.

[thomwolf]

Well because I don't know the internal of numpy as well as you I guess hahahah, do you want to try to open a PR proposing a solution?

[BramVanroy]

@thomwolf @AndreasMadsen I think this is a terrible idea, n/o, and I am very much against it. Modifying internals of an array in such a hacky way is bound to run into other (user) issues down the line. To users it would not be clear at all what is going on e.g. when they check for write access (which will return True) but then they get a warning that the array is not writeable. That's extremely confusing.

If your only goal is to get rid of warnings in your code, then you can just use a simplefilter for UserWarnings in your own code. Changing the code-base in such an intuitive way does not seem like a good way to go and sets a bad precedent, imo.

(Feel free to disagree, of course.)

IMO a warning can stay (as they can be filtered by users anyway), but it can be clarified why the warning takes place.

[AndreasMadsen]

To users it would not be clear at all what is going on e.g. when they check for write access (which will return True) but then they get a warning that the array is not writeable. That's extremely confusing.

Confusion can be resolved with a helpful error message. In this case, that error message can be controlled by huggingface/datasets. The right argument here is that if code depends on .flags.writable being truthful (not just for warnings), then it will cause unavoidable errors. Although, I can't imagine such a use-case.

If your only goal is to get rid of warnings in your code, then you can just use a simplefilter for UserWarnings in your own code. Changing the code-base in such an intuitive way does not seem like a good way to go and sets a bad precedent, imo.

I don't want to ignore all UserWarnings, nor all warnings regarding non-writable arrays. Ignoring warnings leads to hard to debug issues.

IMO a warning can stay (as they can be filtered by users anyway), but it can be clarified why the warning takes place.

Plain use cases should really not generate warnings. It teaches developers to ignore warnings which is a terrible practice.

---

The best solution would be to allow non-writable arrays in DataLoader, but that is a PyTorch issue.

[BramVanroy]

The right argument here is that if code depends on .flags.writable being truthful (not just for warnings), then it will cause unavoidable errors. Although, I can't imagine such a use-case.

That's exactly the argument in my first sentence. Too often someone "cannot think of a use-case", but you can not foresee the use-cases of a whole research community.

I don't want to ignore all UserWarnings, nor all warnings regarding non-writable arrays. Ignoring warnings leads to hard to debug issues.

That's fair.

Plain use cases should really not generate warnings. It teaches developers to ignore warnings which is a terrible practice.

But this is not a plain use-case (because Pytorch does not support these read-only tensors). Manually setting the flag to writable will solve the issue on the surface but is basically just a hack to compensate for something that is not allowed in another library.

What about an "ignore_warnings" flag in set_format that when True wraps the offending code in a block to ignore userwarnings at that specific step in _convert_outputs? Something like:

def _convert_outputs(..., ignore_warnings=True):
    ...
    with warnings.catch_warnings():
        if ignore_warnings:
            warnings.simplefilter("ignore", UserWarning)
        return torch.tensor(...)
# continues without warning filter after context manager...

[AndreasMadsen]

But this is not a plain use-case (because Pytorch does not support these read-only tensors).

By "plain", I mean the recommended way to use datasets with PyTorch according to the datasets documentation.