Farm: Finetune LM Preprocessing - Multiprocessing Error?

[lingsond]

I'm trying to finetune with old german texts (using bert-base-german-case model), running the script in Kubernetes in Docker image. I have some questions:

• Sometimes the pod running the script will produce an error like this:

Preprocessing Dataset /data/german_old_texts/processed/lm/train.txt:   2%|1         | 7/451 [02:39<2:48:40, 22.79s/ Dicts]
Traceback (most recent call last):
  File "finetune_lm.py", line 61, in <module>
    max_processes=20
  File "/home/user/farm/data_handler/data_silo.py", line 105, in __init__
    self._load_data()
  File "/home/user/farm/data_handler/data_silo.py", line 205, in _load_data
    self.data["train"], self.tensor_names = self._get_dataset(train_file)
  File "/home/user/farm/data_handler/data_silo.py", line 176, in _get_dataset
    for dataset, tensor_names in results:
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 735, in next
    raise value
multiprocessing.pool.MaybeEncodingError: Error sending result: '(<torch.utils.data.dataset.TensorDataset object at 0x7f3341f74f98>, ['input_ids', 'padding_mask', 'segment_ids', 'lm_label_ids', 'nextsentence_label_ids'])'. Reason: 'RuntimeError('unable to write to file </torch_70_633414359>',)'

However, this error doesn't occur everytime. What can I do to avoid this error?

• I run two version of the script (one with a huge training file with a lot of texts) and one with smaller training file (where I only put a small portion of the texts into the train.txt).
  The huge train.txt is more than 400MB in size, and the smaller one is only around 7MB in size. Both scripts use same parameters (Training file are in separate folders, the cache and save dir are also different so there shouldn't be any conflicts). However, both scripts will freeze (I don't know if it's because the size of the dataset or it is really freezing, but the cpu/gpu load is minimal, so we assume it is freezing) after showing the second sample at this line:

nextsentence_label_ids: [0]
_____________________________________________________

(This happened whether I run just one version of script at a time, or both running parallel)
Is it still processing or is it freezing? What cause this problem?

• Although both scripts are using same parameters, the one with smaller training file will only assign one worker even though I set max_processes=20 for the DataSilo parameter.

Setting in YAML file:

resources:
            limits:
              nvidia.com/gpu: "2"
              cpu: "20"
              memory: "32Gi"

But when using the big training file, 20 parallel workers were assigned to process the dataset. Why doesn't it assign more than 1 worker when processing the small training file?

• Should the script work without error/freezing, how long should I expect the training to run for the small training file (the one with about 7MB in size) so I know it is processing instead of freezing?

Additional Info:
When I tried to add this to the code:

from multiprocessing import set_start_method
set_start_method("spawn")

after showing the second sample and after the same line as in previous question, what happened is this:

```
nextsentence_label_ids: [0] _____________________________________________________ 07/13/2020 22:11:28 - INFO - transformers.file_utils - PyTorch version 1.5.0 available. 07/13/2020 22:11:28 - INFO - farm.utils - device: cuda n_gpu: 2, distributed training: False, automatic mixed precision training: None
07/13/2020 22:12:12 - INFO - transformers.file_utils - PyTorch version 1.5.0 available. 07/13/2020 22:12:12 - INFO - farm.utils - device: cuda n_gpu: 2, distributed training: False, automatic mixed precision training: None

```
And then it freezes again.
We added this code in the hope to solve the first problem (with the multiprocessing error). I don't know if this information will be useful, but I just add it here.

Please help. I've been trying to do this finetuning in kubernetes since weeks already without success (I first tested the script for finetuning on Google Colab with a really small training data and it has worked without error). I don't know what I should do anymore to make it work. Please let me know if I should add any information. Thanks in advance.

[Timoeller]

Hey @lingsond good that you contacted us, I can relate to the confusion and difficulties you went through when working on large datafiles.

I think there are two very quick solutions to your issues:

1. Please try out the StreamingDataSilo to load your data as used in our train from scratch example. Be aware that you have to change the way the number of batches are computed for the optimizer when using the StreamingDataSilo n_batches=len(stream_data_silo.get_data_loader("train"))
2. Your train.txt as well as your smaller file seem to have very large single documents inside of them. Please separate documents inside these files by a newline.

Explanations to the solutions
When not using streaming approach all data will be loaded into memory, which is quite a lot. You "just" assigned 32GB to your machine, in some of our use cases more than 200GB is needed. Your particular error though possibly comes from very large multiprocessing chunks. So your data is split into chunks (determined by max_multiprocessing_chunksize parameter). If these chunks are very large (possibly due to your train.txt file contains large documents) an error like "'RuntimeError('unable to write to file ',)'" occurs.

Hope that can help you. We will be able to assist if further problems occur.
Maybe post your small train file for a quick check if you can?

[lingsond]

Hi @Timoeller , thanks for your reply.

• I'm trying to change my code to use the StreamingDataSilo. This might be stupid, and I think what I write should be the same as in the example, but I did get an import error.
  Here's my imports:

# from multiprocessing import set_start_method

import logging
import os
import json
import math
from pathlib import Path
from tqdm import tqdm, trange

from farm.modeling.tokenization import Tokenizer
from farm.data_handler.data_silo import DataSilo, StreamingDataSilo
from farm.data_handler.processor import BertStyleLMProcessor
from farm.data_handler.utils import split_file
from farm.modeling.adaptive_model import AdaptiveModel
from farm.modeling.language_model import LanguageModel
from farm.modeling.optimization import initialize_optimizer
from farm.modeling.prediction_head import BertLMHead, NextSentenceHead
from farm.train import Trainer, EarlyStopping
from farm.utils import set_all_seeds, initialize_device_settings

import torch

And this is the error I'm getting:

Traceback (most recent call last):
  File "finetune_lm_split_sleep.py", line 13, in <module>
    from farm.data_handler.utils import split_file
ImportError: cannot import name 'split_file'

Did I miss anything?

• The big file contains about 450+ text files, and the small training file contains only 20 text files (which individually has the size of about 300KB to 400KB). In the combining process, a blank/new line is added to separate the documents (so it is not just one big document because when I tried to just use one document as a training file, it will cause another error, so I assume it is not possible to train just one single document).
• Here are additional parameters information:
  max_seq_len=512 (for BertStyleLMProcessor), batch_size=1, max_multiprocessing_chunksize=20, max_processes=20 (for DataSilo).
  The length of the text in each line is quite long, thus the need for max_seq_len=512. But because of that I have decrease the batch_size to 1. Is the setting for max_multiprocessing_chunksize=20 still too big? What number should I ideally use for it then?
• Posting the file online is out of the question because of data privacy etc as these data belong to the university I worked for. However, I can ask my superior if it's okay to send the file privately to you should your solution with the StreamingDataSilo still doesn't work.

Thanks for the help.

[lingsond]

Never mind the import error. It turns out that I was still using version 0.4.3 where the split_file hasn't been implemented.

[lingsond]

I've change the code accordingly like the example. but I still got an error:

Traceback (most recent call last):
  File "finetune_lm_split_sleep.py", line 223, in <module>
    main()
  File "finetune_lm_split_sleep.py", line 218, in main
    train_batch()
  File "finetune_lm_split_sleep.py", line 170, in train_batch
    n_batches=len(data_silo.get_data_loader["train"]),
TypeError: 'method' object is not subscriptable

That line is the one in the initialize_optimizer:

# 5. Create an optimizer
        warmup_proportion = 0.05
        grad_acc = 1
        model, optimizer, lr_schedule = initialize_optimizer(
            model=model,
            learning_rate=2e-5,
            schedule_opts={"name": "LinearWarmup", "warmup_proportion": warmup_proportion},
            grad_acc_steps=grad_acc,
            device=device,
            n_batches=len(data_silo.get_data_loader["train"]),
            n_epochs=n_epochs,
        )

But it's a great progress since I never get so far before :)
What cause this error and how can I fix it?

[Timoeller]

Nice self service about the import error : ) The next error is about method vs data index. As I have written you should use: n_batches=len(stream_data_silo.get_data_loader("train")). there get_data_loader is a function that takes an argument. In the normal datasilo case it is a dictionary that takes keys.

About your dataset, I think you do not need to send it via pm, I think your explanations agree with our hypothesis. If you do not have many docs in your txt file only one (or a few) processes are used. This is because each process needs to select random documents for the next sentence prediction task of Bert finetuning. If you only have 20 docs all of them will be assigned to one process.

[lingsond]

I didn't change the variable data_silo to stream_data_silo but i still use the StreamingDataSilo.

data_loader_workers = 10
data_silo = StreamingDataSilo(processor=processor, batch_size=batch_size, dataloader_workers=data_loader_workers)

[lingsond]

Just to complete the information, here's the processor:

 processor = BertStyleLMProcessor(
            data_dir=Path('/data/german_old_texts/processed/lm_batch'),
            tokenizer=tokenizer,
            max_seq_len=512,
            max_docs=None,
            next_sent_pred_style=next_sent_pred_style,
            train_filename=file_name,
            dev_filename=None,
            test_filename=None,
            dev_split=0.1
        )

next_sent_pred_style is 'bert-style', and for tokenizer I used 'bert-base-german-cased' as languange model.

[Timoeller]

I didn't change the variable data_silo to stream_data_silo but i still use the StreamingDataSilo.

that is why you have to change how n_batches is computed. One is a method call with ("train") the other one a dictionary access with ["train"]. I agree this handling is confusing and we should unify the usage.

Does it work with the StreamingDataSilo then?

[lingsond]

Hi @Timoeller , it seems that you have misunderstood me. What I mean is, I didn't change the variable name because I only use one data silo in the script. But I did change the code to use StreamingDataSilo as you can see in the code I posted above. Besides, if using the normal DataSilo, it uses data_silo.loaders instead of get_data_loader.

However, from your explanation I've realized my mistake. In my code, I changed the data_silo.loaders to data_silo.get_data_loader, but I didn't notice the bracket's difference, so I didn't change the them from [ ] to ( ). After changing the brackets, everything work as perfectly. It is currently doing the training, but everything looks good for now. And the speed is even faster than I expected. Wunderbar!

Vielen Dank. I've been struggling with this problem for a long time already.

Btw, I didn't use the split_file yet because it caused this error:

Splitting file ...:   5%|5         | 127877/2407713 [00:00<00:02, 869200.61it/s]
Traceback (most recent call last):
  File "finetune_lm.py", line 43, in <module>
    split_file(data_dir / "train.txt", output_dir=Path('/data/german_old_texts/processed/lm/split_files'), docs_per_file=20)
  File "/home/user/farm/data_handler/utils.py", line 785, in split_file
    write_file.writelines(lines_to_write)
UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 62: ordinal not in range(128)

From the source, the default should be using utf-8 for the encoding.

If I'm not mistaken, in the source code of the split_file method, when it open read the file, it uses the encoding, but during the writing process, the encoding was omitted. Perhaps that caused the error?

[Timoeller]

Nice, always happy to help.

About the split_file encoding issue. This might be a bug and your proposed solution seems reasonable.
Would you like to create a separate issue with this bug and maybe a minimal example showing the error? If you want you could even contribute a PR adding the encoding to the writing?

[lingsond]

Ok, I'll create a separate issue for the split_file. However, I'm still not too familiar with git so I don't know how to contribute a PR.

Since the original issue with the multiprocessing problem is solved, should I close this issue?

[Timoeller]

I will close this one for you.

About creating a PR: here is a nice tutorial, starting from step 2: https://medium.com/@jenweber/your-first-open-source-contribution-a-step-by-step-technical-guide-d3aca55cc5a6