Farm: Accessing dataset index after evaluation

[malteos]

Question

Hi all,

thanks for your great work! I was wondering why the dataset samples don't have an ID set. See https://github.com/deepset-ai/FARM/blob/89826c1f4269a5c0d0b5a6193ece94a3401766d9/farm/data_handler/processor.py#L492

Is there any way to obtain the ID/index/line number of a particular sample in the results of Inferencer or Evaluator?

Or is it ensured that the order of the results is the same as in the test file, i.e. index of results list is equal to line number of test.tsv?

Best,
Malte

Additional context
-

[Timoeller]

Hey @malteos thanks for the kudos : )

Your ID issue addresses a valid concern.
Short answer: You can disable multiprocessing by setting max_processes=1 in the data_silo, then the order of results is ensured.

Long answer: We want to fill this ID value with sensible values and are working on a solution that works for all downstream tasks. In particular, we are not happy with the format of results in our Inferencer (nested dicts) and will transition to interface classes with defined attributes - one of them will be the ID.
We cannot really commit to a timeline on this feature, though I hope the disabling of multiprocessing is a valid solution for you?

[malteos]

Can I set a different max_processes for train and test? Processing all training data with a single process would be a time killer.. but just test set would be fine ;)

[Timoeller]

I thought you wanted to use the Inferencer to get model predictions from a model that has been trained before (the Inferencer needs a trained model anyways)?

So you could use multiprocessing during training for creating and saving the model.
Then you can load it without multiprocessing, so the order of results is assured with:
model = Inferencer.load("path/to/FARM_model",num_processes=0)
//you might have to update FARM, the num_processes inside the load() method is only a week old

You can do inference from dicts (with the structure from the example script) or inference from file, if you already divided train and test set.

[brandenchan]

So we just tested out the inferencer even with multiprocessing and found that the output is in fact in the same order as the input. This is because we are calling self.process_pool.imap() and not imap_unordered() (see Inferencer._inference_with_multiprocessing()).

As for why the Sample ids are not set, this is because the id is actually assigned after TextClassificationProcessor.dict_to_samples. If you're interested in how this is done, have a look at Processor._init_samples_in_baskets().

If you would still like the output to include the FARM internal id, you could have a look at TextClassificationHead.formatted_preds(). You would need to extract the id from each sample and then add it to the dict that gets returned. If you would like any help with that, just let me know!

[brandenchan]

Closing now since the Inferencer retains the order of the input. If you have any issues along this vein, please feel free to open up the issue again.