Transformers: how we can replace/swap the Wikipedia data with our custom data for knowledge retrieval in the RAG model and the format of the retrieval data.

[Madhuri05thorat]

❓ Questions & Help

Details

A link to original question on the forum/Stack Overflow:

[iliemihai]

I am also playing with RAG model and I am trying to understand how to replace the Wikipedia data with my custom data. From transformes/model-cards/facebook/rag-sequence-nq I see that the train dataset is wiki_dpr. So I loaded it with the following
from datasets import load_dataset dataset = load_dataset("wiki_dpr")
The dataset is loaded with arrow into RAM(it's prety big, 75 GB). I was wandering, the custom dataset must have the same format as wiki_dpr ? If you can help with a tutorial on how to replace wiki dataset with a custom one, it will be very helpful. Thank you :D

[patrickvonplaten]

I think @lhoestq is working on this right now?

[lhoestq]

Yes indeed. I'll create the PR later today to allow users to use their own data. I'll also add code examples

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.