Modernmt: How long does it take for new terms to register?

[LoekvanKooten]

How long does it take for new terms to register, i.e. the tensors to realign themselves where needed? Are newly added translations incorporated in the neural network immediately, or are they put in some kind of queue? In the latter case, how long does it take (doesn't need to be precise, just a very general estimate like minutes, quarters or hours) for a new translation to "register"?

I know that sometimes terms need to be added multiple times in multiple contexts for the vectors to actually realign enough to come up with the new translation in suggestions, so that is not my question. I'm merely talking about the actual processing of a newly added sentence.

Thank you in advance!

[davidecaroselli]

Hi @LoekvanKooten !

So, quite simple. Every contribution is added to an internal queue and eventually they are added to the model. How long does it take? A single contribution is almost immediate, while for a 1M source words TMX it should take 30/40 seconds (depending on the hardware of your machine).

Most importante, there is a method to know exactly when a contribution has been added: Get Import Job API. Every time you add a single contribution or an entire TMX, the result is an import job ID:

{
    "data": {
        "begin": 0,
        "dataChannel": 1,
        "memory": 2,
        "end": 0,
        "id": "80000001-0000-0002-0000-000000000000",
        "progress": 0.0,
        "size": 1
    },
    "status": 200
}

If you take the id 80000001-0000-0002-0000-000000000000, you can use it to retrieve the progress of the import job with the Get Import Job API. When the progress become 1 the contribution (or TMX) has been added to the model and the engine can use it for adaptation.

[EtienneAb3d]

As far as I understand, these sentences are added to the Translation Memory, thus certainly used in the on-the-fly re-training. Right ? Is there a time when the NMT model is itself updated in a long-time way ?

[LoekvanKooten]

Thank you for your explanation, @davidecaroselli ! Much appreciated.

[davidecaroselli]

@EtienneAb3d :

As far as I understand, these sentences are added to the Translation Memory, thus certainly used in the on-the-fly re-training

Exactly, and they are also added to the Context Analyzer for the context analysis.

Is there a time when the NMT model is itself updated in a long-time way ?

Not automatically. If you would like to re-train the whole engine including any new content, this has to be done manually.