Ml-agents: Problem with JobTempAlloc has allocations that are more than 4 frames old

[SestoAle]

Hi,

when training my env at random time I got this warning and my training stops:

**Internal: JobTempAlloc has allocations that are more than 4 frames old - this is not allowed and likely a leak** (Filename: ./Runtime/Allocator/ThreadsafeLinearAllocator.cpp Line: 372) To Debug, enable the define: TLA_DEBUG_STACK_LEAK in ThreadsafeLinearAllocator.cpp. This will output the callstacks of the leaked allocations

It only appears in training mode and I don not know what causes it.

I implemented my game with hybrid ECS, but I disable all my parallel job system when the game starts, so in training mode there aren't any job active.

Anyone knows how can I resolve this?

[awjuliani]

Hi @SestoAle,

I have not seen this issue before. This sounds like an error with ECS, rather than the ML-Agents package. Can you share more information about your Unity version and OS platform?

[SestoAle]

I use Unity v2018.3.5f1 and the training phase is done in Linux (I built a Linux version of the game in MacOS), and I also think that is related to ECS, but it only appears in training mode so I don't know how to replicate it. Also, the only systems that use ParallelJob are disabled when the game starts, so there are no active jobs and I don't know why this error comes.

[awjuliani]

@VilmantasBal Have you seen this error before?

[jo3w4rd]

The error is from the JobSystem and is pretty much what the error says. The JobSystem allows you to allocate memory with different "longevity," the fastest, most efficient for temporary memory is Allocator.Temp -- which has this safety restriction. See https://docs.unity3d.com/Manual/JobSystemNativeContainer.html for more info.

I'm not sure at this point how hybrid ECS works with FixedUpdate. Possibly, the Time Scale/ Target Framerate changes ML-Agents does during training is fooling the Job System somehow.

[VilmantasBal]

I haven’t seen this error and ml agents dont use ECS, not even in hybrid
mode.

Can getting temporary render texture cause this error ?

Kind regards,
Vilmantas

On Friday, February 15, 2019, Joe Ward [e-mail] wrote:

The error is from the JobSystem and is pretty much what the error says.
The JobSystem allows you to allocate memory with different "longevity," the
fastest, most efficient for temporary memory is Allocator.Temp -- which has
this safety restriction. See https://docs.unity3d.com/Manual/
JobSystemNativeContainer.html for more info.

I'm not sure at this point how hybrid ECS works with FixedUpdate.
Possibly, the Time Scale/ Target Framerate changes ML-Agents does during
training is fooling the Job System somehow.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Unity-Technologies/ml-agents/issues/1710#issuecomment-464162703,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AqAp9m-fzoWxCY6mOV7p32sVEUa5N5rHks5vNwS2gaJpZM4a7rfj
.

[jo3w4rd]

@VilmantasBal The project that triggered this report is using ECS and ML-Agents together.

[SestoAle]

I see that if I start the training rendering the game, it goes obviously a little slower but the warning disappears. So is there a way to limit the speed of execution a bit to test if it’s really a speed problem?

[awjuliani]

Hi all,

Due to inactivity, I am closing this issue for now. Please feel free to re-open if you deem it necessary.

[jjxtra]

Seeing this with a blank project just LWRP, shader graph and post process. Latest Unity 2019.1.

[nukadelic]

What worked for me is to Install a different Unity version