Containers-roadmap: Feature request: override mount points for each task

[dinvlad]

Hi Team,

Would it be possible to introduce an option to override volume mount path for each task? Right now, we have to register a new task definition just so we could change the mount path on a per-container basis. This introduces additional headache in how to keep track of various definitions, and is also rate limited by the task registration rate.

Our use case is batch processing of various datasets, where the container parameters don't change but the location of the dataset on the local host must be unique for each container/task. E.g. we would have all of our datasets under /datasets/<job_id> on the host, where job_id is unique for each task. One workaround is to pass an environmental variable JOB_ID and have each container read/write to its prefixed location, e.g. if /datasets is mounted to /data in all containers, then each container would gracefully access only /data/<JOB_ID>. However, this provides only minimal isolation of tasks in untrusted environments. I.e. if one task 'misbehaves', it can access everything under /data, including datasets used by other tasks.

It seems like Docker _run_'s -v option maps directly to the containerPath option of a task definition, so is there any reason it cannot be currently overridden for each task, just like environmental variables? We wouldn't even need to change the volume itself, which can remain an immutable part of the task definition.

Thanks

[jahkeup]

@dinvlad thanks for the suggestion and identifying a use case for it! We are tracking this as a feature request.

[tobinski]

Would be nice tho have this feature. I have a similar usecase and need to define separated taskdefinitions for each dataset

[GitShaffi]

@jahkeup We do have a similar usecase. Please let us know there is some progress in this feature request or if there's a workaround for this.

[eoinmurray]

Any update on this @jahkeup - I want to start jupyterlab data-science environments and have each container in its own directory on EFS, so isolating each container like this is really important.

I could register new tasks - but the rate limit of 1/s is too low, any thoughts?

[jsding]

Any progress on this case? we have similar use case.
We are creating directories in EFS for each task and mount these directories for each fargate container.