Jib: Make layering of extraDirectory configurable

[remmeier]

Files in the extra layer can be:

• quickly changing project resources
• rather static project resources
• third-party resources
• potentially have a substantial size

So it is thinkable to place extraDirectories inbetween any other layer for optimial layering. A configuration flag for this would be great. Or maybe even the possiblity to have multiple extra layers.

[coollog]

Hi @remmeier , thanks for the suggestion. We will definitely consider supporting multiple extra layers though the order will most likely not be configurable since each layer is pushed independently (and thus ordering does not matter). I think for multiple extra directories, we can support something like src/main/jib2, src/main/jib3, etc. as conventions for extra extra directories.

[coollog]

@GoogleContainerTools/java-tools Any comments on supporting extra extra directories (src/main/jib2)?

[TadCordle]

@coollog Would the same convention be used for custom extra directories?

e.g. extraDirectory = file('custom/extra/dir')
Layered structure is custom/extra/dir, custom/extra/dir2, custom/extra/dir3, etc.

[coollog]

Yep, that could be added too, so any number 2 and onwards (consecutively) appended to the configured extra directory path will be used as extra directory layers in the numbered order.

[chanseokoh]

I just thought about a (highly unlikely) edge case of extraDirectory = file('F:\'). Might need to be careful not to fail in that case.

And another edge case: what if src/main/jib doesn't exist but src/main/jib2 does? Or, src/main/jib and src/main/jib4 exist but no jib2 or jib3.

Might just be cleaner to get a list of extra directory paths in the config. However, these are edge cases that may work in one way or another anyway, so I also see some good things about the simple approach of jib, jib2, etc.

[coollog]

For F:, Jib would try to look for F:2, but that doesn't exist, so it stops.
For src/main/jib2 exists but src/main/jib doesn't, Jib looks for src/main/jib, doesn't find it, and then stops. Essentially, it should go in sequence.

[briandealwis]

This seems to me to be a case where we abandon convention for configuration. @remmeier's examples don't cleanly map to 2, 3, 4, ... and I would guess that some of those files would be things generated or best copies into target/ rather than into the source tree.

Perhaps now's the time to flesh out an <extras> / extras mapping:

<extras>
  <dir>src/main/jib</dir>
  <dir>target/downloads</dir>
</extras>

extras = ['src/main/jib', 'build/downloads']

[foal]

One more point. Now If I select the extraDirectory parameter selected folder is used instead of standard. It isn't very clever - the configured folder have to be used along with the standard folder.

Standard folder used for resources belong to the project. extraDirectory for the resources produced during a build.

[chanseokoh]

One more point. Now If I select the extraDirectory parameter selected folder is used instead of standard. It isn't very clever - the configured folder have to be used along with the standard folder.

I don't agree on this. For example, Maven has the convention that the default value for <sourceDirectory> is src/main/java. When you omit it, it's src/main/java. And when you configure it to something else, it uses the new directory as the source directory rather than adding it as an extra source directory . Jib parameters are working in the same manner.

[foal]

I see - you look on the extraDirectory parameter as on folder with extra data. But I understood the extra as one more directory with usual data - that is a point of confusing :)
Ok, agree - from your POV the implementation follows the Maven convention, but anyway it will better to have a possibility to configure several folders for the extra data.

[TadCordle]

@remmeier @foal version 1.2.0 has been released with the multiple extra directories feature! (jib.extraDirectories(.paths|.permissions) / <extraDirectories>(<paths><path>|<permisssions>)

[remmeier]

are there plans to make the position of extra layers configurable? we have use cases of third-party files in extra layers that are large in size and ideally placed below project resources.

[loosebazooka]

The order of layers shouldn't matter with the registry/image format we are using. No invalidation is happening, if a layer doesn't change, it is not rebuilt or repushed.

(vs docker which must handle arbitrarily executed steps)

[ninpuabi]

the documentation says

<extraDirectories>
<paths></paths>

which puts them at the root. Is there a way to put them somewhere specific?

[ninpuabi]

the documentation says:
<extraDirectories>
<paths></paths>
but does not give the option to where to put them.

[loosebazooka]

ah, so currently it's sort of attached to your local filesystem layout.

if you have a directory some/path/extra-dir/
with contents

 |_ potato.txt
 |_ some-dir
       |_ tomato.txt

and you do

<extraDirectories>
<paths>
  <path>some/path/extra-dir</path>
</paths>
</extraDirectories>

what goes into the root of the container is the contents of extra-dir, which means potato.txt and some-dir, some-dir/tomato.txt are all at /