Kaniko: Multistage build: Application directory is empty, no files copied from previous stage

[fhemberger]

I'm trying to build an image in GitLab CI using the latest debug tag of kaniko (should be commit 9f65174cb8391e01b4fbeda178d0e9b63dcabd75 and gcr.io/kaniko-project/executor@sha256: 0445151f95a2bffc69d629683290a945c7309ac96c2f4cbd487aae2e5b9021e9).

I'm modifying both the official Microsoft .Net SDK and Runtime image and store them in a private registry as base images.

Using a multistage build off these two images, the application directory is empty at the end of the process.

Custom .Net SDK image:

FROM microsoft/dotnet:2.1-sdk-alpine
WORKDIR /app
COPY ./Nuget.config /root/.nuget/NuGet/NuGet.Config

Custom .Net Runtime image:

FROM microsoft/dotnet:2.1-aspnetcore-runtime-alpine
WORKDIR /app

RUN apk update \
  && apk add ca-certificates \ 
  && rm -rf /var/cache/apk/*

COPY rootCA.crt /usr/local/share/ca-certificates/ca.crt
RUN update-ca-certificates

ENV USER dotnet
RUN addgroup -g 1000 -S $USER \
  && adduser -u 1000 -S $USER -G $USER

RUN chown -R $USER:$USER /app
USER $USER

EXPOSE 8080
EXPOSE 8443
EXPOSE 8099

ENV ASPNETCORE_URLS http://*:8080
ENV Startup__Monitoring__Port 8099

Multistage build file for application:

ARG CI_REGISTRY
FROM ${CI_REGISTRY}/aspnetcore-sdk:2.1-custom-alpine AS build-env

COPY . .
RUN dotnet publish -c Release -o /app/out

# build runtime image
FROM ${CI_REGISTRY}/aspnetcore-runtime:2.1-custom-alpine

EXPOSE 1500

COPY --from=build-env /app/out .

ENTRYPOINT ["dotnet", "MyService.dll"]

The build process is passing:

INFO[0000] Downloading base image <redacted>/aspnetcore-sdk:2.1-custom-alpine 
2019/02/05 17:13:05 No matching credentials were found, falling back on anonymous
INFO[0000] Unpacking rootfs as cmd RUN dotnet publish -c Release -o /app/out requires it. 
INFO[0013] Taking snapshot of full filesystem...        
INFO[0014] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0014] Skipping paths under /cache, as it is a whitelisted directory 
INFO[0014] Skipping paths under /builds, as it is a whitelisted directory 
INFO[0014] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0014] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0014] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0014] Skipping paths under /busybox, as it is a whitelisted directory 
INFO[0019] Using files from context: [/builds/MyService] 
INFO[0019] COPY ./ ./                                   
INFO[0019] Taking snapshot of files...                  
INFO[0020] RUN dotnet publish -c Release -o /app/out    
INFO[0020] cmd: /bin/sh                                 
INFO[0020] args: [-c dotnet publish -c Release -o /app/out] 
Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  <redacted>
  MyService -> /app/bin/Release/netcoreapp2.1/MyService.dll
  MyService -> /app/bin/Release/netcoreapp2.1/MyService.Views.dll
  MyService -> /app/out/
INFO[0034] Taking snapshot of full filesystem...        
INFO[0034] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0034] Skipping paths under /cache, as it is a whitelisted directory 
INFO[0034] Skipping paths under /builds, as it is a whitelisted directory 
INFO[0034] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0034] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0034] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0034] Skipping paths under /busybox, as it is a whitelisted directory 
INFO[0035] ignoring socket CoreFxPipe_root.0FRcgFW7zUBYLPE43n5AGU1e4, not adding to tar 
INFO[0037] ignoring socket CoreFxPipe_root.F.sBEGC1dqcKw5AyvRtvF8Vl0rg, not adding to tar 
INFO[0045] Storing source image from stage 0 at path /kaniko/stages/0 
INFO[0065] Deleting filesystem...                       
INFO[0067] Downloading base image <redacted>/aspnetcore-runtime:2.1-custom-alpine 
2019/02/05 17:14:13 No matching credentials were found, falling back on anonymous
INFO[0067] Taking snapshot of full filesystem...        
INFO[0068] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0068] Skipping paths under /cache, as it is a whitelisted directory 
INFO[0068] Skipping paths under /builds, as it is a whitelisted directory 
INFO[0068] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0068] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0068] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0068] Skipping paths under /busybox, as it is a whitelisted directory 
INFO[0068] EXPOSE 1500                                  
INFO[0068] cmd: EXPOSE                                  
INFO[0068] Adding exposed port: 1500/tcp                
INFO[0068] COPY --from=build-env /app/out .             
INFO[0068] Taking snapshot of full filesystem...        
INFO[0068] Skipping paths under /kaniko, as it is a whitelisted directory 
INFO[0068] Skipping paths under /cache, as it is a whitelisted directory 
INFO[0068] Skipping paths under /builds, as it is a whitelisted directory 
INFO[0068] Skipping paths under /sys, as it is a whitelisted directory 
INFO[0068] Skipping paths under /dev, as it is a whitelisted directory 
INFO[0068] Skipping paths under /proc, as it is a whitelisted directory 
INFO[0068] Skipping paths under /busybox, as it is a whitelisted directory 
INFO[0068] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0068] ENTRYPOINT ["dotnet", "MyService.dll"] 

However, the second last line makes me wonder:

INFO[0068] No files were changed, appending empty layer to config. No layer added to image. 

And indeed, if I run the resulting container, /app/ is empty.
Running the exact same Dockerfiles with docker (18.09.1) works for me.

[julian7]

Have you checked whether /app/out contains the files you're looking for when you build only the build-env image?

[jonathancole-sr]

We just encountered this issue in one of our builds. Typical builder/app multi-stage build based on node:alpine where only node_modules wasn't getting copied over into the final stage. We fixed the issue by ensuring we were setting a WORKDIR in the builder stage (previously nothing was set, leading to things being done in /, the default workdir) and building things in a subdir created in the Dockerfile during the builder stage.

[mashail]

@fhemberger try to remove out from .dockerignore it worked for me :)

[clhodapp]

I believe that this should be a P1, as it definitely meets the criteria, "Basic need feature compatibility with docker build. we should be working on this next." (this is a pretty serious problem with docker build compatibility for people using multi-stage builds).

[woz5999]

Thanks to the tip from @malmuzaini, I was able to fix my issue by updating my .dockerignore