Docker-airflow: Unable to install openJdk8 on the container

[ravieze]

Version: version: '2.1'
command to bring up the containers: docker-compose -f docker-compose-CeleryExecutor.yml up -d

Things are working fine till here. I am able to see the flower UI, DAG, run jobs etc. Sofar all good.

I want to use spark submit operator which requires me to have spark pakages installed -- which inturn need open jdk 8 to be present.

Now i tried all possible documents google can give me (eg: https://xmoexdev.com/wordpress/installing-openjdk-8-debian-jessie/) to install the open jdk 8 --but it was of no use. The installation always endsup in the below error. People report that they are able to install the openJDK when they use native Debian-8 (jessie); however, even though the docker container tells its Debian-8 Jessie, OPENJDk8 isnot installing.

The error i get during the installation is as follows. Any ideas how to fix this?

Setting up openjdk-8-jre-headless:amd64 (8u162-b12-1~bpo8+1) ...
update-alternatives: using /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/rmid to provide /usr/bin/rmid (rmid) in auto mode
update-alternatives: error: error creating symbolic link `/usr/share/man/man1/rmid.1.gz.dpkg-tmp': No such file or directory
dpkg: error processing package openjdk-8-jre-headless:amd64 (--configure):
subprocess installed post-installation script returned error exit status 2
dpkg: dependency problems prevent configuration of ca-certificates-java:
ca-certificates-java depends on openjdk-7-jre-headless | java7-runtime-headless; however:
Package openjdk-7-jre-headless is not installed.
Package java7-runtime-headless is not installed.
Package openjdk-8-jre-headless:amd64 which provides java7-runtime-headless is not configured yet.

dpkg: error processing package ca-certificates-java (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of openjdk-8-jre:amd64:
openjdk-8-jre:amd64 depends on openjdk-8-jre-headless (= 8u162-b12-1~bpo8+1); however:
Package openjdk-8-jre-headless:amd64 is not configured yet.

dpkg: error processing package openjdk-8-jre:amd64 (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of openjdk-8-jdk-headless:amd64:
openjdk-8-jdk-headless:amd64 depends on openjdk-8-jre-headless (= 8u162-b12-1~bpo8+1); however:
Package openjdk-8-jre-headless:amd64 is not configured yet.

dpkg: error processing package openjdk-8-jdk-headless:amd64 (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of openjdk-8-jdk:amd64:
openjdk-8-jdk:amd64 depends on openjdk-8-jre (= 8u162-b12-1~bpo8+1); however:
Package openjdk-8-jre:amd64 is not configured yet.
openjdk-8-jdk:amd64 depends on openjdk-8-jdk-headless (= 8u162-b12-1~bpo8+1); however:
Package openjdk-8-jdk-headless:amd64 is not configured yet.

dpkg: error processing package openjdk-8-jdk:amd64 (--configure):
dependency problems - leaving unconfigured
Setting up libgtk-3-common (3.14.5-1+deb8u1) ...
Setting up libgtk-3-0:amd64 (3.14.5-1+deb8u1) ...
Setting up libgtk-3-bin (3.14.5-1+deb8u1) ...
Setting up adwaita-icon-theme (3.14.0-2) ...
Processing triggers for systemd (230-7~bpo8+2) ...
Processing triggers for libc-bin (2.19-18+deb8u10) ...
Processing triggers for libgdk-pixbuf2.0-0:amd64 (2.31.1-2+deb8u7) ...
Processing triggers for dbus (1.8.22-0+deb8u1) ...
Processing triggers for sgml-base (1.26+nmu4) ...
Errors were encountered while processing:
openjdk-8-jre-headless:amd64
ca-certificates-java
openjdk-8-jre:amd64
openjdk-8-jdk-headless:amd64
openjdk-8-jdk:amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)

[Jonathan34]

may be that will help you. This is my docker image with Java 8 installed
I created my own docker image based on that one and put this inside:

FROM puckel/docker-airflow:1.9.0
COPY requirements.txt /

# install Java
USER root
RUN mkdir -p /usr/share/man/man1 && \
    (echo "deb http://http.debian.net/debian jessie-backports main" > /etc/apt/sources.list.d/backports.list) && \
    apt-get update -y && \
    apt-get install -t jessie-backports openjdk-8-jdk -y

RUN apt-get install unzip -y && \
    apt-get autoremove -y

USER airflow

[anthony-o]

Thanks to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199 you just have to execute mkdir -p /usr/share/man/man1 before apt install -y openjdk-8-jdk command in order to install Open JDK 8 on Debian Stretch.

[darshanmehta10]

@Jonathan34 docker image built with the above script does not start up. It doesn't log anything either.
Also, when I try with the new version of the base image (i.e. 1.10.1), I get the following error:

The following packages have unmet dependencies:
 openjdk-8-jdk : Depends: openjdk-8-jre (= 8u171-b11-1~bpo8+1) but it is not going to be installed
                 Depends: openjdk-8-jdk-headless (= 8u171-b11-1~bpo8+1) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
The command '/bin/sh -c mkdir -p /usr/share/man/man1 &&     (echo "deb http://http.debian.net/debian jessie-backports main" > /etc/apt/sources.list.d/backports.list) &&     apt-get update -y &&     apt-get install -t jessie-backports openjdk-8-jdk -y' returned a non-zero code: 100

[darshanmehta10]

@Jonathan34 @anthony-o following worked for me:

FROM puckel/docker-airflow:1.10.1

# install Java
USER root
RUN mkdir -p /usr/share/man/man1 && \
    apt-get update -y && \
    apt-get install -y openjdk-8-jdk

RUN apt-get install unzip -y && \
    apt-get autoremove -y

USER airflow

[Jonathan34]

@darshanmehta10 yes we updated to that couple of months ago. openjdk was not available when i posted the image above. thanks for correcting!

[andrew-womeldorf]

As of version 1.10.9, the base image has moved to python:3.7-slim-buster. Debian Buster appears to have removed openjdk-8 (see last message of https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199)

Here's a modified version of the aforementioned Dockerfile to install openjdk-8-jdk:

FROM puckel/docker-airflow:1.10.9

# install Java
USER root
RUN echo "deb http://security.debian.org/debian-security stretch/updates main" >> /etc/apt/sources.list                                                   
RUN mkdir -p /usr/share/man/man1 && \
    apt-get update -y && \
    apt-get install -y openjdk-8-jdk

RUN apt-get install unzip -y && \
    apt-get autoremove -y

USER airflow

There may be a better repository to add, but this one works.

[chencivalue]

As of version 1.10.9, the base image has moved to python:3.7-slim-buster. Debian Buster appears to have removed openjdk-8 (see last message of https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199)

Here's a modified version of the aforementioned Dockerfile to install openjdk-8-jdk:

FROM puckel/docker-airflow:1.10.9

# install Java
USER root
RUN echo "deb http://security.debian.org/debian-security stretch/updates main" >> /etc/apt/sources.list                                                   
RUN mkdir -p /usr/share/man/man1 && \
    apt-get update -y && \
    apt-get install -y openjdk-8-jdk

RUN apt-get install unzip -y && \
    apt-get autoremove -y

USER airflow

There may be a better repository to add, but this one works.

It works! thanks!

[kantharajucn]

As of version 1.10.9, the base image has moved to python:3.7-slim-buster. Debian Buster appears to have removed openjdk-8 (see last message of https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=863199)

Here's a modified version of the aforementioned Dockerfile to install openjdk-8-jdk:

FROM puckel/docker-airflow:1.10.9

# install Java
USER root
RUN echo "deb http://security.debian.org/debian-security stretch/updates main" >> /etc/apt/sources.list                                                   
RUN mkdir -p /usr/share/man/man1 && \
    apt-get update -y && \
    apt-get install -y openjdk-8-jdk

RUN apt-get install unzip -y && \
    apt-get autoremove -y

USER airflow

There may be a better repository to add, but this one works.

This works. Thanks:)