Docker-airflow: SSH Hook Problem - Failed to create remote temp file

[bdrjonny]

Hi Puckel,

i´ve got a Problem with Airflow SSH Hook. When i start my dag, i got an error, that Airflow "Failed to create remote temp file". I want to open a ssh-connection via Airflow. I set all settings in Airflow Connections and my DAG. Maybe you can help me ?!
When i run this DAG (sshjonas.py) outside of a docker-container on a local Airflow-Installation then everythink works...

BR,
Jonas

dag.txt

[puckel]

Hi,

Did you test a ssh connection inside container (outside airflow dag) ?

[sdikby]

Hi @puckel ,
The ssh connection was tested inside the container.
We have tried this method as we don't know how to make airflow docker container communicate with external hosts. For example to execute some hdfs or spark functions.

[expz]

This error occurred for me when the permissions for the private key file were not sufficiently strict.

It was solved by running a command like chmod 600 id_rsa inside the container.

For reference, the working configuration has an SSHHook created with

from airflow.contrib.hooks import SSHHook
ssh_hook = SSHHook(conn_id='ssh_conn')

where the connection ssh_conn has properties:

conn_id='ssh_conn'
conn_type='SSH'
host='172.17.0.1'
login='username'
port=2222
extra='{"key_file": "/usr/local/airflow/.ssh/id_rsa", "no_host_key_check": true}'

[expz]

(And it required creating a new docker container based on puckel/docker-airflow that installs the package openssh-client.)

[sdikby]

@expz thank you for your answer. But there is something that still unclear. do you generate the id_rsa key inside the container (worker?scheduler?webserver? or all?) or you generate it on the host machine and you define a volume in the docker-compose-CeleryExecutor file for each service to the id_rsa key that is found on the host machine ?

[expz]

Generate id_rsa on the host and mount the .ssh directory as a volume inside the container. This will guarantee persistence of the key. Otherwise the key will disappear if you destroy and remake the container.

[sdikby]

@expz docker is on a host with ip @ 1.2.3.4 and the host that i want my docker conatiner to coneect to is my hadoop cluster with ip @ 1.2.3.5.. when i use you suggested ip @ 172.17.0.1 it will connect to the docker0 but the docker it self is not connected somehow to the 2sn host. Did i miss something? how to make docker conect to the seconf host without install it there?

[sid88in]

@bdrjonny - did you resolve this? I am seeing the same error.

I am trying to use SSHExecuteOperator https://github.com/OpenGov/airflow/blob/master/airflow/contrib/operators/ssh_execute_operator.py#L74

and I create a task like :
self.arg_dev_task = SSHExecuteOperator(
task_id="arg_dev_task",
bash_command='sh /hostname/directory/test.sh',
ssh_hook=sshHook,
dag=dag)
but I get an error:
TemplateNotFound: sh /hostname/directory/test.sh
any idea whats happening? I am trying to execute a shell script on a remote host
(P.S: I have already created a SSH connection for the remote host in the airflow UI and I am passing that connection id in the ssh_hook)

when I pass bash_command = 'echo $hostname' then it gives me an error Failed to create remote temp file

[blpabhishek]

@sid88in @bdrjonny Did anyone found the solution.I have similar problem.
It says "Failed to create remote temp file"

[nathairtras]

may not be directly related, just wasted five minutes on this myself, so will restate for others. known_hosts

make sure not to miss the extra json that @expz listed for disabling known host check. without that ( or a valid entry in known_hosts ) airflow will fail with the remote file error - it is stuck on the known hosts prompt.

if checking the remote key is required, you will need to request/accept the key as part of provisioning or persist the known_hosts file with your SSH key.

this can sneak by manual testing, as you might test the connection, accept the prompt "confirming" that it works, retry in airflow _again_ confirming that it works, and then lose the known hosts file on next spin-up.

this is not specific to docker-airflow - it applies to any airflow environment that gets rebuilt with any frequency. for long-lasting environments, after the first success this is a non-issue, since known_hosts will never need to be regenerated.