Docker-transmission-openvpn: Container keep crashing with fatal error

[shawn8888]

Synology, PIA, port forwarding enabled, latest version 3.01

After I updated to the latest version, the container just keep crashing every few minutes. It used to be very stable with the same configuration.
Can you please take a look at the log below?

waiting for rebind loop.................
token expiry 1609792612
remaining = 5417995
the port has been bound to 41345  Mon Nov  2 23:06:57 -05 2020
the port has been bound to 41345  Mon Nov  2 23:36:58 -05 2020
Mon Nov  2 23:48:07 2020 event_wait : Interrupted system call (code=4)
Mon Nov  2 23:48:07 2020 SIGUSR1[hard,] received, process restarting
Mon Nov  2 23:48:12 2020 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Mon Nov  2 23:48:12 2020 TCP/UDP: Preserving recently used remote address: [AF_INET]154.3.42.71:1198
Mon Nov  2 23:48:12 2020 UDP link local: (not bound)
Mon Nov  2 23:48:12 2020 UDP link remote: [AF_INET]154.3.42.71:1198
Mon Nov  2 23:48:12 2020 [toronto424] Peer Connection Initiated with [AF_INET]154.3.42.71:1198
Mon Nov  2 23:48:13 2020 AUTH: Received control message: AUTH_FAILED
RTNETLINK answers: No such process
Mon Nov  2 23:48:13 2020 ERROR: Linux route delete command failed: external program exited with error status: 2
RTNETLINK answers: No such process
Mon Nov  2 23:48:13 2020 ERROR: Linux route delete command failed: external program exited with error status: 2
Mon Nov  2 23:48:13 2020 /sbin/ip addr del dev tun0 10.7.112.123/24
Mon Nov  2 23:48:13 2020 /etc/openvpn/tunnelDown.sh tun0 1500 1622 10.7.112.123 255.255.255.0 init
Sending kill signal to transmission-daemon
Mon Nov  2 23:48:13 2020 SIGTERM[soft,auth-failure] received, process exiting

[11jwolfe2]

I am also getting something very similar and can not figure out how to resolve it.

Using OpenVPN provider: PIA
Starting OpenVPN using config Romania.ovpn
Setting OPENVPN credentials...
adding route to local network 10.142.0.0/24 via 10.42.4.1 dev eth0
Tue Nov  3 13:34:04 2020 OpenVPN 2.4.7 arm-unknown-linux-gnueabihf [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Feb 20 2019
Tue Nov  3 13:34:04 2020 library versions: OpenSSL 1.1.1d  10 Sep 2019, LZO 2.10
Tue Nov  3 13:34:04 2020 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Tue Nov  3 13:34:04 2020 TCP/UDP: Preserving recently used remote address: [AF_INET]188.240.220.26:1198
Tue Nov  3 13:34:04 2020 UDP link local: (not bound)
Tue Nov  3 13:34:04 2020 UDP link remote: [AF_INET]188.240.220.26:1198
Tue Nov  3 13:35:04 2020 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Tue Nov  3 13:35:04 2020 TLS Error: TLS handshake failed
Tue Nov  3 13:35:04 2020 SIGUSR1[soft,tls-error] received, process restarting
Tue Nov  3 13:35:09 2020 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Tue Nov  3 13:35:09 2020 TCP/UDP: Preserving recently used remote address: [AF_INET]185.210.218.103:1198
Tue Nov  3 13:35:09 2020 UDP link local: (not bound)
Tue Nov  3 13:35:09 2020 UDP link remote: [AF_INET]185.210.218.103:1198
Tue Nov  3 13:36:09 2020 TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Tue Nov  3 13:36:09 2020 TLS Error: TLS handshake failed
Tue Nov  3 13:36:09 2020 SIGUSR1[soft,tls-error] received, process restarting
Tue Nov  3 13:36:14 2020 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Tue Nov  3 13:36:14 2020 TCP/UDP: Preserving recently used remote address: [AF_INET]94.176.148.34:1198
Tue Nov  3 13:36:14 2020 UDP link local: (not bound)
Tue Nov  3 13:36:14 2020 UDP link remote: [AF_INET]94.176.148.34:1198

[haugene]

@11jwolfe2 You seem to be running an old version of the container. It is not using the new PIA servers and certificates. They are now shutting them down and you need to upgrade to the newest version of this image (currently 3.0.1). Ref: https://www.privateinternetaccess.com/blog/private-internet-access-legacy-vpn-network-sunset-announcement-30-september

@shawn8888 You are getting AUTH_FAILED. I'll copy part of my response in #1451:

About the AUTH_FAILED. I have not seen a case yet where this does not mean that your credentials are wrong.
On the old servers (that are due to shut down) there is a separate username/password pair that you can find in your control panel after logging in to PIA. It is listed under "PPTP/L2TP/SOCKS Username and Password" and username typically starts with an x. And for the nextgen servers they have changed this and are now using the same credentials for the ovpn connections as you use to sign in to the client control panel: https://www.privateinternetaccess.com/pages/client-sign-in
If you still have issues, it might be that the values you pass in are not persisted correctly and the wrong values are used by the container. In that case I want to know so that we can fix it. Have a look in /config/openvpn-credentials.txt inside the container and see if it matches what you set in the variables.

[11jwolfe2]

I am using latest-armhf as my tag to pull when running. Shouldn't this pull the latest 3.0.1?

Thanks in advance for the help.

[chrishilbert]

Still getting AUTH_FAILED after updating my username/password combo. Container keeps restarting over and over.

[dhitchcock]

I was having it restart also, using docker-compose pull ; docker-compose up -d claimed it recreated it with latest but it didn't work still. Then I brought the stack down docker-compose down and back up docker-compose up, and now it's working fine.

I don't get it, may be total fluke, but something you can try if you're using docker-compose

[chrishilbert]

I was having it restart also, using docker-compose pull ; docker-compose up -d claimed it recreated it with latest but it didn't work still. Then I brought the stack down docker-compose down and back up docker-compose up, and now it's working fine.

I don't get it, may be total fluke, but something you can try if you're using docker-compose

I'm using the docker run method.

[haugene]

@11jwolfe2 That was not my response to you. Again, I believe you're running an older version of the container. You need to pull the image again.

@chrishilbert Have you had a look in the file I'm referring to in my last comment? Does it look OK?

To know that you're running the actually latest build you can run this command:

docker image inspect haugene/transmission-openvpn --format '{{ .RepoDigests }}'

Or if you're running latest-armhf:

docker image inspect haugene/transmission-openvpn:latest-armhf --format '{{ .RepoDigests }}'

[chrishilbert]

@11jwolfe2 That was not my response to you. Again, I believe you're running an older version of the container. You need to pull the image again.

@chrishilbert Have you had a look in the file I'm referring to in my last comment? Does it look OK?

To know that you're running the actually latest build you can run this command:

docker image inspect haugene/transmission-openvpn --format '{{ .RepoDigests }}'

Or if you're running latest-armhf:

docker image inspect haugene/transmission-openvpn:latest-armhf --format '{{ .RepoDigests }}'

Made some progress. CA Toronto appears to be bad. I flipped to CA Montreal and AUTH_FAILED went away. Testing now.

I did a docker stop/rm/pull and start as my initial trial. Looks like I'm running current version.

[11jwolfe2]

Fixed my issue by changing latest-armhf to 3.0.1-armhf in my yaml file. Then rerunning the install. Up and running now! Thanks for the help. Any ideas why using latest-armhf was not working for some reason?

[chrishilbert]

I am now resolved too. Kristian, see if you have issues with Toronto...seems that is not good config.

[haugene]

@chrishilbert Tested Toronto here, don't see any issues :man_shrugging:

@11jwolfe2 Well, I believe the latest-armhf works :crossed_fingers: Can't know for sure but I have a feeling that you haven't pulled the image lately? Remember that latest-armhf today is not the same as latest yesterday, or a month or a year ago. If you run latest tag on your machine it will never go out and check if there is a newer latest than last time. I think a lot of people think that latest will always be the absolute latest but it isn't. It's the latest at the point you first ran it. Not to be presumptive but it's a very wide range of familiarity with Docker on these issues :sweat_smile:

[shawn8888]

Same here, I didn't change any password settings, but after I switched from CA Toronto to CA Montreal, the problem went away.

[shawn8888]

@ haugene if latest is not guaranteed to be the latest, how do I find out the version of my running container?

[haugene]

Yeah. It's an issue. You can do it by running the docker image inspect commands I posted above :point_up:

That will give the sha256 hash of the image as it's seen on the registry. This can be tracked back to the commit that produced it. But it's tricky. People could choose to lock into a version. Like 3.0, 3.0.1 or 3.1 when it comes. But then you won't get updates.

I think the best you can do is to just keep running :latest. And I'm going to add labels to the docker image so that it can be logged on startup what commit you're running and when it's from.

[shawn8888]

I just tried the command docker image inspect and it is NOT straightforward.
How about put the version# in the start up logs, so that we can check the terminal logs to find out the version.

[haugene]

Yes. As I said. I will.

[haugene]

Allright! So I just pushed this commit https://github.com/haugene/docker-transmission-openvpn/commit/cd9a5694cf313521e7414991b83a62b2c1254ec1 that adds the revision of the code as the first line that is printed when starting the container. That will hopefully ease the process going forward when errors are reported.

It sounds like you all now have the container running :tada: So I guess this issue can be closed?

[haugene]

Ohh. Need to add the build-arg to CircleCI that builds our images. Just a sec.

EDIT: Done :heavy_check_mark:

[mjw210]

Getting some fatal error issues, but it seems to be working when it initially starts up.
Transmission.txt

[ravensvoice]

@haugene "If you run latest tag on your machine it will never go out and check if there is a newer latest than last time. I think a lot of people think that latest will always be the absolute latest but it isn't. It's the latest at the point you first ran it. Not to be presumptive but it's a very wide range of familiarity with Docker on these issues

Just FYI I have watchtower running and it actually upgraded the latest this morning (the 4th at 4 am Pacfici Time) and when I swapped from Toronto to Montreal it fixed the issue.

[shawn8888]

Allright! So I just pushed this commit cd9a569 that adds the revision of the code as the first line that is printed when starting the container.

I just updated and I don't see the revision in the log. Is it because I am still running the old version?
The logs and the inspect result is attached.
TM.log.txt
inspect.log.txt

[haugene]

@shawn8888 based on the inspect log you're running the newest image. But the other log you posted is not the complete logs. The fatal error you're seeing is because the port updater script fails. It will probably take some time until it's stable. In the meantime you can use DISABLE_PORT_UPDATER=yes to bypass it.

[haugene]

@shawn8888 Can you try now using the 3.1 tag?

[shawn8888]

@shawn8888 Can you try now using the 3.1 tag?

I always use the latest tag. I downloaded the 3.1 tag image, but my container is on the latest image. I don't know how to switch to 3.1, keep the same setting without re-creating and re-setting the container.

Anyway, I updated using the "latest" tag and here are the logs. I still don't see the version info in the startup log. And the port forwarding fails after a couple of days running.

I added DISABLE_PORT_UPDATER=true to the environment. Will see how it goes.

---

update:
I added DISABLE_PORT_UPDATER to the first line of environment and port forwarding stopped working. I have to remove it.

tail.log
inspection.log

[haugene]

I see from the inspect log that you're on the latest version, so no need to run on 3.1.
Setting DISABLE_PORT_UPDATER will stop port forwarding from working, that is the idea. But if that would make the container run stable and not crash then it would be valuable information.

Can you try to reduce the number of env vars that you set? With the latest version almost all changes can be done in the UI and will stick between restarts.

    TRANSMISSION_HOME=/data/transmission-home \
    TRANSMISSION_RPC_PORT=9091 \
    TRANSMISSION_DOWNLOAD_DIR=/data/completed \
    TRANSMISSION_INCOMPLETE_DIR=/data/incomplete \
    TRANSMISSION_WATCH_DIR=/data/watch \

are the only ones that will be overridden by a default if you don't have it set in your environment.

[shawn8888]

I thought the new version display the version# on the startup logs, NO?
I don't need DISABLE_PORT_UPDATER because it is working after I switch to CA Montreal
I like to save all the settings in env vars, so I don't have to set it in UI every time I restart the container.

[ilike2burnthing]

With the latest version almost all changes can be done in the UI and will stick between restarts.

[shawn8888]

After switching to CA Montreal from CA Toronto, no more crash.
There are still some small issues such as port forwarding not stable, container version # doesn't display on startup logs, but OK to close this issue. Thanks, haugene!