Teslamate: Unable to run teslamate in IPv6-only environment

[tohojo]

Describe the bug

I am trying to run teslamate in an IPv6-only environment, however, when connecting to the (remote) database, I get the following error:

/usr/src/teslamate/_build/prod/rel/teslamate/bin/teslamate eval "TeslaMate.Release.migrate"
21:11:05.430 [error] Postgrex.Protocol (#PID<0.238.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (db.example.org:5432): non-existing domain - :nxdomain

This is from a manual install, and with DATABASE_HOST=db.example.org set in the environment.

Expected behavior

I would expect the connection to succeed. I have verified that the connection works with the regular 'psql' database client.

How to reproduce it (as minimally and precisely as possible):

1. Install teslamate on an IPv6-only host
2. Try to connect to database

Relevant entries from the logs
As above:

/usr/src/teslamate/_build/prod/rel/teslamate/bin/teslamate eval "TeslaMate.Release.migrate"
21:11:05.430 [error] Postgrex.Protocol (#PID<0.238.0>) failed to connect: ** (DBConnection.ConnectionError) tcp connect (db.example.org:5432): non-existing domain - :nxdomain

Environment

• TeslaMate version: v1.20.0
• Type of installation: Manual
• OS TeslaMate is installed on: Debian Buster
• User OS & Browser: N/A

[adriankumpf]

@tohojo Can you try this PR and see if that fixes your issue: #1006? With DATABASE_IPV6=true IPv6 should be used to connect to Postgres.

[tohojo]

Thank you for the swift reply!

Yes, that fixes the issue and lets me run the initial migration + start the daemon (although I had to cherry-pick the commit, as compilation wasn't working when just switching to that branch).

However, when I try to login (with admin:admin), I get an "Error: unknown". I see a couple of 'inet_gethost 4' processes in the systemd daemon monitoring, but I'm not sure if those are related.

As an aside, an explicit config seems a bit like an invitation to playing whack-a-mole? :)
Can't the elixir runtime just do a regular getaddrinfo() and respect the address family preference of the OS?

[adriankumpf]

Are you saying that the login to Grafana fails? Can you create a datasource in Grafana that is able to connect to your postgres instance?

[tohojo]

Ah, no, Grafana works and can connect to the database, and I can import the dashboards. But obviously there's no data in them, and the teslafi login page itself (where I input my Tesla credentials) just gives me that "Error: unknown".

I also got this in the logs now:

[warn] Update check failed: %Mint.TransportError{reason: :enetunreach}

So I guess it has the same issue connecting to the Tesla API? And maybe it'll also have trouble with MQTT once it gets that far?

[adriankumpf]

Gotcha! Yeah, the update check fails because GitHub does not support IPv6. And the Tesla Onwer API also does not seem to support it.

[tohojo]

Ah, but they do! :)

The host is behind a DNS64/NAT64 gateway. E.g.,:

$  curl -v https://api.github.com/
*   Trying 2a00:cens:ored:6464::8c52:7905...
* TCP_NODELAY set
* Expire in 149974 ms for 3 (transfer 0x55852aff2920)
* Expire in 200 ms for 4 (transfer 0x55852aff2920)
* Connected to api.github.com ( 2a00:cens:ored:6464::8c52:7905) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=*.github.com
*  start date: Jun 22 00:00:00 2020 GMT
*  expire date: Aug 17 12:00:00 2022 GMT
*  subjectAltName: host "api.github.com" matched cert's "*.github.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
*  SSL certificate verify ok.
> GET / HTTP/1.1
> Host: api.github.com
> User-Agent: curl/7.64.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/1.1 200 OK
< date: Tue, 13 Oct 2020 16:36:13 GMT
< server: GitHub.com
< status: 304 Not Modified
< cache-control: public, max-age=60, s-maxage=60
< vary: Accept, Accept-Encoding, Accept, X-Requested-With, Accept-Encoding
< access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset
< access-control-allow-origin: *
< strict-transport-security: max-age=31536000; includeSubdomains; preload
< x-frame-options: deny
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
< content-security-policy: default-src 'none'
< content-type: application/json; charset=utf-8
< x-github-media-type: github.v3; format=json
< ETag: W/"27278c3efffccc4a7be1bf315653b901b14f2989b2c2600d7cc2e90a97ffbf60"
< X-Ratelimit-Limit: 60
< X-Ratelimit-Remaining: 58
< X-Ratelimit-Reset: 1602610515
< X-Ratelimit-Used: 2
< Accept-Ranges: bytes
< Content-Length: 2308
< X-GitHub-Request-Id: D650:624B:E3D0189:10B59D84:5F85D786
< 
{
  "current_user_url": "https://api.github.com/user",
  "current_user_authorizations_html_url": "https://github.com/settings/connections/applications{/client_id}",
  "authorizations_url": "https://api.github.com/authorizations",
  "code_search_url": "https://api.github.com/search/code?q={query}{&page,per_page,sort,order}",
  "commit_search_url": "https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}",
  "emails_url": "https://api.github.com/user/emails",
  "emojis_url": "https://api.github.com/emojis",
  "events_url": "https://api.github.com/events",
  "feeds_url": "https://api.github.com/feeds",
  "followers_url": "https://api.github.com/user/followers",
  "following_url": "https://api.github.com/user/following{/target}",
  "gists_url": "https://api.github.com/gists{/gist_id}",
  "hub_url": "https://api.github.com/hub",
  "issue_search_url": "https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}",
  "issues_url": "https://api.github.com/issues",
  "keys_url": "https://api.github.com/user/keys",
  "label_search_url": "https://api.github.com/search/labels?q={query}&repository_id={repository_id}{&page,per_page}",
  "notifications_url": "https://api.github.com/notifications",
  "organization_url": "https://api.github.com/orgs/{org}",
  "organization_repositories_url": "https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}",
  "organization_teams_url": "https://api.github.com/orgs/{org}/teams",
  "public_gists_url": "https://api.github.com/gists/public",
  "rate_limit_url": "https://api.github.com/rate_limit",
  "repository_url": "https://api.github.com/repos/{owner}/{repo}",
  "repository_search_url": "https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}",
  "current_user_repositories_url": "https://api.github.com/user/repos{?type,page,per_page,sort}",
  "starred_url": "https://api.github.com/user/starred{/owner}{/repo}",
  "starred_gists_url": "https://api.github.com/gists/starred",
  "user_url": "https://api.github.com/users/{user}",
  "user_organizations_url": "https://api.github.com/user/orgs",
  "user_repositories_url": "https://api.github.com/users/{user}/repos{?type,page,per_page,sort}",
  "user_search_url": "https://api.github.com/search/users?q={query}{&page,per_page,sort,order}"
}
* Connection #0 to host api.github.com left intact

So really this seems to be the same issue as with the database without the fix...

[adriankumpf]

Any success with the latest commit on #1006?

[tohojo]

Yes, much better, thanks! I can connect and get data from the car now; only thing that's not working as far as I can tell is MQTT publishing (MQTT_HOST is also set to an IPv6-only hostname).

[tohojo]

Managed to work around the MQTT issue by proxying it, but it would be good to have that just work over IPv6 as well. I'm also getting these in the logs:

Oct 13 20:06:17 teslamate teslamate[33668]: 22:06:17.646 [warn] Elevation query failed: %SRTM.Error{message: "HTTP request failed", reason: %Mint.TransportError{reason: :enetunreach}}
Oct 13 20:06:17 teslamate teslamate[33668]: 22:06:17.713 [warn] Elevation query failed: %SRTM.Error{message: "HTTP request failed", reason: %Mint.TransportError{reason: :enetunreach}}
Oct 13 20:06:17 teslamate teslamate[33668]: 22:06:17.730 [warn] Elevation query failed: %SRTM.Error{message: "HTTP request failed", reason: %Mint.TransportError{reason: :enetunreach}}

though I'm not sure if it actually fails permanently, or if it's just a transient error - there are just a lot of:

Oct 13 20:06:28 teslamate teslamate[33668]: 22:06:28.892 [info] Adding elevation to 1000 positions ...

afterwards.

[tohojo]

Also, not sure if this is related, or if it's normal, but I'm seeing a lot of connection messages in the log:

Oct 13 22:02:51 teslamate teslamate[33668]: 00:02:51.976 [info] Starting logger for 'My Car Name'
Oct 13 22:02:54 teslamate teslamate[33668]: 00:02:54.721 car_id=1 [info] Start / :online
Oct 13 22:02:54 teslamate teslamate[33668]: 00:02:54.728 car_id=1 [info] Connecting ...
Oct 13 22:02:55 teslamate teslamate[33668]: 00:02:55.646 car_id=1 [info] Start / :online
Oct 13 22:02:55 teslamate teslamate[33668]: 00:02:55.653 car_id=1 [info] Connecting ...
Oct 13 22:02:57 teslamate teslamate[33668]: 00:02:57.621 car_id=1 [info] Start / :online
Oct 13 22:02:57 teslamate teslamate[33668]: 00:02:57.628 car_id=1 [info] Connecting ...
Oct 13 22:03:00 teslamate teslamate[33668]: 00:03:00.668 car_id=1 [info] Start / :online
Oct 13 22:03:00 teslamate teslamate[33668]: 00:03:00.681 car_id=1 [info] Connecting ...
Oct 13 22:03:03 teslamate teslamate[33668]: 00:03:03.949 car_id=1 [info] Start / :online
Oct 13 22:03:03 teslamate teslamate[33668]: 00:03:03.956 car_id=1 [info] Connecting ...
Oct 13 22:03:04 teslamate teslamate[33668]: 00:03:04.584 car_id=1 [info] Start / :online
Oct 13 22:03:04 teslamate teslamate[33668]: 00:03:04.590 car_id=1 [info] Connecting ...
Oct 13 22:03:05 teslamate teslamate[33668]: 00:03:05.325 [info] Starting logger for 'My Car Name'
Oct 13 22:03:06 teslamate teslamate[33668]: 00:03:06.072 car_id=1 [info] Start / :online
Oct 13 22:03:06 teslamate teslamate[33668]: 00:03:06.078 car_id=1 [info] Connecting ...
Oct 13 22:03:07 teslamate teslamate[33668]: 00:03:07.090 car_id=1 [info] Start / :online
Oct 13 22:03:07 teslamate teslamate[33668]: 00:03:07.097 car_id=1 [info] Connecting ...
Oct 13 22:03:07 teslamate teslamate[33668]: 00:03:07.724 car_id=1 [info] Start / :online
Oct 13 22:03:07 teslamate teslamate[33668]: 00:03:07.734 car_id=1 [info] Connecting ...
Oct 13 22:03:10 teslamate teslamate[33668]: 00:03:10.595 car_id=1 [info] Start / :online
Oct 13 22:03:10 teslamate teslamate[33668]: 00:03:10.602 car_id=1 [info] Connecting ...
Oct 13 22:03:11 teslamate teslamate[33668]: 00:03:11.203 car_id=1 [info] Start / :online
Oct 13 22:03:11 teslamate teslamate[33668]: 00:03:11.210 car_id=1 [info] Connecting ...
Oct 13 22:03:14 teslamate teslamate[33668]: 00:03:14.690 car_id=1 [info] Start / :online
Oct 13 22:03:14 teslamate teslamate[33668]: 00:03:14.696 car_id=1 [info] Connecting ...

[adriankumpf]

Those are the attempts to connect to the WebSocket based streaming API. The library used seems to set up the socket for IPv4 by default and does not support passing socket options. The same applies to the library used to get elevation data. I suspect we will not get much further without modifying these libraries.

[tohojo]

Right, fair enough. The IPv6 support for the database connection is the most important thing to me (and MQTT if that is straight-forward to fix as well). I can arrange for v4 connectivity to the internet for the API connectivity... Indeed doing so fixes the constant reconnections :)

[adriankumpf]

IPv6 support for the MQTT connection should theoretically be straightforward, as the client library allows the passing of the required options. However, there is a bug in the passing of this specific option which would need to be fixed by the author of the library first. I will create a issue in his repo when I have time.

Edit: There is already a ticket for it: https://github.com/gausby/tortoise/issues/111

[tohojo]

Awesome! I'll keep the proxy in the meantime - thanks :)

[adriankumpf]

Can you try the master branch now? If MQTT_IPV6 is set to true it should connect to your MQTT broker via IPv6.

[tohojo]

Adrian Kumpf [e-mail] writes:

Can you try the master branch now? If MQTT_IPV6 is set to true it
should connect to your MQTT broker via IPv6.

Tried master and couldn't get it to compile:

# mix deps.get --only prod
** (CompileError) config/config.exs:23: undefined function config_env/0
    (stdlib 3.13.2) lists.erl:1358: :lists.mapfoldl/3
    (elixir 1.10.4) expanding macro: Kernel.to_string/1
    config/config.exs:23: (file)
    (elixir 1.10.4) expanding macro: Config.import_config/1
    config/config.exs:23: (file)

[adriankumpf]

Yep, master requires Elixir v1.11.

[tohojo]

Adrian Kumpf [e-mail] writes:

Yep, master requires Elixir v1.11.

Ah, right, gotcha! After upgrading Elixir I am now running the master
branch with native MQTT and database connections - thanks a bunch! :)