Teleport: Upgrade to 4.0.x failed: "specify directory path to the database"

[mpitt]

What happened:

I tried upgrading my auth server from 3.2.6 to 4.0.15. The teleport service failed to start with:

error: specify directory path to the database using 'path' parameter, initialization failed

What you expected to happen:

I expected the teleport service to start correctly.

How to reproduce it (as minimally and precisely as possible):

Upgrade from Teleport 3.2.6 to 4.0.15 with a configuration similar to this:

teleport:
  nodename: foo.example.com
  pid_file: /var/run/teleport.pid
  auth_servers:
  connection_limits:
    max_connections: 1000
    max_users: 250
  log:
    output: stderr
    severity: WARN
  storage:
    type: bolt
    region: eu-west-1
    audit_events_uri:
      - file:///var/lib/teleport/log
      - dynamodb://teleport.foo.audit_events
    audit_sessions_uri: s3://internal-foo-teleport-logs-1/foo-main
  data_dir: /var/lib/teleport
auth_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3025
  oidc_connectors: []
  cluster_name: foo-main
  tokens:
    - "node:REDACTED"
ssh_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3022
  public_addr: localhost:3022
proxy_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3023
  web_listen_addr: 0.0.0.0:3080
  tunnel_listen_addr: 0.0.0.0:3024
  public_addr: foo.example.com
  https_key_file: /etc/ssl/foo/foo.key
  https_cert_file: /etc/ssl/foo/foo.crt

Environment:

• Teleport version (use teleport version): 4.0.15
• Tsh version (use tsh version): n/d
• OS (e.g. from /etc/os-release): Ubuntu 14.04.5 LTS

[mpitt]

It looks like adding memory: true under storage solves this. Why is this not documented? Why do we always have to guess the configuration? If I search for "memory" in https://gravitational.com/teleport/docs/admin-guide/ I get nothing.

[webvictim]

Assuming that you're just intending to use the built-in database, then you should remove type: bolt from the config.

This means Teleport will assume the default type (which is type: dir) and also the default path (which is path: /var/lib/teleport/backend)

Alternatively, if you want to actually be specific about this:

teleport:
  storage:
    type: dir
    path: /var/lib/teleport/backend

type: bolt hasn't been in the docs since Teleport 2.4. Specifying a storage type now also requires the use of path to describe where it should go.

I'll look to get this docs updated to make this more obvious.

[mpitt]

You're using type: bolt as well which doesn't appear anywhere in the documentation. Why?

Because we have been using Teleport since 2.x, bolt was deprecated and then removed (as far as I can see) in 4.1, we didn't want to recreate our users and we saw no clear path for upgrading the storage while retaining data.

you should remove type: bolt from the config

Will that keep our existing state or do we need to wipe it?

[webvictim]

Looking at https://github.com/gravitational/teleport/issues/1741, it doesn't appear that there is a supported automatic migration path from bolt to dir (which uses sqlite)

If you want to continue using the bolt backend, I would take a backup of your /var/lib/teleport directory then try this config:

teleport:
  storage:
    type: bolt
    path: /var/lib/teleport

If it works, good - but you should probably plan to migrate away from bolt as the project is no longer actively maintained and doesn't work with Teleport's graceful restarts. I don't know how long we will continue to support its use (EDIT: seems like we stopped supporting it in Teleport 4.1)

We added a feature in Teleport 4.1 to allow a cluster's config to be exported - see https://gravitational.com/teleport/docs/admin-guide/#backing-up-teleport - this may support the bolt backend as it uses internal Teleport API calls to get the data from whatever storage backend is being used.

You could theoretically upgrade to Teleport 4.1, export/backup your data as per the instructions above, then create a new cluster using the dir SQLite backend and bootstrap it with your exported data.

If this doesn't work, I would probably advise that you create a new cluster and migrate over to that instead.

[mpitt]

You could theoretically upgrade to Teleport 4.1, export/backup your data as per the instructions above

That's a nice plan, except 4.1 is also the first version that doesn't support bolt :)

If this doesn't work, I would probably advise that you create a new cluster and migrate over to that instead

That's what we are planning to do, yeah.

Thanks for your answers anyway!

[webvictim]

No problem. I didn't realise that anyone was still using bolt as a backend. Changing over is definitely the right course of action.

[benarent]

I'm going to close out this issue, as we won't plan on supporting bolt. As per webvictim recommendations we recommend building a new cluster and migrating.

[ap-wtioit]

just to document what i did (because i couldn't find a documentation for this anywhere else but this issue lead me to the solution, thanks @mpitt @webvictim )
migrating from 3.2.16 bolt (settings coming from 2.x) to 4.4 dir, and keeping the data:

• upgrade to 4.0.16
  
  
  
  • changing teleport.yaml storage
    
    from:
    
    teleport: storage: type: bolt
    
    to
    
    teleport: storage: type: bolt path: /var/lib/teleport
  
  
• upgrade to 4.1.11
  
  
  
  • changing teleport.yaml storage to
    
    teleport: storage: type: dir path: /var/lib/teleport
  
  
• upgrade to 4.2.12
• upgrade to 4.3.7
• upgrade to 4.4.1

[webvictim]

Thanks for the clear list of steps!