Teleport: Error about prefix of etcd key

[masa213f]

What happened:

When using etcd backend, the following 2 issues occurred.

1. Even if I specified the prefix of etcd key in teleport.yaml, the default prefix (/teleport) has been used.
   https://github.com/gravitational/teleport/blob/v4.0.2/lib/backend/etcdbk/etcd.go#L124
   https://github.com/gravitational/teleport/blob/v4.0.2/lib/backend/etcdbk/etcd.go#L302

2. When I omitted the prefix of etcd key in teleport.yaml, it causes the following error.

ERROR REPORT:
Original Error: *trace.BadParameterError etcd: missing "prefix" setting

What you expected to happen:

When the prefix is specified in teleport.yaml, the specified prefix will be used.
If the prefix is omitted, the default prefix is used.

Environment:

• Teleport version (use teleport version): 4.0.2
• Tsh version (use tsh version): 4.0.2

[benarent]

Thanks for reporting @masa213f this looks like a bug. I've assigned a label to it and it'll go into the queue.

[awly]

This was supported until around Dec 2018.
The fix is easy, but the upgrade path is unclear.

If we _only_ fix the issue - an upgrade picking it up will switch to the prefix specified in the config and "forget" all data under the hardcoded prefix (/teleport).

We could move all data from the hardcoded prefix to the config prefix at startup (a one-time migration). This assumes that the user doesn't store anything else valuable under the old hardcoded /teleport prefix.
And there's HA - how to you upgrade multiple auth servers without downtime.

I propose the following:

• in 4.3
  
  
  
  • all writes go to both hardcoded and config prefixes
  
  
  • all reads come from the config prefix
  
  
  • at startup and periodically after (say every 30min) sync data from hardcoded prefix to config prefix
  
  
  • this means hardcoded prefix is source of truth for the sync
  
  
  • skip this if hardcoded prefix is empty
  
  
• in 4.4 (or 5.0)
  
  
  
  • all writes and reads go to config prefix
  
  
  • no periodic syncing
  
  

What can go wrong:

• a write going through 4.2 auth server isn't reflected by 4.3 auth servers for up to 30min (resync period) - temporary data inconsistency based on load balancer routing
  
  
  
  • this is less of a problem if auth servers get updated quickly
  
  
• out-of-band changes to hardcoded prefix will get mirrored into the config prefix
• after 4.4, the data under hardcoded prefix remains, unused

@russjones @benarent thoughts?

[awly]

Ping @russjones @benarent, WDYT about the above migration plan?

[benarent]

We did review the ticket and I think RJ had some more questions, but I think the proposed migration plan from 4.3 -> 4.4 ( 5.0 ) seems like a reasonable approach. It might be good to sync with @travelton and try to find which our current customers use etcd and pass the idea passed them in various support channels.

[awly]

Talked to @russjones about the plan.
We'll change one thing about it: no periodic resyncs in 4.3.
The user is expected to downsize to 1 auth server before the upgrade, upgrade that auth server and scale back up.
This upgraded auth server will perform a one-time blocking sync at startup and start using the correct prefix afterwards (with double-writing to the old prefix).

[stmuraka]

@awly in 4.3, will Teleport write to both the prefix and /teleport? even for new deployments?
part of the main issue is that if multiple Teleport clusters want to share the etcd instance, the additional clusters will end up overwriting entries in /teleport. I like the notion of during an upgrade to 4.3, the keys are migrated to the "new" prefix and /teleport is deleted/no longer used.

[awly]

@stmuraka yes, 4.3 will write to both /teleport and the custom prefix from config file (if they are different). The problem is: if you decide to downgrade back to 4.2 for any reason, the data still needs to exist in /teleport prefix to avoid loss.

If you have a new 4.3 deployment, pointing multiple teleport clusters to a shared etcd cluster with different prefixes shouldn't be a problem. They will corrupt /teleport prefix data between each other, but none of them will _read_ from that prefix. So it should be fine, as long as _all_ teleport clusters use a non-default prefix.

In 4.4, /teleport will be ignored. I'm not super comfortable with deleting /teleport data automatically because it could be a valid prefix that some other cluster uses.

[stmuraka]

the delete comment was only for migrating to a user specified prefix. if the /teleport prefix was specified (or left empty and defaults to /teleport), you wouldn't delete it. for 4.4, why would you keep the /teleportprefix around if another prefix is specified in the config? It may lead to confusion why it's in etcd?

in 4.3, if we take periodic backups of etcd, do we need to replicate /teleport as well?
e.g. Would Teleport still come up if we take a backup without /teleport and in the case of etcd failure, we re-load the backup (without /teleport) prior to restarting Teleport?

[awly]

for 4.4, why would you keep the /teleport prefix around if another prefix is specified in the config?

Here's a problematic scenario:

• teleport cluster A using 4.3 and /cluster-a prefix
• teleport cluster B using 4.3 and no custom prefix (defaulting to /teleport) in _the same_ etcd
• teleport cluster A gets updated to 4.4
• on restart, cluster A notices that there's a /teleport prefix and deletes it
• cluster B lost all data

in 4.3, if we take periodic backups of etcd, do we need to replicate /teleport as well?

Good question! If /teleport is completely empty after backup restore and Teleport restart, it will be fine and Teleport will skip the sync.
If /teleport has some data after restore, things get trickier. We can probably compare the revisions of both prefixes to see which is newer and skip sync if /teleport is older.

[stmuraka]

if you will be supporting the user specified prefix in 4.3, there shouldn't be a need to sync/migrate (and delete) if migrating to 4.4. I think any migration support (if any) would be for 4.2 -> 4.3 (only)

in your scenario above, it would be:

1. teleport cluster A using 4.3 and /cluster-a prefix
2. teleport cluster B using 4.3 and no custom prefix (defaulting to /teleport) in the same etcd
3. teleport cluster A gets updated to 4.4
4. on restart, cluster A notices that there's a /teleport prefix and deletes it continues to use /cluster-a prefix
5. cluster B lost all data continues using /teleport

But in addition to above:

• If teleport Cluster C (or any other cluster for that matter) using 4.3+ and no custom prefix (defaulting to /teleport) in the same etcd, should fail to start if it finds that the prefix (/teleport) is already in use

[awly]

I think we had a small misunderstanding :)
I assumed that by for 4.4, why would you keep the /teleport prefix around if another prefix is specified in the config? you meant that 4.4 should actively clean up /teleport. Sounds like you actually meant it should ignore that prefix.
And I agree, 4.4 will not be aware of this whole migration, only 4.3 will. There won't be any automatic deletion.

If teleport Cluster C (or any other cluster for that matter) using 4.3+ and no custom prefix (defaulting to /teleport) in the same etcd, should fail to start if it finds that the prefix (/teleport) is already in use

This is an interesting idea. We could probably add something like this independently of this migration, thanks for the suggestion!

[russjones]

Fixed in https://github.com/gravitational/teleport/pull/3798.