Solana: Automate user/pubkey addition to data center nodes

[danpaul000]

We need a way to quickly and without typos create users with passwordless sudo and pubkey based auth so that various engineering team members can SSH to the data center nodes.

Perhaps a source controlled script that one of us can run that adds this info to each server in kind.

Also should consider where to store the pubkeys of the team members such that a malicious actor cannot edit a pubkey and then gain access to the systems.

Would also be nice to have an analogous feature to remove access from all systems in the event someone no longer needs to use them or leaves the team.

[mvines]

Let's fold https://github.com/solana-labs/solana/blob/master/net/scripts/add-solana-user-authorized_keys.sh into whatever we do here (for testnet ssh access)

[TristanDebrunner]

@mvines what do you mean? Should that list be added to, or should those pubkeys be given access to something on the machines?

[mvines]

Use one list. I don't care if it's that list or some other list

[TristanDebrunner]

I was thinking each person who needed access to the DC nodes would get their own persistent username with associated pubkey for login. Then we just need scripts to create and delete those users on all the nodes. AFAICT we wouldn't need to maintain a list of pubkeys.

[mvines]

Yeah that's cool. But then I'd want to slurp up all those pubkeys and grant them all access to the testnet nodes too :)

It's the other way for the pubkey though, I don't get a pubkey. I give you a pubkey

[t-nelson]

Probably easiest to get both functionalities by making the script work off a users dir, each file inside named the DC node username and containing a list of keys.

Then you get something like..
add_user.sh

echo "pubkeys" > users/new_user
for h in $DC_HOSTS; do
  ssh "$h" -C "useradd [flags] new_user"
  scp users/new_user "$h:/home/new_user/.ssh/authorized_keys"
done

add_dc_host.sh

for u in users/*; do
  ssh new_host -C "useradd [flags] $u"
  scp users/$u "new_host:/home/$u/.ssh/authorized_keys"
done

update_testnet_users.sh

rm -f /tmp/testnet_authorized_keys
for u in users/*; do
  cat users/$u >> /tmp/testnet_authorized_keys
done
for h in $TESTNET_HOSTS; do
  scp /tmp/testnet_authorized_keys "$h:/home/solana/.ssh/authorized_keys"
done
rm -f /tmp/testnet_authorized_keys

revoke_user.sh

rm users/bad_user
./update_testnet_users.sh
for h in $DC_HOSTS; do
  ssh $h -C "userdel [flags] bad_user"
done

Except with error handling and stuff... :wink:

[TristanDebrunner]

To @danpaul000's point, I'm not sure that storing all the pubkeys in git is the best option for security. For testnets run on the DC machines, we can use the machines' user/pubkey lists, but for other testnets I'm not sure of a good way to provide that list with higher security.

[mvines]

Why? it's a public key. Everybody knows it, that's the point

[mvines]

Let's think about how these machines will be actually used. We seem to be thinking they'll be multi-user machines, but I don't think that's the case. We want them to run a node, either as a part of an existing testnet or a dev testnet. That's a single user. So maybe we treat them like instances a user can allocate, like a AWS or GCP, machine. Once allocated, that user must release them for another user to pick it up

[TristanDebrunner]

The concern is if someone gets one of their pubkeys into the list

[mvines]

How?

[TristanDebrunner]

We have also been using dumoulin for benchmarking, not sure if that's planned to continue.

[TristanDebrunner]

How?

That would give them access to our testnets and the DC machines. We have already had issues with malware being installed on one machine.

[mvines]

But that's also single user right? You allocate the machine when you want it for benchmarking, then deallocate it later with a rm -rf /home/solana in-between.

[mvines]

We have already had issues with malware being installed on one machine.

Well don't use u:test p:test to avoid that. That incident is unrelated really;)

[TristanDebrunner]

Well don't use u:test p:test to avoid that. That incident is unrelated really;)

But if they have access via pubkey (esp with sudo privileges) they can cause similar troubles

[mvines]

How? :)
Let's take this to #devops perhaps?