Cli-microsoft365: Add support for persisting credentials

[waldekmastykarz]

Currently, connection information is persisted in memory while running the CLI. In order for the CLI to support non-interactive mode in the future and be used for building scripts, the connection information must be persisted so that it can be used across the different commands, where each command runs in a separate process.

Azure CLI has implementation for cross-platform storage of credentials (https://github.com/Azure/azure-xplat-cli/blob/bea341b53149246fb61b95883aa34b72a7e980f2/lib/util/authentication/adalAuth.js#L32). We could reuse that code to implement storing credentials in the O365 CLI as well. Once we have that, we could adapt the CLI to run in non-immersive mode which can be used for building scripts.

• [x] implement persisting connection state on macOS
• [x] implement persisting connection state on Windows
• [x] implement persisting connection state on Linux
• [x] implement restoring connecting state when starting the CLI (both interactive and non-interactive)
• [x] fix broken tests
• [x] prove that new code is working
• [x] check that telemetry is being submitted in non-interactive mode
• [x] check that telemetry is not being submitted when running unit tests
• [x] add documentation pages about how persisting connection state works on the different OSes

[waldekmastykarz]

I've done some research over the weekend what's possible with regard to persisting credentials/connection info. Here is the current solution I have in mind:

• in the OS password store (Keychain for macOS, Windows Creds Store or a plain-text file on the file system for Linux. We need this last option to support running the CLI in a Docker container), store the refresh token. Azure AAD refresh tokens are multi-resource so we only need to store one and use it for retrieving access tokens to all O365 services. Refresh token is stored and updated each time you run the o365 spo connect <url> command
• the first time you run the o365 spo connect command on your machine, and no refresh token is present in the OS secure store, the connection process will be interactive (you get device code, go to aka.ms/devicelogin and complete the authN). Once the refresh token is available and valid, the connection process will be non-interactive and will automatically retrieve an access token for the specified resource
• once the connection is established, serialize the connection information (eg. access token, site URL, tenant info) to JSON and store in a shell variable (similar to env variable but persisted only for the duration of the shell session)
• all commands can dehydrate the connection information from the shell variable
• it turns out that node.js cannot access shell variables (it can only access global variables but these have to be set outside of node.js). So plan B is to persist the connection information in OS password store after all

Storing the connection information in a shell variable is necessary to speed up running CLI commands in a script. Without it, for each command to be run, we'd have to:

1. access OS secure store to retrieve the refresh token
2. contact AAD to exchange refresh token for access token
3. run command

Alternatively, we could store all connection information in the OS secure store, but retrieving information from it, isn't very fast (~90ms to retrieve password from macOS Keychain) and might have a negative impact on running a script using CLI commands.

What do you think @andrewconnell @estruyf @VelinGeorgiev

[waldekmastykarz]

I've just finished working on persisting connection state in macOS Keychain. You can find the current implementation in my fork @ https://github.com/waldekmastykarz/office365-cli/commit/afcf6a4c63db19ef2c061c02ab581db81e2bb230. Next, I'll be working on an implementation for Windows and Linux.

[waldekmastykarz]

First version of persisting connection state on Windows is ready as well. Current implementation is available in my fork @ https://github.com/waldekmastykarz/office365-cli/commit/161a0d56deda44ddc4b6d850bbfd9726428d1f3d. Next, I'll be working on an implementation based on the file system for Linux/Docker containers. Once that's done, I'll clean it up add test and merge to dev so that we can have a beta version of the CLI for testing.

[VelinGeorgiev]

lol Waldek you are fast!

I agree with the above except the plain text for docker, but I have no experience with docker and containers so I am absolutely confident that since you do it, there isnt other option.

Is it an option the container image to come with passwords store or it is too much to ask?

[waldekmastykarz]

Thanks @VelinGeorgiev! To me it feels like it's taking way too long, but making progress nevertheless and slowly seeing the end of the tunnel. The hardest part was to make it fit the CLI and support both interactive and non-interactive mode, but once that was done when implementing support for macOS, implementing Windows support was easier (it has some quirks of itself, but luckily the Azure guys solved them in their x-plat CLI).

With regards to Docker, I'm currently thinking there isn't really a point in encrypting it, because the container file system is read-only. The moment the container is stopped, all changes outside of the folders mounted from the host are gone. The only potential issue might be, if someone got access to your stopped container and would access its file system. But even then I wonder, if they couldn't extract the password from the encrypted store. So if you start your container with the --rm flag (remove container after stopping), there should be no trace of your credentials anywhere.

[waldekmastykarz]

I've just finished adding support for persisting connection state on the file system on Linux. You can find the latest changes in my fork @ https://github.com/waldekmastykarz/office365-cli/commit/59ca5010e9373da001510848ecaccba4bffd71c1. I'm going to verify that it's working as it should in Docker and will then continue with cleaning up the code and adding tests.

[waldekmastykarz]

I'll leave this here: the CLI works in Docker like a charm! You can connect to SPO, connection is persisted across commands, and after you disconnect there is only an empty file left in the container which is gone after the container is closed.

[waldekmastykarz]

Just added the last unit test for the new code. See my fork @ https://github.com/waldekmastykarz/office365-cli/commit/fdae130e557e5f305a7211c1e9151caa6f8bf6a3. Need to check a few more things and write docs and will merge for releasing a beta.

[waldekmastykarz]

Just released this in a beta version of the CLI. You can test it out by grabbing @pnp/office365-cli@next 🚀