Pyrevit: Deploying private extensions (git) using CLI

[Dre-Tas]

We are currently trying to implement pyRevit at a global level in our company right now.
Our intention is to store the code for our extension on Bitbucket in a private (at least initially) repo and ideally what would happen is that pyRevit connects to that repo to grab the the code from there and keep everyone's tools up to date to the changes that happen on the repo.

From reading this issue here it seems to me this is possible, but I don't understand how to do it myself. Is there any specific documentation on that? Am I off track?

I think that I have to store the bitbucket credentials in the config ini file, but then where is the url of the repo stored? Is it in the extensions.json file? It looks odd that you have url and creds in two different places.

Thanks!

[DuncanLithgow]

Personally I think development should focus on projects that add public functionality to pyRevit rather than making it easier to run 'private' scripts. Linking/establishing a public script repos however would be great. That preserves the possibility for organisations to make their own custom scripts while making it easy for others to take inspiration from them.

[Dre-Tas]

Hi Duncan. While I'm big on open source (one of my projects for example) I think that it is fundamental to give flexibility for platforms like pyRevit.
In this particular case, Woods Bagot too is always a keen open source contributor, for a greater good in the industry. Having private repos does not mean keeping secrets and trying to advance in a selfish way. More often it is a matter of sensible data and not willing to release tools that are not ready to be released.
The reason why we we need to grab the extensions we're working on from a private repo is that so we can test it. We're still planning to release the tools that we believe are useful to the industry in the future.

[Dre-Tas]

UPDATE:
I actually made it to reproduce manually the process to grab the extension code from the private repo (as Gui was doing in the other issue). I'm not sure what I was doing wrong yesterday...

The the process though is quite manual because once I open revit I have to go to the pyRevit tab > pyRevit panel > _Extensions_ tool > click on the extension I want to get from bitbucket > click _Install Extension_
I would like to deploy these tools to multiple people in multiple offices and I would like to automate all this. I tried with the CLI command
pyrevit extend ui MyFirstExtension https://bitbucket.org/andreatas/test_pyrevit_priv.git --dest=C:\temp\MyExtensions
but I get a 401 error (Unauthorized), meaning I can't access the repo because I don't have the credentials.
Does it mean pyRevit is not reading the pyRevit_config.ini file from the CLI? I find it odd that if I do it manually it gets my credentials but if I do it from the CLI it doesn't.
Am I doing something wrong? How do I fix it?

Also, in case it helps, the same thing happens if I try to update the extension. If I click the _Update_ button under the pyRevit panel, my extension gets updated smoothly.
Although, if I do it from from the CLI with pyrevit extensions update MyFirstExtension I get another 401.

[eirannejad]

@Dre-Tas Yes the pyRevit CI at the moment lacks the functionality to authenticate to private repos. I'll add this very soon though and will keep you posted. Follow this issue please.

[Dre-Tas]

Thanks for letting me know mate. Makes sense now!

Is this something I can help developing? More than happy to contribute.

[eirannejad]

Of course. Any help is always appreciated.

We need to determine a way to pass credentials (--user: --pass: options or --apikey:) to the pyrevit clone command. The credentials need to be passed down to the CreateClone method and ends up going to the git wrapper module Clone method.

Once cloned, the credentials should be stored in the ini file so the pyrevit runtime can keep the clone updated.

[MY.extension]
disabled = false
private_repo = true
username = "username"
password = "password"

[htlcnn]

I think we should not save credentials in pyrevit.ini. Instead, username and password can be inserted when typing pyrevit clone. SSH key can be retrieve from user's .ssh folder like git bash.

[Dre-Tas]

@eirannejad shouldn't it work the same way it works from the pyRevit UI within Revit when you click Install Extension?
Shouldn't they actually call the same method?

Also, yeah I tend to agree with @htlcnn on where to store those credentials. User and pass shouldn't be in plain text, visible to the Revit user.
We can probably have a look at doing it with the ini file for now as a test and then, when it works, implement the more secure way?

[eirannejad]

@Dre-Tas @htlcnn
Completely agree. A bit of historical info: I implemented the pyRevit extension manager with git support back in 2016. The pyRevit CLI is much newer and was developed in 2018 and is based on the pyRevitLabs APIs that pyRevit runtime uses. I haven't gotten around to switch the python-based GUI extension manager to use the pyRevitLabs. This is an outstanding item that needs to be taken care of.

I also agree that storing credentials in plain text is not a good idea. This was also done to get the extension manager out the door.

Moving the python-based GUI extension manager to use the pyRevitLabs is quite a task however. To make it work for now, I'd suggest implementing the authentication into the CLI so the setup scripts can deploy the extensions from private repos. We can maybe salt the credentials and store in the config file, or automatically create ssh keys that allow the GUI extension manager to update the deployed extensions as well?

Lemme know if any part of this is not clear. I'm completely open to suggestions.

[Dre-Tas]

@eirannejad I had a deeper look at the code today and I have a better understanding now.

In regards to what you were saying here: isn't the CreateClone method meant for installing pyRevit itself?

From my understanding this issue is about when the user uses the pyrevit extend (ui | lib) <extension_name> <repo_url> [--dest=<dest_path>] [--branch=<branch_name>] [--log=<log_file>] command in the CLI. Right?
Hence I think it's the Extend method that should get the credentials and pass them down to InstallExtension that should at this point call the Clone method. Clone should probably be expanded by adding the CloneOptions as shown here in the LibGit2Sharp docs.

How to pass those credentials: probably the best thing would be
1- Check it the ini file has credentials already stored (i.e. the manager that is about to deploy is using his/her own settings) and, in case, use those;
2- if not, read the --user: --pass: options from the command line.

This is all of course regarding the temporary "for now" solution before we dig deeper into the .ssh stuff (which I'm quite ignorant about).

What do you think?

[Dre-Tas]

Hi @eirannejad !
I'm trying to implement these changes (described above) to to the CLI but I don't understand how to debug the pyRevit CLI. I followed the instruction here but it doesn't work, even if it seems it working. Kinda looks like it can't find all the dependencies.
How do you usually do that?

[eirannejad]

Just pass the --debug to the CLI and it will spit out the debug information

[Dre-Tas]

I actually mean how I debug the code I've written from visual studio?
I forked the project, implemented my code to see if I can clone a private repo through the CLI and now I want to see if the code works.
I thought I had to build following the instructions under the Dev folder but that didn't work...

[eirannejad]

Can you build from Visual Studio?

[Dre-Tas]

It built now! I promise it was't working before! I'm not crazy 😆
Could it be that it wasn't working because I didn't have some of the packages required in that bash script?

[eirannejad]

Implemented in 4.7.5