Spyder: Remote Python Development using Spyder

[abhijeetbadhe]

Hello,

I was wondering if it is possible to use the Spyder locally (on my mac) for development on remote Linux server with all the features (such as debugging, linting, variable-explorer etc.) available? All my project files would be on the server and the code would also run on the server...

From Spyder documentation and Carlos' posts on several forums, it's clear that it's possible to run a program developed locally on a remote machine (by launching spyder kernels, etc. ), but I have been unable to find whether or not remote development and execution is possible locally?

I would appreciate it if you could let me know...

Thanks & regards,

Abhijeet

[goanpeca]

Not at the moment but we do plan to add this capability for a future version.

[fohria]

I'm not sure I understand correctly what you want @abhijeetbadhe, but if you have SSH access to your server, you could mount the folder on your local machine using SSHFS: https://www.digitalocean.com/community/tutorials/how-to-use-sshfs-to-mount-remote-file-systems-over-ssh

I haven't done this with Spyder, as I mostly develop locally and sometimes connect to a remote kernel when I need some more computational power. But I've done it successfully with other editors in the past, and usually works like a charm!

[ccordoba12]

@fohria, we'd like to provide is a more integrated experience. So users would only need to provide their SSH credentials and then we'd create a remote kernel for them, show Files placed on the remote project dir and allow files to be edited in our editor.

[goanpeca]

Using something like this https://github.com/althonos/fs.sshfs @ccordoba12 ?

[ccordoba12]

Yep, seems like the right project to use.

[CAM-Gerlach]

@ccordoba12 Back when we discussed this at considerable length in the context of me writing a grant for Spyder 5 to "Army Labs" (by which I assume you meant either DARPA or ARL), with this as the headline feature, you found spurplus and recommended we use that, since it was a higher level solution than just SSHFS or paramiko alone. Might be worth digging that out if we want to implement something like this since we spent a lot of time discussing the high-level details of the implementation and how we would propose implementing it in stages.

By the way, after all the work we both put in on that grant, what happened to it? Did they have any feedback/suggestions for us about this remote development project? Did it even get submitted? Although I remember bugging you about it a lot, I don't recall if I ever got a clear answer; I know you were very busy at the time.

[ccordoba12]

you found spurplus and recommended we use that, since it was a higher level solution than just SSHFS or paramiko alone

Both projects seem similar. We'll have to evaluate which one is better maintained and more useful for us.

By the way, after all the work we both put in on that grant, what happened to it?

Sorry but support from TDK Micronas and Kite came in and it was much better than what we proposed to Army Labs. Besides, I couldn't make the contract with them through Quansight, which was a major issue for me. So I decided not to continue talking with them.

[CAM-Gerlach]

Both projects seem similar. We'll have to evaluate which one is better maintained and more useful for us.

Yeah, and I'm not sure fs.sshfs even existed back when we had that conversation for you to consider as an option.

So I decided not to continue talking with them.

Considering I put several straight days of my volunteer time into it at your request and you emphasized its urgency and importance to me at a time when I was very busy with other high-priority things, I would really have appreciated that at the time as a straight, honest answer to my many questions about it. But at the end of the day, that's almost a year in the past and we've both learned and grown a lot since then, so hopefully our communication has improved to a point where it won't be an issue in the future. I hope that grant can still be useful, and be submitted somewhere else at some point to get this funded.

[ccordoba12]

I would really have appreciated that at the time as a straight, honest answer to my many questions about it

Yeah, sorry about that. I was just in the middle of our initial talks with Kite and TDK people, so I forgot about it.

I hope that grant can still be useful, and be submitted somewhere else at some point to get this funded.

Well, basically all the ideas on how to improve our remote workflow capabilities came from it, so it was very useful just for that.

[fkloosterman]

Related, it may nice for Spyder to work with the Jupyter Enterprise Gateway to launch and connect to kernels remotely?

[ccordoba12]

@fkloosterman, very nice suggestion! We'll see what we can do about it.

[jsh9]

Just curious: RStudio Cloud is a browser-based RStudio -- the front end is elements (such as buttons, scroll bars, etc) that are rendered in the browser, and the back end is the R kernel on the server side.

Is it possible for Spyder to do something similar? I don't know enough about Spyder's front end to know the feasibility.

I know the scope of this issue is only limited to connecting to a remote back end, but I'm just wondering about this possibility for the future. :)

[ccordoba12]

No, it's not possible, sorry.

[CAM-Gerlach]

To expand a bit on @ccordoba12 's response, it is _possible_, but only in the sense that it would be a completely new frontend, sharing only the Spyder name (and perhaps some backend elements, but those are mostly lower in the stack anyway). It was possible for Rstudio because it was built on a web framework (just like VSCode is based on Electron), so it was relatively easy to port it to the web (but conversely, the Rstudio desktop interface does inherit some limitations as a consequence compared to a "true" desktop application.

While some of the panes (Help, Online Help, and the future Viewer pane) do leverage web-based rendering, Spyder as a whole is written entirely in pure Python, and the UI uses PyQt (a Qt binding for Python), powerful, full-featured desktop GUI application development library. This offers considerable advantages of performance, OS integration, and a native experience users expect, alongside being written in the same language it is primarily used to develop to make it easy for users to contribute features and write plugins, but isn't as readily portable to the web.

However, if funding or community support is available, there have been plans to considerably expand Spyder's remote development features, including potential integration with Jupyter. Perhaps @ccordoba12 can comment on the current status of that; we've been talking about it for years but money and developer time has always been a limiting factor compared to far more lavishly funded projects like Jupyter.

[jsh9]

Thanks for both your answers @ccordoba12 @CAM-Gerlach .

I didn't know that Jupyter is better funded than Spyder. If I had known, I probably would have gone there and ask them: "can Jupyter implement a good variable explorer and a proper debugger, just like what they do in Spyder?" I'm surprised that they still don't (have a variable explorer or a debugger), after this many years.

I would say the scope of this issue (i.e., kernel on the server side + front end on the client side) covers 95% of our use cases, and that's quite enough. (The only remaining 5% use cases where we desire a full browser-based IDE is that we somehow want to code on tablets.)

[CAM-Gerlach]

They get $100 000s or millions from large companies and foundations, I'm not entirely certain where all that money goes, vs. Spyder is developed and in large part crowdfunded by the international community of its users, although it has attracted a rapidly increasing amount of grant money and corporate donations in the past couple years, mostly to implement the most-requested features by users (the two top ones which were better code completion/introspection/analysis and a better debugger, which were the two biggest major enhancements delivered in Spyder 4).

I'm surprised that they still don't (have a variable explorer or a debugger), after this many years.

Yeah, I'm not sure, considering even VSCode is getting at least basic support for that sort of thing now. It does help that Spyder is more mature than e.g. JupyterLab and written in Python, attracting a ready pool of talent and avoiding some of the hurdles of web-based development.

[ccordoba12]

To expand a bit on @ccordoba12 's response, it is possible

I'd like to clarify that this is possible in theory, but we haven't considered to follow that direction at all. So, in practical terms, you're not going to see a purely web-based Spyder in the future.

However, Spyder can run on Binder. That's the closest you'll get to "have Spyder inside my browser" from us.