Libertem: Easier transition between GUI and API scripting

[uellue]

Discussion with @sk1p : Using both GUI and scripting together works very well in practice: GUI for navigation and quick pre-analysis, scripting for reproducible results and more complex pipelines.

We should think how we make this transition easier.

Ideas, to be discussed:

• [x] In the GUI: Similar to a "Share" functionality, we offer with each analysis a code snippet that reproduces this specific analysis with a Python script -- cluster connection, dataset path and parameters, job definition etc. The user can just copy&paste this snippet in a Jupyter notebook.
• [x] Perhaps even open a new Jupyter notebook with a link?
• [ ] In the script: function to extract an identifier for a data set (GUI URL, cluster and path) that can be copied and pasted into a browser window to open this data set in the GUI.
• [ ] Perhaps allow elements of the GUI to run in a Jupyter cell?

Your feedback and ideas are highly appreciated!

[BisariaUtkarsh]

Hello

I am looking forward to contribute to this idea for GSOC 2020. I have installed Libertem and have been going through the documentation to make myself more clear about Libertem. I will be looking out for issues to solve for this week to help me with the codebase. So any help to get me started with this idea would be appreciated.

Thanks

[sk1p]

Hi @BisariaUtkarsh and welcome to LiberTEM! Note that another student, @twentyse7en, has already expressed their interest in this project in our gitter chat. It would be best to choose another idea. You are of course free to write a proposal for this project idea, too, but the likelihood it will be accepted is lower.

Please have a look at our ideas page again, or even our full issue list. If anything on there is of interest for you, we can discuss about making that your GSoC project.

To get started with LiberTEM, the best way is to pick one or more issues marked with "good first issue" and submit PR(s). Let us know if you have any more questions!

[BisariaUtkarsh]

Hello @sk1p I will go through the list again and get back to you if I find something of interest. Thanks anyways.

[kunal12298]

Hello, I am Kunal Gupta and am looking forward to contribute for this project.

[sk1p]

Hello, I am Kunal Gupta and am looking forward to contribute for this project.

Hi @kunal12298 - and welcome to LiberTEM! Please see the comment above - having more than one student work towards a proposal for this idea may reduce the likelihood your proposal will be accepted. I see you are also interested in #546 - I would suggest to concentrate on writing one excellent proposal, rather than putting the work into two or more proposals. This is only my personal recommendation, you can of course write as many proposals as you like.

[twentyse7en]

Python is mentioned as only skill required for this project. Does this also require react and typescript ?

[sk1p]

Python is mentioned as only skill required for this project. Does this also require react and typescript ?

Actually it will require to learn a tiny bit of TypeScript / react, too. In the easiest variant, this could be integrating some kind of "button" that the user clicks on, which triggers showing the Python code. It really depends on what you write into the proposal - for example, instead of perfecting the GUI part, you could put more focus on the Python part, by investigating and implementing the Jupyterhub / jupyter notebook / ... integration. There is quite a bit of freedom what you put into your proposal, as long as it aligns with the goals of LiberTEM and our user's requirements.

[twentyse7en]

Hey @uellue and @sk1p need help regarding some clarification of project. I have send a mail with a rough design. Design is in its intial state. Feedback would be very helpful for building the proposal 😊

[sk1p]

I have send a mail with a rough design

Thank you for reaching out for feedback, I'll have a detailed look on Monday!

[twentyse7en]

Thank you 😊

[twentyse7en]

I'm stuck at how GUI pass parameter to context.api ? Then I could use the same method to trigger another function which could save the parameters passed and functions used.

For example : If I'm using Ring analysis in GUI, I have to pass ri, ro to create_ring_analysis(). How parameters are passed into function ? which part of the code base should I have to look ?

[sk1p]

I'm stuck at how GUI pass parameter to context.api

A good way to check this is actually to use the development tools of the browser. In this case, especially the "network" tab. If you have a close look at what happens if you start an analysis, you will notice a PUT request to a URL like http://localhost:9000/api/jobs/dfb8bcd7-e4ab-40db-8f05-366c2c8e6afc/. Now, you can check the "URL configuration" for that pattern - you will notice that the URL above matches the pattern for JobDetailHandler. So the JobDetailHandler will be of most interest for you - if you carefully read that, you will find out how the web API works with regards to passing parameters. Also closely look at how the network request mentioned above is structured and how the JSON payload looks like.

[twentyse7en]

>

In the script: function to extract an identifier for a data set (GUI URL, cluster and path) that can be copied and pasted into a browser window to open this data set in the GUI.

Can you provide more details about this

[sk1p]

In the script: function to extract an identifier for a data set (GUI URL, cluster and path) that can be copied and pasted into a browser window to open this data set in the GUI.

Can you provide more details about this

The idea is to provide a way to quickly open a dataset you already have opened in your script with the LiberTEM GUI. This could work, for example, by encoding the dataset parameters in a URL and pointing a browser there. I see two challenges with this:

1) The script needs to know about the LiberTEM GUI/Web API, which it currently does not. This could be solved by some kind of service discovery like dbus on Linux (note this is only a possibility, not a concrete suggestion!), if we only want to support the use case of having the LiberTEM Web API running in the same desktop session as the script is executing in. In the case of a JupyterHUB environment, one could think about starting up a libertem service on-demand, which re-uses the jupyterhub authentication etc.
2) The GUI cannot connect to the same dask scheduler, if the scheduler was started as part of a local cluster. This is a more fundamental problem, as it depends on the IPC mechanism used by the dask cluster whether we can connect from the outside or not.

I think one has to closely think about different scenarios here, and how/if we can support them. Like:

• The executor can be in-process/"local" or external, and an external executor can be on the same computer or on a remote computer
• Script execution can be local or remote
• The "script" can execute in different environments (python or ipython repl, embedded into another app, jupyter notebook)

This can also be solved in a creative way, like having a libertem-server as part of Context, which you can "tell" about your datasets, or other crazy ideas.

[sk1p]

GUI-wise, the "transition to scripting" part could integrate with the new download dialog from #665 - it could show the download options, and in addition, something like "-or- continue working in a scripting environment" (wording to be improved, of course).

[r-ujjwal]

Hello Sir @uellue @sk1p,
I am Ujjwal Richhariya from India,
I have introduced myself to LiberTEM and its codebase.
I am willing and ready to contribute to this project.
I would be highly thankful if my proposal for this project would be accepted.

[sk1p]

@r-ujjwal Hi and welcome to LiberTEM! As a first starting point, I recommend thoroughly reading https://python-gsoc.org/#gettingstarted and https://google.github.io/gsocguides/student/ - they contain a lot of information on how GSoC works. Then, a good way to get started is to pick an issue marked "good first issue" and try to fix it - this will give you a first impression of the LiberTEM code base and our development process. As the deadline for proposal submissions is coming closer, I would advice to start to write the proposal ASAP - and ask any questions you have about the project!

Just as a heads up, note that there is already a student writing a proposal for this project idea. This is inevitable as more students apply, but I wanted to let you know.

I would be highly thankful if my proposal for this project would be accepted

Does that mean that you already sent a proposal? If so, let us know where you sent it and/or re-send it to both @uellue and me via email to the addresses listed on our GSoC page.

[r-ujjwal]

Thank you sir @sk1p for the useful suggestions

Does that mean that you already sent a proposal? If so, let us know where you sent it and/or re-send it to both @uellue and me via email to the addresses listed on our GSoC page.

I have prepared an outline for the proposal. I will send it to you as soon as it is completed.

[uellue]

Should we close here, or are the remaining two items still relevant? :-)

[sk1p]

Good call, I'm going to close this now. We can see about the remaining ideas if there is more concrete user demand. Thanks again to @twentyse7en !

[uellue]

Congrats @twentyse7en for closing such a long-standing issue! 💪

[twentyse7en]

Thx for your support @sk1p @uellue :)