Notebook: Convert JS tests to Selenium

[takluyver]

We have a number of Javascript tests:

• https://github.com/jupyter/notebook/tree/master/notebook/tests/base
• https://github.com/jupyter/notebook/tree/master/notebook/tests/notebook
• https://github.com/jupyter/notebook/tree/master/notebook/tests

We would like to convert these to use Selenium, in this folder. To run selenium tests, you will need the Selenium Python package and a 'driver' (a backend for it to talk to your browser). Once you have those, you can run:

py.test -v notebook/tests/selenium

This is quite a big task, and it probably won't be done in one pull request, but rather several smaller ones converting individual bits of the tests.

I'm marking this 'good first issue', though it's a bit borderline. It requires a decent understanding of Javascript and Python, but you don't need to know much about Jupyter to translate the tests. It should get easier as more of the tests are translated, because there will be more examples to refer to.

[parkerduckworth]

I would like to take this on. Getting started on it now

[parkerduckworth]

Hmm. It appears that the Selenium IDE is no longer being supported in Firefox:
https://addons.mozilla.org/en-US/firefox/addon/selenium-ide/

There is a chrome extension, but it is still under development and has a lot of issues. Have we considered Endtest? Are there any other testing frameworks we would consider using?

[takluyver]

The Selenium IDE isn't important to us - I haven't used it at all. We're using Selenium Webdriver, and the Firefox backend is still supported.

[parkerduckworth]

Ah, ok! Can I be assigned this issue?

[varshneydevansh]

I want to contribute to this good first issue.
I am proficient in Python, C++ and Good at JS but need help to get started.

[takluyver]

Github only lets us assign issues to people with push access to the repo, so just commenting that you plan to work on it is sufficient - especially since this is one a large task which multiple people can work on.

You can see the existing JS tests, for example this file. By comparison, here's the initial Selenium test. We want to add a Selenium (Python) version of each JS test, check that it passes, and then delete the JS test.

[takluyver]

PhantomJS has suspended development, providing some extra impetus to make this switch.

[shubhsherl]

I see this is a good first issue.
Can you guide me how can I begin with this?

[Sheshtawy]

If no one is working in this issue I'd be happy to help

[takluyver]

This is an issue that can be split up into small pieces, so multiple people can work on it at once. However, as I describe in the first message, it's not the easiest issue. You need to look at the Javascript tests (see the links in the first message), pick a test and understand what it's doing. Then look at the Selenium tests (also linked in the first message), and understand how they work. Then translate the Javascript test you chose to Selenium.

This should get easier as we convert more of the tests, because there will be more examples to look at. @mpacer and I are working on converting some of them, so if you're not sure what to do, I can hopefully point you to an example pull request soon.

[Sheshtawy]

From the example selenium test you posted in the description I got an initial good idea. I'm familiar with selenium and webdrivers and I'm willing to invest time understanding the phantomjs tests so I can start working on https://github.com/jupyter/notebook/tree/master/notebook/tests/notebook if that's okay. And once your pull request is ready I can use it as a guide to make sure my code is consistent with yours and also in case I got stuck or something.

But before I do that I have a question, you would want all tests from the three directories you linked above in the same tests/selenium directory? or you want them separated as they are now? so that the selenium directory is something like:

selenium/base/
selenium/notebook/

Let me know if it's okay to start tackling the notebook directory and what structure do you prefer inside the new tests/selenium directory and I can start working on immediately.

[takluyver]

For now, let's keep the tests/selenium directory flat, placing files directly in there. We can easily rearrange it later if we need to divide them up, but I'm not convinced that the current way the JS tests are divided is that useful.

[Sheshtawy]

Okay great! I'll start tackling the test files in the tests/notebook then, okay?

[takluyver]

Sounds good. @mpacer was looking at converting markdown.js from that folder, so if you want to start with another file, that would be good.

We also hit one or two fiddly bits with opening a new notebook and switching between edit and command mode. @mpacer was writing a couple of utility functions to do those things.

[mpacer]

In order to support multiple tests in the same notebook I’m running into some issues with the utility functions as they are currently. I’m going to try to make something a bit more comprehensive for manipulating the notebook on the page. If you want to do any tests on the tree page though (not on notebooks themselves) I won’t be touching that outside of #3412 until I get these utilities working.

[danagilliann]

@takluyver Hi! Do you mind giving a brief overview as to what the example tests do? To me they look like they're doing something quite different, and I'm a bit confused

[anand0427]

I would like to contribute as well. Need some help getting started.

[takluyver]

You mean the test that's already converted to Selenium?

The changes in #3412 make it easier to follow, so I'll describe this version of it

It gets authenticated_browser from a py.test fixture defined in conftest.py. That deals with starting a notebook server in a separate process, starting a Selenium driver, and navigating it to the files list of the notebook server. The Selenium driver is authenticated_browser inside the test code.

The first loop clicks down into directories in the list, until it reaches one with no subdirectories. At each level, it waits for something with the HTML class item_link to be visible on the page. Then it makes a list of such elements and stores it in a dictionary by the URL. If there's only one item, we assume that's .. (go up a level); otherwise we get the second item, click it, and assert that the new URL is what the link pointed to.

The second loop goes back up the levels, using browser history. Again, at each step it waits for something with class item_link, then builds a list of those items, and checks it against the list of items it found on the page on the way in. Then it calls authenticated_browser.back() to go back a step in history.

This may not seem like much of a test, but we recently made it navigate directories without reloading the page, using the history API in Javascript, and there were a few bugs in that.

[danagilliann]

Ah thank you for explaining. Just for clarification though, this Selenium test
and this JS test tests the same thing, right? If so which JS file/function is it testing?

[takluyver]

Nope, those are different. I removed the equivalent Javascript test when I added the Selenium, but you can still see it in history: this Javascript test was the equivalent of this Selenium test in master.

[anand0427]

So basically write python code that is equivalent to the JS file that you already have? Which one can i start with?

[takluyver]

Yup, that's about right. However, it's somewhat blocked at the moment because @mpacer is working on some utilities for the test framework to interact with the notebook UI, and there was only one test for the file list, which was already converted.

The tests in the base and security directories mostly don't seem to use the DOM, so these could be converted to use a regular JS test framework like Mocha, and be run in Node without needing to control a browser. That should make them faster, and if there are failures they will probably be easier to debug.

Alternatively, you could write new Selenium tests for parts of the UI other than the notebook editor - e.g. the 'running' tab in the dashboard, or authentication (visit .../logout, check that we can't get back to .../tree, try an invalid token, then the correct one).

[takluyver]

I had a go at creating a mocha test loading base/js/utils. Running this in node wasn't entirely successful. It looks like too much of the code assumes it's in a browser to easily test in node. Mocha can run in the browser, which might be an option.

[Sheshtawy]

@mpacer I'm trying to convert deletecell.js. I'll try to make it self-included until the utilities module is ready and I can use it to update the tests. I'm sorry if I seem to overcommunicate stuff, I'm just to trying to keep you guys updated and kind of get feedback if I'm working in the right direction and following the right process. Let me know if need to do something else or even completely wait until a further notice from you.

[Sheshtawy]

Also, I have a question. In order to use selenium to test notebook, we're gonna create python wrappers (using webdriver API) to access notebook APIs like get_cell(), insert_cell_at_bottom() etc.? Or is already a Python API for that?

[takluyver]

The aim with Selenium is to test the user interface, so where practical the tests and utilities are being written in terms of user interactions (click this thing, press these keys) rather than Javascript APIs.

I've just done another review pass on @mpacer's test utilities PR #3458 - I think it's nearly ready to merge.

[Sheshtawy]

So, should I wait for this PR to merge in order to use it in mine?

[takluyver]

I think that's easiest - sorry for the holdup.

[Sheshtawy]

Not at all. Thanks for being responsive and sorry if I ask too many questions

[takluyver]

Don't worry about asking questions - that how we find stuff out :-)

[takluyver]

I've just merged #3458 - that should make it easier for people to get started with testing the notebook editing interface using Selenium.

[takluyver]

And I've opened #3475 as an example of converting one JS test to one Selenium test.

[Sheshtawy]

Do you have any idea when this pull request will be merged since I checked the code and I believe I will need the changes you made in order to finialise my PR? For now I just copies the changes I need to make tests pass

[takluyver]

In theory I wait for someone else to review it. In practice, everyone else may well be too busy, and I'll merge it myself in a few days. It's fine to copy the bits and pieces you need for now. :-)

[lmcdon14]

I would like to try my hand at converting interrupt.js if nobody else is already working on it.

[takluyver]

Go for it :-)

[lmcdon14]

I just created https://github.com/jupyter/notebook/pull/3518 . I ran into an issue accessing the interrupt message once cell execution was interrupted, so I had to create a slight work-around for this test. Please check it out and let me know what you think.

[arovit]

@takluyver I just created PR 3554 for converting a small JS test to selenium. Please check it out and let me know. I chose this small JS test to familiarize myself with the code. I have some idea now, will work on other tests as well.

[arovit]

@takluyver did some work on this. Filed PR 3601
Also, just to get some more context, can you help me understand why are we trying to move to selenium from JS ?

[emstaple]

Hi, wondering which files have not been converted yet as I would like to contribute to this task. Is it just any js files still left in the current directories that aren't in the selenium directory? Or is there a list of the files or commits which people have converted already? Thanks

[arovit]

@emstaple Yes, its the js files which are left under tests/notebook. Generally I have seen that people (atleast me) have converted and removed from tests/notebook in same commit (but I might be wrong)

[takluyver]

Yep, I've been trying to ensure that people remove the js files in the PR where they've done the conversion. If there are any js files remaining for tests that already have a Selenium version, feel free to make a PR that just deletes the JS version.

[pranaynagpure321]

hey @takluyver I am trying a conversion of the save.js file to python but I am getting this error
ModuleNotFoundError:IPython.notebook.set_notebook_name(nbname);
how can I set the notebook name, and I am a newbie here if you could provide me already converted files that would be great for me.

[Sheshtawy]

Here are the files that have been converted so far https://github.com/jupyter/notebook/tree/master/notebook/tests/selenium

Also if you'd like an example PR this one is great

[takluyver]

Thanks @Sheshtawy !

[pranaynagpure321]

thanks @Sheshtawy

[Sheshtawy]

If anyone is interested in this ticket, I could use some help with a related PR https://github.com/jupyter/notebook/pull/3710 .. I'm stuck with a selenium issue

[emiliotl]

Hello 👋, I would like to keep converting tests, I just have some questions about this issue:

• If I run all Selenium tests, the test_dualmode_insertcell.py is failing, could someone please double-check to confirm that? if that's the case, I can try to solve it and send the fix on my next PR.
• Just for curiosity, wouldn't be better if we have the driver set as headless? in order to run the tests faster and without opening a Firefox window each time we run some of them. And in that way, we could use the current way to debug only.
• Would you prefer to get PR for a single test? or wouldn't be a problem if someone sends one PR of 2 or more converted tests?

Thanks for letting me contribute : )

[takluyver]

• It seems to be passing on Travis, but if it's failing for you, feel free to investigate.
• You can set an environment variable MOZ_HEADLESS=1 to make Firefox headless for the tests. I like this because having to explicitly hide the window makes it obvious what you have to do to see it again for debugging, and being able to see it is really valuable when you're writing/changing tests.
• I'd suggest doing one test file per PR to keep it manageable for review, but it's not a strict rule.

[emiliotl]

Alright, I'll take a look on the issued test and take another one to convert. Thank you for the answer and clarification on the use of headless mode.

[vallme2003]

Hi, i am interested in contributing. What can i work on?

[AmanPal-Singh]

I am also interested in contributing, what tasks are recommended to work on?

[takluyver]

Hi all - so long as there are JS test files left (there are - see the links in the initial issue description), there are things to do. Briefly:

• Pick a JS test
• Write an equivalent test in Selenium
• Check that the Selenium test runs and passes
• Remove the JS test
• Make a pull request

There are several which have been done and merged already which can serve as examples - see the "this was referenced" links in this issue.

I marked this as 'good first issue' because it doesn't require deep understanding of Jupyter, and it's fairly unlikely to clash with other work people might be doing. But it's not easy: you'll need to set up Selenium, understand the intent behind some Javascript code, and figure out how to check the same thing from Selenium. Maybe 'good second issue' would be more accurate, but we don't have a tag like that.

[dSchurch]

Hello. I'm interested in convert some JS to selenium. Can I start working on the move_multiselection.js test ?

[jimakr]

Hello. i would like to contribute by converting Javascript tests to Selenium. I am going to start with safe_append_output.js

[nfelger]

Hi, I've converted display_id.js (#5394) and notifications.js (#5455).

[06fahad]

Hi, is there a particular reason why none of the JS tests inside the tests/base folder have been converted? If not, I'd like to start working on converting security.js to selenium.