Keystone-classic: Admin UI e2e test status

[webteckie]

This issue covers the current status of the ui/ux/functional e2e tests coverage. Need lots of help here for anyone that's interested! We are using nightwatch to handle e2e testing. All Admin UI e2e tests are under test/e2e/adminUI These are broken into groups and group related tests within those groups. The groups have a sequence number in them to control the order of execution by nightwatch. To run the tests locally use npm run test-e2e. Make sure you have done a successful npm install for the keystone directory first! If you pick a group/test please mention it here to avoid duplication. Ideally we should focus both on positive and negative testing but at this point positive testing is more important to quickly detect regressions!

• [X] group001Login (ui) -- @webteckie
• [X] group001Login (ux) -- @webteckie
• [X] group002Home (ui) -- @webteckie
• [X] group002Home (ux) -- @webteckie
• [X] group003List (ui) -- @webteckie
• [X] group003List (ux) -- @webteckie
• [X] group004Item (ui) -- @webteckie
• [X] group004Item (ux) -- @webteckie
• [ ] group005Fields -- some fields may have different states and the tests should cover all states (e.g., a boolean field can be checked or unchecked)
  
  
  
  • [ ] azurefile (ui)
  
  
  • [ ] azurefile (ux)
  
  
  • [X] boolean (ui) -- @joerter
  
  
  • [X] boolean PO refactoring (ui) -- @jstockwin
  
  
  • [X] boolean (ux) -- @joerter
  
  
  • [X] boolean PO refactoring (ux) -- @jstockwin
  
  
  • [X] cloudinaryimage (ui) -- @jstockwin
  
  
  • [X] cloudinaryimage (ux) -- @jstockwin
  
  
  • [X] cloudinaryimages (ui) -- @jstockwin
  
  
  • [X] cloudinaryimages (ux) -- @jstockwin
  
  
  • [X] code (ui) -- @webteckie
  
  
  • [X] code (ux) -- @webteckie
  
  
  • [X] color (ui) -- @jstockwin
  
  
  • [X] color (ux) -- @jstockwin
  
  
  • [X] date (ui) -- @jstockwin
  
  
  • [X] date (ux) -- @jstockwin
  
  
  • [X] datearray (ui) -- @jstockwin
  
  
  • [X] datearray (ux) -- @jstockwin
  
  
  • [X] datetime (ui) -- @jstockwin
  
  
  • [X] datetime (ux) -- @jstockwin
  
  
  • [X] email (ui) -- @webteckie
  
  
  • [X] email PO refactoring (ui) -- @jstockwin
  
  
  • [X] email (ux) -- @webteckie
  
  
  • [X] email PO refactoring (ux) -- @jstockwin
  
  
  • [ ] embedly (ui)
  
  
  • [ ] embedly (ux)
  
  
  • [X] geopoint (ui)
  
  
  • [X] geopoint (ux)
  
  
  • [X] html (ui) -- @jstockwin
  
  
  • [X] html (ux) -- @jstockwin
  
  
  • [X] key (ui) -- @jstockwin
  
  
  • [X] key (ux) -- @jstockwin
  
  
  • [X] localfile (ui) -- @jstockwin
  
  
  • [X] localfile (ux) -- @jstockwin
  
  
  • [X] localfiles (ui) -- @jstockwin
  
  
  • [X] localfiles (ux) -- @jstockwin
  
  
  • [X] location (ui) -- @jstockwin
  
  
  • [X] location (ux) -- @jstockwin
  
  
  • [X] markdown (ui) -- @jstockwin
  
  
  • [X] markdown (ux) -- @jstockwin
  
  
  • [X] money (ui) -- @jstockwin
  
  
  • [X] money (ux) -- @jstockwin
  
  
  • [X] name (ui) -- @webteckie
  
  
  • [X] name (ux) -- @webteckie
  
  
  • [X] number (ui) -- @joerter
  
  
  • [X] number PO refactoring (ui) -- @jstockwin
  
  
  • [X] number (ux) -- @joerter
  
  
  • [X] number PO refactoring (ux) -- @jstockwin
  
  
  • [X] numberarray (ui) -- @jstockwin
  
  
  • [X] numberarray (ux) -- @jstockwin
  
  
  • [X] password (ui) -- @jstockwin
  
  
  • [X] password (ux) -- @jstockwin
  
  
  • [X] relationship (ui)
  
  
  • [X] relationship (ux)
  
  
  • [ ] s3file (ui)
  
  
  • [ ] s3file (ux)
  
  
  • [X] select (ui) -- @webteckie
  
  
  • [X] select (ux) -- @webteckie
  
  
  • [X] text (ui) -- @jstockwin
  
  
  • [X] text (ux) -- @jstockwin
  
  
  • [X] textarea (ui) -- @jstockwin
  
  
  • [X] textarea (ux) -- @jstockwin
  
  
  • [X] textarray (ui) -- @jstockwin
  
  
  • [X] textarray (ux) -- @jstockwin
  
  
  • [X] url (ui) -- @jstockwin
  
  
  • [X] url (ux) -- @jstockwin
  
  
• [ ] group006Relationships (ui)
• [ ] group006Relationships (ux)
• [ ] Cover Lists with noedit:true
• [ ] Cover Lists with nocreate:true
• [ ] Cover Lists with nodelete:true
• [ ] Cover Fields with noedit:true
• [ ] Cover list filtering
• [ ] Cover list columns
• [ ] Cover list download
• [ ] Cover list expand table width
• [ ] Cover negative testing behavior
• [ ] add config/run tests in Chrome browser
• [ ] add config/run tests in IE browser

[jstockwin]

@webteckie Mark me down for some more simple ones i.e. textarea and url for now.

Out of interest, and maybe for future reference, how many checks need to happen on the more complex types? For example, markdown should display with buttons to make text bold and add links etc. Do we check these are there, and working, and then maybe also somehow check the output html is correct, or do we simply just add some text in the field, without many extra checks.

[webteckie]

@jstockwin alright will do! You can add as much testing as you need. Currently we've been adding simple testing just to get the basic stuff covered. But we should really try to cover as much of it as possible.

[jstockwin]

@webteckie #2709 should be okay to merge now. I'll probably do datetime later, so put me down for a few more. Uhh html, markdown and password?

Presumably the verifyInput checks on password will be a bit different, as it shouldn't display the password in plaintext.

When we get on to the ones with uploads, e.g. cloudinary, s3 and local files, how does this work? Is there a way to upload a file, or do we just check the correct fields are there?

[mxstbr]

Amazing progress here by the way, This is one of the most important issues we have open currently, it'll massively improve our iteration speed.

Thanks for working on this @joerter @webteckie and @jstockwin, I can't wait to close this issue!

[webteckie]

@jstockwin you got the next fields assigned to you...thanks for the awesome work! With regards to the upload ones most likely we'll just do the ui tests until we figure a good way to do the ux tests.

[jstockwin]

For group 005 tests, boolean, email and number all need refactoring to the new pattern of using page objects, so perhaps need unticking or something to indicate this.

[jstockwin]

@webteckie All field tests have now been re-factored. At the time of writing this, all the check boxes are up to date.

I just tried safe-deleting adminUI.js, but it's still referenced from the relationships tests in group006, and also in the readme.

I can't see any progress being made for embedly, s3file or azureFiles because these all need config options to be set with valid access tokens/keys etc. Maybe they need to be striked out or something?

Will try and get round to geopoint, numberarray, and textarray in the next few days.

Regarding relationship field tests, since there is a group006 for this, should relationship be removed from group005?

[webteckie]

@jstockwin freel free to comment out the adminUI.js references in group006 since that group is TODO. Thank you for getting us this far...amazing!

[jstockwin]

@webteckie Just a quick one - currently the relationship field tests are for many: false relationships. We should perhaps also add many: true ones, unless you think it's best to just leave them for group006.

IMO there is quite a difference in behaviour between the two cases, so we should treat them as two fields. I'm not going to have time to do this any time soon, but thought I should point it out.

[webteckie]

@jstockwin if you are talking about testing more options in the field that should happen in the field tests. If we are talking about app scenarios then should probably be in group007. We should probably get rid of group006 and rename group007 to group006.

[jstockwin]

@webteckie - yeah, so I think I mean we need to add some more relationship stuff in group005. We need a new model to test many: true relationships, since we don't currently do this.

[webteckie]

@jstockwin yeah, that's the plan....feel free :-)

[webteckie]

btw, you don't really need a new model. You can just keep adding fields with the different options and expand the field PO, if needed.

[jstockwin]

We also need to extend the html/markdown tests to include e.g. wysiwyg options.