Polis: e2e tests fail on dev on mac / document e2e test expectations for new contributors
Expected behavior:
e2e tests pass
Actual behavior:
e2e tests fail
To Reproduce:
git clone pol-is/polis && cd polis
docker-compose up --detach
cd e2e
npm install
CYPRESS_BASE_URL=http://127.0.0.1 npm run e2e:standalone
Device information:
macOS Catalina 10.15.7
Now using node v8.17.0 (npm v6.13.4)
micahstubbs
All 37 comments
thanks @ballPointPenguin for the detailed steps to test!
at the npm install step, it tells me that my nodejs is too new:
npm WARN npm npm does not support Node.js v15.6.0
npm WARN npm You should probably upgrade to a newer version of node as we
npm WARN npm can't make any promises that npm will work with this version.
npm WARN npm Supported releases of Node.js are the latest release of 4, 6, 7, 8.
npm WARN npm You can find the latest version at https://nodejs.org/
npm ERR! cb.apply is not a function
npm ERR! A complete log of this run can be found in:
npm ERR! /Users/m/.npm/_logs/2021-01-27T09_17_49_442Z-debug.log`
It would be nice to specify somewhere in the e2e directory what version of nodejs is expected.
The root project's package.json does specify, so lets use that:
"engines": {
"node": "8.17.0"
},
I'm doing this locally with nvm:
nvm install 8.17.0
node -v
# v8.17.0
now let's try:
npm install && CYPRESS_BASE_URL=http://127.0.0.1 npm run e2e:standalone
when I run npm install
I see this error:
e2e git:(bower-to-webpack-alt) npm install
> [email protected] postinstall /Users/m/workspace/polis/e2e/node_modules/cypress
> node index.js --exec install
/Users/m/workspace/polis/e2e/node_modules/fs-extra/lib/mkdirs/make-dir.js:85
} catch {
^
SyntaxError: Unexpected token {
at createScript (vm.js:80:10)
at Object.runInThisContext (vm.js:139:10)
at Module._compile (module.js:617:28)
at Object.Module._extensions..js (module.js:664:10)
at Module.load (module.js:566:32)
at tryModuleLoad (module.js:506:12)
at Function.Module._load (module.js:498:3)
at Module.require (module.js:597:17)
at require (internal/module.js:11:18)
at Object.<anonymous> (/Users/m/workspace/polis/e2e/node_modules/fs-extra/lib/mkdirs/index.js:3:44)
at Module._compile (module.js:653:30)
at Object.Module._extensions..js (module.js:664:10)
at Module.load (module.js:566:32)
at tryModuleLoad (module.js:506:12)
at Function.Module._load (module.js:498:3)
at Module.require (module.js:597:17)
npm WARN [email protected] requires a peer of [email protected] - 6.x but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] No repository field.
npm WARN [email protected] No license field.
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] postinstall: `node index.js --exec install`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] postinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! /Users/m/.npm/_logs/2021-01-27T09_23_23_310Z-debug.log
Ok, so depcheck.yml uses nodejs v 14.4.0, so let's give that version a try:
nvm install 14.4.0
now let's try again:
npm install && CYPRESS_BASE_URL=http://127.0.0.1 npm run e2e:standalone
npm install worked with node v14.4.0.
The Cypress e2e tests ran. I saw test failures. It looks like the Cypress tests aren't able to talk to the backend running in the Docker container. I'm not sure if this is a problem with this PR or some user error on my part.
Access control
1) Cannot access /user/:id
cy:request ✔ POST /api/v3/auth/deregister
Status: 200
Response body: <EMPTY>
cy:request ✔ POST /api/v3/auth/login
Status: 200
Response body: {
"uid": 3,
"email": "[email protected]",
"token": "5BS7ni3g5zMxOAeM5L6v"
}
cy:command ✔ visit /user/7kvdfwfrmf
cy:xhr ⓘ GET http://127.0.0.1/user/_domainWhitelistError_api/v3/users?errIfNoAuth=true
cy:command ✘ get #no-permission-warning
cy:command ✘ assert expected **#no-permission-warning** to exist in the DOM
2) Cannot access /user/:id/share
cy:request ✔ POST /api/v3/auth/deregister
Status: 200
Response body: <EMPTY>
cy:request ✔ POST /api/v3/auth/login
Status: 200
Response body: {
"uid": 3,
"email": "[email protected]",
"token": "hQ0UhDuuo4sWG50YXP3I"
}
cy:command ✔ visit /user/7kvdfwfrmf/share
cy:xhr ⓘ GET http://127.0.0.1/user/7kvdfwfrmf/_domainWhitelistError_api/v3/users?errIfNoAuth=true
cy:command ✘ get #no-permission-warning
cy:command ✘ assert expected **#no-permission-warning** to exist in the DOM
3) Cannot access /user/:id/comments
cy:request ✔ POST /api/v3/auth/deregister
Status: 200
Response body: <EMPTY>
cy:request ✔ POST /api/v3/auth/login
Status: 200
Response body: {
"uid": 3,
"email": "[email protected]",
"token": "NUz96WUvfUdDBUHJHkRS"
}
cy:command ✔ visit /user/7kvdfwfrmf/comments
cy:xhr ⓘ GET http://127.0.0.1/user/7kvdfwfrmf/_domainWhitelistError_api/v3/users?errIfNoAuth=true
cy:command ✘ get #no-permission-warning
cy:command ✘ assert expected **#no-permission-warning** to exist in the DOM
4) Cannot access /user/:id/comments/rejected
cy:request ✔ POST /api/v3/auth/deregister
Status: 200
Response body: <EMPTY>
cy:request ✔ POST /api/v3/auth/login
Status: 200
Response body: {
"uid": 3,
"email": "[email protected]",
"token": "1yretoQBy3WDHHQF7Nue"
}
cy:command ✔ visit /user/7kvdfwfrmf/comments/rejected
cy:xhr ⓘ GET http://127.0.0.1/user/7kvdfwfrmf/comments/_domainWhitelistError_api/v3/users?errIfNoAuth=true
cy:command ✘ get #no-permission-warning
cy:command ✘ assert expected **#no-permission-warning** to exist in the DOM
5) Cannot access /user/:id/stats
cy:request ✔ POST /api/v3/auth/deregister
Status: 200
Response body: <EMPTY>
cy:request ✔ POST /api/v3/auth/login
Status: 200
Response body: {
"uid": 3,
"email": "[email protected]",
"token": "cAkpdX5PnERtdr7mrKLM"
}
cy:command ✔ visit /user/7kvdfwfrmf/stats
cy:xhr ⓘ GET http://127.0.0.1/user/7kvdfwfrmf/_domainWhitelistError_api/v3/users?errIfNoAuth=true
cy:command ✘ get #no-permission-warning
cy:command ✘ assert expected **#no-permission-warning** to exist in the DOM
6) Cannot access /user/:id/reports
cy:request ✔ POST /api/v3/auth/deregister
Status: 200
Response body: <EMPTY>
cy:request ✔ POST /api/v3/auth/login
Status: 200
Response body: {
"uid": 3,
"email": "[email protected]",
"token": "0stdoUnJRGmqZ43NyGlb"
}
cy:command ✔ visit /user/7kvdfwfrmf/reports
cy:xhr ⓘ GET http://127.0.0.1/user/7kvdfwfrmf/_domainWhitelistError_api/v3/users?errIfNoAuth=true
cy:command ✘ get #no-permission-warning
cy:command ✘ assert expected **#no-permission-warning** to exist in the DOM
0 passing (58s)
6 failing
1) Access control
Cannot access /user/:id:
AssertionError: Timed out retrying after 4000ms: Expected to find element: `#no-permission-warning`, but never found it.
at Context.eval (http://127.0.0.1/__cypress/tests?p=cypress/integration/polis/client-admin/convo_access.spec.js:112:38)
2) Access control
Cannot access /user/:id/share:
AssertionError: Timed out retrying after 4000ms: Expected to find element: `#no-permission-warning`, but never found it.
at Context.eval (http://127.0.0.1/__cypress/tests?p=cypress/integration/polis/client-admin/convo_access.spec.js:116:38)
3) Access control
Cannot access /user/:id/comments:
AssertionError: Timed out retrying after 4000ms: Expected to find element: `#no-permission-warning`, but never found it.
at Context.eval (http://127.0.0.1/__cypress/tests?p=cypress/integration/polis/client-admin/convo_access.spec.js:120:38)
4) Access control
Cannot access /user/:id/comments/rejected:
AssertionError: Timed out retrying after 4000ms: Expected to find element: `#no-permission-warning`, but never found it.
at Context.eval (http://127.0.0.1/__cypress/tests?p=cypress/integration/polis/client-admin/convo_access.spec.js:124:38)
5) Access control
Cannot access /user/:id/stats:
AssertionError: Timed out retrying after 4000ms: Expected to find element: `#no-permission-warning`, but never found it.
at Context.eval (http://127.0.0.1/__cypress/tests?p=cypress/integration/polis/client-admin/convo_access.spec.js:128:38)
6) Access control
Cannot access /user/:id/reports:
AssertionError: Timed out retrying after 4000ms: Expected to find element: `#no-permission-warning`, but never found it.
at Context.eval (http://127.0.0.1/__cypress/tests?p=cypress/integration/polis/client-admin/convo_access.spec.js:132:38)
(Results)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 6 │
│ Passing: 0 │
│ Failing: 6 │
│ Pending: 0 │
│ Skipped: 0 │
│ Screenshots: 6 │
│ Video: true │
│ Duration: 57 seconds │
│ Spec Ran: polis/client-admin/convo_access.spec.js │
└───────────────────────────────────────────────────────
overall results:
(Run Finished)
Spec Tests Passing Failing Pending Skipped
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ✖ polis/client-admin/convo_access.spe 00:57 6 - 6 - - │
│ c.js │
├────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ✖ polis/client-admin/create_user.spec 00:21 2 - 2 - - │
│ .js │
├────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ✔ polis/client-admin/home.spec.js 923ms 2 2 - - - │
├────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ✔ polis/client-admin/public_pages.spe 876ms 2 2 - - - │
│ c.js │
├────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ✖ polis/client-admin/routes.spec.js 00:48 5 - 5 - - │
├────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ✖ polis/client-admin/share.spec.js 00:06 2 - 1 - 1 │
├────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ✖ polis/client-admin/signin.spec.js 00:06 1 - 1 - - │
├────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ✔ polis/client-participation/i18n.spe 00:10 11 11 - - - │
│ c.js │
├────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ✖ polis/client-participation/kitchens 00:07 1 - 1 - - │
│ ink.spec.js │
├────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ✖ polis/client-participation/social-l 00:05 4 - 1 - 3 │
│ ogin.spec.js │
├────────────────────────────────────────────────────────────────────────────────────────────────┤
│ ✖ polis/emails.spec.js 00:15 7 - 2 5 - │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
✖ 8 of 11 failed (73%) 03:01 43 15 19 5 4
Let's see if I can successfully run the e2e tests on the dev branch:
git checkout dev
# rebuild
docker-compose up --build
cd e2e
npm install
CYPRESS_BASE_URL=http://127.0.0.1 npm run e2e:standalone
The tests also fail for me on dev, in the same way as abovie. This makes me think that there is some local setup or docker networking issue.
http://127.0.0.1/home does render the Polis home page, and I do see the page render inside of the Cypress tests, so _part_ of what is running locally inside the docker container is working.
micahstubbs
on 15 Feb 2021
related comment https://github.com/pol-is/polis/pull/734#issuecomment-768275431
micahstubbs
on 15 Feb 2021
I'm def expecting a local failure in e2e tests unless you dissect the canonical dock-compose setup and get all pieces running locally in the same way they run in docker.
This would probably be good to document, so that new contributors aren't confused. It's okay if local e2e tests aren't really supported now, we should just document that.
micahstubbs
on 15 Feb 2021
I like this idea from @ballPointPenguin
I wonder if it makes more sense to run the Cypress tests within docker than it does to run them outside of docker and thus be subject to the unique environment of each user's machine.
https://github.com/pol-is/polis/pull/734#issuecomment-768315873
micahstubbs
on 15 Feb 2021
Hey Micah! Thanks man -- I feel bad that docker keeps giving you hassle. I've been in the situation where one tool keeps me from contributing, and I personally get frustrated real quick by it -- almost unreasonably so 😄
Would you be down to jump on a quick call sometime? I feel like this is probably something we could catch real quick, given that a fresh environment on CI is consistently working.
Also, seems there are sanctioned way to run cypress inside containers: 🚀 https://docs.cypress.io/examples/examples/docker.html#Examples
Though again, I don't think this is the issue. I believe it's the polis containers failing here for some reason, not cypress itself (in my understanding, they've put a ton of work into cypress running beautifully cross-platform). It looks like the tests passing tests are the ones that don't access admin via UI 🤔
patcon
on 15 Feb 2021
I'm reproducing this now with a completely fresh install (since I recently reformatted my drive).
I'm starting with
docker-compose pull
docker-compose up
since I'm using the 'dev' branch, which has up-to-date docker images pre-built on https://hub.docker.com/u/polisdemo (as specified in the docker-compose file)
ballPointPenguin
on 16 Feb 2021
ballPointPenguin
on 16 Feb 2021
hmmm... maybe our pulled images no longer work. i've been building, and haven't used the docker hub ones in months. We may have broken those (or let their build process go stale)
EDIT: how about a fresh build without pulled images. they work?
patcon
on 16 Feb 2021
my results are nearly identical to @micahstubbs
ballPointPenguin
on 16 Feb 2021
I had similar results with built images last time, as did micah, so I don't suspect the pulled images in this case
ballPointPenguin
on 16 Feb 2021
- what hostname do you use? i exclusively use 123.45.67.89.xip.io (ip being the ip of my digital ocean machine that runs docker)
EDIT:
- as sanity check, is it just e2e tests that fail, or can you even log in manually?
- can you try 127.0.0.1.xip.io via manual and e2e test base url?
patcon
on 16 Feb 2021
Cannot GET /_domainWhitelistError_api/v3/auth/new
I see this error when trying to manually create a new user
ballPointPenguin
on 16 Feb 2021
what hostname do you use? i exclusively use 123.45.67.89.xip.io (ip being the ip of my digital ocean machine that runs docker)
http:://127.0.0.1
running locally
I can view the app in my browser at http://localhost or http://127.0.0.1
ballPointPenguin
on 16 Feb 2021
I can view the app in my browser
And do things like create new convos, etc? (also, hello email readers! we put slack in your github in your inbox! sorry not sorry)
patcon
on 16 Feb 2021
I can view the app in my browser
And do things like create new convos, etc? (also, hello email readers! we put slack in your github in your inbox! sorry not sorry)
No. I can't create a user (see above error), so I can't do anything really. I think the tests are correctly failing because in its current default configuration, the docker setup is failing somewhere.
ballPointPenguin
on 16 Feb 2021
something isn't happening here like it needs to
https://github.com/pol-is/polis/blob/dev/client-admin/src/util/url.js
ballPointPenguin
on 16 Feb 2021
not sure if you saw my ask above, but what happens when using 127.0.0.1.xip.io? final important Q before dig in :)
patcon
on 16 Feb 2021
AHA!
so http://localhost works
as does http://127.0.0.1.xip.io
but not http://127.0.0.1 because it's not in the whitelist
(by "works" I mean that I can create a user without the above error; test results TBD)
ballPointPenguin
on 16 Feb 2021
Yesss i am glad for how easy the fix might be -- but yeah, this was a stuuuupid frustrating thing that i'm sure many ppl ran into. Thanks @micahstubbs!
patcon
on 16 Feb 2021
New result: 2 of 14 failed
x polis/client-participation/embeds.spec.js
x polis/client-participation/integration.spec.js
ballPointPenguin
on 16 Feb 2021
both of those are the new ones that use iframes, and will need some love to make them less fincky. i've had to re-run them a few times on CI. It might be that your workstation is labouring enough that default timeouts are overflowing
Maybe try running em with cypress open UI, which will allow you to poke around the test run environment after a fail. if you see it load after failure, then we'll know it's a timing thing :)
patcon
on 16 Feb 2021
Same results with newly built docker images.
I'll try those two failing tests with cypress open.
ballPointPenguin
on 16 Feb 2021
this line never finds its match:
cy.enter(#polis_${this.siteId}_${pageId}).then(cyframe => {
And in debugging and poking around I do not see anything in the DOM with an ID like that.
ballPointPenguin
on 16 Feb 2021
you don't see that on the wrapping embed page? (It wouldn't be on the polis convo page itself, to be clear)
I'm guessing the issue may be that it's failing to capture the this.siteId value from /integrate page... so maybe use this
cy.log(this.siteId)
cy.log(this.pageId)
to see which ones missing
Thinking maybe this is failing to match for some reason:
patcon
on 16 Feb 2021
Yeah, seems this is what gets matched from my <pre> tag, so it should be that full string in siteId:
https://regex101.com/r/3gUKbo/1
patcon
on 16 Feb 2021
@patcon curious, do you want to send a separate PR for the selector issue with these two tests?
x polis/client-participation/embeds.spec.js
x polis/client-participation/integration.spec.js
It looks like #837 only fixes the domain issue.
I'm excited to get these tests reliably passing locally. Progress!
micahstubbs
on 17 Feb 2021
I suggest that this issue can be closed with https://github.com/pol-is/polis/pull/837 and maybe an update to the e2e README about what xip.io is and how to use it (@patcon ?)
ballPointPenguin
on 17 Feb 2021
Good call! Yes: Progress!
General improvements for e2e tests exists: https://github.com/pol-is/polis/issues/854
Re-ticketed e2e embed test fixes here: https://github.com/pol-is/polis/issues/839
Re-ticketed README improvements here:
@micahstubbs ready to close out if that sounds like it captures everything 🎸
patcon
on 17 Feb 2021
Pretty sure this is good to close, but re-open if mistaken!
patcon
on 23 Feb 2021
will re-open this until PRs are merged
micahstubbs
on 24 May 2021
okay, after reading more, the new ticket is good. will close this again. https://github.com/compdemocracy/polis/issues/839
micahstubbs
on 24 May 2021
Related issues
myguidingstar
·
9Comments
patcon
·
9Comments
patcon
·
6Comments
patcon
·
10Comments
patcon
·
10Comments
Most helpful comment
Same results with newly built docker images.
I'll try those two failing tests with
cypress open.