Freecodecamp: Detect and add notification if tests fail to run.

[erictleung]

Challenge Name

Mainly JavaScript challenges

Issue Description

We've had many coding issues made on GitHub (a list of some from the past few months: #9532, #9137, #9118, #8940, #9058, #8436, #8266, #8156, #8092 [comment within this issue], #7655, #7441, #7629, #7470, #7453, #7442, #5290, #4819, #4733, #4686, #4187, #3755, #1345) on why their code doesn't work, which turn out to be because they have a global variable they mutate with their function.

Proposed Fix (Updated)

Somewhere early in the JavaScript challenges have a note on how the tests work, in that they will only run your function and any global variables mutated in the process will keep their state.

• Detect unsafe global variable code and wrap them in isolated blocked scope.
• Display a notification in such cases if applicable and tests fail to run.

cc/ @FreeCodeCamp/moderators

[BerkeleyTrue]

@erictleung Most of these should be fixed in the new react version. We now destroy and recreate the iframe on each test pass.

Actually looking at the first couple it seems like campers don't realize that we run their function multiple times.

[Bouncey]

I saw a comment early in a coding game (I think it was code wars) about how they ran multiple tests in the background, and that it was in effort to stop players from hard coding the results just to advance a level.

I think something to this effect at the start of js would be good.

[raisedadead]

Yeah a generic intro challenge should suffice.

Also, how about an animated progress bar that corresponds to the no, of tests executing, just between the run tests button and the tests? This could be empty or set to no of passed tests.

[erictleung]

@raisedadead I like the idea of a progress bar, or at least showing the user if/when tests are run. Right now, it is unclear on whether the tests ran or your code is still wrong. Maybe progress bar or any other visuals on the tests working could be another issue.

@HKuz do you have any comment on where this could go? I was thinking adding some instructions before the Basic Algorithm Challenges would be good because each challenge there requires the use of functions. The Basic JavaScript section does have some functions but it strongly suggests where the camper can insert their code with suggestions like // Change code below this line

[QuincyLarson]

@erictleung I suspect an explanation will not suffice. Many campers will forget it, or skip reading it all together. Instead, we need a way to detect that tests will not run and tell them why the tests aren't running.

[Bouncey]

Currently in staging the output window cycles between running tests and
tests complete when the camper runs the tests.

Also the test suit has an initial state that is different from a failing
and passing test. The 'scroll the test suite into view when tests have run'
animation still needs to be implemented into the react version of the site

Do we really need more hints that the tests have run?

On Thursday, 1 December 2016, Eric Leung [e-mail] wrote:

@raisedadead https://github.com/raisedadead I like the idea of a
progress bar, or at least showing the user if/when tests are run. Right
now, it is unclear on whether the tests ran or your code is still wrong.
Maybe progress bar or any other visuals on the tests working could be
another issue.

@HKuz https://github.com/HKuz do you have any comment on where this
could go? I was thinking adding some instructions before the Basic
Algorithm Challenges would be good because each challenge there requires
the use of functions. The Basic JavaScript section does have some functions
but it strongly suggests where the camper can insert their code with
suggestions like // Change code below this line

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/FreeCodeCamp/FreeCodeCamp/issues/9063#issuecomment-264047502,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ARtk5tPI77i_TnvgSpAA7ts2GG79zGJJks5rDh1HgaJpZM4IyjVW
.

[QuincyLarson]

@Bouncey as long as it has a message like "Your tests couldn't run. Please double check your code. " or something like that, I think this will be sufficient.

[raisedadead]

we need a way to detect that tests will not run

Sadly this covers a wide area of test cases, combinations, and if implemented will result in more complicated bugs.

I am open to the idea of detecting global variables so we may show a notif, but the problem here is they are global vars and mutate the result of the test cases following the first one.

It would be really challenging to come up with a robust way to know what went wrong.

An ideal solution would be to take each test case and execute them in separate isolated containers. we currently use eval to do so, its kinda dangerous, but gives us bugs such as these.

[Bouncey]

It could be done with an iframe for each function call, which is destroyed
immediately after the test has run. But implementation would have a certain
element of complexity and testing overhaul.

On Thursday, 1 December 2016, mrugesh [e-mail] wrote:

we need a way to detect that tests will not run

Sadly this covers a wide area of test cases, combinations, and if
implemented will result in more complicated bugs.

I am open to the idea of detecting global variables so we may show a
notif, but the problem here is they are global vars and mutate the result
of the test cases following the first one.

It would be really challenging to come up with a robust way to know what
went wrong.

An ideal solution would be to take each test case and execute them in
separate isolated containers. we currently use eval to do so, its kinda
dangerous, but gives us bugs such as these.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/FreeCodeCamp/FreeCodeCamp/issues/9063#issuecomment-264093079,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ARtk5i8yvRYUVjEmly66wtAS4-TWogj5ks5rDm8kgaJpZM4IyjVW
.

[raisedadead]

iframe

They still have the context of their parent, and would only increase the complexity, and remember that they are going to make things slower, as well. Not a favored choice.

[Bouncey]

Agreed Mrugesh

On Thursday, 1 December 2016, Stuart Taylor [e-mail] wrote:

It could be done with an iframe for each function call, which is
destroyed immediately after the test has run. But implementation would have
a certain element of complexity and testing overhaul.

On Thursday, 1 December 2016, mrugesh <[email protected]

we need a way to detect that tests will not run

Sadly this covers a wide area of test cases, combinations, and if
implemented will result in more complicated bugs.

I am open to the idea of detecting global variables so we may show a
notif, but the problem here is they are global vars and mutate the
result of the test cases following the first one.

It would be really challenging to come up with a robust way to know what
went wrong.

An ideal solution would be to take each test case and execute them in
separate isolated containers. we currently use eval to do so, its kinda
dangerous, but gives us bugs such as these.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/FreeCodeCamp/FreeCodeCamp/issues/9063#issuecomment-264093079,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ARtk5i8yvRYUVjEmly66wtAS4-TWogj5ks5rDm8kgaJpZM4IyjVW
.

[Bouncey]

We could wrap each function call in some closure, each with its own copy of
the campers code. So a global in the campers code becomes block scoped for
the that test.

On Thursday, 1 December 2016, Stuart Taylor [e-mail] wrote:

Agreed Mrugesh

On Thursday, 1 December 2016, Stuart Taylor <[email protected]

It could be done with an iframe for each function call, which is
destroyed immediately after the test has run. But implementation would have
a certain element of complexity and testing overhaul.

On Thursday, 1 December 2016, mrugesh [e-mail] wrote:

we need a way to detect that tests will not run

Sadly this covers a wide area of test cases, combinations, and if
implemented will result in more complicated bugs.

I am open to the idea of detecting global variables so we may show a
notif, but the problem here is they are global vars and mutate the
result of the test cases following the first one.

It would be really challenging to come up with a robust way to know what
went wrong.

An ideal solution would be to take each test case and execute them in
separate isolated containers. we currently use eval to do so, its kinda
dangerous, but gives us bugs such as these.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/FreeCodeCamp/FreeCodeCamp/issues/9063#issuecomment-264093079,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ARtk5i8yvRYUVjEmly66wtAS4-TWogj5ks5rDm8kgaJpZM4IyjVW
.

[raisedadead]

Yup, that could be an approach. Updated the OP and the title. Tagging as help wanted.