Cli-microsoft365: Bug report: multiple failing tests on Windows

[appieschot]

Description

Besides the failing test described in #1375 there are an additional 12 tests failing on the spfx side of things. All tests that use comparison or assertions on the \\ fails on windows.

Steps to reproduce

Follow path to awesome and run npm test

Expected result

all test pass

Actual result

Be aware that the first failing issue is described in #1375

  7848 passing (14s)
  13 failing

  1) spfx project externalize
       covers all text report branches:

      AssertionError [ERR_ASSERTION]: 124 == 122
      + expected - actual

      -124
      +122

      at Context.<anonymous> (dist\o365\spfx\commands\project\project-externalize.spec.js:666:16)
      at processImmediate (internal/timers.js:439:21)

  2) spfx project upgrade
       e2e: suggests creating small teams app icon using a fixed name for upgrading react web part 1.6.0 project to 1.7.0:

      AssertionError [ERR_ASSERTION]: 'teams\\tab20x20.png' == 'teams/tab20x20.png'
      + expected - actual

      -teams\tab20x20.png
      +teams/tab20x20.png

      at C:\Git\pnp\office365-cli\dist\o365\spfx\commands\project\project-upgrade.spec.js:1389:20
      at SpfxProjectUpgradeCommand.commandAction (dist\o365\spfx\commands\project\project-upgrade.js:239:9)
      at Object.action (dist\o365\spfx\commands\project\project-upgrade.js:63:17)
      at Context.<anonymous> (dist\o365\spfx\commands\project\project-upgrade.spec.js:1387:21)
      at processImmediate (internal/timers.js:439:21)

  3) spfx project upgrade
       e2e: suggests creating large teams app icon using a fixed name for upgrading react web part 1.6.0 project to 1.7.0:

      AssertionError [ERR_ASSERTION]: 'teams\\tab96x96.png' == 'teams/tab96x96.png'
      + expected - actual

      -teams\tab96x96.png
      +teams/tab96x96.png

      at C:\Git\pnp\office365-cli\dist\o365\spfx\commands\project\project-upgrade.spec.js:1397:20
      at SpfxProjectUpgradeCommand.commandAction (dist\o365\spfx\commands\project\project-upgrade.js:239:9)
      at Object.action (dist\o365\spfx\commands\project\project-upgrade.js:63:17)
      at Context.<anonymous> (dist\o365\spfx\commands\project\project-upgrade.spec.js:1395:21)
      at processImmediate (internal/timers.js:439:21)

  4) spfx project upgrade
       e2e: suggests creating small teams app icon using a dynamic name for upgrading react web part 1.7.1 project to 1.8.0:

      AssertionError [ERR_ASSERTION]: 'teams\\7c4a6c24-2154-4dcc-9eb4-d64b8a2c5daa_outline.png' == 'teams/7c4a6c24-2154-4dcc-9eb4-d64b8a2c5daa_outline.png'
      + expected - actual

      -teams\7c4a6c24-2154-4dcc-9eb4-d64b8a2c5daa_outline.png
      +teams/7c4a6c24-2154-4dcc-9eb4-d64b8a2c5daa_outline.png

      at C:\Git\pnp\office365-cli\dist\o365\spfx\commands\project\project-upgrade.spec.js:1505:20
      at SpfxProjectUpgradeCommand.commandAction (dist\o365\spfx\commands\project\project-upgrade.js:239:9)
      at Object.action (dist\o365\spfx\commands\project\project-upgrade.js:63:17)
      at Context.<anonymous> (dist\o365\spfx\commands\project\project-upgrade.spec.js:1503:21)
      at processImmediate (internal/timers.js:439:21)

  5) spfx project upgrade
       e2e: suggests creating large teams app icon using a dynamic name for upgrading react web part 1.7.1 project to 1.8.0:

      AssertionError [ERR_ASSERTION]: 'teams\\7c4a6c24-2154-4dcc-9eb4-d64b8a2c5daa_color.png' == 'teams/7c4a6c24-2154-4dcc-9eb4-d64b8a2c5daa_color.png'
      + expected - actual

      -teams\7c4a6c24-2154-4dcc-9eb4-d64b8a2c5daa_color.png
      +teams/7c4a6c24-2154-4dcc-9eb4-d64b8a2c5daa_color.png

      at C:\Git\pnp\office365-cli\dist\o365\spfx\commands\project\project-upgrade.spec.js:1513:20
      at SpfxProjectUpgradeCommand.commandAction (dist\o365\spfx\commands\project\project-upgrade.js:239:9)
      at Object.action (dist\o365\spfx\commands\project\project-upgrade.js:63:17)
      at Context.<anonymous> (dist\o365\spfx\commands\project\project-upgrade.spec.js:1511:21)
      at processImmediate (internal/timers.js:439:21)

  6) FN018001_TEAMS_folder
       doesn't return notifications if teams folder exists:

      AssertionError [ERR_ASSERTION]: 1 == 0
      + expected - actual

      -1
      +0

      at Context.<anonymous> (dist\o365\spfx\commands\project\project-upgrade\rules\FN018001_TEAMS_folder.spec.js:42:16)
      at processImmediate (internal/timers.js:439:21)

  7) FN018002_TEAMS_manifest
       looks for Teams manifest for a web part using the correct path:

      AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:

  assert(existsSyncFake.calledWith('/usr/tmp/teams/manifest_webpart.json'))

      + expected - actual

      -false
      +true

      at Context.<anonymous> (dist\o365\spfx\commands\project\project-upgrade\rules\FN018002_TEAMS_manifest.spec.js:39:9)
      at processImmediate (internal/timers.js:439:21)

  8) FN018002_TEAMS_manifest
       creates manifest with a unique name following the web part name (single web part):

      AssertionError [ERR_ASSERTION]: 'teams\\manifest_webpart.json' == 'teams/manifest_webpart.json'
      + expected - actual

      -teams\manifest_webpart.json
      +teams/manifest_webpart.json

      at Context.<anonymous> (dist\o365\spfx\commands\project\project-upgrade\rules\FN018002_TEAMS_manifest.spec.js:145:16)
      at processImmediate (internal/timers.js:439:21)

  9) FN018002_TEAMS_manifest
       creates manifest with a unique name following the web part name (multiple web parts):

      AssertionError [ERR_ASSERTION]: Incorrect manifest path for web part 1
      + expected - actual

      -teams\manifest_webpart1.json
      +teams/manifest_webpart1.json

      at Context.<anonymous> (dist\o365\spfx\commands\project\project-upgrade\rules\FN018002_TEAMS_manifest.spec.js:167:16)
      at processImmediate (internal/timers.js:439:21)

  10) FN018003_TEAMS_tab20x20_png
       returns path to icon with the specified name when fixed name used:

      AssertionError [ERR_ASSERTION]: 'teams\\tab20x20.png' == 'teams/tab20x20.png'
      + expected - actual

      -teams\tab20x20.png
      +teams/tab20x20.png

      at Context.<anonymous> (dist\o365\spfx\commands\project\project-upgrade\rules\FN018003_TEAMS_tab20x20_png.spec.js:58:16)
      at processImmediate (internal/timers.js:439:21)

  11) FN018003_TEAMS_tab20x20_png
       returns path to icon with name following web part ID when no fixed name specified:

      AssertionError [ERR_ASSERTION]: 'teams\\c93e90e5-6222-45c6-b241-995df0029e3c_outline.png' == 'teams/c93e90e5-6222-45c6-b241-995df0029e3c_outline.png'
      + expected - actual

      -teams\c93e90e5-6222-45c6-b241-995df0029e3c_outline.png
      +teams/c93e90e5-6222-45c6-b241-995df0029e3c_outline.png

      at Context.<anonymous> (dist\o365\spfx\commands\project\project-upgrade\rules\FN018003_TEAMS_tab20x20_png.spec.js:72:16)
      at processImmediate (internal/timers.js:439:21)

  12) FN018004_TEAMS_tab96x96_png
       returns path to icon with the specified name when fixed name used:

      AssertionError [ERR_ASSERTION]: 'teams\\tab96x96.png' == 'teams/tab96x96.png'
      + expected - actual

      -teams\tab96x96.png
      +teams/tab96x96.png

      at Context.<anonymous> (dist\o365\spfx\commands\project\project-upgrade\rules\FN018004_TEAMS_tab96x96_png.spec.js:58:16)
      at processImmediate (internal/timers.js:439:21)

  13) FN018004_TEAMS_tab96x96_png
       returns path to icon with name following web part ID when no fixed name specified:

      AssertionError [ERR_ASSERTION]: 'teams\\c93e90e5-6222-45c6-b241-995df0029e3c_color.png' == 'teams/c93e90e5-6222-45c6-b241-995df0029e3c_color.png'
      + expected - actual

      -teams\c93e90e5-6222-45c6-b241-995df0029e3c_color.png
      +teams/c93e90e5-6222-45c6-b241-995df0029e3c_color.png

      at Context.<anonymous> (dist\o365\spfx\commands\project\project-upgrade\rules\FN018004_TEAMS_tab96x96_png.spec.js:72:16)
      at processImmediate (internal/timers.js:439:21)

Environment

Windows 10, PowerShell, 2.7.0, Node v12.10.0

[waldekmastykarz]

Ouch! Good catch. Let's try to fix it asap to prevent blocking our contributors on Windows.

[appieschot]

Suggestion to add an or clause to all checks that fail due to this issue to validate against \ as well as to / to support both Windows and Linux. If you agree you can assign to me and I will pick up asap :)

[waldekmastykarz]

Not sure I follow, could you please explain it? 😊

[appieschot]

Not sure I follow, could you please explain it? 😊

We now fail on windows because it handles the comparison on / different. We could add a or clause to all the test that now fails to check either the \ as used in Linux and a // as used in Windows.

[waldekmastykarz]

What if rather than checking specifically just for paths, we ran the whole test suite also on Windows so that we cover these, as well as other kinds of OS-specific errors?

[appieschot]

Not quite sure if I follow, but my take is that it had to do with the path package: import * as path from 'path'; As that is used to construct to create the paths to return that acts different on Windows and OSX and Linux:

Sample:

var filePath = path.join(foo, bar);
// 'foo/bar' on OSX and Linux
// 'foo\\bar' on Windows

I am not sure if the path part actually has an option to return the same value in all scenario's so the tests won't fail. But if there is a better approach I am all up for it :D

[waldekmastykarz]

Shouldn't the path be different on each OS?

[appieschot]

I guess so, but since we assert the path in the tests we should assert on both outcomes right?

[waldekmastykarz]

If use path separator from Node.js and then run tests on both OS-es, we should be good, right?

[waldekmastykarz]

For the teams file paths, should the path actually follow the OS or not? Can you see what the default project contents are on Windows so that we can compare them and make right assumptions?

[appieschot]

Oke sorry for the confusion; turns out the manifest packages expect a / in the format. That does mean that the test are correct on OSX / Linux and are working as expected. The reason they fail on Windows is because we are using path.join in our reports. That means the reports return a incorrect value on Windows (and thus the test fail correctly). We should not update the tests but rather update the code that is making the test fail to not use the path.join. Feelf ree to assign to me.