Flank: IOS GCloud, Flank command parity

[Sloox]

This is a perfectly valid gcloud command:

gcloud alpha firebase test ios run 
--type game-loop --app build/ipa/flankgameloop.ipa --scenario-numbers 1 
--device model=iphone11,version=13.6

This is not:

gcloud:
  app: test_projects/ios/FlankGameLoopExample/ipa/FlankGameLoopExample.ipa
  type: game-loop
  scenario-numbers:
      - 1
  device:
      - model: iphone11
        version: 13.6
        orientation: portrait
        locale: en_US

Flank expects a xctestrunfile & test, as it fails the above command with:
Both of following options must be specified [test, xctestrun-file].

is this necessary?

some changes would be needed to get it to parity.

[bootstraponline]

@Sloox you're right! I think we should change this to get parity.

[jan-gogo]

@Sloox you're right! I think we should change this to get parity.

I agree, Both of following options must be specified [test, xctestrun-file]. was true before the game loop was added. Now it is not necessary if the type is game-loop so the validation of IosArgs should be fixed.

[Sloox]

@jan-gogo unfortunately simply fixing the validation doesn't work as intended. the current implementation is tied very closely with looking for the test, xctestrun-file. I attempted to skip past all the checks for the xctestrun but it fails to upload anything or progress.

[jan-gogo]

I'm almost sure, I know how to implement fix this issue. On my refactored implementation from #1321 with support for xctestplans, it should be quite simple.

[Sloox]

@jan-gogo as we discussed will merge in the test application #1280 & #1321 then we will work on this together

[pawelpasterz]

@Sloox @jan-gogo
Since you have already figured it out, could you give some info on why it's not working and what is the fix? (from a high-level perspective ofc)

[jan-gogo]

My recommendation is to implement abstraction over iOS test types similar to solution used for android (AndroidTestContext), so the GcIosTestMatrix.build function will receive IosTestContext instead of testZipGcsPath, xcTestRun and probably --additional-ipas (it's related to game-loop, right?). After this change IosArgs.runIosTests shouldn't know about exact test type and details, it will be only aware about IosTestContext object.

In steps:

1. Fix validation on test and xctestrun-file to not be required if the type is game-loop.
2. Add IosTestContext sealed class with derivatives data-classes for specific test types, similar to AndroidTestContext.
3. Change implementation and name of IosArgs.xcTestRunFlow. Now it should return flow of IosTestContext instead of ByteArray
4. Pass IosTestContext to GcIosTestMatrix.build as argument and handle it properly.

[Sloox]

@jan-gogo I feel the ticket then needs a re-estimation as that appears to be a large task.
I attempted an implementation where we made a special case for game loop. However it caused trouble due to a lot of the IOS changes that came. We spoke briefly and you mentioned that it could be fixed in an "easier manner".
I unfortunately do not know what that meant or have simply forgotten. Do you remember?

I agree however that you proposal is most likely a better approach.

[jan-gogo]

Of course it can be re-estimated If you feel that is larger than we previously assumed. Here https://github.com/Flank/flank/issues/1341#issuecomment-732900657 I mentioned that it should be quite simple. I meant quite simple in comparastion to situation before #1321, because after merge I was able to specify those in 4 points quite easily, which was not possible on old implementation (for me).