Cocoapods: [RFC] Support for launch arguments and environment variables in test spec schemes

[abbeycode]

• [x] I've read and understood the *CONTRIBUTING guidelines and have done my best effort to follow.

Background

I have a project that produces very verbose logging output as its unit tests run. When I run pod lib lint from inside a CI job, I hit the 4 MB limit and my build gets killed. It looks like the best way to defeat this is by setting an environment variable on my unit testing scheme, but I realized CocoaPods doesn't give you any capability to set launch arguments or environment variables on the scheme it creates, as shown in the screenshot below.

Suggested Solution

I propose adding a couple of items to the Test Spec DSL that would allow Podspec authors to create both launch arguments and environment variables for the tests that run. There are many uses to this flexible solution beyond the one that led me to make this suggestion.

DSL

  s.test_spec 'Tests' do |test_spec|
    ...
    test_spec.test_scheme_launch_args = "--iLoveCocoaPods YES", "--SecondArgument"
    test_spec.test_scheme_env_vars = { "MY_ENVIRONMENT_VARIABLE" => "A Value" }
  end

Generated xcscheme code

Since this change affects a scheme, we would need to represent both new types of data in the xcscheme file generated inside ___.xcodeproj/xcshareddata/xcschemes/. The DSL sample above would produce the following in the scheme file (with irrelevant parts elided).

 <?xml version="1.0" encoding="UTF-8"?>
<Scheme ...>
   ...
   <TestAction>
      ...
      <CommandLineArguments>
         <CommandLineArgument
            argument = "--iLoveCocoaPods YES"
            isEnabled = "YES">
         </CommandLineArgument>
         <CommandLineArgument
            argument = "--SecondArgument"
            isEnabled = "YES">
         </CommandLineArgument>
      </CommandLineArguments>
      <EnvironmentVariables>
         <EnvironmentVariable
            key = "MY_ENVIRONMENT_VARIABLE"
            value = "A Value"
            isEnabled = "YES">
         </EnvironmentVariable>
      </EnvironmentVariables>
   </TestAction>
   ...
</Scheme>

[dnkoutso]

@abbeycode how is that represented in an .xcodeproj? I haven't seen those and not sure if there is any support yet.

[dnkoutso]

@abbeycode this:

    test_spec.pod_target_xcconfig = { "OTHER_CFLAGS" => "$(inherited) -Wno-unguarded-availability"

Should work. Does there is a chance OTHER_CFLAGS is actually being stomped by CocoaPods as found in https://github.com/CocoaPods/CocoaPods/issues/7577

If you try another setting it works? If so I think this is a dup of #7577 and we can fix.

[abbeycode]

@dnkoutso The test spec is working fine as is. I only provided it to give context. I want the OS_ACTIVITY_MODE environment variable to reside in a scheme. Schemes are in ___.xcodeproj/xcshareddata/xcschemes/___.xsscheme. Does CocoaPods have to create a scheme to run the unit tests from a test spec? Or is it able to use a default scheme?

[dnkoutso]

@abbeycode I do not think those are supported currently by CocoaPods. That is a feature request and will mark it as such.

[amorde]

I personally don't see why a Podspec should be setting an environment variable such as OS_ACTIVITY_MODE, as that affects the output displayed in the console when debugging. Do you have an example where this feature would be beneficial?

[dnkoutso]

I think test specs could use this as test specs create xcschemes so it would only be a test spec DSL

[amorde]

Ah I see, that makes sense

[abbeycode]

@amorde The reason I'm looking to do this is because I need a way to limit output or else Travis CI kills my builds (I'm exceeding their 4 MB limit). I would rather turn down the logging level, but the ways I tried doing that with Travis didn't work. This definitely works though, since I'm using the Unified Logging framework, so it would be nice to have a way to do it. I'd be open to making a PR but I'd need some guidance.

[dnkoutso]

@abbeycode this is a bit more complicated. The first step is defining the DSL and it has to be under https://github.com/CocoaPods/Core project and then we have to update this project to integrate the setting properly based on the podspec.

[abbeycode]

For the DSL, I propose the following, the following, to mimic what you can configure in Xcode:

  s.test_spec 'Tests' do |test_spec|
    test_spec.test_scheme_launch_args = "MyFirstArgument", "MySecondArgument"
    test_spec.test_scheme_env_vars = { "MY_ENVIRONMENT_VARIABLE" => "Different Value" }
  end

What's the usual process for proposing it and getting it reviewed before I begin the PR? Or should I try moving forward with the PR and it'll get reviewed there?

[dnkoutso]

https://github.com/CocoaPods/CocoaPods/issues/7134 check this out as a sample.

You can propose it here in this issue.

[abbeycode]

@dnkoutso Done and done.

[amorde]

Another option instead of attributes test_scheme_ prefixed attributes is to have something similar to platform, where the attribute test_scheme exposes attributes of a scheme

s.test_spec 'Tests' do |test_spec|
    test_spec.test_scheme.launch_arguments = ["MyFirstArgument", "MySecondArgument"]
    test_spec.test_scheme.environment_variables = { "MY_ENVIRONMENT_VARIABLE" => "Value" }
end

This would give us the opportunity to further extend it with another scheme, such as the library / framework scheme

# Part of the Pod's scheme that builds the pod
s.build_scheme.launch_arguments = ["MyBuildArg1", "MyBuildArg2"]
s.build_scheme.environment_variables = { "BUILD_VAR" => "Value" }

# Could also be called "scheme" instead of build_scheme

s.test_spec 'Tests' do |test_spec|
    # Only applies to the scheme for tests
    test_spec.test_scheme.launch_arguments = ["TestArg1"]
    test_spec.test_scheme.environment_variables = { "TEST_VARIABLE" => "TESTING" }
end

It would also allow us to support setting other values that are part of the scheme if desired.

[dnkoutso]

interesting. It can be combined then with build_scheme

[abbeycode]

@amorde Your suggestion makes a lot of sense. Should I update my proposal to match that, or is having it only in your comment preferred?

[amorde]

I would let it sit for a bit and see if others have any suggestions or comments - once we add this to the DSL we can't undo it, so its good to get a few ideas on it :)

Once there is some sort of consensus, let's update the proposal to match

[dnkoutso]

@abbeycode , @amorde

I think we should start with test specs first, we can then open it up later. There is another thing thats problematic that will be solved in 1.6.0 and test specs.

Right now if a podspec has more than 1 test spec CocoaPods will group them together by test type (currently only :unit supported). This means that if you add test_spec.build_scheme.launch_args = ["Something"] it will have to merge the two.

If you want to wait for 1.6.0 (or wait until my PR lands) then this will be way easier as each test spec will get its own test bundle and scheme.

[amorde]

Waiting for 1.6.0 sounds good to me. This might take a bit to get implemented regardless.

In the meantime @abbeycode I'm not sure of a way to fix the issue you are currently having

[dnkoutso]

We do not have to wait for all of 1.6.0 to ship, only the part that enables 1-to-1 mapping of each test spec to a test bundle which will be part of 1.6.0.

[abbeycode]

@amorde I'm working on a hack, piping my pod lib lint output to sed and filtering out lines containing xctest. Hopefully I can get that to work in the meantime.

[abbeycode]

Looks like I finally got my workaround working, so this isn't a blocker for me anymore (FWIW).

[abbeycode]

@dnkoutso Are there any changes we should make to labels or anything to encourage more feedback?

[dnkoutso]

@abbeycode not much from me! I am working on updating test specs to actually give you a separate test target per test spec (instead of the grouping that happens today)

want to wait until maybe next week for me to open this PR?

[abbeycode]

@dnkoutso Sounds good!

[dnkoutso]

btw @abbeycode I was just thinking we do not have to necessarily wait for my PR to do one_to_one test spec mapping to a test target.

If you want you can begin work in cocoapods-core to introduce the DSL.

[amorde]

Are we good with this syntax, including the changes I suggested? I'm thinking build_scheme could be renamed to simply scheme. Could also just focus on the test_scheme for now and revisit the build scheme later

[dnkoutso]

I think narrowing down for test_scheme for starters would be good!

[amorde]

Yup, and starting with just environment variables and launch arguments would be good as well

[dnkoutso]

Still a great enhancement.

[dnkoutso]

Marking this for 1.7.0

[dnkoutso]

Thought this about a bit more.

# ...
s.test_spec 'Tests' do |test_spec|
    # Only applies to the scheme for tests
    test_spec.test_scheme.launch_arguments = ["TestArg1"]
    test_spec.test_scheme.environment_variables = { "TEST_VARIABLE" => "TESTING" }
end

This won't work with current attribute and DSL support. We will have to extend it to support other classes and notice that it has multiple declarations of test_spec.test_scheme therefore we would need to memoize the scheme once created.

If we do this outside of the attribute support within the DSL then this means we lose support for multi-platform like test_spec.ios.test_scheme etc.

I propose we stick to the original (and arguably slightly worse) API:

# ...
s.test_spec 'Tests' do |test_spec|
    # Only applies to the scheme for tests
    test_spec.scheme = { 
      :launch_arguments => ["TestArg1"],
      :environment_variables => { "TEST_VARIABLE" => "TESTING" }
    }
end