Swiftgen: [New Feature] Generate xcfilelist for Script Build Phases

[AliSoftware]

Context

In WWDC'18 Session #408 ("Building Faster in Xcode"), they present a new feature of Xcode 10 where you can provide .xcfilelist files to specify the list of input and output files for the build phase, which will allow Xcode to better determine the dependency graph between targets, files having to be rebuilt, etc.

Note: Input & Output files for build phases were already present in previous versions of Xcode to determine when to run the build phases and optimise the build dependency resolution; what's new in Xcode 10 is the ability to put the list of those input/output files in xcfilelist — that can then be generated/updated by external tools without having to modify the xcodeproj for that

Feature Description

It would be cool to make a new swiftgen xcfilelist command that would read the config file and generate an swiftgen-input-files.xcfilelist and swiftgen-output-files.xcfilelist according to the input and output files listed/used by the swiftgen.yml

Obviously the users would still have to:

1. Configure their Script Build Phase invoking "$(PODS_ROOT)/SwiftGen/bin/swiftgen to also use swiftgen-input-files.xcfilelist and swiftgen-output-files.xcfilelist as Input/Output File Lists reps. — But that would be only necessary once, when you configure the Script Build Phase during your installation of SwiftGen into your project
2. Run swiftgen xcfilelist manually every time they update their swiftgen.yml file — because (see 13'50 in the WWDC video) Xcode won't allowed us to modify xcfilelist _during_ the build so we wouldn't be able to make swiftgen (aka swiftgen config run) generate those xcfilelist in addition to generating the other outputs.

That second point is to investigate further though, because it could still be interesting that swiftgen would in fact — like what we do for other generated files — only re-generate-and-override the xcfilelist file if the new content is different than the old one, and not touching the xcfilelist file if nothing has changed — which could lead to Xcode being happy when you didn't change your config file… and probably failing with an (hopefully explicit) error when your xcfilelists are out-of-sync and have been re-generated, which could actually be a good thing…?

Suggested CLI

# Explicit path to files to generate
$ swiftgen xcfilelist --inputs Resources/swiftgen-inputs.xcfilelist --outputs Resources/swiftgen-output.xcfilelist

# Default paths when not specified anywhere: defaults to --inputs ./swiftgen-input-files.xcfilelist --outputs ./swiftgen-output-files.xcfilelist
$ swiftgen xcfilelist 

# Ability to also list the paths in swiftgen.yml to avoid repeating them?
$ swiftgen xcfilelist # uses values from xcfilelists.inputs and xcfilelists.outputs keys from config file

The last mode is debatable, because if the user indeed does have to run swiftgen xcfilelist manually — so assuming we don't re-generate the xcfilelists every time we run swiftgen but only if the content has changed, and we are fine with Xcode failing if the file lists have changed, etc — then having the an xcfilelists entry in the swiftgen.yml to list those paths would likely make most people assume that, because it's in the config file, it will be executed every time, same as all other commands listed in that same config file… and thus if we do that I foresee a lot of false-positive issues/bug reports coming…

Example of generated files

# swiftgen-input-files.xcfilelist

# xcassets
$(SRCROOT)/MyApp/Resources/Images.xcassets
$(SRCROOT)/MyApp/Resources/Logos.xcassets

# fonts
$(SRCROOT)/MyApp/Resources/Fonts/MyFont.ttf
$(SRCROOT)/MyApp/Resources/Fonts/MyOtherFont.ttf

# swiftgen-output-files.xcfilelist

# xcassets
$(SRCROOT)/MyApp/Generated/Images-generated.swift

# fonts
$(SRCROOT)/MyApp/Generated/Fonts-generated.swift

[djbe]

I saw mentions of that new file, haven't seen the session yet though. This replaces the input/output paths list that's currently available in Xcode 9?

Are the input/output xcfilelist paths provided as environment variables during the build phase? If so, we could check and regenerate (or warn the user) as needed.

[AliSoftware]

I saw mentions of that new file, haven't seen the session yet though. This replaces the input/output paths list that's currently available in Xcode 9?

Yes, that's what I wrote in the issue description already (see the note under the screen capture)

Are the input/output xcfilelist paths provided as environment variables during the build phase? If so, we could check and regenerate (or warn the user) as needed.

Yes (that's not mentioned in that WWDC session, but is explained in Xcode 10b1 Release Notes):

Run Script build phases now support declaring input and output files in a .xcfilelist file. This file should contain a newline-seperated list of the file paths for the inputs or outputs. Build setting references may be used in these paths. The path to a copy of the xcfilelist file with resolved build setting references will be provided in the SCRIPT_INPUT_FILE_LIST_(#) and SCRIPT_OUTPUT_FILE_LIST_(#) environment variables of the Run Script phase. Projects which use an .xcfilelist require Xcode 10. (39241255)

[djbe]

Yes, that's what I wrote in the issue description already (see the note under the screen capture)

(woops missed that)

Aha, good to know those'll be available. Wow, you can even define multiple input/output files, we'll have to handle that as well?

[AliSoftware]

Indeed. So in fact file lists doesn't really _replaces_ the old input/output paths we already had in Xcode 9, but comes _in addition_ to them. And if I understand the UI correctly, we can still provide both input files individually (in the "Input Files" section of the Build Phase) _AND_ input file lists (one… or more, indeed), and the full list of input files will, I'm guessing, be the union of them all.

In practice I guess most people will either continue to use "Input Files" one by one as they did in Xcode 9 (because they didn't update their project's Build Phases and kept it the old way, for example) OR use a single "Input File List". But if we want to cover all possible use cases, we should indeed handle the possibility of them having multiple file lists indeed…

[AliSoftware]

But I think if we consider warning the user when the xcfilelist need to be re-generated (because they modified their config files to add new subcommands or new input/output files to their subcommands… or because they added some files in a folder that is one of the input folders listed in the config, like Resources/Fonts/ for example…) then we shouldn't rely on SCRIPT_INPUT_FILE_LIST_(#) env vars, but instead then put the paths to those file lists in the config file (and use the paths from there and not from SCRIPT_INPUT_FILE_LIST_0)

It would be the responsibility of the user, when configuring SwiftGen on their project, to keep its project setup consistant between what they put in their swiftgen.yml's xcfilelist.inputs and xcfilelist.output config keys (= the path of the xcfilelists to generate) and the paths they configured in their Xcode Script Build Phase config.

And it would be the responsibility of SwiftGen to re-generate the xcfilelist files (into paths specified by xcfilelist.inputs and xcfilelist.outputs in the config file) on each run — but only writing that new content to disk if the content has changed, to avoid overriding the file with the same content and resulting in just touch-ing it and making Xcode complain while the content haven't actually changed). Making Xcode probably complain about "this.xcfilelist has been modified during build: build aborted" or something if the file lists had to be updated, which would actually probably be a good thing — and re-running another build right after that failed build would then pass

[samskiter]

+1 to this feature!

It might be worth experimenting with generating the xcfilelists files during a prebuild phase. Also definitely worth making sure the file isn't touched unless it actually changes

[ZvOlCuOk2009]

Tell me how to correctly edit the configuration file. Added from example https://github.com/appwise-labs/AppwiseCore-Example/blob/master/swiftgen.yml when compiling I get an error:

12:9: error: parser: while parsing a block collection in line 11, column 9
did not find expected '-' indicator:
output: Storyboards Scenes.swift
^

[AliSoftware]

@ZvOlCuOk2009 I don't understand your question. This current issue is about discussing for a possible new feature, not implemented yet, about xcfilelists, so I don't see how your question is fitting in there.

If you have an issue using SwiftGen and the configuration file, please read the dedicated documentation about the config file and open a dedicated issue if needed.

[trupin]

@AliSoftware do you think adding a build phase to generate the xcfilelist files prior to the code generation build phase could be an option?

I have been trying this on a code generation tool I'm working on right now, and it seems to work pretty smoothly (see a comment I wrote about that).

[trupin]

Oops, actually just saw @samskiter comment:

It might be worth experimenting with generating the xcfilelists files during a prebuild phase. Also definitely worth making sure the file isn't touched unless it actually changes

Then, to answer to that, my experimentations seem to be positive, for what it's worth.

[AliSoftware]

Hmmm I'm surprised that it works tbh, given that Apple explicitly said it wouldn't during the WWDC session presenting the feature. So not sure how reliable it would be, given that the new build system first resolves build phase dependencies (using xcfilelists among other things) before even running the first phase (that would generate it)…
At least it could work for runs after the very first one but would fail for the first build after a clean (so potentially every time on a CI)…

[AliSoftware]

This is the moment in the WWDC session I'm talking about in my previous comment

The important thing to note here, though, is that these [xcfilelist] files cannot be modified or generated throughout your build process. They are read when your build process starts. And all that information is used then.

[trupin]

I see and I actually figured later that my CI fails to build because of what you mentionned. Too bad Xcode doesn't provide a way to run pre-build scripts. Looks like we'll have to stick with running the xcfilelist command manually.

[AliSoftware]

Actually one way to execute a script before any build phase is to use the "Pre-Actions" of the "Build" action in your Scheme.

This works as it's run before any build, but has the super annoying drawback of not reporting errors (not reporting any output actually), so if/when the script you put in that "Pre-actions" of your scheme fails, it does it completely silently and doesn't tell you anything and doesn't prevent the rest of the build to run…

[trupin]

Interesting. In that particular case, the command could delete the xcfilelist files before regenerating them, which means that in case of error, Xcode would fail reading these files and fail the build. It doesn't give a very explicit error message, but it still breaks the build when something's wrong and points out the direction by saying the xcfilelist files are missing.

[whyAreYouSoShy]

Interesting. In that particular case, the command could delete the xcfilelist files before regenerating them, which means that in case of error, Xcode would fail reading these files and fail the build. It doesn't give a very explicit error message, but it still breaks the build when something's wrong and points out the direction by saying the xcfilelist files are missing.

How to resolve?

[djbe]

Implemented in SwiftGen 6.4.0, which has been released 🎉