Swiftgen: Run commands in parallel

[marcelofabri]

If I understood correctly, this is where SwiftGen starts the process of running commands from configs:

https://github.com/SwiftGen/SwiftGen/blob/1cf0a8fb410b82f30d5615e0e19e0a7b9b033b22/Sources/SwiftGen/Commander/ConfigCommands.swift#L117-L120

Could this be done in parallel threads as an easy way to speed up SwiftGen with multiple commands? Any gotchas if we decide to implement it?

[AliSoftware]

I think that's a really reasonable improvement to provide.

I don't see any blocker from the top of my head which would prevent this from working.

Only points I can think of right now:

• If people misconfigure their config file and use the same output file for multiple config entries. Parallel runs for each entry could then risk trying to write to the same file in parallel. But that would already be a config error from the user to do so, so not sure that's a big deal
• We'd obviously still need a synchronisation point to not make swiftgen end and return before every thread we spawn has finished. So that we don't risk having an invocation of swiftgen in an Xcode script build phase terminate and return control to Xcode before it has finished generating everything
• Not sure how we'd handle individual throws of subcommands. But actually doing those in parallel could actually improve the statu quo here – as currently the first config entry which fails will fail the whole swiftgen execution, while running in parallel could allow other entries to still generate – but we'd still need to ensure to have an exit code that is consistent (be non-zero if _any_ of the parallel thread throws an error

[djbe]

The only thing to be careful about is the logger, that should be synced to a "main" thread/queue, otherwise the user will get garbage output.

[djbe]

I was worried a bit about how this will affect the memory model/access during a template execution. With that I mean that the contents of a context can change during execution (see set tag, or lazy calculation in #700).

Luckily, in the current draft PR we're only parallelising the execution of actions/parsers (colors, fonts, ib, etc...), so:

• Actions parallelisation: #708
• Entries in an action parallelisation: would apply to https://github.com/SwiftGen/SwiftGen/blob/develop/Sources/SwiftGen/Commander/ConfigCommands.swift#L19. ~Could be added in #708, or in a follow-up PR~ Added in #708
• Outputs in an entry parallelisation: would apply to https://github.com/SwiftGen/SwiftGen/blob/develop/Sources/SwiftGen/Commander/ConfigCommands.swift#L62. Not sure if it can easily be done, as the base context variable is shared, although it's copied (and modified) into enriched.

[djbe]

@marcelofabri #708 has been merged into develop, so if you want you can give it a try. If I remember correctly you had some large projects/files that benefited from (amongst others) these changes.

[marcelofabri]

I see some improvements (but I think the other changes are going to be more meaningful for my project):

Before:

$ time Vendor/SwiftGen/bin/swiftgen
Not writing the file as content is unchanged
Not writing the file as content is unchanged
Not writing the file as content is unchanged
Vendor/SwiftGen/bin/swiftgen  3.13s user 0.12s system 58% cpu 5.586 total

After:

$ time /Users/marcelofabri/SwiftGen/build/swiftgen/bin/swiftgen
Not writing the file as content is unchanged
Not writing the file as content is unchanged
Not writing the file as content is unchanged
/Users/marcelofabri/SwiftGen/build/swiftgen/bin/swiftgen  2.60s user 0.08s system 116% cpu 2.289 total

[djbe]

Implemented in SwiftGen 6.2.1 🎉