Cocoapods: Make `pod install` smarter about when it's not required

[keith]

🌈

Currently on our team, our developer's workflows look something like this:

1. Work on a feature
2. Submit a PR
3. Switch branches to work on something else
4. Update development submodules
5. bundle exec pod install
6. Repeat

Steps 4 & 5 are normally required because:

• Files can be added or removed at anytime from our development submodules, required a new pods project generation.
• We can update pods without everyone noticing, requiring a new pods project generation.
• It's annoying to start compiling the project just to find out that they need to sync pods.

This causes a few issues for us that it feels like might be solvable within CocoaPods.

• A no-op bundle exec pod install (without --repo-update) often takes 30 seconds on our project
• A pod install triggers an entire recompile on our app

While we can solve the second issue, it seems like it might be worth solving both of these for everyone in CocoaPods if possible.

We've solved this on our project by creating a script (which runs in about a second) that attempts to decide if a pod install is actually necessary. Currently for our use case it works like this:

1. If Pods doesn't exist, install
2. If Podfile.lock and Pods/Manifest.lock differ, install
3. Since we use development pods, the most complex step here is to determine if the Pods.xcodeproj requires a regen for added or removed files. To do this we compare what's currently in the project with what's on the file system using a small ruby script and xcodeproj. This step is a lot of work since we have to duplicate a lot of CocoaPods logic (like exclude_files in our own script).

I'm sure there are other cases here we are missing for different configurations, but this seems to cover our case pretty well.

So questions from this are:

1. Is this feasible to solve for all cases in CocoaPods?
2. If it's not feasible for _all_ cases, would it make sense to add it behind a command line flag (or other command a la bundle check) of some sort for most cases?
3. Is this something CocoaPods should handle in general?

Thanks for reading!

[endocrimes]

Yep, this is definitely something that would be nice to improve, and some of the changes required for https://github.com/CocoaPods/CocoaPods/issues/6310 would help towards this I think.

The best way to handle this would be some form of pod check a la bundler, but this is difficult to get right, and would require a more detailed plan and RFC to be implemented. https://github.com/square/cocoapods-check exists, but it doesn't work too well with recent CocoaPods versions.

[dnkoutso]

+1 this should be smarter. Maaaybe I can spend sometime on this but no promises at this time. :)

[dnkoutso]

CocoaPods 1.3.1 did a bunch of improvements for the recompile and pod install times. They are still kinda high for large projects. I have a few improvements to upstream to make pod install even faster.

[dnkoutso]

Aiming to try incremental pod install for 1.7.0

[segiddins]

@keith you mentioned in your initial issue you had a script that attempts to diff the FS with the pods project. Any chance you can share that?

[keith]

Yea, so first, to check the easy cases, we ran this script:

#!/bin/bash

if [ ! -d Pods ]; then
  echo "Pod installing because no Pods directory was found"
  exit 1
elif ! diff Podfile.lock Pods/Manifest.lock > /dev/null; then
  echo "Pod installing because Pods have changed"
  exit 1
fi

bundle exec ruby tools/check_development_pods.rb
case $? in
  2)
    exit 2
    ;;
  1)
    echo "Pod installing because development pods added/removed files"
    exit 1
    ;;
  *)
    echo "Pods have not changed"
    exit 0
    ;;
esac

The output of this script being 1 meant we installed (that was managed a level above this) and exiting with 0 is up to date.

The complicated part for development pods was handled by this ruby script:

require "find"
require "xcodeproj"

PROJECT_PATH = "Pods/Pods.xcodeproj"
VALID_SOURCE_EXTENSIONS = [
  ".m",
  ".swift",
]
RESOURCE_EXTENSIONS = [
  ".bin",
  ".lproj",
  ".storyboard",
  ".xcassets",
  ".xib",
]
RESOURCE_TARGETS = [
  "Foo-FooResources",
]
DEVELOPMENT_TARGETS = [
  "Foo",
]
PATH_FOR_TARGET = {
  "Foo" => "Modules",
}
EXCLUSIONS_BY_TARGET = {
  "Foo-watchOS" => [
    "iOSOnlyFile.swift",
  ]
}

def is_resource_bundle_target?(target)
  RESOURCE_TARGETS.any? { |name| target.display_name == name }
end

def is_development_target?(target)
  DEVELOPMENT_TARGETS.any? { |name| target.display_name.start_with?(name) }
end

def path_for_target(target, subdirectory)
  target_name = target.display_name.split("-").first
  root_path = PATH_FOR_TARGET[target_name]
  File.join(root_path, target_name, subdirectory)
end

def local_source_files_for_target(target)
  Find.find(path_for_target(target, "Sources"))
    .select { |path| File.file?(path) }
    .map { |file| File.basename(file) }
    .select { |file| VALID_SOURCE_EXTENSIONS.include? File.extname(file) }
end

def local_header_files_for_target(target)
  Find.find(path_for_target(target, "Sources"))
    .select { |path| File.file?(path) }
    .map { |file| File.basename(file) }
    .select { |file| File.extname(file) == ".h" }
end

def local_resource_files_for_target(target)
  path = path_for_target(target, "Resources")
  if Dir.exist? path
    Find.find(path)
      .map { |file| File.basename(file) }
      .select { |file| RESOURCE_EXTENSIONS.include? File.extname(file) }
  else
    []
  end
end

def compiled_files_for_target(target)
  target
    .source_build_phase
    .files_references
    .map(&:display_name)
end

def header_files_for_target(target)
  target
    .headers_build_phase
    .files_references
    .map(&:display_name)
    .reject { |file| file.end_with?("-umbrella.h") }
end

def xcode_resource_files_for_target(target)
  target
    .resources_build_phase
    .files_references
    .map(&:display_name)
end

def changed_sources_for_target(target)
  exclusions = EXCLUSIONS_BY_TARGET.fetch(target.display_name, [])
  local_files = local_source_files_for_target(target) - exclusions
  dummy_files = ["#{ target.display_name }-dummy.m"]
  compiled_files = compiled_files_for_target(target) - exclusions - dummy_files

  local_files - compiled_files \
    | compiled_files - local_files
end

def changed_headers_for_target(target)
  exclusions = EXCLUSIONS_BY_TARGET.fetch(target.display_name, [])
  local_files = local_header_files_for_target(target) - exclusions
  xcode_headers = header_files_for_target(target)

  local_files - xcode_headers \
    | xcode_headers - local_files
end

def changed_resources_for_target(target)
  exclusions = EXCLUSIONS_BY_TARGET.fetch(target.display_name, [])
  local_resources = local_resource_files_for_target(target) - exclusions
  xcode_resources = xcode_resource_files_for_target(target)

  local_resources - xcode_resources \
    | xcode_resources - local_resources
end

def find_missing_files
  Xcodeproj::Project.open(PROJECT_PATH)
    .targets
    .select { |target| is_development_target?(target) }
    .each do |target|
      if is_resource_bundle_target?(target)
        changed_files = changed_resources_for_target(target)
      else
        changed_files = changed_sources_for_target(target) \
          | changed_headers_for_target(target)
      end

      if changed_files.count > 0
        puts "Missing #{ changed_files } in #{ target.display_name }"
        exit 1
      end
    end
end

begin
  find_missing_files
rescue => error
  puts "Development pods check: #{ error }, please notify #client-productivity"
  exit 2
end

This is definitely a little specific to our project. All sources for our targets are nested in a single source path, in this case Modules/Foo/Sources. And we only care about a few file extensions when we're checking the project vs what is on disk, we would've added more there if we had more. Also we're kinda duplicating the exclude_files attribute from the podspec with the EXCLUSIONS_BY_TARGET for watchOS targets. The biggest downside was separating out the target names explicitly. I'm sure you can find a better way to do a lot of this, but this worked well for a while!

[segiddins]

Thanks for sharing! I’m very interested in building this into CocoaPods as soon as I get time to devote to CocoaPods work

[dnkoutso]

@keith closing this in favor of https://github.com/CocoaPods/CocoaPods/issues/8253. We are actively working on this to be built-into cocoapods.

Would love your feedback as well!