Cocoapods: Size of ~/.cocoapods/repos exceeds 1gb

[chrismanderson]

The size of my ~/.cocoapods/repos folder is over 1gb (1.3gb to be specific). This seems much too large. Can I remove this folder? Or how can I shrink it down? Trying to maximize every iota of space on a MacBook with minimal storage.

[dnkoutso]

@chrismanderson it's because of the specs master repo most likely (https://github.com/CocoaPods/Specs). I do not think anything can be done at this moment to reduce its size but I do understand it is becoming a problem.

[chrismanderson]

Yeah, I know folder came about from http://blog.cocoapods.org/Master-Spec-Repo-Rate-Limiting-Post-Mortem/, but still, I hope there is something that can be done about it in the future. Just too big a folder to require people to keep.

[orta]

We've taken a few attacks that this, but realistically it's a hard problem that no-one has volunteered time to fix and maintain. If this is something you're interested at fixing, there's a few ideas:

• Create a smaller binary version of the specs repo that stays local
• Move trunk specs resolving to a server, similar to how bunder works
• Regularly squash the specs repo

[amorde]

What if two separate repositories existed, one which pod trunk etc. interacted with and retained the full history, and one that is kept in sync but routinely squashed?

It seems like the main issue is that the history of who/when adds specs is useful and worth keeping, but every user of CocoaPods does not need to be downloading the entire history and keeping a copy on their machine.

[orta]

That's what my repo tried to do, but it turns out you don't really gain much from the process alas - you're welcome to give it a fresh perspective though. I could have missed something.

[amorde]

Just tried checking out the Specs repo and removing the .git folder, and it while there is a significant reduction it's most likely not worth the effort.

192,884,971 bytes (773.4 MB on disk) for 404,973 items

Attempting to even create a fresh repo takes quite a long time and increases the size dramatically.

[davidair]

What if (and I know this is probably a crazy idea) the CocoaPods repo was split into multiple repositories based on some spec partition that would optimize for most common cases? Then the pod installation/update could conceivably function like that:

• Sync the smaller "current" repo first, attempt to resolve all dependencies. If it worked, done.
• Keep syncing additional "archival" repos until the dependencies are fully resolved.

Of course, these would have to be all brand-new repositories and pod trunk push would probably have to publish to both the monolithic repo and the new partitioned ones. Managing the relative sizes of the partitioned repos would also become a continuous maintenance task.

[chrismanderson]

What’s the biggest obstacle in implementing a Bundler-type system? The dev work involved, or finding a host/sponsor/etc for server costs?

[orta]

I guess dev work to integrate and having admins around to maintain the server infra long term. Server costs are _probably_ fine with our existing relationships - but that service would be mission critical.

The worrying part for me is that every service we've made so far does not affect someone using CocoaPods, you could always pod install if trunk was down for example, an online resolver requires a different level of commitment.

[stale[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.

[dnkoutso]

Stay.

[shawnce]

Yeah the size of this is getting kinda a problem especially in CI setups, etc. Does anyway exist to have pod evaluate the Podfile and generate local spec repo of just the things you currently need? We could then cache that for CI, etc. and only modify it when the Podfile changes?

[0:524] (master)> du -hd 1
889M    ./Specs
584M    ./.git
1.4G    .

[orta]

With that you're back to building a web service for resolving which brings it back to https://github.com/CocoaPods/CocoaPods/issues/7046#issuecomment-333401676

Which someone can definitely do, but it's a serious commitment

[CrazyPython]

Homebrew doesn't use a centralized server and it doesn't use a lot of space for its build details- perhaps cocoapods could do what homebrew does?

[amorde]

@CrazyPython actually Homebrew basically does the same exact thing that CocoaPods does. The reason it doesn't require as much space is that the number of specs is significantly fewer than the number of Podspecs.

[gautamkrishnar]

@orta perhaps you guys can apply for openshift open source sponsorship: https://blog.openshift.com/openshift-for-non-profits-open-source-and-educational-institutions/ if you are planning to maintain a server for cocoapods specs.

[stale[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.

[ghost]

It's interesting that @orta considers this as an attack and not constructive critisism.

He shut me down here and suggested I was too mean. Maybe I could have re-phrased it slightly but still man...get a thicker skin and man up.

Edit you have pissed me off to the max.

It's like telling an oil company "you used to be a helpful addition to the world, but now there seems to be a few problems selling oil" and what do they do. SHUT YOU DOWN.

Cocoapods is everywhere. It's in my face when I start a new job and I'm the only dev on the team with no documentation and spend the first week onboarding getting the damn thing to compile. It's in almost every open source repo I download and most of them NEVER WORK FIRST TIME.

I have the right to speak up about it !

[orta]

@rob-nash please be more considerate of others feelings on our issues. Constructive criticism is awesome and useful when focused. You definitely have the right provide feedback, that's why we're not deleting or editing your posts - but please stay on topic.

[ghost]

@orta I understand. I can’t open a thread here to discuss the issue and I suspect it has a lot to do with the fact that this is the public domain. Is there a way I can raise this issue internally, so that only those in control of the fate of cocoapods can see and comment on this issue? The issue being quite simply that cocoapods, in my opinion, is getting in the way of creative iOS development. I just want to smash some UI together man. I have spent hundreds of hours repairing podfiles. And I no longer want to download a GIGA 😬 of god knows, to run a simple view controller, with a table view.

[amorde]

The issue being quite simply that cocoapods, in my opinion, is getting in the way of creative iOS development. I just want to smash some UI together man. I have spent hundreds of hours repairing podfiles.

@rob-nash if you have specific examples of things that CocoaPods could improve on, please do submit an issue with a suggestion for an improvement. But saying that CocoaPods is getting in your way doesn't really help us make it better unless you give some specifics :)

[amorde]

Re: 1GB spec repo, the CocoaPods devs also want to fix this. Some possible solutions have been noted here, including possibly making a service similar to RubyGems. Some others suggested by users in this tread:

• Try squashing commits (this appears to not make enough of a difference, unfortunately)
• Shard the spec repo into multiple repos

A web service similar to RubyGems is likely the best option, but it brings with it some concerns:

• Our ability to maintain another web server over time
• If the service goes down, this may break many teams' workflows
• Potential costs of hosting. This may or may not be a concern for CocoaPods with the resources available at this time

[amorde]

Other ways you can mitigate the impact of this to your project right now:

• Create a custom spec repo with just the specs you need, and use that repo in your Podfile
• Commit the Pods/ directory to source control so that pod install isn't required at all when building on CI environments
• Use the :git or :podspec options when declaring pod dependencies in the Podfile to manually load Pods

None of these options are ideal but they can help alleviate some of the problems that the big specs repo causes until we have a solution in place.

[ghost]

Regarding your comment here @amorde i have done. but the issue was closed.

8001

[ashanbrown]

I've been attempting to accelerate the cocoapods download on circleci by caching a shallow copy of the master spec repo. However, when I run pod install --no-repo-update it tries to updates the repo anyway. Here's my log:

> pod install --no-repo-update
....
Updating spec repo `master`
  Performing a deep fetch of the `master` specs repo to improve future performance
    $ /usr/local/bin/git -C /Users/distiller/.cocoapods/repos/master fetch --unshallow
  $ /usr/local/bin/git -C /Users/distiller/.cocoapods/repos/master fetch origin --progress
  $ /usr/local/bin/git -C /Users/distiller/.cocoapods/repos/master rev-parse --abbrev-ref HEAD
  master
  $ /usr/local/bin/git -C /Users/distiller/.cocoapods/repos/master reset --hard origin/master
  fatal: ambiguous argument 'origin/master': unknown revision or path not in the working tree.
  Use '--' to separate paths from revisions, like this:
  'git <command> [<revision>...] -- [<file>...]'
[!] Unable to add a source with url `https://github.com/CocoaPods/Specs.git` named `master-1`.
(CocoaPods was not able to update the `master` repo. If this is an unexpected issue and persists you can inspect it running `pod repo update --verbose`)
You can try adding it manually in `~/.cocoapods/repos` or via `pod repo add`.

/Library/Ruby/Gems/2.3.0/gems/cocoapods-1.5.3/lib/cocoapods/sources_manager.rb:35:in `rescue in find_or_create_source_with_url'
/Library/Ruby/Gems/2.3.0/gems/cocoapods-1.5.3/lib/cocoapods/sources_manager.rb:38:in `find_or_create_source_with_url'
/Library/Ruby/Gems/2.3.0/gems/cocoapods-1.5.3/lib/cocoapods/installer/analyzer.rb:873:in `block in sources'
/Library/Ruby/Gems/2.3.0/gems/cocoapods-1.5.3/lib/cocoapods/installer/analyzer.rb:872:in `map'
/Library/Ruby/Gems/2.3.0/gems/cocoapods-1.5.3/lib/cocoapods/installer/analyzer.rb:872:in `sources'
/Library/Ruby/Gems/2.3.0/gems/cocoapods-1.5.3/lib/cocoapods/installer/analyzer.rb:780:in `block in resolve_dependencies'
/Library/Ruby/Gems/2.3.0/gems/cocoapods-1.5.3/lib/cocoapods/user_interface.rb:64:in `section'
/Library/Ruby/Gems/2.3.0/gems/cocoapods-1.5.3/lib/cocoapods/installer/analyzer.rb:779:in `resolve_dependencies'
/Library/Ruby/Gems/2.3.0/gems/cocoapods-1.5.3/lib/cocoapods/installer/analyzer.rb:88:in `analyze'
/Library/Ruby/Gems/2.3.0/gems/cocoapods-1.5.3/lib/cocoapods/installer.rb:243:in `analyze'
/Library/Ruby/Gems/2.3.0/gems/cocoapods-1.5.3/lib/cocoapods/installer.rb:154:in `block in resolve_dependencies'
/Library/Ruby/Gems/2.3.0/gems/cocoapods-1.5.3/lib/cocoapods/user_interface.rb:64:in `section'
/Library/Ruby/Gems/2.3.0/gems/cocoapods-1.5.3/lib/cocoapods/installer.rb:153:in `resolve_dependencies'
/Library/Ruby/Gems/2.3.0/gems/cocoapods-1.5.3/lib/cocoapods/installer.rb:116:in `install!'
/Library/Ruby/Gems/2.3.0/gems/cocoapods-1.5.3/lib/cocoapods/command/install.rb:41:in `run'
/Library/Ruby/Gems/2.3.0/gems/claide-1.0.2/lib/claide/command.rb:334:in `run'
/Library/Ruby/Gems/2.3.0/gems/cocoapods-1.5.3/lib/cocoapods/command.rb:52:in `run'
/Library/Ruby/Gems/2.3.0/gems/cocoapods-1.5.3/bin/pod:55:in `<top (required)>'
/usr/local/bin/pod:22:in `load'
/usr/local/bin/pod:22:in `<main>'
Exited with code 1

I see there has been a lot of discussion about shallow repos, but my goal is not to pull a shallow repo directly from github. I download the repo from s3 (about 45 seconds in circleci) and then shallow clone it to create the copy I will cache. Is not trying to update the master spec repo something you'd be willing to support? Thanks.

[CrazyPython]

What if JFrog Bintray was used? They have an open-source plan with 1TB/month, which might not be enough, but JFrog might be willing to negotiate due to the prominence CocoaPods holds.

Although BinTray is for application binaries, it could just as easily be for specs.

Their API has package creation, search, and update. I'm imagining the bot that commits to the specs repo could also create or update bintray at the same time.

[reitzig]

This continues to be a serious issue. Usability aside, always retrieving the entire history of all pods that ever were seems ridiculous. So far, I have not read a single argument for the current approach being necessary.

@amorde wrote:

A web service similar to RubyGems is likely the best option, but it brings with it some concerns:

* Our ability to maintain another web server over time

* If the service goes down, this may break many teams' workflows

* Potential costs of hosting. This may or may not be a concern for CocoaPods with the resources available at this time

You could have CI/CD create some form of static website and/or API from the spec repo and host that on Github pages. That way, you don't have to maintain separate infrastructure and the specs can be served efficiently, e.g. via CDNs or caches or whatever Github does to deliver those pages.

[keith]

Our team worked around this problem by hosting an internal specs repo that contained on the specs we consumed. We automatically updated it on a cron with a few simple scripts:

With the specs we need defined in a pods.txt file one per line (including all transitive pods):

...

We download the current specs tarball:

#!/bin/bash

set -euo pipefail

REPOS_LOCATION="/tmp/cocoapods"
MASTER_REPO_LOCATION="$REPOS_LOCATION/master"

tempfile=$(mktemp)

cleanup() {
  echo "Download failed, cleaning up and falling back to standard checkout..."
  rm -rf "$MASTER_REPO_LOCATION"
  rm "$tempfile"
  exit 1
}

trap cleanup ERR

rm -rf "$MASTER_REPO_LOCATION"
mkdir -p "$REPOS_LOCATION"

echo "Downloading CocoaPods master repo..."
curl -L https://github.com/CocoaPods/Specs/archive/master.tar.gz -o "$tempfile"

echo "Uncompressing CocoaPods master repo..."
# We expect the structure with the "master" as the top dir in the archive.
tar -C "$REPOS_LOCATION" -xzf "$tempfile"
mv "$REPOS_LOCATION/Specs-master" "$MASTER_REPO_LOCATION"

echo "Successfully downloaded CocoaPods master repo."
rm "$tempfile"

The we copied over the specs we needed into our own repo

#!/bin/bash

rm -rf Specs
mkdir Specs

while read -r pod
do
  spec_dir="$(find /tmp/cocoapods/master -type d -name "$pod")"
  if [ -n "$spec_dir" ]; then
    output_path="Specs/$pod"
    echo "Copying specs from '$spec_dir' to '$output_path'"
    cp -R "$spec_dir" "$output_path"
  else
    echo "No specs found for '$pod'"
  fi
done < "pods.txt"

And then we committed and pushed this, using only this source in our Podfile. This is a little bit of overhead but it helped us significantly. If you wanted to you could make this setup even smarter to only pull over specific versions of pods you cared about, decreasing the size even more.

[dnkoutso]

We did something very similar. We have a mirror of the open source dependencies that we use instead and never declare the source directive to the Podfile for the open source repo.

Faster repo update update times and installation, more secure since depending on open source dependencies they can change under you.

We have an internal tool that take any podspec zips it up and uploads its sources internally to be hosted and published the podspec to our mirror repo.

[at-daonguyen]

Can we move inactive pods out of master repo?
inactive means number of downloads per year is equal to zero.

[ccorcos]

Just curious, can someone explain why the Spec repo is necessary to have locally?

[segiddins]

It isn’t necessary, you do not need to use the public specs repo to use CocoaPods.

[ccorcos]

Hmm. Then why don't we just not clone it?

[amorde]

You need it if you want to access the public spec repo (the repository of all public pods) - you don't need it if you use private spec repos, or if you manually specify pods with directives like :git or :path

There's work being done to improve this - see #8268 and its accompanying PRs for details

[ccorcos]

I see. Well there's a good idea!

[ffffwh]

An idea for a quick ~fix~

• make a git branch, e.g. empty
  
  
  
  • in the branch, remove Specs dir
  
  
• checkout master before pod <operation>, and checkout empty after.
• make it an option to satisfy both who often pod and who seldom do it or want to save space.

Since git will pack & compress things not in the working tree, it saves space (currently by 1.2GB).

[igor-makarov]

PRs #8280 & Core#469 were merged into master on both repos.
To take advantage of CDN-based spec repo do the following:

• Use a Gemfile and get the latest cocoapods and cocoapods-core gems from master.
• On command line: bundle exec pod repo add-cdn jsDelivr-Specs "https://cdn.jsdelivr.net/cocoa/"
• In your Podfile: source "https://cdn.jsdelivr.net/cocoa/"
• If you already have a Podfile.lock replace the SPEC REPOS key there as well - there's a bug that prevents it from auto-updating.

[stale[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.

[igor-makarov]

CDN-based spec retrieval is in 1.7.0 beta.

Perhaps this issue can be closed?

[stale[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.

[stale[bot]]

This issue will be auto-closed because there hasn't been any activity for a few months. Feel free to open a new one if you still experience this problem :+1:

[strobox]

Use a Gemfile and get the latest cocoapods and cocoapods-core gems from master.
On command line: bundle exec pod repo add-cdn jsDelivr-Specs "https://cdn.jsdelivr.net/cocoa/"
In your Podfile: source "https://cdn.jsdelivr.net/cocoa/"`

@igor-makarov will it be possible to use that with react-native project, how do you think?

Also confused how to change this in Podfile.lock:

SPEC REPOS:
  https://github.com/cocoapods/specs.git:
    - boost-for-react-native

[igor-makarov]

The info in this thread is outdated, go here for official final info:

https://blog.cocoapods.org/CocoaPods-1.7.2/

[braysonjohn148]

[cocoapods] anybody with this ziped folder https://github.com/CocoaPods/Specs to provide us the link, taking long to clone from git. Please help, trying to download for month now.

[jgreen210]

I've not tried using it yet, but I see that cocopods have made the CDN approach the default (in place of github):

http://blog.cocoapods.org/CocoaPods-1.8.0-beta/

[braysonjohn148]

Please some explanations. I installed pods, after that i run pod setup it started cloning the file which never ended. So how do i do this

[FranklinYu]

@braysonjohn148 CocoaPods version? In addition, what’s your Podfile?