Homebrew-core: icu4c: now on git, but need lfs

[srl295]

Please note we will close your issue without comment if you delete, do not read or do not fill out the issue checklist below and provide ALL the requested information. If you repeatedly fail to use the issue template, we will block you from ever submitting issues to Homebrew again.

• [X] are reporting a bug others will be able to reproduce and not asking a question. If you're not sure or want to ask a question do so on our Discourse: https://discourse.brew.sh
• [ ] have a problem with brew install (or upgrade, reinstall) a single, official formula (not cask)? If it's a general brew problem please file this issue at Homebrew/brew: https://github.com/Homebrew/brew/issues/new/choose. If it's a brew cask problem please file this issue at https://github.com/Homebrew/homebrew-cask/issues/new/choose. If it's a tap (e.g. Homebrew/homebrew-php) problem please file this issue at the tap.
• [X] ran brew update and can still reproduce the problem?
• [X] ran brew doctor, fixed all issues and can still reproduce the problem?
• [X] ran brew gist-logs <formula> (where <formula> is the name of the formula that failed) and included the output link?
• [ ] if brew gist-logs didn't work: ran brew config and brew doctor and included their output with your issue?

ICU is now on GitHub

diff --git a/Formula/icu4c.rb b/Formula/icu4c.rb
index b97ad5cca..fc3aed2c0 100644
--- a/Formula/icu4c.rb
+++ b/Formula/icu4c.rb
@@ -5,7 +5,7 @@ class Icu4c < Formula
   mirror "https://downloads.sourceforge.net/project/icu/ICU4C/62.1/icu4c-62_1-src.tgz"
   version "62.1"
   sha256 "3dd9868d666350dda66a6e305eecde9d479fb70b30d5b55d78a1deffb97d5aa3"
-  head "https://ssl.icu-project.org/repos/icu/trunk/icu4c/", :using => :svn
+  head "https://github.com/unicode-org/icu.git"

   bottle do
     cellar :any

… but, though I have LFS installed, brew upgrade can't find it. LFS is needed for an ICU clone.

$ brew upgrade icu4c --HEAD
==> Upgrading 1 outdated package, with result:
icu4c 60.2 -> HEAD
==> Upgrading icu4c
==> Cloning https://github.com/unicode-org/icu.git
Updating /Users/srl/Library/Caches/Homebrew/icu4c--git
==> Checking out branch master
git-lfs filter-process: git-lfs: command not found
fatal: The remote end hung up unexpectedly
Error: Failed to download resource "icu4c"
Failure while executing: git checkout -f master --
bash: : command not found

[chrmoritz]

head do
  url "https://github.com/unicode-org/icu.git"
  depends_on "git-lfs" => :build
end

You need to declare git-lfs as a build dependency (otherwise it gets filtered from the PATH).

But the build will still fail, because the cd "source" do needs to be cd "icu4c/source" do for HEAD builds only.

[srl295]

@chrmoritz thanks. I didn't see how to do that.

Yeah, the cd icu4c/source can be condintionalized on head? I think.

[ilovezfs]

I'm a little confused. The clone seems to work fine without git-lfs.

[ilovezfs]

@srl295 ping

[srl295]

The clone seems to work fine without git-lfs.

When i tried it, the git clone from homebrew failed if git-lfs was not installed. I wondered if it was a PATH thing (/usr/bin/git vs /usr/local/bin/git)

Now, the icu4c build doesn't need any of the LFS content. If there's a way to skip LFS for the build from head, that would be fine also.

[apjanke]

The clone breaks for me without LFS on my pretty vanilla 10.13.6 Homebrew setup. You sure you don't happen to have git-lfs installed, ILZ?

$ brew install -s --HEAD icu4c
==> Cloning https://github.com/unicode-org/icu.git
Cloning into '/Users/janke/Library/Caches/Homebrew/icu4c--git'...
remote: Counting objects: 8027, done.
remote: Compressing objects: 100% (6002/6002), done.
remote: Total 8027 (delta 2740), reused 3757 (delta 1865), pack-reused 0
Receiving objects: 100% (8027/8027), 37.62 MiB | 7.04 MiB/s, done.
Resolving deltas: 100% (2740/2740), done.
git: 'lfs' is not a git command. See 'git --help'.

The most similar command is
    log
error: external filter 'git lfs smudge %f' failed 1
error: external filter 'git lfs smudge %f' failed
fatal: icu4j/main/shared/data/icudata.jar: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'

Error: Failed to download resource "icu4c"
Failure while executing; `git clone --depth 1 --branch master https://github.com/unicode-org/icu.git /Users/janke/Library/Caches/Homebrew/icu4c--git` exited with 128.
[~]
$

[ilovezfs]

yup.

[apjanke]

Got me, then.

Here's what I've got:

$ brew config
HOMEBREW_VERSION: 1.7.0-91-gdb29903
ORIGIN: https://github.com/Homebrew/brew
HEAD: db299035ac8d2b3024d4f32f27761fba5066973f
Last commit: 8 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 47d170afb91a448297dc44331736b3827d0cd8c1
Core tap last commit: 13 hours ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_DEVELOPER: 1
HOMEBREW_NO_AUTO_UPDATE: 1
HOMEBREW_SANDBOX: 1
HOMEBREW_VISUAL: subl -w
CPU: quad-core 64-bit kabylake
Homebrew Ruby: 2.3.7 => /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/bin/ruby
Clang: 9.1 build 902
Git: 2.18.0 => /usr/local/bin/git
Curl: 7.54.0 => /usr/bin/curl
Java: 1.8.0_144
macOS: 10.13.6-x86_64
CLT: 9.4.1.0.1.1528165917
Xcode: 9.4 => /Applications/Xcode-9.4.app/Contents/Developer
XQuartz: 2.7.11 => /opt/X11

[apjanke]

Here's a PR for discussion and testing: https://github.com/Homebrew/homebrew-core/pull/30379

[chrmoritz]

@ilovezfs: Are you using homebrew git or Apple git? I have the same issue without lfs using homebrew git.

[ilovezfs]

It works fine with both. The issue only occurs if you have ever set up git-lfs in your .gitconfig, which has the stanza

[filter "lfs"]
    clean = git-lfs clean -- %f
    smudge = git-lfs smudge -- %f
    process = git-lfs filter-process
    required = true

if you have.

[apjanke]

Oh, yep. I have that in mine.

$ cat ~/.gitconfig | grep lfs                                                                                                                       master
[filter "lfs"]
    clean = git lfs clean %f
    smudge = git lfs smudge %f

So what should we do here to isolate users from that variability? Always require git-lfs? Bypass ~/.gitconfig?

[ilovezfs]

Possibly define a download strategy that sets GIT_LFS_SKIP_SMUDGE=1 since @srl295 says the files aren't needed or we could just remove the head spec.

[srl295]

@ilovezfs yeah , it's _currently_ not needed for icu4c. it's possible but not likely that the direction would change in the future. What's more likely is that for icu4j (java) we will eventually remove the jars from master which could theoretically remove the need for the lfs smudge ICU-12968

Perhaps for now it's better to just enable lfs if building from head if there's no such download strategy (i don't know what that entails).

[ilovezfs]

I've removed the head spec in https://github.com/Homebrew/homebrew-core/pull/30427.

[srl295]

@ilovezfs why? What was a wrong with https://github.com/Homebrew/homebrew-core/issues/30190#issuecomment-405732975

[ilovezfs]

@srl295 because the git fetching behavior would vary based on the user's configuration or lack of configuration of git-lfs outside of Homebrew, and formula behavior shouldn't vary like that.

[srl295]

@ilovezfs ? even if i have git-lfs uninstalled and not mentioned in ~/.gitconfig I get:

$ brew upgrade --HEAD icu4c
==> Upgrading 1 outdated package, with result:
icu4c 60.2 -> HEAD
==> Upgrading icu4c
==> Installing dependencies for icu4c: git-lfs
==> Installing icu4c dependency: git-lfs
==> Downloading https://homebrew.bintray.com/bottles/git-lfs-2.4.2.high_sierra.bottle.tar.gz
Already downloaded: /Users/srl/Library/Caches/Homebrew/git-lfs-2.4.2.high_sierra.bottle.tar.gz
==> Pouring git-lfs-2.4.2.high_sierra.bottle.tar.gz
==> Caveats
Update your git config to finish installation:

  # Update global git config
  $ git lfs install

  # Update system git config
  $ git lfs install --system
==> Summary
🍺  /usr/local/Cellar/git-lfs/2.4.2: 65 files, 8.4MB
==> Installing icu4c --HEAD
==> Cloning https://github.com/unicode-org/icu.git
Updating /Users/srl/Library/Caches/Homebrew/icu4c--git
==> Checking out branch master
Checking out files: 100% (8550/8550), done.
Filtering content:  63% (148/234), 14.48 MiB | 1.24 MiB/s

this is with

diff --git a/Formula/icu4c.rb b/Formula/icu4c.rb
index 7ac202390..2d1f3f687 100644
--- a/Formula/icu4c.rb
+++ b/Formula/icu4c.rb
@@ -5,6 +5,10 @@ class Icu4c < Formula
   mirror "https://downloads.sourceforge.net/project/icu/ICU4C/62.1/icu4c-62_1-src.tgz"
   version "62.1"
   sha256 "3dd9868d666350dda66a6e305eecde9d479fb70b30d5b55d78a1deffb97d5aa3"
+  head do
+    url "https://github.com/unicode-org/icu.git"
+    depends_on "git-lfs" => :build
+  end

   bottle do
     cellar :any
@@ -19,6 +23,7 @@ class Icu4c < Formula
     args = %W[--prefix=#{prefix} --disable-samples --disable-tests --enable-static]
     args << "--with-library-bits=64" if MacOS.prefer_64_bit?

+    cd "icu4c" if head?
     cd "source" do
       system "./configure", *args
       system "make"

with the above patch, I get a complete download (including binary blobs) in all situations.

[srl295]

If the depends_on "git-lfs" is not there, then YES you will have different behavior (successful but missing .jars, or broken) depending on your environment. but - unless i'm missing something - the above patch makes --HEAD work

[chrmoritz]

because the git fetching behavior would vary based on the user's configuration or lack of configuration of git-lfs outside of Homebrew, and formula behavior shouldn't vary like that.

The git fetching behavior would vary only inside the icu4j (java) folder and not inside the icu4c folder as far as I understand it. That's why it shouldn't affect formula behavior.

Whats wrong with using a download strategy, which sets the GIT_LFS_SKIP_SMUDGE=1 env? icu4c --HEAD had 95 install events in the last 90 days (# 3744) according to analytics, which is more than 1 per day and more the most other HEAD specs.

[ilovezfs]

Whats wrong with using a download strategy, which sets the GIT_LFS_SKIP_SMUDGE=1 env

Nothing if someone wants to do a PR for that.

unless i'm missing something

@srl295 the output shows

==> Cloning https://github.com/unicode-org/icu.git
Updating /Users/srl/Library/Caches/Homebrew/icu4c--git

so that's not testing a fresh clone.

[ilovezfs]

With your diff and a fresh clone @srl295

iMac-TMP:homebrew-core joe$ brew instal --HEAD -dv icu4c
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/icu4c.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/git-lfs.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/go.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/ruby.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/pkg-config.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/libyaml.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/autoconf.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/automake.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/libtool.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/openssl.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/makedepend.rb
/usr/local/Homebrew/Library/Homebrew/brew.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/readline.rb
/usr/bin/sandbox-exec -f /private/tmp/homebrew20180725-27450-y41n9m.sb nice /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/bin/ruby -W0 -I /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/gems/2.3.0/gems/did_you_mean-1.0.0/lib:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/site_ruby/2.3.0:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/site_ruby/2.3.0/x86_64-darwin9.0:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/site_ruby/2.3.0/universal-darwin9.0:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/site_ruby:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/vendor_ruby/2.3.0:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/vendor_ruby/2.3.0/x86_64-darwin9.0:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/vendor_ruby/2.3.0/universal-darwin9.0:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/vendor_ruby:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/2.3.0:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/2.3.0/x86_64-darwin9.0:/usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.3.7/lib/ruby/2.3.0/universal-darwin9.0:/usr/local/Homebrew/Library/Homebrew/cask/lib:/usr/local/Homebrew/Library/Homebrew -- /usr/local/Homebrew/Library/Homebrew/build.rb /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/icu4c.rb --verbose --debug --HEAD
/usr/local/Homebrew/Library/Homebrew/build.rb (Formulary::FromPathLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/icu4c.rb
/usr/local/Homebrew/Library/Homebrew/build.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/git-lfs.rb
/usr/local/Homebrew/Library/Homebrew/build.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/go.rb
/usr/local/Homebrew/Library/Homebrew/build.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/ruby.rb
/usr/local/Homebrew/Library/Homebrew/build.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/pkg-config.rb
/usr/local/Homebrew/Library/Homebrew/build.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/libyaml.rb
/usr/local/Homebrew/Library/Homebrew/build.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/autoconf.rb
/usr/local/Homebrew/Library/Homebrew/build.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/automake.rb
/usr/local/Homebrew/Library/Homebrew/build.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/libtool.rb
/usr/local/Homebrew/Library/Homebrew/build.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/openssl.rb
/usr/local/Homebrew/Library/Homebrew/build.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/makedepend.rb
/usr/local/Homebrew/Library/Homebrew/build.rb (Formulary::FormulaLoader): loading /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/readline.rb
==> Cloning https://github.com/unicode-org/icu.git
Updating /Users/joe/Library/Caches/Homebrew/icu4c--git
git config remote.origin.url https://github.com/unicode-org/icu.git
git config remote.origin.fetch +refs/heads/master:refs/remotes/origin/master
git fetch origin
remote: Counting objects: 7, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 7 (delta 6), reused 6 (delta 5), pack-reused 0
Unpacking objects: 100% (7/7), done.
From https://github.com/unicode-org/icu
   472791d..76168da  master     -> origin/master
==> Checking out branch master
git checkout -f master --
Checking out files: 100% (8550/8550), done.
Filtering content: 100% (475/475), 20.06 MiB | 1.67 MiB/s, done.
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
  (use "git pull" to update your local branch)
git reset --hard origin/master
HEAD is now at 76168da ICU-10464 Make installation of icu-config optional (#13)

No binary blobs fetched.

[srl295]

@ilovezfs

so that's not testing a fresh clone

it was fresh, just not in this example.

Filtering content: 100% (475/475), 20.06 MiB | 1.67 MiB/s, done.
no binary blobs fetched

But you have "filtering content" so it should have downloaded 20MiB

So then: find /Users/srl/Library/Caches/Homebrew/icu4c--git -name '*.jar' -ls

you should see jars with substantial sizes

12508534 Jul 25 07:52 /Users/srl/Library/Caches/Homebrew/icu4c--git/icu4j/main/shared/data/icudata.jar

that's what's expected.

[ilovezfs]

Those are fetched regardless of the git-lfs dependency.

Josephs-MacBook-Pro:icu joe$ find . -name '*.jar' -ls
243915085        8 -rwxr-xr-x    1 joe              staff                 133 Jul 25 09:13 ./icu4j/main/shared/data/icudata.jar
243915086        8 -rwxr-xr-x    1 joe              staff                 130 Jul 25 09:13 ./icu4j/main/shared/data/icutzdata.jar
243915088        8 -rwxr-xr-x    1 joe              staff                 131 Jul 25 09:13 ./icu4j/main/shared/data/testdata.jar
243916603        8 -rw-r--r--    1 joe              staff                 131 Jul 25 09:13 ./tools/release/java/lib/serializer.jar
243916604        8 -rw-r--r--    1 joe              staff                 132 Jul 25 09:13 ./tools/release/java/lib/xalan.jar
243916605        8 -rw-r--r--    1 joe              staff                 132 Jul 25 09:13 ./tools/release/java/lib/xercesImpl.jar
Josephs-MacBook-Pro:icu joe$ 

When git-lfs is actually configured, then there is additional fetching.

[ilovezfs]

And note no filtering step now that I removed the lfs bits from /usr/local/etc/gitconfig

==> Cloning https://github.com/unicode-org/icu.git
git clone --depth 1 --branch master https://github.com/unicode-org/icu.git /Users/joe/Library/Caches/Homebrew/icu4c--git
Cloning into '/Users/joe/Library/Caches/Homebrew/icu4c--git'...
remote: Counting objects: 8027, done.
remote: Compressing objects: 100% (4878/4878), done.
remote: Total 8027 (delta 3161), reused 6698 (delta 2989), pack-reused 0
Receiving objects: 100% (8027/8027), 37.25 MiB | 7.85 MiB/s, done.
Resolving deltas: 100% (3161/3161), done.
Checking out files: 100% (8550/8550), done.
git config homebrew.cacheversion 0
==> Checking out branch master
git checkout -f master --
Already on 'master'
Your branch is up to date with 'origin/master'.

[ilovezfs]

And even then

bash-3.2$ find /Users/joe/Library/Caches/Homebrew/icu4c--git -name '*.jar' -ls
773541936        8 -rwxr-xr-x    1 joe              staff                 133 Jul 25 09:22 /Users/joe/Library/Caches/Homebrew/icu4c--git/icu4j/main/shared/data/icudata.jar
773541937        8 -rwxr-xr-x    1 joe              staff                 130 Jul 25 09:22 /Users/joe/Library/Caches/Homebrew/icu4c--git/icu4j/main/shared/data/icutzdata.jar
773541939        8 -rwxr-xr-x    1 joe              staff                 131 Jul 25 09:22 /Users/joe/Library/Caches/Homebrew/icu4c--git/icu4j/main/shared/data/testdata.jar
773543452        8 -rw-r--r--    1 joe              staff                 131 Jul 25 09:22 /Users/joe/Library/Caches/Homebrew/icu4c--git/tools/release/java/lib/serializer.jar
773543453        8 -rw-r--r--    1 joe              staff                 132 Jul 25 09:22 /Users/joe/Library/Caches/Homebrew/icu4c--git/tools/release/java/lib/xalan.jar
773543454        8 -rw-r--r--    1 joe              staff                 132 Jul 25 09:22 /Users/joe/Library/Caches/Homebrew/icu4c--git/tools/release/java/lib/xercesImpl.jar
bash-3.2$ 

[srl295]

i'll look at the skip smudge thing

[srl295]

OK, if icudata.jar is 133 bytes, then it didn't update the blob.

[ilovezfs]

Are we having fun yet?