Conda-forge.github.io: Remove `*.la` files from ecosystem

[scopatz]

It turns out that the libtool created *.la files can cause extraneous overlinkage. This is where every shared object file gets directly linked to each of its dependencies, even if they are far upstream in the dependency chain and not actually a direct dependency.

As discussed in https://github.com/conda-forge/glib-feedstock/pull/31, conda-forge/libxcb-feedstock#9, and conda-forge/staged-recipes#673, the decision is to remove these files. However, this has been done in an incomplete and inconsistent way. We also have no way currently of preventing folks from adding these files to packages in the future.

The following is a listing of all packages which need to have the *.la files removed. Please check them off as the *.la files are removed. Additionally, we need to add docs to this repo explaining why these files are not OK. Finally, an option should be added to conda-build to prevent *.la files from being added to a package.

The command for removing the *.la files should be:

find $PREFIX -name '*.la' -delete

CC @jakirkham @ocefpaf @mingwandroid @pkgw

• [ ] adept
• [ ] alsa-lib
• [ ] argtable2
• [ ] backtrace
• [ ] bdw-gc
• [ ] binutils
• [ ] brial
• [x] cairo
• [ ] cairomm
• [ ] cddlib
• [x] check
• [ ] cliquer
• [ ] cloog
• [ ] coincbc
• [ ] coinmp
• [ ] cppunit
• [ ] cunit
• [x] curl
• [ ] cyrus-sasl
• [ ] czmq
• [ ] dbus
• [ ] eclib
• [ ] ecm
• [ ] ecmwf_grib
• [ ] exempi
• [x] expat
• [ ] fflas-ffpack
• [ ] fftw
• [x] flex
• [x] fontconfig
• [ ] fplll
• [ ] freetds
• [x] freetype
• [x] freexl
• [ ] fswatch
• [x] gdal
• [ ] gdb
• [ ] gdk-pixbuf
• [ ] geoip
• [x] geos
• [x] gettext
• [ ] gf2x
• [ ] giac
• [x] giflib
• [ ] givaro
• [x] glib
• [x] glibmm
• [ ] glog
• [ ] glpk
• [ ] gmp
• [ ] gnutls
• [ ] gobject-introspection
• [ ] gperftools
• [x] graphviz
• [x] gsl
• [ ] gst-plugins-base
• [ ] gstreamer
• [ ] gtk2
• [x] harfbuzz
• [ ] harminv
• [x] hdf4
• [x] hdf5
• [x] hdfeos2
• [x] hdfeos5
• [ ] hunspell
• [ ] igraph
• [x] imagemagick
• [ ] iml
• [ ] inkscape
• [ ] inotify-tools
• [ ] iperf
• [ ] ipopt
• [ ] isl
• [ ] jags
• [ ] jansson
• [x] jasper
• [x] jpeg
• [ ] jq
• [x] json-c
• [ ] judy
• [ ] lal
• [ ] lalframe
• [ ] lalmetaio
• [ ] lalsimulation
• [ ] lalxml
• [ ] leptonica
• [ ] lftp
• [x] libarchive
• [ ] libassuan
• [ ] libatomic_ops
• [ ] libblitz
• [ ] libbrial
• [ ] libcf
• [ ] libcmaes
• [ ] libconfig
• [ ] libctl
• [x] libdap4
• [ ] libedit
• [ ] libev
• [ ] libevent
• [x] libffi
• [ ] libflac
• [ ] libframe
• [ ] libgap
• [ ] libgcrypt
• [x] libgd
• [x] libgdal
• [ ] libglu
• [ ] libgpg-error
• [ ] libgsasl
• [ ] libhwloc
• [x] libiconv
• [ ] libidn11
• [x] libjpeg-turbo
• [ ] libksba
• [ ] libmad
• [ ] libmagic
• [ ] libmatio
• [ ] libmemcached
• [x] libmo_unpack
• [ ] libntlm
• [ ] libogg
• [x] libpng
• [ ] libprotobuf
• [ ] libsecret
• [ ] libsemigroups
• [x] libsigcpp
• [ ] libsigsegv
• [ ] libsndfile
• [ ] libsodium
• [x] libspatialindex
• [x] libspatialite
• [ ] libtasn1
• [ ] libtheora
• [x] libtiff
• [x] libtool
• [ ] libunistring
• [ ] libunwind
• [x] libuuid
• [ ] libuv
• [ ] libvorbis
• [ ] libwebp
• [x] libxcb
• [x] libxml2
• [x] libxmlpp
• [x] libxslt
• [ ] lighttpd
• [ ] linbox
• [ ] log4cpp
• [ ] lrcalc
• [ ] m4ri
• [ ] m4rie
• [ ] mesalib
• [ ] metaio
• [ ] moab
• [ ] modsecurity
• [ ] mono
• [ ] mpb
• [ ] mpc
• [ ] mpfi
• [ ] mpfr
• [ ] mpg123
• [ ] mpich
• [ ] mpir
• [x] nco
• [x] netcdf-cxx4
• [x] netcdf-fortran
• [ ] neuron
• [ ] nfft
• [ ] nlopt
• [ ] npth
• [ ] ntbtls
• [ ] ntl
• [ ] numcosmo
• [ ] ocl-icd
• [ ] octave
• [ ] oniguruma
• [ ] openfst
• [ ] openmotif-dev
• [ ] openmpi
• [ ] ossuuid
• [ ] pango
• [x] pcre
• [ ] pcre2
• [x] pixman
• [ ] planarity
• [ ] pocketsphinx
• [ ] popt
• [ ] postgis
• [ ] ppl
• [ ] primesieve
• [x] proj4
• [ ] protobuf
• [ ] pygobject
• [ ] pymeep
• [ ] pynac
• [ ] python-ecmwf_grib
• [ ] qd
• [ ] qpdf
• [ ] qt
• [ ] qtlocation
• [ ] qtwebkit
• [ ] rw
• [ ] sdl2
• [ ] sdl2_gfx
• [ ] sdl2_image
• [ ] sdl2_mixer
• [ ] sdl2_net
• [ ] sdl2_ttf
• [ ] singular
• [ ] singularity
• [ ] smpeg2
• [ ] snappy
• [ ] source-highlight
• [ ] sox
• [ ] speex
• [ ] sphinxbase
• [x] sqlite
• [ ] sstp-client
• [ ] tesseract
• [ ] texinfo
• [ ] texlive-core
• [ ] trmm_rsl
• [x] udunits
• [x] udunits2
• [ ] unixodbc
• [x] xerces-c
• [x] xorg-libice
• [x] xorg-libsm
• [x] xorg-libx11
• [x] xorg-libxau
• [x] xorg-libxaw
• [x] xorg-libxaw3d
• [x] xorg-libxcb
• [x] xorg-libxcursor
• [x] xorg-libxdmcp
• [x] xorg-libxext
• [x] xorg-libxfixes
• [x] xorg-libxft
• [x] xorg-libxi
• [x] xorg-libxmu
• [x] xorg-libxp
• [x] xorg-libxpm
• [x] xorg-libxrandr
• [x] xorg-libxrender
• [x] xorg-libxt
• [x] xorg-libxtst
• [ ] xraylib
• [x] xz
• [ ] yaml
• [ ] zbar
• [ ] zeromq
• [ ] zookeeper-c

[isuruf]

Note that, once the .la files are removed, you can probably remove some pinned dependencies.

[mingwandroid]

I'm considering adding a check to see that lib packages identified as run requirements are contributing a .so that's actually linked to, issuing a warning otherwise.

[CJ-Wright]

We could potentially write a migrator for this although we'd need to hard code that list somewhere.

[scopatz]

@CJ-Wright the tricky thing is that we'd need to modify the build.sh script. Is this something we can do from the migrators? Also @pkgw - what was the removal command you used on Windows?

[CJ-Wright]

Yes, we can do the modification fro the migrators. We have the full feedstock repo when we go to run the actual migrate method. (On a side note we might consider stuffing those script tests into the graph, either by discovering them during the migration or by explicitly pulling them in the make_graph stage)

[pkgw]

@scopatz The only Windows packages for which I've dealt with .la files are the X.org ones, which use a POSIX-y environment to use the autotools build scripts — bld.bat actually invokes build.sh. There are a bunch of hoops to jump through with environment variables (the distinction between $PREFIX and $PREFIX\Library matters) but in the end I use almost the same command. Example build.sh, example PR removing the Windows .la files.

[pkgw]

PS. And I did deal with all of the X.org library packages, so you can cross those off of the big list.

[scopatz]

@pkgw - I'll let you cross them off, since I don't know which ones you did.

[pkgw]

@scopatz Yeah .... I can't figure out how to cross things off by myself. Maybe some missing permission? Anyway, I did every package whose name begins with xorg, and libxcb.

[scopatz]

Hmm weird, you can't just click the boxes?

[pkgw]

Nope.

[scopatz]

Weird, I guess you must not have push rights to this repo. I'll go ahead and click the boxes.

[halldc]

I can't click the checkbox, but I've done imagemagick.

[scopatz]

@halldc - thanks! Just imagemagick? Or any of its deps?

[halldc]

Just imagemagick

[epruesse]

Sounds all nice. What do I do if my builds now fail with

/bin/bash ./libtool  --tag=CXX   --mode=link g++ -std=c++11  -g -O2 -w -L/home/travis/miniconda/lib/arb/lib -Wl,-rpath -Wl,/home/travis/miniconda/lib/arb/lib -Wl,-rpath -Wl,/home/travis/miniconda/lib   -o src/libsina.la -rpath /home/travis/install/lib src/align.lo src/aligned_base.lo src/alignment_stats.lo src/cseq.lo src/cseq_comparator.lo src/famfinder.lo src/mseq.lo src/pseq.lo src/query_arb.lo src/query_pt.lo src/rw_fasta.lo src/rw_arb.lo src/log.lo src/search_filter.lo src/tray.lo src/kmer_search.lo -lz -lARBDB -lCORE -L/home/travis/miniconda/lib -lglib-2.0 -lboost_program_options -lboost_thread -lpthread -lboost_serialization -lboost_system -ltbb -ltbbmalloc -lrt 
/bin/grep: /home/travis/miniconda/lib/libiconv.la: No such file or directory
/home/travis/miniconda/bin/sed: can't read /home/travis/miniconda/lib/libiconv.la: No such file or directory
libtool:   error: '/home/travis/miniconda/lib/libiconv.la' is not a valid libtool archive
make[1]: *** [src/libsina.la] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: Leaving directory `/home/travis/build/epruesse/SINA'
make: *** [all] Error 2

?

Best I can come up with is manually creating the missing .la file. :(

[epruesse]

For others running into above issue:

find $PATH_TO_MINICONDA -name '*.la' -delete

before building code fixes it.

The whole thing is a consequence of the intermediate stage in which some packages have had the .la files removed but others have not. If a still existing .la file references an already deleted .la file, libtool will try to inspect the deleted file and die failing to do so.

[ocefpaf]

PR https://github.com/conda/conda-build/pull/3102 was merged and as soon as we get a new release of conda-build we won't need

find $PATH_TO_MINICONDA -name '*.la' -delete

anymore. We still need to rebuild those packages though!

[ocefpaf]

@scopatz I marked gdal and udunits b/c those were renamed to libgdal and udunits2 respectively and we won't act on the legacy packages.

[CJ-Wright]

I think this is now being done automatically via conda-build thanks to @ocefpaf and @msarahan! Can this be closed?

[ocefpaf]

Can this be closed?

In theory this issue still exist, only after we rebuild stuff it will be solved.

With that said I am OK closing it b/c :smile: one less item to keep track of.

[CJ-Wright]

Is this done now that the compiler migration is done?

[ocefpaf]

I believe we can safely close this one. Thanks @CJ-Wright.