Cryptopp: make clean does not delete neon_simd.o

[lordadamson]

Trying to cross-compile to iOS on a macOS

• State the operating system and version (Ubutnu 17 x86_64, Windows 7 Professional x64, etc)

macOS 11.1 Big Sur

• State the version of the Crypto++ library (Crypto++ 7.0, Master, etc)

Master

• State how you built the library (Makefile, Cmake, distro, etc)

export IOS_SDK=iPhoneOS
export IOS_CPU=arm64

source TestScripts/setenv-ios.sh

export CXXFLAGS="-DNDEBUG -g2 -O3 -fPIC -pipe -fembed-bitcode"
make -f GNUmakefile-cross

• Show a typical command line (the output of the compiler for cryptlib.cpp)

clang++ -DNDEBUG -g2 -O3 -fPIC -pipe -fembed-bitcode -Wall -arch arm64 -miphoneos-version-min=6 -stdlib=libc++ --sysroot "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.2.sdk" -c cryptlib.cpp

• Show the link command (the output of the linker for libcryptopp.so or cryptest.exe)

This is where I discovered the issue because neon_simd.o was an old version from a previous build

libtool -static -o libcryptopp.a cryptlib.o cpu.o integer.o 3way.o adler32.o algebra.o algparam.o allocate.o arc4.o aria.o aria_simd.o ariatab.o asn.o authenc.o base32.o base64.o basecode.o bfinit.o blake2.o blake2b_simd.o blake2s_simd.o blowfish.o blumshub.o camellia.o cast.o casts.o cbcmac.o ccm.o chacha.o chacha_avx.o chacha_simd.o chachapoly.o cham.o cham_simd.o channels.o cmac.o crc.o crc_simd.o darn.o default.o des.o dessp.o dh.o dh2.o dll.o donna_32.o donna_64.o donna_sse.o dsa.o eax.o ec2n.o eccrypto.o ecp.o elgamal.o emsa2.o eprecomp.o esign.o files.o filters.o fips140.o fipstest.o gcm.o gcm_simd.o gf256.o gf2_32.o gf2n.o gf2n_simd.o gfpcrypt.o gost.o gzip.o hc128.o hc256.o hex.o hight.o hmac.o hrtimer.o ida.o idea.o iterhash.o kalyna.o kalynatab.o keccak.o keccak_core.o keccak_simd.o lea.o lea_simd.o luc.o mars.o marss.o md2.o md4.o md5.o misc.o modes.o mqueue.o mqv.o nbtheory.o neon_simd.o oaep.o osrng.o padlkrng.o panama.o pkcspad.o poly1305.o polynomi.o pssr.o pubkey.o queue.o rabbit.o rabin.o randpool.o rc2.o rc5.o rc6.o rdrand.o rdtables.o rijndael.o rijndael_simd.o ripemd.o rng.o rsa.o rw.o safer.o salsa.o scrypt.o seal.o seed.o serpent.o sha.o sha3.o sha_simd.o shacal2.o shacal2_simd.o shake.o shark.o sharkbox.o simeck.o simon.o simon128_simd.o skipjack.o sm3.o sm4.o sm4_simd.o sosemanuk.o speck.o speck128_simd.o square.o squaretb.o strciphr.o tea.o tftables.o threefish.o tiger.o tigertab.o ttmac.o tweetnacl.o twofish.o vmac.o wake.o whrlpool.o xed25519.o xtr.o xtrcrypt.o xts.o zdeflate.o zinflate.o zlib.o

• Show the exact error message you are receiving (copy and paste it); or

error: libtool: file: neon_simd.o is not an object file (not allowed in a library)
make: *** [libcryptopp.a] Error 1

• Clearly state the undesired behavior (and state the expected behavior)

If I do a make clean like this:

make -f GNUmakefile-cross clean
make clean

it won't delete neon_simd.o

Which leads to all sorts of confusing errors while linking because neon_simd.o is from a previous build.

I understand basic makefiles but I actually don't know how to fix this one because the make file is too advanced and complicated for me lol

If you direct me to where the issue might be I could open a PR with a fix.

For anyone looking for an immediate fix: just delete neon_simd.o and build again.

[noloader]

If I do a make clean like this:

make -f GNUmakefile-cross clean
make clean

it won't delete neon_simd.o

That's weird. What are the permissions on neon_simd.o? ls -Al neon_simd.* should show them.

---

This is where I discovered the issue because neon_simd.o was an old version from a previous build

So to reproduce, build iPhone/armv7, then make distclean, and then build iPhone/armv8?

[noloader]

This is a related check-in for GNUmakefile-cross: commit a783d2bf0742.

It fixes a missing CPPFLAGS that some folks may use. It does not fix your issue, however.

[lordadamson]

➜  cryptopp git:(master) ✗ ls -Al neon_simd.*
-rw-r--r--  1 adam  staff   5521 Dec 31 14:47 neon_simd.cpp
-rw-r--r--  1 adam  staff  54132 Jan  1 17:56 neon_simd.o

But it's not different from any other object file. For example:

➜  cryptopp git:(master) ✗ ls -Al integer.*
-rw-r--r--  1 adam  staff   133752 Dec 31 14:47 integer.cpp
-rw-r--r--  1 adam  staff    38699 Dec 31 14:47 integer.h
-rw-r--r--  1 adam  staff  3762732 Jan  1 17:54 integer.o

Here's the output of the clean commands:

➜  cryptopp git:(master) ✗ make -f GNUmakefile-cross clean
Here's what we found... IS_X86: 0, IS_X64: 0, IS_ARM32: 0, IS_ARMV8: 0

rm -f adhoc.cpp.o adhoc.cpp.proto.o cryptlib.o cpu.o integer.o 3way.o adler32.o algebra.o algparam.o allocate.o arc4.o aria.o aria_simd.o ariatab.o asn.o authenc.o base32.o base64.o basecode.o bfinit.o blake2.o blake2b_simd.o blake2s_simd.o blowfish.o blumshub.o camellia.o cast.o casts.o cbcmac.o ccm.o chacha.o chacha_avx.o chacha_simd.o chachapoly.o cham.o cham_simd.o channels.o cmac.o crc.o crc_simd.o darn.o default.o des.o dessp.o dh.o dh2.o dll.o donna_32.o donna_64.o donna_sse.o dsa.o eax.o ec2n.o eccrypto.o ecp.o elgamal.o emsa2.o eprecomp.o esign.o files.o filters.o fips140.o fipstest.o gcm.o gcm_simd.o gf256.o gf2_32.o gf2n.o gf2n_simd.o gfpcrypt.o gost.o gzip.o hc128.o hc256.o hex.o hight.o hmac.o hrtimer.o ida.o idea.o iterhash.o kalyna.o kalynatab.o keccak.o keccak_core.o keccak_simd.o lea.o lea_simd.o luc.o mars.o marss.o md2.o md4.o md5.o misc.o modes.o mqueue.o mqv.o nbtheory.o oaep.o osrng.o padlkrng.o panama.o pkcspad.o poly1305.o polynomi.o pssr.o pubkey.o queue.o rabbit.o rabin.o randpool.o rc2.o rc5.o rc6.o rdrand.o rdtables.o rijndael.o rijndael_simd.o ripemd.o rng.o rsa.o rw.o safer.o salsa.o scrypt.o seal.o seed.o serpent.o sha.o sha3.o sha_simd.o shacal2.o shacal2_simd.o shake.o shark.o sharkbox.o simeck.o simon.o simon128_simd.o skipjack.o sm3.o sm4.o sm4_simd.o sosemanuk.o speck.o speck128_simd.o square.o squaretb.o strciphr.o tea.o tftables.o threefish.o tiger.o tigertab.o ttmac.o tweetnacl.o twofish.o vmac.o wake.o whrlpool.o xed25519.o xtr.o xtrcrypt.o xts.o zdeflate.o zinflate.o zlib.o  rdrand-*.o adhoc.o test.o bench1.o bench2.o bench3.o datatest.o dlltest.o fipsalgt.o validat0.o validat1.o validat2.o validat3.o validat4.o validat5.o validat6.o validat7.o validat8.o validat9.o validat10.o regtest1.o regtest2.o regtest3.o regtest4.o 

➜  cryptopp git:(master) ✗ make clean
rm -f adhoc.cpp.o adhoc.cpp.proto.o cryptlib.o cpu.o integer.o 3way.o adler32.o algebra.o algparam.o allocate.o arc4.o aria.o aria_simd.o ariatab.o asn.o authenc.o base32.o base64.o basecode.o bfinit.o blake2.o blake2b_simd.o blake2s_simd.o blowfish.o blumshub.o camellia.o cast.o casts.o cbcmac.o ccm.o chacha.o chacha_avx.o chacha_simd.o chachapoly.o cham.o cham_simd.o channels.o cmac.o crc.o crc_simd.o darn.o default.o des.o dessp.o dh.o dh2.o dll.o donna_32.o donna_64.o donna_sse.o dsa.o eax.o ec2n.o eccrypto.o ecp.o elgamal.o emsa2.o eprecomp.o esign.o files.o filters.o fips140.o fipstest.o gcm.o gcm_simd.o gf256.o gf2_32.o gf2n.o gf2n_simd.o gfpcrypt.o gost.o gzip.o hc128.o hc256.o hex.o hight.o hmac.o hrtimer.o ida.o idea.o iterhash.o kalyna.o kalynatab.o keccak.o keccak_core.o keccak_simd.o lea.o lea_simd.o luc.o mars.o marss.o md2.o md4.o md5.o misc.o modes.o mqueue.o mqv.o nbtheory.o oaep.o osrng.o padlkrng.o panama.o pkcspad.o poly1305.o polynomi.o pssr.o pubkey.o queue.o rabbit.o rabin.o randpool.o rc2.o rc5.o rc6.o rdrand.o rdtables.o rijndael.o rijndael_simd.o ripemd.o rng.o rsa.o rw.o safer.o salsa.o scrypt.o seal.o seed.o serpent.o sha.o sha3.o sha_simd.o shacal2.o shacal2_simd.o shake.o shark.o sharkbox.o simeck.o simon.o simon128_simd.o skipjack.o sm3.o sm4.o sm4_simd.o sosemanuk.o speck.o speck128_simd.o square.o squaretb.o sse_simd.o strciphr.o tea.o tftables.o threefish.o tiger.o tigertab.o ttmac.o tweetnacl.o twofish.o vmac.o wake.o whrlpool.o xed25519.o xtr.o xtrcrypt.o xts.o zdeflate.o zinflate.o zlib.o rdrand-*.o adhoc.o test.o bench1.o bench2.o bench3.o datatest.o dlltest.o fipsalgt.o validat0.o validat1.o validat2.o validat3.o validat4.o validat5.o validat6.o validat7.o validat8.o validat9.o validat10.o regtest1.o regtest2.o regtest3.o regtest4.o cryptlib.export.o cpu.export.o integer.export.o 3way.export.o adhoc.export.o adler32.export.o algebra.export.o algparam.export.o allocate.export.o arc4.export.o aria.export.o aria_simd.export.o ariatab.export.o asn.export.o authenc.export.o base32.export.o base64.export.o basecode.export.o bench1.export.o bench2.export.o bench3.export.o bfinit.export.o blake2.export.o blake2b_simd.export.o blake2s_simd.export.o blowfish.export.o blumshub.export.o camellia.export.o cast.export.o casts.export.o cbcmac.export.o ccm.export.o chacha.export.o chacha_avx.export.o chacha_simd.export.o chachapoly.export.o cham.export.o cham_simd.export.o channels.export.o cmac.export.o crc.export.o crc_simd.export.o darn.export.o datatest.export.o default.export.o des.export.o dessp.export.o dh.export.o dh2.export.o dll.export.o dlltest.export.o donna_32.export.o donna_64.export.o donna_sse.export.o dsa.export.o eax.export.o ec2n.export.o eccrypto.export.o ecp.export.o elgamal.export.o emsa2.export.o eprecomp.export.o esign.export.o files.export.o filters.export.o fips140.export.o fipsalgt.export.o fipstest.export.o gcm.export.o gcm_simd.export.o gf256.export.o gf2_32.export.o gf2n.export.o gf2n_simd.export.o gfpcrypt.export.o gost.export.o gzip.export.o hc128.export.o hc256.export.o hex.export.o hight.export.o hmac.export.o hrtimer.export.o ida.export.o idea.export.o iterhash.export.o kalyna.export.o kalynatab.export.o keccak.export.o keccak_core.export.o keccak_simd.export.o lea.export.o lea_simd.export.o luc.export.o mars.export.o marss.export.o md2.export.o md4.export.o md5.export.o misc.export.o modes.export.o mqueue.export.o mqv.export.o nbtheory.export.o oaep.export.o osrng.export.o padlkrng.export.o panama.export.o pkcspad.export.o poly1305.export.o polynomi.export.o pssr.export.o pubkey.export.o queue.export.o rabbit.export.o rabin.export.o randpool.export.o rc2.export.o rc5.export.o rc6.export.o rdrand.export.o rdtables.export.o regtest1.export.o regtest2.export.o regtest3.export.o regtest4.export.o rijndael.export.o rijndael_simd.export.o ripemd.export.o rng.export.o rsa.export.o rw.export.o safer.export.o salsa.export.o scrypt.export.o seal.export.o seed.export.o serpent.export.o sha.export.o sha3.export.o sha_simd.export.o shacal2.export.o shacal2_simd.export.o shake.export.o shark.export.o sharkbox.export.o simeck.export.o simon.export.o simon128_simd.export.o skipjack.export.o sm3.export.o sm4.export.o sm4_simd.export.o sosemanuk.export.o speck.export.o speck128_simd.export.o square.export.o squaretb.export.o sse_simd.export.o strciphr.export.o tea.export.o test.export.o tftables.export.o threefish.export.o tiger.export.o tigertab.export.o ttmac.export.o tweetnacl.export.o twofish.export.o validat0.export.o validat1.export.o validat10.export.o validat2.export.o validat3.export.o validat4.export.o validat5.export.o validat6.export.o validat7.export.o validat8.export.o validat9.export.o vmac.export.o wake.export.o whrlpool.export.o xed25519.export.o xtr.export.o xtrcrypt.export.o xts.export.o zdeflate.export.o zinflate.export.o zlib.export.o cryptlib.import.o cpu.import.o integer.import.o 3way.import.o adler32.import.o algebra.import.o algparam.import.o allocate.import.o arc4.import.o aria.import.o aria_simd.import.o ariatab.import.o asn.import.o authenc.import.o base32.import.o base64.import.o basecode.import.o bfinit.import.o blake2.import.o blake2b_simd.import.o blake2s_simd.import.o blowfish.import.o blumshub.import.o camellia.import.o cast.import.o casts.import.o cbcmac.import.o ccm.import.o chacha.import.o chacha_avx.import.o chacha_simd.import.o chachapoly.import.o cham.import.o cham_simd.import.o channels.import.o cmac.import.o crc.import.o crc_simd.import.o darn.import.o default.import.o des.import.o dessp.import.o dh.import.o dh2.import.o dll.import.o donna_32.import.o donna_64.import.o donna_sse.import.o dsa.import.o eax.import.o ec2n.import.o eccrypto.import.o ecp.import.o elgamal.import.o emsa2.import.o eprecomp.import.o esign.import.o files.import.o filters.import.o fips140.import.o fipstest.import.o gcm.import.o gcm_simd.import.o gf256.import.o gf2_32.import.o gf2n.import.o gf2n_simd.import.o gfpcrypt.import.o gost.import.o gzip.import.o hc128.import.o hc256.import.o hex.import.o hight.import.o hmac.import.o hrtimer.import.o ida.import.o idea.import.o iterhash.import.o kalyna.import.o kalynatab.import.o keccak.import.o keccak_core.import.o keccak_simd.import.o lea.import.o lea_simd.import.o luc.import.o mars.import.o marss.import.o md2.import.o md4.import.o md5.import.o misc.import.o modes.import.o mqueue.import.o mqv.import.o nbtheory.import.o oaep.import.o osrng.import.o padlkrng.import.o panama.import.o pkcspad.import.o poly1305.import.o polynomi.import.o pssr.import.o pubkey.import.o queue.import.o rabbit.import.o rabin.import.o randpool.import.o rc2.import.o rc5.import.o rc6.import.o rdrand.import.o rdtables.import.o rijndael.import.o rijndael_simd.import.o ripemd.import.o rng.import.o rsa.import.o rw.import.o safer.import.o salsa.import.o scrypt.import.o seal.import.o seed.import.o serpent.import.o sha.import.o sha3.import.o sha_simd.import.o shacal2.import.o shacal2_simd.import.o shake.import.o shark.import.o sharkbox.import.o simeck.import.o simon.import.o simon128_simd.import.o skipjack.import.o sm3.import.o sm4.import.o sm4_simd.import.o sosemanuk.import.o speck.import.o speck128_simd.import.o square.import.o squaretb.import.o sse_simd.import.o strciphr.import.o tea.import.o tftables.import.o threefish.import.o tiger.import.o tigertab.import.o ttmac.import.o tweetnacl.import.o twofish.import.o vmac.import.o wake.import.o whrlpool.import.o xed25519.import.o xtr.import.o xtrcrypt.import.o xts.import.o zdeflate.import.o zinflate.import.o zlib.import.o adhoc.import.o test.import.o bench1.import.o bench2.import.o bench3.import.o datatest.import.o dlltest.import.o fipsalgt.import.o validat0.import.o validat1.import.o validat2.import.o validat3.import.o validat4.import.o validat5.import.o validat6.import.o validat7.import.o validat8.import.o validat9.import.o validat10.import.o regtest1.import.o regtest2.import.o regtest3.import.o regtest4.import.o dlltest.dllonly.o

neon_simd.o is missing in both clean commands

[noloader]

OK, I think I see what is going on.

export IOS_SDK=iPhoneOS
export IOS_CPU=arm64
source TestScripts/setenv-ios.sh
export CXXFLAGS="-DNDEBUG -g2 -O3 -fPIC -pipe -fembed-bitcode"

Now, make distclean. Notice neon_simd.o is not included in the file list.

We actually have to make -f GNUmakefile-cross distclean. Notice neon_simd.o is included in the file list.

The reason the regular makefile running on x86 does not include neon_simd.o during clean is this piece of goodness in GNUmakefile : 1135:

# Remove unneeded arch specific files to speed build time.
ifeq ($(IS_PPC32)$(IS_PPC64),00)
  SRCS := $(filter-out ppc_%,$(SRCS))
endif
ifeq ($(IS_ARM32)$(IS_ARMV8),00)
  SRCS := $(filter-out arm_%,$(SRCS))
  SRCS := $(filter-out neon_%,$(SRCS))
endif
ifeq ($(IS_X86)$(IS_X32)$(IS_X64),000)
  SRCS := $(filter-out sse_%,$(SRCS))
endif

The source files are filtered from SRCS, and then the object files are missing from OBJS:

OBJS := $(SRCS:.cpp=.o)
OBJS := $(OBJS:.S=.o)

Later, make clean uses LIBOBJS, DLLOBJS, etc to clean the artifacts:

.PHONY: clean
clean:
    -$(RM) adhoc.cpp.o adhoc.cpp.proto.o $(LIBOBJS) rdrand-*.o $(TESTOBJS) $(DLLOBJS) $(LIBIMPORTOBJS) $(TESTIMPORTOBJS) $(DLLTESTOBJS)
    ...

[lordadamson]

aah good catch.

Feels like the kind of bug that is caused by caching. If we're not saving a significant build time with this trick then I'd recommend we get rid of it completely.

It's always nice to have things as simple and straight forward as possible.

[noloader]

I'm wondering if we should just rm *.o to sidestep it.

In the past I did not want to do it in case there were non-cryptopp object files there.

Ping @mouse07410. Mouse07410, are we allowed to whack all object files in our directory during a make clean or make distclean? Or are we limited to just the object files produced by the library?

I think the controlling document is GNU Coding Standards | Standard Targets. The clean and distclean recipes only authorize deleting object files "... that are created by configuring or building the program...".

[lordadamson]

Honestly rm *.o is the kind of low-tech solution I'd go for :D :D

[noloader]

OK, this should handle the problem: commit a184ce1aba1a. The commit builds a list of all potential object files and cleans them. It does not filter-out object files. It does not matter which makefile you use.

[noloader]

@lordadamson,

You'll need this one too. I missed it on the original commit. Commit 7315a6f907a1.

Hold on, that broke things. Reverting that one.

[noloader]

@lordadamson,

The difference between rm *.o and:

CLEAN_SRCS := $(wildcard *.cpp)
CLEAN_OBJS := $(CLEAN_SRCS:.cpp=.o)

is that *.o whacks all object files. CLEAN_OBJS only cleans object files produced by the library.

If you compile your own my_test.cxx, then its object file will be skipped because it is not included in CLEAN_SRCS. In fact, I use the *.cxx trick for test programs. Each *.cxx can have a main that would otherwise result in duplicate symbols main if they were included in CLEAN_SRCS.

[noloader]

This is the commit that handles Cryptogams object files: Commit ae20711372b1.

[mouse07410]

My personal take: make clean is allowed to (and should?) remove all the object files in our directory. make distcleanshould also remove all the source files that the build (or configuration) process generated.

[noloader]

Thanks @mouse07410,

If this ever surfaces again we'll just whack all of the object files.