Sqldelight: Non-deterministic builds between two machines

[Giszmo]

1328 sounds as if sqldelight was supposed to deterministically build but we get diffs in every(?) sqldelight generated file between two devs, both running linux and equal Dockers. In fact, using or not the Docker container makes no difference on the respective machines but somehow we have a huge diff between the two machines.

The code is this commit and the diff is
this.

[Giszmo]

Apparently sqldelight relies on file order as per underlying file system. I can locally reproduce the issue:

mkdir /tmp/sort /tmp/rev
disorderfs --sort-dirents=yes --reverse-dirents=no $PWD /tmp/sort
disorderfs --sort-dirents=yes --reverse-dirents=yes $PWD /tmp/rev

(cd /tmp/sort ; ./gradlew clean :mbw:assProdRel; apktool d -o fromSorted mbw/build/outputs/apk/prodnet/release/mbw-prodnet-release.apk)
(cd /tmp/rev ; ./gradlew clean :mbw:assProdRel; apktool d -o fromReversed mbw/build/outputs/apk/prodnet/release/mbw-prodnet-release.apk)

diff --brief --recursive fromSorted fromReversed

produces this output:

Files fromSorted/original/META-INF/CERT.RSA and fromReversed/original/META-INF/CERT.RSA differ
Files fromSorted/original/META-INF/CERT.SF and fromReversed/original/META-INF/CERT.SF differ
Files fromSorted/original/META-INF/MANIFEST.MF and fromReversed/original/META-INF/MANIFEST.MF differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountBackingQueriesImpl$deleteTransaction$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountBackingQueriesImpl$deleteTransaction$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountBackingQueriesImpl$deleteTransaction$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountBackingQueriesImpl$deleteTransaction$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountBackingQueriesImpl$insertTransaction$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountBackingQueriesImpl$insertTransaction$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountBackingQueriesImpl$insertTransaction$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountBackingQueriesImpl$insertTransaction$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountBackingQueriesImpl$select$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountBackingQueriesImpl$select$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountBackingQueriesImpl$select$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountBackingQueriesImpl$select$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountBackingQueriesImpl$Select$execute$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountBackingQueriesImpl$Select$execute$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountBackingQueriesImpl$Select.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountBackingQueriesImpl$Select.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountBackingQueriesImpl$updateTransaction$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountBackingQueriesImpl$updateTransaction$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountBackingQueriesImpl$updateTransaction$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountBackingQueriesImpl$updateTransaction$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountBackingQueriesImpl.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountBackingQueriesImpl.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$delete$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$delete$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$delete$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$delete$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$insert$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$insert$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$insert$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$insert$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$insertFullObject$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$insertFullObject$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$insertFullObject$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$insertFullObject$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$selectAll$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$selectAll$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$selectAll$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$selectAll$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$selectByUUID$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$selectByUUID$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$selectByUUID$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$selectByUUID$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$SelectByUUID$execute$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$SelectByUUID$execute$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$SelectByUUID.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$SelectByUUID.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$update$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$update$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$update$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl$update$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/AccountContextQueriesImpl.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$deleteTransaction$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$deleteTransaction$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$deleteTransaction$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$deleteTransaction$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$insertTransaction$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$insertTransaction$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$insertTransaction$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$insertTransaction$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$selectTransactionSummaries$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$selectTransactionSummaries$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$selectTransactionSummaries$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$selectTransactionSummaries$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$SelectTransactionSummaries$execute$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$SelectTransactionSummaries$execute$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$selectTransactionSummariesSince$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$selectTransactionSummariesSince$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$selectTransactionSummariesSince$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$selectTransactionSummariesSince$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$SelectTransactionSummariesSince$execute$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$SelectTransactionSummariesSince$execute$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$SelectTransactionSummariesSince.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$SelectTransactionSummariesSince.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$SelectTransactionSummaries.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$SelectTransactionSummaries.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$selectTransactionSummaryById$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$selectTransactionSummaryById$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$selectTransactionSummaryById$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$selectTransactionSummaryById$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$SelectTransactionSummaryById$execute$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$SelectTransactionSummaryById$execute$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$SelectTransactionSummaryById.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl$SelectTransactionSummaryById.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthAccountBackingQueriesImpl.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$delete$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$delete$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$delete$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$delete$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$insert$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$insert$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$insert$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$insert$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$insertFullObject$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$insertFullObject$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$insertFullObject$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$insertFullObject$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$selectAll$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$selectAll$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$selectAll$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$selectAll$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$selectByUUID$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$selectByUUID$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$selectByUUID$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$selectByUUID$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$SelectByUUID$execute$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$SelectByUUID$execute$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$SelectByUUID.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$SelectByUUID.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$update$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$update$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$update$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl$update$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/EthContextQueriesImpl.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/FeeEstimationsQueriesImpl$insertFullObject$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/FeeEstimationsQueriesImpl$insertFullObject$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/FeeEstimationsQueriesImpl$insertFullObject$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/FeeEstimationsQueriesImpl$insertFullObject$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/FeeEstimationsQueriesImpl$selectByCurrency$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/FeeEstimationsQueriesImpl$selectByCurrency$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/FeeEstimationsQueriesImpl$selectByCurrency$2.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/FeeEstimationsQueriesImpl$selectByCurrency$2.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/FeeEstimationsQueriesImpl$SelectByCurrency$execute$1.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/FeeEstimationsQueriesImpl$SelectByCurrency$execute$1.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/FeeEstimationsQueriesImpl$SelectByCurrency.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/FeeEstimationsQueriesImpl$SelectByCurrency.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/FeeEstimationsQueriesImpl.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/FeeEstimationsQueriesImpl.smali differ
Files fromSorted/smali_classes2/com/mycelium/generated/wallet/database/walletcore/WalletDBImpl$Schema.smali and fromReversed/smali_classes2/com/mycelium/generated/wallet/database/walletcore/WalletDBImpl$Schema.smali differ

[JakeWharton]

It's quite possible, yes! I suspect a lot of Gradle plugins fall into this trap.

[Giszmo]

@JakeWharton does your reaction mean that it's an acknowledged issue that should get fixed in an upcoming release? Or is more info needed? Should I bake a minimal example instead of using our project?

I am glad we can work around this but we can't expect external auditors to work around this, given I have not yet figured out how to hide the disorderfs voodoo inside a Dockerized build script so I would very much appreciate a fix from sqldelight and awareness for the issue, so it doesn't return in a later update.

[JakeWharton]

I certainly acknowledge it and that it needs fixed, yes. We probably just
need to issue a sort() somewhere.

On Sat, Jan 11, 2020 at 12:13 AM Leo Wandersleb [e-mail]
wrote:

@JakeWharton https://github.com/JakeWharton does your reaction mean
that it's an acknowledged issue that should get fixed in an upcoming
release? Or is more info needed? Should I bake a minimal example instead of
using our project?

I am glad we can work around this but we can't expect external auditors to
work around this, given I have not yet figured out how to hide the
disorderfs voodoo inside a Dockerized build script so I would very much
appreciate a fix from sqldelight and awareness for the issue, so it doesn't
return in a later update.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/cashapp/sqldelight/issues/1548?email_source=notifications&email_token=AAAQIELSBHYKXLVEBMUWMZ3Q5FIQLA5CNFSM4KEYENNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIVZTZY#issuecomment-573282791,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAQIEIAFVBEGSMWTY56MSTQ5FIQLANCNFSM4KEYENNA
.

[Giszmo]

How is it going? This is still a huge issue for me and others.

[Giszmo]

I suspect a lot of Gradle plugins fall into this trap.

For me that means reproducibility and thus auditability is not getting enough attention. There are many applications that could do a lot of harm if the binary was infected with a backdoor, yet projects appear to not care. Any release manager of any app could under distress or unknowingly inject bad stuff in a build. My focus is Bitcoin wallets and even there the consciousness is very low.

[Giszmo]

I really care about this issue and although I thought cashapp has the resources to get this done quickly, my project Walletscrutiny is now also offering a $500 bug bounty.

[AlecStrong]

if there's a diff in the smali i assume theres a diff in the generated kotlin code, are you able to upload that? I assume its the query identifiers but want to confirm that is the case

other things that would help push this forward is having a failing test in a PR that we can then fix, it sounds like you're able to reproduce by changing the filesystem ordering, is this something that could be done in one of our gradle integration tests?

[Giszmo]

it sounds like you're able to reproduce by changing the filesystem ordering, is this something that could be done in one of our gradle integration tests?

Disorderfs is designed to do such tests. For reproducible test results you would mount the project to an FS sorted alphabetically and to an FS with the sort order reversed. If the result is the same, you should be good.

I'm no expert on disorderfs but for reproducing Mycelium Wallet builds I run these lines.

Essentially

mkdir /tmp/sorted
sudo disorderfs --sort-dirents=yes --reverse-dirents=no --multi-user=yes path/to/my/repo /tmp/sorted
cd /tmp/sorted
./gradlew build

If I remember correctly, sudo and the --multi-user=yes part can be omitted if you don't want to run the build inside a docker container.