Grdb.swift: Xcode error "No such module 'CSQLite'" when embedding GRDB as a Swift package

[janine-knowink]

What did you do?

Added GRDB to my iOS app via the Swift package manager and added one "import GRDB" (nothing else). The project builds fine, but the tests fail to link with the error "Missing required module 'CSQLite'" on every file which includes "@testable import "

I made a demo project where I just added GRDB and imported it into ContentView.swift. This exhibited the problem. I then found and installed the CSQLite package and imported that also into ContentView.swift, but the problem remains.

What did you expect to happen?

If the main project builds ok, the tests should also.

What happened instead?

Described above.

Environment

GRDB flavor(s): (GRDB, SQLCipher, Custom SQLite build?)

Just GRDB

GRDB version:

4.5.0 "up to next major"

Installation method: (CocoaPods, SPM, manual?)

SPM

Xcode version:

11.1

Swift version:

5

Platform(s) running GRDB: (iOS, macOS, watchOS?)

iOS

macOS version running Xcode:

10.15.1

Demo Project

GRDBTest.zip

[groue]

Hello @janine-knowink,

Thanks for the report and the demo project. The solution is to remove CSQLite from the packages of your Xcode project.

[janine-knowink]

Sorry if I was not clear - the problem happens without CSQLite installed. I had installed it only to see if it would help, but it did not.

My co-worker worked around the problem by adding GRDB to the "Link Binary With Libraries" build rule for our test target, but it seems like that should not be required? GRDB isn't even used in any of the tests, it's only coming in via the testable import of the main project. And I haven't needed to add any of the packages which are used in the tests, so this seems wrong to me.

[groue]

Sorry if I was not clear

The attached demo projets starts successfully compile (both app target, and test target) after I have removed CSQLite from SPM packages.

My co-worker worked around the problem by adding GRDB to the "Link Binary With Libraries" build rule for our test target. but it seems like that should not be required?

GRDB has to be linked to the target(s) that uses it. That is normal.

Now I don't know if you talk about your application or test target. It your tests needs some objects that need GRDB indirectly, then yes your tests need GRDB.

But let's focus: the Missing required module 'CSQLite' error means that something is still referring to CSQLite. But whatever is referring to CSQLite should stop:

• GRDB 4.5.0 does not refer to CSQLite.
• Your app may still refer to CSQLite (see my previous comment).
• Some Xcode caches may also still refer to CSQLite: clean up everything, erase your Derived Data folder.

Make sure nothing at all refers to CSQLite, anywhere. This should remove your issue.

[janine-knowink]

There must be something different between your setup and mine. I tried again to build my sample app:

I deleted derived data:

janine@casey DerivedData % ls -d GRDBTest*
GRDBTest-gbrvwpmiukkzeidfvbfvrdyztjly   GRDBTest-gcxqjzigycqhyzbevkrwyhbcvuoe
janine@casey DerivedData % rm -rf GRDBTest*
janine@casey DerivedData % !ls
ls -d GRDBTest*
zsh: no matches found: GRDBTest*

Checked for references to CSQLite:

janine@casey GRDBTest % cd ~/Desktop/GRDBTest
janine@casey GRDBTest % find . -type f -exec fgrep CSQLite {} \;

Also verified that it's not being included by the package manager.

Closed the app window in Xcode and reopened it so that the package manager would check out GRDB again, and then built. Building the app works, but building the tests still fails with "Missing required module 'CSQLite'".

In order to get the tests to build, I have to comment out the setup function, which uses GRDB, the import of GRDB, and the testable import of the main app.

In our real app, this happens even though GRDB is not used at all in the tests. We get the error about CSQLite simply from the testable import of the main app, where it is used, and the only way to get rid of it is to manually link GRDB for the tests.

I am using Xcode 11.1 and macOS 10.15.1. Are you using something different?

[groue]

All right @janine-knowink, let's reopen the issue. GRDB used to depend on a CSQLite package, and this dependency has been removed in #627. I may have not done this properly: I'll check it.

Meanwhile, I suggest you downgrade GRDB to 4.4.0.

[groue]

After further investigation, I still can not reproduce your issue. I'll keep this open a few days, in order to let you exhibit a way to reproduce the "Missing required module 'CSQLite'" error. For example, make your demo project fail with this error. Meanwhile, I personally stop doing any investigation.

[janine-knowink]

The issue seems to exist only in Xcode 11.1. Using 11.2 the tests build and run fine. I have no idea why it's different, but that is what I am seeing.

[groue]

Good! Maybe Xcode 11.1 had a bug that was solved in Xcode 11.2? We'll never know. Thanks for telling.

[groue]

For the record, this conversation may contain interesting hints, provided we are able to reproduce the "Missing required module 'CSQLite'" error. Left as an exercice for the unfortunate.

[ApplebaumIan]

I'm now receiving this bug as well on Xcode Version 11.2.1 (11B500) using the SPM.

What did you expect to happen?

If the main project builds ok, the tests should also. Tests break on the server during integration testing and on local Mac

What happened instead?

Environment

GRDB flavor(s): (GRDB, SQLCipher, Custom SQLite build?)

Just GRDB

GRDB version:

4.5.0 "up to next major"

Installation method: (CocoaPods, SPM, manual?)

SPM

Xcode version:

11.2.1

Swift version:

5

Platform(s) running GRDB: (iOS, macOS, watchOS?)

iOS

macOS version running Xcode:

10.14.5

The build only fails for testing.
Build target ColorAssistantTests_2019-11-27T18-38-16.txt
Build target ColorAssistantUITests_2019-11-27T18-38-16.txt

[groue]

Hello @ApplebaumIan. I still can't reproduce this issue, so there's nothing I can do. Please re-read above comments. Thank you!

[ApplebaumIan]

Would it help If I zipped my repository and sent it to you? It only fails when you try to run tests. Or at least left it as an example?

[groue]

If you can not investigate and understand the cause of your issue, then yes, asking for help is more efficient if you can provide a reproducible case that other people can inspect.

Without any reproducible case, people just don't know what you are talking about, and can't help.

So, first try to investigate yourself, read, learn about Swift packages and Xcode, search Google, try hard on your own time. Only when you think you are stuck and really need the time of somebody else, ask for help, and make it easier for people who will help with a reproducible case.

[simonbirt]

Weirdly this just started happening for me as well, after a minor unrelated change. What fixed it for me was removing the workspace file UserInterfaceState.xcuserstate (which was the only file I could find in the workspace that contained the string 'CSQLite') and restarting Xcode.

[groue]

Thank you for the tip, @simonbirt! @ApplebaumIan, @janine-knowink, does this trick resolve the problem for you too?

[rcasula]

Hi @groue, I'm facing the same issue.
When using the library with SPM the target build fails. I've tried all of the above but nothing worked.

I'm trying with my personal project but also with GRDBDemoiOS installing GRDB with SPM. Same results.
The build fails in all GRDB files that import CSQLite (e.g. Core/Configuration.swift)

GRDB v4.8.1
Xcode 11.3.1

[groue]

Hi @robcas3. Until there is a way to reproduce this issue, nobody will be able to help. This has been said several times above already.

[rcasula]

Hi @groue I've found a hint.
If you test a brand new project 'iOS App with Watch App' and you include the library in the Watch Extension you can get the error stated above.
I hope it would be of some help.

[Edit]
Here is a sample repo that shows the issue: https://github.com/robcas3/TestGRDB

I've tested with 4.9.0, 4.8.1, 4.5.0, 4.4.0 and it does not work with either of these versions.

It seems like the modulemap, when built with watchOS target, cannot be found.

[groue]

Thank you @robcas3, I could finally reproduce the error from your sample repo 👍

The error happens with the watch extension target. I could successfully create and run UI and Unit testing targets that use GRDB.

This is an Xcode bug, not a GRDB bug.

Now, things being what they are, it would be nice to have a workaround. I could not find any.

I have tried to downgrade to GRDB 4.4.0, at a time when CSQLite was still an independent repo (before #627), so that I could add https://github.com/groue/CSQLite as an explicit dependency to the Xcode project. Alas! This does not make it possible to explicitly link it to the watch extension target.

For the record, the CSQLite SPM target is needed for a single GRDB feature: the global error log.

My current suggestion is:

1. Do file a bug on https://feedbackassistant.apple.com
2. Use another installation technique (such as CocoaPods)

For my part, I'll keep this issue open, just in case someone would have a brilliant idea that would restore support for SPM in multi-targets Xcode projects. And I'll add a warning in the SPM section of the documentation.

[haikusw]

I know nothing about this project (but plan to start using it this month). I was looking through the open issues to see how stable/good it is and found this open issue.

This doesn't look like an Xcode bug to me. I downloaded the test repo https://github.com/robcas3/TestGRDB and built it with Xcode 11.3 on macOS 10.15.2

This comment seems to indicate that there should be no further dependency on CSQLite:

All right @janine-knowink, let's reopen the issue. GRDB used to depend on a CSQLite package, and this dependency has been removed in #627. I may have not done this properly: I'll check it.

Meanwhile, I suggest you downgrade GRDB to 4.4.0.

However the error in Xcode I get is in the file GRDB/CORE/Configuration.swift which clearly has still an import for CSQLite on line 4. This import is conditional on SWIFT_PACKAGE (so only occurs if you import GRDB via the package manager I guess?).

Actually, there are a ton of references to CSQLite in the repo on the master branch, so maybe I'm misunderstanding this thread? Apologies if so.

https://github.com/groue/GRDB.swift/search?q=CSQLite&unscoped_q=CSQLite

It seems like the Package.swift file needs to list CSQLite as a dependency given these conditional import statements.

[groue]

Hello @haikusw. The GRDB target depends on a system library: https://github.com/groue/GRDB.swift/blob/v4.9.0/Package.swift#L19

This system library is defined there: https://github.com/groue/GRDB.swift/tree/v4.9.0/Sources/sqlite3

It defines a module named CSQLite, to which you find many references in GRDB source code.

For more information, see https://github.com/groue/GRDB.swift/issues/642#issuecomment-575994093

To me, this remains an Xcode bug, uneasy to define, and for which I know no workaround.

[haikusw]

Thank you for your patience with my ignorance. I have done additional reading and learned more :)

I actually have gotten it to build a few times but it doesn't seem to reliably do so.
In one case I added GRDB as a parallel project to the TestGRDB project in a workspace and built all the targets of GRDB manually and then built the TestGRDB app and it built. But it is not reliably buildable.

I thought it might be the ".apt" provider entry for "sqlite3" since that seems like a unix thing not an iOS thing, but substituting "path: "Sources/sqlite3" there didn't resolve the issue for me.

Need to make a simple sample that shows the same issue and get a bug filed at bugs.swift.org I guess.

[haikusw]

Still think .apt in https://github.com/groue/GRDB.swift/blob/master/Package.swift on line 16 is incorrect since I've confirmed in the SPM source that .apt is handled via apt-get which isn't part of the default macOS install nor added by installing Xcode command line tools. Maybe it doesn't matter because it'll always just be ignored (see source below), but I wonder if being flagged as not available for this platform is part of the problem around the CSQLite error this thread is about:

/// Check if the provider is available for the current platform.
var isAvailable: Bool {
    guard let platform = Platform.currentPlatform else { return false }
    switch self {
    case .brew:
        if case .darwin = platform {
            return true
        }
    case .apt:
        if case .linux(.debian) = platform {
            return true
        }
        if case .android = platform {
            return true
        }
    }
    return false
}

from: https://github.com/apple/swift-package-manager/blob/b6349d516d2f9b2f26ddae9de2c594ede24af7d6/Sources/PackageLoading/Target%2BPkgConfig.swift

There are likely other elements contributing to this issue, but this seems like a place to start.

Mostly just making this note here as a note to myself or anyone else looking into this (I intend to revisit it when time permits).

[groue]

.apt indeed targets Linux. There is a strong YAGNI smell here, I agree, since Linux is not officially supported.

But I don't want to purposely ruin the preliminary work of https://github.com/groue/GRDB.swift/pull/354.

[haikusw]

Agreed. Thanks for letting me know that Linux support is not yet supported but is hoped for some day. Sounds good.

Two thoughts:
a) if this .systemLibrary entry isn't doing anything on macOS / iOS / ... non-linux builds, is this maybe part of root cause for the CSQLite issues that this ticket is written around? Seems possible and even likely.

b) There are techniques to put this linux-only requirement in the Package.swift file if doing a linux build, namely remove it from where it is now and add it after this Package initializer using something like:

#if os(linux)
package.targets.append(.systemLibrary(name: "sqlite3", providers: [.apt(["libsqlite3-dev"])]))
#endif

// typed into this field so might not be quite syntactically 100%.

If it's ignored as is then could potentially just add the macOS/iOS etc version of this .systemLIbrary entry and leave the linux one as is.

Anyway, these are some threads to pull on to fix this open issue. It would be nice to get GRDB 100% working as a Swift package. I will try to make time to continue to tease this apart.

[groue]

Thanks for your dedication, this is a much useful contribution.

[groue]

Closing due to lack of activity.

[groue]

I can't reproduce the issue any longer with Xcode 11.4

[haikusw]

Sorry I haven't had time to look into this more. Incorporating GRDB into our project got deferred (we'll get there someday) and things ramped up so haven't had time to investigate this on the side. Still intend to eventually…

[groue]

No problem, @haikusw.

This issue has plenty of workarounds. There are other installation methods than SPM, and I can no longer reproduce it with Xcode 11.4.

On top of that, the GRDB horizon is now the forthcoming GRDB 5, which will require Xcode 11.4.

This issue is thus no longer an "urgent" one, an issue of general interest. More: once GRDB 5 has shipped, I will not ship a fix for GRDB 4. Instead, I will ask users to switch to GRDB 5.

Indeed, supporting old versions of GRDB, for the few teams that can't upgrade for some reason, is out of the scope of this freely available open source library. Such level of support requires adequate sponsoring (to be discussed).

[shawnkoh]

@groue We are still facing this issue with GRDB 5.0.0-beta.2 on Xcode 11.5

[groue]

Hello @shawnkoh. I am no longer able reproduce the issue. Will you please investigate and share a minimal setup which creates this error? Like a recipe. "Do this, then that, finally this, and witness the error." Thank you very much for your contribution.

[xplus2g4]

@groue I am able to reproduce this bug, but it seems like a Xcode bug, here are the steps:

1. Create a standalone swift package in your project
2. Create MyLibrary and MyLibrary2 in the inner package
   a) Import GRDB in MyLibrary
   b) Import MyLibrary and GRDB in MyLibrary2
3. In Package.swift of the standalone package:
   a) add the two libraries
   b) add GRDB to the dependencies
   c) Under targets, set GRDB as the dependencies to MyLibrary
   d) Do not set dependencies for MyLibrary2 (Ideally, MyLibrary should be set as the dependency)
4. Build MyLibrary
5. Build MyLibrary2
   -> Missing required module 'CSQLite'

[groue]

Thanks, @xplus2g4! Is it your setup, @shawnkoh?

[shawnkoh]

Yup @groue
Thanks for the really quick replies, we really appreciate your work! Its an incredibly well written library!

[groue]

Thanks @shawnkoh! But you know that this does not mean I have unlimited time and skills, don't you? I appreciate flattery as anybody else, but users who investigate such bugs and properly report them to Apple are even sweeter. THIS is your best option today, not waiting for me.

[ojkelly]

I just had this issue with Xcode 12 beta.

Honestly it's impossible to pin down, but a few interesting things. It happened on a Multiplatform App (new project template in xcode12), in a project that's already had a bit of work, many builds and some other SPM packages.

A brand new multiplatform app it worked.

So I cleaned everything up, build folder, that didn't work.

What did work:
_Removed your Derived Data folder._

[mhrakoto999]

Hello all,
I encountered the same issue when building UITests with both Xcode 12 and Xcode 12.2 beta.
I cleaned up DerivedData directory to no avail.

I inspected the ModuleCache.noindex directory under the DerivedData and noticed that 2 files CSQliteXXX.pcm and CSQliteXXX.pcm.timestamp are generated (along with files named after Apple-related frameworks such as SwiftUI, CoreAudio and so on) each time Xcode builds the main app.

Hope it will help

[mhrakoto999]

Additional info:
Main app and unit tests are built successfully

[mhrakoto999]

Hi all,

I didn't realize the case is closed as I was coming from a google search.
Last info:
I had retroverted to GRDB 4.14.0 and tried to build UITests again, it built well with Xcode 12.2 beta

[citelao]

I found this thread via Google search when trying to set up unit tests that referenced my app.

Since UI tests (the built-in tests in the Multiplatform template) simply launch the app, they don't need references to GRDB etc. However, as soon as I tried to import my app I started getting this error (missing CSQLite).

All-in-all, the fix was to ensure we link the GRDB library when we finish compiling (I also added my other dependencies, since they are needed in my case):

Deleting DerivedData didn't work, but this did. Hope this helps someone, even if I'm a bit necro-ing a thread here.