Realm-cocoa: Crash: RLMGenerator released early(?) when iterating over RLMResults using flatMap()

[nevil]

I'm using realm-cocoa in a Swift application.
This is necessary as I can not yet switch to using frameworks.

I'm experiencing a crash when using flatMap on RLMResults when building with "-Owholemodule".

Building with Xcode 7.3.1 and AddressSanitizer enabled points out a heap-use-after-free in [RLMResults countByEnumeratingWithState:objects:count].
It seems that the RLMGenerator instance is released before flatMap() is finished calling countByEnumeratingWithState...

The issue is reproducing on device and in the simulator.

A workaround is to use the attribute @inline(never) on RLMGenerator.next().
This is possibly a swift optimizer issue...

Steps to Reproduce

• Download the sample from the link below.
• Go to RLMGeneratorCrash/ folder
• Install Realm using cocoa pods: 'pod install'
• Open OptimizationBug.xcworkspace
• Run in a simulator
  Result:
  SUMMARY: AddressSanitizer: heap-use-after-free RLMResults.mm:145 in -[RLMResults countByEnumeratingWithState:objects:count:]
  

  Code Sample

A sample project is available at https://github.com/nevil/Bugs/tree/master/RLMGeneratorCrash

Realm version: 1.0.1
Xcode version: 7.3.1
Cocoapods version: 0.39.0

[jpsim]

Thanks for the great bug report and repro case! I could confirm this locally.

It seems to me like this is a Swift bug, so I wanted to check if it was still an issue with other Swift versions.

But I can't check against Swift 2.3 since it doesn't support address sanitizer and can't check against Swift 3 snapshot because of the following swiftc assertion failure:

LLVM ERROR: unsupported relocation with subtraction expression, symbol ... can not be undefined in a subtraction expression

In any case, let's keep this open so we can further investigate if this is a bug in Realm or Swift.

[nevil]

I reported the optimization issue to the Swift project https://bugs.swift.org/browse/SR-1768

I included a copy of NSFastGenerator from Swift stdlib in RLMSupport.swift, prevented the functions from being inlined, added a deinit and some logging.
This clearly shows that next() is called after deinit of the instance.

final public class MyFastGenerator : GeneratorType {
    var enumerable: NSFastEnumeration
    var state: [NSFastEnumerationState]
    var n: Int
    var count: Int

    /// Size of ObjectsBuffer, in ids.
    var STACK_BUF_SIZE: Int { return 4 }

    /// Must have enough space for STACK_BUF_SIZE object references.
    struct ObjectsBuffer {
        var buf = (COpaquePointer(), COpaquePointer(),
                   COpaquePointer(), COpaquePointer())
    }
    var objects: [ObjectsBuffer]

    @inline(never)
    public func next() -> AnyObject? {
        print("MyFastGenerator next")
        if n == count {
            // FIXME: Is this check necessary before refresh()?
            if count == 0 { return .None }
            refresh()
            if count == 0 { return .None }
        }
        let next : AnyObject = state[0].itemsPtr[n]!
        n += 1
        return next
    }

    @inline(never)
    func refresh() {
        print("MyFastGenerator refresh")
        n = 0
        count = enumerable.countByEnumeratingWithState(
            state._baseAddressIfContiguous,
            objects: AutoreleasingUnsafeMutablePointer(
                objects._baseAddressIfContiguous),
            count: STACK_BUF_SIZE)
    }

    @inline(never)
    public init(_ enumerable: NSFastEnumeration) {
        print("MyFastGenerator init")
        self.enumerable = enumerable
        self.state = [ NSFastEnumerationState(
            state: 0, itemsPtr: nil,
            mutationsPtr: _fastEnumerationStorageMutationsPtr,
            extra: (0, 0, 0, 0, 0)) ]
        self.objects = [ ObjectsBuffer() ]
        self.n = -1
        self.count = -1
    }

    deinit {
        print("MyFastGenerator deinit")
    }
}

It gives the output:
MyFastGenerator init
MyFastGenerator next
MyFastGenerator refresh
MyFastGenerator deinit
MyFastGenerator next
MyFastGenerator refresh

The output is same also with Swift 2.3.

I'm getting the same error as you using Swift 3 targeting the simulator.
Targeting device I get:

Undefined symbols for architecture arm64:
  "nominal type descriptor for __ObjC.RLMResults", referenced from:
      l_protocol_conformances in RLMSupport.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[isoiphone]

Any updates on a fix for this?
It looks like it is still happening as of v2.1.0 release using Xcode 8.0 (8A218a) (whole module optimization is on)

I can confirm adding @inline(never) to public func next() -> RLMObject? resolves the crash.

My stacktrace is not useful for this other then pinpointing crash at RLMSupport.swift:40
I can private message you the offending realm + flatMap statement if it is of any value.

[jpsim]

Any updates on a fix for this?

I've commented on SR-1768 to follow up.

@isoiphone would you advise against adding @inline(never) to next() in RLMSupport.swift directly? If that works around the issue, that may be the best way to resolve it short term.

[isoiphone]

Adding that worked and fixed the crash I am aware of. There might be other crashes I'm not aware of.

I've left that statement in as a precaution, and just completely disabled swift whole module optimization. Does not seem ready for production use yet.

[jpsim]

... just completely disabled swift whole module optimization. Does not seem ready for production use yet.

Sounds like you should be filing bugs at https://bugs.swift.org rather than here, for maximum effect.

[jpsim]

The initial issue reported here appears to be resolved with Swift 3: https://github.com/nevil/Bugs/pull/1

I think the issue reported by @isoiphone is unrelated. So I'll be closing this issue now.

[nevil]

Thanks for following up!
I've since switched to RealmSwift (and recently swift 3) so no longer had the issue.