Realm-cocoa: Database growing significantly
Hi all
I tried asking this question on stack overflow but no answer yet and this issue is delay the release of our app until we can solve it.
http://stackoverflow.com/questions/41642267/realm-cocoa-database-growing-significantly
Using realm in a multithreaded app and I'm finding that the app is growing significantly every few seconds while I update percentages in my downloads using the session download api. Code to update the bytes downloaded looks like this.
[[RLMRealm defaultRealm] transactionWithBlock:^{
if(!restoreAsset.isInvalidated)
restoreAsset.bytesDownloaded = totalBytesWritten;
}];
I'm running multiple threads, up to about 32 in some cases between uploads and downloads.
Am I doing something wrong here by using the default realm or how can I avoid all these threads duplicating their data because they are all accessing the same table or whatever the problem would be.
The database eventually gets to over 2 Gigs and crashes the app. I compact the database when the app is restarted but this doesn't seem to make any difference to the size of it. There is not many rows in the database, approx 1-2K rows max. Sometimes a lot less. Doesn't seem to matter.
Any help or guidelines on how to access the Realm across multiple threads is welcome because I can't seem to get this to work as expected.
doniekelly
All 38 comments
@donie-inhance are you using autoreleasepool { ... } block around the Realm instance you open on background threads to force close it at the end of execution?
Zhuinden
on 16 Jan 2017
@donie-inhance could you please provide the whole code sample that is executed on each thread? I guess restoreAsset is a Realm object, how do you create or get it?
/cc @bdash, @tgoyne
stel
on 16 Jan 2017
Hi @Zhuinden. Yes, I'm using auto releases pools everywhere... This code has a download queue of items it needs to download from our server. It tries to keep up to 8 downloads going at any one time (SIMULTAENOUS_DOWNLOADS_FOR_RESTORE). If we are doing uploads at the same time it can be doing up to 24 uploads at once too, all accessing the same Realm.
However, this growing database problem is restricted to downloads using the code below. At the moment the CTXRestoreAsset table has 505 rows. Small items in columns. CTXDeviceAsset has 146 rows. The database has grown to 100 Megs already after a few downloads.
Let me know if you need more.
D
- (void)URLSession:(NSURLSession *)session downloadTask:(NSURLSessionDownloadTask *)downloadTask didWriteData:(int64_t)bytesWritten totalBytesWritten:(int64_t)totalBytesWritten totalBytesExpectedToWrite:(int64_t)totalBytesExpectedToWrite
{
@autoreleasepool {
// Update the DB with the progress
NSString *md5 = downloadTask.taskDescription;
CTXRestoreAsset *restoreAsset= [[CTXCloudPersistence shared] restoreItemForMD5:md5];
if(restoreAsset)
{
[[RLMRealm defaultRealm] transactionWithBlock:^{
if(!restoreAsset.isInvalidated)
restoreAsset.bytesDownloaded = totalBytesWritten;
}];
float percentage = ((float)totalBytesWritten/(float)totalBytesExpectedToWrite)*100;
if(percentage > 95.0)
NSLog(@"Downloading: %@/%@ %.0f%%", restoreAsset.tagCode, downloadTask.taskDescription, percentage);
}
}
}
- (void)URLSession:(NSURLSession *)session downloadTask:(NSURLSessionDownloadTask *)task didFinishDownloadingToURL:(NSURL *)url
{
@autoreleasepool {
NSString *md5 = task.taskDescription;
CTXRestoreAsset *restoredAsset = [[CTXCloudPersistence shared] restoreItemForMD5:md5];
NSInteger statusCode = ((NSHTTPURLResponse *)task.response).statusCode;
switch(statusCode)
{
case 200:
if(restoredAsset)
{
AssetOperations *assetOps = [AssetOperations new];
// Must move the asset before this call completes as it will get deleted otherwise. See NSURLSession docs
[restoredAsset moveDownloadToTempFolder:url];
[assetOps addAssetToDevice:restoredAsset withCompletion:^(NSString *md5, NSString *newIdentifier) {
CTXRestoreAsset *threadSafeAsset = [[CTXCloudPersistence shared] restoreItemForMD5:md5];
// Remove the asset from disk as we don't need it anymore
[AssetOperations removeFileFromDisk:[threadSafeAsset getLocalFilename]];
// update the asset status
[threadSafeAsset markAsRestored];
[threadSafeAsset copyAssetToDeviceTableWithIdentifier:newIdentifier];
// Let the UI know we've downloaded another item
[self notifyListeners];
}];
}
break;
case 401:
case 404:
{
NSLog(@"Asset does not exist in cloud %@", md5);
// Update the status of the asset to failed
[restoredAsset markAsRestoreFailed:statusCode];
break;
}
default:
[[CTXCloudPersistence shared] resetRestoreAssetForReDownload:restoredAsset];
break;
}
// Check if we need more assets retored
[self continueRestore];
}
}
-(void) continueRestore
{
@synchronized (self) {
if(!self.ctxCloudRestoreQueue)
self.ctxCloudRestoreQueue = [NSOperationQueue new];
CreateDownloadSessionsOperations *createSessions = [[CreateDownloadSessionsOperations alloc] initWithName:@"Create download sessions"];
createSessions.delegate = self;
[self.ctxCloudRestoreQueue addOperation:createSessions];
}
}
-(CTXRestoreAsset *) restoreItemForMD5:(NSString *)md5
{
@autoreleasepool {
return [[CTXRestoreAsset objectsWhere:@"md5 == %@", md5] firstObject];
}
}
-(void) markAsRestored
{
@autoreleasepool {
RLMRealm *realm = [RLMRealm defaultRealm];
[realm beginWriteTransaction];
self.status = RestoreAssetStatusOnDevice;
self.restoreTaskID = 0;
self.errorCode = 200;
self.selectedForRestore = NO;
[realm commitWriteTransaction];
// Mark restore as finished if this was the last item
[self checkIfWeAreFinshedRestore];
}
}
-(void) copyAssetToDeviceTableWithIdentifier:(NSString *)identifier
{
@autoreleasepool {
CTXDeviceAsset *deviceAsset = [CTXDeviceAsset new];
RLMRealm *realm = [RLMRealm defaultRealm];
[realm beginWriteTransaction];
// Move asset to upload asset queue
deviceAsset.localIdentifier = identifier;
deviceAsset.filename = self.filename;
deviceAsset.title = self.title;
deviceAsset.mimeType = self.mimeType;
deviceAsset.assetSize = self.assetSize;
deviceAsset.md5 = self.md5;
deviceAsset.lastModified = self.lastModified;
deviceAsset.created = self.created;
deviceAsset.orientation = self.orientation;
[realm addObject:deviceAsset];
[realm commitWriteTransaction];
}
}
Operation queue to create more download sessions from my download queue
@implementation CreateDownloadSessionsOperations
-(void) start
{
@autoreleasepool
{
if(self.isCancelled) // Check if we are cancelled
[super operationCancelled];
@synchronized ([CTXCloudManager shared]) {
// This is a long running task after the user selects what they want to resore as we have to generate MD5's. This will give us another three minutes in the background if the user happens to press home button before it finished. If we don't finish within the time we need to figure out what to do...
[self endBackgroundHandling];
self.taskId = [[UIApplication sharedApplication] beginBackgroundTaskWithName:@"Restore Selection Processing" expirationHandler:^{
// TODO - Clean up the task we started or figure out a way to resume it later if the app opened again
[self endBackgroundHandling];
}];
NSArray *allRestorableAssets;
BOOL keepProcessing = YES;
while(keepProcessing)
{
allRestorableAssets = [[CTXCloudPersistence shared] getBatchOfAssetsAwaitingDownload:SIMULTAENOUS_DOWNLOADS_FOR_RESTORE];
// If we have no assets to process then break out at this point as we have filled the download queue
if(allRestorableAssets.count == 0)
keepProcessing = NO;
// Scan through the assets that are ready for download and check for duplicates before committing to a download session
for(CTXRestoreAsset *assetToRestore in allRestorableAssets)
{
if(self.isCancelled) // Check if we are cancelled
[super operationCancelled];
// Do we have enough disk space to restore asset
if([assetToRestore isEnoughDiskSpaceForRestoringAsset])
{
// If asset not already on device then download it
if(![self isAssetOnDevice:assetToRestore])
{
NSLog(@"Creating download task for %@", assetToRestore.md5);
// Asset does not exist locally so go ahead and download it
[assetToRestore createSessionDownloadTask];
}
else
{
NSLog(@"Item %@ already exists. Not restoring", assetToRestore.md5);
// Mark item as restored as it already exists locally
[assetToRestore markAsRestored];
// Continue checking for restorable assets
[[CTXCloudManager shared].restoreManager continueRestore];
}
}
else
{
// Mark in DB as Insuffient storage avaialble on local device
[assetToRestore markAsRestoreFailed:507];
NSLog(@"Postpone assset download (%@) until disk space becomes available", assetToRestore.md5);
}
}
}
// Stop the background handler if we don't need it anymore
[self endBackgroundHandling];
[super operationComplete];
}
}
}
-(BOOL) isAssetOnDevice:(CTXRestoreAsset *)restoringAsset
{
CTXDeviceAsset *deviceAsset = [[CTXCloudPersistence shared] deviceItemForMD5:restoringAsset.md5];
if(deviceAsset)
return YES;
else
return NO;
}
-(void) endBackgroundHandling
{
if(self.taskId != UIBackgroundTaskInvalid)
{
[[UIApplication sharedApplication] endBackgroundTask:self.taskId];
self.taskId = UIBackgroundTaskInvalid;
}
}
@end
It seems to me that the code here
for(CTXRestoreAsset *assetToRestore in allRestorableAssets)
{
if(self.isCancelled) // Check if we are cancelled
[super operationCancelled];
// Do we have enough disk space to restore asset
if([assetToRestore isEnoughDiskSpaceForRestoringAsset])
{
// If asset not already on device then download it
if(![self isAssetOnDevice:assetToRestore])
{
NSLog(@"Creating download task for %@", assetToRestore.md5);
// Asset does not exist locally so go ahead and download it
[assetToRestore createSessionDownloadTask];
}
else
{
NSLog(@"Item %@ already exists. Not restoring", assetToRestore.md5);
// Mark item as restored as it already exists locally
[assetToRestore markAsRestored];
// Continue checking for restorable assets
[[CTXCloudManager shared].restoreManager continueRestore];
}
}
else
{
// Mark in DB as Insuffient storage avaialble on local device
[assetToRestore markAsRestoreFailed:507];
NSLog(@"Postpone assset download (%@) until disk space becomes available", assetToRestore.md5);
}
}
is essentially
FOR(ALL ELEMENTS IN LIST) {
BEGIN TRANSACTION
CHANGE ELEMENT
COMMIT TRANSACTION
}
Which results in many many transactions; it's better if you move the begin and the commit around the for loop so that markAsRestored doesn't open its own tranasction per element
Zhuinden
on 16 Jan 2017
Hi @Zhuinden,
I noticed the issue was only happening when we had the UI open that shows a table view of the assets we are downloading.
We use the realm notification API to update the view as it changes. The code that populates the table for the view is the one below.
With the [realm refresh] removed the database grows as we are downloading the items referenced in the table as we update the bytes downloaded etc.
If I add [realm refresh] in the loop as shown below the database stays at 1.4 Megs.
How can this code lock up the table like this? Either way, adding the refresh seems to have fixed this instance of this problem.
Thanks
Donie
-(void) createCloudAssetListForTagcode:(NSString *)tagCode withResults:(id)results completion:(void(^)(void))completion
{
@autoreleasepool {
RLMRealm *realm = [RLMRealm defaultRealm];
[realm beginWriteTransaction];
for (NSDictionary *itemJson in results) {
// Ignore items marked for restore as they are in progress
CTXRestoreAsset *restoreAsset = [CTXRestoreAsset itemWithJSON:itemJson];
CTXRestoreAsset *existingAsset = [self restoreItemForMD5:restoreAsset.md5];
if(existingAsset.selectedForRestore == YES)
continue;
[realm addOrUpdateObject:restoreAsset];
[realm refresh];
}
[realm commitWriteTransaction];
completion();
}
}
Guys, I'm considering throwing out Realm as I cannot get it to behave itself. I've done everything by the book. The app is very much dependant on being concurrent with many threads running doing uploads and downloads.
Yesterday I though I has it tamed as I had combined more and more transactions together to avoid doing multiple begin/commit transactions per thread and this is causing me a headache to have to re-archiect the app.
It was behaving itself for a while and the database stayed at 320k. Then all of a sudden for no real apparent reason as nothing new was happening as regards what the threads are doing it started to increase the size of the database on disk. It went from 320k to over 100 megs in a about a minute as transaction were running as they always do.
It won't compact so they transactions are being pinned as you seem to call it. Why is it doing this and why do I have to care? Our whole project is in jeopardy due to this now. I can't easily switch to core data or some other database as it would cause us to have to migrate old clients to a new DB and we don't have time to code that.
How can I debug what is gone wrong in Realm? Every time I seem to fix one problem is starts leaking in a previously working thread.
I'm tearing my hair out at this stage!!!
IS there any way to debug your Realm database to tell me what transactions are being pinned so that I can see if I can re-organise it some way to stop it happening. This thing was supposed to be thread safe but it seems far from it unless I'm doing something really crazy and I don't think I am...
Please help...
doniekelly
on 19 Jan 2017
@donie-inhance I figured realm.refresh() after the transaction wouldn't do much. I'd typically put it after Realm.getDefaultInstance() if anywhere.
I'm not a Realm member, I use Realm-Android, but the core concepts are the same. I think I can give you some insight which might be able to help you figure out a way to reduce the occurrance of this problem.
Please make sure you read this blob of text thoroughly, everything in it is important to know and understand.
Realm is thread-safe but it does that by showing an "immutable view of the current version of the Realm at a given time on a given thread". This also means that while a Realm version exists at a given moment in time on a thread, it cannot be released. A version gets released if no Realm instances point to it on any thread. When this happens, the related data is cleared, and it gets reused by later versions of Realm. This "space" left behind is what is removed via Realm.compactRealm(), that way you don't need to "write into it again to reuse it".
Now, if you think about it, a new Realm version is created when you commit a transaction on any thread. Every committed transaction creates a new version..
Now if you have 32 threads and these threads randomly create new versions (in loops, so creating N new versions per N objects) and close/reopen Realm instances at "intermittent versions" while your loop creates ~100 Realm versions, you randomly end up retaining 32 different versions of the database at the same time!
Now if you have enough data, this means your database can take up 32x more space than it normally would.
So with that in mind, a Realm version is released when no threads point to it, right?
A Realm instance is updated in two cases:
- Realm.refresh() is called
- A transaction is opened and the Realm is put into "write mode", because you need to see the latest version of the Realm to start creating a new one
The Realm auto-updates because it sits on top of the runloop of the underlying framework (in Android the looper, in iOS the event loop); and this event loop is a queue which can receive messages, and can receive the message that "there was a commit in a background thread, please update your Realm instance and notify all Realm results/objects that they have been updated"
So you can do the following:
- call Realm.refresh() after
getDefaultInstance() - do not create multiple transactions per thread
- limit the number of concurrent threads so that you don't have the possibility of creating 32 versions of the Realm at the same time
- (use auto-updating
RunLoops instead of threads, although that seems unnecessarily hacky) - make sure that
Realm.getDefaultInstance()is wrapped in anautoreleasepoolwhen used withdispatch_async
P.S.: transactions don't get pinned. Realm instances (belonging to a given thread) do. But autoreleasepool { ... } should counteract that.
Zhuinden
on 19 Jan 2017
Sorry for a delay with reply @donie-inhance. We will get back to you as soon as possible. Please provide a version of Realm/Xcode you use. It would also be great if you can send us a simplified Xcode project that reproduces this case.
stel
on 19 Jan 2017
Everything is wrapped in a autorelease pool as I've split out the code into a separate api and hence why some calls are wrapped in their own begin/end. It seems like that was a bad idea and I need to lump all the code into one place or pass the realm around for the current threads to avoid creating new transactions.
This seems really unwieldy as it will now require a lot of rework to make it such that every thread can only contain one transaction commit per run.
The threads can run for some time and it needs to update at start and at the end. Some of the threads interact with the Photo library and some of the callbacks out of that come back on a different thread so it makes it even more complicated.
I just assumed Realm made it easier. I presume it does for light work but with so many threads going around I've architected this wrong I think based on how I need to interact with Realm.
doniekelly
on 19 Jan 2017
@stel I'm not sure I can reproduce this problem easily as the project is a bit of a monster build wise etc. I've tried to see if I can give you access to the source in GitHub but I'm not allowed. If I could would that help?
doniekelly
on 19 Jan 2017
@donie-inhance I'm not sure if it's a good idea in this case, let's start from the Realm version you use.
stel
on 19 Jan 2017
2.2.0
doniekelly
on 19 Jan 2017
Any chance you guys could do up an example of an operation queue with multiple parallel threads writing something to the database? Just increment a number. And show us the correct implementation of what you are talking about.
I've tried a few things. Even calling invalidate when finished with the realm. But for some reason I get the realm is invalidated in the next transaction on that thread.
It's getting very frustrating...
...Because everything I'm trying is having no effect. I'm calling refresh before each transaction to bring the database up to date. I have the autorelease pool in place. I am calling refresh during long operations.
At the moment the database is growing at about 10 megs per second while I'm doing these operations that I've checked and double checked.
I'm obviously missing something here?
doniekelly
on 19 Jan 2017
@donie-inhance, if you're to get us access to the source it'd certainly help us understand where things are going wrong. If that's not possible, and you've indicated it may not be, then a prebuilt copy of the app that reproduces the problem would also allow us to look into it.
bdash
on 19 Jan 2017
Hi @bdash
OK, some progress finally. I was running with the old database in place but I just now removed the app and started from scratch. The database has grown to 1.2 megs and and held there after uploading and downloading across on average 16 threads.
I've seen no growth at all apart from the amount of items in the table which is around 1200 rows.
Is this possible that the old database having realms pinned was causing further pinning?
doniekelly
on 19 Jan 2017
It is possible that the old Realm file itself was problematic, but only if you also had it open in another application at the same time as you were doing your testing. This can happen if you're watching your Realm contents via the Realm Browser.
When we speak of a version in the Realm being pinned, what that means is that somewhere in a running application there is a read transaction open that points at that version of the data. A given version is unpinned when all read transactions that were pointing at it have advanced to newer versions of the data (typically via the Realm being automatically refreshed to the latest version on each runloop iteration).
The list of versions that are pinned are tracked in the lock file. This ensures that if multiple processes are working with a given file, they won't try to reclaim versions that the other process is still making use of. The fact that the set of pinned versions is tracked on disk does mean that if an application crashes (or is force quit) with a version pinned, the version will remain marked as pinned in the lock file despite the process no longer having an open read transaction (since it is no longer running). This pinned state can only be reset the next time the Realm file is opened _while no other applications have the Realm file open_, as at that point we know that no-one else can possibly have any open read transactions on the Realm file.
bdash
on 19 Jan 2017
Ok, I now added this code back in and the problem is back
[self.ctxCloudBackupQueue addOperationWithBlock:^{
NSString *md5 = task.taskDescription;
NSUInteger taskID = task.taskIdentifier;
CTXUploadAsset* uploadedAsset= [[CTXCloudPersistence shared] uploadItemForMD5:md5 withTaskID:taskID];
if(uploadedAsset)
{
[[RLMRealm defaultRealm] refresh];
[[RLMRealm defaultRealm] transactionWithBlock:^{
if(!uploadedAsset.invalidated)
uploadedAsset.bytesUploaded = uploadedAsset.bytesUploaded + bytesSent;
}];
float percentage = ((float)uploadedAsset.bytesUploaded/(float)uploadedAsset.assetSize)*100;
if(percentage > 95.0)
NSLog(@"Uploading: %@/%@ %.0f%%", _Settings.deviceTag, task.taskDescription, percentage);
}
}];
@bdash I never have seen the file reduce in size since this problem has started. I even compress the file when the app starts but it's having no effect.
doniekelly
on 19 Jan 2017
Just wondering if there is any way to see what the lock file holds? Would it tell me anything about what locks are being held?
doniekelly
on 19 Jan 2017
One thing I notice is the absence of an autorelease pool around that block of work. You could be seeing a RLMRealm or other managed object making it into a dispatch queue's autorelease pool, keeping a version pinned for longer than expected.
Is ctxCloudBackupQueue a concurrent queue? If so, what is its max concurrent operation count set to? How often is that block being submitted to the queue?
Is it possible that the block is being submitted to a concurrent queue a large number of times and then being executed a number of times in parallel?
I never have seen the file reduce in size since this problem has started. I even compress the file when the app starts but it's having no effect.
What do you mean by "compress the file"?
Just wondering if there is any way to see what the lock file holds? Would it tell me anything about what locks are being held?
There's no easy way to see which versions are in use in the lock file. And if there was, you'd still have the challenge of determining what behavior in your code is resulting in those versions remaining pinned.
bdash
on 19 Jan 2017
When we speak of a version in the Realm being pinned, what that means is that somewhere in a running application there is a read transaction open that points at that version of the data. A given version is unpinned when all read transactions that were pointing at it have advanced to newer versions of the data (typically via the Realm being automatically refreshed to the latest version on each runloop iteration).
@bdash Does that mean every time I read data I'm pinning my thread? How do I release that lock? I do a lot of reads throughout my code. Some of those reads are on background threads. Well, probably most. If I didn't do a commit trans or a refresh on that thread after a read would that pin the read?
doniekelly
on 19 Jan 2017
How do I release that lock?
On background threads, the answer is that read transactions end once the RLMRealm is either invalidated or deallocated (note that the RLMRealm is kept alive by accessors such as RLMObject or RLMResults instances). Hence the advice to ensure that all work on background queues is contained within explicit autorelease pools to bound the lifetime of the objects involved in the work, ensuring that the RLMRealm will be deallocated and any read transaction closed at the conclusion of the work, rather than at some indeterminate point in the future when the dispatch queue eventually drains its autorelease pool.
On the main thread, the read transaction is automatically advanced to the latest version on each runloop iteration. This means it's somewhat less important to worry about object lifetimes on the main thread.
bdash
on 19 Jan 2017
One thing I notice is the absence of an autorelease pool around that block of work. You could be seeing a RLMRealm or other managed object making it into a dispatch queue's autorelease pool, keeping a version pinned for longer than expected.
Sorry, I didn't show all the code, there is one in that method just outside that block I show there.
Is ctxCloudBackupQueue a concurrent queue? If so, what is its max concurrent operation count set to? How often is that block being submitted to the queue?
I've set the maxConcurrentOperationCount to 1 for the mo. It triggers each time the callback fro the URL session downloads a block of data. It's kinda experimental to see would it make a difference if moved off the thread it was on. I thought it might have been on the main thread from that callback but it's not. I can remove it as it doesn't do anything constructive.
What do you mean by "compress the file"?
I meant compact, this is the code below. Not having much effect in this app. It normally does.
-(void) compactDatabase
{
@autoreleasepool {
// Compacting the database reduces the file size taken up by the DB
RLMRealm *realm = [RLMRealm defaultRealm];
NSURL *origDatabase = realm.configuration.fileURL;
NSURL *tmpDatabase = [NSURL URLWithString:[NSString stringWithFormat:@"file://%@/%@", NSTemporaryDirectory(), @"tmpRealm.db"]];
NSError *error;
NSDictionary *attribs = [[NSFileManager defaultManager] attributesOfItemAtPath:origDatabase.path error:nil];
NSLog(@"Size before compacting: %@", [attribs valueForKey:NSFileSize]);
[[NSFileManager defaultManager] removeItemAtURL:tmpDatabase error:&error];
[[RLMRealm defaultRealm] writeCopyToURL:tmpDatabase encryptionKey:nil error:&error];
// Move the copy back to where the original was
if(!error)
{
[[NSFileManager defaultManager] removeItemAtURL:origDatabase error:&error];
[[NSFileManager defaultManager] moveItemAtURL:tmpDatabase toURL:origDatabase error:&error];
}
else
[[NSFileManager defaultManager] removeItemAtURL:tmpDatabase error:&error];
attribs = [[NSFileManager defaultManager] attributesOfItemAtPath:origDatabase.path error:nil];
NSLog(@"Size after compacting: %@", [attribs valueForKey:NSFileSize]);
// Set file protection on the database to avoid locking when device locked
NSURL *folder = origDatabase.URLByDeletingLastPathComponent;
[[NSFileManager defaultManager] setAttributes:@{NSFileProtectionKey: NSFileProtectionCompleteUntilFirstUserAuthentication} ofItemAtPath:folder.path error:nil];
}
}
One thing I notice is the absence of an autorelease pool around that block of work. You could be seeing a RLMRealm or other managed object making it into a dispatch queue's autorelease pool, keeping a version pinned for longer than expected.
Sorry, I didn't show all the code, there is one in that method just outside that block I show there.
I'm not sure what you mean by this. Is the autorelease pool inside the block you're passing to -addOperationWithBlock:?
bdash
on 19 Jan 2017
I'm not sure what you mean by this. Is it inside the block you're passing to -addOperationWithBlock:?
Here is the full code...
- (void)URLSession:(NSURLSession *)session task:(NSURLSessionTask *)task didSendBodyData:(int64_t)bytesSent totalBytesSent:(int64_t)totalBytesSent totalBytesExpectedToSend:(int64_t)totalBytesExpectedToSend
{
@autoreleasepool {
if([[CTXCloudPersistence shared] isQueueOnStartupHold])
return;
[self.ctxCloudBackupQueue addOperationWithBlock:^{
NSString *md5 = task.taskDescription;
NSUInteger taskID = task.taskIdentifier;
CTXUploadAsset* uploadedAsset= [[CTXCloudPersistence shared] uploadItemForMD5:md5 withTaskID:taskID];
if(uploadedAsset)
{
[[RLMRealm defaultRealm] refresh];
[[RLMRealm defaultRealm] transactionWithBlock:^{
if(!uploadedAsset.invalidated)
uploadedAsset.bytesUploaded = uploadedAsset.bytesUploaded + bytesSent;
}];
float percentage = ((float)uploadedAsset.bytesUploaded/(float)uploadedAsset.assetSize)*100;
if(percentage > 95.0)
NSLog(@"Uploading: %@/%@ %.0f%%", _Settings.deviceTag, task.taskDescription, percentage);
}
}];
}
}
Thanks. The autorelease pool around the body of the function only helps for objects that are autoreleased within the function. The block you're submitting to the operation queue needs its own autorelease pool, since its body is evaluated outside the context of your -URLSession:task:didSendBodyData:… function.
bdash
on 19 Jan 2017
OK, Let me check if I've done that throughout the app, that was a kind of temporary fix there. I'm just checking all my operations have autorelease pools.
doniekelly
on 19 Jan 2017
@donie-inhance the docs do say that you need to wrap the asynchronous operation in the autoreleasepool, not when you add it to the queue - see example
Zhuinden
on 20 Jan 2017
@Zhuinden Yes, I understand that and that's what I'm doing as per the example. Apart from this below...
@bdash I found an operation that was created in a nested callback that I needed to add as a dependency as the Photo library is asynchronous for video export and I created a new operation block that was dependant on that being complete. There was no autorelease around the operation block so that may have been the culprit. I also put autoreleases around all helper function that read from the realm just to be sure.
Let's see if I can make it grow again...
doniekelly
on 20 Jan 2017
@bdash I have a few instances where I am be doing the following in a single thread due to the complexity of the code and the fact that some of these are helpers.
@autoreleasepool {
RLMRealm *realm = [RLMRealm defaultRealm];
[realm refresh]; [realm beginWriteTransaction];
........{first nested op }
[realm commitWriteTransaction];
}
@autoreleasepool {
RLMRealm *realm = [RLMRealm defaultRealm];
[realm refresh]; [realm beginWriteTransaction];
........{second nested op }
[realm commitWriteTransaction];
}
Is it ok to use multiple transactions in the one thread if I release them as above?
doniekelly
on 20 Jan 2017
Yes, it's fine to do multiple transactions in one thread. It's usually preferable to combine writes into a single transaction as there's a certain amount of per-transaction overhead, but it shouldn't affect correctness to use multiple transactions like that.
bdash
on 20 Jan 2017
ok, good !
doniekelly
on 20 Jan 2017
I had a similar rampant growth problem recently but for me the cause was very simple. I replaced the default.realm file but left the existing default.realm.lock file in the folder. My db went from 2mb to 147 in under a minute. I removed both files; copied an old db into place and everything was fine. Is it possible you have an out of synch lock file?
Mozahler
on 20 Jan 2017
No. I can reproduce after a fresh install.
doniekelly
on 20 Jan 2017
Ok. Sorry I couldn't help.
Mozahler
on 20 Jan 2017
Just to follow up. Taking on all the comments and understanding more of why it leaks I think I've solved the majority of issues. I noted it still climbs under certain circumstances and I'm trying to locate these but in general, it has held its size under almost all circumstances.
Hoping I can find the remaining leaks but that's an exercise for me. Thanks for all your help here guys, you saved my sanity as I was under pressure to finish this project.
d0n13
on 22 Jan 2017
Thanks, @d0n13! I'm closing this issue for now, please feel free to reopen it or create a new one if you need any further help.
stel
on 23 Jan 2017
Thanks Stel, the problems went away once I got all bits under control. Thanks for the help
d0n13
on 25 Feb 2017
Related issues
TheHmmka
·
3Comments
jpsim
·
3Comments
dmorrow
·
3Comments
matteodanelli
·
3Comments
duribreux
·
3Comments