Robomongo: Scripts end prematurely and randomly when using loop

[Galan2X]

When running a script within Robomongo that includes document enumeration, such as:
db.MyCollection.find().forEach(function(doc) {
// do some work, or print some data
});

If the amount of work is not trivial (such as a 'printjson(doc)' only) and the collection has hundreds of documents, the enumeration ends at random times before the end is reached, as if the cursor is sometimes advanced more than one step, or it believes it has reached the end. There are no errors thrown. The same script run locally on the server with the MongoDB shell does not exhibit this behavior.

This is for Robomongo 0.9.0-RC8 on Mac. It did not occur on earlier versions of Robomongo, like probably pre-0.9.0.

[simsekgokhan]

Hi @Galan2X, we have reproduced the issue, we are now investigating the problem.
Thanks a lot for reporting.

Problem Summary:

Long running scripts (inserting) from shell ends before the end of script.

Investigation Details:

Steps to reproduce:
var document = {"link": {"status":65},"clients": {"enddt" : "test", "type" : "testtype", "clentid" : 1}};
for (var j=0; j<500; j++) // for j=100 working
{
db.aa6.insert(document)
}

Next Actions:

• Investigate logs.
• Check roboshell timeouts.

[sthammadi-spsc]

Just to add, this issue also happen on Windows OS.

[simsekgokhan]

Thanks for the input Sandeep. We will investigate the issue as soon as possible.

[juliashibalko]

@Galan2X, @sthammadi-spsc, @oschrenk For Robomongo _0.9.x_ version as workaround let me suggest you next solution:

1. Make sure Robomongo is closed

2. Open robomongo.json configuration file .

Please mind that for Robomongo _0.9.x_ version configuration file location differs from app _0.8.x_ version:

  0.9.x
  C:\Users\<user>\.config\robomongo\0.9\robomongo.json
  0.8.x
  C:\Users\<user>\.config\robomongo\robomongo.json   
MAC
    0.9.x
    /Users/<user>/.config/robomongo/0.9/robomongo.json
    0.8.x
   /Users/<user>/.config/robomongo/robomongo.json     
Linux
    0.9.x
    /home/<user>/.config/robomongo/0.9/robomongo.json
    0.8.x
    /home/<user>/.config/robomongo/robomongo.json

1. Change "shellTimeoutSec" attribute value to higher number in seconds. (i.e. "shellTimeoutSec" : 60)
2. Save config file and re-launch Robomongo app.
3. Run necessary script.
   If script isn't executed entirely, try to increase shellTimeoutSec value.
   Please let me know in case of any questions

[sthammadi-spsc]

@Galan2X, thanks for the workaround. Unfortunately, I couldn't find 'shellTimeoutSec' attribute in my robomongo.json file. I'm assuming I could add it in.

Below is my config:
{
"autoExec" : true,
"autoExpand" : false,
"autocompletionMode" : 1,
"batchSize" : 50,
"connections" : [
// Removed other connections here
{
"connectionName" : "Localhost",
"credentials" : [
{
"databaseName" : "admin",
"enabled" : false,
"mechanism" : "SCRAM-SHA-1",
"userName" : "",
"userPassword" : ""
}
],
"defaultDatabase" : "",
"serverHost" : "localhost",
"serverPort" : 27017
}
],
"disableConnectionShortcuts" : false,
"lineNumbers" : false,
"loadMongoRcJs" : false,
"style" : "Native",
"textFontFamily" : "",
"textFontPointSize" : -1,
"timeZone" : 0,
"toolbars" : {
"connect" : true,
"exec" : true,
"explorer" : true,
"logs" : false,
"open_save" : true
},
"uuidEncoding" : 0,
"version" : "2.0",
"viewMode" : 1
}

[juliashibalko]

@sthammadi-spsc For version _0.9.0-RC8_ or _0.9.0-RC9_ this parameter usually set by default.
Let me offer you just to double-check version of Robomongo using and configuration file location on C:\Users\<user>\.config\robomongo\0.9\robomongo.json.
If there is still no "shellTimeoutSec" parameter, you could add both missed attributes manually after "loadMongoRcJs" : false, line:

 "mongoTimeoutSec" : 10,
 "shellTimeoutSec" : 60,

Then save _robomongo.json_ file and start Robomongo.
Please let me know if you have any questions.

[teliatko]

@juliashibalko hi, what does the mongoTimeoutSec mean? How is this timeout different from shellTimeoutSec? Where is which one used?

Btw. can you check my comment in https://github.com/paralect/robomongo/issues/1098#issuecomment-231986010

Thanks.

[juliashibalko]

Hi @teliatko,
Thanks for interest in Robomongo! As you know Robomongo embeds mongo shell. So we've set two different timeout value attributes for MongoDB shell and for Robomongo application itself. Therefore shellTimeoutSec is used for scripts running in the shell only (e.g. db.getCollection().find() executing) and mongoTimeoutSec is used for actions executed outside the shell (e.g. expanding collections list).
Hope I've answered your question!

[snarlynarwhal]

I just ran into a similar, if not the same problem. When attempting to iterate through a collection of 100,000 documents, it would stop about 1/4 of the way through. I ported my code to the C# driver and it worked like a charm.

[ClarenceL]

Ya it happens to me when I'm running a long mongo script, usually iterating over a cursor and updating records, kind of annoying to have to sit there and re-execute the script multiple times.

It seems to terminate after a random amount of time too.

[juliashibalko]

@ClarenceL thanks for reporting! Yes, there is an issue about random terminating (please check #1232). We are investigating the problem.

[jfamme]

So, upping my timeout values to 300 did the trick for me...I guess the ask for me would be: Obviously the application can properly complete my query, if given enough time...it would be nice if the app told you that it timed out, and maybe presented a way to increase that wait, without having to manually find and edit config files. Thanks!

[simsekgokhan]

Hi @jfamme, thanks a lot for suggestions. All make perfect sense.
We have created two issues for this problem. We will work on these when we will have chance.
https://github.com/paralect/robomongo/issues/1254
https://github.com/paralect/robomongo/issues/1253

[snarlynarwhal]

This has been an issue for about 8 months.. and it's a pretty significant issue. :/ Can we get a status update? Has any progress been made?

[simsekgokhan]

Hi @snarlynarwhal , currently priority is delivering critical features like Replica Sets and Export/Import. And the number of contributors of Robomongo is not too many. This issue is tagged as high vote and major, for sure it will be one of the first to work after delivering features mentioned above.

The workaround here https://github.com/paralect/robomongo/issues/1106#issuecomment-230258348 seems to be working for lots people. Can you let us know if this helps you for now? If you have another problem, we will be happy to help.

[snarlynarwhal]

Okay sounds good, thank you very much! :)

[jeremycooper1]

Hi I tried up shellTimeoutSec" : 120 - the query I'm trying to run in version 9 only takes 6 seconds, yet it still won't appear running:
db.currentOp().inprog.forEach( function(op) { if(op.secs_running > 5) printjson(op); } )
but does in version 8 - not sure this is the same issue of this thread

[juliashibalko]

@jeremycooper1 Yes, you are correct - it's another issue. So let's continue investigation in scope of #1259 card.

[jsheely]

@simsekgokhan I'd consider that this is a higher priority issue than that of new features.

Given this example:
In its current form you run a script that loops over a collection of say 10,000 items and you insert them into another collection. The script says it ran successfully, no warnings, no errors.. But you only get 300 items in the new collection.

Now you're in a position where you could lose a significant amount of data. This is something I would urge you to re-consider the importance of if time permits.

The current work around still has some significant issues. Unless you set these timeouts to be in terms of hours it is still possible to get false positive of completion if your script would happen to run longer than the timeout.

[simsekgokhan]

Hi @jsheely, I totally agree with you. Thanks for the example and reminding us.

[roberterdin]

I agree with @jsheely. The fact that there is no warning is very dangerous. I was just seconds away from a data loss that would have been very expensive to us. Data that was purchased from a third party was almost lost because two collections were supposedly merged and the obsolete collection deleted. Luckily I double-checked with a document at the end of the target collection so this was avoided.

[snarlynarwhal]

I think this needs to be resolved asap also.

[gauravkalal]

@juliashibalko This solution works. Thank you.

[RokoMijic]

It might be a good idea to edit the error message that is shown to make the user aware of what happened and what the resolution is. I can't imagine that it would take long to edit the text of the error message to say something more sensible.

[simsekgokhan]

@RokoMijic , thanks a lot for your interest and suggestion. We will consider.

[simsekgokhan]

Hoping the related code changes for this problem and UI option to configure shell timeout which are included in the latest version Robo 3t 1.1 will solve this problem.

More:
Blog section: Fix for Robo 3T Shell Timeout Issue
All sections: http://blog.robomongo.org/robomongo-is-robo-3t

[RokoMijic]

@simsekgokhan Looks good!

[afirdousi]

In 2019 at Robo 3T 1.2.1, still seeing the same issue while running aggregation queries. My shell timeout is set to 360.