Salt: bson.errors.InvalidDocument: key must not contain '.'

[krishneil]

hi

i am seeing bson.errors.InvalidDocument: key must not contain '.' errors when running salt-run jobs.active . i am using mongodb for storing jobs and returners. Any help on how to solve this other than changing my minions name ?

InvalidDocument: key 'auto.foo.bar.com' must not contain '.'
Traceback (most recent call last):
  File "/usr/bin/salt-run", line 10, in <module>
    salt_run()
  File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 354, in salt_run
    client.run()
  File "/usr/lib/python2.7/dist-packages/salt/cli/run.py", line 41, in run
    runner.run()
  File "/usr/lib/python2.7/dist-packages/salt/runner.py", line 181, in run
    False)  # Don't daemonize
  File "/usr/lib/python2.7/dist-packages/salt/client/mixins.py", line 407, in _proc_function
    return self.low(fun, low)
  File "/usr/lib/python2.7/dist-packages/salt/client/mixins.py", line 360, in low
    mminion=self.mminion,
  File "/usr/lib/python2.7/dist-packages/salt/utils/job.py", line 88, in store_job
    mminion.returners[fstr](load)
  File "/usr/lib/python2.7/dist-packages/salt/returners/mongo_future_return.py", line 188, in returner
    mdb.saltReturns.insert_one(sdata.copy())
  File "/usr/local/lib/python2.7/dist-packages/pymongo/collection.py", line 549, in insert_one
    return InsertOneResult(self._insert(sock_info, document),
  File "/usr/local/lib/python2.7/dist-packages/pymongo/collection.py", line 469, in _insert
    check_keys, manipulate, write_concern, op_id)
  File "/usr/local/lib/python2.7/dist-packages/pymongo/collection.py", line 453, in _insert_one
    check_keys=check_keys)
  File "/usr/local/lib/python2.7/dist-packages/pymongo/pool.py", line 206, in command
    self._raise_connection_failure(error)
  File "/usr/local/lib/python2.7/dist-packages/pymongo/pool.py", line 333, in _raise_connection_failure
    raise error
bson.errors.InvalidDocument: key 'auto.foo.bar.com' must not contain '.'

 salt --versions-report
Salt Version:
           Salt: 2015.8.1

Dependency Versions:
         Jinja2: 2.7.2
       M2Crypto: Not Installed
           Mako: 0.9.1
         PyYAML: 3.10
          PyZMQ: 14.4.0
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.4
           cffi: Not Installed
       cherrypy: 3.2.2
       dateutil: 1.5
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
        libnacl: 1.4.3
   msgpack-pure: Not Installed
 msgpack-python: 0.3.0
   mysql-python: 1.2.3
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: 0.8.2
        timelib: Not Installed

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.16.0-53-generic
         system: Ubuntu 14.04 trusty

[krishneil]

forgot to mention:
you need to have an active job and a minion with dot(.) in the name to see this.

i used the following to generate a job:

import salt.client
local = salt.client.LocalClient()
local.run_job('*', 'test.sleep', [600])

[jfindlay]

@krishneil, thanks for the report. This looks like an issue with pymongo. I am unfamiliar with pymongo and why it would have such a restriction. Do you have any ideas here, @rhealitycheck, @Lothiraldan?

[rhealitycheck]

hmm @krishneil that's strange because we have minions with dots in the name like linuxbox123.company.com and those have been fine in the returner. the mongo_future_return specifically removes dots from any keys so i would have thought this would have been handled. i'll try to take a look over the weekend and see if i find anything. can you also post your pymongo version, salt doesn't report that out when it lists out its libraries (just want to rule it out as a factor).

actually giving this a bit more thought, i'm puzzled by your set up because it's throwing an error on keys. your minion value should never be a key it should only be a value so i'm not sure how that's happening. can you tell me what version of mongo you're running as well? //edit: oh wait, i see you're using a runner, i haven't tested what the returns are for the runners, i'll look into that.

@jfindlay it's not pymongo it's actually mongodb which doesn't allow dots in keys. in fact most nosql dbs are like this and do not allow dots in keys, elasticsearch is moving to this model as well as of 2.2 i believe

[krishneil]

pymongo version is 3.1.1.

Just a note: Job and returners logs is successfully being added to the Mongodb for the job i initiated but the above error only pops up if i have a active job and run "salt-run jobs.active"

[rhealitycheck]

what about your mongodb version? i want to try to clone most of your environment when i look into this.

[krishneil]

using Mongodb version 3.0.7,

Salt master config:

return: mongo
master_job_cache: mongo
mongo.db: salt-jobs
mongo.host: 127.0.0.1
mongo.user:
mongo.password:
mongo.port: 27017
mongo.indexes: true

[kenju254]

@krishneil were you able to solve this eventually . I am having the same with Pymongo

[sjatkins]

I am also having this problem when storing a sub document dictionary value using pymongo where some of the keys have a '.' in them. I am very surprised by this. It is not a Salt problem obviously.

[sara-02]

I am also getting the same error with a key name 'socket.io' it is giving the following error.

bson.errors.InvalidDocument: key 'socket.io' must not contain '.'

I am using Mongodb: version 3.2.8
Pymongo: Version: 3.2.2 (pip installed)

[sara-02]

Setting check_keys=False while inserting seems to be a work around, as per this Stack overflow answer

[kedare]

How the returner can be configured to use the check_keys=False ?
Maybe today there is a better workaround ? (Salt 2017.7.2)

[sara-02]

@kedare Not working on Mongo anymore, but if see the Stackoverflow link I posted, it has a sample snippet of how to use it.
And it seems to be getting used here https://github.com/mongodb/mongo-python-driver/blob/master/pymongo/collection.py#L591

[greed2411]

check this out, Im not sure if its of any help tho.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.