Couchdb: couch_peruser is broken in 2.1.0

[iamdanthedev]

So I've compiled 2.1.0 under docker image. Launched it. Created DBs, switch cors on, set peruser = true in local.ini. Commit, run the newly commited image, couchdb crashes :-(

[error] 2017-08-12T17:18:07.625357Z nonode@nohost <0.194.0> -------- CRASH REPORT Process  (<0.194.0>) with 0 neighbors exited with reason: {{shutdown,{failed_to_start_child,couch_secondary_services,{shutdown,{failed_to_start_child,couch_peruser,{badarg,[{ets,member,[mem3_openers,<<"_users">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,for_db,2,[{file,"src/mem3_shards.erl"},{line,59}]},{mem3_shards,local,1,[{file,"s..."},...]},...]}}}}},...} at application_master:init/4(line:133) <= proc_lib:init_p_do_apply/3(line:237); initial_call: {application_master,init,['Argument__1','Argument__2',...]}, ancestors: [<0.193.0>], messages: [{'EXIT',<0.195.0>,normal}], links: [<0.193.0>,<0.7.0>], dictionary: [], trap_exit: true, status: running, heap_size: 987, stack_size: 27, reductions: 157
[info] 2017-08-12T17:18:07.626827Z nonode@nohost <0.7.0> -------- Application couch exited with reason: {{shutdown,{failed_to_start_child,couch_secondary_services,{shutdown,{failed_to_start_child,couch_peruser,{badarg,[{ets,member,[mem3_openers,<<"_users">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,for_db,2,[{file,"src/mem3_shards.erl"},{line,59}]},{mem3_shards,local,1,[{file,"s..."},...]},...]}}}}},...}
[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
{"Kernel pid terminated",application_controller,"{application_start_failure,couch,{{shutdown,{failed_to_start_child,couch_secondary_services,{shutdown,{failed_to_start_child,couch_peruser,{badarg,[{ets,member,[mem3_openers,<<\"_users\">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,\"src/mem3_shards.erl\"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,\"src/mem3_shards.erl\"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,\"src/mem3_shards.erl\"},{line,378}]},{mem3_shards,for_db,2,[{file,\"src/mem3_shards.erl\"},{line,59}]},{mem3_shards,local,1,[{file,\"src/mem3_shards.erl\"},{line,156}]},{couch_peruser,init,0,[{file,\"src/couch_peruser.erl\"},{line,58}]},{couch_peruser,init,1,[{file,\"src/couch_peruser.erl\"},{line,187}]}]}}}}},{couch_app,start,[normal,[]]}}}"}

Crash dump:
https://gist.github.com/rasdaniil/8fe6e94a68e5730f56404d65dcd2ff9e

Dockerfile
https://gist.github.com/rasdaniil/fada64e4015ba1a1a3a651a3ad41e4fa

[wohali]

This is not a bug with CouchDB, but your Docker setup.

The _users database must exist before couch_peruser can be enabled.

[iamdanthedev]

Thanks for quick reply @wohali
I've just double checked (done it again) - I'm creating the _users db before setting couch peruser enable = true.. So the db exists prior to setting the peruser setting and restarting the service
I do this not in Dockerfile , but manually, committing changes to the image

But still the error exists
Error in process <0.325.0> with exit value: {database_does_not_exist,[{mem3_shards,load_shards_from_db,"_users",[{file,"src/mem3_shards.erl"},{line,403}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk...

Btw!!, is there an option to create databases in the Dockerfile ?

Thanks

[wohali]

That error is saying you still didn't create the _users database.

I know zero about the Docker setup. I can tell you that testing it locally I do not see these errors.

[iamdanthedev]

You see, I checked sources, if the auth db does not exist, it should show error:
couch_peruser can't proceed as underlying database (name) is missing, disables itself.

Now if I change the auth db to a db which really doesn't exist like

[couch_httpd_auth]
authentication_db = i_dont_exist

then I can see a proper error
[warning] 2017-08-14T07:27:10.966328Z nonode@nohost <0.231.0> -------- couch_peruser can't proceed as underlying database (i_dont_exist) is missing, disables itself.

Now if I either create the i_dont_exist db or switch auth db back to _users (which of course exists before settings peruser to true), in this case couch db doesn't show a proper error like above but _crashes_ with the error in the first message

So it's unfair to close the issue

[garrensmith]

Can you recreate this issue outside of docker? If you cannot it is a docker issue.

On 14 Aug 2017, 9:31 AM +0200, Dan Goody [e-mail], wrote:

You see, I checked sources, if the auth db does not exist, it should show error:
couch_peruser can't proceed as underlying database (name) is missing, disables itself.
Now if I change the auth db to a db which really doesn't exist like
[couch_httpd_auth]
authentication_db = i_dont_exist
then I can see a proper error
[warning] 2017-08-14T07:27:10.966328Z nonode@nohost <0.231.0> -------- couch_peruser can't proceed as underlying database (i_dont_exist) is missing, disables itself.
Now if I either create the i_dont_exist db or switch auth db back to _users (which of course exists before settings peruser to true), in this case couch db doesn't show a proper error like above but crashes with the error in the first message
So it's unfair to close the issue
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[iamdanthedev]

Yes alright, I'll give it a try

On 14 August 2017 at 10:42, garren smith [e-mail] wrote:

Can you recreate this issue outside of docker? If you cannot it is a
docker issue.

On 14 Aug 2017, 9:31 AM +0200, Dan Goody [e-mail],
wrote:

You see, I checked sources, if the auth db does not exist, it should
show error:
couch_peruser can't proceed as underlying database (name) is missing,
disables itself.
Now if I change the auth db to a db which really doesn't exist like
[couch_httpd_auth]
authentication_db = i_dont_exist
then I can see a proper error
[warning] 2017-08-14T07:27:10.966328Z nonode@nohost <0.231.0> --------
couch_peruser can't proceed as underlying database (i_dont_exist) is
missing, disables itself.
Now if I either create the i_dont_exist db or switch auth db back to
_users (which of course exists before settings peruser to true), in this
case couch db doesn't show a proper error like above but crashes with the
error in the first message
So it's unfair to close the issue
—
You are receiving this because you are subscribed to this thread.

Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/couchdb/issues/749#issuecomment-322121068, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AXVWwlqtZVC-1ZMDjtG6yhnDctmfH5P4ks5sX_pagaJpZM4O1obs
.

[iamdanthedev]

Unfortunately, after repeating the steps on my local ubuntu machine (without docker), the problem persists
If the auth db exists (_users or custom defined), couchdb then crashes on start
Thanks for your time
Dan

danthedev@danthedevpc:~/Projects/github/couchdb/rel/couchdb$ bin/couchdb 
[info] 2017-08-14T08:15:09.581860Z couchdb@localhost <0.9.0> -------- Application couch_log started on node couchdb@localhost
[info] 2017-08-14T08:15:09.588925Z couchdb@localhost <0.9.0> -------- Application folsom started on node couchdb@localhost
[info] 2017-08-14T08:15:09.653730Z couchdb@localhost <0.9.0> -------- Application couch_stats started on node couchdb@localhost
[info] 2017-08-14T08:15:09.653848Z couchdb@localhost <0.9.0> -------- Application khash started on node couchdb@localhost
[info] 2017-08-14T08:15:09.667989Z couchdb@localhost <0.9.0> -------- Application couch_event started on node couchdb@localhost
[info] 2017-08-14T08:15:09.675875Z couchdb@localhost <0.9.0> -------- Application ibrowse started on node couchdb@localhost
[info] 2017-08-14T08:15:09.696017Z couchdb@localhost <0.9.0> -------- Application ioq started on node couchdb@localhost
[info] 2017-08-14T08:15:09.696075Z couchdb@localhost <0.9.0> -------- Application mochiweb started on node couchdb@localhost
[info] 2017-08-14T08:15:09.710956Z couchdb@localhost <0.204.0> -------- Apache CouchDB 2.1.0-f6156c3d1 is starting.

[info] 2017-08-14T08:15:09.711073Z couchdb@localhost <0.205.0> -------- Starting couch_sup
[error] 2017-08-14T08:15:09.863924Z couchdb@localhost <0.217.0> -------- CRASH REPORT Process  (<0.217.0>) with 0 neighbors exited with reason: bad argument in call to ets:member(mem3_openers, <<"users">>) at mem3_shards:maybe_spawn_shard_writer/3(line:486) <= mem3_shards:load_shards_from_db/2(line:389) <= mem3_shards:load_shards_from_disk/1(line:378) <= mem3_shards:for_db/2(line:59) <= mem3_shards:local/1(line:156) <= couch_peruser:init/0(line:58) <= couch_peruser:init/1(line:187); initial_call: {couch_peruser,init,['Argument__1']}, ancestors: [couch_secondary_services,couch_sup,<0.204.0>], messages: [], links: [<0.213.0>,<0.93.0>], dictionary: [], trap_exit: false, status: running, heap_size: 2586, stack_size: 27, reductions: 2273
[error] 2017-08-14T08:15:09.865047Z couchdb@localhost <0.213.0> -------- Supervisor couch_secondary_services had child couch_peruser started with couch_peruser:start_link() at undefined exit with reason bad argument in call to ets:member(mem3_openers, <<"users">>) at mem3_shards:maybe_spawn_shard_writer/3(line:486) <= mem3_shards:load_shards_from_db/2(line:389) <= mem3_shards:load_shards_from_disk/1(line:378) <= mem3_shards:for_db/2(line:59) <= mem3_shards:local/1(line:156) <= couch_peruser:init/0(line:58) <= couch_peruser:init/1(line:187) in context start_error
[error] 2017-08-14T08:15:09.866105Z couchdb@localhost <0.205.0> -------- Supervisor couch_sup had child couch_secondary_services started with couch_secondary_sup:start_link() at undefined exit with reason {shutdown,{failed_to_start_child,couch_peruser,{badarg,[{ets,member,[mem3_openers,<<"users">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,for_db,2,[{file,"src/mem3_shards.erl"},{line,59}]},{mem3_shards,local,1,[{file,"src/mem3_shards.erl"},{line,156}]},{couch_peruser,init,0,[{file,...},...]},...]}}} in context start_error
[error] 2017-08-14T08:15:09.866652Z couchdb@localhost <0.204.0> -------- Error starting Apache CouchDB:

    {error,{shutdown,{failed_to_start_child,couch_secondary_services,{shutdown,{failed_to_start_child,couch_peruser,{badarg,[{ets,member,[mem3_openers,<<"users">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,for_db,2,[{file,"src/mem3_shards.erl"},{line,59}]},{mem3_shards,local,1,[{file,"src/mem3_shards.erl"},{line,156}]},{couch_peruser,init,0,[{file,"src/couch_peruser.erl"},{line,58}]},{couch_peruser,init,1,[{file,"src/couch_peruser.erl"},{line,187}]}]}}}}}}

[wohali]

We need more time to investigate this one. Reopening until we have time to do so.

[realulim]

I'm having the same issue, but without docker. Compiled from sources, ensured that I have a working CouchDB (via an integration test suite) and can add and remove users. All that with the couch_peruser feature disabled. Once I enable it and restart CouchDB I get an error in the logfile (see attached file
log.txt
).

[wohali]

We nave a possible workaround for @realulim @rasdaniil . Can you try removing this line from your default.ini file?

couch_peruser={couch_peruser, start_link, []}

It should be line 247.

If you remove that line and try again, does it work?

[realulim]

It's line 236 in my default.ini, which is unchanged from the download. After commenting it out, CouchDB starts up and my test suite works. I haven't checked yet, whether the per user DB is actually created, will do that later and report back.

[realulim]

I created a user and saw in Fauxton that the user was actually contained in _users, but there was no additional database. So to me it appears that the workaround avoids the crash, but the per user DB feature does not work.

[wohali]

So it looks like couch_peruser in 2.1.0 is broken. Sorry, everyone. We'll get this fixed in the next release of CouchDB.

[realulim]

Thank you very much, that would be terrific.

[sirhoe]

Thanks for the confirmation. We really look forward to have this feature working. Adding my 2bits to the bug

[orolle]

Thanks for the confirmation.

[DanielMaly]

Could we get a rough time estimate when this could be fixed? I'm stuck on 1.6.1 at the moment, because couch_peruser is broken in 2.0 as well.

[wohali]

@chewbranca 's analysis says that the issues are not trivial. Russell, can you expand a bit on what needs to be done?

[arcadius]

Thanks @wohali and @chewbranca for looking into this

[budiadiono]

Any temporary solution for this? I'm having same issue when running on Centos 7.

[domachine]

My solution is currently that I reimplemented this feature using a separate
process (Node.js) that listens to the _user database changes. If you're
interested I can try to setup a repo to demo this approach.

Am 24.09.2017 12:28 nachm. schrieb "Budi Adiono" [e-mail]:

Any temporary solution for this? I'm having same issue when running on
Centos 7.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/apache/couchdb/issues/749#issuecomment-331700973, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAZOGPbsejOY8rFkVwcE0-chA_HUymvdks5sli65gaJpZM4O1obs
.

[budiadiono]

@domachine That's is interesting! -- so, the idea is to turn of the per user feature and create a listener that manually creates/drops user's db once _users database changes? I was thinking that too, but I don't know if it's good idea or not.

[domachine]

I don't know if this is a good idea, either :blush:. But it works for me.

[budiadiono]

Haha.. seems we've got to choose this way though. I'm interest to see your workaround.

[domachine]

I'm currently on the road but get back this evening.

[budiadiono]

no rush 😊

[pierrekilly]

I have found a workaround:

As setting the couch_peruser to true from Fauxton works well, I simply activate it "by hand" after the DB is started.

In the script that starts the DB, I wait 10 seconds (to leave some time to the DB to start) and then:

curl \
    -X PUT \
    --user "${adminUser}:${adminPass}" \
    --data '"true"' \
    "http://localhost:5984/_node/nonode@nohost/_config/couch_peruser/enable"

(The admin username and password are taken from env variables in my case)

Hope this helps!

[wohali]

There's a lot more wrong with couch_peruser in 2.1.0 - I strongly recommend you wait for our imminent 2.1.1 release before using this feature.

[dynamite-ready]

Is this issue still open?

I set the config property (couch_peruser = true), created a new user, and expected to find an associated DB in the list of databases on Fauxton. I'm on version 2.2.0.

[dynamite-ready]

I have found a workaround:

As setting the couch_peruser to true from Fauxton works well, I simply activate it "by hand" after the DB is started.

In the script that starts the DB, I wait 10 seconds (to leave some time to the DB to start) and then:

curl \
    -X PUT \
    --user "${adminUser}:${adminPass}" \
    --data '"true"' \
    "http://localhost:5984/_node/nonode@nohost/_config/couch_peruser/enable"

(The admin username and password are taken from env variables in my case)

Hope this helps!

Actually, I can confirm that the above works.
So the issue appears to centre around how/when it's initialised.