Origin: Deleting and recreating a user fails with 500
Working on a logging test, found deleting a user and then recreating causes a 500 server error:
oc v3.9.0-alpha.3+78ddc10
kubernetes v1.9.1+a0ce1bc657
features: Basic-Auth GSSAPI Kerberos SPNEGO
Server https://127.0.0.1:8443
openshift v3.7.1+a8deba5-34
kubernetes v1.7.6+a08f5eeb62
oc login
Authentication required for https://127.0.0.1:8443 (openshift)
Username: foo
Password:
Login successful.
You don't have any projects. You can try to create a new project, by running
oc new-project <projectname>
[root@openshiftdevel origin-aggregated-logging]# oc delete user
error: resource(s) were provided, but no name, label selector, or --all flag specified
[root@openshiftdevel origin-aggregated-logging]# oc delete user foo
Error from server (Forbidden): User "foo" cannot list clusterrolebindings.authorization.openshift.io at the cluster scope: User "foo" cannot list all clusterrolebindings.authorization.openshift.io in the cluster (get clusterrolebindings.authorization.openshift.io)
[root@openshiftdevel origin-aggregated-logging]# oc login -u system:admin
Logged into "https://127.0.0.1:8443" as "system:admin" using existing credentials.
You have access to the following projects and can switch between them with 'oc project <projectname>':
* default
kube-public
kube-system
logging
myproject
openshift
openshift-infra
openshift-node
Using project "default".
[root@openshiftdevel origin-aggregated-logging]# oc delete user foo
user "foo" deleted
[root@openshiftdevel origin-aggregated-logging]# oc login
Authentication required for https://127.0.0.1:8443 (openshift)
Username: foo
Password:
error: Internal error occurred: unexpected response: 500 - verify you have provided the correct host and port and that the server is currently running.
Error from server (InternalError): Internal error occurred: unexpected response: 500
jcantrill
All 14 comments
I've run into something similar to this before, I think it's related to the fact that when you delete user "foo" the identity for foo remains
$ oc delete user foo
user "foo" deleted
$ oc get identities
NAME IDP NAME IDP USER NAME USER NAME USER UID
anypassword:developer anypassword developer developer ec2e120b-27ae-11e8-b202-507b9dac9a27
anypassword:foo anypassword foo foo bcb48b19-27af-11e8-b202-507b9dac9a27
$ oc delete identities/anypassword:foo
identity "anypassword:foo" deleted
$ oc login -u foo
Authentication required for https://127.0.0.1:8443 (openshift)
Username: foo
Password:
Login successful.
Should deleting a user also delete the identity for that user?
damemi
on 14 Mar 2018
Update, looks like this is intended https://github.com/openshift/origin/blob/a2ec32460a7351e6ddef482f995f375e99fdef25/pkg/user/reaper/user.go#L115-L117
Could add a flag or something to delete the user's identities too, if that's something that would be useful I can work on it
damemi
on 15 Mar 2018
@openshift/sig-security
jwforres
on 20 Mar 2018
no matter what we shouldn't have things result in 500's, so there is a bug to fix here
jwforres
on 20 Mar 2018
@enj is a 500 expected ? I tend to agree with @jwforres that a 500 should never happen.
simo5
on 20 Mar 2018
Should deleting a user also delete the identity for that user?
No, the user object is owned by OpenShift. The identity object(s) are owned by the IDP(s). Deleting the identity objects can cause us to lose state which we have no way to recover.
no matter what we shouldn't have things result in 500's, so there is a bug to fix here
No, this is very much by design. If you start deleting users or identities, then you take ownership of fixing what you just screwed up. Online has dealt with this from the very beginning because they take full ownership of both the user and identity objects.
@enj is a 500 expected ?
It is expected.
I tend to agree with @jwforres that a 500 should never happen.
I have no issue with the server returning 500s if you break API invariants.
@liggitt any thoughts? I am fine with closing this since it is working as designed.
enj
on 21 Mar 2018
Agree. An orphaned identity pointing to a user with a different uid is a server error that must be resolved by the cluster admin. A 5xx error is appropriate
liggitt
on 21 Mar 2018
If you check the server log, there should be errors around login indicating the issue. If there are not, we should add better logging there.
liggitt
on 21 Mar 2018
@enj @liggitt so should we close/wontfix this one ?
simo5
on 22 Mar 2018
Issues go stale after 90d of inactivity.
Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.
If this issue is safe to close now please do so with /close.
/lifecycle stale
openshift-bot
on 20 Jun 2018
Stale issues rot after 30d of inactivity.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.
If this issue is safe to close now please do so with /close.
/lifecycle rotten
/remove-lifecycle stale
openshift-bot
on 20 Jul 2018
Rotten issues close after 30d of inactivity.
Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.
/close
openshift-bot
on 20 Aug 2018
Here I have found the solution:
It will remove only the username from the htpasswd file by default it won’t remove user identity so we should remove user identity as well if not – whenever new user created with old username it will give error:
[root@master ~]# oc login
Authentication required for https://master.lab.example.com:443 (openshift)
Username: joe
Password:
Error from server (InternalError): Internal error occurred: unexpected response: 500
[root@master ~]# oc login -u joe -p zaldebra https://master.lab.example.com:443
Error from server (InternalError): Internal error occurred: unexpected response: 500
[root@master ~]# oc login --loglevel=10
I0830 17:02:14.427946 104015 loader.go:357] Config loaded from file /root/.kube/config
I0830 17:02:14.429201 104015 round_trippers.go:417] curl -k -v -XHEAD https://master.lab.example.com:443/
I0830 17:02:14.491327 104015 round_trippers.go:436] HEAD https://master.lab.example.com:443/ 403 Forbidden in 62 milliseconds
I0830 17:02:14.491383 104015 round_trippers.go:442] Response Headers:
I0830 17:02:14.491398 104015 round_trippers.go:445] Cache-Control: no-store
I0830 17:02:14.491412 104015 round_trippers.go:445] Content-Type: application/json
I0830 17:02:14.491438 104015 round_trippers.go:445] X-Content-Type-Options: nosniff
I0830 17:02:14.491452 104015 round_trippers.go:445] Content-Length: 238
I0830 17:02:14.491465 104015 round_trippers.go:445] Date: Sun, 30 Aug 2020 11:32:14 GMT
I0830 17:02:14.493237 104015 round_trippers.go:417] curl -k -v -XGET -H "X-Csrf-Token: 1" https://master.lab.example.com:443/.well-known/oauth-authorization-server
I0830 17:02:14.493966 104015 round_trippers.go:436] GET https://master.lab.example.com:443/.well-known/oauth-authorization-server 200 OK in 0 milliseconds
I0830 17:02:14.494079 104015 round_trippers.go:442] Response Headers:
I0830 17:02:14.494251 104015 round_trippers.go:445] Date: Sun, 30 Aug 2020 11:32:14 GMT
I0830 17:02:14.494357 104015 round_trippers.go:445] Cache-Control: no-store
I0830 17:02:14.494462 104015 round_trippers.go:445] Content-Type: application/json
I0830 17:02:14.494558 104015 round_trippers.go:445] Content-Length: 543
I0830 17:02:14.498855 104015 round_trippers.go:417] curl -k -v -XGET -H "X-Csrf-Token: 1" https://master.lab.example.com/oauth/authorize?client_id=openshift-challenging-client&code_challenge=Z61r66ZF2OoHrHNlqP-9dxRyAAkqF9Nyj25xkIjJ5x8&code_challenge_method=S256&redirect_uri=https%3A%2F%2Fmaster.lab.example.com%2Foauth%2Ftoken%2Fimplicit&response_type=code
I0830 17:02:14.501594 104015 round_trippers.go:436] GET https://master.lab.example.com/oauth/authorize?client_id=openshift-challenging-client&code_challenge=Z61r66ZF2OoHrHNlqP-9dxRyAAkqF9Nyj25xkIjJ5x8&code_challenge_method=S256&redirect_uri=https%3A%2F%2Fmaster.lab.example.com%2Foauth%2Ftoken%2Fimplicit&response_type=code 401 Unauthorized in 2 milliseconds
I0830 17:02:14.501739 104015 round_trippers.go:442] Response Headers:
I0830 17:02:14.501865 104015 round_trippers.go:445] Content-Type: text/plain; charset=utf-8
I0830 17:02:14.501975 104015 round_trippers.go:445] Content-Length: 0
I0830 17:02:14.502069 104015 round_trippers.go:445] Date: Sun, 30 Aug 2020 11:32:14 GMT
I0830 17:02:14.502164 104015 round_trippers.go:445] Www-Authenticate: Basic realm="openshift"
Authentication required for https://master.lab.example.com:443 (openshift)
Username: joe
Password:
I0830 17:02:40.060501 104015 round_trippers.go:417] curl -k -v -XGET -H "Authorization: Basic am9lOnphbGRlYnJh" -H "X-Csrf-Token: 1" https://master.lab.example.com/oauth/authorize?client_id=openshift-challenging-client&code_challenge=Z61r66ZF2OoHrHNlqP-9dxRyAAkqF9Nyj25xkIjJ5x8&code_challenge_method=S256&redirect_uri=https%3A%2F%2Fmaster.lab.example.com%2Foauth%2Ftoken%2Fimplicit&response_type=code
I0830 17:02:40.067033 104015 round_trippers.go:436] GET https://master.lab.example.com/oauth/authorize?client_id=openshift-challenging-client&code_challenge=Z61r66ZF2OoHrHNlqP-9dxRyAAkqF9Nyj25xkIjJ5x8&code_challenge_method=S256&redirect_uri=https%3A%2F%2Fmaster.lab.example.com%2Foauth%2Ftoken%2Fimplicit&response_type=code 500 Internal Server Error in 6 milliseconds
I0830 17:02:40.067053 104015 round_trippers.go:442] Response Headers:
I0830 17:02:40.067061 104015 round_trippers.go:445] Content-Type: text/plain; charset=utf-8
I0830 17:02:40.067067 104015 round_trippers.go:445] Content-Length: 46
I0830 17:02:40.067073 104015 round_trippers.go:445] Date: Sun, 30 Aug 2020 11:32:40 GMT
I0830 17:02:40.072129 104015 helpers.go:201] server response object: [{
"metadata": {},
"status": "Failure",
"message": "Internal error occurred: unexpected response: 500",
"reason": "InternalError",
"details": {
"causes": [
{
"message": "unexpected response: 500"
}
]
},
"code": 500
}]
F0830 17:02:40.072201 104015 helpers.go:119] Error from server (InternalError): Internal error occurred: unexpected response: 500
[root@master ~]# grep -A5 joe .kube/config
user: joe/master-lab-example-com:443
name: /master-lab-example-com:443/joe
- context:
cluster: master-lab-example-com:443
user: prabhanjan/master-lab-example-com:443
name: /master-lab-example-com:443/prabhanjan
- context:
- name: joe/master-lab-example-com:443
user:
token: xOtUGSiLpA_yM4vtRBLhHge6fVOBDXUnoBFeySvq3pc - name: prabhanjan/master-lab-example-com:443
user:
token: SgYuyRvyVib72xmlSt-pGVV21i_Se7n2zr4ebaXHZ9A
[root@master ~]# oc login -u joe -p redhat https://master.lab.example.com:443
Login failed (401 Unauthorized)
Verify you have provided correct credentials.
[root@master ~]# oc login -u joe -p password https://master.lab.example.com:443
Error from server (InternalError): Internal error occurred: unexpected response: 500
solution:
[root@master ~]# oc delete identity htpasswd_auth:joe
identity "htpasswd_auth:joe" deleted
[root@master ~]# oc login -u joe -p password https://master.lab.example.com:443
Login successful.
prrabbhanjon
on 30 Aug 2020
Deleting the identity works for me, too. Thanks for sharing that.
josephtingiris
on 30 Sep 2020
Related issues
slmzig
·
3Comments
thescouser89
·
3Comments
syui
·
3Comments
davivd
·
4Comments
alikhajeh1
·
3Comments
Most helpful comment
I've run into something similar to this before, I think it's related to the fact that when you delete user "foo" the identity for foo remains
Should deleting a user also delete the identity for that user?