Google-cloud-go: asn1: structure error: tags don't match

[karalabe]

Client

Datastore

Describe Your Environment

Own server

Expected Behavior

I can connect and query data. I'm using a service account json credential downloaded from the cloud console.

store, _ := datastore.NewClient(context.Background(), project, option.WithCredentialsFile("credentials.json"))
keys, _ := store.GetAll(context.Background(), datastore.NewQuery("Entity").Limit(256), &results)

{
  "type": "service_account",
  "project_id": "...",
  "private_key_id": "...",
  "private_key": "-----BEGIN PRIVATE KEY----- ... -----END PRIVATE KEY-----\n",
  "client_email": "[email protected]",
  "client_id": "...",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/...%40....iam.gserviceaccount.com"
}

Actual Behavior

Query fails with rpc error: code = Unauthenticated desc = transport: private key should be a PEM or plain PKCS1 or PKCS8; parse error: asn1: structure error: tags don't match (16 vs {class:0 tag:13 length:45 isCompound:true}) {optional:false explicit:false application:false defaultValue:<nil> tag:<nil> stringType:0 timeType:0 set:false omitEmpty:false} pkcs1PrivateKey @2

[karalabe]

Meh, here's the entire revoked private key. Go's pem paser rejects this:

-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDHxZaeBzu7JOqY
HV6PKCMb+tsiFRsFiY1kGyexJXzLYy39sRL6Xsg1jxMUx6TdHRzSP06N225DbUs3
D1vbEZAp91qdJouvK9Mbf+1eQ5d4yeDQZMqdJJoBbmckCGcR/QhXwZ2wn0Gw0AhH
Af4EsCX3d8Cxtu5zxzR0GIE6YUONt3RZdBgpddQzlFCRGZBkz+c7jUejT7iBdwix
DaW7itqYO5TETaqwF1bzefD1BSrIjATfdcyUdpMtMTNKfVTZ9Eh3eKy+qIBdTtH/
avvf4AVc2yN1K/+occ7doEclSd3W7vTi51w3VT8TdA9aFVz0RbxP3Y2px0zWX12f
unLeBSWbAgMBAAECggEARfVZagzha4ehidSbJSnqpaVDMRvQAy/o7loeG8ije7xH
QlTM7xXbKfppNbk2cGJ+EdiuoznpUr6G/QipY72yTSf8uRTjDNydiL9TelPUSy3z
RzdMxxwmvIKTpwg0RBXm4oiAtvYGdKtdgrRdZvniyddLiVClD7F+mntsYevm0s0C
AgOFwfz/pGun7EIKGaDCo418u9nHbhQUyscfqmZEx9lTNRwcGAbxoi4EC0OVifDr
k+yoFwXjJG1mcMxZUaJTJYwP65rCUSviMUvzd7BGfenNLZTxSiHEZ/hHa3glLfVy
ggf1WA+74Mu/0KBV4Ya6ZAkHHw6hVET+8+faG2h1mQKBgQDo11lW7k3nflHCS9qD
5FUsZeesE+1sQjCCpA7TvwWnJ3avYP8R1dyHiwPXH5kMl9YKn7N/DxOqzsvFC0Qh
1UBJQjWlm2YPK5I3FYX66lwkKNibaU3sq+Ly1CFCGqrKZa839otd5sVOol30uplF
018G8aTiqhHuyc1fjyx+8jSn8wKBgQDbpDhT6W+HdFrDO5TVae6pGHMO4GMrmvQm
R8IHeMDuU4bN4XL1Uj1skp+apcqcAareFItIi03VyZnf7koGQujqgFLEZK2naUXA
IOD2gAz0YZEiZbFeXKLdVX1DnAvz6cbhRp2qpR+Ky+vOHEjV6mY+YuKDz/bzgMSe
tkc47NnduQKBgQDaX8ZpcnzcJSvW9z9MvaRoTHbYe6QMCZPnoqhJTXmmyKtWVrlC
5/m5odaLNxZaqjjTo+47t08xvlt8RVG0DYYKby9TT4iLp8itIuGSb6TVQP3N3Bh6
ZMcoCW3bypjt1CpeaTtSaTIZyswlz7Aavd/86jtDXlANTXTxL52CvfRGowKBgQCW
RGr5Fbrk/DjgWxH/VEMg0wZcxi1y9sdUrUFU5Utxghm3HygMKKC3eDTTk9vjEcz5
tSp5jjzJJ+0rZBam4/3/+Z0mmg6oe4Bp6tSeMIssYtftpY9MlKokLUnPCKKw1F7p
XuudhOzog40nbPhzybL7uaFpNs2oWI+sWd6uVnTTmQKBgF450f5zWvUFv/00WUY+
1YWZtYDWV/tzwauz4+7pNC4MNGQQpOFEVEucPVfKMX/vbm1h1DJTVPzBgdkw7nfk
ckD3dVK4nva0qzdpfKOX/QHX/21Y0ivELswnmZ488IYrwWJpJta0ZvD1/SpvAgJb
+VLBOI3lpGHlxgh2ZTcanfsV
-----END PRIVATE KEY-----

[jadekler]

@broady Do you happen to know anything about this?

[karalabe]

I've double checked and the key can successfully be decoded, so my above pem report is moot. Still trying to figure out what rejects it in the end.

    block, _ := pem.Decode([]byte("-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQDHxZaeBzu7JOqY\nHV6PKCMb+tsiFRsFiY1kGyexJXzLYy39sRL6Xsg1jxMUx6TdHRzSP06N225DbUs3\nD1vbEZAp91qdJouvK9Mbf+1eQ5d4yeDQZMqdJJoBbmckCGcR/QhXwZ2wn0Gw0AhH\nAf4EsCX3d8Cxtu5zxzR0GIE6YUONt3RZdBgpddQzlFCRGZBkz+c7jUejT7iBdwix\nDaW7itqYO5TETaqwF1bzefD1BSrIjATfdcyUdpMtMTNKfVTZ9Eh3eKy+qIBdTtH/\navvf4AVc2yN1K/+occ7doEclSd3W7vTi51w3VT8TdA9aFVz0RbxP3Y2px0zWX12f\nunLeBSWbAgMBAAECggEARfVZagzha4ehidSbJSnqpaVDMRvQAy/o7loeG8ije7xH\nQlTM7xXbKfppNbk2cGJ+EdiuoznpUr6G/QipY72yTSf8uRTjDNydiL9TelPUSy3z\nRzdMxxwmvIKTpwg0RBXm4oiAtvYGdKtdgrRdZvniyddLiVClD7F+mntsYevm0s0C\nAgOFwfz/pGun7EIKGaDCo418u9nHbhQUyscfqmZEx9lTNRwcGAbxoi4EC0OVifDr\nk+yoFwXjJG1mcMxZUaJTJYwP65rCUSviMUvzd7BGfenNLZTxSiHEZ/hHa3glLfVy\nggf1WA+74Mu/0KBV4Ya6ZAkHHw6hVET+8+faG2h1mQKBgQDo11lW7k3nflHCS9qD\n5FUsZeesE+1sQjCCpA7TvwWnJ3avYP8R1dyHiwPXH5kMl9YKn7N/DxOqzsvFC0Qh\n1UBJQjWlm2YPK5I3FYX66lwkKNibaU3sq+Ly1CFCGqrKZa839otd5sVOol30uplF\n018G8aTiqhHuyc1fjyx+8jSn8wKBgQDbpDhT6W+HdFrDO5TVae6pGHMO4GMrmvQm\nR8IHeMDuU4bN4XL1Uj1skp+apcqcAareFItIi03VyZnf7koGQujqgFLEZK2naUXA\nIOD2gAz0YZEiZbFeXKLdVX1DnAvz6cbhRp2qpR+Ky+vOHEjV6mY+YuKDz/bzgMSe\ntkc47NnduQKBgQDaX8ZpcnzcJSvW9z9MvaRoTHbYe6QMCZPnoqhJTXmmyKtWVrlC\n5/m5odaLNxZaqjjTo+47t08xvlt8RVG0DYYKby9TT4iLp8itIuGSb6TVQP3N3Bh6\nZMcoCW3bypjt1CpeaTtSaTIZyswlz7Aavd/86jtDXlANTXTxL52CvfRGowKBgQCW\nRGr5Fbrk/DjgWxH/VEMg0wZcxi1y9sdUrUFU5Utxghm3HygMKKC3eDTTk9vjEcz5\ntSp5jjzJJ+0rZBam4/3/+Z0mmg6oe4Bp6tSeMIssYtftpY9MlKokLUnPCKKw1F7p\nXuudhOzog40nbPhzybL7uaFpNs2oWI+sWd6uVnTTmQKBgF450f5zWvUFv/00WUY+\n1YWZtYDWV/tzwauz4+7pNC4MNGQQpOFEVEucPVfKMX/vbm1h1DJTVPzBgdkw7nfk\nckD3dVK4nva0qzdpfKOX/QHX/21Y0ivELswnmZ488IYrwWJpJta0ZvD1/SpvAgJb\n+VLBOI3lpGHlxgh2ZTcanfsV\n-----END PRIVATE KEY-----\n"))

    fmt.Println(x509.ParsePKCS8PrivateKey(block.Bytes))

[karalabe]

This is getting annoying. I've deleted my entire service account and created a new one. Now I get a different error: rpc error: code = PermissionDenied desc = Missing or insufficient permissions.

[jadekler]

What permissions did you give your service account? You might try Project Owner, which I believe is the highest permission you can grant, and whittle it down from there.

[karalabe]

I did that, just in case, still received the error above.

[karalabe]

If it helps anything, my datastore is "Cloud Firestore in Datastore mode"

[jadekler]

This may seem silly, but could you double check that your service account is in the same project as what you're specifying in the NewClient call? I manage a lot of projects and sometimes mess that up.. :/

[karalabe]

Yes, it is in the same one. If I delete the service account the error immediately switches over to rpc error: code = Unauthenticated desc = transport: oauth2: cannot fetch token: 400 Bad Request\nResponse: {\n \"error\": \"invalid_grant\",\n \"error_description\": \"Not a valid email or user ID.\"\n}

[jadekler]

Aha, ok. This is baffling. Credential logic has not recently changed, and is working as-usual for me, so I suspect this is not directly related to client libraries. Let me know if you have reason to suspect otherwise, though, such as seeing different behavior on an older version of the libraries.

As to fixing your issue, I'm out of debugging tips, because I'm an auth noob. :) @broady Perhaps you have some ideas?

Also, @karalabe, you might find want to browse around or open an issue on https://stackoverflow.com/questions/tagged/google-cloud-platform, or contact our support teams.

[karalabe]

Huh, if I create a key on top of the "App Engine default service account", all permissions are fine. If I use a separate service account, my requests get rejected.

[karalabe]

Either way, my secondary issue is now solved. I'm still baffled by the original key rejection though. However I deleted that key while playing around, so can't reproduce it apart from the private key I had above. Perhaps that's worth a closer look, since it seems weird to be rejected.

[jadekler]

That's quite odd. Thanks for keeping us updated with your findings. Btw, are you running on AppEngine? (original issue says no, but just want to double check)

[broady]

This is very puzzling to me. As far as I know, there is nothing particularly different about the GAE default service account.

I'll try to reproduce, but I'm about to head out of office.

Does this code run fine locally, outside of GAE?

[karalabe]

My setup if a fresh new project, I just made 2 tiny javascript cloud functions (I assume this is where GAE comes from) that inject data into the datastore (operating in "Cloud Firestore in Datastore mode", whatever that specifically means, I chose that because the wizard told me that's best :D).

From within the cloud functions, everything worked nicely, but that's to be expected with the permissions baked into GAE. Beside my 2 tiny cloud funcs however, I also wanted to access the database from the outside (from Go, I just used JavaScript for my tiny testing funcs) from a remote server that periodically scrapes the data collected by the cloud funcs.

Apparently using a plain service account is not enough, I needed specifically the GAE one. I'm certain that it might be some weird hidden permission I don't know about, but it's very odd that creating a service account with "Owner" permissions is still not enough.

[karalabe]

Ahahahaha, this is getting funny now :D I may have found a reserved keyword :P

I created 2 brand new (!!) service accounts, same empty stuff with project owner permissions. I named one of them "tester" and I named the other one "tracker" (I used tracker as the name previously too).

If I use the json from the tester, everything works. If I use the json from the tracker, it errors out with rpc error: code = PermissionDenied desc = Missing or insufficient permissions..

---

Just to verify that I didn't mess something up, I created a brand new google cloud project, added some test data and did the same 2 new service account. Same results!! :D The one called tracker gets a permission denied, while any other random name works.

[jadekler]

This is super bizarre. I just tried to reproduce and wasn't able to - tracker works fine for me. o_O

Is there any other details you can think to provide?

ps: Glad you were able to get it working, though! That's positive at least. :)

[jadekler]

We've asked some clarifying questions and haven't heard back. Closing this for now, but please let me know if you'd like to resume the conversation! And, I'll keep an eye out for similarly strange reserved-words kind of errors.

[tetriscode]

This issue appears to happen for me when the service-account.json I pass in to authenticate works if the file is pretty-printed json and does not work if I remove spaces and line breaks from the json and not the key. printf "%s" '${GCLOUD_CREDS}' > service-account.json. Putting JSON with spaces and line breaks will result in printf "%s" "{ " > service-account.json

[MrJoy]

~I've been seeing this exact PEM parsing error happen intermittently. So far I've seen it 172 times today (although I'm sub-sampling 5% of errors blindly because of the volume of work my system does). There's only one file the credentials are in, which is generated via confd -- and one codepath for parsing and using that data. For scale, I'm attempting token fetches (and thus this code path) about 100 times _per minute_. No process restarts, manual intervention on the node, automated confd regeneration (it's manually triggered), etc. It's just that somehow or other, the PEM parsing fails from time to time.~

~We've only recently started seeing this problem, although the blind sub-sampling of errors might have masked it out if it was happening infrequently before. We're on Go 1.11.10, and v0.39.0 of cloud.google.com/go.~

~Not super useful, I know -- but I figured it might be a useful datapoint. Happy to provide more info, if you can think of anything useful I might be able to add.~

Problem was happening 100% of the time -- in our staging environment. And it had nothing to do with this library. My apologies for adding to the noise floor.