Gogs: error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502 Bad Gateway

Created on 18 Mar 2017  ·  48Comments  ·  Source: gogs/gogs
  • Gogs version (or commit ref): 0.10.18.0313
  • Git version: 2.11.1
  • Operating system: Ubuntu 16.04 64bits
  • Database (use [x]):

    • [ ] PostgreSQL

    • [ ] MySQL

    • [ ] MSSQL

    • [x] SQLite

  • Can you reproduce the bug at https://try.gogs.io:

    • [ ] Yes (provide example URL)

    • [x] No

    • [ ] Not relevant

  • Log gist (usually found in log/gogs.log):
2017/03/18 00:43:07 [TRACE] Template: repo/home
2017/03/18 00:43:15 [TRACE] Session ID: 680247266044030b
2017/03/18 00:43:15 [TRACE] CSRF Token: XqrlvbmA3VRS46nV_Uufk3YAt_A6MTQ4OTgxMjE5NTE4NTY5ODg3Mg==
2017/03/18 00:43:15 [TRACE] Session ID: cba1c7a688bd14dc
2017/03/18 00:43:15 [TRACE] CSRF Token: PdAbkOr0oYgJst6rQpmkfuU1rbI6MTQ4OTgxMjE5NTQ4NzI2MTIyNA==

Description

Getting:

Mar 18 00:38:03 quark gogs[17459]: [Macaron] 2017-03-18 00:38:03: Completed POST /david/collantes.me.git/git-receive-pack 401 Unauthorized in 4.46717ms

On git client (Tower.app, macOS).

Full log:

Pushing to https://[email protected]/david/collantes.me.git
Counting objects: 31, done.
Delta compression using up to 8 threads.
Compressing objects:   3% (1/27)   
Compressing objects:   7% (2/27)   
Compressing objects:  11% (3/27)   
Compressing objects:  14% (4/27)   
Compressing objects:  18% (5/27)   
Compressing objects:  22% (6/27)   
Compressing objects:  25% (7/27)   
Compressing objects:  29% (8/27)   
Compressing objects:  33% (9/27)   
Compressing objects:  37% (10/27)   
Compressing objects:  40% (11/27)   
Compressing objects:  44% (12/27)   
Compressing objects:  48% (13/27)   
Compressing objects:  51% (14/27)   
Compressing objects:  55% (15/27)   
Compressing objects:  59% (16/27)   
Compressing objects:  62% (17/27)   
Compressing objects:  66% (18/27)   
Compressing objects:  70% (19/27)   
Compressing objects:  74% (20/27)   
Compressing objects:  77% (21/27)   
Compressing objects:  81% (22/27)   
Compressing objects:  85% (23/27)   
Compressing objects:  88% (24/27)   
Compressing objects:  92% (25/27)   
Compressing objects:  96% (26/27)   
Compressing objects: 100% (27/27)   
Compressing objects: 100% (27/27), done.
Writing objects:   3% (1/31)   
Writing objects:   6% (2/31)   
Writing objects:   9% (3/31)   
Writing objects:  12% (4/31)   
Writing objects:  16% (5/31)   
Writing objects:  19% (6/31)   
Writing objects:  22% (7/31)   
Writing objects:  25% (8/31)   
Writing objects:  29% (9/31)   
Writing objects:  32% (10/31)   
Writing objects:  35% (11/31)   
Writing objects:  38% (12/31)   
Writing objects:  41% (13/31)   
Writing objects:  48% (15/31)   
Writing objects:  51% (16/31)   
Writing objects:  54% (17/31)   
Writing objects:  58% (18/31)   
Writing objects:  61% (19/31)   
Writing objects:  64% (20/31)   
Writing objects:  67% (21/31)   
Writing objects:  70% (22/31)   
Writing objects:  74% (23/31)   
Writing objects:  77% (24/31)   
Writing objects:  80% (25/31)   
Writing objects:  83% (26/31)   
Writing objects:  87% (27/31)   
Writing objects:  90% (28/31)   
Writing objects:  93% (29/31)   
Writing objects:  96% (30/31)   
Writing objects: 100% (31/31)   
Writing objects: 100% (31/31), 359.70 KiB | 0 bytes/s, done.
Total 31 (delta 3), reused 0 (delta 0)
POST git-receive-pack (368506 bytes)
error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502 Bad Gateway
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
Everything up-to-date

When trying to push a commit. I can commit one small file, no problems. If I try to add several files, or directories, the error occurs. From server's /var/log/syslog:

Mar 18 00:50:04 quark gogs[17459]: [Macaron] 2017-03-18 00:50:04: Started GET /david/collantes.me.git/info/refs?service=git-receive-pack for XXX.XXX.XXX.XXX
Mar 18 00:50:04 quark gogs[17459]: [Macaron] 2017-03-18 00:50:04: Completed GET /david/collantes.me.git/info/refs?service=git-receive-pack 200 OK in 18.748727ms
Mar 18 00:50:04 quark gogs[17459]: [Macaron] 2017-03-18 00:50:04: Started POST /david/collantes.me.git/git-receive-pack for XXX.XXX.XXX.XXX
Mar 18 00:50:04 quark gogs[17459]: [Macaron] 2017-03-18 00:50:04: Completed POST /david/collantes.me.git/git-receive-pack 401 Unauthorized in 3.706604ms
needs reproduce 💊 bug 🙇‍♂️ help wanted
Source
picture Fastidious

Most helpful comment

Like @netlander the workaround of bypassing the Apache proxy worked for me. It was crashing at that step: "POST git-receive-pack (7200216 bytes)", but this works normally without the proxy.

Might well be related to the 401: with the Apache mod_proxy, I did not receive an authentication request, but I received one without the proxy and had to type user/password for the commit to be accepted.

Apache error log (multiple identical lines):

[Sat Jan 12 09:12:33.710124 2019] [proxy:error] [pid 25787] (32)Broken pipe: [client 37.58.212.212:59962] AH01084: pass request body failed to 127.0.0.1:3000 (127.0.0.1)
[Sat Jan 12 09:12:33.710162 2019] [proxy_http:error] [pid 25787] [client 37.58.212.212:59962] AH01097: pass request body failed to 127.0.0.1:3000 (127.0.0.1) from 37.58.212.212 ()

Gogs log/hooks/post-receive.log (9:20 is without the proxy, nothing visible at 9:12 with the proxy):

2019/01/12 09:20:41 [ERROR] [...ogs/models/action.go:493 CommitRepoAction()] UpdateIssuesCommit: invalid issue reference [ref: 09/01/2019]

Other log/hooks/* files are touched but empty

Portion of the log/gogs.log which appears relevant, without the proxy:

2019/01/12 09:20:41 [TRACE] HTTPGit - Authenticated user: fgm
2019/01/12 09:20:41 [TRACE] Session ID: d8376360d674a0ea
2019/01/12 09:20:41 [TRACE] CSRF Token: FKyzeZELo1j4eoGE-TKp3yDmyc06MTU0NzI4NDg0MTU0MjMxMjM2MA==
2019/01/12 09:20:41 [TRACE] Detected encoding: UTF-8 (fast)
2019/01/12 09:20:41 [TRACE] Template: repo/home
2019/01/12 09:20:41 [TRACE] Session ID: 9e0fc1184eb4970b
2019/01/12 09:20:41 [TRACE] CSRF Token: M0fhCH0kbQAhgfPKdmaXSga1DPk6MTU0NzI4NDg0MTkxMjIwMDAwOA==
2019/01/12 09:20:41 [TRACE] TriggerTask 'assets__prague2013_blocks/blocks-1-8' by 'fgm'
2019/01/12 09:20:41 [TRACE] DeliverHooks [repo_id: 112]
2019/01/12 09:20:41 [TRACE] AddTestPullRequestTask [head_repo_id: 112, head_branch: blocks-1-8]: finding pull requests
2019/01/12 09:20:41 [TRACE] AddTestPullRequestTask [base_repo_id: 112, base_branch: blocks-1-8]: finding pull requests

More interestingly, updating my HTTPS Vhost to look like this, as suggested on https://httpd.apache.org/docs/2.4/en/mod/mod_proxy.html#envsettings , fixed the issue for me. It causes the browser to ask for credentials (triggering keychain access in macOS), and succeeds:

  ProxyPreserveHost On
  ProxyRequests off
  ProxyPass / http://127.0.0.1:3000/
  ProxyPassReverse / http://127.0.0.1:3000/
  SetEnv force-proxy-request-1.0 1
  SetEnv proxy-nokeepalive 1

This might suggest a problem with HTTP/1.1 keepalive handling in Gogs or its dependencies.

picture fgm on 12 Jan 2019
👍2

All 48 comments

I was able to create another repository, and commit a rather large set of files and directories, no issues. The problems is showing with this repository.

Fastidious picture Fastidious on 18 Mar 2017

More info:

git push --verbose
Pushing to https://gits.quark.business/david/collantes.us.git
Counting objects: 534, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (525/525), done.
Writing objects: 100% (534/534), 6.56 MiB | 0 bytes/s, done.
Total 534 (delta 4), reused 0 (delta 0)
POST git-receive-pack (6876323 bytes)
error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502 Bad Gateway
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
Everything up-to-date
Fastidious picture Fastidious on 18 Mar 2017

Another tidbit. I installed Gitea (the bastard fork 😂) and it works fine, so it seems the issue is related to Gogs.

Fastidious picture Fastidious on 18 Mar 2017
😄1

Please post logs from files under log/hooks and log/serv.log.

I was able to create another repository, and commit a rather large set of files and directories, no issues. The problems is showing with this repository.
Another tidbit. I installed Gitea (the bastard fork 😂) and it works fine

Combine two, I don't see any logic here.

unknwon picture unknwon on 18 Mar 2017

@Unknwon there are no log/hooks and the log/gogs.log reads:

2017/03/18 14:59:26 [TRACE] Log Mode: File (Trace)
2017/03/18 14:59:26 [ INFO] Gogs 0.10.18.0313
2017/03/18 14:59:26 [ INFO] Cache Service Enabled
2017/03/18 14:59:26 [ INFO] Session Service Enabled
2017/03/18 14:59:26 [ INFO] Mail Service Enabled
2017/03/18 14:59:26 [ INFO] Register Mail Service Enabled
2017/03/18 14:59:26 [ INFO] Notify Mail Service Enabled
2017/03/18 14:59:26 [ INFO] Git Version: 2.11.1
2017/03/18 14:59:26 [ INFO] Git config user.name set to Gogs
2017/03/18 14:59:26 [ INFO] Git config user.email set to [email protected]
2017/03/18 14:59:26 [ INFO] SQLite3 Supported
2017/03/18 14:59:26 [ INFO] Run Mode: Production
2017/03/18 14:59:26 [TRACE] Doing: CheckRepoStats
2017/03/18 14:59:26 [ INFO] First-time run install finished!
2017/03/18 14:59:27 [TRACE] Session ID: 03e55b5502dc22ee
2017/03/18 14:59:27 [TRACE] CSRF Token: kFF1VCCz6eBWxDsd7ldIJ57puXc6MTQ4OTg2MzU2NzAwMTMzNDU2MA==
2017/03/18 14:59:27 [TRACE] Session ID: 03e55b5502dc22ee
2017/03/18 14:59:27 [TRACE] CSRF Token: kFF1VCCz6eBWxDsd7ldIJ57puXc6MTQ4OTg2MzU2NzAwMTMzNDU2MA==
2017/03/18 14:59:27 [TRACE] Template: user/dashboard/dashboard
2017/03/18 14:59:35 [TRACE] Session ID: 03e55b5502dc22ee
2017/03/18 14:59:35 [TRACE] CSRF Token: kFF1VCCz6eBWxDsd7ldIJ57puXc6MTQ4OTg2MzU2NzAwMTMzNDU2MA==
2017/03/18 14:59:35 [TRACE] Template: user/dashboard/dashboard
2017/03/18 14:59:36 [TRACE] Session ID: 03e55b5502dc22ee
2017/03/18 14:59:36 [TRACE] CSRF Token: kFF1VCCz6eBWxDsd7ldIJ57puXc6MTQ4OTg2MzU2NzAwMTMzNDU2MA==
2017/03/18 14:59:36 [TRACE] Template: user/dashboard/dashboard
2017/03/18 14:59:41 [TRACE] Session ID: 03e55b5502dc22ee
2017/03/18 14:59:41 [TRACE] CSRF Token: kFF1VCCz6eBWxDsd7ldIJ57puXc6MTQ4OTg2MzU2NzAwMTMzNDU2MA==
2017/03/18 14:59:41 [TRACE] Template: admin/dashboard
2017/03/18 15:01:31 [TRACE] Log Mode: File (Trace)
2017/03/18 15:01:31 [ INFO] Gogs 0.10.18.0313
2017/03/18 15:01:36 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:01:36 [TRACE] CSRF Token: T7kONwJf0QhEWTDI1NWNnfoOORU6MTQ4OTg2MzY5NjUwODAyNjA2Mg==
2017/03/18 15:01:36 [TRACE] Template: home
2017/03/18 15:01:38 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:01:38 [TRACE] CSRF Token: T7kONwJf0QhEWTDI1NWNnfoOORU6MTQ4OTg2MzY5NjUwODAyNjA2Mg==
2017/03/18 15:01:38 [TRACE] Template: user/auth/signin
2017/03/18 15:01:44 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:01:44 [TRACE] CSRF Token: T7kONwJf0QhEWTDI1NWNnfoOORU6MTQ4OTg2MzY5NjUwODAyNjA2Mg==
2017/03/18 15:01:44 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:01:44 [TRACE] CSRF Token: AoiY2AbDW3uYFMg2JmZHYO8_Fzs6MTQ4OTg2MzcwNDkxNzgzNzcxNw==
2017/03/18 15:01:44 [TRACE] Template: user/dashboard/dashboard
2017/03/18 15:01:49 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:01:49 [TRACE] CSRF Token: AoiY2AbDW3uYFMg2JmZHYO8_Fzs6MTQ4OTg2MzcwNDkxNzgzNzcxNw==
2017/03/18 15:01:49 [TRACE] Template: user/dashboard/dashboard
2017/03/18 15:01:50 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:01:50 [TRACE] CSRF Token: AoiY2AbDW3uYFMg2JmZHYO8_Fzs6MTQ4OTg2MzcwNDkxNzgzNzcxNw==
2017/03/18 15:01:50 [TRACE] Template: user/dashboard/issues
2017/03/18 15:01:51 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:01:51 [TRACE] CSRF Token: AoiY2AbDW3uYFMg2JmZHYO8_Fzs6MTQ4OTg2MzcwNDkxNzgzNzcxNw==
2017/03/18 15:01:51 [TRACE] Template: user/dashboard/issues
2017/03/18 15:01:52 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:01:52 [TRACE] CSRF Token: AoiY2AbDW3uYFMg2JmZHYO8_Fzs6MTQ4OTg2MzcwNDkxNzgzNzcxNw==
2017/03/18 15:01:52 [TRACE] Template: explore/repos
2017/03/18 15:01:55 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:01:55 [TRACE] CSRF Token: AoiY2AbDW3uYFMg2JmZHYO8_Fzs6MTQ4OTg2MzcwNDkxNzgzNzcxNw==
2017/03/18 15:01:55 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:01:55 [TRACE] CSRF Token: 559yq4OOOAu8VtJuu2N3b3mexeY6MTQ4OTg2MzcxNTY3NTAwOTc0OA==
2017/03/18 15:01:55 [TRACE] Template: home
2017/03/18 15:01:58 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:01:58 [TRACE] CSRF Token: 559yq4OOOAu8VtJuu2N3b3mexeY6MTQ4OTg2MzcxNTY3NTAwOTc0OA==
2017/03/18 15:01:58 [TRACE] Template: user/auth/signup
2017/03/18 15:02:29 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:02:29 [TRACE] CSRF Token: 559yq4OOOAu8VtJuu2N3b3mexeY6MTQ4OTg2MzcxNTY3NTAwOTc0OA==
2017/03/18 15:02:29 [TRACE] Account created: david
2017/03/18 15:02:29 [TRACE] NewMessageFrom (htmlBody):
<!DOCTYPE html>
<html>
<head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
        <title>david, please activate your account</title>
</head>

<body>
        <p>Hi <b>david</b>, thanks for registering at Gogs!</p>
        <p>Please click the following link to verify your e-mail address within <b>3 hours</b>:</p>
        <p><a href="https://git.quark.business/user/activate?code=201703181502000180e14c8b4903bdd04a463ae28da54ef86e869de1e26461766964">https://git
.quark.business/user/activate?code=201703181502000180e14c8b4903bdd04a463ae28da54ef86e869de1e26461766964</a></p>
        <p>Not working? Try copying and pasting it to your browser.</p>
        <p>© 2017 <a target="_blank" href="https://git.quark.business/">Gogs</a></p>
</body>
2017/03/18 15:02:29 [TRACE] Template: user/auth/activate
2017/03/18 15:02:29 [TRACE] New e-mail sending request [[email protected]]: UID: 2, activate account
2017/03/18 15:02:29 [TRACE] E-mails sent [[email protected]]: UID: 2, activate account
2017/03/18 15:02:39 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:02:39 [TRACE] CSRF Token: 559yq4OOOAu8VtJuu2N3b3mexeY6MTQ4OTg2MzcxNTY3NTAwOTc0OA==
2017/03/18 15:02:39 [TRACE] User activated: david
2017/03/18 15:02:39 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:02:39 [TRACE] CSRF Token: h1swrtfLEk3r4qh98IvjO-O-rPE6MTQ4OTg2Mzc1OTk5ODA3MzM1Ng==
2017/03/18 15:02:39 [TRACE] Template: user/dashboard/dashboard
2017/03/18 15:02:49 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:02:49 [TRACE] CSRF Token: h1swrtfLEk3r4qh98IvjO-O-rPE6MTQ4OTg2Mzc1OTk5ODA3MzM1Ng==
2017/03/18 15:02:49 [TRACE] Template: user/profile
2017/03/18 15:02:51 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:02:51 [TRACE] CSRF Token: h1swrtfLEk3r4qh98IvjO-O-rPE6MTQ4OTg2Mzc1OTk5ODA3MzM1Ng==
2017/03/18 15:02:51 [TRACE] Template: user/profile
2017/03/18 15:02:52 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:02:52 [TRACE] CSRF Token: h1swrtfLEk3r4qh98IvjO-O-rPE6MTQ4OTg2Mzc1OTk5ODA3MzM1Ng==
2017/03/18 15:02:52 [TRACE] Template: user/profile
2017/03/18 15:02:54 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:02:54 [TRACE] CSRF Token: h1swrtfLEk3r4qh98IvjO-O-rPE6MTQ4OTg2Mzc1OTk5ODA3MzM1Ng==
2017/03/18 15:02:54 [TRACE] Template: repo/create
2017/03/18 15:03:16 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:03:16 [TRACE] CSRF Token: h1swrtfLEk3r4qh98IvjO-O-rPE6MTQ4OTg2Mzc1OTk5ODA3MzM1Ng==
2017/03/18 15:03:16 [TRACE] Repository created [1]: david/Test
2017/03/18 15:03:16 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:03:16 [TRACE] CSRF Token: h1swrtfLEk3r4qh98IvjO-O-rPE6MTQ4OTg2Mzc1OTk5ODA3MzM1Ng==
2017/03/18 15:03:16 [TRACE] Template: repo/home
2017/03/18 15:05:29 [TRACE] Session ID: 96675a747da5817c
2017/03/18 15:05:29 [TRACE] CSRF Token: bH2bxU3vg5O-JtrjGUa0KDAGjoY6MTQ4OTg2MzkyOTM5MTIzOTM1Nw==
2017/03/18 15:02:52 [TRACE] CSRF Token: h1swrtfLEk3r4qh98IvjO-O-rPE6MTQ4OTg2Mzc1OTk5ODA3MzM1Ng==
2017/03/18 15:02:52 [TRACE] Template: user/profile
2017/03/18 15:02:54 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:02:54 [TRACE] CSRF Token: h1swrtfLEk3r4qh98IvjO-O-rPE6MTQ4OTg2Mzc1OTk5ODA3MzM1Ng==
2017/03/18 15:02:54 [TRACE] Template: repo/create
2017/03/18 15:03:16 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:03:16 [TRACE] CSRF Token: h1swrtfLEk3r4qh98IvjO-O-rPE6MTQ4OTg2Mzc1OTk5ODA3MzM1Ng==
2017/03/18 15:03:16 [TRACE] Repository created [1]: david/Test
2017/03/18 15:03:16 [TRACE] Session ID: 28fb123fbabc1e09
2017/03/18 15:03:16 [TRACE] CSRF Token: h1swrtfLEk3r4qh98IvjO-O-rPE6MTQ4OTg2Mzc1OTk5ODA3MzM1Ng==
2017/03/18 15:03:16 [TRACE] Template: repo/home
2017/03/18 15:05:29 [TRACE] Session ID: 96675a747da5817c
2017/03/18 15:05:29 [TRACE] CSRF Token: bH2bxU3vg5O-JtrjGUa0KDAGjoY6MTQ4OTg2MzkyOTM5MTIzOTM1Nw==
2017/03/18 15:05:29 [TRACE] Template: status/404
2017/03/18 15:05:49 [TRACE] Session ID: a2ad6113c8316d7b
2017/03/18 15:05:49 [TRACE] CSRF Token: Fd7-zOaWzkGl2dCaBOybcFSQK-g6MTQ4OTg2Mzk0OTUwNjY5OTE1MQ==
2017/03/18 15:05:49 [TRACE] Session ID: 77649ef57a51b66f
2017/03/18 15:05:49 [TRACE] CSRF Token: GnH7u5xynswbqqcBw_bMTqFVWqk6MTQ4OTg2Mzk0OTgwNzUzMzk2Mw==
2017/03/18 15:05:49 [TRACE] Session ID: a0fcefe816c49e59
2017/03/18 15:05:49 [TRACE] CSRF Token: 85hGdWcVDonjqq84oIRke5tQpK06MTQ4OTg2Mzk0OTg1MzY1NTgyNw==
2017/03/18 15:06:02 [TRACE] Session ID: 5e99f035697d5050
2017/03/18 15:06:02 [TRACE] CSRF Token: r7AMkLQTmChqbZ3UEjDx5lKzwbk6MTQ4OTg2Mzk2MjE2MjM4MDkyMA==
2017/03/18 15:07:25 [TRACE] Session ID: f27f1b618f1a82b5
2017/03/18 15:07:25 [TRACE] CSRF Token: W9Cvi1aFtEYwvB3Bl9SuJOQIMCk6MTQ4OTg2NDA0NTkxNjIwOTc4OA==
2017/03/18 15:07:26 [TRACE] Session ID: b8bfe15c302f7a85
2017/03/18 15:07:26 [TRACE] CSRF Token: 0116_oGeCk0F35KHpoyAzBB7v0k6MTQ4OTg2NDA0NjYwNzA2NDE1NA==
2017/03/18 15:09:00 [TRACE] Session ID: f067053f4d455657
2017/03/18 15:09:00 [TRACE] CSRF Token: 8aAUTaZ8cteLU4HiDz5k-eCMwmY6MTQ4OTg2NDE0MDU4NjgzNjcyOQ==
2017/03/18 15:09:01 [TRACE] Session ID: d1e5e202d6832dac
2017/03/18 15:09:01 [TRACE] CSRF Token: SGk69u-6iGI0BkSdyrM145VY-uw6MTQ4OTg2NDE0MTI2MDg1MjY4NA==

Again, on /var/log/syslog I notice:

Mar 18 15:09:00 rom gogs[43495]: [Macaron] 2017-03-18 15:09:00: Started GET /david/Test.git/info/refs?service=git-receive-pack for 184.89.2.2
Mar 18 15:09:00 rom gogs[43495]: [Macaron] 2017-03-18 15:09:00: Completed GET /david/Test.git/info/refs?service=git-receive-pack 200 OK in 6.695456
ms
Mar 18 15:09:01 rom gogs[43495]: [Macaron] 2017-03-18 15:09:01: Started POST /david/Test.git/git-receive-pack for 184.89.2.2
Mar 18 15:09:01 rom gogs[43495]: [Macaron] 2017-03-18 15:09:01: Completed POST /david/Test.git/git-receive-pack 401 Unauthorized in 1.373096ms

I mentioned Gitea because the problem isn't present in it. I thought that perhaps that would help. I am using Apache2 to proxy to Gogs (same setup for Gitea), following Gogs documentation.

Fastidious picture Fastidious on 18 Mar 2017

If you send me your SSH key, I can grant you access to my test server, if that helps. I really would like to use Gogs.

Fastidious picture Fastidious on 18 Mar 2017

If you're using 0.10.18.0313 and you don't have log/hooks, I suspect your Gogs binary location has been changed, but all the hooks are using old Gogs version. 

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCqTb7KvMLymGy+8TEUnbABaH/405kV32fRERby80auIwqgavLJQsVmJXojjYkb6CcgOA9fvRo98jhSbvs3CYDm3/5wBkK7GcrM3Wp3yD2RiNbQfGzaYBuQMoEobweckIazn+6Fh8H8KPErzD5b7hQp+2oTISLjok9b8GGpo+QjvWFs2qS72JFTf1mDnGmYse+zJSQeJgkPGT7RM1ryqECGUllgS3M7cm0ilshCLKMJtmHCqHmOzHdlVhVqH7HrRcY68ipn+MrCC3iOp6w9P4Hp2Sn/ZQ6PxkpwuBHA2dkJlgoH35dGe+2rvfaMq3qXAVfyiDZu8LRyipRAdS54pSv1 [email protected]
unknwon picture unknwon on 18 Mar 2017

@Unknwon I deleted everything, and started from scratch. Key added.

Fastidious picture Fastidious on 18 Mar 2017

Thanks, I think there is not much I can do on your server, more like a internal auth logic bug.

Is it the only repo returns 401 at the end?

unknwon picture unknwon on 18 Mar 2017

@Unknwon it is the only repo. But I can add as many as I want, the result will be exactly the same.

Fastidious picture Fastidious on 18 Mar 2017

it is the only repo. But I can add as many as I want, the result will be exactly the same.

Any all new repo returns 401?

unknwon picture unknwon on 18 Mar 2017

@Unknwon I am on Freenode, if you want direct exchange, under Quark. By the way, there is a #gogs channel on Freenode. FYI.

Fastidious picture Fastidious on 18 Mar 2017

@Unknwon any repo to which I submit a commit that has more than one or two files.

Fastidious picture Fastidious on 18 Mar 2017

I am on Freenode, if you want direct exchange, under Quark.

I don't know how to use Freenode...

any repo to which I submit a commit that has more than one or two files.

I just pushed two file (one of them around 46MB) via HTTPS and no error... https://try.gogs.io/unknwon/hello

unknwon picture unknwon on 18 Mar 2017

@Unknwon give this one a try, https://git.quark.business. Try using more than one or two files (I don't think it is the size, but the number).

Fastidious picture Fastidious on 18 Mar 2017

I pushed 5 4 files via HTTPS: https://git.quark.business/unknwon/issue-4305

Update: pushed another 30 files... maybe there is something different between our client side environment.

unknwon picture unknwon on 18 Mar 2017

@Unknwon could you push something bigger? Push something like the gogs repo, for example. Do not use mirror, but regular.

Fastidious picture Fastidious on 18 Mar 2017

@Unknwon when I try to push 184 files, all different sizes, but relatively small, I get the error.

Fastidious picture Fastidious on 18 Mar 2017

No error... https://git.quark.business/unknwon/gogs

unknwon picture unknwon on 18 Mar 2017

Have you tried https://github.com/gitlabhq/gitlabhq/issues/3882#issuecomment-17603883?

unknwon picture unknwon on 18 Mar 2017

I tried it, no changes.

This is so weird! This is what I get (on shell, not using Tower.app):

madhatter:Test david$ git push
Counting objects: 184, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (182/182), done.
Writing objects: 100% (184/184), 600.87 KiB | 0 bytes/s, done.
Total 184 (delta 113), reused 0 (delta 0)
error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502 Bad Gateway
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
Everything up-to-date
madhatter:Test david$
Fastidious picture Fastidious on 18 Mar 2017

This is my .gitconfig:

[core]
        excludesfile = /Users/david/.gitignore_global
[user]
        name = David Collantes
        email = [email protected]
        signingkey = B9B67D02
[commit]
        gpgsign = true
[gpg]
        program = /usr/local/Cellar/gnupg21/2.1.18_1/bin/gpg2
[http]
        postBuffer = 524288000
Fastidious picture Fastidious on 18 Mar 2017

@Unknwon what git version do you use?

Fastidious picture Fastidious on 18 Mar 2017

Locally:

➜  ~ git version
git version 2.11.1
unknwon picture unknwon on 18 Mar 2017

@Unknwon same as me (both, local, and on server). This is driving me nuts. I will walk away for a bit, eat something, and come back to test a couple of things. My main puzzle is that the issue is present in Gogs, but not the other. 😠 Will post back my findings.

Fastidious picture Fastidious on 18 Mar 2017

@Fastidious can you try push via SSH, just to make sure if it's a problem with HTTPS specifically.

unknwon picture unknwon on 18 Mar 2017

@Unknwon pushing via SSH works perfectly.

Fastidious picture Fastidious on 18 Mar 2017

Mysterious bug.

unknwon picture unknwon on 18 Mar 2017

@Unknwon indeed. It is happening to me in two VPS, both running Ubuntu 16.04, same git, same gogs version. I guess that, if I want to use gogs, I could only use SSH exclusively, but it really bothers me.

Fastidious picture Fastidious on 18 Mar 2017

It could be the thing you push is somehow different, but I can't reproduce thus can't debug..

unknwon picture unknwon on 18 Mar 2017

@Unknwon I thought so too, so I used different things to push. If I pushed only one file, or two, it worked fine. If I pushed 10, it would work, with certain things. If I pushed many (100+) will never work.

Mysterious...

Fastidious picture Fastidious on 18 Mar 2017

The thing is... still no error with 144 files https://try.gogs.io/unknwon/hello/commit/a8bc43411caefaa0ab00b6ebef9a6cc1ae0617c1 :sob::sob:

unknwon picture unknwon on 18 Mar 2017

@Unknwon do you use proxy? If so, assuming nginx, yes? I am getting on /var/log/syslog:

Mar 18 18:02:06 rom gogs[1249]: [Macaron] 2017-03-18 18:02:06: Started GET /david/Test.git/info/refs?service=git-receive-pack for 184.89.2.2
Mar 18 18:02:06 rom gogs[1249]: [Macaron] 2017-03-18 18:02:06: Completed GET /david/Test.git/info/refs?service=git-receive-pack 200 OK in 4.398879m
s
Mar 18 18:02:07 rom gogs[1249]: [Macaron] 2017-03-18 18:02:07: Started POST /david/Test.git/git-receive-pack for 184.89.2.2
Mar 18 18:02:07 rom gogs[1249]: [Macaron] 2017-03-18 18:02:07: Completed POST /david/Test.git/git-receive-pack 401 Unauthorized in 1.762409ms

And then on Apache2 logs:

[Sat Mar 18 18:02:07.604179 2017] [proxy:error] [pid 33679:tid 140349070575360] (32)Broken pipe: [client 184.89.2.2:48838] AH01084: pass request body failed to 127.0.0.1:3000 (127.0.0.1)
[Sat Mar 18 18:02:07.604237 2017] [proxy_http:error] [pid 33679:tid 140349070575360] [client 184.89.2.2:48838] AH01097: pass request body failed to 127.0.0.1:3000 (127.0.0.1) from 184.89.2.2 ()
Fastidious picture Fastidious on 18 Mar 2017

I use Caddy, without limit to post client size.

unknwon picture unknwon on 19 Mar 2017

@Unknwon installed Caddy, no problems. I am assuming it was an Apache, a mod_proxy or a mod_proxy_http problem. Weird it did not presented itself with other applications, but gogs. Shrugs.

Fastidious picture Fastidious on 19 Mar 2017
😄1

Notabug.org appears to have had a similar issue. I believe that setting SetEnv proxy-sendcl fixed the issue. I'll update this post if it turns out it didn't.

hpvb picture hpvb on 30 Mar 2017

I still have the same issue with the "0.11.4.0405" version of gogs.
Used with Ubuntu 16.04 and apache.
It works well in SSH.

Pofilo picture Pofilo on 14 Aug 2017

@Pofilo have you checked post-body size of your proxy (NGINX or Apache)?

unknwon picture unknwon on 15 Aug 2017

I had to fix the issue by moving from apache to nginx for notabug.org. There's a problem with the way mararoon deals with 401's on a POST request. From what I understand Apache (and the HTTP spec) expects a backend to consume the body of the request even if there is a 401 or other response. It seems that gogs doesn't do this and apache will then return a backend error.

hpvb picture hpvb on 15 Aug 2017

Why would a 401 error turns into 502 error? Doesn't sound right to me... @hpvb

unknwon picture unknwon on 15 Aug 2017

It isn't right. As far as I understand it the sequence of events is this:

Client > mod_proxy - HTTP POST + postdata
Mod_proxy > macaroon - HTTP POST + postdata
Macaroon > mod_proxy - HTTP 401 (without consuming the TCP packets for postdata)
Mod_proxy > client - HTTP 502 ( because of backend timeout)

What macaroon should do is return a 401 only after it has finished reading the entire request. At least, that's how I understand the problem.

hpvb picture hpvb on 15 Aug 2017

@hpvb I see... thanks for the details!

Just to clarify: Macaron is just a router, Gogs is the "one" who deals with request and response data.

I suggest file a new issue and reference to your comment here. Thanks!

unknwon picture unknwon on 16 Aug 2017

On one machine git push works as expected on another machine i am getting exactly the same error.
I am using apache reverse proxy.

git output:

Total 2087 (delta 160), reused 0 (delta 0)error: 
RPC failed; result=22, HTTP code = 502
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
Everything up-to-date

apache log:

[proxy:error] [pid 9098:tid 1870656560] (32)Broken pipe: [client 127.0.0.1:40824] AH01084: pass request body failed to 172.17.0.3:3000 (172.17.0.3)
[proxy_http:error] [pid 9098:tid 1870656560] [client 127.0.0.1:40824] AH01097: pass request body failed to 172.17.0.3:3000 (172.17.0.3) from 127.0.0.1 ()
playmyskay picture playmyskay on 20 Dec 2017

Ran also into this issue today. We have several repos, and only one cannot be pushed anymore.
Using apache2.2, gogs 0.11.34 and reverse proxy.
SetEnv proxy-sendcl 1 didn't help.
Setting the git http.postBuffer to 524288000 didn't help.

The workaround that helped was to bypass the proxy. So I changed gogs to listen on the lan ip and change the remote url to connect to port 3000 directly.
Then I could push, revert above changes, and newly pushed commits are working again.

So this seems to be a strange combination of things. The problematic push had 5 commits, maybe it's a size problem?

Corben78 picture Corben78 on 11 Jan 2018
👍1

@Corben78 workaround fixed it for me. Seems to help reset Apache reverse proxy.

netlander picture netlander on 27 Jun 2018

i Had the same issue , seems to solved for my setup.
i host gogs in my raspberry pi 2B in combination with dataplicity im able to push clone manage my repos from anywhere .
when i create a new repo gogs generate automatically some instructions to follow for push your code into.
in section :
git remote add origin http://:80/user/myrepo.git
for some reason ":80" shoudlnt be there.. i removed and im able to push normal.
PS. yeah my gogs server listen on 80 Port , Dataplicity dont support Port routing except default web and ssh so maybe that was my problem.
anyway if anyone has same problem , maybe will help.

t133 picture t133 on 25 Jul 2018

Like @netlander the workaround of bypassing the Apache proxy worked for me. It was crashing at that step: "POST git-receive-pack (7200216 bytes)", but this works normally without the proxy.

Might well be related to the 401: with the Apache mod_proxy, I did not receive an authentication request, but I received one without the proxy and had to type user/password for the commit to be accepted.

Apache error log (multiple identical lines):

[Sat Jan 12 09:12:33.710124 2019] [proxy:error] [pid 25787] (32)Broken pipe: [client 37.58.212.212:59962] AH01084: pass request body failed to 127.0.0.1:3000 (127.0.0.1)
[Sat Jan 12 09:12:33.710162 2019] [proxy_http:error] [pid 25787] [client 37.58.212.212:59962] AH01097: pass request body failed to 127.0.0.1:3000 (127.0.0.1) from 37.58.212.212 ()

Gogs log/hooks/post-receive.log (9:20 is without the proxy, nothing visible at 9:12 with the proxy):

2019/01/12 09:20:41 [ERROR] [...ogs/models/action.go:493 CommitRepoAction()] UpdateIssuesCommit: invalid issue reference [ref: 09/01/2019]

Other log/hooks/* files are touched but empty

Portion of the log/gogs.log which appears relevant, without the proxy:

2019/01/12 09:20:41 [TRACE] HTTPGit - Authenticated user: fgm
2019/01/12 09:20:41 [TRACE] Session ID: d8376360d674a0ea
2019/01/12 09:20:41 [TRACE] CSRF Token: FKyzeZELo1j4eoGE-TKp3yDmyc06MTU0NzI4NDg0MTU0MjMxMjM2MA==
2019/01/12 09:20:41 [TRACE] Detected encoding: UTF-8 (fast)
2019/01/12 09:20:41 [TRACE] Template: repo/home
2019/01/12 09:20:41 [TRACE] Session ID: 9e0fc1184eb4970b
2019/01/12 09:20:41 [TRACE] CSRF Token: M0fhCH0kbQAhgfPKdmaXSga1DPk6MTU0NzI4NDg0MTkxMjIwMDAwOA==
2019/01/12 09:20:41 [TRACE] TriggerTask 'assets__prague2013_blocks/blocks-1-8' by 'fgm'
2019/01/12 09:20:41 [TRACE] DeliverHooks [repo_id: 112]
2019/01/12 09:20:41 [TRACE] AddTestPullRequestTask [head_repo_id: 112, head_branch: blocks-1-8]: finding pull requests
2019/01/12 09:20:41 [TRACE] AddTestPullRequestTask [base_repo_id: 112, base_branch: blocks-1-8]: finding pull requests

More interestingly, updating my HTTPS Vhost to look like this, as suggested on https://httpd.apache.org/docs/2.4/en/mod/mod_proxy.html#envsettings , fixed the issue for me. It causes the browser to ask for credentials (triggering keychain access in macOS), and succeeds:

  ProxyPreserveHost On
  ProxyRequests off
  ProxyPass / http://127.0.0.1:3000/
  ProxyPassReverse / http://127.0.0.1:3000/
  SetEnv force-proxy-request-1.0 1
  SetEnv proxy-nokeepalive 1

This might suggest a problem with HTTP/1.1 keepalive handling in Gogs or its dependencies.

fgm picture fgm on 12 Jan 2019
👍2

I had this error with Gitlab behind an Nginx proxy with pushing large files.

$ git push origin master
warning: redirecting to http://xxxx.xxxx.xxx/xxxx/xxxx.git/
Uploading LFS objects: 100% (2/2), 408 MB | 0 B/s, done                                                                                                                                                                                                 
Enumerating objects: 8, done.
Counting objects: 100% (8/8), done.
Delta compression using up to 6 threads
Compressing objects: 100% (8/8), done.
Writing objects: 100% (8/8), 386.08 MiB | 8.74 MiB/s, done.
Total 8 (delta 0), reused 0 (delta 0)
error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502
fatal: the remote end hung up unexpectedly
fatal: the remote end hung up unexpectedly
Everything up-to-date

This was my solution for nginx based on @fgm and the keepalive clue:
(SSL terminated at a higher layer, this is for the internal LB in front of docker containers)
Narrowed down the issue to proxy_http_version, keepalive and removing the connection header. 

http {
...
    upstream gitlab {
        keepalive 100;
        server 00.000.0.0:8080;
    }

server {
        listen 80;
        server_name gitlab.entouch.jp;

        location / {
            add_header Strict-Transport-Security "max-age=15768000;";
            keepalive_timeout 120s;

            proxy_read_timeout 3600s;
            proxy_http_version 1.1;

            # Remove Connection header, client might send "close" to close a keepalive connection
            proxy_set_header Connection "";
            proxy_set_header Host $host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto https;
            proxy_set_header X-Forwarded-Host $host:$server_port;
            proxy_set_header X-Forwarded-Server $host;

            proxy_pass http://gitlab/;
        }
    }
...
}

Something else that can help to narrow down the cause:
GIT_CURL_VERBOSE=1 GIT_TRACE=1 git push origin master

aogilvie picture aogilvie on 18 Oct 2019
❤1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

hhenkel picture hhenkel  ·  44Comments
nd2s picture nd2s  ·  45Comments
strk picture strk  ·  41Comments
cameron314 picture cameron314  ·  39Comments
bastiao picture bastiao  ·  125Comments