Gogs: SSH server is not working

[matrixise]

The module SSH is just not working, sorry but with this code:
https://github.com/gogits/gogs/blob/master/modules/ssh/ssh.go#L109 and this line https://github.com/gogits/gogs/blob/master/modules/ssh/ssh.go#L59 . How can it work?

Or I don't understand your ssh module.

[tboerger]

This module is not used...

[matrixise]

and ? can we use ssh ?

[tboerger]

This is just a module where they plan to build some custom minimal SSHd. Currently gogs always uses a separate SSH server that have to be installed on the system

[matrixise]

is there a config or a "how to" for ssh ?

[tboerger]

SSH is just enabled by default. Just start whichever SSH server you want, add an sshkey to your gogs user and configure the ssh port within the config (if the server is not running on port 22)

[matrixise]

SSH enabled ? sorry, but I don't understand, which ssh server, if the ssh module is "deprecated"/"unstable"

If I add a ssh key to a user, where the key will be stored? in the database or in the .ssh/authorized_keys of the "git" user ?

[tboerger]

Gogs is using the SSH daemon from your server. The keys will be stored in the database and automatically written to the authorized_keys file of the user running gogs.

[unknwon]

The module SSH is just not working, sorry but with this code:
https://github.com/gogits/gogs/blob/master/modules/ssh/ssh.go#L109 and this line https://github.com/gogits/gogs/blob/master/modules/ssh/ssh.go#L59 . How can it work?

Sorry about confusion, but this file is not in use currently.

[matrixise]

in this case, remove it, because there is a real confusion.

I suppose than gogs will use the .ssh/authorized_keys will be used by the sshd server and the file will be filed by gogs.

[unknwon]

in this case, remove it, because there is a real confusion.

The problem is it's not junk code, and getting to in use. :sweat_smile:

[unknwon]

Close for now but open for continuing discussion.

[HorseLuke]

Does it mean that even if START_SSH_SERVER = true in v0.7.6 or above, it can not pull/push repo by built-in SSH Server?

I tested in Windows 10, Git Client received error:

Gogs: Internal error
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Gogs console outputted such error:

2015/11/23 11:28:45 [I] SSH server started on :3001
2015/11/23 11:28:45 [I] Listen: https://0.0.0.0:3000
2015/11/23 11:28:50 [T] Connection from 127.0.0.1:52163 (SSH-2.0-PuTTY_Unidentified_Local_Build)
2015/11/23 11:28:50 [T] Payload: git-upload-pack '/admin1/testa.git'
2015/11/23 11:28:50 [...s/modules/ssh/ssh.go:97 func1()] [E] Wait: exit status 1

[unknwon]

@HorseLuke can you check path/to/gogs/log/serv.log?

[EtienneBruines]

@Unknwon As I am experiencing the exactly same problem (as in https://github.com/gogits/gogs/issues/1681#issuecomment-158846295, with prob. the same setup), I'm answering that question:

2015/11/23 15:33:10 [...ts/gogs/cmd/serve.go:155 runServ()] [F] Failed to get repository owner(etiennebruines): resource temporarily unavailable

[unknwon]

@EtienneBruines the error you have seems like failed DB query.

[EtienneBruines]

@Unknwon Isn't that strange, with a new installation?

[unknwon]

@EtienneBruines It is strange, but I think nothing to do with Gogs: http://serverfault.com/questions/687026/su-bin-bash-resource-temporarily-unavailable

[EtienneBruines]

@Unknwon Looking into it now; as I'm using tidb which I selected using the Gogs installation wizard, it at least has something to do with it ;-).

[unknwon]

@EtienneBruines blame application for database error is quite a easy thing.

[EtienneBruines]

@Unknwon on a blank installation, it's usually the case. Whether it a missing dependency that wasn't documented, or something else. Could be a flaw in tidb, though. Debugging now :-). (but as tidb is included in the gogs binary, it's related in some way)

[unknwon]

@EtienneBruines I don't play word game.

[EtienneBruines]

@Unknwon Is everything run under the same USER as the gogs binary? (I could probably find out, but that'd take hours) - trying to cover if it's a permissions error.

[unknwon]

@EtienneBruines Yes.

[EtienneBruines]

@Unknwon I've tried with MySQL, and after fixing some permissions (which I messed up for looking at https://github.com/gogits/gogs/issues/1681#issuecomment-158960829), it works as it should. (Thank you for pointing me in that direction; I did in no way mean to offend you :-( )

Not sure what the tidb error was, though. Maybe tidb and ARM don't go that well together.

[unknwon]

@EtienneBruines :trollface: Tests are passed on ARM for TiDB, but they do not code on ARM and no wish to support at current.

[HorseLuke]

@Unknwon serv.log could not found, only found gogs.log, http.log, and xorm.log.
Seems strange.

I set log.ROOT_PATH on purpose to collect log, but serv.log was missing too.

[log]
ROOT_PATH = log222
MODE = file
LEVEL = Trace

[unknwon]

@HorseLuke Ok..Hmm, what's in http.log?

[HorseLuke]

@Unknwon empty content -_-||

Contents of gogs.log is the same as https://github.com/gogits/gogs/issues/1681#issuecomment-158846295 in which I mentioned

[unknwon]

Maybe you get to the wrong location, if Gogs: Internal error is presented, error is logged...

[HorseLuke]

Er? I change log.ROOT_PATH 3 times to reproduced operation, but every time it really only logged what i reported above, no log that you want was founded.
Give up this problem, but I will keep an eye on it. -_-||

[unknwon]

@HorseLuke OK... I'll test on win10 see if I can reproduce.

[unknwon]

@HorseLuke I have found the bug, and the solution (not sure if it breaks anything else for SSH on Windows).

The log is indeed attempted to write to serv.log, but closes too quick, so not get chance flush to file system.

Haven't got an idea why log cannot be wrote to file system on Windows, all checks show that log is successful. My quick solution to this is to print error to client side when RUN_MODE = dev.

And the actual error is git-upload-pack not found. I'm not quite familiar with Windows so I don't know where to find it either.

But I think git upload-pack is equivalent to git-upload-pack, so I will make a fix to it.

[unknwon]

@HorseLuke pushed fixes to develop.

[pecigonzalo]

This is great as it starts the ssh server without openssh/cygssh!!!

but there is still an issue with:

Sorry, we're not able to verify your SSH key: ssh-keygen -lf: line 1 too long: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCq... C:\\Windows\\TEMP\\keytest672583434 is not a public key file.

when trying to add a key.

[vivace-io]

Make sure when you paste the SSH *.pub key in to the content box, you remove the host name that gets added to the end? Does that help or have you already tried that?

[pecigonzalo]

Thanks @vivace-io, i tried that, i suspected permissions but the Gogs is running as local system on this test instance.
Ill try and find the portion of the code that does the key add and see what it is actually doing.

[pecigonzalo]

Ok, something was wrong on my side and the key was being generated incorrectly.

Thanks

[fanningert]

I also get the same problem.

OS: ArchLinux
DB: SQLite

console output when I am clone via git.

# git clone ssh://[email protected]:9922/user/pass.git
Cloning into 'pass'...
Gogs: Internal error
Failed to get repository owner(fanningert): dial tcp 127.0.0.1:3306: getsockopt: connection refusedfatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

I activate the log level trace and get following output for the clone request.

Nov 29 09:17:02 alarmpi gogs[11110]: 2015/11/29 09:17:02 [T] Connection from 192.168.1.100:60138 (SSH-2.0-OpenSSH_7.1)
Nov 29 09:17:02 alarmpi gogs[11110]: 2015/11/29 09:17:02 [T] Payload: git-upload-pack '/user/pass.git'
Nov 29 09:17:02 alarmpi gogs[11110]: 2015/11/29 09:17:02 [...s/modules/ssh/ssh.go:97 func1()] [E] Wait: exit status 1

XORM.log when I would like to clone the repository

[xorm] [info]  2015/11/29 09:17:02.479744 [sql] SELECT `id`, `owner_id`, `name`, `fingerprint`, `content`, `mode`, `type`, `created`, `updated` FROM `public_key` WHERE content like ? LIMIT 1 [args] [ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDm2dO0iqkucMQdP08T4GArMzYis9zia8Pxgar0VnR9afvoL71CIOzGQ5RjQdpx5fCUO8r8c4y/KG/TZhggJe3vavun0/iY2vMUfEhZdjHYo1Y6OWGN0tJ9Ge7oTOvAJ0v7BQc2i0wzgaqNIrdW5rzCE+AmtXBzpMhXYriOso6rWGmzF0p3UHw4rjsqTl3oSs7BsgibuypmKiRH3cqULbL66jVkSRBGq+ncpKuNQjwbV9A9rndhfjB65FKjiS607SMBcuUuG2mYzssg728RualaDYoXlbc/TXAzdD1grFSlhtM/rSGyiaoERvpVGNv+LWYpAY5wpYJVoayFqoRerZn/PbKrwq81hsCxKKo0qvsVGAFjlXHNpquLwEzeHkevO9AeRn9wqgGerF7DMUYf951RynSmkeO5QNzgEDX2p5wbWDKHrzsDZ+0fAS300yQq0MiohqNsZ2v0quqeRXdo97y/V9C5PRTLEy2rLdwVmdTjVYLiEYCuWdGALRtjteHoXAT1+ZL0RQQE9kffHPhe2INuRwf6D+491wI2zH7tEoJL3QHGTO62NW2rKitg94SHdvPRvvucQkhBiOd4SDaNjN7wDz/V9gnxTh8RR2b8+RMUxaP9QX/IDZuoXy+i3ZYCiRgvU1qeOEmAMf59cAJM7BsHhW1A3d7ZpanZazDnPaKPIw==%]

Can be the problem the % at the end of the public key in the args? In the public key that I add it, there is no % after the last ==.

[unknwon]

Can be the problem the % at the end of the public key in the args? In the public key that I add it, there is no % after the last ==.

No the % is on purpose for LIKE statement.

However, the error shows Gogs tries to connect to MySQL other than SQLite3, can you make sure your update hook is up-to-date with Gogs binary path?

[fanningert]

What did you mean with "can you make sure your update hook is up-to-date with Gogs binary path?".
I installed the current master branch on my laptop and activated the same port (9922) for ssh.

I copied the clone url from the webfrontend.

% git clone ssh://gogs@localhost:9922/fanningert/test.git                                       :(
Cloning into 'test'...
ssh: connect to host localhost port 9922: Connection refused
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

But I get the same error.

Nov 30 17:57:44 tfux32vd gogs[13298]: 2015/11/30 17:57:44 [T] Connection from [::1]:56920 (SSH-2.0-OpenSSH_7.1)
Nov 30 17:57:44 tfux32vd gogs[13298]: 2015/11/30 17:57:44 [T] Payload: git-upload-pack '/fanningert/test.git'
Nov 30 17:57:44 tfux32vd gogs[13298]: 2015/11/30 17:57:44 [...s/modules/ssh/ssh.go:97 func1()] [E] Wait: exit status 1

[unknwon]

@fanningert

[unknwon]

The new error you post:

ssh: connect to host localhost port 9922: Connection refused

Looks like the SSH server is not started at all.

[fanningert]

Sorry, I saw now that on my laptop (master branch) I get a different error then on my server (release 0.7.22). But could it possible that the current problem is that the ssh server only listen in ipv6 and not also on ipv4. #2055
I checked that the ssh server listen (so it is running) on the ipv6 port 9922 but not on the ipv4 port 9922.

I will run the script on my server and laptop. The result I will post here.

[fanningert]

Ok, I start the current master branch on my laptop. Here the console output.

Nov 30 19:14:55 tfux32vd gogs[1105]: 2015/11/30 19:14:55 [I] Gogs: Go Git Service 0.7.22.1125 Beta
Nov 30 19:14:55 tfux32vd gogs[1105]: 2015/11/30 19:14:55 [I] Log Mode: Console(Trace)
Nov 30 19:14:55 tfux32vd gogs[1105]: 2015/11/30 19:14:55 [I] Cache Service Enabled
Nov 30 19:14:55 tfux32vd gogs[1105]: 2015/11/30 19:14:55 [I] Session Service Enabled
Nov 30 19:14:55 tfux32vd gogs[1105]: 2015/11/30 19:14:55 [I] Git Version: 2.6.2
Nov 30 19:14:55 tfux32vd gogs[1105]: 2015/11/30 19:14:55 [I] SQLite3 Supported
Nov 30 19:14:55 tfux32vd gogs[1105]: 2015/11/30 19:14:55 [I] Run Mode: Production
Nov 30 19:14:55 tfux32vd gogs[1105]: 2015/11/30 19:14:55 [T] Doing: CheckRepoStats
Nov 30 19:14:55 tfux32vd gogs[1105]: 2015/11/30 19:14:55 [I] SSH server started on :9922
Nov 30 19:14:55 tfux32vd gogs[1105]: 2015/11/30 19:14:55 [I] Listen: http://0.0.0.0:3000

And here the listen ports output with the command netstat -tulpn | grep 9922. (No ipv4 port)

tcp6       0      0 :::9922                 :::*                    LISTEN      -

After the start I run the script from your answer.

Ok, now I run git clone.

% git clone ssh://gogs@localhost:9922/fanningert/test.git
Cloning into 'test'...
Gogs: Internal error
Failed to get repository owner(fanningert): dial tcp 127.0.0.1:3306: getsockopt: connection refusedfatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Here the console log.

Nov 30 19:19:26 tfux32vd gogs[1105]: 2015/11/30 19:19:26 [T] Connection from [::1]:55528 (SSH-2.0-OpenSSH_7.1)
Nov 30 19:19:26 tfux32vd gogs[1105]: 2015/11/30 19:19:26 [T] Payload: git-upload-pack '/fanningert/test.git'
Nov 30 19:19:26 tfux32vd gogs[1105]: 2015/11/30 19:19:26 [...s/modules/ssh/ssh.go:97 func1()] [E] Wait: exit status 1

[unknwon]

Have you done:

https://github.com/gogits/gogs/issues/1681#issuecomment-160704485

[fanningert]

Yes I executed the function over the web frontend, before I retried the command git clone. Did I get better log output, when I change from productive to develop mode?

[unknwon]

The error is still that Gogs tries to connect to MySQL other than SQLite3.

PS: I'm hurry to class, will come back later and see where is the problem.

[fanningert]

When you need any information, I will post it or test it. No problem for me.
In the mean time I will retest the release version on my server also.

[unknwon]

@fanningert I think I found the problem, the builtin SSH server does not carry --config argument.

[fanningert]

Ok, i also test it with the 0.7.22. There is the same error message after I run the hook reset script.

[unknwon]

@fanningert I just pushed a commit to develop branch, please help test again!

[fanningert]

No problem. I will install the current develop branch. Give me some minutes.

[fanningert]

@Unknwon: It's looks good. The clone process is now working. Tomorrow I will test some more git commands, with some more data.

git clone output

% git clone ssh://gogs@localhost:9922/fanningert/test.git
Cloning into 'test'...
remote: Counting objects: 3, done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (3/3), done.
Checking connectivity... done.

console log

Nov 30 21:46:00 tfux32vd gogs[4443]: 2015/11/30 21:46:00 [T] Connection from [::1]:53966 (SSH-2.0-OpenSSH_7.1)
Nov 30 21:46:00 tfux32vd gogs[4443]: 2015/11/30 21:46:00 [T] Payload: git-upload-pack '/fanningert/test.git'
Nov 30 21:46:00 tfux32vd gogs[4443]: 2015/11/30 21:46:00 [T] Arguments: [serv key-1 --config=/srv/gogs/conf/app.ini]

[unknwon]

@fanningert Awesome!

[quantonganh]

cat templates/.VERSION
0.8.10.1217

The problem is: the built-in ssh server is just stop working suddenly. Here's what happened when I perform a git fetch/pull:

ssh_exchange_identification: read: Connection reset by peer
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

and the log (at Info level):

2015/12/24 07:41:02 [...s/modules/ssh/ssh.go:99 func1()] [E] Wait: exit status 1
2015/12/24 07:42:08 [...s/modules/ssh/ssh.go:127 listen()] [E] Error on handshaking: ssh: unexpected message type 1 (expected 50)

We can fix it by restarting the gogs daemon.

I've just changed the log level to debug and restart the gogs.
I will post the details log here when it happens again.

[unknwon]

@quantonganh you may want to check path/to/gogs/log/serv.log file as well.

[quantonganh]

Even turning on the Debug log level, the message is not details than the Info log level:

2015/12/30 04:18:35 [...s/modules/ssh/ssh.go:127 listen()] [E] Error on handshaking: EOF
2015/12/30 04:21:04 [...s/modules/ssh/ssh.go:99 func1()] [E] Wait: exit status 1

@quantonganh you may want to check path/to/gogs/log/serv.log file as well.

There are a lot of "Key access denied" errors in the serv.log file (I asked a question here: https://github.com/gogits/gogs/issues/2280)

2015/12/30 04:18:07 [...ts/gogs/cmd/serve.go:212 runServ()] [F] Key access denied: 16-46

I'm not sure if it might be the culprit, something like hosts.deny but the thing is the IP address that caused "Key access denied" error is totally different from some hosts which are working fine, then suddenly stop.

[unknwon]

@quantonganh I think you're using deploy key as public key, is that the case?

[quantonganh]

I think you're using deploy key as public key, is that the case?

I'm not sure I understand your question. If you think that I take the deploy key and paste into the SSH Keys in my account, then the answer is no (Deploy key is for CI server). If not, please clarify.

As I said in the issue #2280, I found the client that caused "Key access denied" error and fixed it, but the hanging problem still happen. What can I do to debug in this case?

[unknwon]

@quantonganh I may describe it wrongly.

The only place for your error: https://github.com/gogits/gogs/blob/master/cmd%2Fserve.go#L212

Is that you are using a deploy key from repo A to pull from repo B.

[quantonganh]

@Unknwon Please ignore the "Key access denied" error because it had been fixed.

Is that you are using a deploy key from repo A to pull from repo B.

No.

The problem is the built-in ssh server seems to be hanging randomly. I cannot reproduce that using something like this:

for i in $(seq 1 100); do
    git pull/fetch ...
done

and there is no details log in gogs.log, serv.log, ... even when log level is Trace.

Do you have any opinion about this?

[unknwon]

Oh... OK.

The problem is the built-in ssh server seems to be hanging randomly. I cannot reproduce that using something like this:

Which Git client are you using? Some people has met this for non-command line Git client tool.

[quantonganh]

Which Git client are you using?

/usr/local/bin/git --version
git version 2.5.0

[unknwon]

Looks right to me...

[quantonganh]

Ah, notice that we have some CI servers that use Git Plugin (version 2.4.0) to clone repository to build job.

[unknwon]

Oh... I suppose it is Java and uses JGit? If so, it is the tool that has problem with builtin SSH server.

[quantonganh]

https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin#GitPlugin-WhyNotJGit

As of 1.2.0, the Git plugin uses git-client-plugin for all Git low-level operation. git-client was extracted from git plugin 1.1.x code base, to ensure SoC and allow other plugins (gerrit, git-parameters...) to directly use and contribute to this one when needed.

The git-client plugin 1.0.4 used JGit by default, while still including the command line Git implementation as an alternate implementation. Initial deployments of the JGit based plugin exposed a number of gaps in the JGit implementation. Those gaps need to be resolved in the JGit implementation before it can be used as the default implementation. Beginning with git-client-plugin 1.0.5, the command line implementation is the default implementation.

As we are using Git client plugin version 1.19.0, so it's the command line, not JGit.

[unknwon]

@quantonganh no ideas right now..

[oludouglas]

Facing the same problem while pushing 1.2Gb repo to remote. Gogs just terminates