Ghidra: Ghidra Server timeout

[Arcnor]

I'm having some trouble running the Ghidra server on a VPS I have (an OpenVZ one).

The server starts, then just times out. If I increase the timeout to something like hours, I can use the server for that period of time without any problems (so it doesn't look like server load or memory problems, the VPS is doing literally nothing else), everything seems to work great on the Ghidra side, but I'd love to be able to run this without having to set a timeout of days, if possible. I'm adding my log here to see if it shows something, although I can't really see anything interesting.

Also, my server.conf is the stock one, I haven't changed anything although I've tried disabling the "fork hack" and running as another user (I'm running as root now). I also don't have SELinux enabled.

Thanks in advance.

12865/0|[INFO] StandardFileSystemManager - Using "/tmp/vfs_cache" as temporary files store.
12865/0|INFO  Using log config file: jar:file:/root/ghidra_9.0/Ghidra/Framework/Generic/lib/Generic.jar!/generic.log4j.xml (LoggingInitialization)
12865/0|INFO  Using log file: /root/ghidra_9.0/repositories/server.log (LoggingInitialization)
12865/0|INFO  Initializing SSL Context (SSLContextInitializer)
12865/0|INFO  Initializing Random Number Generator... (SecureRandomFactory)
12865/0|INFO  Random Number Generator initialization complete: NativePRNGNonBlocking (SecureRandomFactory)
12865/0|INFO  Trust manager disabled, cacerts have not been set (ApplicationTrustManagerFactory)
12865/0|INFO  Using self-signed certificate: CN=GhidraServer (ApplicationKeyManagerFactory$ApplicationKeyManager)
12865/0|INFO     defaultsigkey: GhidraServer, issued by GhidraServer, S/N 165cf7e8, expires Sun Mar 14 06:43:27 EDT 2021 (ApplicationKeyStore)
12865/0|INFO  Ghidra Server 9.0 (GhidraServer)
12865/0|INFO     Server bound to XX.XX.XX.XX (GhidraServer)
12865/0|INFO     RMI Registry port: 13100 (GhidraServer)
12865/0|INFO     RMI SSL port: 13101 (GhidraServer)
12865/0|INFO     Block Stream port: 13102 (GhidraServer)
12865/0|INFO     Block Stream compression: enabled (GhidraServer)
12865/0|INFO     Root: ./repositories (GhidraServer)
12865/0|INFO     Auth: Password File (GhidraServer)
12865/0|INFO     Prompt for user ID: no (GhidraServer)
12865/0|INFO     Anonymous server access: disabled (GhidraServer)
12865/0|INFO  root starting Ghidra Server... (GhidraServer)
12865/0|INFO  Instantiating Repository Manager for /root/ghidra_9.0/repositories (RepositoryManager)
12865/0|INFO  Instantiating User Manager (w/password management) (UserManager)
12865/0|INFO  User file contains 2 entries (UserManager)
12865/0|INFO  Known Users: (UserManager)
12865/0|INFO     test (UserManager)
12865/0|INFO     arcnor (UserManager)
12865/0|INFO  Known Repositories: (RepositoryManager)
12865/0|INFO     TEstingRepo (RepositoryManager)
12865/0|INFO  Loading TEstingRepo ... (Repository)
12865/0|INFO     ... loading 0 files ... (Repository)
12865/0|INFO     TEstingRepo load complete.  (Repository)
12865/0|INFO  Started Block Stream Server on XX.XX.XX.XX:13102 (BlockStreamServer)
12865/0|INFO  Registering Ghidra Server... (GhidraServer)
12865/0|INFO  Registered Ghidra Server. (GhidraServer)
12865/0|startup of java application timed out. if this is due to server overload consider increasing wrapper.startup.timeout
wrapper|Shutting down Wrapper
12865/0|no connection to wrapper during 10 seconds -> System.exit(-1)
12865/0|if this is due to server overload consider increasing yajsw configuration property wrapper.startup.timeout
wrapper|waitpid 12865 65280
wrapper|exit code posix process: 65280 application(status/signal): 255/0

[ghost]

Hey @Arcnor,

Did you already uncomment the following lines in server.conf?

# Uncomment to enable remote debug support
#wrapper.java.additional.9=-Xdebug
#wrapper.java.additional.10=-Xnoagent
#wrapper.java.additional.11=-Djava.compiler=NONE
#wrapper.java.additional.12=-Xrunjdwp:transport=dt_socket\,server=y\,suspend=n\,address=*:18200
#wrapper.startup.timeout=0
#wrapper.ping.timeout=0

I also suggest you edit your post to mask your server's IP address.

Do you have the required ports open? and could you post your server.conf here?

[Arcnor]

Thanks @quosego , didn't realize my IP was showing there. As far as I understood, those are for debugging the app and not the wrapper, right? Not sure what that will accomplish in this situation? As I said the app itself works. But I'll try that, of course.

What I found is that there are options to log debugging info of the wrapper itself:

wrapper.debug = true
wrapper.debug.level = 3

With this, I get the following extra info when the wrapper is closing, although I'm not closer to the solution:

14221/0|INFO  Registering Ghidra Server... (GhidraServer)
14221/0|INFO  Registered Ghidra Server. (GhidraServer)
14221/0|exit on main terminate -1
14221/0|WrapperManger did not log on within timeout of 10000
14221/0|Controller State: WAITING -> STARTUP_TIMEOUT
14221/0|startup of java application timed out. if this is due to server overload consider increasing wrapper.startup.timeout
wrapper|giving up after 0 retries
wrapper|set state RUNNING->STATE_ABORT
wrapper|stop from Thread yajsw.controller.scheduler-0 reason: USER
wrapper|process not in state RUNNING -> Delaying stop
wrapper|set state STATE_ABORT->IDLE
14221/0|Controller State: STARTUP_TIMEOUT -> WAITING_CLOSED
14221/0|session closed -> waiting
wrapper|Shutting down Wrapper
14221/0|no connection to wrapper during 10 seconds -> System.exit(-1)
14221/0|if this is due to server overload consider increasing yajsw configuration property wrapper.startup.timeout
14221/0|INFO  Ghidra server terminated. (GhidraServer)
wrapper|waitpid 14221 65280
wrapper|exit code posix process: 65280 application(status/signal): 255/0
14221/0|process terminated
14221/0|Controller State: WAITING_CLOSED -> PROCESS_KILLED
wrapper|giving up after 0 retries
wrapper|set state IDLE->STATE_ABORT
wrapper|set state STATE_ABORT->IDLE

I do have the required ports opened (I've checked with nmap, they are listening) and I can post my server.conf, of course:

#********************************************************************
# Service Wrapper Properties
#
# NOTE! The following properties contained within this file may be
# modified by the ghidraSvr script:
#   - Wrapper.fork_hack
#********************************************************************

# Initial Working Directory (i.e., absolute installation directory path)
wrapper.working.dir=${ghidra_home}

# Mac OS X launchd plist directory
wrapper.launchd.dir=/Library/LaunchDaemons

# Java Application
wrapper.java.command=${java}

# Establish default permissions for generated files
wrapper.java.umask=027

# Java Classpath
include=${classpath_frag}

# Java Library Path (location of native authentication support libraries)
wrapper.java.library.path.1=${os_dir}

# Java Additional Parameters
wrapper.java.additional.1=-Djava.net.preferIPv4Stack=true

# Establishes a minimum number of rolling server.log backups to be maintained
wrapper.java.additional.2=-DApplicationRollingFileAppender.maxBackupIndex=10

# Ensure that classpath_frag is defined for service startup
wrapper.java.additional.3=-Dclasspath_frag=${classpath_frag}

# A suitable cacerts file must be installed when using PKI authentication
#wrapper.java.additional.4=-Dghidra.cacerts=./Ghidra/cacerts

# If Ghidra clients must authenticate the server, the server will need to install
# a server key/certificate in a secure location (e.g., /etc/pki/...)
# and specify the location and password via the properties below.
# Be sure to properly set permissions on the Ghidra installation and this file
# if using these settings.
#wrapper.java.additional.5=-Dghidra.keystore=
#wrapper.java.additional.6=-Dghidra.password=

# Temporary Directory Setting - uncomment the following setting to override the Java default.
# This may be necessary on certain Windows platforms when installing as a service.
#wrapper.java.additional.7=-Djava.io.tmpdir=C:\\Windows\\Temp

# Enable/Disable use of compression for DataBuffer serialization and Block Streams
wrapper.java.additional.8=-Ddb.buffers.DataBuffer.compressedOutput=true

# Uncomment to enable remote debug support
#wrapper.java.additional.9=-Xdebug
#wrapper.java.additional.10=-Xnoagent
#wrapper.java.additional.11=-Djava.compiler=NONE
#wrapper.java.additional.12=-Xrunjdwp:transport=dt_socket\,server=y\,suspend=n\,address=*:18200
wrapper.startup.timeout=10
wrapper.ping.timeout=10

# Optional debug enablement instead of using the wrapper.java.additional arguments above
# This will cause application to start in a suspended state
#wrapper.java.debug.port=18200

# Uncomment to enable remote use of jvisualvm for profiling
# See JMX documentation for more information: http://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html
#wrapper.java.additional.13=-Dcom.sun.management.jmxremote.port=9010
#wrapper.java.additional.14=-Dcom.sun.management.jmxremote.local.only=false
#wrapper.java.additional.15=-Dcom.sun.management.jmxremote.authenticate=false
#wrapper.java.additional.16=-Dcom.sun.management.jmxremote.ssl=false

# Do not use process fork - use exec instead (MUST BE ENABLED FOR Mac OS X)
#  -Should be "true" for Mac OS X and Linux
#  -Should be commented-out or "false" for Windows
#  -Should remain commented-out in source control and initial distribution
wrapper.fork_hack=true

# Pipe server output to console/log
#wrapper.console.pipestreams=true

# Monitor for Deadlock
wrapper.java.monitor.deadlock = true

# Main server application class
wrapper.java.app.mainclass=ghidra.server.remote.GhidraServer

# Initial Java Heap Size (in MB)
wrapper.java.initmemory=396

# Maximum Java Heap Size (in MB)
# See svrREADME.txt file for advice (Server Memory Considerations)
wrapper.java.maxmemory=768

# Specify the directory used to store repositories. This directory must be dedicated to this
# Ghidra Server instance and may not contain files or folders not produced by the
# Ghidra Server or its administrative scripts.  Relative paths originate from the
# Ghidra installation directory, although an absolute path is preferred if not using default.
# This variable is also used by the svrAdmin script.

ghidra.repositories.dir=./repositories

# Ghidra server startup parameters.
#
# Command line parameters: (Add command line parameters as needed and renumber each starting from .1)
#   [-ip###.###.###.###] [-p#] [-a#] [-anonymous] [-ssh] [-d<ntDomain>] [-e<days>] [-u] [-n] <repositories_path>
#
#   -ip###.###.###.### : ip address to bound to (by default, uses IP address bound to hostname)
#   -p# : base TCP port to be used (default: 13100)
#   -a# : an optional authentication mode where # is a value 0 or 2
#         0 - Private user password
#         1 - NT Login Password (domain authentication requires -d option)
#         2 - PKI Authentication
#         3 - NT Login Password (domain authentication requires -d option) with optional private user password support
#   -anonymous : enables anonymous repository access (see svrREADME.html for details)
#   -ssh : enables SSH authentication for headless clients
#   -d<ntDomain> : specifies NT Domain to be used for authentication (-a1 and -a3 mode only))
#   -e<days> : specifies default password expiration time in days (-a0 mode only, default is 1-day)
#   -u   : enable users to be prompted for user ID (does not apply to -a2 PKI mode)
#   -n   : enable reverse name lookup for IP addresses when logging (requires proper configuration
#          of reverse lookup by your DNS server)
#   ${ghidra.repositories.dir} : config variable (defined above) which identifies the directory
#                       used to store repositories.  Use of this variable to define the
#                       repositories directory must be retained.
wrapper.app.parameter.1=-a0
wrapper.app.parameter.2=${ghidra.repositories.dir}

# Establish server process owner
# This should only be used when installing as a service using a nologin
# local user account.  Establishing a suitable account is left as a
# system administration task.  NOTE: the use of this feature is not
# yet supported for Windows installations.
#wrapper.app.account=ghidra

#********************************************************************
# Service Wrapper Logging Properties
#********************************************************************
# Format of output for the console.  (See docs for formats)
wrapper.console.format=PM

# Log Level for console output.  (use INFO to see Ghidra Server activity)
wrapper.console.loglevel=INFO

# Provide additional wrapper debug logging info
wrapper.debug=true
wrapper.debug.level=3

# Log file to use for wrapper output logging.
wrapper.logfile=wrapper.log

# Format of output for the log file.  (See docs for formats)
wrapper.logfile.format=LPTM

# Log Level for wrapper.log file output.  (See docs for log levels)
wrapper.logfile.loglevel=INFO

# Maximum size that the log file will be allowed to grow to before
#  the log is rolled. Size is specified in bytes.  The default value
#  of 0, disables log rolling.  May abbreviate with the 'k' (kb) or
#  'm' (mb) suffix.  For example: 10m = 10 megabytes.
wrapper.logfile.maxsize=10m

# Maximum number of rolled log files which will be allowed before old
#  files are deleted.  The default value of 0 implies no limit.
wrapper.logfile.maxfiles=10

#********************************************************************
# Service Wrapper Windows Properties
#********************************************************************
# Title to use when running as a console
wrapper.console.title=Ghidra Server

#********************************************************************
# Service Wrapper Windows NT/2000/XP Service Properties
#********************************************************************
# WARNING - Do not modify any of these properties when an application
#  using this configuration file has been installed as a service.
#  Please uninstall the service before modifying this section.  The
#  service can then be reinstalled.

# Name of the service
wrapper.ntservice.name=ghidraSvr

# Display name of the service
wrapper.ntservice.displayname=Ghidra Server

# Description of the service
wrapper.ntservice.description=Repository server for Ghidra data files.

# Service dependencies.  Add dependencies as needed starting from 1
wrapper.ntservice.dependency.1=

# Mode in which the service is installed.
wrapper.ntservice.starttype=AUTO_START

# Linux service daemon priority for Ghidra Server (start/stop)
# It is important that the network interface has started and any file-system
# dependencies are mounted prior to the Ghidra Server starting.
# NOTE: uninstall the Ghidra Server service using svrUninstall script before changing
# the property wrapper.daemon.update_rc or wrapper.daemon.run_level_dir property.
wrapper.daemon.update_rc= 98 05

# Linux service daemon link directories
wrapper.daemon.run_level_dir=/etc/rcX.d

# Allow the service to interact with the desktop.
wrapper.ntservice.interactive=false

# Restart failed service after 1 minute delay
wrapper.ntservice.failure_actions.actions=RESTART
wrapper.ntservice.failure_actions.actions_delay=60000

[ghidra1]

What Linux version are you using?

A POSIX process spawn is the default for Linux with the fork_hack setting being ignored (recent change to YAJSW). For Ghidra 9.0.1 the fork_hack use will be enabled for OS X only since YAJSW has not filled-out the POSIX support for OS X.

Although I would have thought the default POSIX spawn would have worked, you can try using the fork_hack by disabling the POSIX spawn option which is assumed true:

wrapper.posix_spawn=false
wrapper.fork_hack=true

[Arcnor]

Just tried that, but unfortunately it made no difference, although let me reiterate that the process itself is starting (I can use the server), but the wrapper itself is just closing it after the timeout for some reason.

Thanks for the help in any case, hopefully we'll get to the bottom of this

[ghidra1]

I suspect this is a networking related issue with how YAJSW monitors the spawned process. If you are ambitious, YAJSW is an open source project and I would suspect other uses of it in environments such as yours may raise some discussion on the internet. In addition, the source is available and understanding how the process "ping" is performed may shed some light.

[colinrgodsey]

For me, I had to set the correct RMI hostname in the arguments. Finally dug around a bit through the logs and saw RMI was involved (le sigh) and the horrors of past RMI dealings came rushing back...

wrapper.java.additional.4=-Djava.rmi.server.hostname=ec2-w-x-y-z.us-east-2.compute.amazonaws.com

EDIT: in general, im pretty confident sayings that: the system must have a hostname that is sensible to the client (cant yet figure out if either simple hostname or the FQDN hostname must make sense), and im pretty sure the IP the hostname resolves to (and binds to) must also be sensible to the client. I know RMI does this (it assumes that sending hostname and/or IP over the wire is a sane thing to do), and guessing there might be something else in YAJSW also doing this.

[ryanmkurtz]

Closing this question out due to inactivity. We can reopen it if there is still a need.