Kong: [ERR] Cassandra error: connection refused

Created on 13 Jul 2015  路  41Comments  路  Source: Kong/kong

I retried the whole installation for Kong and it all went well.

But now I get the following:

$ kong start
[INFO] Using configuration: /etc/kong/kong.yml
[INFO] Kong version.......0.3.2
       Proxy HTTP port....8000
       Proxy HTTPS port...8443
       Admin API port.....8001
       dnsmasq port.......8053
       Database...........cassandra keepalive=60000 hosts=localhost port=9042 timeout=1000 keyspace=kong
[INFO] Connecting to the database...
[ERR] Cassandra error: connection refused

I am totally new with kong, and haven't changed the /etc/kong/kong.yml from the defaults.

Can anyone help me with what I am doing wrong?

picture marktopper

Most helpful comment

Okay run:

$ cassandra

Wait for it to start and then:

$ cqlsh 127.0.0.1 9042
picture subnetmarco on 14 Jul 2015
👍4

All 41 comments

Usually this errors happens when Kong cannot communicate with Cassandra. By default the configuration expects Cassandra to be running on 127.0.0.1 on the default 9042 port. Did you also install and start Cassandra on 127.0.0.1?

If you did't install Cassandra, one option is to install it, and another option is to use kongdb.org to provision a testing Cassandra instance (remember to change the connection settings in kong.yml with the right values).

subnetmarco picture subnetmarco on 13 Jul 2015

I have really no idea how to setup or use Cassandra.
But I have used the default configurations.

Then I tried to start Cassandra with the following methods:

forge@kong:~$ sudo service cassandra start
forge@kong:~$ sudo service cassandra restart
 * Restarting Cassandra cassandra                                                                                                                        [ OK ]
forge@kong:~$ sudo service cassandra stop
forge@kong:~$ sudo service cassandra start

But still I get:

$ kong start
[INFO] Using configuration: /etc/kong/kong.yml
[INFO] Kong version.......0.3.2
       Proxy HTTP port....8000
       Proxy HTTPS port...8443
       Admin API port.....8001
       dnsmasq port.......8053
       Database...........cassandra keepalive=60000 hosts=localhost port=9042 timeout=1000 keyspace=kong
[INFO] Connecting to the database...
[ERR] Cassandra error: connection refused

NOTE: Sometimes when restarting Cassandra I get this:

$ sudo service cassandra restart
 * Restarting Cassandra cassandra
start-stop-daemon: warning: failed to kill 21417: No such process

And even when using the Kongdb settings from kongdb.org I get an error:

$ kong start
[INFO] Using configuration: /etc/kong/kong.yml
[INFO] Kong version.......0.3.2
       Proxy HTTP port....8000
       Proxy HTTPS port...8443
       Admin API port.....8001
       dnsmasq port.......8053
       Database...........cassandra keepalive=60000 hosts=ec2-54-172-252-55.compute-1.amazonaws.com port=32824 timeout=2000 keyspace=kong
[INFO] Connecting to the database...
[ERR] Cassandra error: connection refused
marktopper picture marktopper on 14 Jul 2015

I found a stack overflow with the issue, and wondered if that could have anything to do with it.

So I found rpc_address: localhost and rpc_port: 9160 in /etc/cassandra/cassandra.yaml.

Could it maybe be that the server blocks that port for some reason?
I also tried to check if it was actually listening to the port 9042, but seems not to, since I get nothing from $ cat /etc/services | grep 9042.

Also a good knowledge note is that my server has a public and a private IP and a firewall installed.

marktopper picture marktopper on 14 Jul 2015

My bad, that above command is not supposed to return any input.

But $ sudo netstat -anp | grep 9042and $ sudo lsof -i | grep 9042 also returns empty.

marktopper picture marktopper on 14 Jul 2015

@marktopper What OS are you using?

subnetmarco picture subnetmarco on 14 Jul 2015

Linux 14.04

marktopper picture marktopper on 14 Jul 2015

To check if cassandra is running you can execute:

$ cqlsh

Which is the shell client of Cassandra. If Cassandra is running on 127.0.0.1, you should see something like:

Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 2.1.3 | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
cqlsh>
subnetmarco picture subnetmarco on 14 Jul 2015

I followed this to get it installed: https://gist.github.com/marktopper/b9c45b1b7c1f7e03030e

marktopper picture marktopper on 14 Jul 2015

Running cqlsh returns:

Connection error: Could not connect to localhost:9160
marktopper picture marktopper on 14 Jul 2015
👍1

Can you try with:

cqlsh 127.0.0.1 9042

Also, are you using Cassandra version >= 2.1.5 ?

subnetmarco picture subnetmarco on 14 Jul 2015 
$ cqlsh 127.0.0.1 9042
Connection error: Could not connect to 127.0.0.1:9042
marktopper picture marktopper on 14 Jul 2015

Which makes be believe Cassandra didn't start for some reason. Can you try to execute:

ps aux | grep cassandra
subnetmarco picture subnetmarco on 14 Jul 2015

Ohh. Seems to be Cassandra v. Version: 2.0.11.

I will get it updated and get back.

marktopper picture marktopper on 14 Jul 2015

If you still need the other informations.

$ ps aux | grep cassandra
forge    22250  0.0  0.1  11740   940 pts/0    S+   10:59   0:00 grep --color=auto cassandra
marktopper picture marktopper on 14 Jul 2015

Upgraded to Cassandra Version: 2.1.8

marktopper picture marktopper on 14 Jul 2015

Okay run:

$ cassandra

Wait for it to start and then:

$ cqlsh 127.0.0.1 9042
subnetmarco picture subnetmarco on 14 Jul 2015
👍4 
$ cassandra
forge@kong:~$ 05:03:12,627 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/etc/cassandra/logback.xml]
05:03:12,766 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
05:03:12,779 |-INFO in ReconfigureOnChangeFilter{invocationCounter=0} - Will scan for changes in [[/etc/cassandra/logback.xml]] every 60 seconds.
05:03:12,779 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - Adding ReconfigureOnChangeFilter as a turbo filter
05:03:12,797 |-INFO in ch.qos.logback.classic.joran.action.JMXConfiguratorAction - begin
05:03:12,876 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
05:03:12,881 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [FILE]
05:03:12,967 |-INFO in ch.qos.logback.core.rolling.FixedWindowRollingPolicy@18dba48d - Will use zip compression
05:03:13,005 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
05:03:13,096 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Active log file name: /var/log/cassandra/system.log
05:03:13,096 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - File property is set to [/var/log/cassandra/system.log]
05:03:13,106 |-ERROR in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - openFile(/var/log/cassandra/system.log,true) call failed. java.io.FileNotFoundException: /var/log/cassandra/system.log (Permission denied)
    at java.io.FileNotFoundException: /var/log/cassandra/system.log (Permission denied)
    at  at java.io.FileOutputStream.open(Native Method)
    at  at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
    at  at ch.qos.logback.core.recovery.ResilientFileOutputStream.<init>(ResilientFileOutputStream.java:28)
    at  at ch.qos.logback.core.FileAppender.openFile(FileAppender.java:150)
    at  at ch.qos.logback.core.FileAppender.start(FileAppender.java:108)
    at  at ch.qos.logback.core.rolling.RollingFileAppender.start(RollingFileAppender.java:86)
    at  at ch.qos.logback.core.joran.action.AppenderAction.end(AppenderAction.java:96)
    at  at ch.qos.logback.core.joran.spi.Interpreter.callEndAction(Interpreter.java:317)
    at  at ch.qos.logback.core.joran.spi.Interpreter.endElement(Interpreter.java:196)
    at  at ch.qos.logback.core.joran.spi.Interpreter.endElement(Interpreter.java:182)
    at  at ch.qos.logback.core.joran.spi.EventPlayer.play(EventPlayer.java:62)
    at  at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:149)
    at  at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:135)
    at  at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:99)
    at  at ch.qos.logback.core.joran.GenericConfigurator.doConfigure(GenericConfigurator.java:49)
    at  at ch.qos.logback.classic.util.ContextInitializer.configureByResource(ContextInitializer.java:75)
    at  at ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:150)
    at  at org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85)
    at  at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:55)
    at  at org.slf4j.LoggerFactory.bind(LoggerFactory.java:128)
    at  at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:107)
    at  at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:295)
    at  at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:269)
    at  at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:281)
    at  at org.apache.cassandra.service.CassandraDaemon.<clinit>(CassandraDaemon.java:73)
05:03:13,106 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
05:03:13,114 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
05:03:13,126 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
05:03:13,137 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
05:03:13,137 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [FILE] to Logger[ROOT]
05:03:13,138 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT]
05:03:13,139 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [com.thinkaurelius.thrift] to ERROR
05:03:13,139 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
05:03:13,139 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@4e58c0be - Registering current configuration as safe fallback point

INFO  09:03:13 Hostname: kong
INFO  09:03:13 Loading settings from file:/etc/cassandra/cassandra.yaml
INFO  09:03:13 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=Test Cluster; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/var/lib/cassandra/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=localhost; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=localhost; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; saved_caches_directory=/var/lib/cassandra/saved_caches; seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, parameters=[{seeds=127.0.0.1}]}]; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000]
INFO  09:03:13 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO  09:03:13 Global memtable on-heap threshold is enabled at 59MB
INFO  09:03:13 Global memtable off-heap threshold is enabled at 59MB
INFO  09:03:14 Loading settings from file:/etc/cassandra/cassandra.yaml
INFO  09:03:14 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=Test Cluster; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/var/lib/cassandra/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=localhost; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=localhost; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; saved_caches_directory=/var/lib/cassandra/saved_caches; seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, parameters=[{seeds=127.0.0.1}]}]; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000]
INFO  09:03:14 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_80
INFO  09:03:14 Heap size: 250544128/251592704
INFO  09:03:14 Code Cache Non-heap memory: init = 2555904(2496K) used = 712128(695K) committed = 2555904(2496K) max = 50331648(49152K)
INFO  09:03:14 Eden Space Heap memory: init = 51249152(50048K) used = 51249152(50048K) committed = 51249152(50048K) max = 51249152(50048K)
INFO  09:03:14 Survivor Space Heap memory: init = 6356992(6208K) used = 6356992(6208K) committed = 6356992(6208K) max = 6356992(6208K)
INFO  09:03:14 CMS Old Gen Heap memory: init = 192937984(188416K) used = 1043616(1019K) committed = 192937984(188416K) max = 193986560(189440K)
INFO  09:03:14 CMS Perm Gen Non-heap memory: init = 21757952(21248K) used = 15567520(15202K) committed = 21757952(21248K) max = 85983232(83968K)
INFO  09:03:14 Classpath: /etc/cassandra:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.8.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.8.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.9.2.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar::/usr/share/cassandra/lib/jamm-0.3.0.jar
WARN  09:03:14 Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
WARN  09:03:14 JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
ERROR 09:03:14 Directory /var/lib/cassandra/data doesn't exist
ERROR 09:03:14 Has no permission to create /var/lib/cassandra/data directory

$ cqlsh 127.0.0.1 9042
Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})

Noticed the following:

WARN  09:02:31 Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
WARN  09:02:31 JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
ERROR 09:02:31 Directory /var/lib/cassandra/data doesn't exist
ERROR 09:02:31 Has no permission to create /var/lib/cassandra/data directory
marktopper picture marktopper on 14 Jul 2015

And

05:03:13,106 |-ERROR in ch.qos.logback.core.rolling.RollingFileAppender[FILE] - openFile(/var/log/cassandra/system.log,true) call failed. java.io.FileNotFoundException: /var/log/cassandra/system.log (Permission denied)

Should I run it with sudo?

marktopper picture marktopper on 14 Jul 2015

Yes please.

subnetmarco picture subnetmarco on 14 Jul 2015 
forge@kong:~$ sudo cassandra
forge@kong:~$ INFO  09:05:03 Hostname: kong
INFO  09:05:03 Loading settings from file:/etc/cassandra/cassandra.yaml
INFO  09:05:04 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=Test Cluster; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/var/lib/cassandra/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=localhost; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=localhost; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; saved_caches_directory=/var/lib/cassandra/saved_caches; seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, parameters=[{seeds=127.0.0.1}]}]; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000]
INFO  09:05:04 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO  09:05:04 Global memtable on-heap threshold is enabled at 59MB
INFO  09:05:04 Global memtable off-heap threshold is enabled at 59MB
INFO  09:05:04 Loading settings from file:/etc/cassandra/cassandra.yaml
INFO  09:05:04 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=Test Cluster; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/var/lib/cassandra/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=localhost; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=localhost; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; saved_caches_directory=/var/lib/cassandra/saved_caches; seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, parameters=[{seeds=127.0.0.1}]}]; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000]
INFO  09:05:04 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_80
INFO  09:05:04 Heap size: 250544128/251592704
INFO  09:05:04 Code Cache Non-heap memory: init = 2555904(2496K) used = 710080(693K) committed = 2555904(2496K) max = 50331648(49152K)
INFO  09:05:04 Eden Space Heap memory: init = 51249152(50048K) used = 51249152(50048K) committed = 51249152(50048K) max = 51249152(50048K)
INFO  09:05:04 Survivor Space Heap memory: init = 6356992(6208K) used = 6356992(6208K) committed = 6356992(6208K) max = 6356992(6208K)
INFO  09:05:04 CMS Old Gen Heap memory: init = 192937984(188416K) used = 1050144(1025K) committed = 192937984(188416K) max = 193986560(189440K)
INFO  09:05:04 CMS Perm Gen Non-heap memory: init = 21757952(21248K) used = 15571264(15206K) committed = 21757952(21248K) max = 85983232(83968K)
INFO  09:05:04 Classpath: /etc/cassandra:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.8.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.8.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.9.2.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar::/usr/share/cassandra/lib/jamm-0.3.0.jar

forge@kong:~$ cqlsh 127.0.0.1 9042
Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})
forge@kong:~$ sudo cqlsh 127.0.0.1 9042
Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})
marktopper picture marktopper on 14 Jul 2015

It may take a while for Cassandra to start. Can you run sudo cassandra, wait 30 seconds, and then run cqlsh 127.0.0.1 9042 ?

subnetmarco picture subnetmarco on 14 Jul 2015
👍1

I will give it a try and wait for a while.

Thanks so far for your time, will keep you updated.

marktopper picture marktopper on 14 Jul 2015

Tried running sudo cassandra and waited 10 minutes before running cqlsh 127.0.0.1 9042.

But still without luck.

You got any other ideas?

forge@kong:~$ sudo cassandra
forge@kong:~$ INFO  09:08:50 Hostname: kong
INFO  09:08:50 Loading settings from file:/etc/cassandra/cassandra.yaml
INFO  09:08:50 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=Test Cluster; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/var/lib/cassandra/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=localhost; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=localhost; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; saved_caches_directory=/var/lib/cassandra/saved_caches; seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, parameters=[{seeds=127.0.0.1}]}]; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000]
INFO  09:08:51 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
INFO  09:08:51 Global memtable on-heap threshold is enabled at 59MB
INFO  09:08:51 Global memtable off-heap threshold is enabled at 59MB
INFO  09:08:51 Loading settings from file:/etc/cassandra/cassandra.yaml
INFO  09:08:51 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_snapshot=true; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=Test Cluster; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/var/lib/cassandra/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=32; concurrent_reads=32; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; cross_node_timeout=false; data_file_directories=[/var/lib/cassandra/data]; disk_failure_policy=stop; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; endpoint_snitch=SimpleSnitch; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=localhost; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=256; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=localhost; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; saved_caches_directory=/var/lib/cassandra/saved_caches; seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, parameters=[{seeds=127.0.0.1}]}]; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; write_request_timeout_in_ms=2000]
INFO  09:08:51 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_80
INFO  09:08:51 Heap size: 250544128/251592704
INFO  09:08:51 Code Cache Non-heap memory: init = 2555904(2496K) used = 724096(707K) committed = 2555904(2496K) max = 50331648(49152K)
INFO  09:08:51 Eden Space Heap memory: init = 51249152(50048K) used = 51249152(50048K) committed = 51249152(50048K) max = 51249152(50048K)
INFO  09:08:51 Survivor Space Heap memory: init = 6356992(6208K) used = 6356984(6207K) committed = 6356992(6208K) max = 6356992(6208K)
INFO  09:08:51 CMS Old Gen Heap memory: init = 192937984(188416K) used = 1047632(1023K) committed = 192937984(188416K) max = 193986560(189440K)
INFO  09:08:51 CMS Perm Gen Non-heap memory: init = 21757952(21248K) used = 15557416(15192K) committed = 21757952(21248K) max = 85983232(83968K)
INFO  09:08:51 Classpath: /etc/cassandra:/usr/share/cassandra/lib/ST4-4.0.8.jar:/usr/share/cassandra/lib/airline-0.6.jar:/usr/share/cassandra/lib/antlr-runtime-3.5.2.jar:/usr/share/cassandra/lib/commons-cli-1.1.jar:/usr/share/cassandra/lib/commons-codec-1.2.jar:/usr/share/cassandra/lib/commons-lang3-3.1.jar:/usr/share/cassandra/lib/commons-math3-3.2.jar:/usr/share/cassandra/lib/compress-lzf-0.8.4.jar:/usr/share/cassandra/lib/concurrentlinkedhashmap-lru-1.4.jar:/usr/share/cassandra/lib/disruptor-3.0.1.jar:/usr/share/cassandra/lib/guava-16.0.jar:/usr/share/cassandra/lib/high-scale-lib-1.0.6.jar:/usr/share/cassandra/lib/jackson-core-asl-1.9.2.jar:/usr/share/cassandra/lib/jackson-mapper-asl-1.9.2.jar:/usr/share/cassandra/lib/jamm-0.3.0.jar:/usr/share/cassandra/lib/javax.inject.jar:/usr/share/cassandra/lib/jbcrypt-0.3m.jar:/usr/share/cassandra/lib/jline-1.0.jar:/usr/share/cassandra/lib/jna-4.0.0.jar:/usr/share/cassandra/lib/json-simple-1.1.jar:/usr/share/cassandra/lib/libthrift-0.9.2.jar:/usr/share/cassandra/lib/logback-classic-1.1.2.jar:/usr/share/cassandra/lib/logback-core-1.1.2.jar:/usr/share/cassandra/lib/lz4-1.2.0.jar:/usr/share/cassandra/lib/metrics-core-2.2.0.jar:/usr/share/cassandra/lib/netty-all-4.0.23.Final.jar:/usr/share/cassandra/lib/reporter-config-2.1.0.jar:/usr/share/cassandra/lib/slf4j-api-1.7.2.jar:/usr/share/cassandra/lib/snakeyaml-1.11.jar:/usr/share/cassandra/lib/snappy-java-1.0.5.2.jar:/usr/share/cassandra/lib/stream-2.5.2.jar:/usr/share/cassandra/lib/super-csv-2.1.0.jar:/usr/share/cassandra/lib/thrift-server-0.3.7.jar:/usr/share/cassandra/apache-cassandra-2.1.8.jar:/usr/share/cassandra/apache-cassandra-thrift-2.1.8.jar:/usr/share/cassandra/apache-cassandra.jar:/usr/share/cassandra/cassandra-driver-core-2.0.9.2.jar:/usr/share/cassandra/netty-3.9.0.Final.jar:/usr/share/cassandra/stress.jar::/usr/share/cassandra/lib/jamm-0.3.0.jar

forge@kong:~$ cqlsh 127.0.0.1 9042
Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})
marktopper picture marktopper on 14 Jul 2015

Can you execute java -version?

subnetmarco picture subnetmarco on 14 Jul 2015 
$ java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)
marktopper picture marktopper on 14 Jul 2015

Okay so this is very weird. Let's try one more thing. In your Kong configuration can you use the following Cassandra properties:

cassandra:
  properties:
    hosts: ec2-52-7-33-160.compute-1.amazonaws.com
    port: 32809
    timeout: 2000
    keyspace: kong
    keepalive: 60000

For testing, also run: 

$ cqlsh ec2-52-7-33-160.compute-1.amazonaws.com 32809
subnetmarco picture subnetmarco on 14 Jul 2015

Then I get

$ kong start
[INFO] Using configuration: /etc/kong/kong.yml
[INFO] Kong version.......0.3.2
       Proxy HTTP port....8000
       Proxy HTTPS port...8443
       Admin API port.....8001
       dnsmasq port.......8053
       Database...........cassandra keepalive=60000 hosts=ec2-52-7-33-160.compute-1.amazonaws.com port=32809 timeout=2000 keyspace=kong
[INFO] Connecting to the database...
[ERR] Permission denied /usr/local/kong/
marktopper picture marktopper on 14 Jul 2015

Wait, with sudo kong start I get

[INFO] Using configuration: /etc/kong/kong.yml
[INFO] Kong version.......0.3.2
       Proxy HTTP port....8000
       Proxy HTTPS port...8443
       Admin API port.....8001
       dnsmasq port.......8053
       Database...........cassandra keepalive=60000 hosts=ec2-52-7-33-160.compute-1.amazonaws.com port=32809 timeout=2000 keyspace=kong
[INFO] Connecting to the database...
[INFO] dnsmasq started
[WARN] ulimit is currently set to "1024". For better performance set it to at least "4096" using "ulimit -n"
[OK] Started
marktopper picture marktopper on 14 Jul 2015

So, it might be a error related to my Cassandra and not Kong.

marktopper picture marktopper on 14 Jul 2015

More tests:

forge@kong:~$ cqlsh ec2-52-7-33-160.compute-1.amazonaws.com 32809
Connected to Test Cluster at ec2-52-7-33-160.compute-1.amazonaws.com:32809.
[cqlsh 5.0.1 | Cassandra 2.1.4 | CQL spec 3.2.0 | Native protocol v3]
Use HELP for help.
cqlsh> exit;
forge@kong:~$ cqlsh 127.0.0.1 9042
Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})
marktopper picture marktopper on 14 Jul 2015

The local Cassandra installation seems to be broken somehow. It's 3am here, I will follow up tomorrow on this thread.

subnetmarco picture subnetmarco on 14 Jul 2015

I will try to build another server and look for a guide for installation for Linux 14.04.

Sleep well and thanks for your time.

marktopper picture marktopper on 14 Jul 2015

@marktopper any news?

subnetmarco picture subnetmarco on 14 Jul 2015

@marktopper I wrote a guide for installing Kong on a base Ubuntu 14.04 image awhile back, it includes a section for getting Cassandra running locally. Here it is: 

# Install Cassandra 
wget http://apache.mirrors.tds.net/cassandra/2.1.5/apache-cassandra-2.1.5-bin.tar.gz
tar xvzf apache-cassandra-2.1.5-bin.tar.gz
./apache-cassandra-2.1.5/bin/cassandra

You could also put the bin directory in your path or make use of tools like init or upstart to start, stop and restart Cassandra as well as keep it running in case of reboots or crashes.

montanaflynn picture montanaflynn on 15 Jul 2015

I will close this issue because it's specific to Cassandra and not Kong. But we can keep commenting on it to fix the problem.

subnetmarco picture subnetmarco on 16 Jul 2015

I managed to get it to work.
It seemed like Cassandra was broken so I installed it this way:
https://gist.github.com/marktopper/57af0232db757971e5f7

marktopper picture marktopper on 16 Jul 2015

Thanks for writing that Gist file, I will take it and put it on the getkong.org website just in case other people wonder how to install Cassandra.

subnetmarco picture subnetmarco on 16 Jul 2015

:+1: Thanks to you as well.

marktopper picture marktopper on 16 Jul 2015

Hi thefosk

muppudathis-tzl picture muppudathis-tzl on 16 Feb 2016

How to configure my own Cassandra in kong.yml file

muppudathis-tzl picture muppudathis-tzl on 16 Feb 2016

@muppudathis-tzl https://getkong.org/docs/0.6.x/configuration/#cassandra

subnetmarco picture subnetmarco on 16 Feb 2016
Was this page helpful?
0 / 5 - 0 ratings

Related issues

jeremyjpj0916 picture jeremyjpj0916  路  59Comments
jeremyjpj0916 picture jeremyjpj0916  路  34Comments
noamelf picture noamelf  路  36Comments
jason-riddle picture jason-riddle  路  40Comments
subnetmarco picture subnetmarco  路  97Comments