Redisgraph: Crash report

Created on 10 Apr 2021  路  5Comments  路  Source: RedisGraph/RedisGraph

Redis crashed while I was running a couple RedisGraph queries concurrently. One had been running about 2 hours and the other about 25 mins.

This is on an Ubuntu 20.04 server, running Redis 6.2.1 and redisgraph.so built on git SHA 92046602a775dcba2396d493597b60043f0e39bb. If you want the RDB file that has the graph data let me know; it's about 10MB in size.

=== REDIS BUG REPORT START: Cut & paste starting from here ===
32319:M 10 Apr 2021 21:22:12.057 # Redis 6.2.1 crashed by signal: 11, si_code: 1
32319:M 10 Apr 2021 21:22:12.057 # Accessing address: 0xffff8
32319:M 10 Apr 2021 21:22:12.057 # Killed by PID: 1048568, UID: 0
32319:M 10 Apr 2021 21:22:12.057 # Crashed running the instruction at: 0x7f116f1d727b

------ STACK TRACE ------
EIP:
/etc/redis/modules/redisgraph.so(+0x1f327b)[0x7f116f1d727b]

Backtrace:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f117144e3c0]
/etc/redis/modules/redisgraph.so(+0x1f327b)[0x7f116f1d727b]
/etc/redis/modules/redisgraph.so(DataBlock_AllocateItem+0xd2)[0x7f116f1d7582]
/etc/redis/modules/redisgraph.so(_ResultSet_ConsumeRecord+0x37)[0x7f116f1cb657]
/etc/redis/modules/redisgraph.so(ResultSet_AddRecord+0x3e)[0x7f116f1cb6fe]
/etc/redis/modules/redisgraph.so(+0x1c693b)[0x7f116f1aa93b]
/etc/redis/modules/redisgraph.so(ExecutionPlan_Execute+0x50)[0x7f116f19f670]
/etc/redis/modules/redisgraph.so(+0x1b607d)[0x7f116f19a07d]
/etc/redis/modules/redisgraph.so(+0x1f3daf)[0x7f116f1d7daf]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609)[0x7f1171442609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f1171367293]

------ REGISTERS ------
32319:M 10 Apr 2021 21:22:12.059 # 
RAX:0000000000000000 RBX:00007f0ac020b0c0
RCX:00007f0e300000c0 RDX:0000000000000000
RDI:00007f1149e0b020 RSI:0000000000000004
RBP:0000000000020000 RSP:00007f116c7dc380
R8 :0000000000000004 R9 :00007f1170e076a0
R10:0000000000000000 R11:000000000000001c
R12:00007f0ac020b0c0 R13:00000000000ffff8
R14:0000000000000004 R15:00007f1170c36140
RIP:00007f116f1d727b EFL:0000000000010202
CSGSFS:002b000000000033
32319:M 10 Apr 2021 21:22:12.059 # (00007f116c7dc38f) -> 00007f116f1cb6fe
32319:M 10 Apr 2021 21:22:12.060 # (00007f116c7dc38e) -> 00007f0ac0215020
32319:M 10 Apr 2021 21:22:12.060 # (00007f116c7dc38d) -> 00007f0ac020e070
32319:M 10 Apr 2021 21:22:12.060 # (00007f116c7dc38c) -> 00007f0ac02a4ab0
32319:M 10 Apr 2021 21:22:12.060 # (00007f116c7dc38b) -> 00007f0ac020e070
32319:M 10 Apr 2021 21:22:12.060 # (00007f116c7dc38a) -> 00007f0ac022c5a0
32319:M 10 Apr 2021 21:22:12.060 # (00007f116c7dc389) -> 00007f116f1cb657
32319:M 10 Apr 2021 21:22:12.060 # (00007f116c7dc388) -> 00007f0ac020e070
32319:M 10 Apr 2021 21:22:12.060 # (00007f116c7dc387) -> 00007f0ac020e070
32319:M 10 Apr 2021 21:22:12.060 # (00007f116c7dc386) -> 00007f0ac02a4ab0
32319:M 10 Apr 2021 21:22:12.060 # (00007f116c7dc385) -> 00007f116f1d7582
32319:M 10 Apr 2021 21:22:12.060 # (00007f116c7dc384) -> 0000000000000004
32319:M 10 Apr 2021 21:22:12.060 # (00007f116c7dc383) -> 00007f0ac02a4ab0
32319:M 10 Apr 2021 21:22:12.060 # (00007f116c7dc382) -> 0000000000000001
32319:M 10 Apr 2021 21:22:12.060 # (00007f116c7dc381) -> 0000000000000000
32319:M 10 Apr 2021 21:22:12.060 # (00007f116c7dc380) -> 00007f0ac020b0c0

------ INFO OUTPUT ------
# Server
redis_version:6.2.1
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:81a84d60f8ac9371
redis_mode:standalone
os:Linux 5.4.0-1038-aws x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:9.3.0
process_id:32319
process_supervised:systemd
run_id:b5f30d561292d13652ec03597f0ab4a0a3138e41
tcp_port:6379
server_time_usec:1618089731977086
uptime_in_seconds:7250
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:7476995
executable:/usr/bin/redis-server
config_file:/etc/redis/redis.conf
io_threads_active:0

# Clients
connected_clients:2
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:324
client_recent_max_output_buffer:0
blocked_clients:2
tracking_clients:0
clients_in_timeout_table:0

# Memory
used_memory:64480163224
used_memory_human:60.05G
used_memory_rss:110383509504
used_memory_rss_human:102.80G
used_memory_peak:64482812616
used_memory_peak_human:60.05G
used_memory_peak_perc:100.00%
used_memory_overhead:888696
used_memory_startup:842872
used_memory_dataset:64479274528
used_memory_dataset_perc:100.00%
allocator_allocated:64482686440
allocator_active:65290051584
allocator_resident:65530507264
total_system_memory:535151763456
total_system_memory_human:498.40G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.01
allocator_frag_bytes:807365144
allocator_rss_ratio:1.00
allocator_rss_bytes:240455680
rss_overhead_ratio:1.68
rss_overhead_bytes:44853002240
mem_fragmentation_ratio:1.71
mem_fragmentation_bytes:45902263152
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:41528
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
current_cow_size:0
current_fork_perc:0.00%
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1618082481
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:4
total_commands_processed:4
instantaneous_ops_per_sec:0
total_net_input_bytes:1023
total_net_output_bytes:2623008996
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:19
evicted_keys:0
keyspace_hits:4
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:6
total_writes_processed:32046
io_threaded_reads_processed:0
io_threaded_writes_processed:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:97eda557fcb14d92d237891bc8e9696e0a03cb17
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:411.522780
used_cpu_user:8406.164504
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:157.008063
used_cpu_user_main_thread:1304.781339

# Modules
module:name=graph,ver=999999,api=1,filters=0,usedby=[],using=[],options=[]

# Commandstats
cmdstat_graph.QUERY:calls=4,usec=243,usec_per_call=60.75,rejected_calls=0,failed_calls=0

# Errorstats

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=81,expires=0,avg_ttl=0

------ CLIENT LIST OUTPUT ------
id=6 addr=127.0.0.1:34538 laddr=127.0.0.1:6379 fd=8 name= age=7184 idle=7184 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=212 obl=0 oll=0 omem=0 tot-mem=20724 events=r cmd=graph.QUERY user=default redir=-1
id=15 addr=127.0.0.1:34576 laddr=127.0.0.1:6379 fd=9 name= age=1471 idle=1471 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=292 obl=0 oll=0 omem=0 tot-mem=20804 events=r cmd=graph.QUERY user=default redir=-1

------ MODULES INFO OUTPUT ------
# graph_executing commands
graph_command:GRAPH.QUERY MATCH (c:Committee)-[:contributesTo*0..2]-(c2:Committee)-[:contributesTo]->(r:Committee) WHERE r.party = 'D' AND c2.party IS NULL AND c.party IS NULL AND c <> c2 RETURN DISTINCT c.id, c2.id, r.id
graph_command:GRAPH.QUERY MATCH (c:Committee)-[:contributesTo]-(c2:Committee)-[:contributesTo]-(c3:Committee)-[:contributesTo]->(r:Committee) WHERE r.party = 'D' AND c.party IS NULL AND c2.party IS NULL AND c3.party IS NULL AND c <> c2 AND c <> c3 AND c2 <> c3 RETURN DISTINCT c.id, c2.id, c3.id, r.id

------ FAST MEMORY TEST ------
32319:M 10 Apr 2021 21:22:12.061 # main thread terminated
32319:M 10 Apr 2021 21:22:12.061 # Bio thread for job type #0 terminated
32319:M 10 Apr 2021 21:22:12.061 # Bio thread for job type #1 terminated
32319:M 10 Apr 2021 21:22:12.061 # Bio thread for job type #2 terminated

Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: (null) (base: (nil))
Module: /etc/redis/modules/redisgraph.so (base 0x7f116efe4000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=(nil) -D -b binary -m i386:x86-64 /tmp/dump.bin
------

=== REDIS BUG REPORT END. Make sure to include from START to END. ===
picture abevoelker

Most helpful comment

@abevoelker, I've looked into both the data and the query
I believe the crash is caused due to the system running out of memory.

I've created an index: GRAPH.QUERY SchedB "create index on :Committee(party)"
and modified the query a bit to take advantage of the index: 

"MATCH (r:Committee {party:'D'}) WITH r MATCH (c:Committee)-[:contributesTo]-(c2:Committee)-[:contributesTo]-(c3:Committee)-[:contributesTo]->(r) WHERE c.party IS NULL AND c2.party IS NULL AND c3.party IS NULL AND c <> c2 AND c <> c3 AND c2 <> c3 RETURN DISTINCT c.id, c2.id, c3.id, r.id"

Now by removing one hop: (c:Committee)-[:contributesTo]-(c2:Committee)-[:contributesTo]->(r)
I'm getting ~27M results, which probably means the original query (3 hops) will response with hundreds of millions of rows

the reason for the high memory consumption is due to the fact that RedisGraph is unable to send responses as they're being constructed, it has to accumulate the entire response in-memory and only once the entire result-set is computed it is being transmitted, if I were to change the projection a bit to say RETURN count(c) we see almost no change in the overall memory consumption.

I would suggest (if possible) to use both SKIP and LIMIT to consume this query in batches, in order to keep memory consumption low.

Lastly is the bidirectional traversal: (c:Committee)-[:contributesTo]-(c2:Committee)-[:contributesTo]-(c3:Committee)
on purpose? otherwise I would suggest specifying directions, e.g.: (c:Committee)-[:contributesTo]->(c2:Committee)-[:contributesTo]->(c3:Committee)

picture swilly22 on 13 Apr 2021
🎉2

All 5 comments

Crashed again, this time it was just with the one query running for about 24 mins (the same one that was running for 25 mins last crash):

=== REDIS BUG REPORT START: Cut & paste starting from here ===
32841:M 10 Apr 2021 22:01:01.960 # Redis 6.2.1 crashed by signal: 11, si_code: 1
32841:M 10 Apr 2021 22:01:01.960 # Accessing address: 0xffff8
32841:M 10 Apr 2021 22:01:01.960 # Killed by PID: 1048568, UID: 0
32841:M 10 Apr 2021 22:01:01.960 # Crashed running the instruction at: 0x7f672d5d727b

------ STACK TRACE ------
EIP:
/etc/redis/modules/redisgraph.so(+0x1f327b)[0x7f672d5d727b]

Backtrace:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f672f8f13c0]
/etc/redis/modules/redisgraph.so(+0x1f327b)[0x7f672d5d727b]
/etc/redis/modules/redisgraph.so(DataBlock_AllocateItem+0xd2)[0x7f672d5d7582]
/etc/redis/modules/redisgraph.so(_ResultSet_ConsumeRecord+0x37)[0x7f672d5cb657]
/etc/redis/modules/redisgraph.so(ResultSet_AddRecord+0x3e)[0x7f672d5cb6fe]
/etc/redis/modules/redisgraph.so(+0x1c693b)[0x7f672d5aa93b]
/etc/redis/modules/redisgraph.so(ExecutionPlan_Execute+0x50)[0x7f672d59f670]
/etc/redis/modules/redisgraph.so(+0x1b607d)[0x7f672d59a07d]
/etc/redis/modules/redisgraph.so(+0x1f3daf)[0x7f672d5d7daf]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609)[0x7f672f8e5609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f672f80a293]

------ REGISTERS ------
32841:M 10 Apr 2021 22:01:01.962 # 
RAX:0000000000000000 RBX:00007f66fd80b0c0
RCX:00007f66fda000c0 RDX:0000000000000000
RDI:00007f67081ad008 RSI:0000000000000001
RBP:0000000000020000 RSP:00007f672c3df380
R8 :0000000000000001 R9 :00007f672f207448
R10:0000000000000000 R11:000000000000001c
R12:00007f66fd80b0c0 R13:00000000000ffff8
R14:0000000000000004 R15:00007f672f036140
RIP:00007f672d5d727b EFL:0000000000010202
CSGSFS:002b000000000033
32841:M 10 Apr 2021 22:01:01.962 # (00007f672c3df38f) -> 00007f672d5cb6fe
32841:M 10 Apr 2021 22:01:01.962 # (00007f672c3df38e) -> 00007f66fd815020
32841:M 10 Apr 2021 22:01:01.962 # (00007f672c3df38d) -> 00007f66fd80e070
32841:M 10 Apr 2021 22:01:01.962 # (00007f672c3df38c) -> 00007f66fd8a4bb0
32841:M 10 Apr 2021 22:01:01.962 # (00007f672c3df38b) -> 00007f66fd80e070
32841:M 10 Apr 2021 22:01:01.962 # (00007f672c3df38a) -> 00007f66fd82c5a0
32841:M 10 Apr 2021 22:01:01.962 # (00007f672c3df389) -> 00007f672d5cb657
32841:M 10 Apr 2021 22:01:01.962 # (00007f672c3df388) -> 00007f66fd80e070
32841:M 10 Apr 2021 22:01:01.962 # (00007f672c3df387) -> 00007f66fd80e070
32841:M 10 Apr 2021 22:01:01.962 # (00007f672c3df386) -> 00007f66fd8a4bb0
32841:M 10 Apr 2021 22:01:01.962 # (00007f672c3df385) -> 00007f672d5d7582
32841:M 10 Apr 2021 22:01:01.962 # (00007f672c3df384) -> 0000000000000004
32841:M 10 Apr 2021 22:01:01.962 # (00007f672c3df383) -> 00007f66fd8a4bb0
32841:M 10 Apr 2021 22:01:01.962 # (00007f672c3df382) -> 0000000000000001
32841:M 10 Apr 2021 22:01:01.962 # (00007f672c3df381) -> 0000000000000000
32841:M 10 Apr 2021 22:01:01.962 # (00007f672c3df380) -> 00007f66fd80b0c0

------ INFO OUTPUT ------
# Server
redis_version:6.2.1
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:81a84d60f8ac9371
redis_mode:standalone
os:Linux 5.4.0-1038-aws x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:9.3.0
process_id:32841
process_supervised:systemd
run_id:8626a7b40ff22cbb4cd331f6d94dc07ec2693adb
tcp_port:6379
server_time_usec:1618092061949433
uptime_in_seconds:2325
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:7479325
executable:/usr/bin/redis-server
config_file:/etc/redis/redis.conf
io_threads_active:0

# Clients
connected_clients:1
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:324
client_recent_max_output_buffer:0
blocked_clients:1
tracking_clients:0
clients_in_timeout_table:0

# Memory
used_memory:43008420160
used_memory_human:40.05G
used_memory_rss:78958608384
used_memory_rss_human:73.54G
used_memory_peak:43009737928
used_memory_peak_human:40.06G
used_memory_peak_perc:100.00%
used_memory_overhead:863780
used_memory_startup:842872
used_memory_dataset:43007556380
used_memory_dataset_perc:100.00%
allocator_allocated:43010363640
allocator_active:43548344320
allocator_resident:43672436736
total_system_memory:535151763456
total_system_memory_human:498.40G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.01
allocator_frag_bytes:537980680
allocator_rss_ratio:1.00
allocator_rss_bytes:124092416
rss_overhead_ratio:1.81
rss_overhead_bytes:35286171648
mem_fragmentation_ratio:1.84
mem_fragmentation_bytes:35949123008
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:20804
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence
loading:0
current_cow_size:0
current_fork_perc:0.00%
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:80
rdb_bgsave_in_progress:0
rdb_last_save_time:1618089736
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0

# Stats
total_connections_received:8
total_commands_processed:9
instantaneous_ops_per_sec:0
total_net_input_bytes:2851
total_net_output_bytes:34789
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:5
evicted_keys:0
keyspace_hits:1
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:16
total_writes_processed:8
io_threaded_reads_processed:0
io_threaded_writes_processed:0

# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:a57ecfebb32ad94ebd8b1f378302b2c6db5c9a33
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:144.562218
used_cpu_user:1296.627570
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:143.093987
used_cpu_user_main_thread:1295.005730

# Modules
module:name=graph,ver=999999,api=1,filters=0,usedby=[],using=[],options=[]

# Commandstats
cmdstat_del:calls=1,usec=38,usec_per_call=38.00,rejected_calls=0,failed_calls=0
cmdstat_info:calls=1,usec=85,usec_per_call=85.00,rejected_calls=0,failed_calls=0
cmdstat_keys:calls=5,usec=113,usec_per_call=22.60,rejected_calls=0,failed_calls=0
cmdstat_graph.QUERY:calls=1,usec=30,usec_per_call=30.00,rejected_calls=0,failed_calls=0
cmdstat_command:calls=1,usec=519,usec_per_call=519.00,rejected_calls=0,failed_calls=0

# Errorstats

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=1,expires=0,avg_ttl=0

------ CLIENT LIST OUTPUT ------
id=6 addr=127.0.0.1:34586 laddr=127.0.0.1:6379 fd=8 name= age=1438 idle=1438 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=292 obl=0 oll=0 omem=0 tot-mem=20804 events=r cmd=graph.QUERY user=default redir=-1

------ MODULES INFO OUTPUT ------
# graph_executing commands
graph_command:GRAPH.QUERY MATCH (c:Committee)-[:contributesTo]-(c2:Committee)-[:contributesTo]-(c3:Committee)-[:contributesTo]->(r:Committee) WHERE r.party = 'D' AND c.party IS NULL AND c2.party IS NULL AND c3.party IS NULL AND c <> c2 AND c <> c3 AND c2 <> c3 RETURN DISTINCT c.id, c2.id, c3.id, r.id

------ FAST MEMORY TEST ------
32841:M 10 Apr 2021 22:01:01.963 # main thread terminated
32841:M 10 Apr 2021 22:01:01.963 # Bio thread for job type #0 terminated
32841:M 10 Apr 2021 22:01:01.963 # Bio thread for job type #1 terminated
32841:M 10 Apr 2021 22:01:01.963 # Bio thread for job type #2 terminated

Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: (null) (base: (nil))
Module: /etc/redis/modules/redisgraph.so (base 0x7f672d3e4000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=(nil) -D -b binary -m i386:x86-64 /tmp/dump.bin
------

=== REDIS BUG REPORT END. Make sure to include from START to END. ===
abevoelker picture abevoelker on 11 Apr 2021

Hi @abevoelker,
Can you please provide both the RDB and queries you're running?
Also can you please specify how much memory the machine you're running on have?
Thank you!

swilly22 picture swilly22 on 11 Apr 2021

RDB: https://www.dropbox.com/s/podx7zhb6q7byb4/crash-db.rdb?dl=0

Query:

GRAPH.QUERY SchedB "MATCH (c:Committee)-[:contributesTo]-(c2:Committee)-[:contributesTo]-(c3:Committee)-[:contributesTo]->(r:Committee) WHERE r.party = 'D' AND c.party IS NULL AND c2.party IS NULL AND c3.party IS NULL AND c <> c2 AND c <> c3 AND c2 <> c3 RETURN DISTINCT c.id, c2.id, c3.id, r.id"

The machine had 512GB of RAM (htop reported 498GB). It was an r5dn.16xlarge on AWS EC2 to be precise. Spot instances were $1.14/hr yesterday when I was testing

abevoelker picture abevoelker on 11 Apr 2021
👍1

@abevoelker, I've looked into both the data and the query
I believe the crash is caused due to the system running out of memory.

I've created an index: GRAPH.QUERY SchedB "create index on :Committee(party)"
and modified the query a bit to take advantage of the index: 

"MATCH (r:Committee {party:'D'}) WITH r MATCH (c:Committee)-[:contributesTo]-(c2:Committee)-[:contributesTo]-(c3:Committee)-[:contributesTo]->(r) WHERE c.party IS NULL AND c2.party IS NULL AND c3.party IS NULL AND c <> c2 AND c <> c3 AND c2 <> c3 RETURN DISTINCT c.id, c2.id, c3.id, r.id"

Now by removing one hop: (c:Committee)-[:contributesTo]-(c2:Committee)-[:contributesTo]->(r)
I'm getting ~27M results, which probably means the original query (3 hops) will response with hundreds of millions of rows

the reason for the high memory consumption is due to the fact that RedisGraph is unable to send responses as they're being constructed, it has to accumulate the entire response in-memory and only once the entire result-set is computed it is being transmitted, if I were to change the projection a bit to say RETURN count(c) we see almost no change in the overall memory consumption.

I would suggest (if possible) to use both SKIP and LIMIT to consume this query in batches, in order to keep memory consumption low.

Lastly is the bidirectional traversal: (c:Committee)-[:contributesTo]-(c2:Committee)-[:contributesTo]-(c3:Committee)
on purpose? otherwise I would suggest specifying directions, e.g.: (c:Committee)-[:contributesTo]->(c2:Committee)-[:contributesTo]->(c3:Committee)

swilly22 picture swilly22 on 13 Apr 2021
🎉2

Thanks, this was very helpful to me! I did not intend to do the bidirectional traversal. I was jumping around between Cypher tutorials and I must've missed that distinction, but glad to know it now. Thanks for your help! Closing this now.

abevoelker picture abevoelker on 30 Apr 2021
Was this page helpful?
0 / 5 - 0 ratings

Related issues

mdazartash picture mdazartash  路  3Comments
esokullu picture esokullu  路  3Comments
dandago picture dandago  路  3Comments
sidenAlex picture sidenAlex  路  3Comments
WitchsCat picture WitchsCat  路  10Comments