StackExchange.Redis.RedisTimeoutException: Timeout performing HSCAN (10000ms) ubunutu

[KamranShahid]

I have an .net core (3.1) application which is running fine in windows. Now i am trying to run it in ubunutu 18
My application is running fine and even inserting entries in redis. But during get i am getting above error.
StackExchange.Redis version i am using is 2.0.601

RedisManager.zip

[KamranShahid]

Anybody looking into this repository?

[Rodrigo-Andrade]

If you relax the timeout values, does the problem go away?

[mgravell]

@KamranShahid yes, but speaking for myself: support here comes from my own personal time. This isn't something I do as my day job, which means it needs to wait on my availability.

[KamranShahid]

If you relax the timeout values, does the problem go away?

I think 10 seconds is more then enough. Strange thing is write operation have no problem.

[tombatron]

How big is the hash that you're scanning?

Does this time out occur every time?

If this timeout doesn't occur every time, how long does the operation take when it succeeds?

[KamranShahid]

It's a small hash. I have a windows 10 machine with Redis 3.0 of windows install. I were not able to connect with it. what eventually i have done is install local redis on that ubuntu and work.

[KamranShahid]

Any update on this?
Just got another error logs

Timeout performing ZADD (10000ms), next: HSCAN InProccessingMsisdnTransaction_, inst: 35, qu: 0, qs: 178, aw: False, rs: ReadAsync, ws: Idle, in: 1690, in-pipe: 0, out-pipe: 0, serverEndpoint: 10.10.10.10:6379, mgr: 10 of 10 available, clientName: SafeRedisConnection, IOCP: (Busy=0,Free=1000,Min=10,Max=1000), WORKER: (Busy=59,Free=32708,Min=2,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

Timeout performing ZADD (10000ms), next: HSCAN InProccessingMsisdnTransaction_, inst: 35, qu: 0, qs: 178, aw: False, rs: ReadAsync, ws: Idle, in: 1690, in-pipe: 0, out-pipe: 0, serverEndpoint: 10.10.10.10:6379, mgr: 10 of 10 available, clientName: SafeRedisConnection, IOCP: (Busy=0,Free=1000,Min=10,Max=1000), WORKER: (Busy=59,Free=32708,Min=2,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

[KamranShahid]

Redis details is as following

"# Server
redis_version:4.0.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:606540f254e6adde
redis_mode:standalone
os:Linux 4.9.85-37.55.amzn1.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:4.8.5
process_id:2532
run_id:9c2c6338d329d60803f6bf999ae8913f05591962
tcp_port:6379
uptime_in_seconds:19857
uptime_in_days:0
hz:10
lru_clock:15777407
executable:/usr/local/bin/redis-server
config_file:/etc/redis/redis.conf

# Clients
connected_clients:10
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:2369616
used_memory_human:2.26M
used_memory_rss:6017024
used_memory_rss_human:5.74M
used_memory_peak:2728528
used_memory_peak_human:2.60M
used_memory_peak_perc:86.85%
used_memory_overhead:1033176
used_memory_startup:765624
used_memory_dataset:1336440
used_memory_dataset_perc:83.32%
total_system_memory:2098429952
total_system_memory_human:1.95G
used_memory_lua:37888
used_memory_lua_human:37.00K
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
mem_fragmentation_ratio:2.54
mem_allocator:jemalloc-4.0.3
active_defrag_running:0
lazyfree_pending_objects:0

# Persistence
loading:0
rdb_changes_since_last_save:7610
rdb_bgsave_in_progress:0
rdb_last_save_time:1576058203
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:667648
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0

# Stats
total_connections_received:5113
total_commands_processed:83224
instantaneous_ops_per_sec:1
total_net_input_bytes:17903084
total_net_output_bytes:44580487
instantaneous_input_kbps:0.04
instantaneous_output_kbps:1.51
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:659
evicted_keys:0
keyspace_hits:39460
keyspace_misses:5512
pubsub_channels:1
pubsub_patterns:0
latest_fork_usec:148
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

# Replication
role:master
connected_slaves:0
master_replid:8a05024149f1cb5c5bd73a05871404048e0a7515
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:11.62
used_cpu_user:9.34
used_cpu_sys_children:0.09
used_cpu_user_children:0.43

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=713,expires=573,avg_ttl=3078364011427
db1:keys=3,expires=0,avg_ttl=0
db2:keys=130,expires=2,avg_ttl=1678549

[KamranShahid]

If some one even point me out for stackexchange.redis altenative for .net core 3.0/3.1 then it would also help.

[KamranShahid]

What i have done is set min thread to 200 in application startup project file as I have read in some post.

<ServerGarbageCollection>false</ServerGarbageCollection>
<ConcurrentGarbageCollection>true</ConcurrentGarbageCollection>
<ThreadPoolMinThreads>200</ThreadPoolMinThreads>

Now i am not getting any timeout error so far with my load. So basically it's look like threading issue on linux machine.

[mgravell]

Hmmm. I can see some Linux VMs in my future. It is very true that my own testing is very Windows centric we should fix that.

[NickCraver]

Indeed threading under heavy load (so that any .NET process can handle the work it's sharing with thread occupation), but since that's already explicitly called out in the docs we link to from the exception: https://stackexchange.github.io/StackExchange.Redis/Timeouts#are-you-seeing-a-high-number-of-busyio-or-busyworker-threads-in-the-timeout-exception I don't think there's additional action to take here.

Additionally, we're now running the test suite on Ubuntu every build as well to help catch issues there.

Glad you're up and running! Going to close this out to cleanup :)

[KamranShahid]

Thanks Nick.
I saw Stack StackExchange.Redis v2.1.0-preview.23 in nuget package update list. Hope there will be a release soon with some fixes (if found) as per Linux version load testing