Hi Team,
We notice below redis timeout error in application logs -
Redis LogRetryData - Retry operation. RetryCount = [1]. Sleep duration = [00:00:05]. Causing exception:
[StackExchange.Redis.RedisTimeoutException: Timeout performing HMGET (5000ms), next: HMGET 0f21kpmv0wrwvnntd5cf3ah2:Cache, inst: 1, qu: 0, qs: 26, aw: False, rs: ReadAsync, ws: Idle, in: 188, in-pipe: 35, out-pipe: 762, serverEndpoint:, mgr: 10 of 10 available, clientName: , IOCP: (Busy=48,Free=952,Min=4,Max=1000), WORKER: (Busy=49,Free=32718,Min=4,Max=32767), v: 2.0.601.3402 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server)
at StackExchange.Redis.RedisBase.ExecuteSync[T](Message message, ResultProcessor`1 processor, ServerEndPoint server)
at StackExchange.Redis.RedisDatabase.HashGet(RedisKey key, RedisValue[] hashFields, CommandFlags flags)
at Terrasoft.Redis.StackExchangeAdapters.RedisClientAdapter.GetValuesFromHash[TValue](String hashId, String[] keys)
at Terrasoft.Redis.BaseRedisStore.<>c__DisplayClass22_0.b__0()
at Polly.Policy.<>c__DisplayClass119_0`1.b__0(Context ctx, CancellationToken ct)
at Polly.Policy.<>c__DisplayClass129_0`1.b__0(Context ctx, CancellationToken ct)
at Polly.RetrySyntax.<>c__DisplayClass12_1.b__1(Context ctx, CancellationToken ct)
at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, IEnumerable`1 shouldRetryExceptionPredicates, IEnumerable`1 shouldRetryResultPredicates, Func`1 policyStateFactory)]
A few articles found online suggested extending timeout on redis client using connectTimeout to more than default 5000ms. The timeout parameter was added in the connection string and set to 10000. Yet, we see the same type of timeout error referencing 5000ms.
Connection string for redis as follows-
; db=; port=6379; connectTimeout=10000; maxReadPoolSize=250; maxWritePoolSize=250; " />
Has anyone come across this issue before? Any insight is very much appreciated.
Like
Greetings!
This issue occurs due to a heavy load on the website.
We noticed that you recently contacted our support regarding this matter. After thoroughly examining the information you provided, we concluded that the issue likely arises because there is insufficient processing power during operations. You need to monitor performance. This error is related to performance, CPU load, or it could be a general (temporary server issue) due to server overload. At some point, certain operations slow down the system and degrade performance, affecting everything—logs and Redis included.
For example, there could be throttling on the website, insufficient CPU resources, or many concurrent threads, which could lead to a lack of retries. If you carefully review your error, you'll notice that typically only one retry is attempted—meaning the operation failed once and then a retry occurred. In this case, these are not critical errors but rather a consequence of what’s happening on the server. So, this issue is a result of a performance problem, and it could also be related to a filled Redis database. However, you need to analyze the overall website performance.
I also noticed that you changed the connectionStrings, but it is set in the web.config file in the Redis configuration.
In the Redis configuration, the connectionStringName="redis", the default timeout is set to 30 seconds.
Regards,
Orkhan