Webhook Handler Under High Load Causes Idle Queries and DB Connection Limit Issues
Hi everyone,
I’m currently facing a technical issue with a webhook handler I’ve implemented in Creatio.
Background:
- I have a webhook endpoint that gets called by an external system at a rate of approximately 150 hits per second.
- The Creatio database has a maximum connection limit of 400 connections.
The Problem:
During high load, I observed that the number of open database connections increases rapidly, and some queries are left idle (idle in transaction) even after the handler completes its process.
This leads to:
- Exhaustion of available database connections
- Timeout errors on other operations
- Overall system unresponsiveness
What I’ve Tried So Far:
- I modified the data access code by changing queries from ESQ (EntitySchemaQuery) to direct Entity usage.
- I manually handled database connections by using:
- using (DBConnection dbConnection = new DBConnection(...)) { ... }
- Explicit calls to .Close() and .Dispose() on connections to ensure cleanup.
Despite these adjustments, I'm still seeing idle queries accumulating under high load.
My Questions:
- What could be the possible reasons why queries remain idle even after the handler has completed?
- Are there any known best practices in Creatio for managing database connections under high request volumes like this?
- Is there a proper way to ensure UserConnection or underlying connections are completely released, especially in high-frequency webhook scenarios?
I would really appreciate any insights, suggestions, or shared experiences that could help resolve this issue.
Thank you in advance!
Like
Hello.
1. There are many potential reasons for this behavior:
- Blocking at the DB level (when a large number of requests block each other), such requests may be displayed as "idle in transaction".
- Incorrect operation in the code that executes requests, but "a lot" of code is executed between the start of the transaction and the commit.
- There is not enough capacity of the DB server to process requests (including releasing the connection after the request is completed).
There is also a probability that the issue lies in the handler itself, and it doesn't close the database connection once the query is executed, but the issue may also be in the server configuration.
2. The options that you use are suitable for such purposes (but of course it all depends on what code and what requests are being generated).
3. At the code level, they must be completely released. At the DB level, the DBMS itself is engaged in the release, with sufficient capacity it usually copes well.
Without knowing what code is executed, what requests, what capacities with such activity it is difficult to accurately name the cause and how it can be fixed.
Best regards,
Antonii.