In this blog, I’d like to summarize and share how I designed and implemented a fair queue to isolate tens of millions of workflows. It was an interesting and complex technical system.
Background
It was at the end of 2023, after I joined the Automation team of a Low-Code product. At that time, as the AI era had just started, many companies were transitioning to AI and incubating various AI products internally. Our team was responsible for a workflow system that supported Lark and was going to support Coze. At that time, hundreds of thousands of tenants were using the workflow system.
The problem
When I joined the team, the system had been running for more than 6 years. As the number of tenants was increasing rapidly at that time, the servers began to be blocked for long periods or crash suddenly. At first, a team member scaled the servers from 20 to 30 and adjusted the rate limit from 20 to 10. But the problem was still there. So I was assigned to solve the problem.
Analyzing
After I analyzed the system using pprof and traces, I found that more and more tenants had begun to build very complex workflows. Some of them ran for more than 15 minutes, and a few even ran for over an hour, using lots of memory. And there were often many instances of long-running workflows operating concurrently. So RPS or latency were no longer suitable metrics, and rate limiting or scaling the servers was no longer effective.
Solve Quickly
It did not make sense to restrict tenants from creating complex workflows. One good approach would be to allocate a quota to each tenant, like cloud services do, but because of the limitations of our architecture, the cost was too high. So I suggested a solution to control the number of concurrent instances for every workflow, limiting the number of complex workflow instances that tenants could submit. Considering the stability and the increase in tenant complaints, I decided to fix it quickly.
In the first version, I adopted an approach that counted the number of instances for each workflow, but I couldn’t simply increment and decrement the number. Even using a defer statement was not completely safe. Imagine the server crashed for some reason, or a long-running workflow was stopped by a deployment or a server restart. The number would never be decremented.
So I used the Redis hash structure to maintain the IDs of workflow instances and store the start time of each instance, controlling the concurrency through the number of fields in the hash. To avoid the problem mentioned above, I added an expiration time for each instance ID. After analyzing the data, 90% of workflows would finish within 15 minutes, so I set the expiration time to 30 minutes and added an alarm for workflows running longer than that.
Why is this method temporary?
Although this approach could fix the problem quickly and increase stability, there were some drawbacks.
- 01The expiration time was not completely safe. If users submitted a long-running workflow instance after the previous ones expired, the limit would not work. Though there was an alarm, people had to review and deal with it manually.
- 02The noisy neighbours. Users submitted their tasks into a Kafka queue first. If a tenant submitted a long-running workflow many times, it would block other tenants’ tasks. For example, if the limit was 5 and workflow A ran for 15 minutes, imagine that there were 4 instances of workflow A running, while other tasks from other tenants were waiting to run in the queue. The 5th instance of workflow A was submitted but would wait to run at the end of the queue, and the tenant submitted more than 20 instances of workflow A at that time. So it would block other tenants’ workflow tasks when they were running.
To resolve these drawbacks, what we needed was a fair queue to isolate the traffic across workflows. In the next blog, I will share why we needed to develop a fair queue and how I designed and implemented one to support tens of millions of workflows, with each workflow requiring its own queue.
End of article 4 min read
More posts