You can limit the concurrent requests that can be handled at a time to 2000 and limit memory per request handler to 2MB.
So you allocate 2000 x 2MB at program start.
Then you crate an arena allocator with (2MB limit) when starting to handle a request.
It will just fail if a request uses more memory than 2MB. This means you will basically never have OOM issues or a request using unexpected amount of memory to crash the server.
You also have performance benefits like:
- Allocation operation is basically an integer increment.
- Memory for a request is laid out linearly in memory and likely is pre-populated so you won't have page fault into the kernel.
- It is very unlikely to have double free or other memory management mistakes.
You can also fuzz the system in several ways to make sure requests can really never exceed that amount of memory usage so you don't have surprises in runtime.
If a request by a user really exceeds that much memory usage, you can just fail that request and log it so it will be debugged and fixed later.
You also don't need data structures like SmallVec (rust version of the thing) that are hybrid stack/heap allocated.
There are many intricacies to this approach but it works much better on scale when you have a large program that handles a large amount of memory.
It is super difficult to debug and fix memory usage issues on global allocator only programs in my experience.
I tried to fix one such issue on a database-like program in rust and it was impossible.
Also this means you don't absolutely need ownership/lifetime safety.
I am using this pattern in a C-like C++ codebase I have and it works really well.