In this article I am going to present few rules I use when selecting prefetch
count value for RabbitMQ. Those rules are based on the experience I gained when working with RabbitMQ. You should treat them as a guidance and starting point, remembering that each application, network and queue topology is different. Let’s start with quick explanation what prefetch
count is.
Prefetchcount
RabbitMQ allows consumers to specify the size of the limit of unacknowledged messages on a queue basis. This value is used to specify how many messages is send to the consumer and cached by RabbitMQ client library. Once the buffer is full the RabbitMQ will wait with delivering new messages to that consumer until it sends ACKs / NACKs. The main purpose for pre fetching messages is to optimise message delivering. The ideal situation is where there is a message delivered and waiting for the processor to be consumed.
Of course, caching messages on the consumer side has repercussions. All pre-fetched messages are removed from the queue and become invisible to other processors. Hence setting prefetch
count to a high value will skew message distribution between different consumers.
Consumer types
Below are rules for selecting prefetch
count depending on the consumer type.
Fast, single consumer
If there is only one consumer processing messages quickly you definitely want all messages to be pre-fetched. The optimum value will probably be something higher than 20, but setting it to anything above 50 will rarely delivery more value. Remember that pre-fetched messages are cached on the client side consuming its resources. High value will also restrict your ability to peek messages.
Multiple fast consumers
When you have multiple consumers which are processing messages fast you need to find value which keeps messages waiting for consumers as well as allows for balanced message delivery between consumers. Low value will penalise consumers as they need to wait for messages. High value will damage load balancing. I usually use anything between 20 and 30.
Slow consumers
For slow consumers message delivery time is negligible. My advice is to set prefetch
count to 1 and let RabbitMQ load balance message between consumers.
Application crash and buffered messages
Let’s see what happens to buffered messages when the application crashes. Because RabbitMQ automatically puts all not ACKed messages back to the queue, if the application crashes then all cached messages will simply go back to the top of the queue.
Setting prefetch count
To set ‘prefetch’ count in RabbitMQ you have to execute basic.qos command[1].
Setting prefetch
count in EasyNetQ
You can set prefetch
count value in EasyNetQ by including it in the bus’ connection string. Default value used by EasyNetQ is 50.
Below example sets prefetch
count to 30:
var bus = RabbitHutch.CreateBus("host=localhost;prefetchcount=30");
Resources
You may find more information about prefetch count in following articles:
For example of usage check Work Queues tutorial. ↩︎