This post is a follow up to the UK Azure User Group presentation I gave yesterday about running RabbitMQ in Azure.
When deciding on messaging system to use in your cloud solution, one of the consideration will be the cost. Azure offers Service Bus Queues and Topics as PaaS, but it also offers Virtual Machines (as IaaS) which you can use to host other messaging brokers, meaning you can host your own RabbitMQ cluster in Azure.
RabbitMQ
The cost of running a RabbitMQ cluster in Azure can be calculated as a monthly cost of Virtual Machines which consist the cluster. If you want to build a cluster with 3 large instance (4 vCPU, 7GB RAM) running Linux OS, then your monthly cost will be £340.91 or $535.68 (check Azure Calculator for other currencies).
With 3 node cluster you can replicate the same capabilities for you messaging broker as in Service bus: messages being replicate to 3 nodes and high cluster availability. It also gives you very decent performance of few thousand “reliable” messages per second, or tens of thousand “transient” messages a second.
Bear in mind that I haven’t included any cost related to maintaining Virtual Machines. In your calculations you will need to include that as well, but the cost depends much on the efficiency of a person maintaining Linux and RabbitMQ.
Azure Service Bus queue
Now, let’s see how Azure Service Bus compares to that. First, few assumptions:
- Your application is using Push delivery model. Because pull delivery model incurs cost (each pull request is one message operation, even when no message was delivered), you will be better of with push model.
- Messages are less than 64 KB in size. Each message is divided into 64 KB chunks and you are billed for operations on chunks (sending and delivering). If your message size is 128 KB than it will cost you twice as much as 64 KB message.
- The messages are delivered to one queue only. Azure Service Bus charges are based on message operations. Sending a message to the broker is one operations, and delivering message to the queue is another operation. If the message gets delivered to 4 queues, Service Bus will charge 5 message operations (1 send and 4 deliveries)
- And finally, I assume perfect messages flow, meaning none of the messages get abandoned or dead lettered.
To spend £340.91 you need to do 536 M message operations in a month. Using above assumptions that means processing 268 M messages per month (processing is defined here as publishing and consuming a message). 268 million message a month translates to:
- 8.81 million messages a day
- 367,123 messages per hour
- 6,119 messages per minute
- or, 102 messages per second
When processing with maximum throughput of 2 000 messages per second[1], you will use up the credit in about 37 hours.
You can process the same number of messages with RabbitMQ in under 2 hours[2].
Economics
As you can see, simple cost comparison shows that to get your Return Of Investment from RabbitMQ, you need to have a system processing perfectly at least 268 million messages a month. It may sound a lot at first, but in reality many applications are dealing with much more than that – it is only 6K messages per minute. The number will be even lower once you start using more advanced messaging when one message is delivered to more than one queue.
Scaling down RabbitMQ cluster
The cluster I have chosen to make a comparison is quite a powerful, and that translates directly to cost. Fortunately, you have many ways to scale RabbitMQ broker. First you can use Medium instances (A2). This will half the power (2 CPUs and 3.5 GB RAM) but it will also drop down your monthly bill to £170.46.
If you are willing to drop few per mills from the availability, you can use two nodes instead of three. Together with smaller instance size this will reduce monthly cost down to £113.64.
The £113 cost is an equivalent of 89 million messages processed per month in Azure Service Bus, or:
- 2.93 million messages a day
- 48,767 messages per hour
- 813 messages per minute
- or, just 13.5 per second
I strongly advice against using smaller instance size than A2. Lack of CPU and RAM will completely cripple the broker. It is also not advised to use only one instance of RabbitMQ in the cloud.
Finally, during the first stage when your application doesn’t generate heavy load you may decide to deploy processors to the same virtual machines which are running RabbitMQ cluster. Probably this will mean using Windows OS which is slightly more expensive, but you will save money on worker roles hosting processors.
Occasional processing
Another option to get savings with RabbitMQ installation is in the systems which doesn’t have to run 24/7. If your application does processing periodically, such as processing reports daily at 3am, than you may run the cluster only for few minutes a day. And a billing unit for charging Azure Virtual Machines is a minute! Below table shows comparison of Azure Service Bus and RabbitMQ cost for occasional processing:
| NO.OF MESSAGES | SB TIME | SB COST | R TIME | R COST |
|---|---|---|---|---|
| 100,000 | 50 secs | £0.06 | 2 mins | £0.02 |
| 500,000 | 4 mins | £0.32 | 2.5 mins | £0.02 |
| 1,000,000 | 8 mins | £0.64 | 3 mins | £0.02 |
| 5,000,000 | 42 mins | £3.19 | 5 mins | £0.04 |
| 10,000,000 | 83 mins | £6.37 | 7 mins | £0.05 |
| 50,000,000 | 7 hours | £31.82 | 23 mins | £0.18 |
* R Time and T Cost show data for RabbitMQ
You also should consider the cost associated with running worker roles for processing messages. RabbitMQ gives you much better messages throughput meaning more messages per second. If you can consume messages quickly enough, you may save on time required to run your worker roles.
Summary
I hope that I could shed some light on costs of running RabbitMQ cluster in Azure and this article will help you when thinking about costs. As you could see, there are many aspects to consider: how much faster you can process messages, how big cluster to use, or whether to run processors on same VMs as RabbitMQ cluster.
And do not forget that this comparison shows only the running cost related to keeping message broker. As for any application, the total cost is a combination of development, deployment, maintenance and running cost.
Whereas development cost may be comparable between the two, RabbitMQ will have higher deployment and maintenance cost. You need to deploy and maintain Virtual Machines as well as RabbitMQ server. I must say that it is incredibly easy to deploy Linux VM in Azure and install RabbitMQ cluster, but still it is something you have to do.
And finally, think about future scalability requirements for the application. Azure Service Bus queue are very restricted from scalability point of view, which may cause an issue when the application grows rapidly and you need to scale your messaging system. Inability to scale application along with user demand may have very high cost, so my advice is not to be scared and think about it. The cloud is for elasticity.
A throughput of 2000 messages per second was taken from Azure Queues and Service Bus Queues – Compared and Contrasted page. I haven’t run performance test against Azure Service bus queues to verify it, but many people told me that it is rather optimistic value. ↩︎
Calculation based on the performance of 40,000 messages per second. ↩︎