ActiveMQ flow control and Apache Camel transacted route gotchas

We have a system that uses Apache Camel and ActiveMQ. It handles periodic bursts of 20,000 messages. The end consumer is slow and it takes about an hour to process all the messages. On their route, messages are passed from one queue to another. On our production system we stumbled onto an unexpected problem. After finishing few thousand messages the whole system would freeze.

The problem was with ActiveMQ flow control and transacted routes in Camel.

When you have slow consumers, to prevent queues from growing infinitely, ActiveMQ has a limit of how many messages you can put in a queue. When limit is reached, producer is, by default, forced to wait until the resources free. You can set these limits on system level or per queue level. Problem arises when you have transacted Camel route and system limit is reached.

AMQ Flow Control

Route that moves messages from queue A to queue B is inside a JMS transaction – you can’t remove message from queue A until the message is successfully placed on queue B.

If system limit is reached, no new messages can be sent to any queue. So, producers are forced to wait, transaction doesn’t complete, messages can’t be taken off queue A and no resource gets freed. TheĀ  whole processing freezes.

There are numerous ways you can work around this problem. You can turn off flow control and potentially let queues grow indefinitely.

In our case, solution was to set per-queue limits so that system limits can never be reached. Sum of limits for all queues needs to be less than the system limit. That way, as consumer takes messages from queue B, new messages can come in, transactions can complete and messages can be taken off queue A. Messages are consumed from queues A and B at the same pace and the whole system works fine.

In our case, I’ve set memoryLimit to 10m for our 13 queues and system memoryUsage to 180m.

For details see :

Category: Software | Tags: , , , , 2 comments »

2 Responses to “ActiveMQ flow control and Apache Camel transacted route gotchas”

  1. Christian

    Thanks for sharing your experience with us. We are running into exactly the same issue. With your post, you hopefully safe much time on our side.

    Thanks again,
    Christian – a Camel rider ;-)

  2. Sabri

    Hi,

    thanks for this nice article. I have one question pls :
    can I control the number of messages that hasn’t be acknowledged through flow control in apache camel ?

    in fact, I used apache camel in particular activemq and smpp connector and I want to stop the sent of messages to SMS-C if the number of unacknowledged messages reach a given number

    Thanks a lot for your help,

    –Sabri


Leave a Reply



 

Back to top