[rabbitmq-discuss] stomp+ssl message send works, receive hangs after around 200-300 messages

Thu May 24 17:08:09 BST 2012

Michael,

Thank you for the further information, this has got us a bit puzzled.

It was a long shot, and my colleagues don't think that ACKs ought to
have been blocked, but I'm glad you at least have a work-around. Are the
SUBSCRIBER STOMP clients SENDing anything (or writing other things) on
the same connection, by any chance? If they are this could explain it,
and the problem ought to be fixed in the next release. The
hanging/jamming problem occurs (apparently) only when sending STOMP
frames that have content and an explicit length header (and other
'high-load' things happen at the same time). But I asked you to try it
anyway.

> Non-SSL connections pass the same tests without problems.

This may be because an SSL connection has a higher 'loading' on
resources, but I'm not sure why this should be.

> Setting prefetch-count=2 fixed it, many thanks ! Could this also affect
> transactions, where a commit is sent instead of ack frames?

If it is the problem I think it is, then yes, but your question makes me
wonder if you remember that ACKs are transactional, not the consuming of
the messages, so you should not think of commit as doing an implicit
ACK: it doesn't. In particular, rollback does not re-queue un-acked
messages. Rollback will 'undo' any acknowledgements made in this
transaction (and, in RabbitMQ, any rejects, too). I'm sure you know
this, but your use of the word 'instead' got me worried. :-)

The bug that I know we've fixed is likely to 'hang' a STOMP frame that
occurs at the end of a buffer, after a particular 'throttling' effect
occurs. This ought not to be frame-specific, so whether we are
transactional or not shouldn't be relevant.

However, I know I'm not speaking for all of us, here, and it still may
not be this bug you are experiencing.

Steve Powell  (a happy bunny)
----------yet more definitions from the SPD----------
corrugate (n.) T.V. soap scandal.
olympic (n.) A camp road-digger.
jamboree (n.) A conserve made from French cheese.

On 24 May 2012, at 13:10, Michael Justin wrote:

> Am 23.05.2012 17:52, Steve Powell wrote:
>> Michael,
>> 
>> Thanks for the update. Can you be a little more specific about the
>> set-up that fails?
> 
> Many thanks for your answer and your suggestion, it is working now - with prefetch-count=2.
> 
> Maybe it is helpful, here is some information from my tests and answers to your questions:
> 
> 
>> In particular I would like to know:
>> 
>> -  the version of Erlang you are using, and the (v)hardware you are
>>    running RabbitMQ on (a copy of the startup log would be nice);
> 
> The startup log is at the end, Erlang version is 5.9.1, machine is a Compaq 6820s Laptop (Intel Core 2 Duo T8100 2.1 GHz)
> 
>> -  if the problem is noticed on a non-ssl connection (if you can try
>>    this) -- we haven't seen this particular problem on our tests;
> 
> Non-SSL connections pass the same tests without problems
> 
>> -  are there any messages in the rabbitmq log around the time the
>>    message delivery stops?
> 
> The log is very small, see attached file (after running > 10 tests)
> 
>> -  can you provide more details about the topic subscription --
>>    is this a durable subscription, for example;
> 
> It is a normal queue destination, using persistent messages
> 
>> -  and do the acknowledgements actually get through (you can tell this
>>    if messages you think you have consumed are redelivered on
>>    reconnect)?
> 
> I can see after retrieving around 240 messages (when the client does not longer receive more messages) that about 120 messages are still on the broker, so the acknowledgement of 120 messages did not reach the broker.
> 
> Further debugging showed that the client hangs when sending the acknowledgement frame. Example:
> 
> received:
> MESSAGE
> content-type:text/plain
> subscription:{00AC21DB-D5A3-40C9-97C4-00FCB902634B}
> destination:/queue/ExampleQueue
> message-id:T_{00AC21DB-D5A3-40C9-97C4-00FCB902634B}@@session-ArgQuHJyTPiO2ge46jB
> 4eD@@245
> content-length:255
> 
> 
> Received: Message: 244 sent at: 24.05.2012 13:52:01         ...
> send:
> ACK
> message-id:T_{00AC21DB-D5A3-40C9-97C4-00FCB902634B}@@session-ArgQuHJyTPiO2ge46jB
> 4eD@@245
> subscription:{00AC21DB-D5A3-40C9-97C4-00FCB902634B}
> 
> <-- client frame sent but the socket write operation hangs
> 
>> We have recently discovered (and fixed) a bug in the STOMP client that
>> will stop the broker seeing the last message sent (under load). It is
>> possible that, if pre-fetch is set to 1, this is causing you to jam
>> up -- the STOMP code may not be detecting your last acknowledgement. To
>> confirm this, please try this with a pre-fetch count of zero (or 2 or
>> more) and see if you still get the lock-up.
>> 
> 
> Setting prefetch-count=2 fixed it, many thanks ! Could this also affect transactions, where a commit is sent instead of ack frames?
> 
> Regards
> 
> Michael Justin
> 
> ---
> 
> Startup log
> 
> Activating RabbitMQ plugins ...
> 7 plugins activated:
> * amqp_client-2.8.2
> * mochiweb-1.3-rmq2.8.2-git
> * rabbitmq_management-2.8.2
> * rabbitmq_management_agent-2.8.2
> * rabbitmq_mochiweb-2.8.2
> * rabbitmq_stomp-2.8.2
> * webmachine-1.7.0-rmq2.8.2-hg
> 
> 
> +---+   +---+
> |   |   |   |
> |   |   |   |
> |   |   |   |
> |   +---+   +-------+
> |                   |
> | RabbitMQ  +---+   |
> |           |   |   |
> |   v2.8.2  +---+   |
> |                   |
> +-------------------+
> AMQP 0-9-1 / 0-9 / 0-8
> Copyright (C) 2007-2012 VMware, Inc.
> Licensed under the MPL.  See http://www.rabbitmq.com/
> 
> node           : rabbit at MJ-PC
> app descriptor : c:/Java/rabbitmq_server-2.8.2/sbin/../ebin/rabbit.app
> home dir       : C:\Users\mj
> config file(s) : c:/Users/mj/AppData/Roaming/RabbitMQ/rabbitmq.config
> cookie hash    : WjOr7YEHHESuUkR1mMuR+g==
> log            : C:/Users/mj/AppData/Roaming/RabbitMQ/log/rabbit at MJ-PC.log
> sasl log       : C:/Users/mj/AppData/Roaming/RabbitMQ/log/rabbit at MJ-PC-sasl.log
> database dir   : c:/Users/mj/AppData/Roaming/RabbitMQ/db/rabbit at MJ-PC-mnesia
> erlang version : 5.9.1
> 
> -- rabbit boot start
> starting file handle cache server ...done
> starting worker pool ...done
> starting database ...done
> starting database sync ...done
> starting codec correctness check ...done
> -- external infrastructure ready
> starting plugin registry ...done
> starting auth mechanism cr-demo ...done
> starting auth mechanism amqplain ...done
> starting auth mechanism plain ...done
> starting statistics event manager ...done
> starting logging server ...done
> starting exchange type direct ...done
> starting exchange type fanout ...done
> starting exchange type headers ...done
> starting exchange type topic ...done
> -- kernel ready
> starting alarm handler ...done
> starting node monitor ...done
> starting cluster delegate ...done
> starting guid generator ...done
> starting memory monitor ...done
> -- core initialized
> starting management agent ...done
> starting exchange, queue and binding recovery ...done
> starting configured definitions ...done
> starting empty DB check ...done
> starting mirror queue slave sup ...done
> starting adding mirrors to queues ...done
> -- message delivery logic ready
> starting error log relay ...done
> starting networking ...done
> starting direct client ...done
> starting notify cluster nodes ...done
> 
> broker running
> ** Found 0 name clashes in code paths
> 
> 
> -- 
> Michael Justin
> habarisoft - Enterprise Messaging Software for Delphi
> http://www.habarisoft.com/
> <rabbit at MJ-PC.log>