[rabbitmq-discuss] Unexpected Behavior When Using the "X-Consistent-Hash" Exchange Type

[Richard Raseley]

We have encountered some unexpected behavior when using the
"x-consistent-hash" exchange type that I am hoping someone on the list can
help us understand.

At a high level, the issue we're experiencing is that in our testing of the
"x-consistent-hash" exchange we aren't able to get even (or roughly even)
distribution of messages between the queues bound to the exchange.

Here is a link to the full broker configuration export (
http://pastebin.com/nkcXskpY ) but at a high level we are doing the
following:

1) Topic exchange named "logging.topic" is bound to hash exchange
"error.hash" with routing key "error.#" (our eventual goal is to have
several of these).
2) Hash exchange "error.hash" is configured with parameters "hash-header:
message-distribution-hash".
3) Queues named "error01" through "error16" are bound to the "error.hash"
exchange each with routing key of "1" (to ensure they each get an equal
slice of the hash range).

When executing a simple test script ( http://pastebin.com/e7EjZ1TD ), which
sends 500K messages to the "logging.topic" with a
"message-distribution-hash" header that a value that is a UUID converted to
a string, I see quite unequal distribution of the messages across the
queues. I ran three tests of 500K messages (resetting and reloading the
broker configuration between each test) and I get the following results:

**FIRST TEST**

Queue Name,Message Count
error01,73616
error02,65789
error03,12986
error04,15029
error05,12026
error06,51006
error07,465
error08,95926
error09,34857
error10,16858
error11,19075
error12,25155
error13,25002
error14,14994
error15,8878
error16,28338
TOTAL,500000

**SECOND TEST**

Queue Name,Message Count
error01,23124
error02,5122
error03,29047
error04,24405
error05,1467
error06,1312
error07,28045
error08,35258
error09,164437
error10,25851
error11,40669
error12,24821
error13,17316
error14,18611
error15,41183
error16,19332
TOTAL,500000

**THIRD TEST**

Queue Name,Message Count
error01,26655
error02,77116
error03,405
error04,24106
error05,10543
error06,7734
error07,37476
error08,25048
error09,32156
error10,43750
error11,92675
error12,14325
error13,50265
error14,18449
error15,2665
error16,36632
TOTAL,500000

As you can see, some queues get *many* more messages than other queues for
example in the third test we see that one queue (error03) received only 405
messages (0.081% of the total) while another (error11) received 92675
(18.53% of the total). With all this in mind, my questions are as follows:

1) Why am I seeing such uneven distribution of messages between the queues
bound to the x-consistent-hash exchange?

2) What can I do to ensure the most even distribution of messages as
possible across the queues bound to the x-consistent-has exchange?

Regards,

Richard Raseley
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20131016/aca11d3d/attachment.htm>