[rabbitmq-discuss] reading time extending on persistent messages

[Matthew Sackman]

Hi Arno,

On Wed, Jun 29, 2011 at 03:34:08PM +0200, Arno Puschmann wrote:
> I'm using rabbitmq 2.4.1 on a Ubuntu 10.04. I tested to write  a large
> number (300k) of messages to an persistent queue. Alle messages are
> the same and have a size of 1KB. After this I restartet my system and
> rabbitmq, to go save that nothing is on the memory but all on disk.
> After restarting all, I startet a consumer to readout the messages.
> Now I noticed that reading out the persistent messages took always
> longer. I measured the time for reading out every interval of 3k
> messages. For example, to readout the first 3k messages it took under
> 1sec. After 45k messages it took over 10sec. to readout the next 3k
> messages. After reading 165k messages it took about 30sec. to read the
> next 3000 messages. I have repeat these test several times always with
> a similar result.

I've used your consuming code that you supplied and repeated your test.
I did not reboot, but I did shut down rabbit and cleared out the caches.
I checked and Rabbit does sit there reading at 20MB/s off disk on
consuming.

I sent in 300,000 1KB persistent messages to a durable queue, then
safely shutdown Rabbit and restarted it, after clearing caches. I used
the Erlang client to publish and the Java client (with your code) to
consume the messages. The output I see is below.

Note that this is with the latest from default which will pretty much be
the same as 2.5.1. There haven't been any performance impacting changes
to the persistence layer for a little while. That said, there was quite
a gap between 2.4.1 and 2.5.0 so it may be something came in there.
Checking the release notes for 2.5.0 -
http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-June/013249.html
- there are a number of performance related items but nothing that
strikes as relevant to your test.

Are there any indicators in your RabbitMQ logs that suggest Rabbit's
under memory pressure? Is there contention for the disk drive? Maybe try
upgrading to 2.5.1 and see if that helps?

Best wishes,

Matthew

The last 3000 Messages needed 332
The last 3000 Messages needed 314
The last 3000 Messages needed 310
The last 3000 Messages needed 383
The last 3000 Messages needed 449
The last 3000 Messages needed 436
The last 3000 Messages needed 309
The last 3000 Messages needed 320
The last 3000 Messages needed 363
The last 3000 Messages needed 340
The last 3000 Messages needed 377
The last 3000 Messages needed 369
The last 3000 Messages needed 404
The last 3000 Messages needed 379
The last 3000 Messages needed 408
The last 3000 Messages needed 445
The last 3000 Messages needed 379
The last 3000 Messages needed 357
The last 3000 Messages needed 434
The last 3000 Messages needed 325
The last 3000 Messages needed 327
The last 3000 Messages needed 508
The last 3000 Messages needed 334
The last 3000 Messages needed 360
The last 3000 Messages needed 326
The last 3000 Messages needed 339
The last 3000 Messages needed 313
The last 3000 Messages needed 458
The last 3000 Messages needed 355
The last 3000 Messages needed 341
The last 3000 Messages needed 358
The last 3000 Messages needed 331
The last 3000 Messages needed 514
The last 3000 Messages needed 339
The last 3000 Messages needed 363
The last 3000 Messages needed 355
The last 3000 Messages needed 353
The last 3000 Messages needed 372
The last 3000 Messages needed 417
The last 3000 Messages needed 375
The last 3000 Messages needed 350
The last 3000 Messages needed 438
The last 3000 Messages needed 339
The last 3000 Messages needed 554
The last 3000 Messages needed 342
The last 3000 Messages needed 340
The last 3000 Messages needed 368
The last 3000 Messages needed 372
The last 3000 Messages needed 372
The last 3000 Messages needed 420
The last 3000 Messages needed 392
The last 3000 Messages needed 358
The last 3000 Messages needed 362
The last 3000 Messages needed 347
The last 3000 Messages needed 509
The last 3000 Messages needed 402
The last 3000 Messages needed 441
The last 3000 Messages needed 437
The last 3000 Messages needed 373
The last 3000 Messages needed 359
The last 3000 Messages needed 657
The last 3000 Messages needed 345
The last 3000 Messages needed 461
The last 3000 Messages needed 384
The last 3000 Messages needed 459
The last 3000 Messages needed 648
The last 3000 Messages needed 361
The last 3000 Messages needed 369
The last 3000 Messages needed 396
The last 3000 Messages needed 407
The last 3000 Messages needed 436
The last 3000 Messages needed 376
The last 3000 Messages needed 369
The last 3000 Messages needed 409
The last 3000 Messages needed 407
The last 3000 Messages needed 378
The last 3000 Messages needed 542
The last 3000 Messages needed 391
The last 3000 Messages needed 430
The last 3000 Messages needed 389
The last 3000 Messages needed 384
The last 3000 Messages needed 480
The last 3000 Messages needed 391
The last 3000 Messages needed 434
The last 3000 Messages needed 417
The last 3000 Messages needed 387
The last 3000 Messages needed 409
The last 3000 Messages needed 609
The last 3000 Messages needed 411
The last 3000 Messages needed 472
The last 3000 Messages needed 388
The last 3000 Messages needed 399
The last 3000 Messages needed 456
The last 3000 Messages needed 430
The last 3000 Messages needed 400
The last 3000 Messages needed 420
The last 3000 Messages needed 394
The last 3000 Messages needed 426
The last 3000 Messages needed 434
The last 3000 Messages needed 416