[rabbitmq-discuss] reading time extending on persistent messages

Fri Jul 1 15:54:08 BST 2011

Hi Matthew,

thanks for response.

Today I tried RabbitMQ 2.5.1 out and get the same results as with RabbitMQ 2.4.1.

Here is a picture from my systemmonitoring on my german Ubuntu.
http://img405.imageshack.us/img405/4730/rabbitmqreceiving.png

I monitored also my disk drive and saw that rabbitmq starts with 500KB/s reading than after a while slow down and change always between 244 KB/s and 122 KB/s and at least it reads nothing or only with 122 KB/s. The systemmonitoring have always the same output.

I looked into all edited logfiles on the path /var/log/rabbitmq/ but I found no error.

Then I tried the same thing on a other system with Ubuntu 9.10 on a Single Core 2,3GHZ and only 512MB RAM and RabbitMQ 2.5.1. With this system I get almost the same results as you.

So what's the different between these machines?
The only thing I have found was that on my first machine with a slow reading I have the new erlang version of all erlang packages 1:13.b.3-dfsg-2ubuntu2.1 and on my fast reading machine I have the old erlang package version 1:13.b.1-dfsg-2ubuntu1.1. 

Could this be the problem?

Heres my output from my slow reading machine.

The last 3000 Messages needed 914
The last 3000 Messages needed 1466
The last 3000 Messages needed 1249
The last 3000 Messages needed 2564
The last 3000 Messages needed 3621
The last 3000 Messages needed 4506
The last 3000 Messages needed 5351
The last 3000 Messages needed 6027
The last 3000 Messages needed 7063
The last 3000 Messages needed 6809
The last 3000 Messages needed 7624
The last 3000 Messages needed 7886
The last 3000 Messages needed 8881
The last 3000 Messages needed 9712
The last 3000 Messages needed 10641
The last 3000 Messages needed 11635
The last 3000 Messages needed 10726
The last 3000 Messages needed 11339
The last 3000 Messages needed 12146
The last 3000 Messages needed 13149
The last 3000 Messages needed 14087
The last 3000 Messages needed 14049
The last 3000 Messages needed 13640
The last 3000 Messages needed 14688
The last 3000 Messages needed 15729
The last 3000 Messages needed 16540
The last 3000 Messages needed 17639
The last 3000 Messages needed 6200
The last 3000 Messages needed 16503
The last 3000 Messages needed 17880
The last 3000 Messages needed 18972
The last 3000 Messages needed 19816
The last 3000 Messages needed 20342
The last 3000 Messages needed 19970
The last 3000 Messages needed 21067
The last 3000 Messages needed 22279
The last 3000 Messages needed 22468
The last 3000 Messages needed 23510
The last 3000 Messages needed 24441
The last 3000 Messages needed 25701
The last 3000 Messages needed 26766
The last 3000 Messages needed 29099
The last 3000 Messages needed 25323
The last 3000 Messages needed 23663
The last 3000 Messages needed 22400
The last 3000 Messages needed 23353
The last 3000 Messages needed 24372
The last 3000 Messages needed 25267
The last 3000 Messages needed 26440
The last 3000 Messages needed 25486
The last 3000 Messages needed 25698
The last 3000 Messages needed 27725
The last 3000 Messages needed 28518
The last 3000 Messages needed 29462
The last 3000 Messages needed 29866
The last 3000 Messages needed 31868
The last 3000 Messages needed 32597
The last 3000 Messages needed 33528
The last 3000 Messages needed 35098
The last 3000 Messages needed 33956
The last 3000 Messages needed 26621
The last 3000 Messages needed 32605
The last 3000 Messages needed 34035
The last 3000 Messages needed 34754
The last 3000 Messages needed 35799
The last 3000 Messages needed 35499
The last 3000 Messages needed 35755
The last 3000 Messages needed 36829
The last 3000 Messages needed 37611
The last 3000 Messages needed 38507
The last 3000 Messages needed 39548
The last 3000 Messages needed 32433
The last 3000 Messages needed 41676
The last 3000 Messages needed 42586
The last 3000 Messages needed 41508
The last 3000 Messages needed 41379
The last 3000 Messages needed 37894
The last 3000 Messages needed 43723
The last 3000 Messages needed 44153
The last 3000 Messages needed 44836
The last 3000 Messages needed 45985
The last 3000 Messages needed 43398
The last 3000 Messages needed 43834
The last 3000 Messages needed 46645
...and so on

-----Original Message-----________________________________________
Von: rabbitmq-discuss-bounces at lists.rabbitmq.com [rabbitmq-discuss-bounces at lists.rabbitmq.com] im Auftrag von Matthew Sackman [matthew at rabbitmq.com]
Gesendet: Donnerstag, 30. Juni 2011 14:58
An: rabbitmq-discuss at lists.rabbitmq.com
Betreff: Re: [rabbitmq-discuss] reading time extending on persistent messages

Hi Arno,

On Wed, Jun 29, 2011 at 03:34:08PM +0200, Arno Puschmann wrote:
> I'm using rabbitmq 2.4.1 on a Ubuntu 10.04. I tested to write  a large
> number (300k) of messages to an persistent queue. Alle messages are
> the same and have a size of 1KB. After this I restartet my system and
> rabbitmq, to go save that nothing is on the memory but all on disk.
> After restarting all, I startet a consumer to readout the messages.
> Now I noticed that reading out the persistent messages took always
> longer. I measured the time for reading out every interval of 3k
> messages. For example, to readout the first 3k messages it took under
> 1sec. After 45k messages it took over 10sec. to readout the next 3k
> messages. After reading 165k messages it took about 30sec. to read the
> next 3000 messages. I have repeat these test several times always with
> a similar result.

I've used your consuming code that you supplied and repeated your test.
I did not reboot, but I did shut down rabbit and cleared out the caches.
I checked and Rabbit does sit there reading at 20MB/s off disk on
consuming.

I sent in 300,000 1KB persistent messages to a durable queue, then
safely shutdown Rabbit and restarted it, after clearing caches. I used
the Erlang client to publish and the Java client (with your code) to
consume the messages. The output I see is below.

Note that this is with the latest from default which will pretty much be
the same as 2.5.1. There haven't been any performance impacting changes
to the persistence layer for a little while. That said, there was quite
a gap between 2.4.1 and 2.5.0 so it may be something came in there.
Checking the release notes for 2.5.0 -
http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2011-June/013249.html
- there are a number of performance related items but nothing that
strikes as relevant to your test.

Are there any indicators in your RabbitMQ logs that suggest Rabbit's
under memory pressure? Is there contention for the disk drive? Maybe try
upgrading to 2.5.1 and see if that helps?

Best wishes,

Matthew

The last 3000 Messages needed 332
The last 3000 Messages needed 314
The last 3000 Messages needed 310
The last 3000 Messages needed 383
The last 3000 Messages needed 449
The last 3000 Messages needed 436
The last 3000 Messages needed 309
The last 3000 Messages needed 320
The last 3000 Messages needed 363
The last 3000 Messages needed 340
The last 3000 Messages needed 377
The last 3000 Messages needed 369
The last 3000 Messages needed 404
The last 3000 Messages needed 379
The last 3000 Messages needed 408
The last 3000 Messages needed 445
The last 3000 Messages needed 379
The last 3000 Messages needed 357
The last 3000 Messages needed 434
The last 3000 Messages needed 325
The last 3000 Messages needed 327
The last 3000 Messages needed 508
The last 3000 Messages needed 334
The last 3000 Messages needed 360
The last 3000 Messages needed 326
The last 3000 Messages needed 339
The last 3000 Messages needed 313
The last 3000 Messages needed 458
The last 3000 Messages needed 355
The last 3000 Messages needed 341
The last 3000 Messages needed 358
The last 3000 Messages needed 331
The last 3000 Messages needed 514
The last 3000 Messages needed 339
The last 3000 Messages needed 363
The last 3000 Messages needed 355
The last 3000 Messages needed 353
The last 3000 Messages needed 372
The last 3000 Messages needed 417
The last 3000 Messages needed 375
The last 3000 Messages needed 350
The last 3000 Messages needed 438
The last 3000 Messages needed 339
The last 3000 Messages needed 554
The last 3000 Messages needed 342
The last 3000 Messages needed 340
The last 3000 Messages needed 368
The last 3000 Messages needed 372
The last 3000 Messages needed 372
The last 3000 Messages needed 420
The last 3000 Messages needed 392
The last 3000 Messages needed 358
The last 3000 Messages needed 362
The last 3000 Messages needed 347
The last 3000 Messages needed 509
The last 3000 Messages needed 402
The last 3000 Messages needed 441
The last 3000 Messages needed 437
The last 3000 Messages needed 373
The last 3000 Messages needed 359
The last 3000 Messages needed 657
The last 3000 Messages needed 345
The last 3000 Messages needed 461
The last 3000 Messages needed 384
The last 3000 Messages needed 459
The last 3000 Messages needed 648
The last 3000 Messages needed 361
The last 3000 Messages needed 369
The last 3000 Messages needed 396
The last 3000 Messages needed 407
The last 3000 Messages needed 436
The last 3000 Messages needed 376
The last 3000 Messages needed 369
The last 3000 Messages needed 409
The last 3000 Messages needed 407
The last 3000 Messages needed 378
The last 3000 Messages needed 542
The last 3000 Messages needed 391
The last 3000 Messages needed 430
The last 3000 Messages needed 389
The last 3000 Messages needed 384
The last 3000 Messages needed 480
The last 3000 Messages needed 391
The last 3000 Messages needed 434
The last 3000 Messages needed 417
The last 3000 Messages needed 387
The last 3000 Messages needed 409
The last 3000 Messages needed 609
The last 3000 Messages needed 411
The last 3000 Messages needed 472
The last 3000 Messages needed 388
The last 3000 Messages needed 399
The last 3000 Messages needed 456
The last 3000 Messages needed 430
The last 3000 Messages needed 400
The last 3000 Messages needed 420
The last 3000 Messages needed 394
The last 3000 Messages needed 426
The last 3000 Messages needed 434
The last 3000 Messages needed 416

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss