[rabbitmq-discuss] System CPU spikes, multi-second latency

Ben Rady benrady at gmail.com
Thu Oct 13 04:41:25 BST 2011


We've been having some latency problems with RabbitMQ that seem kinda out of
the ordinary given what I've read on this list and in the documentation
about it's capabilities. In some cases, the message rate for multiple
exchanges simultaneously drops for up to a minute. This is followed by a
giant spike as the messages finally rush through. Although no messages
appear to be lost, the latency is a problem for us, and any help would be
much appreciated.

== About the environment ==

- RabbitMQ 2.5.1 (Compiled from source tarball), web admin plugin installed,
otherwise out of the box configuration
- erlang-otp R14B03
- Redhat Enterprise 6.0
- IBM 3550, 12 cores, 192G memory (no, that's not a typo)
- A single instance of RabbitMQ, receiving roughly 1200 messages per second
across 70 (or so) fanout exchanges. Each exchange has 2 queues with one
associated consumer per queue. The queues are non-persistant and non-ack.
- There is one publisher per exchange, a Ruby process using the AMQP gem.
The consumers also use ruby with the AMQP gem (one is JRuby, the other MRI).

== What we're seeing ==

I've been able to capture a lot of stats during once of these events, and
I've attached some of the resulting data from one that lasted about 12
seconds. What happens (in roughly temporal order) is this:

1) As reported by top, one of the RabbitMQ threads (shown individually in
top using the 'H' option) spikes to 100% CPU. By comparing the last used CPU
in top to the per-cpu user/system/wait breakdown, it appears the thread is
spending all it's time in system calls (see attached screenshot).
2) As reported by netstat, TCP receive queues build up on the incoming
RabbitMQ socket connections. This happens in just a few seconds. See the
attached netstat.txt file for detailed stats from a particular episode.
3) The web admin plugin stops responding to API requests.
4) The incoming message rate across multiple exchanges drops dramatically.
The system can stay in this state for 10-30 seconds or more.
5) The CPU levels return to normal (sometimes a bit above normal), and the
message rate jumps. Some queuing on the consumers happens as they try to
catch up with flood of messages.
6) The consumers work off the extra messages, and the system returns to
normal (40%, or so, cpu for rabbit, < 5% for the consumers).

These episodes last from just a couple of seconds up to a minute, and seem
to happen randomly, sometimes coming one right after another, sometimes
hours apart. We've never seen it less than once a day.

==What we've already done==

1) Troubleshooting using strace
Since the thread seems to be pegged in system calls, we've been using strace
to try and capture the system call made by the offending thread during the
event. Unfortunately, the best we've been able to do is capture the system
calls immediately following the event. For example, here are the first
system calls captured for two threads, once of which can be seen with a
pegged CPU in the attached screenshot (pid 21860).

$ head rabbit-thread.21860
21:13:53.109706 futex(0x1f2bc20, FUTEX_WAKE_PRIVATE, 1) = 1 <0.000021>
21:13:53.109805 futex(0x1f2bb60, FUTEX_WAKE_PRIVATE, 1) = 1 <0.000018>
21:13:53.109853 futex(0x899240, FUTEX_WAIT_PRIVATE, 2, NULL) = 0 <0.006322>

$ head rabbit-thread.21862
21:13:48.663530 futex(0x1f2bce0, FUTEX_WAIT_PRIVATE, 4294967295, NULL)
  = 0 <4.446544>
21:13:53.110118 sched_yield() = 0 <0.000102>
21:13:53.110334 futex(0x1f856e8, FUTEX_WAIT_PRIVATE, 2, NULL) = 0 <0.006364>
21:13:53.116735 futex(0x1f856e8, FUTEX_WAKE_PRIVATE, 1) = 1 <0.000092>
21:13:53.117139 recvfrom(204, "

The first system call logged for 21860 occurs at 21:13:53. However, notice
the timestamp and 4.4 second execution time for the futex call in the other
thread (21862). All of the straces for the threads were started at the same
time (using the -ff option) and a number of other threads have a similar
call around 21:13:48.6635 as their first logged system call (within a few
milliseconds). This leads me to believe that whatever 21860 is doing during
this time, it's working in a single system call rather
than repeatedly calling in a loop.

2) Reproducing the problem in a controlled environment

This problem only seems to occur in our production environments, but not in
any test environments. The software and hardware in the two environments are
identical. We've used data recorded from our prod environment to try and
reproduce the problem in test (with higher load, even), but no luck so far.

3) Monitoring message rates through the web API

In the second attached screenshot, you can see a graph generated from data
pulled from the web API for this episode. Each trace in the graph represents
the incoming message rate for an individual exchange as reported by the API.
For example, the green line at the top shows the message rate dropping from
223 messages/sec to 32 messages per second. Not all the exchanges

Note that the sample rate for this query is every 5 seconds, and the
straight line between the points at 21:13:40 and 21:13:53 indicate that the
script pulling the data was unable to get a response from rabbit during this
time.

4) Restarting Rabbit

It's not clear that restarting rabbit has any effect on this problem.
Because it's so random, it's hard to tell if a rabbit restart makes things
better.


I've got more data on this if anyone wants to hear it. Thanks in advance for
your help.

Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111012/591902e5/attachment.htm>
-------------- next part --------------
Proto Recv-Q Send-Q Local Address               Foreign Address             State
Wed Oct 12 21:13:01 CDT 2011
Wed Oct 12 21:13:04 CDT 2011
Wed Oct 12 21:13:06 CDT 2011
Wed Oct 12 21:13:08 CDT 2011
Wed Oct 12 21:13:10 CDT 2011
Wed Oct 12 21:13:13 CDT 2011
tcp     1581      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
Wed Oct 12 21:13:15 CDT 2011
Wed Oct 12 21:13:17 CDT 2011
tcp      733      0 localhost.localdomain:amqp  localhost.localdomain:47756 ESTABLISHED 
Wed Oct 12 21:13:20 CDT 2011
tcp     1467      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
Wed Oct 12 21:13:22 CDT 2011
Wed Oct 12 21:13:24 CDT 2011
Wed Oct 12 21:13:26 CDT 2011
tcp     1053      0 localhost.localdomain:47508 localhost.localdomain:amqp  ESTABLISHED 
Wed Oct 12 21:13:29 CDT 2011
Wed Oct 12 21:13:32 CDT 2011
Wed Oct 12 21:13:34 CDT 2011
tcp      352      0 localhost.localdomain:47591 localhost.localdomain:amqp  ESTABLISHED 
Wed Oct 12 21:13:37 CDT 2011
Wed Oct 12 21:13:39 CDT 2011
Wed Oct 12 21:13:41 CDT 2011
Wed Oct 12 21:13:44 CDT 2011
tcp    15936      0 localhost.localdomain:amqp  localhost.localdomain:47736 ESTABLISHED 
tcp    21679      0 localhost.localdomain:amqp  localhost.localdomain:47761 ESTABLISHED 
tcp    74232      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp      600      0 localhost.localdomain:amqp  localhost.localdomain:47792 ESTABLISHED 
tcp     3640      0 localhost.localdomain:amqp  localhost.localdomain:47592 ESTABLISHED 
tcp    13376      0 localhost.localdomain:amqp  localhost.localdomain:47623 ESTABLISHED 
tcp     4235      0 localhost.localdomain:amqp  localhost.localdomain:47756 ESTABLISHED 
tcp      259      0 localhost.localdomain:amqp  localhost.localdomain:47459 ESTABLISHED 
tcp      954      0 localhost.localdomain:amqp  localhost.localdomain:47566 ESTABLISHED 
tcp    40460      0 localhost.localdomain:amqp  localhost.localdomain:47826 ESTABLISHED 
tcp    18722      0 localhost.localdomain:amqp  localhost.localdomai:bacnet ESTABLISHED 
tcp    32314      0 localhost.localdomain:amqp  localhost.localdomain:47655 ESTABLISHED 
tcp    10383      0 localhost.localdomain:amqp  localhost.localdomain:47878 ESTABLISHED 
tcp    10947      0 localhost.localdomain:amqp  localhost.localdomain:47573 ESTABLISHED 
tcp     5798      0 localhost.localdomain:amqp  localhost.localdomain:47850 ESTABLISHED 
tcp    14231      0 localhost.localdomain:amqp  localhost.localdomain:47872 ESTABLISHED 
tcp    13748      0 localhost.localdomain:amqp  localhost.localdomain:47814 ESTABLISHED 
tcp     2916      0 localhost.localdomain:amqp  localhost.localdomain:47626 ESTABLISHED 
tcp    65939      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
tcp     4018      0 localhost.localdomain:amqp  localhost.localdomain:47812 ESTABLISHED 
tcp    36104      0 localhost.localdomain:amqp  localhost.localdomain:47583 ESTABLISHED 
tcp     2656      0 localhost.localdomain:amqp  localhost.localdomain:47787 ESTABLISHED 
tcp    11890      0 localhost.localdomain:amqp  localhost.localdomain:47844 ESTABLISHED 
tcp     3441      0 localhost.localdomain:amqp  localhost.localdomain:47619 ESTABLISHED 
tcp    41800      0 localhost.localdomain:amqp  localhost.localdomain:47671 ESTABLISHED 
tcp      270      0 localhost.localdomain:amqp  localhost.localdomain:47638 ESTABLISHED 
tcp    46458      0 localhost.localdomain:amqp  localhost.localdomain:47871 ESTABLISHED 
tcp     5460      0 localhost.localdomain:amqp  localhost.localdomain:47784 ESTABLISHED 
tcp     2917      0 localhost.localdomain:amqp  localhost.localdomain:47480 ESTABLISHED 
tcp    20745      0 localhost.localdomain:amqp  localhost.localdomain:47767 ESTABLISHED 
tcp     7536      0 localhost.localdomain:amqp  localhost.localdomain:47837 ESTABLISHED 
Wed Oct 12 21:13:46 CDT 2011
tcp    28460      0 localhost.localdomain:amqp  localhost.localdomain:47736 ESTABLISHED 
tcp    48715      0 localhost.localdomain:amqp  localhost.localdomain:47761 ESTABLISHED 
tcp   209606      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp     2400      0 localhost.localdomain:amqp  localhost.localdomain:47792 ESTABLISHED 
tcp     7839      0 localhost.localdomain:amqp  localhost.localdomain:47592 ESTABLISHED 
tcp    29473      0 localhost.localdomain:amqp  localhost.localdomain:47623 ESTABLISHED 
tcp    11309      0 localhost.localdomain:amqp  localhost.localdomain:47756 ESTABLISHED 
tcp      518      0 localhost.localdomain:amqp  localhost.localdomain:47459 ESTABLISHED 
tcp     1908      0 localhost.localdomain:amqp  localhost.localdomain:47566 ESTABLISHED 
tcp   102870      0 localhost.localdomain:amqp  localhost.localdomain:47826 ESTABLISHED 
tcp    64155      0 localhost.localdomain:amqp  localhost.localdomai:bacnet ESTABLISHED 
tcp    84905      0 localhost.localdomain:amqp  localhost.localdomain:47655 ESTABLISHED 
tcp    25611      0 localhost.localdomain:amqp  localhost.localdomain:47878 ESTABLISHED 
tcp    25791      0 localhost.localdomain:amqp  localhost.localdomain:47573 ESTABLISHED 
tcp    12454      0 localhost.localdomain:amqp  localhost.localdomain:47850 ESTABLISHED 
tcp    43832      0 localhost.localdomain:amqp  localhost.localdomain:47872 ESTABLISHED 
tcp    41396      0 localhost.localdomain:amqp  localhost.localdomain:47814 ESTABLISHED 
tcp     7078      0 localhost.localdomain:amqp  localhost.localdomain:47626 ESTABLISHED 
tcp   170929      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
tcp     9673      0 localhost.localdomain:amqp  localhost.localdomain:47812 ESTABLISHED 
tcp    88569      0 localhost.localdomain:amqp  localhost.localdomain:47583 ESTABLISHED 
tcp     5251      0 localhost.localdomain:amqp  localhost.localdomain:47787 ESTABLISHED 
tcp    41544      0 localhost.localdomain:amqp  localhost.localdomain:47844 ESTABLISHED 
tcp    10038      0 localhost.localdomain:amqp  localhost.localdomain:47619 ESTABLISHED 
tcp    91198      0 localhost.localdomain:amqp  localhost.localdomain:47671 ESTABLISHED 
tcp      809      0 localhost.localdomain:amqp  localhost.localdomain:47638 ESTABLISHED 
tcp   144476      0 localhost.localdomain:amqp  localhost.localdomain:47871 ESTABLISHED 
tcp     7878      0 localhost.localdomain:amqp  localhost.localdomain:47784 ESTABLISHED 
tcp     5829      0 localhost.localdomain:amqp  localhost.localdomain:47480 ESTABLISHED 
tcp    64986      0 localhost.localdomain:amqp  localhost.localdomain:47767 ESTABLISHED 
tcp    10902      0 localhost.localdomain:amqp  localhost.localdomain:47837 ESTABLISHED 
Wed Oct 12 21:13:49 CDT 2011
tcp    44641      0 localhost.localdomain:amqp  localhost.localdomain:47736 ESTABLISHED 
tcp    91471      0 localhost.localdomain:amqp  localhost.localdomain:47761 ESTABLISHED 
tcp   427156      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp     3600      0 localhost.localdomain:amqp  localhost.localdomain:47792 ESTABLISHED 
tcp    14675      0 localhost.localdomain:amqp  localhost.localdomain:47592 ESTABLISHED 
tcp    45688      0 localhost.localdomain:amqp  localhost.localdomain:47623 ESTABLISHED 
tcp    17949      0 localhost.localdomain:amqp  localhost.localdomain:47756 ESTABLISHED 
tcp      777      0 localhost.localdomain:amqp  localhost.localdomain:47459 ESTABLISHED 
tcp     3339      0 localhost.localdomain:amqp  localhost.localdomain:47566 ESTABLISHED 
tcp   176459      0 localhost.localdomain:amqp  localhost.localdomain:47826 ESTABLISHED 
tcp   116508      0 localhost.localdomain:amqp  localhost.localdomai:bacnet ESTABLISHED 
tcp   158258      0 localhost.localdomain:amqp  localhost.localdomain:47655 ESTABLISHED 
tcp    45869      0 localhost.localdomain:amqp  localhost.localdomain:47878 ESTABLISHED 
tcp    49992      0 localhost.localdomain:amqp  localhost.localdomain:47573 ESTABLISHED 
tcp    20884      0 localhost.localdomain:amqp  localhost.localdomain:47850 ESTABLISHED 
tcp    69021      0 localhost.localdomain:amqp  localhost.localdomain:47872 ESTABLISHED 
tcp    71917      0 localhost.localdomain:amqp  localhost.localdomain:47814 ESTABLISHED 
tcp     7795      0 localhost.localdomain:amqp  localhost.localdomain:47626 ESTABLISHED 
tcp   363634      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
tcp    18200      0 localhost.localdomain:amqp  localhost.localdomain:47812 ESTABLISHED 
tcp   182609      0 localhost.localdomain:amqp  localhost.localdomain:47583 ESTABLISHED 
tcp    19031      0 localhost.localdomain:amqp  localhost.localdomain:47787 ESTABLISHED 
tcp    62316      0 localhost.localdomain:amqp  localhost.localdomain:47844 ESTABLISHED 
tcp    13194      0 localhost.localdomain:amqp  localhost.localdomain:47619 ESTABLISHED 
tcp   156987      0 localhost.localdomain:amqp  localhost.localdomain:47671 ESTABLISHED 
tcp     1348      0 localhost.localdomain:amqp  localhost.localdomain:47638 ESTABLISHED 
tcp   283881      0 localhost.localdomain:amqp  localhost.localdomain:47871 ESTABLISHED 
tcp     9492      0 localhost.localdomain:amqp  localhost.localdomain:47784 ESTABLISHED 
tcp    10333      0 localhost.localdomain:amqp  localhost.localdomain:47480 ESTABLISHED 
tcp   105166      0 localhost.localdomain:amqp  localhost.localdomain:47767 ESTABLISHED 
tcp      346      0 localhost.localdomain:amqp  localhost.localdomain:47706 ESTABLISHED 
tcp    17632      0 localhost.localdomain:amqp  localhost.localdomain:47837 ESTABLISHED 
Wed Oct 12 21:13:51 CDT 2011
tcp    57163      0 localhost.localdomain:amqp  localhost.localdomain:47736 ESTABLISHED 
tcp   120173      0 localhost.localdomain:amqp  localhost.localdomain:47761 ESTABLISHED 
tcp   604265      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp     4800      0 localhost.localdomain:amqp  localhost.localdomain:47792 ESTABLISHED 
tcp    19344      0 localhost.localdomain:amqp  localhost.localdomain:47592 ESTABLISHED 
tcp    64057      0 localhost.localdomain:amqp  localhost.localdomain:47623 ESTABLISHED 
tcp    25192      0 localhost.localdomain:amqp  localhost.localdomain:47756 ESTABLISHED 
tcp     1036      0 localhost.localdomain:amqp  localhost.localdomain:47459 ESTABLISHED 
tcp     4293      0 localhost.localdomain:amqp  localhost.localdomain:47566 ESTABLISHED 
tcp   242307      0 localhost.localdomain:amqp  localhost.localdomain:47826 ESTABLISHED 
tcp   159940      0 localhost.localdomain:amqp  localhost.localdomai:bacnet ESTABLISHED 
tcp   225302      0 localhost.localdomain:amqp  localhost.localdomain:47655 ESTABLISHED 
tcp    60735      0 localhost.localdomain:amqp  localhost.localdomain:47878 ESTABLISHED 
tcp    68294      0 localhost.localdomain:amqp  localhost.localdomain:47573 ESTABLISHED 
tcp    26739      0 localhost.localdomain:amqp  localhost.localdomain:47850 ESTABLISHED 
tcp    98622      0 localhost.localdomain:amqp  localhost.localdomain:47872 ESTABLISHED 
tcp    96540      0 localhost.localdomain:amqp  localhost.localdomain:47814 ESTABLISHED 
tcp    10084      0 localhost.localdomain:amqp  localhost.localdomain:47626 ESTABLISHED 
tcp   516585      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
tcp    23854      0 localhost.localdomain:amqp  localhost.localdomain:47812 ESTABLISHED 
tcp   257739      0 localhost.localdomain:amqp  localhost.localdomain:47583 ESTABLISHED 
tcp    26798      0 localhost.localdomain:amqp  localhost.localdomain:47787 ESTABLISHED 
tcp    93474      0 localhost.localdomain:amqp  localhost.localdomain:47844 ESTABLISHED 
tcp    16635      0 localhost.localdomain:amqp  localhost.localdomain:47619 ESTABLISHED 
tcp   210562      0 localhost.localdomain:amqp  localhost.localdomain:47671 ESTABLISHED 
tcp     1887      0 localhost.localdomain:amqp  localhost.localdomain:47638 ESTABLISHED 
tcp   402725      0 localhost.localdomain:amqp  localhost.localdomain:47871 ESTABLISHED 
tcp    11396      0 localhost.localdomain:amqp  localhost.localdomain:47784 ESTABLISHED 
tcp    13210      0 localhost.localdomain:amqp  localhost.localdomain:47480 ESTABLISHED 
tcp   149404      0 localhost.localdomain:amqp  localhost.localdomain:47767 ESTABLISHED 
tcp      698      0 localhost.localdomain:amqp  localhost.localdomain:47706 ESTABLISHED 
tcp    21076      0 localhost.localdomain:amqp  localhost.localdomain:47837 ESTABLISHED 
Wed Oct 12 21:13:53 CDT 2011
tcp    33430      0 localhost.localdomain:amqp  localhost.localdomain:47736 ESTABLISHED 
tcp   113449      0 localhost.localdomain:amqp  localhost.localdomain:47761 ESTABLISHED 
tcp   862376      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp    40638      0 localhost.localdomain:amqp  localhost.localdomain:47623 ESTABLISHED 
tcp   266511      0 localhost.localdomain:amqp  localhost.localdomain:47826 ESTABLISHED 
tcp   168676      0 localhost.localdomain:amqp  localhost.localdomai:bacnet ESTABLISHED 
tcp   380701      0 localhost.localdomain:amqp  localhost.localdomain:47655 ESTABLISHED 
tcp    36208      0 localhost.localdomain:amqp  localhost.localdomain:47878 ESTABLISHED 
tcp   264931      0 localhost.localdomain:amqp  localhost.localdomain:47573 ESTABLISHED 
tcp    78198      0 localhost.localdomain:amqp  localhost.localdomain:47872 ESTABLISHED 
tcp    80677      0 localhost.localdomain:amqp  localhost.localdomain:47814 ESTABLISHED 
tcp   758881      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
tcp   411550      0 localhost.localdomain:amqp  localhost.localdomain:47583 ESTABLISHED 
tcp    71906      0 localhost.localdomain:amqp  localhost.localdomain:47844 ESTABLISHED 
tcp   212783      0 localhost.localdomain:amqp  localhost.localdomain:47671 ESTABLISHED 
tcp   623353      0 localhost.localdomain:amqp  localhost.localdomain:47871 ESTABLISHED 
tcp   139246      0 localhost.localdomain:amqp  localhost.localdomain:47767 ESTABLISHED 
Wed Oct 12 21:13:55 CDT 2011
tcp   879697      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp    76291      0 localhost.localdomain:amqp  localhost.localdomain:47826 ESTABLISHED 
tcp      289      0 localhost.localdomain:amqp  localhost.localdomai:bacnet ESTABLISHED 
tcp   824278      0 localhost.localdomain:amqp  localhost.localdomain:47655 ESTABLISHED 
tcp   224023      0 localhost.localdomain:amqp  localhost.localdomain:47573 ESTABLISHED 
tcp   1187576      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
tcp   529432      0 localhost.localdomain:amqp  localhost.localdomain:47583 ESTABLISHED 
tcp     1048    323 localhost.localdomain:amqp  localhost.localdomain:47676 ESTABLISHED 
tcp     6040      0 localhost.localdomain:amqp  localhost.localdomain:47671 ESTABLISHED 
tcp      270      0 localhost.localdomain:amqp  localhost.localdomain:47638 ESTABLISHED 
tcp   1080443      0 localhost.localdomain:amqp  localhost.localdomain:47871 ESTABLISHED 
Wed Oct 12 21:13:58 CDT 2011
tcp   952681      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp   483169      0 localhost.localdomain:amqp  localhost.localdomain:47655 ESTABLISHED 
tcp      441      0 localhost.localdomain:amqp  localhost.localdomain:47573 ESTABLISHED 
tcp   1368987      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
tcp   376058      0 localhost.localdomain:amqp  localhost.localdomain:47583 ESTABLISHED 
tcp   1089092      0 localhost.localdomain:amqp  localhost.localdomain:47871 ESTABLISHED 
Wed Oct 12 21:14:00 CDT 2011
tcp   953637      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp      733      0 localhost.localdomain:amqp  localhost.localdomain:47756 ESTABLISHED 
tcp   238139      0 localhost.localdomain:amqp  localhost.localdomain:47655 ESTABLISHED 
tcp     2350      0 localhost.localdomain:amqp  localhost.localdomain:47573 ESTABLISHED 
tcp   1268676      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
tcp   175759      0 localhost.localdomain:amqp  localhost.localdomain:47583 ESTABLISHED 
tcp   755472      0 localhost.localdomain:amqp  localhost.localdomain:47871 ESTABLISHED 
Wed Oct 12 21:14:02 CDT 2011
tcp   950537      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp      342      0 localhost.localdomain:amqp  localhost.localdomain:47826 ESTABLISHED 
tcp     1713      0 localhost.localdomain:amqp  localhost.localdomain:47655 ESTABLISHED 
tcp     2083      0 localhost.localdomain:amqp  localhost.localdomain:47573 ESTABLISHED 
tcp   930418      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
tcp      314      0 localhost.localdomain:amqp  localhost.localdomain:47583 ESTABLISHED 
tcp   538627      0 localhost.localdomain:amqp  localhost.localdomain:47871 ESTABLISHED 
Wed Oct 12 21:14:05 CDT 2011
tcp   965601      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp   802603      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
tcp   229943      0 localhost.localdomain:amqp  localhost.localdomain:47871 ESTABLISHED 
Wed Oct 12 21:14:07 CDT 2011
tcp   957953      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp      130      0 localhost.localdomain:amqp  localhost.localdomain:47655 ESTABLISHED 
tcp     2383      0 localhost.localdomain:amqp  localhost.localdomain:47872 ESTABLISHED 
tcp   470143      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
Wed Oct 12 21:14:09 CDT 2011
tcp   951273      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp     1631      0 localhost.localdomain:amqp  localhost.localdomai:bacnet ESTABLISHED 
tcp    61814      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
Wed Oct 12 21:14:11 CDT 2011
tcp    11763      0 localhost.localdomain:amqp  localhost.localdomain:47736 ESTABLISHED 
tcp    33022      0 localhost.localdomain:amqp  localhost.localdomain:47761 ESTABLISHED 
tcp   970902      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp     1200      0 localhost.localdomain:amqp  localhost.localdomain:47792 ESTABLISHED 
tcp     4667      0 localhost.localdomain:amqp  localhost.localdomain:47592 ESTABLISHED 
tcp    15576      0 localhost.localdomain:amqp  localhost.localdomain:47623 ESTABLISHED 
tcp     6331      0 localhost.localdomain:amqp  localhost.localdomain:47756 ESTABLISHED 
tcp      259      0 localhost.localdomain:amqp  localhost.localdomain:47459 ESTABLISHED 
tcp     1431      0 localhost.localdomain:amqp  localhost.localdomain:47566 ESTABLISHED 
tcp    61176      0 localhost.localdomain:amqp  localhost.localdomain:47826 ESTABLISHED 
tcp    39199      0 localhost.localdomain:amqp  localhost.localdomai:bacnet ESTABLISHED 
tcp    45894      0 localhost.localdomain:amqp  localhost.localdomain:47655 ESTABLISHED 
tcp    11861      0 localhost.localdomain:amqp  localhost.localdomain:47878 ESTABLISHED 
tcp    23041      0 localhost.localdomain:amqp  localhost.localdomain:47573 ESTABLISHED 
tcp     6448      0 localhost.localdomain:amqp  localhost.localdomain:47850 ESTABLISHED 
tcp    21916      0 localhost.localdomain:amqp  localhost.localdomain:47872 ESTABLISHED 
tcp    24337      0 localhost.localdomain:amqp  localhost.localdomain:47814 ESTABLISHED 
tcp     3553      0 localhost.localdomain:amqp  localhost.localdomain:47626 ESTABLISHED 
tcp   109572      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
tcp     6152      0 localhost.localdomain:amqp  localhost.localdomain:47812 ESTABLISHED 
tcp    48431      0 localhost.localdomain:amqp  localhost.localdomain:47583 ESTABLISHED 
tcp     9145      0 localhost.localdomain:amqp  localhost.localdomain:47787 ESTABLISHED 
tcp    20772      0 localhost.localdomain:amqp  localhost.localdomain:47844 ESTABLISHED 
tcp     3156      0 localhost.localdomain:amqp  localhost.localdomain:47619 ESTABLISHED 
tcp    51892      0 localhost.localdomain:amqp  localhost.localdomain:47671 ESTABLISHED 
tcp      539      0 localhost.localdomain:amqp  localhost.localdomain:47638 ESTABLISHED 
tcp    70331      0 localhost.localdomain:amqp  localhost.localdomain:47871 ESTABLISHED 
tcp      261      0 localhost.localdomain:amqp  localhost.localdomain:47784 ESTABLISHED 
tcp     2893      0 localhost.localdomain:amqp  localhost.localdomain:47480 ESTABLISHED 
tcp    38389      0 localhost.localdomain:amqp  localhost.localdomain:47767 ESTABLISHED 
tcp      352      0 localhost.localdomain:amqp  localhost.localdomain:47706 ESTABLISHED 
tcp     3436      0 localhost.localdomain:amqp  localhost.localdomain:47837 ESTABLISHED 
Wed Oct 12 21:14:14 CDT 2011
tcp    25682      0 localhost.localdomain:amqp  localhost.localdomain:47736 ESTABLISHED 
tcp    74742      0 localhost.localdomain:amqp  localhost.localdomain:47761 ESTABLISHED 
tcp   970902      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp     2400      0 localhost.localdomain:amqp  localhost.localdomain:47792 ESTABLISHED 
tcp    10642      0 localhost.localdomain:amqp  localhost.localdomain:47592 ESTABLISHED 
tcp    31060      0 localhost.localdomain:amqp  localhost.localdomain:47623 ESTABLISHED 
tcp    12133      0 localhost.localdomain:amqp  localhost.localdomain:47756 ESTABLISHED 
tcp      518      0 localhost.localdomain:amqp  localhost.localdomain:47459 ESTABLISHED 
tcp     2385      0 localhost.localdomain:amqp  localhost.localdomain:47566 ESTABLISHED 
tcp   120723      0 localhost.localdomain:amqp  localhost.localdomain:47826 ESTABLISHED 
tcp    77975      0 localhost.localdomain:amqp  localhost.localdomai:bacnet ESTABLISHED 
tcp    98046      0 localhost.localdomain:amqp  localhost.localdomain:47655 ESTABLISHED 
tcp    29099      0 localhost.localdomain:amqp  localhost.localdomain:47878 ESTABLISHED 
tcp    40974      0 localhost.localdomain:amqp  localhost.localdomain:47573 ESTABLISHED 
tcp    16055      0 localhost.localdomain:amqp  localhost.localdomain:47850 ESTABLISHED 
tcp    47105      0 localhost.localdomain:amqp  localhost.localdomain:47872 ESTABLISHED 
tcp    48952      0 localhost.localdomain:amqp  localhost.localdomain:47814 ESTABLISHED 
tcp     4237      0 localhost.localdomain:amqp  localhost.localdomain:47626 ESTABLISHED 
tcp   246403      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
tcp    14179      0 localhost.localdomain:amqp  localhost.localdomain:47812 ESTABLISHED 
tcp   126854      0 localhost.localdomain:amqp  localhost.localdomain:47583 ESTABLISHED 
tcp    11803      0 localhost.localdomain:amqp  localhost.localdomain:47787 ESTABLISHED 
tcp    41544      0 localhost.localdomain:amqp  localhost.localdomain:47844 ESTABLISHED 
tcp     6914      0 localhost.localdomain:amqp  localhost.localdomain:47619 ESTABLISHED 
tcp   108956      0 localhost.localdomain:amqp  localhost.localdomain:47671 ESTABLISHED 
tcp      808      0 localhost.localdomain:amqp  localhost.localdomain:47638 ESTABLISHED 
tcp   174661      0 localhost.localdomain:amqp  localhost.localdomain:47871 ESTABLISHED 
tcp     5721      0 localhost.localdomain:amqp  localhost.localdomain:47784 ESTABLISHED 
tcp     5787      0 localhost.localdomain:amqp  localhost.localdomain:47480 ESTABLISHED 
tcp    71406      0 localhost.localdomain:amqp  localhost.localdomain:47767 ESTABLISHED 
tcp      352      0 localhost.localdomain:amqp  localhost.localdomain:47706 ESTABLISHED 
tcp    10185      0 localhost.localdomain:amqp  localhost.localdomain:47837 ESTABLISHED 
Wed Oct 12 21:14:16 CDT 2011
tcp     1170      0 localhost.localdomain:amqp  localhost.localdomain:47578 ESTABLISHED 
tcp    37767      0 localhost.localdomain:amqp  localhost.localdomain:47736 ESTABLISHED 
tcp   117706      0 localhost.localdomain:amqp  localhost.localdomain:47761 ESTABLISHED 
tcp   970902      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp     3900      0 localhost.localdomain:amqp  localhost.localdomain:47792 ESTABLISHED 
tcp    14838      0 localhost.localdomain:amqp  localhost.localdomain:47592 ESTABLISHED 
tcp    46320      0 localhost.localdomain:amqp  localhost.localdomain:47623 ESTABLISHED 
tcp    18206      0 localhost.localdomain:amqp  localhost.localdomain:47756 ESTABLISHED 
tcp     1036      0 localhost.localdomain:amqp  localhost.localdomain:47459 ESTABLISHED 
tcp     3339      0 localhost.localdomain:amqp  localhost.localdomain:47566 ESTABLISHED 
tcp   186578      0 localhost.localdomain:amqp  localhost.localdomain:47826 ESTABLISHED 
tcp   121427      0 localhost.localdomain:amqp  localhost.localdomai:bacnet ESTABLISHED 
tcp   160900      0 localhost.localdomain:amqp  localhost.localdomain:47655 ESTABLISHED 
tcp    46592      0 localhost.localdomain:amqp  localhost.localdomain:47878 ESTABLISHED 
tcp    58943      0 localhost.localdomain:amqp  localhost.localdomain:47573 ESTABLISHED 
tcp    22513      0 localhost.localdomain:amqp  localhost.localdomain:47850 ESTABLISHED 
tcp    72864      0 localhost.localdomain:amqp  localhost.localdomain:47872 ESTABLISHED 
tcp    73951      0 localhost.localdomain:amqp  localhost.localdomain:47814 ESTABLISHED 
tcp     5667      0 localhost.localdomain:amqp  localhost.localdomain:47626 ESTABLISHED 
tcp   395569      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
tcp    19831      0 localhost.localdomain:amqp  localhost.localdomain:47812 ESTABLISHED 
tcp   198582      0 localhost.localdomain:amqp  localhost.localdomain:47583 ESTABLISHED 
tcp    14401      0 localhost.localdomain:amqp  localhost.localdomain:47787 ESTABLISHED 
tcp    72702      0 localhost.localdomain:amqp  localhost.localdomain:47844 ESTABLISHED 
tcp    13511      0 localhost.localdomain:amqp  localhost.localdomain:47619 ESTABLISHED 
tcp   163556      0 localhost.localdomain:amqp  localhost.localdomain:47671 ESTABLISHED 
tcp     1347      0 localhost.localdomain:amqp  localhost.localdomain:47638 ESTABLISHED 
tcp   296571      0 localhost.localdomain:amqp  localhost.localdomain:47871 ESTABLISHED 
tcp     7868      0 localhost.localdomain:amqp  localhost.localdomain:47784 ESTABLISHED 
tcp     9428      0 localhost.localdomain:amqp  localhost.localdomain:47480 ESTABLISHED 
tcp   105453      0 localhost.localdomain:amqp  localhost.localdomain:47767 ESTABLISHED 
tcp      352      0 localhost.localdomain:amqp  localhost.localdomain:47706 ESTABLISHED 
tcp    14360      0 localhost.localdomain:amqp  localhost.localdomain:47837 ESTABLISHED 
Wed Oct 12 21:14:18 CDT 2011
tcp     1170      0 localhost.localdomain:amqp  localhost.localdomain:47578 ESTABLISHED 
tcp    51258      0 localhost.localdomain:amqp  localhost.localdomain:47736 ESTABLISHED 
tcp   153314      0 localhost.localdomain:amqp  localhost.localdomain:47761 ESTABLISHED 
tcp   970902      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp     5100      0 localhost.localdomain:amqp  localhost.localdomain:47792 ESTABLISHED 
tcp    19811      0 localhost.localdomain:amqp  localhost.localdomain:47592 ESTABLISHED 
tcp    64777      0 localhost.localdomain:amqp  localhost.localdomain:47623 ESTABLISHED 
tcp    23957      0 localhost.localdomain:amqp  localhost.localdomain:47756 ESTABLISHED 
tcp     1295      0 localhost.localdomain:amqp  localhost.localdomain:47459 ESTABLISHED 
tcp     4293      0 localhost.localdomain:amqp  localhost.localdomain:47566 ESTABLISHED 
tcp   243685      0 localhost.localdomain:amqp  localhost.localdomain:47826 ESTABLISHED 
tcp   169201      0 localhost.localdomain:amqp  localhost.localdomai:bacnet ESTABLISHED 
tcp   231517      0 localhost.localdomain:amqp  localhost.localdomain:47655 ESTABLISHED 
tcp    62584      0 localhost.localdomain:amqp  localhost.localdomain:47878 ESTABLISHED 
tcp    76915      0 localhost.localdomain:amqp  localhost.localdomain:47573 ESTABLISHED 
tcp    29885      0 localhost.localdomain:amqp  localhost.localdomain:47850 ESTABLISHED 
tcp    98622      0 localhost.localdomain:amqp  localhost.localdomain:47872 ESTABLISHED 
tcp   100245      0 localhost.localdomain:amqp  localhost.localdomain:47814 ESTABLISHED 
tcp     5667      0 localhost.localdomain:amqp  localhost.localdomain:47626 ESTABLISHED 
tcp   544151      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
tcp    27355      0 localhost.localdomain:amqp  localhost.localdomain:47812 ESTABLISHED 
tcp   271094      0 localhost.localdomain:amqp  localhost.localdomain:47583 ESTABLISHED 
tcp    15669      0 localhost.localdomain:amqp  localhost.localdomain:47787 ESTABLISHED 
tcp    93474      0 localhost.localdomain:amqp  localhost.localdomain:47844 ESTABLISHED 
tcp    16350      0 localhost.localdomain:amqp  localhost.localdomain:47619 ESTABLISHED 
tcp   215103      0 localhost.localdomain:amqp  localhost.localdomain:47671 ESTABLISHED 
tcp     1886      0 localhost.localdomain:amqp  localhost.localdomain:47638 ESTABLISHED 
tcp   419203      0 localhost.localdomain:amqp  localhost.localdomain:47871 ESTABLISHED 
tcp     9494      0 localhost.localdomain:amqp  localhost.localdomain:47784 ESTABLISHED 
tcp    12871      0 localhost.localdomain:amqp  localhost.localdomain:47480 ESTABLISHED 
tcp   149655      0 localhost.localdomain:amqp  localhost.localdomain:47767 ESTABLISHED 
tcp      698      0 localhost.localdomain:amqp  localhost.localdomain:47706 ESTABLISHED 
tcp    22846      0 localhost.localdomain:amqp  localhost.localdomain:47837 ESTABLISHED 
Wed Oct 12 21:14:20 CDT 2011
tcp    17883      0 localhost.localdomain:amqp  localhost.localdomain:47736 ESTABLISHED 
tcp   177245      0 localhost.localdomain:amqp  localhost.localdomain:47761 ESTABLISHED 
tcp   958410      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp    36263      0 localhost.localdomain:amqp  localhost.localdomain:47623 ESTABLISHED 
tcp   259518      0 localhost.localdomain:amqp  localhost.localdomain:47826 ESTABLISHED 
tcp   155674      0 localhost.localdomain:amqp  localhost.localdomai:bacnet ESTABLISHED 
tcp   425744      0 localhost.localdomain:amqp  localhost.localdomain:47655 ESTABLISHED 
tcp    26357      0 localhost.localdomain:amqp  localhost.localdomain:47878 ESTABLISHED 
tcp   256227      0 localhost.localdomain:amqp  localhost.localdomain:47573 ESTABLISHED 
tcp    70898      0 localhost.localdomain:amqp  localhost.localdomain:47872 ESTABLISHED 
tcp    73109      0 localhost.localdomain:amqp  localhost.localdomain:47814 ESTABLISHED 
tcp   761979      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
tcp   473563      0 localhost.localdomain:amqp  localhost.localdomain:47583 ESTABLISHED 
tcp      530    577 localhost.localdomain:amqp  localhost.localdomain:47676 ESTABLISHED 
tcp    61686      0 localhost.localdomain:amqp  localhost.localdomain:47844 ESTABLISHED 
tcp   214324      0 localhost.localdomain:amqp  localhost.localdomain:47671 ESTABLISHED 
tcp   691459      0 localhost.localdomain:amqp  localhost.localdomain:47871 ESTABLISHED 
tcp   121978      0 localhost.localdomain:amqp  localhost.localdomain:47767 ESTABLISHED 
Wed Oct 12 21:14:23 CDT 2011
tcp   996686      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp    41877      0 localhost.localdomain:amqp  localhost.localdomain:47826 ESTABLISHED 
tcp   823505      0 localhost.localdomain:amqp  localhost.localdomain:47655 ESTABLISHED 
tcp   216406      0 localhost.localdomain:amqp  localhost.localdomain:47573 ESTABLISHED 
tcp   1212223      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
tcp   838518      0 localhost.localdomain:amqp  localhost.localdomain:47583 ESTABLISHED 
tcp   1109464      0 localhost.localdomain:amqp  localhost.localdomain:47871 ESTABLISHED 
Wed Oct 12 21:14:25 CDT 2011
tcp   1030094      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp   596756      0 localhost.localdomain:amqp  localhost.localdomain:47655 ESTABLISHED 
tcp    32166      0 localhost.localdomain:amqp  localhost.localdomain:47573 ESTABLISHED 
tcp   1427227      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
tcp   638147      0 localhost.localdomain:amqp  localhost.localdomain:47583 ESTABLISHED 
tcp   1260456      0 localhost.localdomain:amqp  localhost.localdomain:47871 ESTABLISHED 
Wed Oct 12 21:14:27 CDT 2011
tcp   1069334      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp   327272      0 localhost.localdomain:amqp  localhost.localdomain:47655 ESTABLISHED 
tcp     1120      0 localhost.localdomain:amqp  localhost.localdomain:47878 ESTABLISHED 
tcp   1405311      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
tcp   260070      0 localhost.localdomain:amqp  localhost.localdomain:47583 ESTABLISHED 
tcp   954564      0 localhost.localdomain:amqp  localhost.localdomain:47871 ESTABLISHED 
Wed Oct 12 21:14:29 CDT 2011
tcp   1067210      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp    98646      0 localhost.localdomain:amqp  localhost.localdomain:47655 ESTABLISHED 
tcp   1105244      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
tcp    50081      0 localhost.localdomain:amqp  localhost.localdomain:47583 ESTABLISHED 
tcp   695523      0 localhost.localdomain:amqp  localhost.localdomain:47871 ESTABLISHED 
Wed Oct 12 21:14:32 CDT 2011
tcp   1069462      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp   912047      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
tcp     1461      0 localhost.localdomain:amqp  localhost.localdomain:47583 ESTABLISHED 
tcp   633875      0 localhost.localdomain:amqp  localhost.localdomain:47871 ESTABLISHED 
Wed Oct 12 21:14:34 CDT 2011
tcp   1067010      0 localhost.localdomain:amqp  localhost.localdomain:47506 ESTABLISHED 
tcp      320      0 localhost.localdomain:amqp  localhost.localdomain:47623 ESTABLISHED 
tcp      259      0 localhost.localdomain:amqp  localhost.localdomain:47459 ESTABLISHED 
tcp      918      0 localhost.localdomain:amqp  localhost.localdomain:47826 ESTABLISHED 
tcp      450      0 localhost.localdomain:amqp  localhost.localdomain:47655 ESTABLISHED 
tcp      313      0 localhost.localdomain:amqp  localhost.localdomain:47573 ESTABLISHED 
tcp     3843      0 localhost.localdomain:amqp  localhost.localdomain:47872 ESTABLISHED 
tcp   919325      0 localhost.localdomain:amqp  localhost.localdomain:47628 ESTABLISHED 
tcp   664134      0 localhost.localdomain:47649 localhost.localdomain:amqp  ESTABLISHED 
tcp   606917      0 localhost.localdomain:47508 localhost.localdomain:amqp  ESTABLISHED 
Wed Oct 12 21:14:36 CDT 2011
Wed Oct 12 21:14:38 CDT 2011
Wed Oct 12 21:14:41 CDT 2011
tcp     1997      0 localhost.localdomain:47877 localhost.localdomain:amqp  ESTABLISHED 
tcp      307      0 localhost.localdomain:47766 localhost.localdomain:amqp  ESTABLISHED 
Wed Oct 12 21:14:43 CDT 2011
Wed Oct 12 21:14:45 CDT 2011
Wed Oct 12 21:14:47 CDT 2011
Wed Oct 12 21:14:50 CDT 2011
-------------- next part --------------
A non-text attachment was scrubbed...
Name: incoming-messages-per-exchange.png
Type: image/png
Size: 266627 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111012/591902e5/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rabbit-threads-top.png
Type: image/png
Size: 74431 bytes
Desc: not available
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111012/591902e5/attachment-0001.png>


More information about the rabbitmq-discuss mailing list