[rabbitmq-discuss] rabbitmq-discuss Digest, Vol 57, Issue 30

Wed Feb 29 13:00:51 GMT 2012

----- Original Message -----
From: rabbitmq-discuss-request at lists.rabbitmq.com [mailto:rabbitmq-discuss-request at lists.rabbitmq.com]
Sent: Wednesday, February 29, 2012 07:00 AM
To: rabbitmq-discuss at lists.rabbitmq.com <rabbitmq-discuss at lists.rabbitmq.com>
Subject: rabbitmq-discuss Digest, Vol 57, Issue 30

Send rabbitmq-discuss mailing list submissions to
        rabbitmq-discuss at lists.rabbitmq.com

To subscribe or unsubscribe via the World Wide Web, visit
        https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
or, via email, send a message with subject or body 'help' to
        rabbitmq-discuss-request at lists.rabbitmq.com

You can reach the person managing the list at
        rabbitmq-discuss-owner at lists.rabbitmq.com

When replying, please edit your Subject line so it is more specific
than "Re: Contents of rabbitmq-discuss digest..."

Today's Topics:

   1. Re: rabbitmq-discuss Digest, Vol 57, Issue 28 (Jose Jimenez Ortiz)
   2. |Spam| ACCIDENTES DEL TRABAJO "RESPONSABILIDAD CIVIL Y PENAL"
      (CONFERENCIA)
   3. Re: Apache MINA Integration (Gary Russell)
   4. Query a queue status from the client (Christopher Lefevre)
   5. Re: Regd: Spring AMQP versus Java Client Library (Gary Russell)
   6. Re: Query a queue status from the client (Christopher Lefevre)
   7. Re: RabbitMQ and MySQL (Jerry Kuch)
   8. Re: Design Help? (Jerry Kuch)
   9. Re: Design Help? (James Aimonetti)
  10. Re: RabbitMQ and MySQL (PepMad)
  11. Re: RabbitMQ and MySQL (Jerry Kuch)
  12. [help] [beginner] testing with pika: some questions and
      complaints (Alistair Bayley)
  13. (no subject) (Ashutosh Mathur)
  14. Re: rabbitmq-discuss Digest, Vol 57, Issue 28 (Simon MacMullen)
  15. Re: [help] [beginner] testing with pika: some questions and
      complaints (Simon MacMullen)
  16. Re: (no subject) (Simon MacMullen)

----------------------------------------------------------------------

Message: 1
Date: Tue, 28 Feb 2012 08:56:00 -0600
From: Jose Jimenez Ortiz <j-jimenez at finamex.com.mx>
Subject: Re: [rabbitmq-discuss] rabbitmq-discuss Digest, Vol 57, Issue
        28
To: "rabbitmq-discuss at lists.rabbitmq.com"
        <rabbitmq-discuss at lists.rabbitmq.com>
Message-ID:
        <AFFA651B612BDE4EB570E34F0571834D0E170F8B17 at mail.vfinamex.local>
Content-Type: text/plain; charset="us-ascii"

Hi all,

I installed rabbitmq 2.7.1 and enable Rabbitmq_Management plugin but in the explorer I don't see anything on OVERVIEW panel.

Any idea?

Regards,

-----Mensaje original-----
De: rabbitmq-discuss-bounces at lists.rabbitmq.com [mailto:rabbitmq-discuss-bounces at lists.rabbitmq.com] En nombre de rabbitmq-discuss-request at lists.rabbitmq.com
Enviado el: Lunes, 27 de Febrero de 2012 06:00
Para: rabbitmq-discuss at lists.rabbitmq.com
Asunto: rabbitmq-discuss Digest, Vol 57, Issue 28

Send rabbitmq-discuss mailing list submissions to
        rabbitmq-discuss at lists.rabbitmq.com

To subscribe or unsubscribe via the World Wide Web, visit
        https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
or, via email, send a message with subject or body 'help' to
        rabbitmq-discuss-request at lists.rabbitmq.com

You can reach the person managing the list at
        rabbitmq-discuss-owner at lists.rabbitmq.com

When replying, please edit your Subject line so it is more specific than "Re: Contents of rabbitmq-discuss digest..."

Today's Topics:

   1. (no subject) (jay nanavati)
   2. Apache MINA Integration (Uday Subbarayan)
   3. Problem with using rabbitmqctl on EC2 (Codevally)
   4. Re: [help] [beginner] server stops sending messages; publish
      (in transaction) hangs on commit (Alistair Bayley)
   5. dead connection lingers around? (stone)
   6. Re: Problems with STOMP & access control (Steve Powell)
   7. Re: Unexpected behaviour with STOMP & NACK (Steve Powell)
   8. Re: Consumers and exceptions (Jon Hill)
   9. Re: Problems with STOMP & access control (Lionel Cons)
  10. Re: Ensuring low latency for publishers (Simon MacMullen)
  11. Re: STOMP doesn't include the routing-key on      MESSAGE frames
      (Steve Powell)
  12. Re: dead connection lingers around? (Simon MacMullen)
  13. Re: ??:  Passive queue declaration and channel closure
      (Simon MacMullen)
  14. Re: Apache MINA Integration (Michael Bridgen)
  15. Re: [help] [beginner] server stops sending messages; publish
      (in transaction) hangs on commit (Marek Majkowski)

----------------------------------------------------------------------

Message: 1
Date: Sun, 26 Feb 2012 18:45:39 +0000
From: jay nanavati <jaysnanavati at hotmail.co.uk>
Subject: [rabbitmq-discuss] (no subject)
To: <rabbitmq-discuss at lists.rabbitmq.com>
Message-ID: <BAY154-W539870CC6FC7A6D83362DC99680 at phx.gbl>
Content-Type: text/plain; charset="windows-1252"

Hi everyone,
I have followed this guide to setup rabbitMQ for android.http://simonwdixon.wordpress.com/2011/06/03/getting-started-with-rabbitmq-on-android-part-1/
The problem I am facing is On the line mConsumer.connectToRabbitMQ() in the ActivityHome, I have changed it to:
mOutput.append(?\n?+mConsumer.connectToRabbitMQ());
now when I have the wrong server, it prints false as expected however when I have the write server it does not print anything. In fact it does not even print the @string/hello which it should anyways..
The app does not consume messages and I think this could be the reason.. Also I have used loger and found out that the error must be on this method because I can logg before the call however I cannot logg anything after the call.
Any help would be greatly appreciated.. Thanks!

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120226/4bea75c7/attachment.html>

------------------------------

Message: 2
Date: Sun, 26 Feb 2012 12:51:53 -0800 (PST)
From: Uday Subbarayan <uday.subbarayan at yahoo.com>
Subject: [rabbitmq-discuss] Apache MINA Integration
To: "rabbitmq-discuss at lists.rabbitmq.com"
        <rabbitmq-discuss at lists.rabbitmq.com>
Message-ID:
        <1330289513.57728.YahooMailNeo at web44910.mail.sp1.yahoo.com>
Content-Type: text/plain; charset="iso-8859-1"

Hi All,
? ? ?Has anyone integrated Apache MINA with Rabbit MQ to publish and receive messages?

Thanks,
-Uday.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120226/55d812e1/attachment-0001.htm>

------------------------------

Message: 3
Date: Sun, 26 Feb 2012 19:00:15 -0800 (PST)
From: Codevally <codevally at gmail.com>
Subject: [rabbitmq-discuss] Problem with using rabbitmqctl on EC2
To: rabbitmq-discuss at lists.rabbitmq.com
Message-ID:
        <17970f1d-15cf-42a2-9b5a-f431ffd74a6d at v2g2000vbx.googlegroups.com>
Content-Type: text/plain; charset=ISO-8859-1

Hi Experts

I have installed RabbitMQ 2.7.1 on EC2 (host name is ec2test2) and server getting started without any issue.

AMQP 0-9-1 / 0-9 / 0-8
Copyright (C) 2007-2011 VMware, Inc.
Licensed under the MPL.  See http://www.rabbitmq.com/

node           : rabbit at ec2test2
app descriptor : /app/rabbitmq_server-2.7.1/sbin/../ebin/rabbit.app
home dir       : /root
config file(s) : (none)
cookie hash    : 4lZvwoYL5ZJbkEiBx9CgDA==
log            : /var/log/rabbitmq/rabbit at ec2test2.log
sasl log       : /var/log/rabbitmq/rabbit at ec2test2-sasl.log
database dir   : /app/rabbitmq-config/lib/rabbitmq/mnesia/
rabbit at ec2test2
erlang version : 5.9

But when I try to use rabbitmqctl, I am getting the below error.

ec2test2:/app/rabbitmq/sbin # sh rabbitmqctl {error_logger,{{2012,2,27},{2,55,8}},"Can't set long node name!
\nPlease check your configuration\n",[]} {error_logger,{{2012,2,27},{2,55,8}},crash_report,[[{initial_call,
{net_kernel,init,['Argument__1']}},{pid,<0.20.0>},{registered_name,[]},
{error_info,{exit,{error,badarg},[{gen_server,init_it,6,
[{file,"gen_server.erl"},{line,313}]},{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,227}]}]}},{ancestors,
[net_sup,kernel_sup,<0.9.0>]},{messages,[]},{links,[<0.17.0>]},
{dictionary,[{longnames,true}]},{trap_exit,true},{status,running},
{heap_size,6765},{stack_size,24},{reductions,2361}],[]]}

Host name is correctly configured and it displays:
ec2test2:/app/rabbitmq/sbin # hostname
ec2test2
ec2test2:/app/rabbitmq/sbin #

Could someone please explain this erroneous behavior?

Thanks.

------------------------------

Message: 4
Date: Mon, 27 Feb 2012 16:37:24 +1300
From: Alistair Bayley <alistair at abayley.org>
Subject: Re: [rabbitmq-discuss] [help] [beginner] server stops sending
        messages; publish (in transaction) hangs on commit
To: Simon MacMullen <simon at rabbitmq.com>
Cc: rabbitmq-discuss at lists.rabbitmq.com
Message-ID:
        <CAKYODbip9zOg2=7xNukG6oMAGVqZHdU1ibza=5+98xvUTQtzow at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

On 25 February 2012 00:38, Simon MacMullen <simon at rabbitmq.com> wrote:
>>
>> OK. Can I have some guidance as to how much memory we should allocate?
>
> This is a real how-long-is-a-piece-of-string question I'm afraid. 512M
> would already be a significant step towards the comfort zone though.

rabbitmqctl report shows a large number of channels (in the thousands) and memory usage seems to correlate with the large number of channels.
I've left the server running and shut down each of the clients, and the channels have vanished from the report output, so I suspect a dodgy client here. We're using python kombu over ampqlib, so maybe kombu is doing something undesirable. A quick glance at the source shows that it has a channel pool, so maybe that's to blame. Anyway, more investigation required...

Once the number of channels is down to less than 10, the memory usage is around 32M, which seems fine to me. Maybe I don't need to allocate more, if I can solve this channels problem.

>> Is there an
>> option to get the server to return something that says that it is not
>> accepting messages?
>
> In ancient history we used to do this; there's a method in AMQP 0-9-1
> called channel.flow which the server should be able to use to tell a
> client to shut up. In practice it was unreliable (client may not
> support it, or may take too long to take notice of it).

I will do some more testing. Maybe the immediate flag on basic_publish will do something useful. python amqplib is protocol version 0.8, so whatever I do must fit in with that.

Alistair

------------------------------

Message: 5
Date: Mon, 27 Feb 2012 10:25:37 +0100
From: stone <zmstone at gmail.com>
Subject: [rabbitmq-discuss] dead connection lingers around?
To: rabbitmq-discuss at lists.rabbitmq.com
Message-ID:
        <CAPNZrFUMw1ZarWq-cuooqw8g6FJXhCNM7xgaBuiYS=a+cwHHQw at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Hi

Have a problem while testing my Erlang client based application.
I have no idea if it's due to some bad behavior in my app or whatsoever, hope I can get some useful information here.

In my case, I have:
three rabbit nodes in a cluster;
one non-durable topic exchange;
two connections established;
several channels and queues (exclusive & non-exclusive) declared.

While i'm testing, sometimes the connections they just linger around even the client has quit already.

For instance, here is one or my dead connections (copied from admin page):

AMQP 0-9-110.15.36.246:36526rabbit at test-app20B/s(1.9kB total) 0B/s(1.1kB total)4guestrunning I'm pretty sure that the client on 10.15.36.246 has been killed already, but the connection is still in "running" state.

/stone
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120227/3187eb08/attachment-0001.htm>

------------------------------

Message: 6
Date: Mon, 27 Feb 2012 09:44:56 +0000
From: Steve Powell <steve at rabbitmq.com>
Subject: Re: [rabbitmq-discuss] Problems with STOMP & access control
To: Lionel Cons <lionel.cons at cern.ch>
Cc: rabbitmq-discuss at lists.rabbitmq.com
Message-ID: <7245AC14-4662-4F85-B38A-EAFF206DD7EF at rabbitmq.com>
Content-Type: text/plain; charset=us-ascii

Lionel,

Apologies for tardy reply.

Presently there is no way to set access control based upon the STOMP names, though I agree this would be desirable. Does the STOMP specification talk about access control at all?

I think the destination names aren't all entirely opaque, so it might be possible to exert access control on some resources, but a comprehensive solution is not available, I'm afraid.

Steve Powell
steve at rabbitmq.com
[wrk: +44-2380-111-528] [mob: +44-7815-838-558]

On 21 Feb 2012, at 13:44, Lionel Cons wrote:

> I've read http://www.rabbitmq.com/access-control.html and
> http://www.rabbitmq.com/stomp.html and I did not find any practical
> way to use access control with STOMP.
>
> In the STOMP world (in fact, since STOMP is broker behavior agnostic,
> this is rather tied to the JMS world), one can use destinations like
> /queue/test.foo or /topic/test.bar. It would be natural to use these
> names in the access control regexps to allow, for instance, a given
> user to access all queues matching "test\..*".
>
> Unfortunately, the STOMP destinations names get mapped to AMQP
> resource names that loose the original names, for instance
> amq.gen-AXVr2gFuBTO4duQ5OEC9b9.
>
> Is there a way in RabbitMQ to use access control based on the STOMP
> destination names?
>
> Cheers,
>
> Lionel
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

------------------------------

Message: 7
Date: Mon, 27 Feb 2012 10:21:39 +0000
From: Steve Powell <steve at rabbitmq.com>
Subject: Re: [rabbitmq-discuss] Unexpected behaviour with STOMP & NACK
To: Lionel Cons <lionel.cons at cern.ch>
Cc: rabbitmq-discuss at lists.rabbitmq.com
Message-ID: <C71F5D87-E010-4007-8CEB-7560CCEBFFD7 at rabbitmq.com>
Content-Type: text/plain; charset=us-ascii

Lionel,

Apologies for the tardy reply.

Thank you for reporting this. What is happening is that the NACK sets the 'requeue=true' flag, which, by definition, puts the message back on its queue.
It is then available for redelivery. Whether it gets redelivered to the same consumer depends on lots of things, but is entirely possible, and is expected when there is only one consumer.

Currently, we expect this behaviour and did not think it was prohibited by the specification.

If this is determined to violate the specification we could set requeue=false on NACK, then neither this client, nor any other, will see the message; which is allowable server behaviour. There is currently no mechanism for re-routing messages that are NACKd, but this may change in the near future.

Steve Powell
steve at rabbitmq.com
[wrk: +44-2380-111-528] [mob: +44-7815-838-558]

On 22 Feb 2012, at 12:24, Lionel Cons wrote:

> I get an unexpected behaviour with STOMP & NACK with RabbitMQ 2.7.1.
>
> Here is my test, using an empty queue.
> - client1 connects, sends a message to the queue and disconnects
> - client2 connects, subscribes to the queue with ack:client
> - client2 receives the message (so far so good) but does not ack it
> - client3 connects, subscribes to the queue
> - client2 sends a NACK frame for the message
> - client2 receives the message once more
> - client3 receives nothing
>
> I would have expected the other client (client3) to receive the
> message that has not been consumed by client2. In fact, this is how
> ActiveMQ and Apollo work.
>
> FWIW, the STOMP 1.1 spec mentions:
>
> | NACK is the opposite of ACK. It is used to tell the server that the
> | client did not consume the message. The server can then either send
> | the message to a different client, discard it, or put it in a dead
> | letter queue. The exact behavior is server specific.
>
> So, although the behavior is not strictly defined, re-sending the
> message to the same client does not look appropriate to me.
>
> What do you think?
>
> Cheers,
>
> Lionel
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

------------------------------

Message: 8
Date: Mon, 27 Feb 2012 02:30:15 -0800 (PST)
From: Jon Hill <jon.hill at comshen.com>
Subject: Re: [rabbitmq-discuss] Consumers and exceptions
To: rabbitmq-discuss at lists.rabbitmq.com
Message-ID:
        <3b4977f4-5f27-4b7b-97bc-64606fc3acc9 at hs8g2000vbb.googlegroups.com>
Content-Type: text/plain; charset=ISO-8859-1

Steve

Many thanks for the reply. That makes sense, and I am now progressing.

All the best

Jon

------------------------------

Message: 9
Date: Mon, 27 Feb 2012 11:31:45 +0100
From: Lionel Cons <lionel.cons at cern.ch>
Subject: Re: [rabbitmq-discuss] Problems with STOMP & access control
To: Steve Powell <steve at rabbitmq.com>
Cc: rabbitmq-discuss at lists.rabbitmq.com
Message-ID: <20299.23441.992131.685344 at mail.cern.ch>
Content-Type: text/plain; charset="us-ascii"

Steve Powell writes:
 > Does the STOMP specification talk about access control at all?

No. The specification does not describe the broker behavior. Even something like "/topic/foo" means a JMS-style topic named "foo" is outside of the spec.

 > I think the destination names aren't all entirely opaque, so it  > might be possible to exert access control on some resources, but a  > comprehensive solution is not available, I'm afraid.

It seems that things might work with queues since a STOMP destination of "/queue/foo" will AFAIK use a queue named "foo". To be confirmed.

Topics are more problematic but this does not seem to be STOMP specific.

In AMQP, how would you configure the RabbitMQ ACLs so that client A can only send to amq.topic with a routing key matching A.* while client B is restrcited to routing keys matching B.*?

The http://www.rabbitmq.com/access-control.html page does not really describe what a reource is. Would a syntax like amq.topic:A.* make sense to perform routing key based access control?

Cheers,

Lionel

------------------------------

Message: 10
Date: Mon, 27 Feb 2012 10:42:00 +0000
From: Simon MacMullen <simon at rabbitmq.com>
Subject: Re: [rabbitmq-discuss] Ensuring low latency for publishers
To: rabbitmq-discuss at lists.rabbitmq.com
Message-ID: <4F4B5DF8.5090500 at rabbitmq.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

On 24/02/12 18:07, Eric wrote:
> In reality, the throttling case is extremely unlikely anyway, right?
> You basically have to create a case where publishers are outpacing
> consumers and/or the speed of disk writes on the broker.  So you'd
> have to publish something like 100mb/s with little or no consumption,
> and our application is roughly 5mb/minute.

With small messages in fact the first thing you run into is being CPU-bound making routing decisions. But the general principle holds, yes.

> In conclusion, I finally did what I should have done in the first
> place.  I mocked up the threading scenario we're likely to see in
> production and benchmarked it against Java's LinkedBlockingQueue and
> ArrayBlockingQueue.  Compared to each producer thread having its own
> linked list (the completely uncontended scenario) the blocking queues
> introduced virtually no additional waiting - a microsecond or two at
> most.  That's the case even at rates 50x or more than we'd see in
> production.  So blocking queue it is, and that's a comfortingly simple
> solution.
>
> Thanks for all the help and apologies for under-researching what I was
> doing.

Ah, good.

I had a quick look at implementing non-blocking publish on Java on Friday. Unfortunately it's not possible to do this without NIO on Java, so a prerequisite becomes "port the Java client to NIO", which... makes it less likely to happen I'm afraid.

Cheers, Simon

--
Simon MacMullen
RabbitMQ, VMware

------------------------------

Message: 11
Date: Mon, 27 Feb 2012 10:56:09 +0000
From: Steve Powell <steve at rabbitmq.com>
Subject: Re: [rabbitmq-discuss] STOMP doesn't include the routing-key
        on      MESSAGE frames
To: Tony Garnock-Jones <tonygarnockjones+rabbitmq at gmail.com>
Cc: RabbitMQ Discuss <rabbitmq-discuss at lists.rabbitmq.com>
Message-ID: <5E2DB83D-C02B-48D6-97C2-FDB76643B855 at rabbitmq.com>
Content-Type: text/plain; charset=us-ascii

Tony,

Read your patch, and prefer this solution to adding extra headers.

I'm raising a bug (24763) to fix our spec-compliance problem (thank you for
pointing this out), and see if we can get this change in.

Steve Powell
steve at rabbitmq.com
[wrk: +44-2380-111-528] [mob: +44-7815-838-558]

On 20 Feb 2012, at 20:36, Tony Garnock-Jones wrote:

> Hi Steve,
>
> How about the attached patch? It attempts to bring the "destination"
> header in MESSAGEs in to line with the STOMP 1.1 spec. I've paid
> cursory attention to binary-and-slash-escaping issues but it could use
> a second glance before being committed, I reckon.
>
> Regards,
>  Tony
>
> On 20 February 2012 14:56, Tony Garnock-Jones
> <tonygarnockjones+rabbitmq at gmail.com> wrote:
>> Adding the routing_key puts the final small missing piece in place. A
>> perhaps more STOMPish way of giving it would be to have a "to" header
>> or similar carried in a MESSAGE, containing the full destination given
>> in the SEND. Oh wait! I've just checked, and actually that's what's
>> required by the 1.1 spec already. So the STOMP adapter isn't quite
>> spec-compliant, in that it's giving the destination from the
>> *subscribe* rather than the *send*.
> --
> Tony Garnock-Jones
> tonygarnockjones at gmail.com
> http://homepages.kcbbs.gen.nz/tonyg/
> <better-destination-patch.patch>

------------------------------

Message: 12
Date: Mon, 27 Feb 2012 11:05:56 +0000
From: Simon MacMullen <simon at rabbitmq.com>
Subject: Re: [rabbitmq-discuss] dead connection lingers around?
To: stone <zmstone at gmail.com>
Cc: rabbitmq-discuss at lists.rabbitmq.com
Message-ID: <4F4B6394.10903 at rabbitmq.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Hi.

Sometimes if a TCP connection gets killed outright the OS never notices
and so we don't know either.

This probably implies you are not closing connections properly? (i.e.
with the AMQP close handshake, by invoking amqp_connection:close/1.)

Enabling AMQP heartbeats will allow RabbitMQ to detect this condition.

Cheers, Simon

On 27/02/12 09:25, stone wrote:
> Hi
>
> Have a problem while testing my Erlang client based application.
> I have no idea if it's due to some bad behavior in my app or whatsoever,
> hope I can get some useful information here.
>
> In my case, I have:
> three rabbit nodes in a cluster;
> one non-durable topic exchange;
> two connections established;
> several channels and queues (exclusive & non-exclusive) declared.
>
> While i'm testing, sometimes the connections they just linger around
> even the client has quit already.
>
> For instance, here is one or my dead connections (copied from admin page):
>
> AMQP 0-9-1    10.15.36.246:36526 <http://10.15.36.246:36526>
> rabbit at test-app2      0B/s_(1.9kB total)      0B/s_(1.1kB total)      4       guest   running
>
> I'm pretty sure that the client on 10.15.36.246 has been killed already,
> but the connection is still in "running" state.
>
> /stone
>
>
>
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-discuss at lists.rabbitmq.com
> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

--
Simon MacMullen
RabbitMQ, VMware

------------------------------

Message: 13
Date: Mon, 27 Feb 2012 11:06:49 +0000
From: Simon MacMullen <simon at rabbitmq.com>
Subject: Re: [rabbitmq-discuss] ??:  Passive queue declaration and
        channel closure
To: Simone Busoli <simone.busoli at gmail.com>
Cc: "rabbitmq-discuss at lists.rabbitmq.com"
        <rabbitmq-discuss at lists.rabbitmq.com>
Message-ID: <4F4B63C9.7070909 at rabbitmq.com>
Content-Type: text/plain; charset=GB2312

On 25/02/12 18:37, Simone Busoli wrote:
> David, I somewhat agree with you, lack of documentation about
> operational aspects of broker behavior is something I have been missing
> as well.

Yeah, I suspect our documentation could still be better. But this is one
of the areas where it's really easy to be blinded by over-familiarity.
What sort of things do you think are particularly missing?

Cheers, Simon

--
Simon MacMullen
RabbitMQ, VMware

------------------------------

Message: 14
Date: Mon, 27 Feb 2012 11:43:12 +0000
From: Michael Bridgen <mikeb at rabbitmq.com>
Subject: Re: [rabbitmq-discuss] Apache MINA Integration
To: rabbitmq-discuss at lists.rabbitmq.com
Message-ID: <4F4B6C50.9070702 at rabbitmq.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

> Has anyone integrated Apache MINA with Rabbit MQ to publish and receive
> messages?

What form would you expect such an integration to take?

------------------------------

Message: 15
Date: Mon, 27 Feb 2012 11:49:18 +0000
From: Marek Majkowski <majek04 at gmail.com>
Subject: Re: [rabbitmq-discuss] [help] [beginner] server stops sending
        messages; publish (in transaction) hangs on commit
To: Alistair Bayley <alistair at abayley.org>
Cc: rabbitmq-discuss <rabbitmq-discuss at lists.rabbitmq.com>
Message-ID:
        <CABzX+qznWcOPTsLob0w+uK0vUg7GBbOqot0EzhbKud12DDLAJQ at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

On Mon, Feb 27, 2012 at 03:37, Alistair Bayley <alistair at abayley.org> wrote:
> On 25 February 2012 00:38, Simon MacMullen <simon at rabbitmq.com> wrote:
>>>
>>> OK. Can I have some guidance as to how much memory we should allocate?
>>
>> This is a real how-long-is-a-piece-of-string question I'm afraid. 512M would
>> already be a significant step towards the comfort zone though.
>
> rabbitmqctl report shows a large number of channels (in the thousands)
> and memory usage seems to correlate with the large number of channels.
> I've left the server running and shut down each of the clients, and
> the channels have vanished from the report output, so I suspect a
> dodgy client here. We're using python kombu over ampqlib, so maybe
> kombu is doing something undesirable. A quick glance at the source
> shows that it has a channel pool, so maybe that's to blame. Anyway,
> more investigation required...

Right. This is definitely possible that channels are the main
thing that consumes memory.

> Once the number of channels is down to less than 10, the memory usage
> is around 32M, which seems fine to me. Maybe I don't need to allocate
> more, if I can solve this channels problem.

Yup, quite possible.

>>> Is there an
>>> option to get the server to return something that says that it is not
>>> accepting messages?
>>
>> In ancient history we used to do this; there's a method in AMQP 0-9-1 called
>> channel.flow which the server should be able to use to tell a client to shut
>> up. In practice it was unreliable (client may not support it, or may take
>> too long to take notice of it).
>
> I will do some more testing. Maybe the immediate flag on basic_publish
> will do something useful. python amqplib is protocol version 0.8, so
> whatever I do must fit in with that.

Nope, "immediate" flag is a completely unrelated:
  http://www.rabbitmq.com/amqp-0-9-1-reference.html#basic.publish

Marek

------------------------------

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

End of rabbitmq-discuss Digest, Vol 57, Issue 28
************************************************

------------------------------

Message: 2
Date: Tue, 28 Feb 2012 12:48:03 -0300
From: "CONFERENCIA" <santiagomonttco at gmail.com>
Subject: [rabbitmq-discuss] |Spam| ACCIDENTES DEL TRABAJO
        "RESPONSABILIDAD CIVIL Y PENAL"
To: <rabbitmq-discuss at lists.rabbitmq.com>
Message-ID: <fb9cc8f8818c16aba67404620019a3af at gmail.com>
Content-Type: text/plain; charset="us-ascii"

An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120228/db899444/attachment.html>

------------------------------

Message: 3
Date: Tue, 28 Feb 2012 07:44:55 -0800
From: Gary Russell <grussell at vmware.com>
Subject: Re: [rabbitmq-discuss] Apache MINA Integration
To: "rabbitmq-discuss at lists.rabbitmq.com"
        <rabbitmq-discuss at lists.rabbitmq.com>
Message-ID: <4F4CF677.7090600 at vmware.com>
Content-Type: text/plain; charset="iso-8859-1"

We have lightweight TCP adapters in Spring Integration; it makes it very easy to bridge tcp to/from Rabbit. I just pushed a very simple sample here.

https://github.com/garyrussell/spring-integration-samples/tree/INTSAMPLES-66/basic/tcp-amqp

It sends data from a telnet client to a netcat server via a rabbit queue.

Gary

On 02/28/2012 12:16 AM, Uday Subbarayan wrote:
I am aware of the general purpose EAI frameworks including Camel. We are building a high performance compute cluster and using Rabbit MQ to pass messages. So, i want something very simple TCP server to receive & send messages to the queue.

Best,
Uday.

________________________________
From: Oleg Zhurakousky <ozhurakousky at vmware.com><mailto:ozhurakousky at vmware.com>
To: Michael Bridgen <mikeb at rabbitmq.com><mailto:mikeb at rabbitmq.com>
Cc: rabbitmq-discuss at lists.rabbitmq.com<mailto:rabbitmq-discuss at lists.rabbitmq.com>
Sent: Monday, February 27, 2012 6:20 AM
Subject: Re: [rabbitmq-discuss] Apache MINA Integration

I believe it is a more general concern and typically handled by integration frameworks, in this case EIP-style (http://www.eaipatterns.com/) frameworks such as Spring Integration (http://www.springsource.org/spring-integration)
These frameworks already have fist class integration with AMQP (e.g., AMQP inbound and outbound channel adapters) as well as other protocols and products including TCP/IP.

Oleg

----- Original Message -----
From: "Michael Bridgen" <mikeb at rabbitmq.com<mailto:mikeb at rabbitmq.com>>
To: rabbitmq-discuss at lists.rabbitmq.com<mailto:rabbitmq-discuss at lists.rabbitmq.com>
Sent: Monday, February 27, 2012 6:43:12 AM
Subject: Re: [rabbitmq-discuss] Apache MINA Integration

> Has anyone integrated Apache MINA with Rabbit MQ to publish and receive
> messages?

What form would you expect such an integration to take?
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com<mailto:rabbitmq-discuss at lists.rabbitmq.com>
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com<mailto:rabbitmq-discuss at lists.rabbitmq.com>
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120228/9d91cb89/attachment-0001.htm>

------------------------------

Message: 4
Date: Tue, 28 Feb 2012 15:53:56 +0000
From: Christopher Lefevre <CLefevre at Transparent.com>
Subject: [rabbitmq-discuss] Query a queue status from the client
To: "rabbitmq-discuss at lists.rabbitmq.com"
        <rabbitmq-discuss at lists.rabbitmq.com>
Message-ID:
        <5798631D5F355D4C99FDF23580295AF8187E009E at TLMAIL1.Transparent.local>
Content-Type: text/plain; charset="us-ascii"

Hello,
In my searches for obtaining the queue's status from a client, I stumbled upon this Stack Overflow question:
http://stackoverflow.com/questions/1038318/check-rabbitmq-queue-size-from-client

The user is asking about .NET, however I need it to work for Python.

I'm using PIka (0-9-1, if I recall correctly), with RabbitMQ 2.7.1

The user on Stack Overflow seems to be posting the exact solution I need as:

name, jobs, consumers = chan.queue_declare(queue=queuename, passive=True)

However, when I use this line, I only receive back an error that a None type object is not iterable. Which is due to the function only returning None.

Does anyone have any experience with this use case? Am I missing something in the queue declare?

Thank you for your time,
-Christopher Lefevre
The information contained in this electronic message and any attached document(s) is intended only for the personal and confidential use of the designated recipients named above. This message may be confidential. If the reader of this message is not the intended recipient, you are hereby notified that you have received this document in error, and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify sender immediately by telephone (603) 262-6300 or by electronic mail immediately. Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120228/ccd8903e/attachment-0001.htm>

------------------------------

Message: 5
Date: Tue, 28 Feb 2012 07:57:41 -0800
From: Gary Russell <grussell at vmware.com>
Subject: Re: [rabbitmq-discuss] Regd: Spring AMQP versus Java Client
        Library
To: "rabbitmq-discuss at lists.rabbitmq.com"
        <rabbitmq-discuss at lists.rabbitmq.com>
Message-ID: <4F4CF975.5090507 at vmware.com>
Content-Type: text/plain; charset="iso-8859-1"

Spring AMQP...

http://static.springsource.org/spring-amqp/reference/html/

...brings, to AMQP, the familiar benefits of Spring (high level abstractions - eliminating boilerplate code, while providing access to lower level APIs when necessary, consistent exception hierarchy across different vendor implementations,  etc).

It also provides a listener container, enabling message-driven POJOs, thus eliminating all infrastructure dependencies from application code.

While clearly aimed at existing Spring users, who are already aware of these benefits, we feel it is equally applicable to those users who are not (yet) using Spring.

Gary

On 02/27/2012 10:13 PM, preeti sp wrote:
Hi
Do we get any advantages If we use Spring AMQP over Java Client Library or vice-versa?

Is it advised to use Spring AMQP If we do not intend to primarily use Spring as such?

Thanks,
Preeti

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120228/9c4c3bf1/attachment-0001.htm>

------------------------------

Message: 6
Date: Tue, 28 Feb 2012 16:51:29 +0000
From: Christopher Lefevre <CLefevre at Transparent.com>
Subject: Re: [rabbitmq-discuss] Query a queue status from the client
To: "rabbitmq-discuss at lists.rabbitmq.com"
        <rabbitmq-discuss at lists.rabbitmq.com>
Message-ID:
        <5798631D5F355D4C99FDF23580295AF8187E00B9 at TLMAIL1.Transparent.local>
Content-Type: text/plain; charset="us-ascii"

I apologize, in my haste to send this out, I didn't get to fully investigate.

If someone else were to come upon this issue, what you need to do is declare a different Callback method, which accepts a variable.

This variable will contain member items for the Queue, Awaiting Messages, Consumers, Channel Number, and Frame Type.

To access these you would need to use:
variable.frame_type
variable.channel_number
variable.method.queue
variable.method.message_count
variable.method.consumer_count

Sorry to clog up the mailing list, and I hope I'm appending to the message correctly, but hopefully this will clear up any questions people may have had like this.

-Christopher Lefevre

From: Christopher Lefevre
Sent: Tuesday, February 28, 2012 10:54 AM
To: 'rabbitmq-discuss at lists.rabbitmq.com'
Subject: Query a queue status from the client

Hello,
In my searches for obtaining the queue's status from a client, I stumbled upon this Stack Overflow question:
http://stackoverflow.com/questions/1038318/check-rabbitmq-queue-size-from-client

The user is asking about .NET, however I need it to work for Python.

I'm using PIka (0-9-1, if I recall correctly), with RabbitMQ 2.7.1

The user on Stack Overflow seems to be posting the exact solution I need as:

name, jobs, consumers = chan.queue_declare(queue=queuename, passive=True)

However, when I use this line, I only receive back an error that a None type object is not iterable. Which is due to the function only returning None.

Does anyone have any experience with this use case? Am I missing something in the queue declare?

Thank you for your time,
-Christopher Lefevre
The information contained in this electronic message and any attached document(s) is intended only for the personal and confidential use of the designated recipients named above. This message may be confidential. If the reader of this message is not the intended recipient, you are hereby notified that you have received this document in error, and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify sender immediately by telephone (603) 262-6300 or by electronic mail immediately. Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120228/0cc5aa63/attachment-0001.htm>

------------------------------

Message: 7
Date: Tue, 28 Feb 2012 11:47:45 -0800 (PST)
From: Jerry Kuch <jerryk at vmware.com>
Subject: Re: [rabbitmq-discuss] RabbitMQ and MySQL
To: PepMad <pepmadon at gmail.com>
Cc: rabbitmq-discuss at lists.rabbitmq.com
Message-ID: <0a594157-bc17-40e1-ab02-505ba0e7cd6e at CoachZ.local>
Content-Type: text/plain; charset=utf-8

Hi, Pep:

Can you tell us a bit more what you're trying to do?  I think
the StackOverflow poster is hinting at perhaps a more messaging-based
design for your application, rather than a quick drop-in hack to make
some side effect beyond the RDBMS happen as a result of a MySQL trigger.

Best regards,
Jerry

----- Original Message -----
From: "PepMad" <pepmadon at gmail.com>
To: rabbitmq-discuss at lists.rabbitmq.com
Sent: Monday, February 20, 2012 2:24:29 PM
Subject: [rabbitmq-discuss] RabbitMQ and MySQL

http://stackoverflow.com/questions/1467369/invoking-a-php-script-from-a-mysql-trigger

I was reading this post on Stack Overflow and the last suggestion was
to use RabbitMQ. Is this possible? I read the documentation on
RabbitMQ, how can a single mysql update be sent to a producer? I have
seen the mysql log file but that encapsulates multiple updates.

Thanks Pep
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

------------------------------

Message: 8
Date: Tue, 28 Feb 2012 11:53:46 -0800 (PST)
From: Jerry Kuch <jerryk at vmware.com>
Subject: Re: [rabbitmq-discuss] Design Help?
To: James Aimonetti <james at 2600hz.com>
Cc: rabbitmq-discuss at lists.rabbitmq.com
Message-ID: <3418300e-6989-4f26-9846-8c0f6682f9e0 at CoachZ.local>
Content-Type: text/plain; charset=utf-8

Hi, James:

Some of this appears to go beyond messaging per se and into more
general distributed systems/services coordination.  Have you contemplated
using something like Apache Zookeeper or Heroku's Doozer?  The primitives
provided by those may lend themselves well to integrating-with/supporting
your application, while the messaging heavy lifting continues using Rabbit
and its features/guarantees in the ways they work best without strange
fetch-peek-ack-nack trickery that will likely make your application
pretty complex and hard to reason about.

http://zookeeper.apache.org/

http://xph.us/2011/04/13/introducing-doozer.html

Doozer is quite new; Zookeeper by virtue of its position in the
Hadoop ecology is older, more mature and well documented both online
and in books...

Best regards,
Jerry

----- Original Message -----
From: "James Aimonetti" <james at 2600hz.com>
To: rabbitmq-discuss at lists.rabbitmq.com
Sent: Monday, February 20, 2012 9:04:12 AM
Subject: [rabbitmq-discuss] Design Help?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hey list,

Curious for feedback, as this design idea I have would possibly extend
to several places in our application, and I'd love to make it generic
enough to do so.

The basic problem:

We have a queue of work to be done, represented as a named queue. For
redundancy, we have 2->many processes (in Erlang) that will read from
the queue and do the work. Easy so far.

However, we also have a list of resources needed to process the work
queue, which needs to be shared across all worker processes. When one
worker retrieves a job from the work queue, it must also get the list of
available resources, match resource to job, and return the list of
resources (with the used resource flagged as such) back to a globally
accessible place (hoping to avoid the DB, but it may be a good-enough
sync point).

So, for example, there are four jobs, two that require a paper shredder,
two that require a wood chipper. There is only one paper shredder and
one wood chipper. There are also 10 workers available to actually shred
stuff.

Worker one consumes the list of available equipment (one paper shredder,
one wood chipper), and then pulls the first job (paper to be shredded).
Worker one grabs the paper shredder, returns the list of resources
having flagged the paper shredder as in use, and ACKs the job. When
worker one finishes shredding paper, it will wait until the resources
list is available and update it with 1 more paper shredder available.

Once a job is actually started, I don't care if the worker crashes or
not; that will be handled later. And I'll have a supervising gen_server
pull the resource list, add the resource its using to its state, and
spawn the actual worker (catching exits to return the resource). I'm
sure other measures will be needed to ensure the resource is added back
eventually, but I'm trying to get the basic idea down.

So once worker one is done retrieving a resource, worker two can next
get the resources list and the next job. If the job is paper shredding,
worker two will nack the job (as there are no paper shredders at the
moment) and return the resource list. I will eventually want to support
being able to requeue the job to the back of the job queue, but for now,
work stalls until the first item in the queue can be processed successfully.

So while the resources list is out of queue, no workers will consume
jobs from the work queue. As soon as it reappears in queue, I want the
next worker in line to grab it and try to process the next job.

Some initial concerns:

No resources available for the job at the front of the queue means a lot
of fetching and nacking of jobs, and fetching/returning the resource list.

Losing the resources list: if whatever node (our Erlang VMs only
communicate via AMQP) goes down while handling the resources list, it
will obviously never be republished. We do usually have a timeout window
for the worker to get a resource for a job (sometimes the actual finding
of the resource takes a while), so if I could coordinate all workers
consuming from the resources_available queue, I could know after timeout
+ a small fudge factor, that someone else should have begun processing.
So maybe the resources_available queue is more a try_to_process_a_job
queue, and each worker process maintains an identical list of resources
available, with a worker that pairs a job to resource publishing the
resource's in-use status.

So perhaps the solution is:

Jobs queue round robins amongst all workers, but only one worker can
fetch/(n)ack. A worker can only fetch from the jobs queue by consuming a
"next" message from a shared coordination queue. All workers publish
updates to resources available to a routing key so all workers'
resources state is in sync. If a worker can't find a resource for a job,
it can (configurable) 1) nack the job and send a 'next' to the
coordination queue; 2) requeue the job at the end of the jobs queue and
send a 'next'; 3) sit on the job (meaning no other worker can fetch from
the jobs queue) until an appropriate resource comes available.

Concerns here:

Starting/recovering the 'next' token.

I'm sure others but am realizing this is getting long.

Sorry for the wall of text and stream of consciousness writing. We have
a lot of places where this pattern is showing up and would love to get a
module or library written to ease doing this kind of coordination. Any
ideas or directions towards appropriate messaging patterns warmly welcome :)

- --
James Aimonetti
Distributed Systems Engineer / DJ MC_

2600hz | http://2600hz.com
sip:james at 2600hz.com
tel: 415.886.7905
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJPQn0MAAoJENc77s1OYoGgh+gH/i/kW2M5EMfQ/xz3dU6soUd4
joGvWC46MdAX2CqlEKyIcP+EsmiTH4Ojjh64Y5MrVv68xV2Zasi8O7W5scFGK31Q
5r1Uc4lLrCANiEA7MYnaKM3J+s+bf4m3NlDwhE/QKCO+OhZoZl/zYfzlzSlsQ3QX
3dIKIxJnvnvmehat/oqbpZFFZO7NX4dKWGr3Ysp4OBjPhQLDGGkZ3Tbjbe3KfL0j
EfdZubwXQQLsTU7oqI7K3HoZc1uWx50VdR28JT1ZTaBdchpUpQ9PXdbq1b++KA++
SkHpwYB37PgTqK9A3gsmuRCsRMmYP8p3iGPrflnP53kuuESZIZwUFuhbthV5DM8=
=MDx7
-----END PGP SIGNATURE-----
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

------------------------------

Message: 9
Date: Tue, 28 Feb 2012 12:23:48 -0800
From: James Aimonetti <james at 2600hz.com>
Subject: Re: [rabbitmq-discuss] Design Help?
To: Jerry Kuch <jerryk at vmware.com>
Cc: rabbitmq-discuss at lists.rabbitmq.com
Message-ID: <4F4D37D4.90307 at 2600hz.com>
Content-Type: text/plain; charset="UTF-8"

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jerry,

Thanks for the suggestions. You're right that this is an abuse of the
messaging patterns, but the way our system is structured seems to
require building these higher level concepts on top of the primitives we
get from AMQP/RabbitMQ.

We have a system spread out over many servers, across datacenters, with
redundant applications running, and we need ways to coordinate work
amongst the redundant workers in this manner. When we don't care about
the order of work, its really easy to have the queue round-robin the
work to each worker.

As we're building more advanced features into our platform, this pattern
of only allowing one worker to process a job, across the whole platform,
is popping up in a few places and we'd like to encapsulate that pattern
into a reusable bit of code (a behaviour in Erlang).

AMQP is the glue that binds all these apps together, so we look to build
on top of its feature set.

I'll let the list know more if I figure a tenable solution :)

On 02/28/2012 11:53 AM, Jerry Kuch wrote:
> Hi, James:
>
> Some of this appears to go beyond messaging per se and into more
> general distributed systems/services coordination.  Have you contemplated
> using something like Apache Zookeeper or Heroku's Doozer?  The primitives
> provided by those may lend themselves well to integrating-with/supporting
> your application, while the messaging heavy lifting continues using Rabbit
> and its features/guarantees in the ways they work best without strange
> fetch-peek-ack-nack trickery that will likely make your application
> pretty complex and hard to reason about.
>
> http://zookeeper.apache.org/
>
> http://xph.us/2011/04/13/introducing-doozer.html
>
> Doozer is quite new; Zookeeper by virtue of its position in the
> Hadoop ecology is older, more mature and well documented both online
> and in books...
>
> Best regards,
> Jerry
>
> ----- Original Message -----
> From: "James Aimonetti" <james at 2600hz.com>
> To: rabbitmq-discuss at lists.rabbitmq.com
> Sent: Monday, February 20, 2012 9:04:12 AM
> Subject: [rabbitmq-discuss] Design Help?
>
> Hey list,
>
> Curious for feedback, as this design idea I have would possibly extend
> to several places in our application, and I'd love to make it generic
> enough to do so.
>
> The basic problem:
>
> We have a queue of work to be done, represented as a named queue. For
> redundancy, we have 2->many processes (in Erlang) that will read from
> the queue and do the work. Easy so far.
>
> However, we also have a list of resources needed to process the work
> queue, which needs to be shared across all worker processes. When one
> worker retrieves a job from the work queue, it must also get the list of
> available resources, match resource to job, and return the list of
> resources (with the used resource flagged as such) back to a globally
> accessible place (hoping to avoid the DB, but it may be a good-enough
> sync point).
>
> So, for example, there are four jobs, two that require a paper shredder,
> two that require a wood chipper. There is only one paper shredder and
> one wood chipper. There are also 10 workers available to actually shred
> stuff.
>
> Worker one consumes the list of available equipment (one paper shredder,
> one wood chipper), and then pulls the first job (paper to be shredded).
> Worker one grabs the paper shredder, returns the list of resources
> having flagged the paper shredder as in use, and ACKs the job. When
> worker one finishes shredding paper, it will wait until the resources
> list is available and update it with 1 more paper shredder available.
>
> Once a job is actually started, I don't care if the worker crashes or
> not; that will be handled later. And I'll have a supervising gen_server
> pull the resource list, add the resource its using to its state, and
> spawn the actual worker (catching exits to return the resource). I'm
> sure other measures will be needed to ensure the resource is added back
> eventually, but I'm trying to get the basic idea down.
>
> So once worker one is done retrieving a resource, worker two can next
> get the resources list and the next job. If the job is paper shredding,
> worker two will nack the job (as there are no paper shredders at the
> moment) and return the resource list. I will eventually want to support
> being able to requeue the job to the back of the job queue, but for now,
> work stalls until the first item in the queue can be processed successfully.
>
> So while the resources list is out of queue, no workers will consume
> jobs from the work queue. As soon as it reappears in queue, I want the
> next worker in line to grab it and try to process the next job.
>
>
> Some initial concerns:
>
> No resources available for the job at the front of the queue means a lot
> of fetching and nacking of jobs, and fetching/returning the resource list.
>
> Losing the resources list: if whatever node (our Erlang VMs only
> communicate via AMQP) goes down while handling the resources list, it
> will obviously never be republished. We do usually have a timeout window
> for the worker to get a resource for a job (sometimes the actual finding
> of the resource takes a while), so if I could coordinate all workers
> consuming from the resources_available queue, I could know after timeout
> + a small fudge factor, that someone else should have begun processing.
> So maybe the resources_available queue is more a try_to_process_a_job
> queue, and each worker process maintains an identical list of resources
> available, with a worker that pairs a job to resource publishing the
> resource's in-use status.
>
> So perhaps the solution is:
>
> Jobs queue round robins amongst all workers, but only one worker can
> fetch/(n)ack. A worker can only fetch from the jobs queue by consuming a
> "next" message from a shared coordination queue. All workers publish
> updates to resources available to a routing key so all workers'
> resources state is in sync. If a worker can't find a resource for a job,
> it can (configurable) 1) nack the job and send a 'next' to the
> coordination queue; 2) requeue the job at the end of the jobs queue and
> send a 'next'; 3) sit on the job (meaning no other worker can fetch from
> the jobs queue) until an appropriate resource comes available.
>
> Concerns here:
>
> Starting/recovering the 'next' token.
>
> I'm sure others but am realizing this is getting long.
>
> Sorry for the wall of text and stream of consciousness writing. We have
> a lot of places where this pattern is showing up and would love to get a
> module or library written to ease doing this kind of coordination. Any
> ideas or directions towards appropriate messaging patterns warmly welcome :)
>
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

- --
James Aimonetti
Distributed Systems Engineer / DJ MC_

2600hz | http://2600hz.com
sip:james at 2600hz.com
tel: 415.886.7905
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJPTTfUAAoJENc77s1OYoGgdq0H/2X7oohyxx0MhsoD06J07l2P
DZm7PdEEZhUCSRGb43cBszNjiFLvYtmbSGgWUEzQMjZvI0idKC6m4XIgZqKfJ7Wk
yVQcvMfeHmE0w1NIyrJLFlXToONHlX2mLStRHnsmhFdMZuDHuAwF41N3rZPX5/wV
JIUIvYg45VccYu0zL4kRq/9nBORJsfXxXcsB8pdKJa0uLxv2OPptDOHFgZVz/wVe
FC99/fBU/WixtDvzhe2D7d+ubn5UT1jf01ZKUTAgkesM3EpcmqdqI51jQR2Q0hbT
VhVODLSlTSrYSYsumStXOaFzZ8kwY7+mT3UDbRqtv+ThIKuWWejCXGFVjDeKcA8=
=NhsO
-----END PGP SIGNATURE-----

------------------------------

Message: 10
Date: Tue, 28 Feb 2012 13:16:06 -0800 (PST)
From: PepMad <pepmadon at gmail.com>
Subject: Re: [rabbitmq-discuss] RabbitMQ and MySQL
To: rabbitmq-discuss at lists.rabbitmq.com
Message-ID:
        <97320990-eca5-46a4-bd47-f6b7e66973a2 at p12g2000yqe.googlegroups.com>
Content-Type: text/plain; charset=ISO-8859-1

Hi Jerry,

A website that the user can submit a job, the job is a script on the
server, the script takes an unknown length of time. I didn't want the
user on the site to wait, I just wanted a return status that the job
was sent to a que. I thought about using mysql udf's or a mysql script
for polling an update table, then I found the StackOverflow post and I
thought RabbitMQ might be a better solution for messaging a que via
the site.

Pep

------------------------------

Message: 11
Date: Tue, 28 Feb 2012 13:20:18 -0800 (PST)
From: Jerry Kuch <jerryk at vmware.com>
Subject: Re: [rabbitmq-discuss] RabbitMQ and MySQL
To: PepMad <pepmadon at gmail.com>
Cc: rabbitmq-discuss at lists.rabbitmq.com
Message-ID: <77adf375-7d45-4988-a92f-0a9c369da3a1 at CoachZ.local>
Content-Type: text/plain; charset=utf-8

Hi, Pep:

That very well might be.  Your application would put together the
specifics of the job, publish them as a message into RabbitMQ, after
which some consumers which do the actual work of the job would dequeue
those tasks and do the work behind the scenes, thus decoupling your
users from having to wait on them.

Check out the tutorials here:

http://www.rabbitmq.com/getstarted.html

For ideas on how you might incorporate such patterns into your app.
It will likely work far better for you than scary database tricks. :-)

You can also look at the book "RabbitMQ in Action" from Manning (now
available as a preview eBook from Manning, print version due out
very soon) for a thorough discussion of messaging and its use in some
example application scenarios that are likely similar to what you have.

Best regards,
Jerry

----- Original Message -----
From: "PepMad" <pepmadon at gmail.com>
To: rabbitmq-discuss at lists.rabbitmq.com
Sent: Tuesday, February 28, 2012 1:16:06 PM
Subject: Re: [rabbitmq-discuss] RabbitMQ and MySQL

Hi Jerry,

A website that the user can submit a job, the job is a script on the
server, the script takes an unknown length of time. I didn't want the
user on the site to wait, I just wanted a return status that the job
was sent to a que. I thought about using mysql udf's or a mysql script
for polling an update table, then I found the StackOverflow post and I
thought RabbitMQ might be a better solution for messaging a que via
the site.

Pep
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

------------------------------

Message: 12
Date: Wed, 29 Feb 2012 14:54:24 +1300
From: Alistair Bayley <alistair at abayley.org>
Subject: [rabbitmq-discuss] [help] [beginner] testing with pika: some
        questions and complaints
To: rabbitmq-discuss at lists.rabbitmq.com
Message-ID:
        <CAKYODbh2qeJC3+j-PCgpsu7VH7HCaMHq2KPbGpZUupRsuY8HBA at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

I've been testing rabbitmq with pika (python) to understand how they
interact and learn about the AMQP protocol. I've some questions and
some complaints; no doubt much of this will be from my
misunderstandings of the AMQP/rabbitmq specs.

Pika (is this better posted to the pika mailing list?):

1.  why is the test in channel.py:102 like so:
        if callback and not isinstance(callback, types.FunctionType) and \
           not isinstance(callback, types.MethodType):
            raise TypeError("callback should be None, a function or method.")

    i.e. why isn't is just a callable() test e.g.
        if callback and not callable(callback):
            raise TypeError("callback should be callable.")

    (is this a python-3 compatibility thing?)

2.  documentation for callback functions is mostly non-existent, and when it
    exists is sometimes wrong e.g. on http://pika.github.com/channel.html :

    (i) channel.basic_get:
        "callback method signature should have 3 parameters:
        The method frame, header frame and the body"
    In fact the callback gets 4 parameters:
        channel, method (Basic.GetOk), properties (Basic.Properties), body

    (ii) channel.add_on_close_callback:
        "The callback function should receive a frame parameter."
    In fact the callback gets 2 parameters: error_code, error_message

3.  why does basic_ack cause this warning?
    CallbackManager.add: Duplicate callback found for "0:Connection.CloseOk"
    i.e. why does it appear to add a(nother) connection close callback?
    I've grubbed through the pika source but can't see how/where it happens.

AMQP/rabbitmq:

4.  why doesn't basic_ack do what I expect?
    If I do a basic_get with no_ack=False, and then a basic_ack,
    the message stays on the queue.
    I can repeatedly fetch-and-ack it, and each time redelivered=true.

5.  if the server reaches a state where it stops accepting messages
    (e.g. memory usage exceeds the high water mark) it does not send
    nack or return responses to basic_publish.
    This is both with transactions, or with channel confirm_delivery
    enabled.

(5) is a concern when one requires guaranteed delivery. Having the
server swallow the message and not respond doesn't seem reliable to
me. Is there some other method I should be using for guaranteed
delivery?

Alistair

------------------------------

Message: 13
Date: Wed, 29 Feb 2012 15:16:54 +0530
From: Ashutosh Mathur <amathur.q at gmail.com>
Subject: [rabbitmq-discuss] (no subject)
To: rabbitmq-discuss at lists.rabbitmq.com
Message-ID:
        <CA+u4LE6Sg4Ou6JAMpP4EKgB0Ep1i_UCRfsmmK4J-tKY3o9rGzQ at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Hi,

I wanted to know:
- whether RabbitMQ 2.7.1 supports AMQP 1.0 or will there be a newer version
of RabbitMQ to support this?
-Can you please throw some light on the JMS-AMQP mapping which is added in
the AMQP 1.0 release.

Thanks in Advance.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20120229/9c33dd27/attachment-0001.htm>

------------------------------

Message: 14
Date: Wed, 29 Feb 2012 10:13:26 +0000
From: Simon MacMullen <simon at rabbitmq.com>
Subject: Re: [rabbitmq-discuss] rabbitmq-discuss Digest, Vol 57, Issue
        28
To: rabbitmq-discuss at lists.rabbitmq.com
Message-ID: <4F4DFA46.4060709 at rabbitmq.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

On 28/02/12 14:56, Jose Jimenez Ortiz wrote:
> I installed rabbitmq 2.7.1 and enable Rabbitmq_Management plugin but
> in the explorer I don't see anything on OVERVIEW panel.

This is a bit of a vague question, making it a bit hard to answer. All I
can say is it works for lots of people :-)

When you say you don't see anything, you mean a completely blank page?
By explorer do you mean MSIE (which version?) Are there any Javascript
errors shown?

What does http://<server_name>:55672/api/overview look like?

What about http://localhost:55672/js/tmpl/overview.ejs ?

Cheers, Simon

--
Simon MacMullen
RabbitMQ, VMware

------------------------------

Message: 15
Date: Wed, 29 Feb 2012 10:28:18 +0000
From: Simon MacMullen <simon at rabbitmq.com>
Subject: Re: [rabbitmq-discuss] [help] [beginner] testing with pika:
        some questions and complaints
To: Alistair Bayley <alistair at abayley.org>
Cc: rabbitmq-discuss at lists.rabbitmq.com
Message-ID: <4F4DFDC2.3020801 at rabbitmq.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

On 29/02/12 01:54, Alistair Bayley wrote:
> I've been testing rabbitmq with pika (python) to understand how they
> interact and learn about the AMQP protocol. I've some questions and
> some complaints; no doubt much of this will be from my
> misunderstandings of the AMQP/rabbitmq specs.
>
> Pika (is this better posted to the pika mailing list?):

Maybe?

> AMQP/rabbitmq:
>
> 4.  why doesn't basic_ack do what I expect?
>      If I do a basic_get with no_ack=False, and then a basic_ack,
>      the message stays on the queue.
>      I can repeatedly fetch-and-ack it, and each time redelivered=true.

That sounds like somehow the ack is malformed somehow (wrong ack tag?)
and instead taking the channel / connection down. Is anything showing up
in the server logs at this point?

> 5.  if the server reaches a state where it stops accepting messages
>      (e.g. memory usage exceeds the high water mark) it does not send
>      nack or return responses to basic_publish.
>      This is both with transactions, or with channel confirm_delivery
>      enabled.
>
>
> (5) is a concern when one requires guaranteed delivery. Having the
> server swallow the message and not respond doesn't seem reliable to
> me. Is there some other method I should be using for guaranteed
> delivery?

The server is not swallowing the message without responding; it's
stopped accepting messages altogether, and will (eventually) send
confirms for everything it's accepted. However, in this scenario you're
presumably hammering the server with messages, and in that case there
can be lots of messages waiting in the OS TCP buffers - RabbitMQ hasn't
seen them at all.

It's not clear what else RabbitMQ could do in this situation - for
guaranteed delivery you need to treat the message as the producer's
responsibility until you see the confirm or tx.commit-ok.

Cheers, Simon

--
Simon MacMullen
RabbitMQ, VMware

------------------------------

Message: 16
Date: Wed, 29 Feb 2012 11:13:35 +0000
From: Simon MacMullen <simon at rabbitmq.com>
Subject: Re: [rabbitmq-discuss] (no subject)
To: Ashutosh Mathur <amathur.q at gmail.com>
Cc: rabbitmq-discuss at lists.rabbitmq.com
Message-ID: <4F4E085F.9040607 at rabbitmq.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

On 29/02/12 09:46, Ashutosh Mathur wrote:
> I wanted to know:
> - whether RabbitMQ 2.7.1 supports AMQP 1.0 or will there be a newer
> version of RabbitMQ to support this?

2.7.1 does not. We have an experimental 1.0 plugin but it requires a
patched server which has not been kept up to date.

We *may* support AMQP 1.0 in the future. But AMQP 1.0 is a huge change
from 0-9-1, there is almost nothing in common. So it's a really
substantial piece of work to support it, and we are a small team. So
we're waiting to see what the demand for it turns out to be.

Cheers, Simon

--
Simon MacMullen
RabbitMQ, VMware

------------------------------

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

End of rabbitmq-discuss Digest, Vol 57, Issue 30
************************************************

This message is for the sole use of the intended recipient. If you received this message in error please delete it and notify us. If this message was misdirected, Structured Portfolio Management, LLC, and its affiliates ("SPM") do not waive any confidentiality or privilege. Opinions expressed are subject to change without notice and may differ or be contrary to the opinions or recommendations of SPM or the opinions expressed by other business areas or groups of SPM as a result of using different assumptions and criteria. SPM is not responsible for any recommendation, solicitation, offer or agreement or any information about any transaction, customer account or account activity contained in this communication.