[rabbitmq-discuss] Tutorial example suggestion: work queues tutorial could have a description of hierarchical shutdown logic

Fri Jul 8 15:17:56 BST 2011

Thanks Ann,

I understand that you folks have a plan for your tutorials and I respect that.  I am really delighted with the quality of the tutorials and wish to thank you very much for the hard work that you have put in to make them useful.  Even though you don't have a C language tutorial, I was able to quickly and easily create test programs using the C language interfaces based on the contents of the python and java tutorials as examples.

I like the way that the tutorials were activity oriented -- that is, rather than putting in a lot of theory up front, the tutorials jump right in with real world examples and build on top of one another.  This is sort of the Stephen King novel version of tutorials.  He always starts with some sort interesting event and then later goes back fills in the details.  

To carry this metaphor forward, I think that the existing tutorials are great but would benefit from the addition of web links at strategic places in the tutorials to "big picture" theory pages.  For example, one such page might be titled "Exchanges vs Queues".   That page could be structured not as a tutorial but as a reference and practical advice on designing applications to take advantage of the contrasting features of exchanges and queues.  Links to these pages could appear as "see also" links in the other tutorials or as highlighted words, linked to the theory pages, within the existing text (my preference).

Thanks for listening and thanks again for doing such a great job on the tutorials.

Lowell

-----Original Message-----
From: Ann Witbrock (c) [mailto:awitbrock at vmware.com] 
Sent: Friday, July 08, 2011 6:24 AM
To: Boggs Jr., Lowell
Cc: rabbitmq-discuss at lists.rabbitmq.com; david at rabbitmq.com
Subject: Re: [rabbitmq-discuss] Tutorial example suggestion: work queues tutorial could have a description of hierarchical shutdown logic

Hi Lowell,
thanks for the suggestion.
I agree that a tutorial on this (shutdown, auto-delete and similar parameters, and related matters) would be useful, but I don't agree that tutorial is the right place for it. I already have that in as a proposed work item. This example could well be helpful in it.

Ann

----- Original Message -----
From: "Lowell Boggs" <Lowell.Boggs at emc.com>
To: david at rabbitmq.com
Cc: rabbitmq-discuss at lists.rabbitmq.com
Sent: Thursday, 7 July, 2011 4:00:47 PM
Subject: [rabbitmq-discuss] Tutorial example suggestion: work queues tutorial could have a description of hierarchical shutdown logic

Thanks for this great advice.  

I looked for an example of how to handle shutdown and other system commands in this tutorial:

   http://www.rabbitmq.com/tutorials/tutorial-two-python.html

But not finding any advice there, I tried to come up with my own idea based on concepts that I gleaned from the rabbitmq-c interfaces themselves and the tutorial examples.  I see the benefits of your suggestion below.

I would like to suggest that the work queues tutorial is a great place to insert the important information you provided in the attached email about exchange hierarchies.

Note that it would not be necessary to provide example code for this to be a helpful addition.  If you understand the work queues logic, you can understand a description of the shutdown logic without seeing it worked out.  

Also, I have not found (maybe I just haven't looked hard enough) a good explanation of why I should prefer one exchange type over the other.  Direct seems to be doing everything I need.  I am a bit concerned about the performance impact of of using topic name patterns.  The shutdown example you give below would be a nice place in the tutorials to comment on which exchange type is best suited for what.

Thanks again!

Lowell

-----Original Message-----
From: David Wragg [mailto:david at rabbitmq.com] 
Sent: Wednesday, July 06, 2011 5:35 PM
To: Boggs Jr., Lowell
Cc: rabbitmq-discuss at lists.rabbitmq.com
Subject: Re: [rabbitmq-discuss] rabbitmq-c: amqp_exchange_declare() always sets auto-delete to no

Hi Lowell,

<Lowell.Boggs at emc.com> writes:
> Is there a reason that you cannot create an exchange with the
> auto_delete property == true?

Auto-delete on exchanges is a AMQP 0-8 feature that is deprecated in
AMQP 0-9-1.  This is why the amqp_exchange_declare function does not
expose it (ever since it was converted to AMQP 0-9-1).

> Is it safe for me as an application devloper to create a clone of this
> function that has the auto_delete property passed in?  Does the
> rabbitmq team have plans to change the way the underlying functions
> work?

It is safe, but not a good idea.

rabbitmq-server still supports AMQP 0-8, including the auto-delete
property on exchanges.  So if you copy amqp_exchange_declare and add
that parameter, it will work today.  But at some point, rabbitmq-server
may drop AMQP 0-8 compatibility, and then it would stop working.

> Is a temporary exchange even a good idea?

Not really.

> I am trying to simplify the shutdown logic of a multi-threaded
> application.  I don't want to have to invent a way of signaling my
> worker threads to shut down.  Instead, I want to send them a message
> that they can read like normal but have a quick way of identifying
> that this is a shutdown command. I am currently writing code for my
> threads that work like this:
>
> void Thread()
> {
>
>     // create the connection
>     // declare the primary command exchange
>     // declare the qrimary queue
>     // declare a "stop command" exchange
>     // declare a "stop command" queue
>     // bind everyone up and call basic_consume on the queues
>
>
>     for(;;)
>      {
>         string exchangeName;
>         string topicName;
>         string messageBody;
>
>          int error = readMessage(exchangeName, topicName, messageBody);
>
>          if(error) break;
>
>         if(exchangeName == "stop")
>            break;
>
>        processMesage(messageBody);
>
>     }
> }
>
> Note that I am having multiple threads in multiple executables on
> multiple machines all reading from the same primary command exchange
> for performance and redundancy reasons -- so I can't really use the
> topic to indicate shutdown.  I might want to shutdown a given
> executable on a given host, not the entire system.
>
> The above logic is working fine, but when I kill a process, I have to
> remember to delete the exchanges because they don't have an autoDelete
> flag.
>
> Is there a better way to handle this shutdown logic?

It sounds like you have a hierarchy of threads, processes, and machines.
And you want to send command messages (including shutdown messages) to
the various levels in that hierarchy: To a thread, to all the threads
in a process, to all the threads in all processes on a host, and to all
threads everywhere.

You can accomplish this by declaring four exchanges. Let's call them
"per-thread", "per-process", "per-host" and "everything".

Each thread declares an auto-delete queue for itself, and binds this
queue to all four exchanges, but with different binding keys.

"per-thread" should be a direct exchange.  A thread binds its queue to
this exchange  with a binding key of the form "<host_id>.<process_id>.<thread_id>".

"per-process" should be a direct exchange.  A thread binds its queue to
this exchange with with a binding key of the form
"<host_id>.<process_id>".

"per-host" should be a direct exchange.  A thread binds its queue to
this exchange with with a binding key of the form "<host_id>"

Finally, "everything" can just be a fanout exchange (and so the binding
key is not used).

In order to send a command, use the exchange for the appropriate level
in the hierarchy, and the corresponding routing key.

Because all four exchanges are long lived, your code should not need to
delete any of them.

-- 
David Wragg
Staff Engineer, RabbitMQ
VMware, Inc.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq-discuss at lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss