[rabbitmq-discuss] JAVA client / non daemon threads / shutdown hook

[Steve Powell]

Bartłomiej,

Thank you for your thoughts. Your solution 1. is what is being currently
proposed, with the (still undecided) feature that it may not apply to
the Consumer callback threads. However, I dislike options and
parameters, principally because we have to explain everything to users,
and it confuses people if there are a lot of obscure parameters -- even
if most people don't set them.

Therefore, I'm considering proposal number 2. This would be a change in
behaviour to make the MainLoop thread (this is the connection management
thread, which receives all the channels' communications) a daemon
thread, and to leave the Consumer callback threads as non-daemon.

What this means is that the users that never register any Consumers
(synchronous consumers use basic.get) will not get any Consumer threads
allocated (unless they do a clean shutdown, in which case there is a
trivial one which is short-lived). So when they exit (without a clean
shutdown) everything will disappear when the Jvm terminates.

Users that want to have things remain will register Consumers, which
means that threads will get allocated, and these will keep everything
alive, even if the main thread terminates, until shutdown, which can be
driven off of the Consumer callback or will be precipitated if the
connection closes. Cleanup will then happen and if the Jvm is waiting to
terminate it will do so then.

This appears to me to fit the bill for both parties, without anyone
having to set any options.

In a complex environment, the way to allow these threads to finish
explicitly is to break the connection. The close mechanisms will then
get a chance to run, hopefully designed to shut other resources down if
the Consumers are managing them.

Publishers wouldn't register Consumers, either, and presumably would
handle receiving of confirms, and so on, in the main thread, so would
not stop everything until ready to do so. Again, no Consumers: no
non-daemon threads.

So, it would seem to me that making MainLoop a daemon thread is the best
of all possible worlds.

Btw, historically the callbacks were executed on the MainLoop thread, so
making it a daemon was not an option. Recently the threading model
changed and so we haven't spotted that now the MainLoop thread can be a
daemon.

Would this meet your requirements?

Steve Powell  (a happy bunny)
----------yet more definitions from the SPD----------
corrugate (n.) T.V. soap scandal.
olympic (n.) A camp road-digger.
jamboree (n.) A conserve made from French cheese.

On 24 May 2012, at 10:41, Bart Prokop wrote:

> Hi Steve,
> 
> Many thanks for your prompt response. I have two proposals about improving it, while keeping all parties happy. Here they are:
> 
> 1. Add a boolean property "daemonBackgroundThreads" in ConnectionFactory. Based on that property the threads created by client library would be either daemon or not. The default would be false, what wouldn't occur any changes in behavior of existing applications.
> 
> 2. Alternatively the library could be rafactored that way that connection/publishing stuff be daemon threads, while listener stuff are non daemon. But as I explain further it is not best solution.
> 
> I see this problem in a bit other way, from perspective of developer, who not always can design clean solutions. Sometimes I have to wire and patch legacy systems ;). I think it is very common task to rabbitMQ users, who have to wire it into existing ecosystem. My case here is to allow some proprietary web server to send some messages during request processing. So basically I'm restricted to expose a single Java method to be called from the web script. I can create my own threads, connections, bookkeeping, connection failover, reconnect etc... but this all is hidden from the interface (single method "post") exposed to web scripting. The most ugly thing is that I have no other way of detecting that WebServer is going to die than adding shutdown hook to VM. And shutdown hook is fired, when all normal threads are about to die (along with daemon threads). This scenario is probably quite common for live integration of RabbitMQ into existing systems.
> 
> In general when dealing with messages we can have 3 basic scenarios:
> 1. Publisher.
> 2. Synchronous consumer.
> 3. Asynchronous consumer (listener on messages).
> 
> The idea of writing RabbitMQ clients (implied by costly and limited TCP/IP sockets) is to share a connection "per application" and write code that can reconnect in case of RabbitMQ node fail. So as soon as I get the connection, I keep it globally (shared). Usually it is my responsibility to clean on shutdown, but there are cases when I'm unable to do this. The shutdown hooks are handy in those scenarios.
> 
> Only the 3rd scenario (asynchronous consumer) may require non daemon threads. But it is from my experience not a live scenario. You usually do not write application, relaying on created behind the scene "main loop". Why? Because even if all what you do is to listen to messages, you still have to control the application state. You must start and stop that application as well as allow user/OS to close it. I personally for such micro-applications use Apache Daemons and run them as Windows services. On Linux you have jsvc to control such application. Surely "main loop" hidden in implementation is bad idea.
> 
> On Wed, May 23, 2012 at 5:18 PM, Steve Powell <steve at rabbitmq.com> wrote:
> Bartłomiej,
> 
> What a very good point. It is clear that the MainLoop thread ought to be
> a daemon thread in your circumstances, thank you for pointing this out.
> The only affect of this will be to allow the JVM to terminate if only
> daemon threads are available.
> 
> However, we have considered this problem before (internal bug 21110) and
> user apps that create a connection and register some consumers (which do
> all the work) ought to be able to terminate without accidentally killing
> the connection processing.
> 
> This means that there are circumstances in which the threads we create
> are required to be non-daemon, and circumstances where you would like
> them to be daemon.
> 
> The hook you implemented ought to work correctly even when the MainLoop
> is not a daemon, because closing the connection ought to make the
> MainLoop thread terminate normally. If it doesn't, there is a bug. Where
> at all possible you should attempt to close an open connection as a part
> of your termination processing as there are system resources that could
> be left high-and-dry if you do not.
> 
> There are other threads in the Java Client -- the executor worker
> threads used for Consumer callbacks. These are non-daemon, too. The hook
> should still work because shutting down the connection ought to shutdown
> the consumer work service, and in turn the executor (and its worker
> threads).
> 
> However, I could make them daemon threads as well, in case the main app
> terminates abruptly and expects to be able to terminate uncleanly as you
> describe.
> 
> Thank you for reporting this. I'll report back here on progress.
> 
> Steve Powell  (a happy bunny)
> ----------yet more definitions from the SPD----------
> corrugate (n.) T.V. soap scandal.
> olympic (n.) A camp road-digger.
> jamboree (n.) A conserve made from French cheese.
> 
> On 18 May 2012, at 16:27, Bartłomiej Prokop wrote:
> 
> > Hi,
> >
> > I'm using JAVA client for RabbitMQ (com.rabbitmq:amqp-client:2.8.1) to write a "jar component" capable of sending messages for some legacy system. The idea is to wrap all code that maintain the connection inside my component. This way, the client software deals only with very simple methods like "post" and is not aware of any connection handling. The connect/reconnect code is written and hidden from the legacy system.
> >
> > The problem I have faced is that Java client creates "behind the scene" some threads to manage connection - like:
> >          lines 299-301 of AMQConnection class.
> >         // start the main loop going
> >         new MainLoop("AMQP Connection " + getHostAddress() + ":" + getPort()).start();
> >         // after this point clear-up of MainLoop is triggered by closing the frameHandler.
> >
> > Unfortunatelly, those threads aren't "daemon" threads. So, when main application ends and appropriate connection closing not occurs, the VM won't terminate. My approach was to add some shutdown hook to close RabbitMQ connections if it is live inside my "jar component". But, due to those non-daemon threads, VM is not going ever to be terminated and shutdown hooks fired.
> >
> > It is a question to RabbitMQ driver developers, if the internal threads could be fired as daemon threads, could it be done in future releases?
> >
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-discuss at lists.rabbitmq.com
> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> 
> 
> 
> 
> -- 
> Bart Prokop
> tel. +48 509 258 502
> 
> This e-mail is intended solely for the addressee(s) and contains confidential information. Unauthorized distribution, modification or disclosure of its contents is unlawful. If you have received this e-mail in error, please notify the sender immediately by return e-mail. Please then delete the e-mail from your system and do not copy it or disclose its contents to any person. Email transmission cannot be guaranteed to be secure or error free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of email transmission. Information, opinions or conclusions contained in this message that do not relate to the official business of the senders employer or principal will be understood as neither given nor endorsed by it.