[rabbitmq-discuss] Managing channels, esp wrt short lived publishers/consumers
James Aimonetti
james.aimonetti at gmail.com
Fri Mar 18 18:52:29 GMT 2011
Hey list,
I'm trying to figure out the best way to manage channel creation in my
app. Currently, any process that needs to publish or consume gets a
channel and I realize it is probably horribly inefficient, but as a
prototype I wasn't too concerned. Now that we're moving forward, I want
to get this cleaned up before I get too much further along.
I have a setup something like:
[App, App, App] <---> [AMQP] <--> Engine <--> Client
So a client makes a request to the Engine, which creates two processes,
one to publish events from the client, and one to consume commands from
the Apps and relay them to the client.
[App, App, App] <--> [AMQP] <-- ReqEvts <-- Client
[App, App, App] <--> [AMQP] --> ReqCtl --> Client
Ideally we'll be handling several hundreds of requests per second from
clients. So the ReqEvts process pumps 10 or so messages in rapid
succession at the beginning of a request, heartbeats every 20 or so
seconds, then pumps another 10 or so messages at the end of the request.
Requests may last 1 second (so setup and tear down bursts happen almost
on top of each other) or may last minutes or hours. Meanwhile, the
ReqCtl receives commands from a targeted exchange from the Apps to relay
to the Client.
Each App has a queue that it binds to a targeted exchange and one or
more topic exchanges. In general it receives most messages off the topic
exchanges and publishes to the targeted exchange.
So, hopefully that gives a broad overview. The Apps and the Engine
pieces are all on different servers.
So right now each server has an instance of rabbitmq running, and is
clustered to all the other servers (Apps and Engine, currently four
total servers but expandable). What I'm not sure about is how to manage
each server's connection and channels. Each server has a gen_server
managing the connection to rabbit, as well as the channels. So when a
pid requests to publish or consume or declare a queue, the gen_server
checks if the pid is known and uses that channel, or creates a new
channel and stores the association. With all of these potentially
short-lived consumers and publishers, I see a LOT of channel
creation/destruction going on and wonder if there's a better way? Should
I not have hidden channels behind a gen_server?
My initial thought is to have a dedicated channel for all publishers on
the server to send over, a dedicated channel for any queue declarations,
and a channel for each process that is consuming from a queue. Am I
serializing too much with this setup? Other ideas for how to manage
channels?
Any guidance is appreciated!
Thanks,
James
More information about the rabbitmq-discuss
mailing list