[rabbitmq-discuss] FW: Multiple consumers

[Ben Hood]

Sorry, I forwarded this accidentally to erlang-questions....

---------- Forwarded message ----------
From: Ben Hood <0x6e6562 at gmail.com>
Date: Aug 1, 2007 12:53 AM
Subject: Re: [rabbitmq-discuss] FW: Multiple consumers
To: Matthias Radestock <matthias at lshift.net>


Matthias,

Here is the latest patch based on the design discussion we have been
having. This demonstrates how the connection initiation, opening
channel 0, opening channel 1 and doing an access request for both the
direct and networked cases. Please bear in mind that I was
concentrating on getting a working prototype to you based on the
design you described, so there are a lot of rough edges that require
refactoring, cleaning up and polishing, as well as going through the
complete spec for each case (direct and network).

You can best see the flow by executing the amqp_client_test module
which has a test case for the direct and network cases.

To get a unified API, what I have done is to parameterize the client
with a direct driver and a network driver that implement the necessary
callbacks to provide connectivity in the 2 scenarios
(amqp_direct_driver and amqp_network_driver). By doing this I can use
amqp_send() and amqp_receive() functions in the higher level client in
a transparent fashion, so the actual protocol handling is common.

In the direct case, a new process for a client channel is spawned and
this maintains connectivity to the server peer.

In the network case, a new process for the initial AMQP network
connection is spawned which handles the connection tuning. This
process handles the socket connectivity (to be exact, there's a reader
process and a writer process). Then when a user calls open_channel, a
new amqp_client process is spawned for that channel, this process is
registered with the relevant socket reader/writer processes, and
normal message passing occurs from here on in. This way the same
send() and recv() for AMQP can be used in the direct and network
cases.

There are a lot of things that require attention, for example

- The way the new channel processes are send to the socket reader (I
have been reading the gen_tcp man pages for a cleaner way to do this)
- Making sure that exits are trapped correctly
- Taking out all of the io:format() statements (is a logging strategy
worth considering?)
- Streamlining the messages that get passed, so the amqp_send/receive
functions are simpler and easier to understand (there's probably
redundancy in there) and to get rid of some case statements that
currently exist.
- General refactoring in the client to make it neater and easier to maintain.
- Considering refactoring commonalities between the client and the
server, such as the header files and the server modules used by the
client
- I patched rabbit_framing.erl, which I have subsequently learned is a
generated file, so that needs cleaning up.
- The patch probably also contains changes to server modules that are
not entirely necessary, these need to be identified and reviewed.
- Do some real testing with it :-)
- Benchmark the direct and network drivers to see what difference it
makes, if any ;-)

But I just wanted to find out if this is going in the right direction
before I do too much on these points. Also, there are some
dependencies on refactoring the server modules.

So I'll just answer some of your comments now:

> > 1. In the direct case, the host and channel would be set in the
> > start() call, so just passing the Pid is fine.
>
> There is no 'host' in the direct case.

Fine. This is how it has been implemented.

> The initialisation sequence for the networked client is:
>
> 1) open socket,
>
> 2) spawn reader process, parameterised by the socket.
>
> 3) open channels by sending a message to the reader process. This should
> result in the spawning a new channel process, parameterised by the
> channel number, ReaderPid and WriterPid, and the necessary book-keeping
> (mapping channel numbers to pids) - i.e. everything
> rabbit_reader:open_channel does currently. The reply message to the
> sender of the 'open_channel' message should contain the Pid of the new
> channel process.

This is IMHO exactly how it works now.

>
> > 2. Should the processes be linked at all? That is, if a channel
> > process dies, should the user process die as well?

This requires some review to check that it is done properly in the
code, but is not really a big deal.

>
> Channel processes should be (and are) linked to their reader process. As
> for linking them to the user process, i.e. the process that initiated
> the opening of a channel, I reckon we should leave that up to the user.

OK.

HTH,

Ben
-------------- next part --------------
A non-text attachment was scrubbed...
Name: erlang_client_3.patch
Type: application/octet-stream
Size: 50801 bytes
Desc: not available
Url : http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20070801/59b50649/attachment.obj