[rabbitmq-discuss] [Minimum Air Induction] Introducing Shovel: An AMQP Relay
dialtone at gmail.com
Sat Sep 20 09:56:01 BST 2008
-----BEGIN PGP SIGNED MESSAGE-----
small intro: I'm trying to use Shovel (LINK HERE) to create a push-
that can move loglines from a remote location to a centralized
storage, being a
remote location means that messages have to cross the internet.
On Sep 20, 2008, at 12:13 AM, Ben Hood wrote:
> On Sat, Sep 20, 2008 at 1:57 AM, Valentino Volonghi <dialtone at gmail.com
> > wrote:
>> Well... I'm currently rewriting the lib_shovel.erl file using a
>> version of it.
> See comment in previous email.
Yes, I am tracking tip, I didn't notice the big changes in lib_amqp.erl
so I'll refactor my code to use them.
>> SOURCE ---> LOCAL RABBITMQ (FORWARD) -----> CENTRAL RABBITMQ
>> And in CENTRAL RABBITMQ there would be all the listeners.
> Sounds like a good idea.
I hope so :). The main problem is the absolute necessity to not lose
any single one of the messages. Nothing can be lost.
>> The other thing I need is absolute reliability, if a message is not
>> transmitted don't remove it from
>> the queue and ack it when it is delivered. And if a connection is
>> instead of crashing it
>> should retry to connect and not deliver until it's connected back.
> Sounds reasonable. Currently Shovel does not have any reliability.
> Maybe you can elaborate on the concrete scenarios you are trying to
> cover with this.
Sure, basically the source of loglines would be a mochiweb server that
relevant requests would send messages to a known exchange.
Here is already the first 'problem', if the known exchange is down the
would be lost forever, this is simple enough though and rabbitmq would
run embedded in mochiweb together with shovel. Every queue durable and
every message too. In this case if mochiweb fails I won't have to
shovel disconnects it won't send lines to anyone and wouldn't even
them from the queue so nothing is lost here, if rabbitmq dies I hope
everything down with itself, traffic is rebalanced on the remaining
nothing is lost.
Right after this component there's another rabbitmq server, that we
local rabbitmq, which is local to the mochiweb server, in the same
This server would collect everything that various mochiweb+rabbitmq
servers send, persist it and forward it to a central location. Again,
durable so there should be no risk of losing messages.
In the central location there would be a final rabbitmq server that
will wait for
data. Attached to it there would be several consumers that fetch data
store it in various databases in small transactions (let's say one
every 50-100 log lines). Even here everything should be durable so there
should be no problem.
So, how should shovel behave:
Well, it should be pretty sure that every message was delivered to the
location, so I think its way of working would be:
1. receive message from embedded consumer
2. publish message to remote host
3. wait for ack
4. ack the rabbitmq container
5. the rabbitmq container at this point can remove the message
Then of course this procedure is repeated in each machine until we
Now... I'm not sure if there's an ack confirmation message so that the
100% sure that the confirmation was received, I suppose there isn't so
that the system will maybe have duplicates at the end and I'll have to
take care of
this somehow (any suggestions?).
Another small problem is the current state of Shovel where it
basically crashes when a
connection is dropped, a change that I would like to make (or I would
like to see) is that
it should be able to reconnect to the remote host with an exponential
backoff so that it
starts retransmitting as soon as possible.
Does this usecase sound reasonable?
I've read a bit of the archives and I see there are some problems with
and rabbitmq simply crashing... I hope this doesn't become a problem
for us, but...
let's say that the web servers generate messages of about 1K each
(could be much
less, but this is for the example's sake), now the local rabbitmq
instance is down for
some reason, basically mochiweb will remain alive as long as it has
means about 4GB/1MB/sec (let's say that the server generates 1000
second) that is about 66 minutes. If the problem instead is in the
basically each local rabbitmq is subject to the traffic of every web
server, if we have
8 webservers we obtain 4000/8MB/sec or about 8 minutes.
So it basically means that we have 8 minutes to react to such a
failure. Does this
also sound reasonable? and if so... What possible fixes can I look for?
Ultimately... does this sound like something that rabbitmq can be good
failures of machines can be anything from hardware failure to anything
>> I took part of the code I'm using now to replace lib_shovel.erl
>> from a guy
>> on IRC #erlang that
>> shared it, I had my version of it which was pretty similar but his
>> more cleaned up so
>> I'm using that now.
> Who was that?
Ah, can't remember.
> BTW a couple of points:
> 1. Bear in mind the refactoring that is being discussed on the list to
> get the core client into a 1.0 state;
Yep, will do.
> 2. I think this discussion should go over the mailing list, because
> this currently has the widest reach, so other people are made aware of
> what is going on and can potentially offer help to you - so if don't
> mind I will turn this is into a list topic;
This email is CCed/reply-to to the ML.
> 3. It would be really cool if your changes were in a github fork, so
> that they are publicly visible (obviously posting the patches to the
> list would be sharing it as well). An example of this is Peter who is
> maintaining a fork of the as3-amqp client, and I regularly fold his
> changes back into the upstream tree.
I'll start this tree too soon.
Valentino Volonghi aka Dialtone
Now running MacOS X 10.5
Home Page: http://www.twisted.it
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
-----END PGP SIGNATURE-----
More information about the rabbitmq-discuss