[rabbitmq-discuss] Dropping out messages from a queue

Mon Jul 6 19:40:49 BST 2009

On Mon, Jul 6, 2009 at 7:04 PM, GAGAN ARORA<gaganarora.itm at gmail.com> wrote:
> Alexis
>
> When are you planning to release version for first option?

As I said below: "I am just checking with the guys to find out ...
when it will appear."

> Can you explain the second option?

Yes of course :-)

AMQP has a notion of 'custom exchange' for cases where fanout, direct,
and topic or headers don't make sense.  Anyone can define any
exchange, so what you could do is define an exchange which ignores
some set of bindings if some predicate (or more simply, flag) is true.
 The predicate you want in your case is something that means 'User A
is offline'.  This has exactly the same effect as using
exchange-exchange but it could be implemented by you today - or by us
- take your pick ;-)

alexis

>
> Thanks
> Gagan
>
> On Mon, Jul 6, 2009 at 11:06 PM, Alexis Richardson
> <alexis.richardson at gmail.com> wrote:
>>
>> Gagan
>>
>> One way to simplify the subscription management and get rid of 'heavy'
>> churn, is to use "exchange to exchange" bindings.  These are something
>> we have looked at - I am just checking with the guys to find out of
>> this solution would work well, and if so then when it will appear.
>>
>> Another option would be to create a custom exchange which used
>> presence to 'show' and 'hide' queue bindings.  This would be quite
>> neat, but it would involve some coding.
>>
>> alexis
>>
>>
>> On Sat, Jul 4, 2009 at 8:07 PM, GAGAN ARORA<gaganarora.itm at gmail.com>
>> wrote:
>> > Alexis,
>> >
>> > Thanks for your wonderful support and solutions.
>> >
>> > Firstly answers to your queries:
>> >
>> > 1. Are you using fanout exchanges for this (if you want one exchange
>> > per user, then the answer should be 'yes').
>> >
>> >>> No I am using topic exchanges because there will be three type of
>> >>> bindings of a queue with an excajnde. One for publishing messages to
>> >>> all
>> >>> users, another for one to one messaging and third for publishing
>> >>> message to
>> >>> a set of users.
>> >
>> > 2. Where is the RabbitMQ server running?  Is it on Server 1 or 2, or
>> > is it on a 3rd server?  It may not be that important but it would be
>> > good to know.
>> >
>> >>> RabbitMQ Server is running on 3rd Server.
>> >
>> >
>> > Back to original problem.
>> >
>> > Actually I thought of the solution provided by you before posting the
>> > query
>> > to you. But if I follow this solution I have to perform following steps
>> > whenever a user comes online:
>> > 1. Create a queue.
>> > 2. Create a binding of newly created queue with all exchanges of other
>> > users
>> > for retrieving messages published by other users.
>> > 3. Creating a binding with exchange of all users for one to one
>> > messaging.
>> > 4. Create a binding with exchanges of a set of users.
>> >
>> > When user goes offline following operation is to be performed:
>> > 1. Delete all the existing bindings of a queue.
>> > 2. Deleting the queue.
>> >
>> > Actually I want my online and offline processes light weighted.
>> > According to
>> > me this solution would have a high impact on performance and will
>> > increase
>> > time to come online and offline.
>> >
>> > Kindly let me know if this can be achieved in some other manner.
>> >
>> > Thanks
>> > Gagan Arora
>> >
>> >
>> >
>> >
>> >
>> > On Sat, Jul 4, 2009 at 3:23 PM, Alexis Richardson
>> > <alexis.richardson at gmail.com> wrote:
>> >>
>> >> Gagan,
>> >>
>> >> Hi - thank-you for sending the diagram and information which was very
>> >> useful.  There are several ways to implement the 'twitter style'
>> >> pattern that you have described.  See below for one way to do it that
>> >> illustrates the AMQP model.  It may not be the most efficient solution
>> >> in terms of performance as I shall point out.
>> >>
>> >> Two quick questions:
>> >>
>> >> 1. Are you using fanout exchanges for this (if you want one exchange
>> >> per user, then the answer should be 'yes').
>> >>
>> >> 2. Where is the RabbitMQ server running?  Is it on Server 1 or 2, or
>> >> is it on a 3rd server?  It may not be that important but it would be
>> >> good to know.
>> >>
>> >> OK - to your cases -
>> >>
>> >> Case1: User B sending his message to UserB Exchange which will route
>> >> message to User A queue but User A is not connected to any of servers.
>> >> At this point of  all messages sent to Queue A should be dropped.
>> >>
>> >> >>> One way to do this is to identify the lifecycle of Queue A with the
>> >> >>> presence of User A on servers 1 and 2.  In other words, when User A
>> >> >>> is not
>> >> >>> connected to server 1, and also not connected to server 2, then
>> >> >>> Queue A does
>> >> >>> not exist.  Note - This is not the only way to achieve what you
>> >> >>> want but it
>> >> >>> may be the least fiddly :-)
>> >>
>> >> Case2: User A connected to server 1 but UserA consumer is not yet
>> >> subscribed to UserA queue. If UserB sends any message to its exchange
>> >> which is routed to Queue of UserA. At this moment of time I want all
>> >> messages to be stored in queue.
>> >>
>> >> >>> One way to do this (continuing from the above) is to create Queue A
>> >> >>> immediately after UserA connects to server 1.  This is an action
>> >> >>> that server
>> >> >>> 1 can initiate on behalf of UserA.  After Queue A has been created,
>> >> >>> then it
>> >> >>> should be bound to all exchanges from which it will receive
>> >> >>> messages, such
>> >> >>> as UserB's exchange.  This will then have the stated effect you
>> >> >>> seek "If
>> >> >>> UserB sends any message to its exchange [it] is routed to Queue of
>> >> >>> UserA.".
>> >> >>>  Also, because UserA's consumer has not yet started consuming
>> >> >>> messages from
>> >> >>> Queue A, the messages will all remain stored in the queue.
>> >>
>> >> >>> In addition, if instead of proceeding to Case 3, we revert to Case
>> >> >>> 1,
>> >> >>> for example User A disconnects from Server 1, then Queue A can
>> >> >>> simply be
>> >> >>> deleted (along with its contents).  (See also Case 5 below)
>> >>
>> >> >>> However - while simple, the above is possibly not the most
>> >> >>> performant
>> >> >>> solution if (a) you are running RabbitMQ in clustered mode and (b)
>> >> >>> your
>> >> >>> Users generally follow a lot of other Users and (c) they connect
>> >> >>> and
>> >> >>> disconnect frequently.  This is because the effect of (c) is to
>> >> >>> frequuently
>> >> >>> update the routing table, possibly with a lot of binding/routing
>> >> >>> data (b),
>> >> >>> and then replicating it across the cluster (a).
>> >>
>> >> >>> Let us know if you want more info about other ways to do this.
>> >>
>> >>
>> >> Case3: User A is connected to both servers and UserA consumer is
>> >> subscribed to UserA queue. If UserB sends any message to its exchange
>> >> which is routed to Queue of UserA. At this moment of time I want all
>> >> messages stored in queue to be delievered plus any other incoming
>> >> message should also be routed.
>> >>
>> >> >>> In any case, to achieve this, just start consuming messages from
>> >> >>> Queue
>> >> >>> A.
>> >>
>> >> >>> I assume by 'delivered', you mean 'delivered to UserA's consumer'.
>> >> >>>  Where is that running?
>> >>
>> >>
>> >>
>> >>
>> >> Case4: User A is connected to both servers and UserA consumer is
>> >> subscribed to UserA queue. Now User disconnects from Server 2. If
>> >> UserB sends any message to its exchange which is routed to Queue of
>> >> UserA. Again I want to store all messages.
>> >>
>> >> >>> To achieve this, just stop consuming messages from Queue A.  They
>> >> >>> will
>> >> >>> be stored in Queue A.
>> >>
>> >>
>> >>
>> >> Case5: If user A is disconnected from both servers. Initial state is
>> >> achieved again.
>> >>
>> >> >>> When User A disconnects from Server 1, then Server 1 can tell
>> >> >>> RabbitMQ
>> >> >>> to delete Queue A (along with its contents)
>> >>
>> >> Cheers,
>> >>
>> >> alexis
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Fri, Jul 3, 2009 at 5:05 PM, GAGAN ARORA<gaganarora.itm at gmail.com>
>> >> wrote:
>> >> > Hi Alexis
>> >> > Attaching a doc having architecture diagram and cases desired. Hope
>> >> > this
>> >> > would make things more clear to you.
>> >> > Thanks
>> >> > Gagan Arora
>> >> >
>> >> > On Fri, Jul 3, 2009 at 8:23 PM, Alexis Richardson
>> >> > <alexis.richardson at gmail.com> wrote:
>> >> >>
>> >> >> Gagan,
>> >> >>
>> >> >> I'm a bit confused about what you are trying to do.  I think what
>> >> >> you
>> >> >> describe is possible but I am not sure what role is being played by
>> >> >> your two servers.  Is RabbitMQ running on both of them, one of them,
>> >> >> or neither?  You may find it easier to just use one RabbitMQ server.
>> >> >>
>> >> >> Can you send a drawing of what your desired configuration is?
>> >> >>
>> >> >> alexis
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Fri, Jul 3, 2009 at 3:08 PM, GAGAN
>> >> >> ARORA<gaganarora.itm at gmail.com>
>> >> >> wrote:
>> >> >> > Hi
>> >> >> > I am developing a system in which a user will connect to two
>> >> >> > servers.
>> >> >> > On
>> >> >> > connecting to first server he will become online and will send his
>> >> >> > presence
>> >> >> > update but he will receive presence updates only once he is
>> >> >> > connected
>> >> >> > to
>> >> >> > other server as his consumer subscribes to his queue after
>> >> >> > connecting
>> >> >> > to
>> >> >> > second server.
>> >> >> > I am using one exchange and one queue per user. So if one user
>> >> >> > comes
>> >> >> > online
>> >> >> > then he will send his presence update to his exchange and users
>> >> >> > bound
>> >> >> > to
>> >> >> > that will receive updates.
>> >> >> > I need to configure a queue which would hold messages for user who
>> >> >> > has
>> >> >> > just
>> >> >> > came online until he is connected to other server. After
>> >> >> > connecting
>> >> >> > to
>> >> >> > second server all messages will be consumed. All messages sent to
>> >> >> > queue
>> >> >> > will
>> >> >> > be dropped once user goes offline.
>> >> >> > Is it possible?
>> >> >> > Thanks
>> >> >> > Gagan Arora
>> >> >> > _______________________________________________
>> >> >> > rabbitmq-discuss mailing list
>> >> >> > rabbitmq-discuss at lists.rabbitmq.com
>> >> >> >
>> >> >> > http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>> >> >> >
>> >> >> >
>> >> >
>> >> >
>> >
>> >
>
>