[rabbitmq-discuss] Use a mysql databse table as the provider for rabbitmq queue

Wed Oct 17 11:40:35 BST 2012

I think I understand what you mean with the shared library.
However, in my case, RabbitMQ would only be installed if need be (meaning
more than one of the apps are present, and two of those need to be
synchronised for part of their data).

That said, using a shared library would require me to "include/import" said
library when I need to, therefore making me change my app code depending of
the situation I'm in.

And said library would only be required when there's a RabbitMQ available
anyway.

Now a bit further in your message you talk about a listener library.
I'd like to know a bit more about this.
How would an external library be able to listen to anything happening
within my app ?
Would it be listening on the DB queries ?

Cheers,
Ryan.

2012/10/17 Tim Watson <tim at rabbitmq.com>

>  Hi
>
> Please keep the list on CC - the info is useful for everyone. :)
>
>
> On 10/17/2012 11:05 AM, Ryan R. wrote:
>
> Hi,
> Thanks for that long and quite detailed answer.
>
> Based on what you're saying plugging the rabbitmq provider directly on the
> table would not be ideal and would most likely bring issues that I'd rather
> avoid.
>
>
> That's certainly my view.
>
>
> That said, could you provide some insight on a potentielly viable solution
> for my project.
> Reminder :
> I want to keep my applications separate in terms of DB and code (avoid
> triggers if possible)
> The reason for this is that depending on the situations, I would deploy
> only one, two, or more applications out of the whole project ( which
> roughly includes about 5 separate applications, each of which needs to be
> fully usable withou the others as well as with, without changing code not
> db config)
>
>
> From what I'm hearing, the primary concern here is that we want to be able
> to deploy any application in isolation and have it still work, regardless
> of whether or not the others are up an running. This is exactly the kind of
> scenario where messaging is supposed to help! If it was me, I would look at
> data first and foremost, and ask myself what information needs to be
> publicised. In each application I would publish that information, not via
> the database (which I would keep for internal use), but by publishing data
> to an exchange on the RabbitMQ broker. Publication is asynchronous and will
> not fail if there are no consumers - the data will simply get routed to a
> queue and sit there until someone eventually 'asks for it'. Then I would
> look at what external information my application(s) might want to consume.
> I would set up each application to consume only data it needs, again from
> the messaging broker not the database. Consuming can be asynchronous or
> synchronous - you can choose which - and worker threads or other similar
> tools can be used make this information available to the 'main application
> code' when required. A nice client library may do some or all of this
> 'worker/listener with callbacks' stuff for you.
>
> Now lets consider that design for a moment. Each application publishes
> data (asynchronously) regardless of whether or not someone is around to
> consume it. If there are no consumers, the data sits around in the queue.
> If you don't want this, then you can have the data expire immediately or
> after some timeout - rabbit will allow you do configure all of these
> patterns quite easily. The consuming part introduces slightly more new
> paths on your code, as you've got to decide how to get received messages
> from the consuming callbacks, back into the application code, but this
> isn't too hard. You can configure rabbit so that each message is consumed
> by only one application, or a copy is sent to all consumers, or you can
> route messages to specific consumers based on routing keys. The
> possibilities are numerous, and many good examples are available in the
> tutorials on our website!!!
>
> Now the only common thing that you've got to deal with in each application
> is plugging in the messaging sub system - i.e., each application needs to
> hook up to RabbitMQ using a client API and write a bit of glue code for
> publishing and consuming. Apart from that common factor, which if your
> applications are all written in the same language, can be delegated to a
> shared library, you've kept those applications completely decoupled.
>
> I hope that helps you to get started thinking about ways in which rabbit
> could help with your project. I do recommend going through the tutorials to
> get an idea of some of what's possible, and do feel free to come back here
> and ask about anything that's unclear or just to discuss ideas with the
> rest of the community.
>
> Cheers,
> Tim
>
>
> Cheers,
> Ryan.
>
> 2012/10/17 Tim Watson <tim at rabbitmq.com>
>
>>  Hi,
>>
>> On 10/17/2012 09:58 AM, Shadowalker wrote:
>>
>> Hi,
>>
>> I've just stumbled upon rabbitmq and I think it could very well help in a
>> project that I'm currently working on. But there's one thing that I'd like
>> to get some input about though :
>>
>> My project consist of one, two or more applications that, though
>> separated, need to keep some of their datas synchronized.
>>
>> For example say I have two use cases that could happen at the same time :
>>
>> First project has a app A and a app B : when app A insert some specific
>> data in its db i need to get a meesage to appB to do the corresponding
>> action on its own db
>> Second project has only app A and when I insert stuff in A's db it
>> shouldn't send anything anywhere.
>>
>> In both cases, I'll like to have app A and app B identical : I don't want
>> to have to change app A's code jsut because app B is here.
>>
>> So my question is : Is there anyway to plug the raabitmq consumer
>> directly onto a specific table in app A's database so that every time
>> something happens appB can come and consume the data.
>>
>>
>> Well, I'm not sure I would choose this approach personally, and I'll
>> explain why. But as I'm rather attention deficit I'll cut to the chase
>> first and say "yes - it is *possible* to do this - but you'll have to put
>> in a lot of work to make it happen" and as I said, I certainly wouldn't
>> recommend doing it. Now I'll explain why it's a bad idea IMHO and then I'll
>> explain how you can do it if you insist on following this path.
>>
>> First of all, you are creating a very tight coupling between the
>> application and the database schema, which will be difficult to work with
>> if the design changes (which most do) over time. Secondly, you are going to
>> be running code *inside the database server* which is fraught with danger.
>> Thirdly, it's going to be very difficult to test and/or debug when things
>> go wrong. Our industry tends to favour integrating systems using messaging
>> technology so much because doing so decouples applications from one
>> another, at least to a certain extent.
>>
>> The deceptively simple alternative to messaging, and it is often
>> ineffective and costly to develop and maintain, is to share data using
>> files and/or a shared database. Both applications share tables and/or
>> schemas and either poll for changes using worker processes (or threads) on
>> the one hand, or utilise notification features of the database system
>> itself on the other (e,g., Microsoft SQL Server Notification Services). But
>> this approach is full of problems, ranging from contention for reads/writes
>> on shared tables, to timing issues where both applications make very subtle
>> implicit assumptions about the order in which the various participants in
>> the system are interacting with the database, leading to functional errors.
>>
>> One reason that messaging based integration has been so successful is
>> that it gets us *away* from these problems, and allows applications to
>> simplify their model of interaction with the outside world. AMQP in
>> particular allows an application developer to think in terms of
>> asynchronously publishing events (for consumption elsewhere) or consuming
>> data in a variety of ways, based purely on the usage pattern required.
>>
>> Let's assume that you did implement this capability and whenever your
>> publishing application writes to the database, a message gets sent to an
>> exchange on a rabbit broker somewhere, which the other application is
>> consuming via some queue. What happens if the machine or data centre the
>> broker is running on crashes? What happens if the broker becomes overloaded
>> and starts to assert tcp back-pressure on the publisher - which, remember,
>> is running inside a database!? What happens if the database server has to
>> be restarted? And each of these 'what happens' questions must be answered
>> from the perspective of
>>
>> 1. the code running inside the database server
>> 2. the application writing to the database
>> 3. the application listening to the queue(s)
>> 4. the RabbitMQ broker itself
>>
>> That is an *awful* lot of moving parts, one of which is critical to your
>> infrastructure (i.e., the database server) and largely not under you
>> control (in so much as the semantics for running user defined code inside
>> the database server and the error handling behaviour of such, are likely to
>> be highly constrained). Even in an environment where you're running your
>> user defined function in an external operating system process (e.g., Oracle
>> UDFs written in java) the failure modes are terrifying. Even in an
>> environment which gives you a very clear picture of the constrained runtime
>> in which you're operating (e.g., custom .NET code running inside MS SQL
>> Server as a UDF or stored procedure) there are questions about how stable
>> and safe this approach is. And even in those environments, where stability
>> and safety *can* be understood and controlled to some extent, there is
>> still the question of semantics. If the publication fails, what does the
>> application writing to the database see? Does the insert/update fail? Doing
>> so would require making a synchronous call in the trigger/callback, which
>> would block the database server and create contention for the table!
>> Failing to make the call synchronous would lead to silent failures however!
>> There are no happy endings here, I can assure you.
>>
>> Now that I hope I've put you off this idea, here's how you can do it if
>> you disagree. Write a user defined function in C (consulting
>> http://dev.mysql.com/doc/refman/5.1/en/adding-udf.html or whatever
>> resources you have available to assist in this) and in your user defined
>> function, use the RabbitMQ native/C client (librabbitmq) to talk to rabbit.
>> The details of doing all of this are left as an exercise to the daring. Do
>> bare in mind that MySQL UDFs *must* be thread safe, that librabbitmq has no
>> explicit support for threading whatsoever and that blocking in a udf is
>> probably the first step on the road to bedlam.
>>
>> Cheers,
>> Tim
>>
>> Cheers,
>> Shadowalker.
>>
>>
>> _______________________________________________
>> rabbitmq-discuss mailing listrabbitmq-discuss at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20121017/81b4be2b/attachment.htm>