[rabbitmq-discuss] erlang client API: transactions

[Valentino Volonghi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


On Sep 23, 2008, at 12:15 PM, Valentino Volonghi wrote:

> And here: http://github.com/dialtone/shovel/tree/master


Using all these changes and improvements I managed to write a  
prototype of what I'll need.
There are still a few problems though, the first is the fact that  
tx.commit is synchronous I think.

When I try to load the server with a big number of requests (like 50k  
with 200 concurrency) I
see that the tx.commit goes to timeout in shovel and on the other side  
I see that the persister
is timing out, I suspect it means that it's simply taking a lot of  
time and erlang thinks that it's
a timeout.

Now... I don't have a clear idea of what's the best solution for this  
problem actually. The good thing
though is that when after the crash I restarted the webserver with  
rabbitmq and shovel embedded
I started receiving lines again on the other side, which means that  
basically this way of doing things
is definitely a step in the right direction.

I suspect part of the reason for this timeout is the use of the same  
machine for running the test and
receiving log lines from the test web server (using nice avoid the  
timeout problem). Maybe this
problem should be dealt with at the supervisor level and it should  
simply stop and restart shovel
after a couple of seconds. Does this sound reasonable?

Another good news is that I see about 2700 req/sec handled if I run ab  
directly on the webserver (
so it could even be higher).

I have another question though: how do transactions work? Are the  
messages in a single transaction
sent together or they are just sent separately? Because if it's the  
latter then gzip cannot really do
much if the size of the packet is less than 1500 bytes, it'll be a bit  
faster but still just a single packet.
Am I wrong?

- - Shovel:

=CRASH REPORT==== 23-Sep-2008::20:49:12 ===
   crasher:
     pid: <0.167.0>
     registered_name: shovel
     exception exit: {timeout,
                         {gen_server,call,[<0.172.0>,{call, 
{'tx.commit'}}]}}
       in function  gen_server:terminate/6
     initial call: gen:init_it(gen_server,<0.166.0>,<0.166.0>,
                               {local,shovel},
                               shovel,"192.168.1.133",[])
     ancestors: [shovel_supervisor,<0.165.0>]

- - Remote RabbitMQ:

=ERROR REPORT==== 23-Sep-2008::13:49:32 ===
** Generic server <0.127.0> terminating
** Last message in was {commit,{{11,<0.288.0>},7028}}
** When Server state == {q,{amqqueue,
                                 
{resource,<<"/">>,queue,<<"impressions">>},
                                true,false,[],
                                [{binding_spec,
                                      
{resource,<<"/">>,exchange,<<"flow">>},
                                     <<"*.impressions">>,[]}],
                                <0.127.0>},
                            none,none,true,28056,
                            {[],[]},
                            {[{<0.233.0>,
                               {consumer,
                                   <<"amq.ctag- 
QdWeELvdBEl2V21Hwz1gMg==">>,
                                   true}}],
                             []}}
** Reason for termination ==
** {timeout,
        {gen_server,call,
            [rabbit_persister,
             {commit_transaction,
                 {{{11,<0.288.0>},7028},
                  {resource,<<"/">>,queue,<<"impressions">>}}}]}}

=INFO REPORT==== 23-Sep-2008::13:49:32 ===
closing TCP connection <0.282.0> from 192.168.1.16:44937

=CRASH REPORT==== 23-Sep-2008::13:49:32 ===
   crasher:
     pid: <0.127.0>
     registered_name: []
     exception exit: {timeout,
                         {gen_server,call,
                             [rabbit_persister,
                              {commit_transaction,
                                  {{{11,<0.288.0>},7028},
                                   {resource,<<"/">>,queue,
                                       <<"impressions">>}}}]}}
       in function  gen_server:terminate/6
     initial call: gen:init_it(gen_server,<0.123.0>,<0.123.0>,
                               rabbit_amqqueue_process,
                               {amqqueue,
                                    
{resource,<<"/">>,queue,<<"impressions">>},
                                   true,false,[],
                                   [{binding_spec,
                                         
{resource,<<"/">>,exchange,<<"flow">>},
                                        <<"*.impressions">>,[]}],
                                   <0.127.0>},
                               [])


- --
Valentino Volonghi aka Dialtone
Now running MacOS X 10.5
Home Page: http://www.twisted.it
http://www.adroll.com

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iEYEARECAAYFAkjZYaIACgkQ9Llz28widGXBpQCgledTB+l04gCFifZ2CxlswgdQ
HnkAn0ILSCDkd4W8Hix3TRQRbvhgoxM2
=IzHa
-----END PGP SIGNATURE-----