[rabbitmq-discuss] RabbitMQ server performance issue (erl.exe)

[Yogesh Ketkar]

> You said in your original message that there would only ever be 100 queues
Actually this is my test-bed. I wouldn't bother deleting the queues if
there were only 100.
Real life scenario, there could be thousands of queues with 5 to 10
messages on
each queue. And that is why I want to delete a queue, once all the
messages on it are consumed.

> It's not clear when call() gets called but
call() gets called by ExecutorService.

> You are declaring and binding queues on every delivery,
I don't have a choice, when delivery of message happens, destination
queue might have been deleted.
Is there a way in client API to find out
- If queue is already declared
- Is there a binding to the queue?

> http://www.rabbitmq.com/extensions.html#queue-leases
I had thought about this, but with x-expires, queues get deleted even
if they have
messages.

> http://www.rabbitmq.com/api-guide.html#tracer
I will and try and debug using this.

I will try and give you runnable code by removing all the scaffolding
code.

regards, Yogesh


On Jan 27, 4:48 pm, Simon MacMullen <si... at rabbitmq.com> wrote:
> I can't run code that is incomplete :-) but some things stand out:
>
> You are declaring and binding queues on every delivery, and then
> deleting them shortly after as well. On a system where a lot's going on
> in parallel this might lead to lots of retried transactions, which could
> lead to high CPU use. It's not clear when call() gets called but if that
> is in a tight loop then you could be declaring and immediately deleting
> queues constantly.
>
> You said in your original message that there would only ever be 100
> queues - if that is the case then can they just be created once? Or use
>   queue leases (http://www.rabbitmq.com/extensions.html#queue-leases) to
> make sure they get tidied up after some period of inactivity?
>
> You might want to connect your application through the Tracer
> (http://www.rabbitmq.com/api-guide.html#tracer) to see what is happening
> at the protocol level.
>
> Cheers, Simon
>
> On 27/01/12 08:52, Yogesh Ketkar wrote:
>
>
>
>
>
>
>
>
>
> > Hello Simon,
> > This is how code looks like
> > main(String[] argv) {
> >      // Read messages from MainQueue
> >      Channel ch = c.createChannel();
> >      ch.basicConsume("MainQueue", false, "", new
> > MainQueueConsumer(ch));
>
> >      // Read messages from dynamically declared queues
> >      ExecutorService es = Executors.newFixedThreadPool(tpSize);
> >      Collection<QueueProcessor>  vAppProcessorCollection = new
> > ArrayList<QueueProcessor>();
> >      while(true) {
> >          // Get the queues currently present using
> >          // http://<rabbitmq-server>:55672/api/queues/%2f/
> >          List<String>  qs = AMQPUtil.getvAppQueues();
> >          vAppProcessorCollection.clear();
> >          for(String q : qs) {
> >              QueueProcessor p = new QueueProcessor(c, q);
> >              vAppProcessorCollection.add(p);
> >          }
> >          es.invokeAll(vAppProcessorCollection);
> >      }
> > }
> > MainQueueConsumer, in handleDelivery does something like
> > public void handleDelivery() {
> >      // Get Queue name from payload
> >      c.queueDeclare(queueName, true, false, false,
> > getQueueProperties());
> >      c.queueBind(queueName, exchange, routingKey);
> >      c.basicPublish(exchange, routingKey, props, body);
> >      ch.basicAck(envelope.getDeliveryTag(), false);
> > }
>
> > And QueueProcessor does this in call() method
> > public Boolean call() {
> >      try {
> >          Channel ch = c.createChannel();
> >          while(true) {
> >              GetResponse r = ch.basicGet(queue, false);
> >              if(r != null) {
> >                  // process message
> >                  ch.basicAck(r.getEnvelope().getDeliveryTag(), false);
> >              } else break;
> >          }
> >          Util.queueDelete(ch, queue, true, true);
> >          if(ch.isOpen()) ch.close();
> >      } catch(Throwable th) {
> >          BaseUtil.logException(logger, th);
> >      }
> > }
>
> > Just couple of things
> > - If I don't delete the queue in call() method, things are much better
> >    and I am even able to 50000 messages on MainQueue without much
> > issues
> > - Ideally I would like to get rid of while(true) { ... basicGet ... }.
> >    Is there any pattern where I can do
> >    basicConsume() on the dynamically generated queue and
> >    remove the consumer and the queue when number of message on it are 0
>
> > regards, Yogesh
>
> > On Jan 25, 7:23 pm, Simon MacMullen<si... at rabbitmq.com>  wrote:
> >> On 25/01/12 08:30, Yogesh Ketkar wrote:
>
> >>> Any clues how to debug this?
>
> >> That sounds odd. 10k messages should be nothing.
>
> >> Can you share your code somehow so I can try to replicate this?
>
> >> Cheers, Simon
>
> >> --
> >> Simon MacMullen
> >> RabbitMQ, VMware
> >> _______________________________________________
> >> rabbitmq-discuss mailing list
> >> rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
> > _______________________________________________
> > rabbitmq-discuss mailing list
> > rabbitmq-disc... at lists.rabbitmq.com
> >https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
> --
> Simon MacMullen
> RabbitMQ, VMware
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq-disc... at lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss