[rabbitmq-discuss] Frequently getting {badmatch, {error, einval} from one of my rabbimq nodes

Tue Oct 25 15:12:22 BST 2011

I believe I've figured out how to duplicate this problem. It manifests when
many connections and channels are opened, send a few messages (around 20),
and are closed very fast (around 100 a second). Obviously this is not
advisable behavior for the library. It was a quick hack in my code that
worked in testing and so was forgotten about. As soon as I went back and
made a persistent connection and channel for sending messages the problem
went away.

-Max

On Fri, Sep 30, 2011 at 8:57 AM, Max Warnock <maxjwarnock at gmail.com> wrote:

> Has anyone had any more thoughts on this? I'm still getting this error and
> it's making my software queue up about 10 for every 100,000 messages in the
> unacked queue. If I restart my server it will process those messages just
> fine.
> -Max
>
>
> On Thu, Aug 18, 2011 at 4:26 PM, Max Warnock <maxjwarnock at gmail.com>wrote:
>
>> I have been getting the same error, you're not crazy Chak.  The exact
>> message for me is:
>>
>> =ERROR REPORT==== 18-Aug-2011::16:17:35 ===
>> ** Generic server <0.28256.0> terminating
>> ** Last message in was {inet_async,#Port<0.29961>,47384,
>>                                    {ok,<<0,10,0,51,206>>}}
>> ** When Server state == {state,#Port<0.29961>,<0.28252.0>,<0.28253.0>,
>>                                 {method,rabbit_framing_amqp_0_9_1},
>>                                {1,0,4}}
>> ** Reason for termination ==
>> ** {{badmatch,{error,einval}},
>>     [{amqp_main_reader,handle_inet_async,2},
>>      {gen_server,handle_msg,5},
>>      {proc_lib,init_p_do_apply,3}]}
>>
>> I get this when I have a large number of messages going over a single
>> connection and channel (topic exchange, subscription based pulling).  I'm
>> running
>> Erlang R13B04 (erts-5.7.5).
>>
>> Regards,
>> -Max
>>
>>
>> On Wed, Jul 27, 2011 at 12:38 PM, Alexandru Scvorţov <
>> alexandru at rabbitmq.com> wrote:
>>
>>> Hi Chak,
>>>
>>> I've been trying for a while, but I can't reproduce this.
>>>
>>> That error is coming from deep inside Erlang's inet driver.  As far as I
>>> can tell, the only way to get that error would be to call
>>> prim_inet:async_recv/3 on a socket with a {packet, Packet} setting
>>> different from 'raw'.  But this can't happen on any version of rabbit.
>>>
>>> Could you give us a more detailed description of the program?  Is it a
>>> plugin?  Is it doing anything special with the connection? (like setting
>>> socket options after the connection is established) Is there any chance
>>> we could see the code?
>>>
>>> You mentioned it starts at the very beginning.  Does this mean the
>>> problem goes away after some time?
>>>
>>> Cheers,
>>> Alex
>>>
>>>
>>> On Thu, Jul 21, 2011 at 07:48:29PM -0700, Chak Hedik wrote:
>>> > Hi Alex,
>>> >
>>> > Seems like this error is still there and it seems like either it
>>> > occurred when worker entering the loop and send another message to the
>>> > same queue within the loop or occurred when function retry, means
>>> > function call itself for a retry (pretty much like loop also).
>>> >
>>> > Thanks,
>>> > Chak
>>> >
>>> > On Jul 21, 6:40 am, Chak Hedik <chakhe... at gmail.com> wrote:
>>> > > Hi Alex,
>>> > >
>>> > > *Ubuntu Server 11.04 64-Bit
>>> > > *RabbitMQ 2.3.1 (Default ubuntu package)
>>> > > *Erlang Client 2.3.1 (Default ubuntu package)
>>> > > *Erlang R13B03 (Default ubuntu package)
>>> > >
>>> > > Is there something in the server logs that corresponds to this error?
>>> > > - That's the only thing appeared in the logs.
>>> > >
>>> > > You say your other nodes don't have similar issues.  Is there
>>> > > something
>>> > > special about this node?
>>> > > - Nothing special, pretty much the same but handle more messages than
>>> > > others. 1 worker handling 1 queue.
>>> > >
>>> > > It is strange behaviour and it happen at the very beginning where the
>>> > > worker up for the first time (no message in queue yet). I've created
>>> > > another node and put this worker there and will wait and see if
>>> > > similar error appear again or not today.
>>> > >
>>> > > Thanks for your reply.
>>> > >
>>> > > Chak
>>> > >
>>> > > On Jul 21, 5:14 am, Alexandru Scvorţov <alexan... at rabbitmq.com>
>>> wrote:
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > > Hi Chak,
>>> > >
>>> > > > Could you please answer the following questions:
>>> > > > * What OS are you using?
>>> > > > * What version of RabbitMQ?
>>> > > > * What version of the Erlang client?
>>> > > > * What version of Erlang?
>>> > >
>>> > > > > ** Reason for termination ==
>>> > > > > ** {{badmatch,{error,einval}},
>>> > > > >     [{amqp_main_reader,handle_inet_async,2},
>>> > > > >      {gen_server,handle_msg,5},
>>> > > > >      {proc_lib,init_p_do_apply,3}]}
>>> > >
>>> > > > Looks like something strange is taking down the socket, which in
>>> turn is
>>> > > > taking down the entire connection.  If you have more than one
>>> > > > connection, the others should be fine.
>>> > >
>>> > > > Is there something in the server logs that corresponds to this
>>> error?
>>> > >
>>> > > > You say your other nodes don't have similar issues.  Is there
>>> something
>>> > > > special about this node?
>>> > >
>>> > > > Cheers,
>>> > > > Alex
>>> > >
>>> > > > On Wed, Jul 20, 2011 at 12:12:32AM -0700, Chak Hedik wrote:
>>> > > > > Hi,
>>> > >
>>> > > > > I frequently getting this terminating error from one of my nodes
>>> :
>>> > >
>>> > > > > ** Generic server <0.28067.2> terminating
>>> > > > > ** Last message in was {inet_async,#Port<0.15349>,9049,
>>> > > > > {ok,<<0,10,0,51,206>>}}
>>> > > > > ** When Server state ==
>>> {state,#Port<0.15349>,<0.28063.2>,<0.28064.2>,
>>> > > > >
>>>  {method,rabbit_framing_amqp_0_9_1},
>>> > > > >                                {1,0,4}}
>>> > > > > ** Reason for termination ==
>>> > > > > ** {{badmatch,{error,einval}},
>>> > > > >     [{amqp_main_reader,handle_inet_async,2},
>>> > > > >      {gen_server,handle_msg,5},
>>> > > > >      {proc_lib,init_p_do_apply,3}]}
>>> > >
>>> > > > > While several other nodes with the same setup didn't have the
>>> same
>>> > > > > issue. May I know what it means and willl it affect my queues?How
>>> do I
>>> > > > > fix this?
>>> > >
>>> > > > > Thank,
>>> > > > > Chak
>>> > > > > _______________________________________________
>>> > > > > rabbitmq-discuss mailing list
>>> > > > > rabbitmq-disc... at lists.rabbitmq.com
>>> > > > >
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>> > >
>>> > > > _______________________________________________
>>> > > > rabbitmq-discuss mailing list
>>> > > > rabbitmq-disc... at lists.rabbitmq.comhttps://
>>> lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>> > >
>>> > > _______________________________________________
>>> > > rabbitmq-discuss mailing list
>>> > > rabbitmq-disc... at lists.rabbitmq.comhttps://
>>> lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>> > _______________________________________________
>>> > rabbitmq-discuss mailing list
>>> > rabbitmq-discuss at lists.rabbitmq.com
>>> > https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>> _______________________________________________
>>> rabbitmq-discuss mailing list
>>> rabbitmq-discuss at lists.rabbitmq.com
>>> https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/attachments/20111025/3eb1cd20/attachment.htm>