too bad you did not gcore it like I suggested, otherwise who knows... That's just the message it prints when the message parsing thread goes up.<div><br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Nov 26, 2012 at 8:25 PM, Tim St. Pierre <span dir="ltr"><<a href="mailto:fs-list@communicatefreely.net" target="_blank">fs-list@communicatefreely.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thanks for all the input everyone!<br>
<br>
I just had it happen again, this time at night when there were only 2 sessions active!<br>
<br>
I was in doing some other work, so I managed to see it happen and found a few interesting<br>
things.<br>
<br>
CPU load next to nothing, load average around 0.12.<br>
I used iSQL to test ODBC connectivity, and I could read and write to the freeswitch<br>
database using the same DSN that freeswitch is using.<br>
<br>
I could delete a SIP registration from the API<br>
<br>
Since I had things on a higher debug level, I saw this:<br>
<br>
freeswitch@stefan> 2012-11-26 21:21:51.914082 [CONSOLE] sofia.c:1144 MSG Thread Started<br>
<br>
As soon as that came up, everything started working again.<br>
<br>
What's that about?<br>
<span class="HOEnZb"><font color="#888888"><br>
-Tim<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
Ken Rice wrote:<br>
> Sofia is not single threaded except for in one spot deep in libsofia,<br>
>>From there, messages are handed off to a number of message queues for FS<br>
> core to handle as needed...<br>
><br>
> Check to see if anything that fs is depending on is blocking on info<br>
> retrieval like the databases or other areas...<br>
><br>
> K<br>
><br>
> On 11/25/12 9:51 AM, "Abaci" <<a href="mailto:abaci64@gmail.com">abaci64@gmail.com</a>> wrote:<br>
><br>
>> you mentioned that you use xml_curl, if your web server hangs it may<br>
>> hang sofia, iirc sofia is running a single thread and it will wait for<br>
>> the xml_curl response before continuing to the next request.<br>
>><br>
>> On 11/23/2012 1:13 PM, Steven Ayre wrote:<br>
>>> Any kind of DB backup running? Or any long-running queries (innotop is<br>
>>> great for highlighting queries that've been running a while, including<br>
>>> on non-innodb tables).<br>
>>><br>
>>> A global read lock, or queries waiting for a lock could block a db<br>
>>> update from the sofia profile thread but still allow read-only queries<br>
>>> (sofia status) to run.<br>
>>><br>
>>> -Steve<br>
>>><br>
>>><br>
>>><br>
>>> On 23 November 2012 15:32, Tim St. Pierre <<a href="mailto:fs-list@communicatefreely.net">fs-list@communicatefreely.net</a>><br>
>>> wrote:<br>
>>>> Hi Steven,<br>
>>>><br>
>>>> Thanks for the suggestions. I'm hoping once I get the upgrade done it will<br>
>>>> all go away.<br>
>>>> I have watched it happen at least once. I was on the phone at the time.<br>
>>>> Console activity<br>
>>>> more or less stopped, except for a few calls hanging up. The console<br>
>>>> remains responsive,<br>
>>>> and my call wasn't dropped for at least a minute or two (media timeout?). I<br>
>>>> was able to<br>
>>>> run sofia status and other commands that use the database, so I'm assuming<br>
>>>> that the<br>
>>>> connection was still working. All our media is runs through the box, so I<br>
>>>> think things<br>
>>>> are fine on the Ethernet level. I do see higher load averages - maybe 3-4,<br>
>>>> but that's the<br>
>>>> only obvious indication. It's not taking CPU beyond 10% or so.<br>
>>>><br>
>>>> We are using MySQL as the core DB and also as the DB backend for each sofia<br>
>>>> profile. This<br>
>>>> is connecting through ODBC of course.<br>
>>>><br>
>>>> If I can get the other kinks worked out, then I will try 1.2 stable in<br>
>>>> production and<br>
>>>> we'll see how it goes.<br>
>>>><br>
>>>> -Tim<br>
>>>><br>
>>>> Steven Ayre wrote:<br>
>>>>> Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)<br>
>>>>><br>
>>>>><br>
>>>>> As you've already acknowledged it's a very old version.<br>
>>>>><br>
>>>>> It's possible that your issue has already been found and fixed, but if<br>
>>>>> it hasn't then the code will have changed significantly since then and<br>
>>>>> you'd really need to reproduce it on the latest code for it to be<br>
>>>>> investigated.<br>
>>>>><br>
>>>>><br>
>>>>> As some general thoughts though, are you able to spot it happening while<br>
>>>>> it's happening or only afterwards?<br>
>>>>><br>
>>>>> If you're able to get on the system during one of those times look at<br>
>>>>> what else is happening. Is the load average/cpu usage/io high? Perhaps<br>
>>>>> something's running that's blocking all access or causing very high IO.<br>
>>>>><br>
>>>>> What DB backend are you using for Sofia? Is it possible that that's<br>
>>>>> hanging for a moment? For example if you're running a backup on the DB<br>
>>>>> that blocks all writes to the DB while Sofia is trying to update the DB<br>
>>>>> that perhaps would cause this.<br>
>>>>><br>
>>>>> Try running a SIP OPTIONS ping your your sofia profile from the<br>
>>>>> localhost during that time, which should exclude it being any issue on<br>
>>>>> the ethernet.<br>
>>>>><br>
>>>>> -Steve<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> On 22 November 2012 19:31, Tim St. Pierre <<a href="mailto:fs-list@communicatefreely.net">fs-list@communicatefreely.net</a><br>
>>>>> <mailto:<a href="mailto:fs-list@communicatefreely.net">fs-list@communicatefreely.net</a>>> wrote:<br>
>>>>><br>
>>>>> Hello,<br>
>>>>><br>
>>>>> I'm having a bit of an odd problem.<br>
>>>>><br>
>>>>> Intermittently, often every 2-3 days or so, Freeswitch stops<br>
>>>>> replying to SIP for about 5<br>
>>>>> minutes. I can't verify if it's EXACTLY 5 minutes, but it seems to<br>
>>>>> be pretty close.<br>
>>>>><br>
>>>>> During this time, no new registrations or invites can happen, but<br>
>>>>> existing calls stay<br>
>>>>> connected for at least a minute or two. In the logs, you can see<br>
>>>>> calls slowly hanging up<br>
>>>>> with "NORMAL_CLEARING". In 5 minutes, everything starts up again<br>
>>>>> with no word about it at<br>
>>>>> all in the logs.<br>
>>>>><br>
>>>>> When calls resume, I notice that the number of sessions returned by<br>
>>>>> the status command is<br>
>>>>> one higher than the actual number sessions returned by show<br>
>>>>> channels, or by looking in the<br>
>>>>> database. Every time this happens, the discrepancy increases by one.<br>
>>>>><br>
>>>>> The interruption happens on all SIP profiles, but calls originated<br>
>>>>> from the socket API<br>
>>>>> still work, insofar as they return with PROGRESS_TIMEOUT since the<br>
>>>>> profiles are still<br>
>>>>> running, but stuck.<br>
>>>>><br>
>>>>> We are using ODBC/MySQL for the core database, and the database<br>
>>>>> server only runs this<br>
>>>>> database and some basic PHP/xml-curl stuff.<br>
>>>>><br>
>>>>> We have 416 endpoints registered, and usually sit at about 30<br>
>>>>> sessions during the day.<br>
>>>>><br>
>>>>> This never happens at night, only during busier times, but not<br>
>>>>> necessarily busy hour.<br>
>>>>><br>
>>>>> I'm running on FreeBSD 8.2-RELEASE AMD 64(2 XEON cores, 4G ram)<br>
>>>>><br>
>>>>> Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)<br>
>>>>><br>
>>>>> Yes, I know it's old and I'm trying to upgrade, but I'm still having<br>
>>>>> some problems getting<br>
>>>>> all my phones to work properly with 1.2 stable. This is a<br>
>>>>> production system, so I can't<br>
>>>>> just blindly put out the newest release. Mostly, I need to buy<br>
>>>>> myself some time so that I<br>
>>>>> can get the kinks worked out of the latest version and then upgrade<br>
>>>>> the production box.<br>
>>>>><br>
>>>>> I'm grateful for any insights as to what could be happening, even if<br>
>>>>> a solution is just a<br>
>>>>> temporary workaround.<br>
>>>>><br>
>>>>> Thanks!<br>
>>>>><br>
>>>>> -Tim<br>
>>>>><br>
>>>>><br>
>>>>> _________________________________________________________________________<br>
>>>>> Professional FreeSWITCH Consulting Services:<br>
>>>>> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a> <mailto:<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a>><br>
>>>>> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
>>>>><br>
>>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
>>>>> <a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
>>>>><br>
>>>>> Official FreeSWITCH Sites<br>
>>>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>>>> <a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
>>>>> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
>>>>><br>
>>>>> FreeSWITCH-users mailing list<br>
>>>>> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
>>>>> <mailto:<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a>><br>
>>>>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
>>>>><br>
>>>>> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
>>>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> ------------------------------------------------------------------------<br>
>>>>><br>
>>>>> _________________________________________________________________________<br>
>>>>> Professional FreeSWITCH Consulting Services:<br>
>>>>> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
>>>>> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
>>>>><br>
>>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
>>>>> <a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
>>>>><br>
>>>>> Official FreeSWITCH Sites<br>
>>>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>>>> <a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
>>>>> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
>>>>><br>
>>>>> FreeSWITCH-users mailing list<br>
>>>>> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
>>>>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
>>>>> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
>>>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>>> _________________________________________________________________________<br>
>>>> Professional FreeSWITCH Consulting Services:<br>
>>>> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
>>>> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
>>>><br>
>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
>>>> <a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
>>>><br>
>>>> Official FreeSWITCH Sites<br>
>>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>>> <a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
>>>> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
>>>><br>
>>>> FreeSWITCH-users mailing list<br>
>>>> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
>>>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
>>>> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
>>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>> _________________________________________________________________________<br>
>>> Professional FreeSWITCH Consulting Services:<br>
>>> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
>>> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
>>><br>
>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
>>> <a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
>>><br>
>>> Official FreeSWITCH Sites<br>
>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>>> <a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
>>> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
>>><br>
>>> FreeSWITCH-users mailing list<br>
>>> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
>>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
>>> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
>>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>><br>
>> _________________________________________________________________________<br>
>> Professional FreeSWITCH Consulting Services:<br>
>> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
>> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
>><br>
>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
>> <a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
>><br>
>> Official FreeSWITCH Sites<br>
>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
>> <a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
>> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
>><br>
>> FreeSWITCH-users mailing list<br>
>> <a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
>> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
>> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
>> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
><br>
<br>
<br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Anthony Minessale II<br><br>FreeSWITCH <a href="http://www.freeswitch.org/">http://www.freeswitch.org/</a><br>ClueCon <a href="http://www.cluecon.com/">http://www.cluecon.com/</a><br>
Twitter: <a href="http://twitter.com/FreeSWITCH_wire">http://twitter.com/FreeSWITCH_wire</a><br><br>AIM: anthm<br><a href="mailto:MSN%3Aanthony_minessale@hotmail.com">MSN:anthony_minessale@hotmail.com</a><br>GTALK/JABBER/<a href="mailto:PAYPAL%3Aanthony.minessale@gmail.com">PAYPAL:anthony.minessale@gmail.com</a><br>
IRC: <a href="http://irc.freenode.net">irc.freenode.net</a> #freeswitch<br><br>FreeSWITCH Developer Conference<br><a href="mailto:sip%3A888@conference.freeswitch.org">sip:888@conference.freeswitch.org</a><br><a href="mailto:googletalk%3Aconf%2B888@conference.freeswitch.org">googletalk:conf+888@conference.freeswitch.org</a><br>
pstn:+19193869900<br>
</div>