[Freeswitch-users] LUKE: I'm hit, but not bad - thread failure

[Tim St. Pierre]

Thanks!

This version was from Dec 31, so I will try a later version.  I can't run stable yet, as 
it has some problems on BSD like systems.

If I have my wits about me next time, I'll run a gcore before I kill it.  There's 
something about knowing your customers were down for an hour and a half that makes you 
think less about investigation, and more about just getting the switch running again.

-Tim

Anestis Mavro wrote:
> Hello Peter,
> 
> I faced a very similar (if not the same) situation a few weeks ago.
> I had exactly the same behavior with the stable branch, but with the CPU
> very very low. This is on CentOS (with a lot of cores and memory)
> 
> I went back to version 89e163a100 (November 11th, 2012) and the problem was
> gone.
> 
> A few days ago I updated to master git 40cbc5c0e5 (January 4th, 2013) and
> the problem doesn't exist anymore.
> 
> For me, it looks like there was something limiting the number of responses
> from FS to the clients. (New configuration for ODBC was already done!)
> 
> I would try to upgrade to latest git.
> 
> Good luck
> 
> 
> 
> 
> -----Original Message-----
> From: freeswitch-users-bounces at lists.freeswitch.org
> [mailto:freeswitch-users-bounces at lists.freeswitch.org] On Behalf Of Peter
> Olsson
> Sent: Friday, January 11, 2013 8:40 AM
> To: FreeSWITCH Users Help
> Subject: Re: [Freeswitch-users] LUKE: I'm hit, but not bad - thread failure
> 
> Please report to JIRA.
> 
> /Peter
> 
> 11 jan 2013 kl. 02:43 skrev "Tim St. Pierre"
> <fs-list at communicatefreely.net>:
> 
>> Okay, so I have a bit more detail, and my first post hasn't posted yet.
>>
>> FreeSWITCH Version 1.3.13b+git~20121230T190855Z~8859eb0b23 (git 8859eb0
> 2012-12-30 19:08:55Z
>> FreeBSD 8.3-RELEASE
>>
>> I start getting thread_failures several times per second.  It was so many
> entries that the 
>> logs started turning over at about 10 minute intervals.
>>
>> A few minutes later, I get the above quotes from Star Wars.
>>
>> After an hour and a half of no phone service, someone finds my pager
> number and I take a look.
>> Our alarms didn't go off, because Freeswitch would still respond to
> sockets, and it was 
>> still acknowledging SIP messages to a certain degree.
>>
>> fs_cli times out.
>> Freeswitch is using 96% CPU and around 900MB of RAM.  Earlier today, it
> was 500MB, but I 
>> have seen it higher than that before.  I have never seen it use more than
> 10% CPU on our 
>> little dual core Xeon machine.  I probably should get more cores, but it
> has never gone 
>> past 10% CPU usage.
>>
>> So I kill -9 Freeswitch, wait a few seconds, and start it up again.
> Everything comes back 
>> after the phones re-register, but I would really like to know why, and how
> to prevent it 
>> from happening again.
>>
>> This isn't an especially high call volume night, and I looked through the
> logs - it 
>> started just after a new session was created for an outgoing call.
> Nothing unusual about 
>> it.  For a while after, I could still see SIP registrations in the logs,
> but there weren't 
>> any SIP registrations in the database.  The process to delete the expired
> ones seems to 
>> have been working, but not the process to update them.
>>
>> This is essentially a forensic investigation at this point, but I wouldn't
> mind some 
>> suggestions as to what I should watch out for.
>>
>> -Tim
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>> !DSPAM:50ef69f032762133211294!
>>
> 
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
> 
> 
> 
> 
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
> 
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
> 
> 
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
> 
> 
> 
> 
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
> 
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org