[Freeswitch-users] LUKE: I'm hit, but not bad - thread failure

[Tim St. Pierre]

Okay, so I have a bit more detail, and my first post hasn't posted yet.

FreeSWITCH Version 1.3.13b+git~20121230T190855Z~8859eb0b23 (git 8859eb0 2012-12-30 19:08:55Z
FreeBSD 8.3-RELEASE

I start getting thread_failures several times per second.  It was so many entries that the 
logs started turning over at about 10 minute intervals.

A few minutes later, I get the above quotes from Star Wars.

After an hour and a half of no phone service, someone finds my pager number and I take a look.

Our alarms didn't go off, because Freeswitch would still respond to sockets, and it was 
still acknowledging SIP messages to a certain degree.

fs_cli times out.
Freeswitch is using 96% CPU and around 900MB of RAM.  Earlier today, it was 500MB, but I 
have seen it higher than that before.  I have never seen it use more than 10% CPU on our 
little dual core Xeon machine.  I probably should get more cores, but it has never gone 
past 10% CPU usage.

So I kill -9 Freeswitch, wait a few seconds, and start it up again.  Everything comes back 
after the phones re-register, but I would really like to know why, and how to prevent it 
from happening again.

This isn't an especially high call volume night, and I looked through the logs - it 
started just after a new session was created for an outgoing call.  Nothing unusual about 
it.  For a while after, I could still see SIP registrations in the logs, but there weren't 
any SIP registrations in the database.  The process to delete the expired ones seems to 
have been working, but not the process to update them.

This is essentially a forensic investigation at this point, but I wouldn't mind some 
suggestions as to what I should watch out for.

-Tim