User-Agent: FreeSWITCH-mod_sofia/1.0.head-git-46f097c 2011-08-07 01-33-26 -0400

That's very old, you should update. There'll be a lot of bugfixes since then, and the age will make it harder to support.

1)      We have a lot of monitoring in place for all our applications, and I could probably alerted our Operations team if FreeSWITCH crashed or some error was logged in the FS log. However I was unable to found anything meaningful in the logs that indicates any sort of error. It just suddenly stopped creating new channels, and we get the alert/complain from the client – ouch!

You could try sending a SIP OPTIONS packet from a monitoring system and check for a reply. Consecutive requests with no reply (timeout) could indicate a problem. One isn't enough in case the packet gets dropped.

2)      I was also not sure what other information I should collect when something like this happens. As the application didn’t crash it didn’t create any coredump files. I know that restarting it fixes the problem, but that is not what the client wants to hear.

Check out gcore - it can create a coredump of a running process. Very useful for debugging.

If you're compiling from git, you'll need to keep the source you compiled from (gdb needs the source to interpret the coredump).

-Steve




On 30 November 2012 17:06, Hector Geraldino <Hector.Geraldino@ipsoft.com> wrote:

Hello everyone,

 

I have a FreeSWITCH instance that has been running for months without an issue, until last Monday. The setup is for a hotline where we receive incoming calls transferred from the Client Cisco UCM to our FreeSWITCH instance, and the calls are handled by a custom Java application via ESL.

 

As I said it was working flawlessly for weeks, but last Monday FreeSWITCH just stopped accepting new calls. I was able to see the SIP messages flowing back and forth from FS<->UCM, but looks like FreeSWITCH was unable to create new channels. Once I restarted FreeSWITCH everything was back to normal. Here are links to the SIP/console logs I was able to capture which covers both scenarios (non-working FS and working FS):

 

Non-working: http://pastebin.freeswitch.org/20275

Working: http://pastebin.freeswitch.org/20274

 

As you can see, in the non-working scenario no channels were created on FreeSWITCH, so no 200 ACK message was sent back to UCM after the INVITE, which caused the UCM to send a CANCEL message after a few seconds and a 487 Request terminated from FS -> UCM. Restarting the application fixed the issue, but I have a couple of questions regarding this:

 

1)      We have a lot of monitoring in place for all our applications, and I could probably alerted our Operations team if FreeSWITCH crashed or some error was logged in the FS log. However I was unable to found anything meaningful in the logs that indicates any sort of error. It just suddenly stopped creating new channels, and we get the alert/complain from the client – ouch!

2)      I was also not sure what other information I should collect when something like this happens. As the application didn’t crash it didn’t create any coredump files. I know that restarting it fixes the problem, but that is not what the client wants to hear.

 

Any recommendations on what to do/look at to proactively detect this kind of issues and/or how to collect the appropriate info to get the root cause will be greatly appreciated.

 

Thank you all for your time,

Hector


_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server
http://www.cudatel.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://wiki.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org