Hello everyone,
I have a FreeSWITCH instance that has been running for months without an issue, until last Monday. The setup is for a hotline where we receive incoming calls transferred from the Client Cisco UCM to our FreeSWITCH instance, and the calls
are handled by a custom Java application via ESL.
As I said it was working flawlessly for weeks, but last Monday FreeSWITCH just stopped accepting new calls. I was able to see the SIP messages flowing back and forth from FS<->UCM, but looks like FreeSWITCH was unable to create new channels.
Once I restarted FreeSWITCH everything was back to normal. Here are links to the SIP/console logs I was able to capture which covers both scenarios (non-working FS and working FS):
Non-working:
http://pastebin.freeswitch.org/20275
Working: http://pastebin.freeswitch.org/20274
As you can see, in the non-working scenario no channels were created on FreeSWITCH, so no 200 ACK message was sent back to UCM after the INVITE, which caused the UCM to send a CANCEL message after a few seconds and a 487 Request terminated
from FS -> UCM. Restarting the application fixed the issue, but I have a couple of questions regarding this:
1)
We have a lot of monitoring in place for all our applications, and I could probably alerted our Operations team if FreeSWITCH crashed or some error was logged in the FS log. However I was unable to found anything meaningful in the logs
that indicates any sort of error. It just suddenly stopped creating new channels, and we get the alert/complain from the client – ouch!
2)
I was also not sure what other information I should collect when something like this happens. As the application didn’t crash it didn’t create any coredump files. I know that restarting it fixes the problem, but that is not what the
client wants to hear.
Any recommendations on what to do/look at to proactively detect this kind of issues and/or how to collect the appropriate info to get the root cause will be greatly appreciated.
Thank you all for your time,
Hector