Hello everyone, I have a FreeSWITCH instance that has been running for months without an issue, until last Monday. The setup is for a hotline where we receive incoming calls transferred from the Client Cisco UCM to our FreeSWITCH instance, and the calls are handled by a custom Java application via ESL. As I said it was working flawlessly for weeks, but last Monday FreeSWITCH just stopped accepting new calls. I was able to see the SIP messages flowing back and forth from FS<->UCM, but looks like FreeSWITCH was unable to create new channels. Once I restarted FreeSWITCH everything was back to normal. Here are links to the SIP/console logs I was able to capture which covers both scenarios (non-working FS and working FS): Non-working: http://pastebin.freeswitch.org/20275 Working: http://pastebin.freeswitch.org/20274 As you can see, in the non-working scenario no channels were created on FreeSWITCH, so no 200 ACK message was sent back to UCM after the INVITE, which caused the UCM to send a CANCEL message after a few seconds and a 487 Request terminated from FS -> UCM. Restarting the application fixed the issue, but I have a couple of questions regarding this: 1) We have a lot of monitoring in place for all our applications, and I could probably alerted our Operations team if FreeSWITCH crashed or some error was logged in the FS log. However I was unable to found anything meaningful in the logs that indicates any sort of error. It just suddenly stopped creating new channels, and we get the alert/complain from the client - ouch! 2) I was also not sure what other information I should collect when something like this happens. As the application didn't crash it didn't create any coredump files. I know that restarting it fixes the problem, but that is not what the client wants to hear. Any recommendations on what to do/look at to proactively detect this kind of issues and/or how to collect the appropriate info to get the root cause will be greatly appreciated. Thank you all for your time, Hector
User-Agent: FreeSWITCH-mod_sofia/1.0.head-git-46f097c 2011-08-07 01-33-26 -0400
That's very old, you should update. There'll be a lot of bugfixes since then, and the age will make it harder to support. 1) **We have a lot of monitoring in place for all our applications,
and I could probably alerted our Operations team if FreeSWITCH crashed or some error was logged in the FS log. However I was unable to found anything meaningful in the logs that indicates any sort of error. It just suddenly stopped creating new channels, and we get the alert/complain from the client – ouch!
You could try sending a SIP OPTIONS packet from a monitoring system and check for a reply. Consecutive requests with no reply (timeout) could indicate a problem. One isn't enough in case the packet gets dropped. 2) **I was also not sure what other information I should collect when
something like this happens. As the application didn’t crash it didn’t create any coredump files. I know that restarting it fixes the problem, but that is not what the client wants to hear.
Check out gcore - it can create a coredump of a running process. Very useful for debugging. If you're compiling from git, you'll need to keep the source you compiled from (gdb needs the source to interpret the coredump). -Steve On 30 November 2012 17:06, Hector Geraldino <Hector.Geraldino@ipsoft.com>wrote:
Hello everyone,****
** **
I have a FreeSWITCH instance that has been running for months without an issue, until last Monday. The setup is for a hotline where we receive incoming calls transferred from the Client Cisco UCM to our FreeSWITCH instance, and the calls are handled by a custom Java application via ESL.* ***
** **
As I said it was working flawlessly for weeks, but last Monday FreeSWITCH just stopped accepting new calls. I was able to see the SIP messages flowing back and forth from FS<->UCM, but looks like FreeSWITCH was unable to create new channels. Once I restarted FreeSWITCH everything was back to normal. Here are links to the SIP/console logs I was able to capture which covers both scenarios (non-working FS and working FS):****
** **
Non-working: http://pastebin.freeswitch.org/20275****
Working: http://pastebin.freeswitch.org/20274****
** **
As you can see, in the non-working scenario no channels were created on FreeSWITCH, so no 200 ACK message was sent back to UCM after the INVITE, which caused the UCM to send a CANCEL message after a few seconds and a 487 Request terminated from FS -> UCM. Restarting the application fixed the issue, but I have a couple of questions regarding this:****
** **
**1) **We have a lot of monitoring in place for all our applications, and I could probably alerted our Operations team if FreeSWITCH crashed or some error was logged in the FS log. However I was unable to found anything meaningful in the logs that indicates any sort of error. It just suddenly stopped creating new channels, and we get the alert/complain from the client – ouch!****
**2) **I was also not sure what other information I should collect when something like this happens. As the application didn’t crash it didn’t create any coredump files. I know that restarting it fixes the problem, but that is not what the client wants to hear.****
** **
Any recommendations on what to do/look at to proactively detect this kind of issues and/or how to collect the appropriate info to get the root cause will be greatly appreciated. ****
** **
Thank you all for your time,****
Hector****
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
If you do get a coredump from gcore, see http://wiki.freeswitch.org/wiki/Reporting_Bugs#Creating_A_Backtrace_With_gdb... You will see in the backtrace what each thread is doing, which may highlight what's causing the non-response. I highly suggest as your very first step you upgrade to 1.2.5.2. This on its own may stop your issue if it's down to a bug that was fixed, and if you're still having the issue will put you on a version that can be supported. On 30 November 2012 18:20, Steven Ayre <steveayre@gmail.com> wrote:
User-Agent: FreeSWITCH-mod_sofia/1.0.head-git-46f097c 2011-08-07 01-33-26
-0400
That's very old, you should update. There'll be a lot of bugfixes since then, and the age will make it harder to support.
1) **We have a lot of monitoring in place for all our applications,
and I could probably alerted our Operations team if FreeSWITCH crashed or some error was logged in the FS log. However I was unable to found anything meaningful in the logs that indicates any sort of error. It just suddenly stopped creating new channels, and we get the alert/complain from the client – ouch!
You could try sending a SIP OPTIONS packet from a monitoring system and check for a reply. Consecutive requests with no reply (timeout) could indicate a problem. One isn't enough in case the packet gets dropped.
2) **I was also not sure what other information I should collect
when something like this happens. As the application didn’t crash it didn’t create any coredump files. I know that restarting it fixes the problem, but that is not what the client wants to hear.
Check out gcore - it can create a coredump of a running process. Very useful for debugging.
If you're compiling from git, you'll need to keep the source you compiled from (gdb needs the source to interpret the coredump).
-Steve
On 30 November 2012 17:06, Hector Geraldino <Hector.Geraldino@ipsoft.com>wrote:
Hello everyone,****
** **
I have a FreeSWITCH instance that has been running for months without an issue, until last Monday. The setup is for a hotline where we receive incoming calls transferred from the Client Cisco UCM to our FreeSWITCH instance, and the calls are handled by a custom Java application via ESL. ****
** **
As I said it was working flawlessly for weeks, but last Monday FreeSWITCH just stopped accepting new calls. I was able to see the SIP messages flowing back and forth from FS<->UCM, but looks like FreeSWITCH was unable to create new channels. Once I restarted FreeSWITCH everything was back to normal. Here are links to the SIP/console logs I was able to capture which covers both scenarios (non-working FS and working FS):****
** **
Non-working: http://pastebin.freeswitch.org/20275****
Working: http://pastebin.freeswitch.org/20274****
** **
As you can see, in the non-working scenario no channels were created on FreeSWITCH, so no 200 ACK message was sent back to UCM after the INVITE, which caused the UCM to send a CANCEL message after a few seconds and a 487 Request terminated from FS -> UCM. Restarting the application fixed the issue, but I have a couple of questions regarding this:****
** **
**1) **We have a lot of monitoring in place for all our applications, and I could probably alerted our Operations team if FreeSWITCH crashed or some error was logged in the FS log. However I was unable to found anything meaningful in the logs that indicates any sort of error. It just suddenly stopped creating new channels, and we get the alert/complain from the client – ouch!****
**2) **I was also not sure what other information I should collect when something like this happens. As the application didn’t crash it didn’t create any coredump files. I know that restarting it fixes the problem, but that is not what the client wants to hear.****
** **
Any recommendations on what to do/look at to proactively detect this kind of issues and/or how to collect the appropriate info to get the root cause will be greatly appreciated. ****
** **
Thank you all for your time,****
Hector****
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Reproduce it on latest GIT HEAD. Then if you catch it happening, gcore the process and post the bt on a jira search FS wiki for debugging FS and look for obtaining a backtrace On Fri, Nov 30, 2012 at 11:06 AM, Hector Geraldino < Hector.Geraldino@ipsoft.com> wrote:
Hello everyone,****
** **
I have a FreeSWITCH instance that has been running for months without an issue, until last Monday. The setup is for a hotline where we receive incoming calls transferred from the Client Cisco UCM to our FreeSWITCH instance, and the calls are handled by a custom Java application via ESL.* ***
** **
As I said it was working flawlessly for weeks, but last Monday FreeSWITCH just stopped accepting new calls. I was able to see the SIP messages flowing back and forth from FS<->UCM, but looks like FreeSWITCH was unable to create new channels. Once I restarted FreeSWITCH everything was back to normal. Here are links to the SIP/console logs I was able to capture which covers both scenarios (non-working FS and working FS):****
** **
Non-working: http://pastebin.freeswitch.org/20275****
Working: http://pastebin.freeswitch.org/20274****
** **
As you can see, in the non-working scenario no channels were created on FreeSWITCH, so no 200 ACK message was sent back to UCM after the INVITE, which caused the UCM to send a CANCEL message after a few seconds and a 487 Request terminated from FS -> UCM. Restarting the application fixed the issue, but I have a couple of questions regarding this:****
** **
**1) **We have a lot of monitoring in place for all our applications, and I could probably alerted our Operations team if FreeSWITCH crashed or some error was logged in the FS log. However I was unable to found anything meaningful in the logs that indicates any sort of error. It just suddenly stopped creating new channels, and we get the alert/complain from the client – ouch!****
**2) **I was also not sure what other information I should collect when something like this happens. As the application didn’t crash it didn’t create any coredump files. I know that restarting it fixes the problem, but that is not what the client wants to hear.****
** **
Any recommendations on what to do/look at to proactively detect this kind of issues and/or how to collect the appropriate info to get the root cause will be greatly appreciated. ****
** **
Thank you all for your time,****
Hector****
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
participants (3)
-
Anthony Minessale
-
Hector Geraldino
-
Steven Ayre