Freeswitch stops handling register requests
Hello, We faced today the following situation: Freeswitch suddenly stopped to respond to answer to REGISTER requests. At the same time it created more and more channels. A lot of them „hung" in status „RINGING“, after some time all calls were rejected with 503 Maximum Calls In Progress. After a restart, everything went back to normal I cannot find any WARNING or ERRROR/CRIT messages in the log. Also there is nothing special in the syslog at the time the issue appeared. We are using freeswitch 1.6.17 on debian jessie installed from debian packages. Calls are handled via esl / mod_event socket. Is there any known issue with this version? Is there anything else I can check? Thanks and Regards Markus
What you describe could happen if your esl script suddenly sent many thousands of originate commands.
On May 5, 2017, at 11:58 AM, Markus Bönke <mbodbg@gmx.net> wrote:
Hello,
We faced today the following situation:
Freeswitch suddenly stopped to respond to answer to REGISTER requests. At the same time it created more and more channels. A lot of them „hung" in status „RINGING“, after some time all calls were rejected with 503 Maximum Calls In Progress.
After a restart, everything went back to normal
I cannot find any WARNING or ERRROR/CRIT messages in the log. Also there is nothing special in the syslog at the time the issue appeared.
We are using freeswitch 1.6.17 on debian jessie installed from debian packages. Calls are handled via esl / mod_event socket.
Is there any known issue with this version? Is there anything else I can check?
Thanks and Regards
Markus
I just checked the logs of our esl client and this was not the case. There is only one parrallel bridge to 8 destinations near the time where the problem occurred. Also the esl socket continued to receive and send events after freeswitch stopped to answer on register events. But maybe it has to do with the esl library like too many open threads or something like that. We recently updated the system from 1.4.26 to 1.6.17, everything was running for 2 days with around 150-200 calls parallel before the problem occurred. I’ve increased debug and monitoring, let’s see if it appears again in the next days. Thanks and regards Markus
Am 05.05.2017 um 19:03 schrieb Michael Jerris <mike@jerris.com>:
What you describe could happen if your esl script suddenly sent many thousands of originate commands.
On May 5, 2017, at 11:58 AM, Markus Bönke <mbodbg@gmx.net> wrote:
Hello,
We faced today the following situation:
Freeswitch suddenly stopped to respond to answer to REGISTER requests. At the same time it created more and more channels. A lot of them „hung" in status „RINGING“, after some time all calls were rejected with 503 Maximum Calls In Progress.
After a restart, everything went back to normal
I cannot find any WARNING or ERRROR/CRIT messages in the log. Also there is nothing special in the syslog at the time the issue appeared.
We are using freeswitch 1.6.17 on debian jessie installed from debian packages. Calls are handled via esl / mod_event socket.
Is there any known issue with this version? Is there anything else I can check?
Thanks and Regards
Markus
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
But maybe it has to do with the esl library like too many open threads or something like that.
Hello Markus, I remember, we had some issues a while ago, when Freeswitch fired a huge amount of esl events under heavy load, and we could not handle them in time. Therefore we built an ESL process which simply and quickly pushed the events to some Queues on a Redis Server. This process worked much quicker, than Freeswitch could fire events, so events may not be stuck inside Freeswitch waiting to be processed. On the other side we then had workers for each Redis Queue, who pulled the events and processed them. That way we can process all events from Freeswitch in time without losing any events. Maybe you experience something similar? Best regards Peter On 05/06/17 09:31, Markus Bönke wrote:
I just checked the logs of our esl client and this was not the case. There is only one parrallel bridge to 8 destinations near the time where the problem occurred. Also the esl socket continued to receive and send events after freeswitch stopped to answer on register events. But maybe it has to do with the esl library like too many open threads or something like that. We recently updated the system from 1.4.26 to 1.6.17, everything was running for 2 days with around 150-200 calls parallel before the problem occurred. I’ve increased debug and monitoring, let’s see if it appears again in the next days.
Thanks and regards
Markus
Am 05.05.2017 um 19:03 schrieb Michael Jerris <mike@jerris.com>:
What you describe could happen if your esl script suddenly sent many thousands of originate commands.
On May 5, 2017, at 11:58 AM, Markus Bönke <mbodbg@gmx.net> wrote:
Hello,
We faced today the following situation:
Freeswitch suddenly stopped to respond to answer to REGISTER requests. At the same time it created more and more channels. A lot of them „hung" in status „RINGING“, after some time all calls were rejected with 503 Maximum Calls In Progress.
After a restart, everything went back to normal
I cannot find any WARNING or ERRROR/CRIT messages in the log. Also there is nothing special in the syslog at the time the issue appeared.
We are using freeswitch 1.6.17 on debian jessie installed from debian packages. Calls are handled via esl / mod_event socket.
Is there any known issue with this version? Is there anything else I can check?
Thanks and Regards
Markus
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- With kind regards Peter Steinbach Telefaks Services GmbH mailto:lists (att) telefaks.de Internet: www.telefaks.de
Is there a way to see waiting ESL events to be processed? 2017-05-07 16:39 GMT+02:00 Peter Steinbach <lists@telefaks.de>:
But maybe it has to do with the esl library like too many open threads or something like that.
Hello Markus,
I remember, we had some issues a while ago, when Freeswitch fired a huge amount of esl events under heavy load, and we could not handle them in time. Therefore we built an ESL process which simply and quickly pushed the events to some Queues on a Redis Server. This process worked much quicker, than Freeswitch could fire events, so events may not be stuck inside Freeswitch waiting to be processed. On the other side we then had workers for each Redis Queue, who pulled the events and processed them. That way we can process all events from Freeswitch in time without losing any events.
Maybe you experience something similar?
Best regards Peter
On 05/06/17 09:31, Markus Bönke wrote:
I just checked the logs of our esl client and this was not the case. There is only one parrallel bridge to 8 destinations near the time where the problem occurred. Also the esl socket continued to receive and send events after freeswitch stopped to answer on register events. But maybe it has to do with the esl library like too many open threads or something like that. We recently updated the system from 1.4.26 to 1.6.17, everything was running for 2 days with around 150-200 calls parallel before the problem occurred. I’ve increased debug and monitoring, let’s see if it appears again in the next days.
Thanks and regards
Markus
Am 05.05.2017 um 19:03 schrieb Michael Jerris <mike@jerris.com>:
What you describe could happen if your esl script suddenly sent many thousands of originate commands.
On May 5, 2017, at 11:58 AM, Markus Bönke <mbodbg@gmx.net> wrote:
Hello,
We faced today the following situation:
Freeswitch suddenly stopped to respond to answer to REGISTER requests. At the same time it created more and more channels. A lot of them „hung" in status „RINGING“, after some time all calls were rejected with 503 Maximum Calls In Progress.
After a restart, everything went back to normal
I cannot find any WARNING or ERRROR/CRIT messages in the log. Also there is nothing special in the syslog at the time the issue appeared.
We are using freeswitch 1.6.17 on debian jessie installed from debian packages. Calls are handled via esl / mod_event socket.
Is there any known issue with this version? Is there anything else I can check?
Thanks and Regards
Markus
____________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/ options/freeswitch-users http://www.freeswitch.org
____________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- With kind regards Peter Steinbach
Telefaks Services GmbH mailto:lists (att) telefaks.de Internet: www.telefaks.de
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Gregor Nanger *CTO* t./f.: 00386 (0) 7 6000 308/309 • m:. 00386 (0)41 756485 • Infomedia d.o.o. • Jerebova 3, Novo mesto, Slovenia • www.infomedia.si
I just searched the subject line of some 28000 messages here - and did not find anyone looking into Real Time(RT)kernels. I remember many years ago using an RT kernel for some audio programs - didn't appear to make any difference - likely everything has just gotten that much faster. If anyone has worked with this - please let me know if you found any differences. -- -------------------------------------------------------------------------------- Karl Schmidt EMail Karl@xtronics.com Transtronics, Inc. WEB http://secure.transtronics.com 3209 West 9th Street Ph (785) 841-3089 Lawrence, KS 66049 FAX (785) 841-0434 Coordination does not run in my family, it stumbles. --------------------------------------------------------------------------------
Hi Karl, RT kernels are not about speed, but about guaranteed biggest latency, i.e. bounded latency in the kernel. That highest latency gets significantly improved if RT is set up correctly (which is not so straightforward). In case of VoIP the most latency in playing audio stream comes from network congestion, dropouts, etc., therefore it is external to the kernel and most likely there is not that many degrees of freedom in this space which real time system could improve. Piotr
We do use realtime priorities on the scheduler, however, so some threads can be elevated to the least possible ignore time from the kernel. On Mon, May 8, 2017 at 2:53 PM, Piotr Gregor <piotrek.gregor@gmail.com> wrote:
Hi Karl,
RT kernels are not about speed, but about guaranteed biggest latency, i.e. bounded latency in the kernel. That highest latency gets significantly improved if RT is set up correctly (which is not so straightforward). In case of VoIP the most latency in playing audio stream comes from network congestion, dropouts, etc., therefore it is external to the kernel and most likely there is not that many degrees of freedom in this space which real time system could improve.
Piotr
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
Hello Peter, thanks for the hint, but can it really happen that events stuck inside freeswitch and freeswitch partially crashed because if this? Or wasn't it more a problem of the Client that event’s were discarded if they haven’t been processed fast enough. How many calls did you handle on your server when the problem occurred, are you using inbound or outbound mode of mod_event_socket? Best regards Markus
Am 07.05.2017 um 16:39 schrieb Peter Steinbach <lists@telefaks.de>:
But maybe it has to do with the esl library like too many open threads or something like that.
Hello Markus,
I remember, we had some issues a while ago, when Freeswitch fired a huge amount of esl events under heavy load, and we could not handle them in time. Therefore we built an ESL process which simply and quickly pushed the events to some Queues on a Redis Server. This process worked much quicker, than Freeswitch could fire events, so events may not be stuck inside Freeswitch waiting to be processed. On the other side we then had workers for each Redis Queue, who pulled the events and processed them. That way we can process all events from Freeswitch in time without losing any events.
Maybe you experience something similar?
Best regards Peter
On 05/06/17 09:31, Markus Bönke wrote:
I just checked the logs of our esl client and this was not the case. There is only one parrallel bridge to 8 destinations near the time where the problem occurred. Also the esl socket continued to receive and send events after freeswitch stopped to answer on register events. But maybe it has to do with the esl library like too many open threads or something like that. We recently updated the system from 1.4.26 to 1.6.17, everything was running for 2 days with around 150-200 calls parallel before the problem occurred. I’ve increased debug and monitoring, let’s see if it appears again in the next days.
Thanks and regards
Markus
Am 05.05.2017 um 19:03 schrieb Michael Jerris <mike@jerris.com>:
What you describe could happen if your esl script suddenly sent many thousands of originate commands.
On May 5, 2017, at 11:58 AM, Markus Bönke <mbodbg@gmx.net> wrote:
Hello,
We faced today the following situation:
Freeswitch suddenly stopped to respond to answer to REGISTER requests. At the same time it created more and more channels. A lot of them „hung" in status „RINGING“, after some time all calls were rejected with 503 Maximum Calls In Progress.
After a restart, everything went back to normal
I cannot find any WARNING or ERRROR/CRIT messages in the log. Also there is nothing special in the syslog at the time the issue appeared.
We are using freeswitch 1.6.17 on debian jessie installed from debian packages. Calls are handled via esl / mod_event socket.
Is there any known issue with this version? Is there anything else I can check?
Thanks and Regards
Markus
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- With kind regards Peter Steinbach
Telefaks Services GmbH mailto:lists (att) telefaks.de Internet: www.telefaks.de
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (7)
-
Anthony Minessale
-
Gregor Nanger
-
Karl Schmidt
-
Markus Bönke
-
Michael Jerris
-
Peter Steinbach
-
Piotr Gregor