freeswitch HANGING (process exists but stops responding)
Hello all. I've been experiencing a problem with freeswitch where it stops responding (doesn't answer any SIP REQUESTS) and produces the following output in the log file every time the symptom happens: 2013-08-12 15:18:31.488906 [WARNING] switch_event.c:552 Create event dispatch thread 1 2013-08-12 15:18:31.578906 [WARNING] switch_event.c:554 Create additional event dispatch thread 2 2013-08-12 15:18:31.598907 [WARNING] switch_event.c:554 Create additional event dispatch thread 3 2013-08-12 15:18:31.648904 [WARNING] switch_event.c:554 Create additional event dispatch thread 4 2013-08-12 15:18:31.728905 [WARNING] switch_event.c:554 Create additional event dispatch thread 5 2013-08-12 15:19:04.348914 [CONSOLE] sofia.c:1602 MSG Thread 4 Started After this, i can see that the process exists and i can get to the fs_cli prompt, although none of the normal command like: show calls work. The command simply hangs without returning back a response. I think this could be memory related, although can't find a reasonable explanation to that. I had 4GB of physical RAM available + 512MB for SWAP on a CentOS x86_64 machine and the OS was using like about 800MB but if i considered the cached amount of memory, it was like a total of 3.8GB which from what i understand is normal in linux, since the kernel tries to cache all the available memory over time. Nevertheless i've decided to give the machine more 4GB RAM and SWAP is now 16GB, so now i've 8GB total for physical RAM and 16GB SWAP which solved the problem at least temporarily and made freeswitch stop getting back into the symptom described above. I can see that the cached memory is again growing over time. I still don't know if I'll end up with the same symptom again when the total used memory (used+ cached) gets near the 8GB. Anyone has had this symptom before too? Some feedback is appreciated. Thanks. -- *Nuno Miguel Reis* | *Unified Communication** Systems* M. +351 913907481 | nreis@wavecom.pt WAVECOM-Soluções Rádio, S.A. Cacia Park | Rua do Progresso, Lote 15 3800-639 AVEIRO | Portugal T. +351 309 700 225 | F. +351 234 919 191 *GPS<http://maps.google.com/maps/ms?msa=0&msid=202333747613191340808.0004b4b227a6144f0df88>| www.wavecom.pt** <http://www.wavecom.pt/>* [image: Description: Description: WavecomSignature]<http://www.wavecom.pt/pt/wavecom/premios.php> [image: Publicity] <http://www.wavecom.pt/pt/mail_eventos.php>
Another eventually important tip: This is happening at least with freeswitch versions 1.2.3 and 1.2.8. Thanks. -- *Nuno Miguel Reis* | *Unified Communication** Systems* M. +351 913907481 | nreis@wavecom.pt WAVECOM-Soluções Rádio, S.A. Cacia Park | Rua do Progresso, Lote 15 3800-639 AVEIRO | Portugal T. +351 309 700 225 | F. +351 234 919 191 *GPS<http://maps.google.com/maps/ms?msa=0&msid=202333747613191340808.0004b4b227a6144f0df88>| www.wavecom.pt** <http://www.wavecom.pt/>* [image: Description: Description: WavecomSignature]<http://www.wavecom.pt/pt/wavecom/premios.php> [image: Publicity] <http://www.wavecom.pt/pt/mail_eventos.php> On Mon, Aug 12, 2013 at 8:29 PM, Nuno Reis <nreis@wavecom.pt> wrote:
Hello all.
I've been experiencing a problem with freeswitch where it stops responding (doesn't answer any SIP REQUESTS) and produces the following output in the log file every time the symptom happens:
2013-08-12 15:18:31.488906 [WARNING] switch_event.c:552 Create event dispatch thread 1 2013-08-12 15:18:31.578906 [WARNING] switch_event.c:554 Create additional event dispatch thread 2 2013-08-12 15:18:31.598907 [WARNING] switch_event.c:554 Create additional event dispatch thread 3 2013-08-12 15:18:31.648904 [WARNING] switch_event.c:554 Create additional event dispatch thread 4 2013-08-12 15:18:31.728905 [WARNING] switch_event.c:554 Create additional event dispatch thread 5 2013-08-12 15:19:04.348914 [CONSOLE] sofia.c:1602 MSG Thread 4 Started
After this, i can see that the process exists and i can get to the fs_cli prompt, although none of the normal command like: show calls work. The command simply hangs without returning back a response. I think this could be memory related, although can't find a reasonable explanation to that. I had 4GB of physical RAM available + 512MB for SWAP on a CentOS x86_64 machine and the OS was using like about 800MB but if i considered the cached amount of memory, it was like a total of 3.8GB which from what i understand is normal in linux, since the kernel tries to cache all the available memory over time. Nevertheless i've decided to give the machine more 4GB RAM and SWAP is now 16GB, so now i've 8GB total for physical RAM and 16GB SWAP which solved the problem at least temporarily and made freeswitch stop getting back into the symptom described above. I can see that the cached memory is again growing over time. I still don't know if I'll end up with the same symptom again when the total used memory (used+ cached) gets near the 8GB.
Anyone has had this symptom before too? Some feedback is appreciated.
Thanks.
--
*Nuno Miguel Reis* | *Unified Communication** Systems* M. +351 913907481 | nreis@wavecom.pt WAVECOM-Soluções Rádio, S.A. Cacia Park | Rua do Progresso, Lote 15 3800-639 AVEIRO | Portugal T. +351 309 700 225 | F. +351 234 919 191 *GPS<http://maps.google.com/maps/ms?msa=0&msid=202333747613191340808.0004b4b227a6144f0df88>| www.wavecom.pt** <http://www.wavecom.pt/>*
[image: Description: Description: WavecomSignature]<http://www.wavecom.pt/pt/wavecom/premios.php>
[image: Publicity] <http://www.wavecom.pt/pt/mail_eventos.php>
You might want to update to lastest and test then it could have already been resolved... On 8/12/13 2:35 PM, "Nuno Reis" <nreis@wavecom.pt> wrote:
Another eventually important tip:
This is happening at least with freeswitch versions 1.2.3 and 1.2.8.
Thanks. --
Nuno Miguel Reis | Unified Communication Systems M. +351 913907481 | nreis@wavecom.pt <mailto:nreis@wavecom.pt> WAVECOM-Soluções Rádio, S.A. Cacia Park | Rua do Progresso, Lote 15 3800-639 AVEIRO | Portugal T. +351 309 700 225 | F. +351 234 919 191 GPS <http://maps.google.com/maps/ms?msa=0&msid=202333747613191340808.0004b4b227a6... 44f0df88> | www.wavecom.pt <http://www.wavecom.pt/> <http://www.wavecom.pt/>
<http://www.wavecom.pt/pt/wavecom/premios.php>
<http://www.wavecom.pt/pt/mail_eventos.php>
On Mon, Aug 12, 2013 at 8:29 PM, Nuno Reis <nreis@wavecom.pt> wrote:
Hello all.
I've been experiencing a problem with freeswitch where it stops responding (doesn't answer any SIP REQUESTS) and produces the following output in the log file every time the symptom happens:
2013-08-12 15:18:31.488906 [WARNING] switch_event.c:552 Create event dispatch thread 1 2013-08-12 15:18:31.578906 [WARNING] switch_event.c:554 Create additional event dispatch thread 2 2013-08-12 15:18:31.598907 [WARNING] switch_event.c:554 Create additional event dispatch thread 3 2013-08-12 15:18:31.648904 [WARNING] switch_event.c:554 Create additional event dispatch thread 4 2013-08-12 15:18:31.728905 [WARNING] switch_event.c:554 Create additional event dispatch thread 5 2013-08-12 15:19:04.348914 [CONSOLE] sofia.c:1602 MSG Thread 4 Started
After this, i can see that the process exists and i can get to the fs_cli prompt, although none of the normal command like: show calls work. The command simply hangs without returning back a response. I think this could be memory related, although can't find a reasonable explanation to that. I had 4GB of physical RAM available + 512MB for SWAP on a CentOS x86_64 machine and the OS was using like about 800MB but if i considered the cached amount of memory, it was like a total of 3.8GB which from what i understand is normal in linux, since the kernel tries to cache all the available memory over time. Nevertheless i've decided to give the machine more 4GB RAM and SWAP is now 16GB, so now i've 8GB total for physical RAM and 16GB SWAP which solved the problem at least temporarily and made freeswitch stop getting back into the symptom described above. I can see that the cached memory is again growing over time. I still don't know if I'll end up with the same symptom again when the total used memory (used+ cached) gets near the 8GB.
Anyone has had this symptom before too? Some feedback is appreciated.
Thanks. --
Nuno Miguel Reis | Unified Communication Systems M. +351 913907481 | nreis@wavecom.pt <mailto:nreis@wavecom.pt> WAVECOM-Soluções Rádio, S.A. Cacia Park | Rua do Progresso, Lote 15 3800-639 AVEIRO | Portugal T. +351 309 700 225 | F. +351 234 919 191 GPS <http://maps.google.com/maps/ms?msa=0&msid=202333747613191340808.0004b4b227a6 144f0df88> | www.wavecom.pt <http://www.wavecom.pt/> <http://www.wavecom.pt/>
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org irc.freenode.net #freeswitch
Hi Nino, I do experience very similar problem with latest stable: FS 1.2.12. Please let me know if you have managed resolving that. So, eventually my FS stops responding, generating events, requesting xml_curl_auth, etc. I can start fs_cli but when I do want to have "show bridged_calls" and enter sh and press tab "sho" appears and fs_cli totally hangs - i have falled back to 1.1beta1 where I do not saw that problem. But I do not see "Create event dispatch thread" messages in the log. In my setup I have quite busy installation with around 1000 active SIP registrations with ~100 simultaneous calls, and ESL controller over the network, so EVENT activity is quite intensive. Regards, Anton. 2013/8/12 Ken Rice <krice@freeswitch.org>
You might want to update to lastest and test then it could have already been resolved...
On 8/12/13 2:35 PM, "Nuno Reis" <nreis@wavecom.pt> wrote:
Another eventually important tip:
This is happening at least with freeswitch versions 1.2.3 and 1.2.8.
Thanks. --
*Nuno Miguel Reis* | *Unified Communication Systems * M. +351 913907481 | nreis@wavecom.pt <mailto:nreis@wavecom.pt<nreis@wavecom.pt>>
WAVECOM-Soluções Rádio, S.A. Cacia Park | Rua do Progresso, Lote 15 3800-639 AVEIRO | Portugal T. +351 309 700 225 | F. +351 234 919 191 *GPS** < http://maps.google.com/maps/ms?msa=0&msid=202333747613191340808.0004b4b227a6144f0df88> | www.wavecom.pt <http://www.wavecom.pt/> * * <http://www.wavecom.pt/> *
<http://www.wavecom.pt/pt/wavecom/premios.php>
<http://www.wavecom.pt/pt/mail_eventos.php>
On Mon, Aug 12, 2013 at 8:29 PM, Nuno Reis <nreis@wavecom.pt> wrote:
Hello all.
I've been experiencing a problem with freeswitch where it stops responding (doesn't answer any SIP REQUESTS) and produces the following output in the log file every time the symptom happens:
2013-08-12 15:18:31.488906 [WARNING] switch_event.c:552 Create event dispatch thread 1 2013-08-12 15:18:31.578906 [WARNING] switch_event.c:554 Create additional event dispatch thread 2 2013-08-12 15:18:31.598907 [WARNING] switch_event.c:554 Create additional event dispatch thread 3 2013-08-12 15:18:31.648904 [WARNING] switch_event.c:554 Create additional event dispatch thread 4 2013-08-12 15:18:31.728905 [WARNING] switch_event.c:554 Create additional event dispatch thread 5 2013-08-12 15:19:04.348914 [CONSOLE] sofia.c:1602 MSG Thread 4 Started
After this, i can see that the process exists and i can get to the fs_cli prompt, although none of the normal command like: show calls work. The command simply hangs without returning back a response. I think this could be memory related, although can't find a reasonable explanation to that. I had 4GB of physical RAM available + 512MB for SWAP on a CentOS x86_64 machine and the OS was using like about 800MB but if i considered the cached amount of memory, it was like a total of 3.8GB which from what i understand is normal in linux, since the kernel tries to cache all the available memory over time. Nevertheless i've decided to give the machine more 4GB RAM and SWAP is now 16GB, so now i've 8GB total for physical RAM and 16GB SWAP which solved the problem at least temporarily and made freeswitch stop getting back into the symptom described above. I can see that the cached memory is again growing over time. I still don't know if I'll end up with the same symptom again when the total used memory (used+ cached) gets near the 8GB.
Anyone has had this symptom before too? Some feedback is appreciated.
Thanks. --
*Nuno Miguel Reis* | *Unified Communication Systems * M. +351 913907481 | nreis@wavecom.pt <mailto:nreis@wavecom.pt<nreis@wavecom.pt>>
WAVECOM-Soluções Rádio, S.A. Cacia Park | Rua do Progresso, Lote 15 3800-639 AVEIRO | Portugal T. +351 309 700 225 | F. +351 234 919 191 *GPS** < http://maps.google.com/maps/ms?msa=0&msid=202333747613191340808.0004b4b227a6144f0df88> | www.wavecom.pt <http://www.wavecom.pt/> * * <http://www.wavecom.pt/> *
<http://www.wavecom.pt/pt/wavecom/premios.php>
<http://www.wavecom.pt/pt/mail_eventos.php>
------------------------------ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken *http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org *irc.freenode.net #freeswitch
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
If this happens again please check jira to see if there is another instance of this happening. If there is not open a jira, and use gcore to get a core file, and then follow the standard info on the wiki for obtaining a stack trace and attach the resultant output to the ticket. Sounds like you might be getting hammered by something like sipvicious On 8/12/13 2:29 PM, "Nuno Reis" <nreis@wavecom.pt> wrote:
Hello all.
I've been experiencing a problem with freeswitch where it stops responding (doesn't answer any SIP REQUESTS) and produces the following output in the log file every time the symptom happens:
2013-08-12 15:18:31.488906 [WARNING] switch_event.c:552 Create event dispatch thread 1 2013-08-12 15:18:31.578906 [WARNING] switch_event.c:554 Create additional event dispatch thread 2 2013-08-12 15:18:31.598907 [WARNING] switch_event.c:554 Create additional event dispatch thread 3 2013-08-12 15:18:31.648904 [WARNING] switch_event.c:554 Create additional event dispatch thread 4 2013-08-12 15:18:31.728905 [WARNING] switch_event.c:554 Create additional event dispatch thread 5 2013-08-12 15:19:04.348914 [CONSOLE] sofia.c:1602 MSG Thread 4 Started
After this, i can see that the process exists and i can get to the fs_cli prompt, although none of the normal command like: show calls work. The command simply hangs without returning back a response. I think this could be memory related, although can't find a reasonable explanation to that. I had 4GB of physical RAM available + 512MB for SWAP on a CentOS x86_64 machine and the OS was using like about 800MB but if i considered the cached amount of memory, it was like a total of 3.8GB which from what i understand is normal in linux, since the kernel tries to cache all the available memory over time. Nevertheless i've decided to give the machine more 4GB RAM and SWAP is now 16GB, so now i've 8GB total for physical RAM and 16GB SWAP which solved the problem at least temporarily and made freeswitch stop getting back into the symptom described above. I can see that the cached memory is again growing over time. I still don't know if I'll end up with the same symptom again when the total used memory (used+ cached) gets near the 8GB.
Anyone has had this symptom before too? Some feedback is appreciated.
Thanks. --
Nuno Miguel Reis | Unified Communication Systems M. +351 913907481 | nreis@wavecom.pt <mailto:nreis@wavecom.pt> WAVECOM-Soluções Rádio, S.A. Cacia Park | Rua do Progresso, Lote 15 3800-639 AVEIRO | Portugal T. +351 309 700 225 | F. +351 234 919 191 GPS <http://maps.google.com/maps/ms?msa=0&msid=202333747613191340808.0004b4b227a6... 44f0df88> | www.wavecom.pt <http://www.wavecom.pt/> <http://www.wavecom.pt/>
<http://www.wavecom.pt/pt/wavecom/premios.php>
<http://www.wavecom.pt/pt/mail_eventos.php>
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org irc.freenode.net #freeswitch
participants (3)
-
Anton VG
-
Ken Rice
-
Nuno Reis