freeswitch HANGING (process exists but stops responding)
Hello all. I've been experiencing a problem with freeswitch where it stops responding (doesn't answer any SIP REQUESTS) and produces the following output in the log file every time the symptom happens: 2013-08-12 15:18:31.488906 [WARNING] switch_event.c:552 Create event dispatch thread 1 2013-08-12 15:18:31.578906 [WARNING] switch_event.c:554 Create additional event dispatch thread 2 2013-08-12 15:18:31.598907 [WARNING] switch_event.c:554 Create additional event dispatch thread 3 2013-08-12 15:18:31.648904 [WARNING] switch_event.c:554 Create additional event dispatch thread 4 2013-08-12 15:18:31.728905 [WARNING] switch_event.c:554 Create additional event dispatch thread 5 2013-08-12 15:19:04.348914 [CONSOLE] sofia.c:1602 MSG Thread 4 Started After this, i can see that the process exists and i can get to the fs_cli prompt, although none of the normal command like: show calls work. The command simply hangs without returning back a response. I think this could be memory related, although can't find a reasonable explanation to that. I had 4GB of physical RAM available + 512MB for SWAP on a CentOS x86_64 machine and the OS was using like about 800MB but if i considered the cached amount of memory, it was like a total of 3.8GB which from what i understand is normal in linux, since the kernel tries to cache all the available memory over time. Nevertheless i've decided to give the machine more 4GB RAM and SWAP is now 16GB, so now i've 8GB total for physical RAM and 16GB SWAP which solved the problem at least temporarily and made freeswitch stop getting back into the symptom described above. I can see that the cached memory is again growing over time. I still don't know if I'll end up with the same symptom again when the total used memory (used+ cached) gets near the 8GB. Anyone has had this symptom before too? Some feedback is appreciated. Thanks. -- *Nuno Miguel Reis* | *Unified Communication** Systems* M. +351 913907481 | nreis@wavecom.pt WAVECOM-Soluções Rádio, S.A. Cacia Park | Rua do Progresso, Lote 15 3800-639 AVEIRO | Portugal T. +351 309 700 225 | F. +351 234 919 191 *GPS<http://maps.google.com/maps/ms?msa=0&msid=202333747613191340808.0004b4b227a6144f0df88>| www.wavecom.pt** <http://www.wavecom.pt/>* [image: Description: Description: WavecomSignature]<http://www.wavecom.pt/pt/wavecom/premios.php> [image: Publicity] <http://www.wavecom.pt/pt/mail_eventos.php>
Another eventually important tip: This is happening at least with freeswitch versions 1.2.3 and 1.2.8. Thanks. -- *Nuno Miguel Reis* | *Unified Communication** Systems* M. +351 913907481 | nreis@wavecom.pt WAVECOM-Soluções Rádio, S.A. Cacia Park | Rua do Progresso, Lote 15 3800-639 AVEIRO | Portugal T. +351 309 700 225 | F. +351 234 919 191 *GPS<http://maps.google.com/maps/ms?msa=0&msid=202333747613191340808.0004b4b227a6144f0df88>| www.wavecom.pt** <http://www.wavecom.pt/>* [image: Description: Description: WavecomSignature]<http://www.wavecom.pt/pt/wavecom/premios.php> [image: Publicity] <http://www.wavecom.pt/pt/mail_eventos.php> On Mon, Aug 12, 2013 at 8:29 PM, Nuno Reis <nreis@wavecom.pt> wrote:
Hello all.
I've been experiencing a problem with freeswitch where it stops responding (doesn't answer any SIP REQUESTS) and produces the following output in the log file every time the symptom happens:
2013-08-12 15:18:31.488906 [WARNING] switch_event.c:552 Create event dispatch thread 1 2013-08-12 15:18:31.578906 [WARNING] switch_event.c:554 Create additional event dispatch thread 2 2013-08-12 15:18:31.598907 [WARNING] switch_event.c:554 Create additional event dispatch thread 3 2013-08-12 15:18:31.648904 [WARNING] switch_event.c:554 Create additional event dispatch thread 4 2013-08-12 15:18:31.728905 [WARNING] switch_event.c:554 Create additional event dispatch thread 5 2013-08-12 15:19:04.348914 [CONSOLE] sofia.c:1602 MSG Thread 4 Started
After this, i can see that the process exists and i can get to the fs_cli prompt, although none of the normal command like: show calls work. The command simply hangs without returning back a response. I think this could be memory related, although can't find a reasonable explanation to that. I had 4GB of physical RAM available + 512MB for SWAP on a CentOS x86_64 machine and the OS was using like about 800MB but if i considered the cached amount of memory, it was like a total of 3.8GB which from what i understand is normal in linux, since the kernel tries to cache all the available memory over time. Nevertheless i've decided to give the machine more 4GB RAM and SWAP is now 16GB, so now i've 8GB total for physical RAM and 16GB SWAP which solved the problem at least temporarily and made freeswitch stop getting back into the symptom described above. I can see that the cached memory is again growing over time. I still don't know if I'll end up with the same symptom again when the total used memory (used+ cached) gets near the 8GB.
Anyone has had this symptom before too? Some feedback is appreciated.
Thanks.
--
*Nuno Miguel Reis* | *Unified Communication** Systems* M. +351 913907481 | nreis@wavecom.pt WAVECOM-Soluções Rádio, S.A. Cacia Park | Rua do Progresso, Lote 15 3800-639 AVEIRO | Portugal T. +351 309 700 225 | F. +351 234 919 191 *GPS<http://maps.google.com/maps/ms?msa=0&msid=202333747613191340808.0004b4b227a6144f0df88>| www.wavecom.pt** <http://www.wavecom.pt/>*
[image: Description: Description: WavecomSignature]<http://www.wavecom.pt/pt/wavecom/premios.php>
[image: Publicity] <http://www.wavecom.pt/pt/mail_eventos.php>
If this happens again please check jira to see if there is another instance of this happening. If there is not open a jira, and use gcore to get a core file, and then follow the standard info on the wiki for obtaining a stack trace and attach the resultant output to the ticket. Sounds like you might be getting hammered by something like sipvicious On 8/12/13 2:29 PM, "Nuno Reis" <nreis@wavecom.pt> wrote:
Hello all.
I've been experiencing a problem with freeswitch where it stops responding (doesn't answer any SIP REQUESTS) and produces the following output in the log file every time the symptom happens:
2013-08-12 15:18:31.488906 [WARNING] switch_event.c:552 Create event dispatch thread 1 2013-08-12 15:18:31.578906 [WARNING] switch_event.c:554 Create additional event dispatch thread 2 2013-08-12 15:18:31.598907 [WARNING] switch_event.c:554 Create additional event dispatch thread 3 2013-08-12 15:18:31.648904 [WARNING] switch_event.c:554 Create additional event dispatch thread 4 2013-08-12 15:18:31.728905 [WARNING] switch_event.c:554 Create additional event dispatch thread 5 2013-08-12 15:19:04.348914 [CONSOLE] sofia.c:1602 MSG Thread 4 Started
After this, i can see that the process exists and i can get to the fs_cli prompt, although none of the normal command like: show calls work. The command simply hangs without returning back a response. I think this could be memory related, although can't find a reasonable explanation to that. I had 4GB of physical RAM available + 512MB for SWAP on a CentOS x86_64 machine and the OS was using like about 800MB but if i considered the cached amount of memory, it was like a total of 3.8GB which from what i understand is normal in linux, since the kernel tries to cache all the available memory over time. Nevertheless i've decided to give the machine more 4GB RAM and SWAP is now 16GB, so now i've 8GB total for physical RAM and 16GB SWAP which solved the problem at least temporarily and made freeswitch stop getting back into the symptom described above. I can see that the cached memory is again growing over time. I still don't know if I'll end up with the same symptom again when the total used memory (used+ cached) gets near the 8GB.
Anyone has had this symptom before too? Some feedback is appreciated.
Thanks. --
Nuno Miguel Reis | Unified Communication Systems M. +351 913907481 | nreis@wavecom.pt <mailto:nreis@wavecom.pt> WAVECOM-Soluções Rádio, S.A. Cacia Park | Rua do Progresso, Lote 15 3800-639 AVEIRO | Portugal T. +351 309 700 225 | F. +351 234 919 191 GPS <http://maps.google.com/maps/ms?msa=0&msid=202333747613191340808.0004b4b227a6... 44f0df88> | www.wavecom.pt <http://www.wavecom.pt/> <http://www.wavecom.pt/>
<http://www.wavecom.pt/pt/wavecom/premios.php>
<http://www.wavecom.pt/pt/mail_eventos.php>
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org irc.freenode.net #freeswitch
participants (2)
-
Ken Rice
-
Nuno Reis