Hi, I have Freeswitch running as a SBC in front of several Asterisk machines and without warning one of the Freeswitch processes will use 100% of one of the CPU cores and stop responding to SIP requests. I'm using the latest svn on Centos 5.4 in a xen vm. Basically everything will be fine for about 3-4 hours and then this happens. [root@sip ~]# cat /proc/cpuinfo processor : 0 vendor_id : AuthenticAMD cpu family : 15 model : 33 model name : AMD Opteron(tm) Processor 275 stepping : 2 cpu MHz : 2204.998 cache size : 1024 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu de tsc msr pae cx8 apic cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni cmp_legacy bogomips : 5514.25 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: ts fid vid ttp processor : 1 vendor_id : AuthenticAMD cpu family : 15 model : 33 model name : AMD Opteron(tm) Processor 275 stepping : 2 cpu MHz : 2204.998 cache size : 1024 KB physical id : 1 siblings : 1 core id : 0 cpu cores : 1 fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu de tsc msr pae cx8 apic cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni cmp_legacy bogomips : 5514.25 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: ts fid vid ttp processor : 2 vendor_id : AuthenticAMD cpu family : 15 model : 33 model name : AMD Opteron(tm) Processor 275 stepping : 2 cpu MHz : 2204.998 cache size : 1024 KB physical id : 2 siblings : 1 core id : 0 cpu cores : 1 fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu de tsc msr pae cx8 apic cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni cmp_legacy bogomips : 5514.25 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: ts fid vid ttp processor : 3 vendor_id : AuthenticAMD cpu family : 15 model : 33 model name : AMD Opteron(tm) Processor 275 stepping : 2 cpu MHz : 2204.998 cache size : 1024 KB physical id : 3 siblings : 1 core id : 0 cpu cores : 1 fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu de tsc msr pae cx8 apic cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow pni cmp_legacy bogomips : 5514.25 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: ts fid vid ttp [root@sip ~]# uname -r 2.6.18-164.11.1.el5xen -- Spencer Thomason 5Nine Solutions LLC e. sales@5ninesolutions.com p. +1.888.271.7959 f. +1.310.510.6980
Are you on the latest SVN? If the problem persists after you update to SVN trunk please read http://wiki.freeswitch.org/wiki/Reporting_Bugs and report the issue to jira. /b On Mar 14, 2010, at 12:49 PM, Spencer Thomason wrote:
I have Freeswitch running as a SBC in front of several Asterisk machines and without warning one of the Freeswitch processes will use 100% of one of the CPU cores and stop responding to SIP requests. I'm using the latest svn on Centos 5.4 in a xen vm. Basically everything will be fine for about 3-4 hours and then this happens.
Yes, I'm using the latest SVN. Spencer On Mar 14, 2010, at 11:00 AM, Brian West wrote:
Are you on the latest SVN?
If the problem persists after you update to SVN trunk please read http://wiki.freeswitch.org/wiki/Reporting_Bugs and report the issue to jira.
/b
On Mar 14, 2010, at 12:49 PM, Spencer Thomason wrote:
I have Freeswitch running as a SBC in front of several Asterisk machines and without warning one of the Freeswitch processes will use 100% of one of the CPU cores and stop responding to SIP requests. I'm using the latest svn on Centos 5.4 in a xen vm. Basically everything will be fine for about 3-4 hours and then this happens.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Do you think its due to resource limitations in your vm? Make sure you check all the config options to give fs full access to all the resouces. If it persists, do top -H and identify the pid with the 100% usage. Then get a gcore and gdb it to capture the output of "thread apply all bt" Report the findings to jira. On Mar 14, 2010 1:23 PM, "Spencer Thomason" <spencer@5ninesolutions.com> wrote: Yes, I'm using the latest SVN. Spencer On Mar 14, 2010, at 11:00 AM, Brian West wrote:
Are you on the latest SVN?
If the problem pe...
No. Its on our hardware, we just use xen to virtualize the machines.. this is the only vm on the machine. Gcore complains about no debugging symbols found, do I need to rebuild? It appears to be a problem when registering to one of my trunks. The very strange thing is the everything will be working perfectly and then all of a sudden this happens, i.e. trunks regged, calls completing, etc. This is a very lightly loaded server. This is the output from the console: freeswitch@internal> 2010-03-14 23:12:15.199983 [NOTICE] sofia_reg.c: 336 Registering bandtel 2010-03-14 23:12:15.223977 [DEBUG] sofia_reg.c:1478 Changing expire time to 475 by request of proxy sip:66.237.65.67 2010-03-14 23:20:05.429613 [NOTICE] sofia_reg.c:336 Registering bandtel 2010-03-14 23:20:08.285756 [ERR] sofia_reg.c:1490 bandtel Registration Failed with status Operation has no matching challenge [904]. failure #1 2010-03-14 23:20:10.043933 [WARNING] sofia_reg.c:381 bandtel Failed Registration, setting retry to 60 seconds. 2010-03-14 23:21:16.332797 [NOTICE] sofia_reg.c:336 Registering bandtel 2010-03-14 23:21:52.528409 [WARNING] sofia_reg.c:381 bandtel Failed Registration, setting retry to 60 seconds. 2010-03-14 23:22:56.945511 [NOTICE] sofia_reg.c:336 Registering bandtel 2010-03-14 23:23:33.083177 [WARNING] sofia_reg.c:381 bandtel Failed Registration, setting retry to 60 seconds. 2010-03-14 23:24:37.398155 [NOTICE] sofia_reg.c:336 Registering bandtel 2010-03-14 23:25:13.631825 [WARNING] sofia_reg.c:381 bandtel Failed Registration, setting retry to 60 seconds. 2010-03-14 23:26:17.978767 [NOTICE] sofia_reg.c:336 Registering bandtel 2010-03-14 23:26:54.172425 [WARNING] sofia_reg.c:381 bandtel Failed Registration, setting retry to 60 seconds. 2010-03-14 23:27:58.579354 [NOTICE] sofia_reg.c:336 Registering bandtel 2010-03-14 23:28:34.605017 [WARNING] sofia_reg.c:381 bandtel Failed Registration, setting retry to 60 seconds. On Mar 14, 2010, at 4:08 PM, Anthony Minessale wrote:
Do you think its due to resource limitations in your vm? Make sure you check all the config options to give fs full access to all the resouces.
If it persists, do top -H and identify the pid with the 100% usage. Then get a gcore and gdb it to capture the output of "thread apply all bt"
Report the findings to jira.
On Mar 14, 2010 1:23 PM, "Spencer Thomason" <spencer@5ninesolutions.com
wrote:
Yes, I'm using the latest SVN.
Spencer
On Mar 14, 2010, at 11:00 AM, Brian West wrote:
Are you on the latest SVN?
If the problem pe...
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Are you sure you're on SVN trunk? /b On Mar 14, 2010, at 7:14 PM, Spencer Thomason wrote:
No. Its on our hardware, we just use xen to virtualize the machines.. this is the only vm on the machine. Gcore complains about no debugging symbols found, do I need to rebuild?
It appears to be a problem when registering to one of my trunks. The very strange thing is the everything will be working perfectly and then all of a sudden this happens, i.e. trunks regged, calls completing, etc. This is a very lightly loaded server.
This is the output from the console:
I rebuilt today. On Mar 14, 2010, at 5:23 PM, Brian West wrote:
Are you sure you're on SVN trunk?
/b
On Mar 14, 2010, at 7:14 PM, Spencer Thomason wrote:
No. Its on our hardware, we just use xen to virtualize the machines.. this is the only vm on the machine. Gcore complains about no debugging symbols found, do I need to rebuild?
It appears to be a problem when registering to one of my trunks. The very strange thing is the everything will be working perfectly and then all of a sudden this happens, i.e. trunks regged, calls completing, etc. This is a very lightly loaded server.
This is the output from the console:
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
That seems suspiciously similar to an old issue we have already fixed, how exactly did you rebuild? On Sun, Mar 14, 2010 at 6:38 PM, Spencer Thomason < spencer@5ninesolutions.com> wrote:
I rebuilt today.
On Mar 14, 2010, at 5:23 PM, Brian West wrote:
Are you sure you're on SVN trunk?
/b
On Mar 14, 2010, at 7:14 PM, Spencer Thomason wrote:
No. Its on our hardware, we just use xen to virtualize the machines.. this is the only vm on the machine. Gcore complains about no debugging symbols found, do I need to rebuild?
It appears to be a problem when registering to one of my trunks. The very strange thing is the everything will be working perfectly and then all of a sudden this happens, i.e. trunks regged, calls completing, etc. This is a very lightly loaded server.
This is the output from the console:
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900
I'm using Centos 5.4, I have a build machine and I got the latest source from SVN, and rebuilt the RPMS from the updates source and then updated the FS machine. We have our own repo setup to update the machines. I disabled most of the modules except mod_console, mod_logfile, mod_cdr_csv, mod_event_scoket, mod_sofia, mod_loopback, mod_commands, mod_dptools, mod_expr, mod_limit, mod_dialplan_xml, and mod_voipcodecs and everything is working great. As soon as I enable the default set of modules, FS will work for about 3-4 hours and then one of the threads goes to 100% CPU. I haven't had the time to load the modules one by one to try to isolate the problem. The build completes fine, and I'm using the stock centos libraries to build against. The centos xen kernel on x86_64 is 100hz, could that be an issue? On Mar 15, 2010, at 8:45 AM, Anthony Minessale wrote:
That seems suspiciously similar to an old issue we have already fixed, how exactly did you rebuild?
On Sun, Mar 14, 2010 at 6:38 PM, Spencer Thomason <spencer@5ninesolutions.com
wrote: I rebuilt today.
On Mar 14, 2010, at 5:23 PM, Brian West wrote:
Are you sure you're on SVN trunk?
/b
On Mar 14, 2010, at 7:14 PM, Spencer Thomason wrote:
No. Its on our hardware, we just use xen to virtualize the machines.. this is the only vm on the machine. Gcore complains about no debugging symbols found, do I need to rebuild?
It appears to be a problem when registering to one of my trunks. The very strange thing is the everything will be working perfectly and then all of a sudden this happens, i.e. trunks regged, calls completing, etc. This is a very lightly loaded server.
This is the output from the console:
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org pstn:+19193869900 _______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
if you can email me privately at consulting@freeswitch.org with the details to log into the box. I can have a look for you. On Mon, Mar 15, 2010 at 1:07 PM, Spencer Thomason < spencer@5ninesolutions.com> wrote:
I'm using Centos 5.4, I have a build machine and I got the latest source from SVN, and rebuilt the RPMS from the updates source and then updated the FS machine. We have our own repo setup to update the machines. I disabled most of the modules except mod_console, mod_logfile, mod_cdr_csv, mod_event_scoket, mod_sofia, mod_loopback, mod_commands, mod_dptools, mod_expr, mod_limit, mod_dialplan_xml, and mod_voipcodecs and everything is working great. As soon as I enable the default set of modules, FS will work for about 3-4 hours and then one of the threads goes to 100% CPU. I haven't had the time to load the modules one by one to try to isolate the problem. The build completes fine, and I'm using the stock centos libraries to build against. The centos xen kernel on x86_64 is 100hz, could that be an issue?
On Mar 15, 2010, at 8:45 AM, Anthony Minessale wrote:
That seems suspiciously similar to an old issue we have already fixed, how exactly did you rebuild?
On Sun, Mar 14, 2010 at 6:38 PM, Spencer Thomason < spencer@5ninesolutions.com> wrote:
I rebuilt today.
On Mar 14, 2010, at 5:23 PM, Brian West wrote:
Are you sure you're on SVN trunk?
/b
On Mar 14, 2010, at 7:14 PM, Spencer Thomason wrote:
No. Its on our hardware, we just use xen to virtualize the machines.. this is the only vm on the machine. Gcore complains about no debugging symbols found, do I need to rebuild?
It appears to be a problem when registering to one of my trunks. The very strange thing is the everything will be working perfectly and then all of a sudden this happens, i.e. trunks regged, calls completing, etc. This is a very lightly loaded server.
This is the output from the console:
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900 _______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> iax:guest@conference.freeswitch.org/888 googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900
participants (3)
-
Anthony Minessale
-
Brian West
-
Spencer Thomason