Sofia crashing when registering
Hello all, I use Bandtel as one of my trunking providers and Freeswitch will work flawlessly for 24-36 hours and then stop responding to SIP requests. I have monit setup to send an options ping once a minute and restart the process if it doesn't get a reply. This has been working ok but the Freeswitch process has to get restarted about once a day on average. Has anybody else had any issues like this? They are using fairly standard issue stuff, Freeswitch registers to SER 0.9.6 and they are using Sippy as a B2BUA on their end. The log shows a call ending, a period of no activity, and Freeswitch attempting to register repeatedly, the process then stops responding to requests and monit restarts Freeswitch. 2010-03-24 17:31:03.897756 [DEBUG] switch_core_state_machine.c:439 (sofia/default/13105106983@208.70.78.37 ) State DESTROY going to sleep 2010-03-24 17:33:16.353544 [NOTICE] sofia_reg.c:336 Registering bandtel 2010-03-24 17:37:58.452249 [NOTICE] sofia_reg.c:336 Registering bandtel 2010-03-24 17:42:40.588167 [NOTICE] sofia_reg.c:336 Registering bandtel 2010-03-24 17:47:22.529220 [NOTICE] sofia_reg.c:336 Registering bandtel 2010-03-24 17:53:15.899570 [INFO] switch_core.c:1734 Shutting down Here is a typical register request. ------------------------------------------------------------------------ 2010-03-24 21:12:59.955693 [NOTICE] sofia_reg.c:336 Registering bandtel send 772 bytes to udp/[66.237.65.67]:5060 at 04:12:59.956245: ------------------------------------------------------------------------ REGISTER sip:66.237.65.67 SIP/2.0 Via: SIP/2.0/UDP 208.70.78.36;rport;branch=z9hG4bKZcD7H9aN1D50j Max-Forwards: 70 From: <sip: 201009000001@66.237.65.67;transport=udp>;tag=3yZ6ZSZUFj3gQ To: <sip:201009000001@66.237.65.67;transport=udp> Call-ID: b1eae39a-37ae-11df-9bcf-f1944b4d3175 CSeq: 128622152 REGISTER Contact: <sip:gw+bandtel@208.70.78.36:5060;transport=udp;gw=bandtel> Expires: 390 User-Agent: FreeSWITCH Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, MESSAGE, UPDATE, INFO, REGISTER, NOTIFY Supported: timer, precondition, path, replaces Authorization: Digest username="201009000001", realm="10.1.2.1", nonce="4baabf0e1e68a1fdfbde46913ca850da0cbecc8a", algorithm=MD5, uri="sip:66.237.65.67 ", response="3974689c5cb3eeb4f09e634716d2239e" Content-Length: 0 ------------------------------------------------------------------------ recv 505 bytes from udp/[66.237.65.67]:5060 at 04:12:59.980247: ------------------------------------------------------------------------ SIP/2.0 200 OK From: <sip: 201009000001@66.237.65.67;transport=UDP>;tag=3yZ6ZSZUFj3gQ To: <sip: 201009000001 @66.237.65.67;transport=UDP>;tag=2e41cc73d384c8f458f34dd410d7744b-b5e6 Call-ID: b1eae39a-37ae-11df-9bcf-f1944b4d3175 CSeq: 128622152 REGISTER Via: SIP/2.0/UDP 208.70.78.36;rport=5060;branch=z9hG4bKZcD7H9aN1D50j PortaBilling: currency:USD Server: Sip EXpress router (0.9.6 (i386/freebsd)) Contact: <sip:gw+bandtel@66.237.65.67:5060;gw=bandtel>;expires=385 Expires: 385 Content-Length: 0 ------------------------------------------------------------------------ 2010-03-24 21:12:59.979821 [DEBUG] sofia_reg.c:1478 Changing expire time to 385 by request of proxy sip:66.237.65.67 What is the best way to debug this? It seems strange that everything works for a while and then stops. Thanks, Spencer -- Spencer Thomason 5Nine Solutions LLC e. sales@5ninesolutions.com p. +1.888.271.7959 f. +1.310.510.6980
are you on SVN trunk? /b On Mar 24, 2010, at 11:28 PM, Spencer Thomason wrote:
Hello all, I use Bandtel as one of my trunking providers and Freeswitch will work flawlessly for 24-36 hours and then stop responding to SIP requests. I have monit setup to send an options ping once a minute and restart the process if it doesn't get a reply. This has been working ok but the Freeswitch process has to get restarted about once a day on average. Has anybody else had any issues like this? They are using fairly standard issue stuff, Freeswitch registers to SER 0.9.6 and they are using Sippy as a B2BUA on their end. The log shows a call ending, a period of no activity, and Freeswitch attempting to register repeatedly, the process then stops responding to requests and monit restarts Freeswitch.
2010-03-24 17:31:03.897756 [DEBUG] switch_core_state_machine.c:439 (sofia/default/13105106983@208.70.78.37) State DESTROY going to sleep 2010-03-24 17:33:16.353544 [NOTICE] sofia_reg.c:336 Registering bandtel 2010-03-24 17:37:58.452249 [NOTICE] sofia_reg.c:336 Registering bandtel 2010-03-24 17:42:40.588167 [NOTICE] sofia_reg.c:336 Registering bandtel 2010-03-24 17:47:22.529220 [NOTICE] sofia_reg.c:336 Registering bandtel 2010-03-24 17:53:15.899570 [INFO] switch_core.c:1734 Shutting down
Here is a typical register request. ------------------------------------------------------------------------ 2010-03-24 21:12:59.955693 [NOTICE] sofia_reg.c:336 Registering bandtel send 772 bytes to udp/[66.237.65.67]:5060 at 04:12:59.956245: ------------------------------------------------------------------------ REGISTER sip:66.237.65.67 SIP/2.0 Via: SIP/2.0/UDP 208.70.78.36;rport;branch=z9hG4bKZcD7H9aN1D50j Max-Forwards: 70 From: <sip:201009000001@66.237.65.67;transport=udp>;tag=3yZ6ZSZUFj3gQ To: <sip:201009000001@66.237.65.67;transport=udp> Call-ID: b1eae39a-37ae-11df-9bcf-f1944b4d3175 CSeq: 128622152 REGISTER Contact: <sip:gw+bandtel@208.70.78.36:5060;transport=udp;gw=bandtel> Expires: 390 User-Agent: FreeSWITCH Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, MESSAGE, UPDATE, INFO, REGISTER, NOTIFY Supported: timer, precondition, path, replaces Authorization: Digest username="201009000001", realm="10.1.2.1", nonce="4baabf0e1e68a1fdfbde46913ca850da0cbecc8a", algorithm=MD5, uri="sip:66.237.65.67", response="3974689c5cb3eeb4f09e634716d2239e" Content-Length: 0
------------------------------------------------------------------------ recv 505 bytes from udp/[66.237.65.67]:5060 at 04:12:59.980247: ------------------------------------------------------------------------ SIP/2.0 200 OK From: <sip:201009000001@66.237.65.67;transport=UDP>;tag=3yZ6ZSZUFj3gQ To: <sip:201009000001@66.237.65.67;transport=UDP>;tag=2e41cc73d384c8f458f34dd410d7744b-b5e6 Call-ID: b1eae39a-37ae-11df-9bcf-f1944b4d3175 CSeq: 128622152 REGISTER Via: SIP/2.0/UDP 208.70.78.36;rport=5060;branch=z9hG4bKZcD7H9aN1D50j PortaBilling: currency:USD Server: Sip EXpress router (0.9.6 (i386/freebsd)) Contact: <sip:gw+bandtel@66.237.65.67:5060;gw=bandtel>;expires=385 Expires: 385 Content-Length: 0
------------------------------------------------------------------------ 2010-03-24 21:12:59.979821 [DEBUG] sofia_reg.c:1478 Changing expire time to 385 by request of proxy sip:66.237.65.67
What is the best way to debug this? It seems strange that everything works for a while and then stops.
Thanks, Spencer
-- Spencer Thomason 5Nine Solutions LLC e. sales@5ninesolutions.com p. +1.888.271.7959 f. +1.310.510.6980
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Yes. freeswitch@internal> version FreeSWITCH Version 1.0.trunk (17097M) Spencer On Mar 24, 2010, at 9:34 PM, Brian West wrote:
are you on SVN trunk?
/b
On Mar 24, 2010, at 11:28 PM, Spencer Thomason wrote:
Hello all, I use Bandtel as one of my trunking providers and Freeswitch will work flawlessly for 24-36 hours and then stop responding to SIP requests. I have monit setup to send an options ping once a minute and restart the process if it doesn't get a reply. This has been working ok but the Freeswitch process has to get restarted about once a day on average. Has anybody else had any issues like this? They are using fairly standard issue stuff, Freeswitch registers to SER 0.9.6 and they are using Sippy as a B2BUA on their end. The log shows a call ending, a period of no activity, and Freeswitch attempting to register repeatedly, the process then stops responding to requests and monit restarts Freeswitch.
2010-03-24 17:31:03.897756 [DEBUG] switch_core_state_machine.c:439 (sofia/default/13105106983@208.70.78.37 ) State DESTROY going to sleep 2010-03-24 17:33:16.353544 [NOTICE] sofia_reg.c:336 Registering bandtel 2010-03-24 17:37:58.452249 [NOTICE] sofia_reg.c:336 Registering bandtel 2010-03-24 17:42:40.588167 [NOTICE] sofia_reg.c:336 Registering bandtel 2010-03-24 17:47:22.529220 [NOTICE] sofia_reg.c:336 Registering bandtel 2010-03-24 17:53:15.899570 [INFO] switch_core.c:1734 Shutting down
Here is a typical register request.
------------------------------------------------------------------------ 2010-03-24 21:12:59.955693 [NOTICE] sofia_reg.c:336 Registering bandtel send 772 bytes to udp/[66.237.65.67]:5060 at 04:12:59.956245:
------------------------------------------------------------------------ REGISTER sip:66.237.65.67 SIP/2.0 Via: SIP/2.0/UDP 208.70.78.36;rport;branch=z9hG4bKZcD7H9aN1D50j Max-Forwards: 70 From: <sip: 201009000001@66.237.65.67;transport=udp>;tag=3yZ6ZSZUFj3gQ To: <sip:201009000001@66.237.65.67;transport=udp> Call-ID: b1eae39a-37ae-11df-9bcf-f1944b4d3175 CSeq: 128622152 REGISTER Contact: <sip:gw+bandtel@208.70.78.36:5060;transport=udp;gw=bandtel
Expires: 390 User-Agent: FreeSWITCH Allow: INVITE, ACK, BYE, CANCEL, OPTIONS, MESSAGE, UPDATE, INFO, REGISTER, NOTIFY Supported: timer, precondition, path, replaces Authorization: Digest username="201009000001", realm="10.1.2.1", nonce="4baabf0e1e68a1fdfbde46913ca850da0cbecc8a", algorithm=MD5, uri="sip:66.237.65.67", response="3974689c5cb3eeb4f09e634716d2239e" Content-Length: 0
------------------------------------------------------------------------ recv 505 bytes from udp/[66.237.65.67]:5060 at 04:12:59.980247:
------------------------------------------------------------------------ SIP/2.0 200 OK From: <sip: 201009000001@66.237.65.67;transport=UDP>;tag=3yZ6ZSZUFj3gQ To: <sip: 201009000001 @66.237.65.67;transport=UDP>;tag=2e41cc73d384c8f458f34dd410d7744b- b5e6 Call-ID: b1eae39a-37ae-11df-9bcf-f1944b4d3175 CSeq: 128622152 REGISTER Via: SIP/2.0/UDP 208.70.78.36;rport=5060;branch=z9hG4bKZcD7H9aN1D50j PortaBilling: currency:USD Server: Sip EXpress router (0.9.6 (i386/freebsd)) Contact: <sip:gw+bandtel@66.237.65.67:5060;gw=bandtel>;expires=385 Expires: 385 Content-Length: 0
------------------------------------------------------------------------ 2010-03-24 21:12:59.979821 [DEBUG] sofia_reg.c:1478 Changing expire time to 385 by request of proxy sip:66.237.65.67
What is the best way to debug this? It seems strange that everything works for a while and then stops.
Thanks, Spencer
-- Spencer Thomason 5Nine Solutions LLC e. sales@5ninesolutions.com p. +1.888.271.7959 f. +1.310.510.6980
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Spencer Thomason <spencer@5ninesolutions.com> wrote:
What is the best way to debug this?
1. Upgrade to the latest version from the svn repository if you haven't done so already. You didn't specify which version of FreeSWITCH you were using, but if it's old, upgrade it and try to reproduce the problem with the latest version. 2. Assuming you can reproduce the problem with the latest code from svn, I suppose it's time for gdb - instructions are on the wiki. Can you run fs_cli and connect to it in this state or does it hang completely?
fs_cli will still connect but you cannot issue any commands to sofia, i.e. restart, etc I am on svn trunk. The problem is I haven't been able to reproduce this in the lab, it only happens when it goes into production which makes debugging a challenge at best, the process has to get restarted as fast as possible. Spencer On Mar 24, 2010, at 9:42 PM, Jason White wrote:
Spencer Thomason <spencer@5ninesolutions.com> wrote:
What is the best way to debug this?
1. Upgrade to the latest version from the svn repository if you haven't done so already. You didn't specify which version of FreeSWITCH you were using, but if it's old, upgrade it and try to reproduce the problem with the latest version.
2. Assuming you can reproduce the problem with the latest code from svn, I suppose it's time for gdb - instructions are on the wiki.
Can you run fs_cli and connect to it in this state or does it hang completely?
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Can you elaborate a bit more on what is going on? Maybe you just have build skew . /b On Mar 24, 2010, at 11:53 PM, Spencer Thomason wrote:
fs_cli will still connect but you cannot issue any commands to sofia, i.e. restart, etc
I am on svn trunk. The problem is I haven't been able to reproduce this in the lab, it only happens when it goes into production which makes debugging a challenge at best, the process has to get restarted as fast as possible.
Spencer
I have rebuilt and rebuilt, once every couple of days. This has been going on for a while. I was using CentOS but I switched to Ubuntu to try to build against some newer libraries. I thought the problem was resolved but it is very intermittent so its hard to tell. The gdb output is at http://pastebin.freeswitch.org/12451 The Freeswitch process just seems to randomly start using 100% of the CPU and stop responding about once every 12-36 hours. Today it had to be restarted twice, somedays not at all, very light load, about 100 calls per day. root@sip:/usr/local/freeswitch/bin# cat /proc/cpuinfo processor : 0 vendor_id : AuthenticAMD cpu family : 15 model : 33 model name : AMD Opteron(tm) Processor 275 stepping : 2 cpu MHz : 2205.000 cache size : 1024 KB fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow rep_good pni hypervisor cmp_legacy bogomips : 4410.00 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: ts fid vid ttp processor : 1 vendor_id : AuthenticAMD cpu family : 15 model : 33 model name : AMD Opteron(tm) Processor 275 stepping : 2 cpu MHz : 2205.000 cache size : 1024 KB fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow rep_good pni hypervisor cmp_legacy bogomips : 4410.00 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: ts fid vid ttp processor : 2 vendor_id : AuthenticAMD cpu family : 15 model : 33 model name : AMD Opteron(tm) Processor 275 stepping : 2 cpu MHz : 2205.000 cache size : 1024 KB fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow rep_good pni hypervisor cmp_legacy bogomips : 4410.00 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: ts fid vid ttp processor : 3 vendor_id : AuthenticAMD cpu family : 15 model : 33 model name : AMD Opteron(tm) Processor 275 stepping : 2 cpu MHz : 2205.000 cache size : 1024 KB fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu de tsc msr pae cx8 cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm 3dnowext 3dnow rep_good pni hypervisor cmp_legacy bogomips : 4410.00 TLB size : 1024 4K pages clflush size : 64 cache_alignment : 64 address sizes : 40 bits physical, 48 bits virtual power management: ts fid vid ttp On Mar 24, 2010, at 9:58 PM, Brian West wrote:
Can you elaborate a bit more on what is going on? Maybe you just have build skew .
/b
On Mar 24, 2010, at 11:53 PM, Spencer Thomason wrote:
fs_cli will still connect but you cannot issue any commands to sofia, i.e. restart, etc
I am on svn trunk. The problem is I haven't been able to reproduce this in the lab, it only happens when it goes into production which makes debugging a challenge at best, the process has to get restarted as fast as possible.
Spencer
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Well that was your first mistake.. why did you feel the need to use Ubuntu? Also what does uname -a say? /b On Mar 25, 2010, at 12:18 AM, Spencer Thomason wrote:
I have rebuilt and rebuilt, once every couple of days. This has been going on for a while. I was using CentOS but I switched to Ubuntu to try to build against some newer libraries. I thought the problem was resolved but it is very intermittent so its hard to tell. The gdb output is at http://pastebin.freeswitch.org/12451 The Freeswitch process just seems to randomly start using 100% of the CPU and stop responding about once every 12-36 hours. Today it had to be restarted twice, somedays not at all, very light load, about 100 calls per day.
I dunno, just thought I'd try something different.. :-) root@sip:/usr/local/freeswitch/bin# uname -r 2.6.31-20-generic I'd much rather use CentOS. The Ubuntu machine is running under Xen 3.4.1 if that matters. On Mar 24, 2010, at 10:23 PM, Brian West wrote:
Well that was your first mistake.. why did you feel the need to use Ubuntu? Also what does uname -a say?
/b
On Mar 25, 2010, at 12:18 AM, Spencer Thomason wrote:
I have rebuilt and rebuilt, once every couple of days. This has been going on for a while. I was using CentOS but I switched to Ubuntu to try to build against some newer libraries. I thought the problem was resolved but it is very intermittent so its hard to tell. The gdb output is at http://pastebin.freeswitch.org/12451 The Freeswitch process just seems to randomly start using 100% of the CPU and stop responding about once every 12-36 hours. Today it had to be restarted twice, somedays not at all, very light load, about 100 calls per day.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Odd nobody else has reported this behavior. Have you viewed the wiki pages on this topic of running under a vm? /b On Mar 25, 2010, at 12:37 AM, Spencer Thomason wrote:
I dunno, just thought I'd try something different.. :-)
root@sip:/usr/local/freeswitch/bin# uname -r 2.6.31-20-generic
I'd much rather use CentOS. The Ubuntu machine is running under Xen 3.4.1 if that matters.
Lucky me :-) I have. Its an extremely basic setup. Asterisk boxes (1-n) -> Freeswitch (as an SBC) -> ITSP -> PSTN. All Freeswitch is doing is bridging calls from Asterisk to the ITSP. All media is routed point to point. The dialplan is very simple as you can imagine. If I disable registrations to Bandtel everything seems to be ok. but to my eye I don't see anything weird about the format of the register that might cause a problem. Spencer On Mar 24, 2010, at 10:43 PM, Brian West wrote:
Odd nobody else has reported this behavior. Have you viewed the wiki pages on this topic of running under a vm? /b
On Mar 25, 2010, at 12:37 AM, Spencer Thomason wrote:
I dunno, just thought I'd try something different.. :-)
root@sip:/usr/local/freeswitch/bin# uname -r 2.6.31-20-generic
I'd much rather use CentOS. The Ubuntu machine is running under Xen 3.4.1 if that matters.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Next time it happens, prior to a restart of the freeswitch process, run the fscore_pb script to get us a set of backtraces of the running process. support-d/fscore_pb gcore <yourname> try running it before hand to ensure you get expected results. Getting a core does pause the process for a short period of time, so do your test when someone won't notice a RTP hickup. On Thu, Mar 25, 2010 at 1:10 AM, Spencer Thomason < spencer@5ninesolutions.com> wrote:
Lucky me :-) I have. Its an extremely basic setup. Asterisk boxes (1-n) -> Freeswitch (as an SBC) -> ITSP -> PSTN. All Freeswitch is doing is bridging calls from Asterisk to the ITSP. All media is routed point to point. The dialplan is very simple as you can imagine. If I disable registrations to Bandtel everything seems to be ok. but to my eye I don't see anything weird about the format of the register that might cause a problem.
Spencer
On Mar 24, 2010, at 10:43 PM, Brian West wrote:
Odd nobody else has reported this behavior. Have you viewed the wiki pages on this topic of running under a vm? /b
On Mar 25, 2010, at 12:37 AM, Spencer Thomason wrote:
I dunno, just thought I'd try something different.. :-)
root@sip:/usr/local/freeswitch/bin# uname -r 2.6.31-20-generic
I'd much rather use CentOS. The Ubuntu machine is running under Xen 3.4.1 if that matters.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- -Rupa
If you recall, I tried to help you with this already once. I spent a long time on your box building out fresh trunk and configuring it to trace the problem. Then you told me you erased it all by accident =/ I suppose we can try again but I would appreciate if if you gather the info for me. edit conf/autoload_configs/logfile.conf.xml look for this line: <map name="all" value="debug,info,notice,warning,err,crit,alert"/> replace it with: <map name="all" value="debug,info,notice,warning,err,crit,alert,console"/> in the cli: sofia profile internal siptrace on sofia loglevel all 9 (repeat for any other profiles) also run wireshark or tcpdump to capture all traffic to that particular provider When it happens, stop FS and the capture and get me access to the resulting files. This is a rare issue triggered by some strange sip message. What we are trying to do here is get a copy of the traffic that triggers it so we can stop it. On Thu, Mar 25, 2010 at 6:13 AM, Rupa Schomaker <rupa@rupa.com> wrote:
Next time it happens, prior to a restart of the freeswitch process, run the fscore_pb script to get us a set of backtraces of the running process.
support-d/fscore_pb gcore <yourname>
try running it before hand to ensure you get expected results. Getting a core does pause the process for a short period of time, so do your test when someone won't notice a RTP hickup.
On Thu, Mar 25, 2010 at 1:10 AM, Spencer Thomason < spencer@5ninesolutions.com> wrote:
Lucky me :-) I have. Its an extremely basic setup. Asterisk boxes (1-n) -> Freeswitch (as an SBC) -> ITSP -> PSTN. All Freeswitch is doing is bridging calls from Asterisk to the ITSP. All media is routed point to point. The dialplan is very simple as you can imagine. If I disable registrations to Bandtel everything seems to be ok. but to my eye I don't see anything weird about the format of the register that might cause a problem.
Spencer
On Mar 24, 2010, at 10:43 PM, Brian West wrote:
Odd nobody else has reported this behavior. Have you viewed the wiki pages on this topic of running under a vm? /b
On Mar 25, 2010, at 12:37 AM, Spencer Thomason wrote:
I dunno, just thought I'd try something different.. :-)
root@sip:/usr/local/freeswitch/bin# uname -r 2.6.31-20-generic
I'd much rather use CentOS. The Ubuntu machine is running under Xen 3.4.1 if that matters.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- -Rupa
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900
since I forgot to mention, restart FS or at leasr mod_logfile after making the changes to the config On Thu, Mar 25, 2010 at 8:26 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
If you recall, I tried to help you with this already once. I spent a long time on your box building out fresh trunk and configuring it to trace the problem. Then you told me you erased it all by accident =/
I suppose we can try again but I would appreciate if if you gather the info for me.
edit conf/autoload_configs/logfile.conf.xml
look for this line:
<map name="all" value="debug,info,notice,warning,err,crit,alert"/>
replace it with:
<map name="all" value="debug,info,notice,warning,err,crit,alert,console"/>
in the cli:
sofia profile internal siptrace on sofia loglevel all 9
(repeat for any other profiles)
also run wireshark or tcpdump to capture all traffic to that particular provider
When it happens, stop FS and the capture and get me access to the resulting files.
This is a rare issue triggered by some strange sip message. What we are trying to do here is get a copy of the traffic that triggers it so we can stop it.
On Thu, Mar 25, 2010 at 6:13 AM, Rupa Schomaker <rupa@rupa.com> wrote:
Next time it happens, prior to a restart of the freeswitch process, run the fscore_pb script to get us a set of backtraces of the running process.
support-d/fscore_pb gcore <yourname>
try running it before hand to ensure you get expected results. Getting a core does pause the process for a short period of time, so do your test when someone won't notice a RTP hickup.
On Thu, Mar 25, 2010 at 1:10 AM, Spencer Thomason < spencer@5ninesolutions.com> wrote:
Lucky me :-) I have. Its an extremely basic setup. Asterisk boxes (1-n) -> Freeswitch (as an SBC) -> ITSP -> PSTN. All Freeswitch is doing is bridging calls from Asterisk to the ITSP. All media is routed point to point. The dialplan is very simple as you can imagine. If I disable registrations to Bandtel everything seems to be ok. but to my eye I don't see anything weird about the format of the register that might cause a problem.
Spencer
On Mar 24, 2010, at 10:43 PM, Brian West wrote:
Odd nobody else has reported this behavior. Have you viewed the wiki pages on this topic of running under a vm? /b
On Mar 25, 2010, at 12:37 AM, Spencer Thomason wrote:
I dunno, just thought I'd try something different.. :-)
root@sip:/usr/local/freeswitch/bin# uname -r 2.6.31-20-generic
I'd much rather use CentOS. The Ubuntu machine is running under Xen 3.4.1 if that matters.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- -Rupa
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900
Sure, I don't mind doing the legwork. Logging now. Again I apologize for the issues before. Spencer On Mar 25, 2010, at 6:27 AM, Anthony Minessale wrote:
since I forgot to mention,
restart FS or at leasr mod_logfile after making the changes to the config
On Thu, Mar 25, 2010 at 8:26 AM, Anthony Minessale <anthony.minessale@gmail.com
wrote: If you recall, I tried to help you with this already once. I spent a long time on your box building out fresh trunk and configuring it to trace the problem. Then you told me you erased it all by accident =/
I suppose we can try again but I would appreciate if if you gather the info for me.
edit conf/autoload_configs/logfile.conf.xml
look for this line:
<map name="all" value="debug,info,notice,warning,err,crit,alert"/>
replace it with:
<map name="all" value="debug,info,notice,warning,err,crit,alert,console"/>
in the cli:
sofia profile internal siptrace on sofia loglevel all 9
(repeat for any other profiles)
also run wireshark or tcpdump to capture all traffic to that particular provider
When it happens, stop FS and the capture and get me access to the resulting files.
This is a rare issue triggered by some strange sip message. What we are trying to do here is get a copy of the traffic that triggers it so we can stop it.
On Thu, Mar 25, 2010 at 6:13 AM, Rupa Schomaker <rupa@rupa.com> wrote: Next time it happens, prior to a restart of the freeswitch process, run the fscore_pb script to get us a set of backtraces of the running process.
support-d/fscore_pb gcore <yourname>
try running it before hand to ensure you get expected results. Getting a core does pause the process for a short period of time, so do your test when someone won't notice a RTP hickup.
On Thu, Mar 25, 2010 at 1:10 AM, Spencer Thomason <spencer@5ninesolutions.com
wrote: Lucky me :-) I have. Its an extremely basic setup. Asterisk boxes (1-n) -> Freeswitch (as an SBC) -> ITSP -> PSTN. All Freeswitch is doing is bridging calls from Asterisk to the ITSP. All media is routed point to point. The dialplan is very simple as you can imagine. If I disable registrations to Bandtel everything seems to be ok. but to my eye I don't see anything weird about the format of the register that might cause a problem.
Spencer
On Mar 24, 2010, at 10:43 PM, Brian West wrote:
Odd nobody else has reported this behavior. Have you viewed the wiki pages on this topic of running under a vm? /b
On Mar 25, 2010, at 12:37 AM, Spencer Thomason wrote:
I dunno, just thought I'd try something different.. :-)
root@sip:/usr/local/freeswitch/bin# uname -r 2.6.31-20-generic
I'd much rather use CentOS. The Ubuntu machine is running under Xen 3.4.1 if that matters.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- -Rupa
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900 _______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I found the problem in the sofia-sip library, there was a typo causing your endless loop in a very rare condition triggered by your provider's proxy. They are using an unsafe version of SIP reg BTW, that is vulnerable to a replay attack. Its fixed in r17114 On Thu, Mar 25, 2010 at 12:16 PM, Spencer Thomason < spencer@5ninesolutions.com> wrote:
Sure, I don't mind doing the legwork. Logging now. Again I apologize for the issues before.
Spencer
On Mar 25, 2010, at 6:27 AM, Anthony Minessale wrote:
since I forgot to mention,
restart FS or at leasr mod_logfile after making the changes to the config
On Thu, Mar 25, 2010 at 8:26 AM, Anthony Minessale < anthony.minessale@gmail.com> wrote:
If you recall, I tried to help you with this already once. I spent a long time on your box building out fresh trunk and configuring it to trace the problem. Then you told me you erased it all by accident =/
I suppose we can try again but I would appreciate if if you gather the info for me.
edit conf/autoload_configs/logfile.conf.xml
look for this line:
<map name="all" value="debug,info,notice,warning,err,crit,alert"/>
replace it with:
<map name="all" value="debug,info,notice,warning,err,crit,alert,console"/>
in the cli:
sofia profile internal siptrace on sofia loglevel all 9
(repeat for any other profiles)
also run wireshark or tcpdump to capture all traffic to that particular provider
When it happens, stop FS and the capture and get me access to the resulting files.
This is a rare issue triggered by some strange sip message. What we are trying to do here is get a copy of the traffic that triggers it so we can stop it.
On Thu, Mar 25, 2010 at 6:13 AM, Rupa Schomaker <rupa@rupa.com> wrote:
Next time it happens, prior to a restart of the freeswitch process, run the fscore_pb script to get us a set of backtraces of the running process.
support-d/fscore_pb gcore <yourname>
try running it before hand to ensure you get expected results. Getting a core does pause the process for a short period of time, so do your test when someone won't notice a RTP hickup.
On Thu, Mar 25, 2010 at 1:10 AM, Spencer Thomason < spencer@5ninesolutions.com> wrote:
Lucky me :-) I have. Its an extremely basic setup. Asterisk boxes (1-n) -> Freeswitch (as an SBC) -> ITSP -> PSTN. All Freeswitch is doing is bridging calls from Asterisk to the ITSP. All media is routed point to point. The dialplan is very simple as you can imagine. If I disable registrations to Bandtel everything seems to be ok. but to my eye I don't see anything weird about the format of the register that might cause a problem.
Spencer
On Mar 24, 2010, at 10:43 PM, Brian West wrote:
Odd nobody else has reported this behavior. Have you viewed the wiki pages on this topic of running under a vm? /b
On Mar 25, 2010, at 12:37 AM, Spencer Thomason wrote:
I dunno, just thought I'd try something different.. :-)
root@sip:/usr/local/freeswitch/bin# uname -r 2.6.31-20-generic
I'd much rather use CentOS. The Ubuntu machine is running under Xen 3.4.1 if that matters.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- -Rupa
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900 _______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com <MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com<PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <sip%3A888@conference.freeswitch.org> googletalk:conf+888@conference.freeswitch.org<googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900
participants (5)
-
Anthony Minessale
-
Brian West
-
Jason White
-
Rupa Schomaker
-
Spencer Thomason