if they went away by themselves they must not have been hung?<br><br><div class="gmail_quote">On Thu, Mar 5, 2009 at 5:39 PM, Nik Middleton <span dir="ltr"><<a href="mailto:nik.middleton@noblesolutions.co.uk">nik.middleton@noblesolutions.co.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Well if it's any consolation, I have a 4 day ish old copy of SVN and I<br>
have around 200 of these hung calls, though after an hour or so they did<br>
seem to clear.<br>
<br>
That said, FS made 138,330 call attempts today, not too shabby, and<br>
through out the call quality was as good as the first one. Not sure how<br>
to debug this one.<br>
<br>
Version: FreeSWITCH Version 1.0.trunk (12276)<br>
<div><div></div><div class="h5"><br>
-----Original Message-----<br>
From: <a href="mailto:freeswitch-users-bounces@lists.freeswitch.org">freeswitch-users-bounces@lists.freeswitch.org</a><br>
[mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org">freeswitch-users-bounces@lists.freeswitch.org</a>] On Behalf Of Eric<br>
Liedtke<br>
Sent: 05 March 2009 23:23<br>
To: <a href="mailto:freeswitch-users@lists.freeswitch.org">freeswitch-users@lists.freeswitch.org</a><br>
Subject: Re: [Freeswitch-users] Hung Channels (SVN Rev 10231)<br>
<br>
Yup, as I mentioned to brian didn't want to clog jira with a bug that's<br>
been fixed or report against a rev 2k+ revs behind. I was trying to work<br>
through it as a learning exercise. And yeah I actually added a bunch of<br>
stuff to the list_sessions function to spit out a variety of associated<br>
variables for each session looking for a pattern somewhere to clue me<br>
into what might be happening.<br>
<br>
No proxy or bypass media here, just defaults.<br>
<br>
I will keep at it and once we update the production systems, if the<br>
problem persists I will open a bug in jira with all the neccessary<br>
goodies.<br>
<br>
Thanks<br>
-e<br>
<br>
It's seems fuzzy now but I think on Thu, Mar 05, 2009 at 05:55:33PM<br>
-0500 , Mathieu Rene said:<br>
> HI,<br>
><br>
> If you suspect a bug, the place to report it is JIRA. See:<br>
<a href="http://wiki.freeswitch.org/wiki/Reporting_Bugs" target="_blank">http://wiki.freeswitch.org/wiki/Reporting_Bugs</a><br>
> .<br>
> This gives the whole team a way of following up on issues.<br>
><br>
> Also can you upgrade to svn trunk? A lot of fixes gets committed<br>
> daily, so its good to stay up to date.<br>
><br>
> As you seem familiar with GDB, you may symlink the .gdbinit file in<br>
> the support-d/ folder to your home directory.<br>
> This will give you some FS-specific macros such as "list_sessions"<br>
> which will dump a list of uuids to session pointers.<br>
><br>
> In your jira, make sure you include "thread apply all bt",<br>
> "list_sessions" and show channels (this one goes in FS) but PLEASE<br>
> update to svn trunk and test again to see if it still happens.<br>
><br>
> Also, are you using proxy/bypass media or just the default?<br>
><br>
> Math<br>
><br>
> On 5-Mar-09, at 5:38 PM, Eric Liedtke wrote:<br>
><br>
> > Greetings,<br>
> ><br>
> > I've been using FS in production on this rev (I realize it's pretty<br>
<br>
> > far<br>
> > behind current) and it's been running well, save 1 issue.<br>
> ><br>
> > The basic setup is an SBC , 2 GiG-E ports, 1 public , 1 private. I<br>
> > have<br>
> > 2 sip profiles created , 1 per ip interface. This is being used to<br>
> > terminate traffic to a provider so calls are only 1 direction. They<br>
<br>
> > come<br>
> > into the private side profile, get routed via dialplan to the<br>
gateway<br>
> > defined in the external profile and on to the vendor. Pretty simple.<br>
> ><br>
> > I have noticed that under load (50 or so cps with ~800-900 bridged<br>
> > calls up)<br>
> > that over time some channels on the public side seem to get<br>
> > "stuck". Due to<br>
> > the nature of how this is being used , I would expect both sip<br>
> > profiles to show<br>
> > the same number of channels in use any time i do a 'sofia<br>
> > status' ( or at least<br>
> > be within a channel or 2 of each other). However after a day of<br>
> > heavy use I had<br>
> > a disparity of ~250 channels. These extra channels also seem to put<br>
<br>
> > some<br>
> > continual load on the 'system cpu' as well , reported via top.<br>
> ><br>
> > Of course due to the load on the box I have to keep logging turned<br>
way<br>
> > down. So I've been trying to troubleshoot it as best I can.<br>
> ><br>
> > Last night I grabbed a core file and started in with GDB today. I<br>
> > found<br>
> > the 120 or so threads that represented real active calls when I took<br>
<br>
> > the<br>
> > corefile, I also found ~250 threads that appeared to be stuck in the<br>
> > CS_NEW state. The backtraces on all of them looks the same,<br>
> > annotated below.<br>
> ><br>
> > I walked through the code path by hand , based on the bt's and I<br>
> > don't see how<br>
> > this could be happening unless it's a locking issue. But as far as<br>
<br>
> > I can tell<br>
> > each session has it's own mutex defined in the<br>
> > switch_core_session_t struct,<br>
> > so I wouldn't think they would be stepping on each other. I also<br>
> > would have expected<br>
> > if it were something of a deadlock nature it would stop processing<br>
> > calls all<br>
> > together.<br>
> ><br>
> > I grabbed the commands from the .gdbinit (super handy btw!!) and<br>
> > have been trolling<br>
> > through the variables to try to ascertain something about why these<br>
<br>
> > threads seem to<br>
> > be stuck, but am not having much luck even coming up with a scenario<br>
<br>
> > to try<br>
> > to replicate the issue.<br>
> ><br>
> > If anyone has any pointers as to where I might look next it would be<br>
<br>
> > greatly<br>
> > appreciated.<br>
> ><br>
> > We will be updating to the newest release soon, however I was hoping<br>
<br>
> > to nail down<br>
> > what is going so I can systematically replicate it and verify by<br>
> > testing in the lab<br>
> > that it is fixed , rather than just pushing the new release to<br>
> > produvction and hoping.<br>
> ><br>
> > Thanks in advance for any tips/pointers anyone may have.<br>
> ><br>
> > -e<br>
> ><br>
> > ......bt and bt full for a single "hung" thread<br>
> ><br>
> ><br>
> > #0 0xb7fd5410 in __kernel_vsyscall ()<br>
> > #1 0xb7d14cb6 in nanosleep () from /lib/tls/i686/cmov/libc.so.6<br>
> > #2 0xb7d4f1dc in usleep () from /lib/tls/i686/cmov/libc.so.6<br>
> > #3 0xb7ee02cd in switch_sleep (t=1000) at src/switch_time.c:143<br>
> > #4 0xb7e9da03 in switch_core_session_run (session=0x95fe270) at<br>
src/<br>
> > switch_core_state_machine.c:462<br>
> > #5 0xb7e9c765 in switch_core_session_thread (thread=0x9ada840,<br>
> > obj=0x95fe270) at src/switch_core_session.c:853<br>
> > #6 0xb7efd916 in dummy_worker (opaque=0x9ada840) at<br>
threadproc/unix/<br>
> > thread.c:138<br>
> > #7 0xb7e034fb in start_thread () from /lib/tls/i686/cmov/<br>
> > libpthread.so.0<br>
> > #8 0xb7d55e5e in clone () from /lib/tls/i686/cmov/libc.so.6<br>
> > (gdb) bt full<br>
> > #0 0xb7fd5410 in __kernel_vsyscall ()<br>
> > No symbol table info available.<br>
> > #1 0xb7d14cb6 in nanosleep () from /lib/tls/i686/cmov/libc.so.6<br>
> > No symbol table info available.<br>
> > #2 0xb7d4f1dc in usleep () from /lib/tls/i686/cmov/libc.so.6<br>
> > No symbol table info available.<br>
> > #3 0xb7ee02cd in switch_sleep (t=1000) at src/switch_time.c:143<br>
> > No locals.<br>
> > #4 0xb7e9da03 in switch_core_session_run (session=0x95fe270) at<br>
src/<br>
> > switch_core_state_machine.c:462<br>
> > exception = 0 '\0'<br>
> > state = <value optimized out><br>
> > endstate = CS_NEW<br>
> > endpoint_interface = <value optimized out><br>
> > driver_state_handler = (const switch_state_handler_table_t *)<br>
<br>
> > 0xb73b1720<br>
> > application_state_handler = <value optimized out><br>
> > thread_id = 3085554955<br>
> > env = {{__jmpbuf = {134603552, -1428248680, -1461722504,<br>
> > 9184, -1210273432, -1210014020}, __mask_was_saved = -1210034895,<br>
> > __saved_mask = {__val = {0, 3084988404, 3084937740, 3086469280,<br>
> > 9184, 1, 2976641592, 2833244792, 3086590960,<br>
> > 168036728, 3084937740, 2833244808, 3085923728, 1, 3086590960,<br>
<br>
> > 2833244840, 3086590960, 0, 134564192, 2833244840, 3085923728,<br>
> > 134564244, 3086590960, 2833244872, 3085887870, 134564240, 168036728,<br>
<br>
> > 3085458203, 3086590960, 2976606624,<br>
> > 134564192, 2833244904}}}}<br>
> > sig = <value optimized out><br>
> > __func__ = "switch_core_session_run"<br>
> > __PRETTY_FUNCTION__ = "switch_core_session_run"<br>
> > #5 0xb7e9c765 in switch_core_session_thread (thread=0x9ada840,<br>
> > obj=0x95fe270) at src/switch_core_session.c:853<br>
> > session = (switch_core_session_t *) 0x95fe270<br>
> > event = <value optimized out><br>
> > event_str = 0x0<br>
> > val = <value optimized out><br>
> > __func__ = "switch_core_session_thread"<br>
> > __PRETTY_FUNCTION__ = "switch_core_session_thread"<br>
> > #6 0xb7efd916 in dummy_worker (opaque=0x9ada840) at<br>
threadproc/unix/<br>
> > thread.c:138<br>
> > No locals.<br>
> > #7 0xb7e034fb in start_thread () from /lib/tls/i686/cmov/<br>
> > libpthread.so.0<br>
> > No symbol table info available.<br>
> > #8 0xb7d55e5e in clone () from /lib/tls/i686/cmov/libc.so.6<br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Freeswitch-users mailing list<br>
> > <a href="mailto:Freeswitch-users@lists.freeswitch.org">Freeswitch-users@lists.freeswitch.org</a><br>
> > <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
> ><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
> > <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
><br>
><br>
> _______________________________________________<br>
> Freeswitch-users mailing list<br>
> <a href="mailto:Freeswitch-users@lists.freeswitch.org">Freeswitch-users@lists.freeswitch.org</a><br>
> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br>
_______________________________________________<br>
Freeswitch-users mailing list<br>
<a href="mailto:Freeswitch-users@lists.freeswitch.org">Freeswitch-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br>
_______________________________________________<br>
Freeswitch-users mailing list<br>
<a href="mailto:Freeswitch-users@lists.freeswitch.org">Freeswitch-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Anthony Minessale II<br><br>FreeSWITCH <a href="http://www.freeswitch.org/">http://www.freeswitch.org/</a><br>ClueCon <a href="http://www.cluecon.com/">http://www.cluecon.com/</a><br>
<br>AIM: anthm<br><a href="mailto:MSN%3Aanthony_minessale@hotmail.com">MSN:anthony_minessale@hotmail.com</a><br>GTALK/JABBER/<a href="mailto:PAYPAL%3Aanthony.minessale@gmail.com">PAYPAL:anthony.minessale@gmail.com</a><br>
IRC: <a href="http://irc.freenode.net">irc.freenode.net</a> #freeswitch<br><br>FreeSWITCH Developer Conference<br><a href="mailto:sip%3A888@conference.freeswitch.org">sip:888@conference.freeswitch.org</a><br><a href="http://iax:guest@conference.freeswitch.org/888">iax:guest@conference.freeswitch.org/888</a><br>
<a href="mailto:googletalk%3Aconf%2B888@conference.freeswitch.org">googletalk:conf+888@conference.freeswitch.org</a><br>pstn:213-799-1400<br>