Hi All, I'm running a freeswitch server with moderate load (at least, in my opinion..). Everything runs fine and stable. There is however one issue I have: occasionally a channel/thread gots stuck (and keeps that way until a full restart). Doing a uuid_kill doesn't help, and a uuid_dump got locked itself as well. I managed to get a core dump from the running process, and I can see a couple of threads in a locked state: "#0 0x00007f3878764be4 in __lll_lock_wait () from /lib/libpthread.so.0" etc. This is where my experience ends. How should I proceed in finding the cause of these locks? I've read these kind of locks can happen when doing 'fancy' things with cdr processing etc. In my setup this doesn't seem te be the case, as this is a 'simple' gateway only using the default modules. (for the stack trace see http://pastebin.freeswitch.org/18526 version: commit 224f4c0f8aefcb969ecab7ff9d3a66dce92bd943 / Tue Jan 24 11:38:41 2012 -0600). regards, Egbert.
I think you start by raising a JIRA; this doesn't sound like planned behaviour and in that case the authors would be very interested to assist. On 27/02/12 01:43, [Redhosting] Egbert Groot wrote:
Hi All,
I'm running a freeswitch server with moderate load (at least, in my opinion..). Everything runs fine and stable. There is however one issue I have: occasionally a channel/thread gots stuck (and keeps that way until a full restart). Doing a uuid_kill doesn't help, and a uuid_dump got locked itself as well. I managed to get a core dump from the running process, and I can see a couple of threads in a locked state: "#0 0x00007f3878764be4 in __lll_lock_wait () from /lib/libpthread.so.0" etc. This is where my experience ends. How should I proceed in finding the cause of these locks? I've read these kind of locks can happen when doing 'fancy' things with cdr processing etc. In my setup this doesn't seem te be the case, as this is a 'simple' gateway only using the default modules.
(for the stack trace see http://pastebin.freeswitch.org/18526 version: commit 224f4c0f8aefcb969ecab7ff9d3a66dce92bd943 / Tue Jan 24 11:38:41 2012 -0600).
regards, Egbert.
On 27-02-12 02:43, [Redhosting] Egbert Groot wrote:
Hi All,
I'm running a freeswitch server with moderate load (at least, in my opinion..). Everything runs fine and stable. There is however one issue I have: occasionally a channel/thread gots stuck (and keeps that way until a full restart). Doing a uuid_kill doesn't help, and a uuid_dump got locked itself as well. I managed to get a core dump from the running process, and I can see a couple of threads in a locked state: "#0 0x00007f3878764be4 in __lll_lock_wait () from /lib/libpthread.so.0" etc. This is where my experience ends. How should I proceed in finding the cause of these locks? I've read these kind of locks can happen when doing 'fancy' things with cdr processing etc. In my setup this doesn't seem te be the case, as this is a 'simple' gateway only using the default modules.
(for the stack trace see http://pastebin.freeswitch.org/18526 version: commit 224f4c0f8aefcb969ecab7ff9d3a66dce92bd943 / Tue Jan 24 11:38:41 2012 -0600).
Bugs should always be submitted to Jira: http://jira.freeswitch.org The developers will probably want to see that backtrace so please include it in the bugreport. Please also checkout the following link so you provide the developers with the information they need: http://wiki.freeswitch.org/wiki/Reporting_Bugs#Reporting_A_Bug_With_JIRA Since your version is more than a month old you might get the request to first update to today's git master and verify if the bug still happens. So I guess you could try that first. Regards, Patrick
Start by upgrading to latest git HEAD. A bug like this was fixed last week, so I'm pretty sure an update will solve your issue. /Peter ________________________________________ Från: freeswitch-users-bounces@lists.freeswitch.org [freeswitch-users-bounces@lists.freeswitch.org] för Patrick Lists [freeswitch-list@puzzled.xs4all.nl] Skickat: den 27 februari 2012 13:33 Till: FreeSWITCH Users Help Ämne: Re: [Freeswitch-users] Help with debugging stuck threads/channels On 27-02-12 02:43, [Redhosting] Egbert Groot wrote:
Hi All,
I'm running a freeswitch server with moderate load (at least, in my opinion..). Everything runs fine and stable. There is however one issue I have: occasionally a channel/thread gots stuck (and keeps that way until a full restart). Doing a uuid_kill doesn't help, and a uuid_dump got locked itself as well. I managed to get a core dump from the running process, and I can see a couple of threads in a locked state: "#0 0x00007f3878764be4 in __lll_lock_wait () from /lib/libpthread.so.0" etc. This is where my experience ends. How should I proceed in finding the cause of these locks? I've read these kind of locks can happen when doing 'fancy' things with cdr processing etc. In my setup this doesn't seem te be the case, as this is a 'simple' gateway only using the default modules.
(for the stack trace see http://pastebin.freeswitch.org/18526 version: commit 224f4c0f8aefcb969ecab7ff9d3a66dce92bd943 / Tue Jan 24 11:38:41 2012 -0600).
Bugs should always be submitted to Jira: http://jira.freeswitch.org The developers will probably want to see that backtrace so please include it in the bugreport. Please also checkout the following link so you provide the developers with the information they need: http://wiki.freeswitch.org/wiki/Reporting_Bugs#Reporting_A_Bug_With_JIRA Since your version is more than a month old you might get the request to first update to today's git master and verify if the bug still happens. So I guess you could try that first. Regards, Patrick _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org !DSPAM:4f4b773032761169318834!
Op 27-2-2012 18:44, Peter Olsson schreef:
Start by upgrading to latest git HEAD. A bug like this was fixed last week, so I'm pretty sure an update will solve your issue.
/Peter
You're probably referring to jira #FS-3932. Looks like the same issue indeed. Freeswitch moves fast! I will update as soon as I can. Apart from this specific issue, I can conclude having locked threads is reason to open a jira issue (if not someone else did already..) Thank you (and John and Patrick) for taking the time to write a reply. regards, Egbert.
________________________________________ Från: freeswitch-users-bounces@lists.freeswitch.org [freeswitch-users-bounces@lists.freeswitch.org] för Patrick Lists [freeswitch-list@puzzled.xs4all.nl] Skickat: den 27 februari 2012 13:33 Till: FreeSWITCH Users Help Ämne: Re: [Freeswitch-users] Help with debugging stuck threads/channels
On 27-02-12 02:43, [Redhosting] Egbert Groot wrote:
Hi All,
I'm running a freeswitch server with moderate load (at least, in my opinion..). Everything runs fine and stable. There is however one issue I have: occasionally a channel/thread gots stuck (and keeps that way until a full restart). Doing a uuid_kill doesn't help, and a uuid_dump got locked itself as well. I managed to get a core dump from the running process, and I can see a couple of threads in a locked state: "#0 0x00007f3878764be4 in __lll_lock_wait () from /lib/libpthread.so.0" etc. This is where my experience ends. How should I proceed in finding the cause of these locks? I've read these kind of locks can happen when doing 'fancy' things with cdr processing etc. In my setup this doesn't seem te be the case, as this is a 'simple' gateway only using the default modules.
(for the stack trace see http://pastebin.freeswitch.org/18526 version: commit 224f4c0f8aefcb969ecab7ff9d3a66dce92bd943 / Tue Jan 24 11:38:41 2012 -0600). Bugs should always be submitted to Jira: http://jira.freeswitch.org The developers will probably want to see that backtrace so please include it in the bugreport.
Please also checkout the following link so you provide the developers with the information they need: http://wiki.freeswitch.org/wiki/Reporting_Bugs#Reporting_A_Bug_With_JIRA
Since your version is more than a month old you might get the request to first update to today's git master and verify if the bug still happens. So I guess you could try that first.
Regards, Patrick
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
!DSPAM:4f4b773032761169318834!
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I'm not sure your backtrace shows actually locked thread, more that the threads are waiting for a lock just at that specific moment, so I'd just update to latest (and greatest) - I'm pretty sure it will solve the issues. /Peter ________________________________________ Från: freeswitch-users-bounces@lists.freeswitch.org [freeswitch-users-bounces@lists.freeswitch.org] för Egbert Groot [egbert@redhosting.nl] Skickat: den 27 februari 2012 20:46 Till: FreeSWITCH Users Help Ämne: Re: [Freeswitch-users] Help with debugging stuck threads/channels Op 27-2-2012 18:44, Peter Olsson schreef:
Start by upgrading to latest git HEAD. A bug like this was fixed last week, so I'm pretty sure an update will solve your issue.
/Peter
You're probably referring to jira #FS-3932. Looks like the same issue indeed. Freeswitch moves fast! I will update as soon as I can. Apart from this specific issue, I can conclude having locked threads is reason to open a jira issue (if not someone else did already..) Thank you (and John and Patrick) for taking the time to write a reply. regards, Egbert.
________________________________________ Från: freeswitch-users-bounces@lists.freeswitch.org [freeswitch-users-bounces@lists.freeswitch.org] för Patrick Lists [freeswitch-list@puzzled.xs4all.nl] Skickat: den 27 februari 2012 13:33 Till: FreeSWITCH Users Help Ämne: Re: [Freeswitch-users] Help with debugging stuck threads/channels
On 27-02-12 02:43, [Redhosting] Egbert Groot wrote:
Hi All,
I'm running a freeswitch server with moderate load (at least, in my opinion..). Everything runs fine and stable. There is however one issue I have: occasionally a channel/thread gots stuck (and keeps that way until a full restart). Doing a uuid_kill doesn't help, and a uuid_dump got locked itself as well. I managed to get a core dump from the running process, and I can see a couple of threads in a locked state: "#0 0x00007f3878764be4 in __lll_lock_wait () from /lib/libpthread.so.0" etc. This is where my experience ends. How should I proceed in finding the cause of these locks? I've read these kind of locks can happen when doing 'fancy' things with cdr processing etc. In my setup this doesn't seem te be the case, as this is a 'simple' gateway only using the default modules.
(for the stack trace see http://pastebin.freeswitch.org/18526 version: commit 224f4c0f8aefcb969ecab7ff9d3a66dce92bd943 / Tue Jan 24 11:38:41 2012 -0600). Bugs should always be submitted to Jira: http://jira.freeswitch.org The developers will probably want to see that backtrace so please include it in the bugreport.
Please also checkout the following link so you provide the developers with the information they need: http://wiki.freeswitch.org/wiki/Reporting_Bugs#Reporting_A_Bug_With_JIRA
Since your version is more than a month old you might get the request to first update to today's git master and verify if the bug still happens. So I guess you could try that first.
Regards, Patrick
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
!DSPAM:4f4b773032761169318834!
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (5)
-
[Redhosting] Egbert Groot
-
Egbert Groot
-
John
-
Patrick Lists
-
Peter Olsson