You should put this in a Jira, this thread is getting very complex to track. From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Andrew Sent: Wednesday, August 27, 2014 4:25 PM To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] high pdd Hi, I know Andre is trying on UNIX but wanted to share what I found.. I too have the same issue. But I am using Master x64 embedded. After 2600 threads the program just keeps adding a ton threads (very fast) and wireshark shows Lots of re invites. My magic number is 1300 ports(bridge). CPS does not seem to matter but its more ports/Threads. Not sure but when doing a dump I see EnterCriticalSection(&cond->csection); is being pointed to in thread_cond.c static APR_INLINE apr_status_t _thread_cond_timedwait(apr_thread_cond_t *cond, apr_thread_mutex_t *mutex, DWORD timeout_ms ) { DWORD res; apr_status_t rv; unsigned int wake = 0; unsigned long generation; EnterCriticalSection(&cond->csection); cond->num_waiting++; generation = cond->generation; LeaveCriticalSection(&cond->csection); apr_thread_mutex_unlock(mutex); do { res = WaitForSingleObject(cond->semaphore, timeout_ms); EnterCriticalSection(&cond->csection); if (cond->num_wake) { if (cond->generation != generation) { cond->num_wake--; cond->num_waiting--; rv = APR_SUCCESS; break; } else { wake = 1; } } else if (res != WAIT_OBJECT_0) { cond->num_waiting--; rv = APR_TIMEUP; break; } LeaveCriticalSection(&cond->csection); if (wake) { wake = 0; ReleaseSemaphore(cond->semaphore, 1, NULL); } } while (1); LeaveCriticalSection(&cond->csection); apr_thread_mutex_lock(mutex); return rv; } From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: Wednesday, August 20, 2014 6:13 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd https://github.com/traviscross/apr On Wed, Aug 20, 2014 at 4:44 PM, Andre <andretodd@verizon.net<mailto:andretodd@verizon.net>> wrote: Thanks Anthony, I can provide the Dump for you it’s 8gb unzipped so I’ll compress it. The tool I use is Visual studio but I’m sure there are other tools too. I can run a the profiler tool in visual studio to find the CPU. Also how do I get the modified copy of the APR? Thanks again guys. Andre From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Anthony Minessale Sent: Wednesday, August 20, 2014 5:26 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd The cond waits are all of your session threads in bypass media mode asleep in the state machine. Its hard to tell with that blurry screen shot what else is going on. Is there a way to produce that data in a way we could browse it? You should be looking for threads in the sofia sip stack and the ones doing originate if you are worried about pdd. Also which threads are costing the most CPU. If conditionals are implemented in windows as waitforsingleobject i have no idea if that's as efficient as it is in linux. You could try forklifting in our modified copy of APR that has been updated to work with their latest release and see if anything has improved in the windows implementations of conditionals. I recall Jeff had to backport in at least one patch for that over the years. On Wed, Aug 20, 2014 at 4:11 PM, Andre <andretodd@verizon.net<mailto:andretodd@verizon.net>> wrote: Ok I’ll try that next. I’ll create xml dialplan to see if that makes a difference. One note, when I didn’t load the deck in my managed code my dial plan did return a 503 in 0 – 20 milliseconds. Not sure if that matters since no bridge was ever called. Thanks again, Andre From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf Of Michael Jerris Sent: Wednesday, August 20, 2014 5:02 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd I would test this without any of your managed code at all, to confirm that it is not the bottleneck. On Aug 20, 2014, at 4:28 PM, Andre <andretodd@verizon.net<mailto:andretodd@verizon.net>> wrote: I added the new code but it doesn’t seem to fix it. :/ From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Andre Sent: Wednesday, August 20, 2014 3:37 PM To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] high pdd Thanks I’ll work on it right a way. From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Peter Olsson Sent: Wednesday, August 20, 2014 3:30 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd It seems it might be stuck in the condition code for Windows. Try this timing patch for Windows, and see if it makes any difference. I've tested in on lower load, but it would be interesting how it performs in this scenario. https://jira.freeswitch.org/secure/attachment/20613/windows-waitable-timer-d... 2014-08-20 21:13 GMT+02:00 Andre <andretodd@verizon.net<mailto:andretodd@verizon.net>>: I will, need to get hardware and figure out how to use mono. In the meantime, I’ve attached some of the call traces. <image001.jpg><image002.jpg> <image003.jpg> From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf OfBrian West Sent: Wednesday, August 20, 2014 3:05 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd Try the same thing on Debian 7 if possible. On Wed, Aug 20, 2014 at 12:35 PM, Andre <andretodd@verizon.net<mailto:andretodd@verizon.net>> wrote: Hi, I’m using the master. Should I try 1.4.7? I’ve done a dump after things started to go bad/ backed up. I see 8193 threads in _thread_cond_timedwait. Looks like its on line 81 of thread_cond.c res = WaitForSingleObject(cond->semaphore, timeout_ms); The value of timeout_ms is 2016850392<tel:2016850392>. Not sure if this means anything. Thanks for the support guys. Andre From: freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org<mailto:freeswitch-users-bounces@lists.freeswitch.org>] On Behalf OfBrian West Sent: Wednesday, August 20, 2014 11:57 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd 1.2.X is EOL, 1.4.7 is the current release. We have releases and tags in git that no longer carry the 'stable' moniker starting with 1.4. On Sun, Aug 17, 2014 at 1:39 PM, Andre Demattia <andretodd@verizon.net<mailto:andretodd@verizon.net>> wrote: Thanks, I'm using the newest master and the last stable 1.2.23 We are willing to pay to get this fixed. Andre ________________________________ From: Anthony Minessale<mailto:anthony.minessale@gmail.com> Sent: 8/17/2014 12:30 PM To: Freeswitch-users<mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] high pdd We dont have a lot of tools to troubleshoot windows. You should mention windows in your subject to have a better chance to get the attention of our windows users. You should consider installing debian 7 on the same hardware and perform similar tests to help determine if your issue is o/s specific or not. You also have not indicated the revision of FS. If you have not worked with latest master, you should be also to rule out improvements. On Aug 17, 2014 9:54 AM, "Andre Demattia" <andretodd@verizon.net<mailto:andretodd@verizon.net>> wrote: Yes that's off and I believe all the databases are off too. No authentication or registration is enabled. The sizes of the database are 0 until I call the shutdown command then its about 90k. I have the DBS in Ra drive too. Windows seems pretty good until the 3000 sessions per min mark. Then I just get long pdd and the threads get backed up then no more sessions can be created. Funny thing is I has 10 profiles and the same results. I also tried two consoles with the same results. FreeSWITCH is great. I'd be grateful for help. I've worked on this for 2 years and hate to give up. If any one from consulting@freeswitch.org<mailto:consulting@freeswitch.org> is willing to help I'd appreciate that. Thanks again, Andre ________________________________ From: Anthony Minessale<mailto:anthony.minessale@gmail.com> Sent: 8/17/2014 10:13 AM To: Freeswitch-users<mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] high pdd Did you comment out manage-presence ? Also we have no real idea how well FS performs in windows under extreme load. On Aug 17, 2014 5:42 AM, "Andre Demattia" <andretodd@verizon.net<mailto:andretodd@verizon.net>> wrote: Thanks Ken, I'll try the consulting again. Andre ________________________________ From: Ken Rice<mailto:krice@freeswitch.org> Sent: 8/17/2014 1:19 AM To: FreeSWITCH Users Help<mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] high pdd if the last post does not work contact consulting@freeswitch.org<mailto:consulting@freeswitch.org> for pro help from one of the developers... thats about as far as the list will take you Ken Sent from my iPad On Aug 16, 2014, at 15:23, "Andre" <andretodd@verizon.net<mailto:andretodd@verizon.net>> wrote: Going with the idea it’s IO related I did a db_cache status. I just disabled limit in my code by removing it but limit should be HASH not a database however I stills how a database. Below is the results of the db cache status. I’m not real sure what its saying. I should have all registration and limit disabled and should not be using any sql database from my understanding. Anyone know what the results mean? I’ve only tested 2 calls on this status. [The entire original message is not included.] _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com<http://www.cudatel.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users [The entire original message is not included.] _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com<http://www.cudatel.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Brian West brian@freeswitch.org<mailto:brian@freeswitch.org> [http://bkw.org/whmcslogo.png] Twitter: @FreeSWITCH , @briankwest http://www.freeswitchbook.com<http://www.freeswitchbook.com/> http://www.freeswitchcookbook.com<http://www.freeswitchcookbook.com/> T:+19184209001<tel:%2B19184209001> | F:+19184209002<tel:%2B19184209002> | M:+1918424WEST (9378) iNUM:+883 5100 1420 9001 | ISN:410*543 | Skype:briankwest _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com<http://www.cudatel.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> -- Brian West brian@freeswitch.org<mailto:brian@freeswitch.org> [http://bkw.org/whmcslogo.png] Twitter: @FreeSWITCH , @briankwest http://www.freeswitchbook.com<http://www.freeswitchbook.com/> http://www.freeswitchcookbook.com<http://www.freeswitchcookbook.com/> T:+19184209001<tel:%2B19184209001> | F:+19184209002<tel:%2B19184209002> | M:+1918424WEST (9378) iNUM:+883 5100 1420 9001 | ISN:410*543 | Skype:briankwest _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com<http://www.freeswitchsolutions.com/> Official FreeSWITCH Sites http://www.freeswitch.org<http://www.freeswitch.org/> http://confluence.freeswitch.org<http://confluence.freeswitch.org/> http://www.cluecon.com<http://www.cluecon.com/> FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com<http://www.cudatel.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org<http://www.freeswitch.org/> _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900<tel:%2B19193869900> _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org<mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org<mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net<http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org<mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900