[Freeswitch-users] high pdd
Anthony Minessale
anthony.minessale at gmail.com
Sat Aug 30 00:45:44 MSD 2014
What about just producing similar load with static dialplans to rule out
user applications?
That should be possible to test on both debian and windows as a baseline.
On Fri, Aug 29, 2014 at 4:32 PM, Andre <andretodd at verizon.net> wrote:
> Thanks all. I put in a ticket and attached the dump images and CPU
> performance as requested.
>
>
>
> I put it under mod_managed but now I think I should of put it under
> something else because the ticket went to Mike.
>
>
>
> https://jira.freeswitch.org/browse/FS-6776
>
>
>
> I’m also working on trying Debian as suggested however the mod_managed
> code needs to be changed. That will take some time.
>
> Thanks everyone for helping.
>
>
>
> Andre
>
>
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
> freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of *Michael
> Jerris
> *Sent:* Thursday, August 28, 2014 11:10 AM
>
> *To:* FreeSWITCH Users Help
> *Subject:* Re: [Freeswitch-users] high pdd
>
>
>
> Most likely it is neither of these. I'm seriously doubting that the
> mutexes are causing problems unless you are actually seeing multiple calls
> blocking each other, which i don't think is happening here. I think you
> need to be looking at what is using the most cpu. Have you looked at
> running in the profiler?
>
>
>
> On Aug 27, 2014, at 6:10 PM, Dave Kompel <drk at drkngs.net> wrote:
>
>
>
> Does anyone still have a dump with the stack trace of those threads in
> them. The more I think about this, the more it “feels” like it’s not the
> mutex code in libapr, but what is calling it. Can someone post one level up
> in the stacktrace?
>
>
>
> --Dave
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org [
> mailto:freeswitch-users-bounces at lists.freeswitch.org
> <freeswitch-users-bounces at lists.freeswitch.org>] *On Behalf Of *Andre
> *Sent:* Wednesday, August 27, 2014 5:34 PM
> *To:* 'FreeSWITCH Users Help'
> *Subject:* Re: [Freeswitch-users] high pdd
>
>
>
> Thanks I’ll do that in the morning.
>
>
>
> We still will try Travis APR code but I’m not sure how to compile it. Any
> hits would be grateful. I’ve read the read me files but I still get errors
> :/
>
>
>
> Also, I’ve installed Debian as suggested and hope to test soon.
>
> Andre
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org [
> mailto:freeswitch-users-bounces at lists.freeswitch.org
> <freeswitch-users-bounces at lists.freeswitch.org>] *On Behalf Of*Moishe
> Grunstein
> *Sent:* Wednesday, August 27, 2014 5:22 PM
> *To:* FreeSWITCH Users Help
> *Subject:* Re: [Freeswitch-users] high pdd
>
>
>
> You should put this in a Jira, this thread is getting very complex to
> track.
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org [
> mailto:freeswitch-users-bounces at lists.freeswitch.org
> <freeswitch-users-bounces at lists.freeswitch.org>] *On Behalf Of*Andrew
> *Sent:* Wednesday, August 27, 2014 4:25 PM
> *To:* 'FreeSWITCH Users Help'
> *Subject:* Re: [Freeswitch-users] high pdd
>
>
>
> Hi,
>
> I know Andre is trying on UNIX but wanted to share what I found..
>
>
>
>
>
> I too have the same issue. But I am using Master x64 embedded.
>
> After 2600 threads the program just keeps adding a ton threads (very fast)
> and wireshark shows
>
> Lots of re invites. My magic number is 1300 ports(bridge). CPS does not
> seem to matter but its more ports/Threads.
>
>
>
> Not sure but when doing a dump I see
> EnterCriticalSection(&cond->csection); is being pointed to in
> thread_cond.c
>
>
>
> static APR_INLINE apr_status_t _thread_cond_timedwait(apr_thread_cond_t
> *cond,
>
> apr_thread_mutex_t
> *mutex,
>
> DWORD timeout_ms )
>
> {
>
> DWORD res;
>
> apr_status_t rv;
>
> unsigned int wake = 0;
>
> unsigned long generation;
>
>
>
> EnterCriticalSection(&cond->csection);
>
> cond->num_waiting++;
>
> generation = cond->generation;
>
> LeaveCriticalSection(&cond->csection);
>
>
>
> apr_thread_mutex_unlock(mutex);
>
>
>
> do {
>
> res = WaitForSingleObject(cond->semaphore, timeout_ms);
>
>
>
> EnterCriticalSection(&cond->csection);
>
>
>
> if (cond->num_wake) {
>
> if (cond->generation != generation) {
>
> cond->num_wake--;
>
> cond->num_waiting--;
>
> rv = APR_SUCCESS;
>
> break;
>
> } else {
>
> wake = 1;
>
> }
>
> }
>
> else if (res != WAIT_OBJECT_0) {
>
> cond->num_waiting--;
>
> rv = APR_TIMEUP;
>
> break;
>
> }
>
>
>
> LeaveCriticalSection(&cond->csection);
>
>
>
> if (wake) {
>
> wake = 0;
>
> ReleaseSemaphore(cond->semaphore, 1, NULL);
>
> }
>
> } while (1);
>
>
>
> LeaveCriticalSection(&cond->csection);
>
> apr_thread_mutex_lock(mutex);
>
>
>
> return rv;
>
> }
>
>
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org [
> mailto:freeswitch-users-bounces at lists.freeswitch.org
> <freeswitch-users-bounces at lists.freeswitch.org>] *On Behalf Of*Anthony
> Minessale
> *Sent:* Wednesday, August 20, 2014 6:13 PM
> *To:* FreeSWITCH Users Help
> *Subject:* Re: [Freeswitch-users] high pdd
>
>
>
> https://github.com/traviscross/apr
>
>
>
>
>
>
>
> On Wed, Aug 20, 2014 at 4:44 PM, Andre <andretodd at verizon.net> wrote:
>
> Thanks Anthony,
>
> I can provide the Dump for you it’s 8gb unzipped so I’ll compress it. The
> tool I use is Visual studio but I’m sure there are other tools too.
>
>
>
> I can run a the profiler tool in visual studio to find the CPU.
>
>
>
> Also how do I get the modified copy of the APR?
>
>
>
> Thanks again guys.
>
>
>
> Andre
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
> freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of*Anthony
> Minessale
> *Sent:* Wednesday, August 20, 2014 5:26 PM
>
>
> *To:* FreeSWITCH Users Help
> *Subject:* Re: [Freeswitch-users] high pdd
>
>
>
> The cond waits are all of your session threads in bypass media mode asleep
> in the state machine.
>
> Its hard to tell with that blurry screen shot what else is going on. Is
> there a way to produce that data in a way we could browse it?
>
>
>
> You should be looking for threads in the sofia sip stack and the ones
> doing originate if you are worried about pdd. Also which threads are
> costing the most CPU. If conditionals are implemented in windows as
> waitforsingleobject i have no idea if that's as efficient as it is in linux.
>
>
>
>
>
> You could try forklifting in our modified copy of APR that has been
> updated to work with their latest release and see if anything has improved
> in the windows implementations of conditionals. I recall Jeff had to
> backport in at least one patch for that over the years.
>
>
>
>
>
>
>
>
>
> On Wed, Aug 20, 2014 at 4:11 PM, Andre <andretodd at verizon.net> wrote:
>
> Ok I’ll try that next. I’ll create xml dialplan to see if that makes a
> difference.
>
>
>
> One note, when I didn’t load the deck in my managed code my dial plan did
> return a 503 in 0 – 20 milliseconds. Not sure if that matters since no
> bridge was ever called.
>
> Thanks again,
>
> Andre
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
> freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of*Michael
> Jerris
> *Sent:* Wednesday, August 20, 2014 5:02 PM
>
>
> *To:* FreeSWITCH Users Help
> *Subject:* Re: [Freeswitch-users] high pdd
>
>
>
> I would test this without any of your managed code at all, to confirm that
> it is not the bottleneck.
>
>
>
> On Aug 20, 2014, at 4:28 PM, Andre <andretodd at verizon.net> wrote:
>
>
>
> I added the new code but it doesn’t seem to fix it. :/
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org [
> mailto:freeswitch-users-bounces at lists.freeswitch.org
> <freeswitch-users-bounces at lists.freeswitch.org>] *On Behalf Of *Andre
> *Sent:* Wednesday, August 20, 2014 3:37 PM
> *To:* 'FreeSWITCH Users Help'
> *Subject:* Re: [Freeswitch-users] high pdd
>
>
>
> Thanks I’ll work on it right a way.
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org [
> mailto:freeswitch-users-bounces at lists.freeswitch.org
> <freeswitch-users-bounces at lists.freeswitch.org>] *On Behalf Of *Peter
> Olsson
> *Sent:* Wednesday, August 20, 2014 3:30 PM
> *To:* FreeSWITCH Users Help
> *Subject:* Re: [Freeswitch-users] high pdd
>
>
>
> It seems it might be stuck in the condition code for Windows.
>
>
>
> Try this timing patch for Windows, and see if it makes any difference.
> I've tested in on lower load, but it would be interesting how it performs
> in this scenario.
>
>
>
>
> https://jira.freeswitch.org/secure/attachment/20613/windows-waitable-timer-diff.txt
>
>
>
> 2014-08-20 21:13 GMT+02:00 Andre <andretodd at verizon.net>:
>
> I will, need to get hardware and figure out how to use mono.
>
> In the meantime, I’ve attached some of the call traces.
>
>
>
>
>
> <image001.jpg><image002.jpg>
>
> <image003.jpg>
>
>
>
>
>
>
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
> freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of*Brian West
> *Sent:* Wednesday, August 20, 2014 3:05 PM
> *To:* FreeSWITCH Users Help
> *Subject:* Re: [Freeswitch-users] high pdd
>
>
>
> Try the same thing on Debian 7 if possible.
>
>
>
> On Wed, Aug 20, 2014 at 12:35 PM, Andre <andretodd at verizon.net> wrote:
>
> Hi, I’m using the master. Should I try 1.4.7?
>
>
>
> I’ve done a dump after things started to go bad/ backed up.
>
>
>
> I see 8193 threads in _*thread_cond*_timedwait. Looks like its on line 81
> of thread_cond.c res = WaitForSingleObject(cond->semaphore, timeout_ms);
>
>
>
> The value of timeout_ms is 2016850392.
>
>
>
> Not sure if this means anything.
>
>
>
> Thanks for the support guys.
>
>
>
> Andre
>
>
>
> *From:* freeswitch-users-bounces at lists.freeswitch.org [mailto:
> freeswitch-users-bounces at lists.freeswitch.org] *On Behalf Of*Brian West
> *Sent:* Wednesday, August 20, 2014 11:57 AM
>
>
> *To:* FreeSWITCH Users Help
> *Subject:* Re: [Freeswitch-users] high pdd
>
>
>
> 1.2.X is EOL,
>
>
>
> 1.4.7 is the current release.
>
>
>
> We have releases and tags in git that no longer carry the 'stable' moniker
> starting with 1.4.
>
>
>
> On Sun, Aug 17, 2014 at 1:39 PM, Andre Demattia <andretodd at verizon.net>
> wrote:
>
> Thanks, I'm using the newest master and the last stable 1.2.23
> We are willing to pay to get this fixed.
> Andre
> ------------------------------
>
> *From: *Anthony Minessale <anthony.minessale at gmail.com>
> *Sent: *8/17/2014 12:30 PM
>
>
> *To: *Freeswitch-users <freeswitch-users at lists.freeswitch.org>
> *Subject: *Re: [Freeswitch-users] high pdd
>
> We dont have a lot of tools to troubleshoot windows. You should mention
> windows in your subject to have a better chance to get the attention of our
> windows users.
>
> You should consider installing debian 7 on the same hardware and perform
> similar tests to help determine if your issue is o/s specific or not.
>
> You also have not indicated the revision of FS. If you have not worked
> with latest master, you should be also to rule out improvements.
>
> On Aug 17, 2014 9:54 AM, "Andre Demattia" <andretodd at verizon.net> wrote:
>
> Yes that's off and I believe all the databases are off too. No
> authentication or registration is enabled. The sizes of the database are 0
> until I call the shutdown command then its about 90k. I have the DBS in Ra
> drive too.
>
> Windows seems pretty good until the 3000 sessions per min mark. Then I
> just get long pdd and the threads get backed up then no more sessions can
> be created.
>
> Funny thing is I has 10 profiles and the same results. I also tried two
> consoles with the same results.
>
> FreeSWITCH is great.
>
> I'd be grateful for help. I've worked on this for 2 years and hate to give
> up.
>
> If any one from consulting at freeswitch.org is willing to help I'd
> appreciate that.
>
> Thanks again,
> Andre
> ------------------------------
>
> *From: *Anthony Minessale <anthony.minessale at gmail.com>
> *Sent: *8/17/2014 10:13 AM
> *To: *Freeswitch-users <freeswitch-users at lists.freeswitch.org>
> *Subject: *Re: [Freeswitch-users] high pdd
>
> Did you comment out manage-presence ? Also we have no real idea how well
> FS performs in windows under extreme load.
>
> On Aug 17, 2014 5:42 AM, "Andre Demattia" <andretodd at verizon.net> wrote:
>
> Thanks Ken, I'll try the consulting again.
> Andre
> ------------------------------
>
> *From: *Ken Rice <krice at freeswitch.org>
> *Sent: *8/17/2014 1:19 AM
> *To: *FreeSWITCH Users Help <freeswitch-users at lists.freeswitch.org>
> *Subject: *Re: [Freeswitch-users] high pdd
>
> if the last post does not work contact consulting at freeswitch.org for pro
> help from one of the developers... thats about as far as the list will take
> you
>
> Ken
>
> Sent from my iPad
>
>
> On Aug 16, 2014, at 15:23, "Andre" <andretodd at verizon.net> wrote:
>
> Going with the idea it’s IO related I did a db_cache status. I just
> disabled limit in my code by removing it but limit should be HASH not a
> database however I stills how a database.
>
>
>
> Below is the results of the db cache status. I’m not real sure what its
> saying. I should have all registration and limit disabled and should not be
> using any sql database from my understanding.
>
>
>
> Anyone know what the results mean? I’ve only tested 2 calls on this status.
>
>
>
> [The entire original message is not included.]
>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
>
>
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
--
Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞
http://twitter.com/FreeSWITCH
☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+
<http://freeswitch.org/g+>*
ClueCon Weekly Development Call
☎ sip:888 at conference.freeswitch.org ☎ +19193869900
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20140829/75b10db3/attachment-0001.html
Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users
mailing list