I run it now with no sql and the sqllite db in a ram drive. Should I run the entire process in ram drive? I’ve done that in the past but I can try again. From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Michael Jerris Sent: Friday, August 15, 2014 3:42 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd depends what its doing, but i suspect you are hitting some I/O wait of some sort. On Aug 15, 2014, at 3:38 PM, Andre <andretodd@verizon.net <mailto:andretodd@verizon.net> > wrote: What would cause very high pdd 12 seconds + after 3000 ports. Wireshark shows reinvites. I have tried many profiles and even many console apps both seem to have the same results after 2500 to 3000 ports (only 300 are connected) The server cpu and memory are ½ the server limits. Any ideas where to look would be great.

What about logging levels (debug?) and CDRs? Are they going to disk that might be causing high I/O? Also consider trying something more powerful than SQLite for the database to see if that helps - eg PostgreSQL/MySQL On 16 August 2014 14:03, Andre Demattia <andretodd@verizon.net> wrote:

Any other hints that might help me figure out what could be the issue? Thanks ------------------------------ From: Andre <andretodd@verizon.net> Sent: ‎8/‎15/‎2014 3:52 PM

To: 'FreeSWITCH Users Help' <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] high pdd

I run it now with no sql and the sqllite db in a ram drive. Should I run the entire process in ram drive? I’ve done that in the past but I can try again.

*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Michael Jerris *Sent:* Friday, August 15, 2014 3:42 PM *To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] high pdd

depends what its doing, but i suspect you are hitting some I/O wait of some sort.

On Aug 15, 2014, at 3:38 PM, Andre <andretodd@verizon.net> wrote:

What would cause very high pdd 12 seconds + after 3000 ports. Wireshark shows reinvites.

I have tried many profiles and even many console apps both seem to have the same results after 2500 to 3000 ports (only 300 are connected)

The server cpu and memory are ½ the server limits.

Any ideas where to look would be great.

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

Oh the log mod are commented off however I type console log level debug. Only errors that up are after the high pdd that say can't create more sessions or can't create more events for the pop code for the cdrs. I have the config set to 30,000 sessions for testing but the problem happens about 3000 sessions and only 300 connected. Thanks -----Original Message----- From: "Andre Demattia" <andretodd@verizon.net> Sent: ‎8/‎16/‎2014 9:48 AM To: "FreeSWITCH Users Help" <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] high pdd Hmm, I have the cdrs going to a rabbit mq on the same box but I did try disabling the cdrs totally and got the same results. For the SQL db I'd like to turn them off. The only one I need I the limit db however the Sofia reg internal still gets created. But it is in a Ram drive if that helps. The switch is just bypass media and I write my own IP auth so no need to register. Is there a way I can disable all dbs to see if that is the problem? I use no SQL now. Thanks so much for your advice. Andre -----Original Message----- From: "Steven Ayre" <steveayre@gmail.com> Sent: ‎8/‎16/‎2014 9:20 AM To: "FreeSWITCH Users Help" <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] high pdd What about logging levels (debug?) and CDRs? Are they going to disk that might be causing high I/O? Also consider trying something more powerful than SQLite for the database to see if that helps - eg PostgreSQL/MySQL On 16 August 2014 14:03, Andre Demattia <andretodd@verizon.net> wrote:

Any other hints that might help me figure out what could be the issue? Thanks ------------------------------ From: Andre <andretodd@verizon.net> Sent: ‎8/‎15/‎2014 3:52 PM

To: 'FreeSWITCH Users Help' <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] high pdd

I run it now with no sql and the sqllite db in a ram drive. Should I run the entire process in ram drive? I’ve done that in the past but I can try again.

*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Michael Jerris *Sent:* Friday, August 15, 2014 3:42 PM *To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] high pdd

depends what its doing, but i suspect you are hitting some I/O wait of some sort.

On Aug 15, 2014, at 3:38 PM, Andre <andretodd@verizon.net> wrote:

What would cause very high pdd 12 seconds + after 3000 ports. Wireshark shows reinvites.

I have tried many profiles and even many console apps both seem to have the same results after 2500 to 3000 ports (only 300 are connected)

The server cpu and memory are ½ the server limits.

Any ideas where to look would be great.

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

[The entire original message is not included.]

-nodb only turns off the core db, it does not turn off all db's that FreeSWITCH may need... you best bet is to just put the entire db dir in a ram drive Ken Sent from my iPad

On Aug 16, 2014, at 15:23, "Andre" <andretodd@verizon.net> wrote:

Going with the idea it’s IO related I did a db_cache status. I just disabled limit in my code by removing it but limit should be HASH not a database however I stills how a database.

Below is the results of the db cache status. I’m not real sure what its saying. I should have all registration and limit disabled and should not be using any sql database from my understanding.

Anyone know what the results mean? I’ve only tested 2 calls on this status.

db_cache status

db="call_limit",type="core_db" Type: CORE_DB Last used: 196 Total used: 2 Flags: Unlocked, Detached(0) Creator: mod_db.c:98 Last User: mod_db.c:98 db="sofia_reg_external",type="core_db" Type: CORE_DB Last used: 183 Total used: 4 Flags: Locked, Attached(1) Creator: sofia_glue.c:2399 Last User: ..\..\src\switch_core_sqldb.c:2022 db="sofia_reg_internal",type="core_db" Type: CORE_DB Last used: 183 Total used: 4 Flags: Locked, Attached(1) Creator: sofia_glue.c:2399 Last User: ..\..\src\switch_core_sqldb.c:2022 db="sofia_reg_internal",type="core_db" Type: CORE_DB Last used: 3 Total used: 16 Flags: Unlocked, Detached(0) Creator: sofia_glue.c:2174 Last User: sofia_glue.c:2399 db="sofia_reg_external",type="core_db" Type: CORE_DB Last used: 3 Total used: 16 Flags: Unlocked, Detached(0) Creator: sofia_glue.c:2174 Last User: sofia_glue.c:2399 5 total. 2 in use.

From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Andre Demattia Sent: Saturday, August 16, 2014 11:50 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd

Oh the log mod are commented off however I type console log level debug. Only errors that up are after the high pdd that say can't create more sessions or can't create more events for the pop code for the cdrs. I have the config set to 30,000 sessions for testing but the problem happens about 3000 sessions and only 300 connected.

Thanks From: Andre Demattia Sent: ‎8/‎16/‎2014 9:48 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd

Hmm, I have the cdrs going to a rabbit mq on the same box but I did try disabling the cdrs totally and got the same results.

For the SQL db I'd like to turn them off. The only one I need I the limit db however the Sofia reg internal still gets created. But it is in a Ram drive if that helps.

The switch is just bypass media and I write my own IP auth so no need to register.

Is there a way I can disable all dbs to see if that is the problem? I use no SQL now.

Thanks so much for your advice. Andre From: Steven Ayre Sent: ‎8/‎16/‎2014 9:20 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd

What about logging levels (debug?) and CDRs? Are they going to disk that might be causing high I/O?

Also consider trying something more powerful than SQLite for the database to see if that helps - eg PostgreSQL/MySQL

On 16 August 2014 14:03, Andre Demattia <andretodd@verizon.net> wrote: Any other hints that might help me figure out what could be the issue? Thanks From: Andre Sent: ‎8/‎15/‎2014 3:52 PM

To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] high pdd

I run it now with no sql and the sqllite db in a ram drive. Should I run the entire process in ram drive? I’ve done that in the past but I can try again.

From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Michael Jerris Sent: Friday, August 15, 2014 3:42 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd

depends what its doing, but i suspect you are hitting some I/O wait of some sort.

On Aug 15, 2014, at 3:38 PM, Andre <andretodd@verizon.net> wrote:

What would cause very high pdd 12 seconds + after 3000 ports. Wireshark shows reinvites.

I have tried many profiles and even many console apps both seem to have the same results after 2500 to 3000 ports (only 300 are connected) The server cpu and memory are ½ the server limits.

Any ideas where to look would be great.

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

[The entire original message is not included.] _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

Thanks Ken, I'll try the consulting again. Andre -----Original Message----- From: "Ken Rice" <krice@freeswitch.org> Sent: ‎8/‎17/‎2014 1:19 AM To: "FreeSWITCH Users Help" <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] high pdd if the last post does not work contact consulting@freeswitch.org for pro help from one of the developers... thats about as far as the list will take you Ken Sent from my iPad

On Aug 16, 2014, at 15:23, "Andre" <andretodd@verizon.net> wrote:

Going with the idea it’s IO related I did a db_cache status. I just disabled limit in my code by removing it but limit should be HASH not a database however I stills how a database.

Below is the results of the db cache status. I’m not real sure what its saying. I should have all registration and limit disabled and should not be using any sql database from my understanding.

Anyone know what the results mean? I’ve only tested 2 calls on this status.

db_cache status

db="call_limit",type="core_db" Type: CORE_DB Last used: 196 Total used: 2 Flags: Unlocked, Detached(0) Creator: mod_db.c:98 Last User: mod_db.c:98 db="sofia_reg_external",type="core_db" Type: CORE_DB Last used: 183 Total used: 4 Flags: Locked, Attached(1) Creator: sofia_glue.c:2399 Last User: ..\..\src\switch_core_sqldb.c:2022 db="sofia_reg_internal",type="core_db" Type: CORE_DB Last used: 183 Total used: 4 Flags: Locked, Attached(1) Creator: sofia_glue.c:2399 Last User: ..\..\src\switch_core_sqldb.c:2022 db="sofia_reg_internal",type="core_db" Type: CORE_DB Last used: 3 Total used: 16 Flags: Unlocked, Detached(0) Creator: sofia_glue.c:2174 Last User: sofia_glue.c:2399 db="sofia_reg_external",type="core_db" Type: CORE_DB Last used: 3 Total used: 16 Flags: Unlocked, Detached(0) Creator: sofia_glue.c:2174 Last User: sofia_glue.c:2399 5 total. 2 in use.

From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Andre Demattia Sent: Saturday, August 16, 2014 11:50 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd

Oh the log mod are commented off however I type console log level debug. Only errors that up are after the high pdd that say can't create more sessions or can't create more events for the pop code for the cdrs. I have the config set to 30,000 sessions for testing but the problem happens about 3000 sessions and only 300 connected.

Thanks From: Andre Demattia Sent: ‎8/‎16/‎2014 9:48 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd

Hmm, I have the cdrs going to a rabbit mq on the same box but I did try disabling the cdrs totally and got the same results.

For the SQL db I'd like to turn them off. The only one I need I the limit db however the Sofia reg internal still gets created. But it is in a Ram drive if that helps.

The switch is just bypass media and I write my own IP auth so no need to register.

Is there a way I can disable all dbs to see if that is the problem? I use no SQL now.

Thanks so much for your advice. Andre From: Steven Ayre Sent: ‎8/‎16/‎2014 9:20 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd

What about logging levels (debug?) and CDRs? Are they going to disk that might be causing high I/O?

Also consider trying something more powerful than SQLite for the database to see if that helps - eg PostgreSQL/MySQL

On 16 August 2014 14:03, Andre Demattia <andretodd@verizon.net> wrote: Any other hints that might help me figure out what could be the issue? Thanks From: Andre Sent: ‎8/‎15/‎2014 3:52 PM

To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] high pdd

I run it now with no sql and the sqllite db in a ram drive. Should I run the entire process in ram drive? I’ve done that in the past but I can try again.

From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Michael Jerris Sent: Friday, August 15, 2014 3:42 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd

depends what its doing, but i suspect you are hitting some I/O wait of some sort.

On Aug 15, 2014, at 3:38 PM, Andre <andretodd@verizon.net> wrote:

What would cause very high pdd 12 seconds + after 3000 ports. Wireshark shows reinvites.

I have tried many profiles and even many console apps both seem to have the same results after 2500 to 3000 ports (only 300 are connected) The server cpu and memory are ½ the server limits.

Any ideas where to look would be great.

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

[The entire original message is not included.] _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

[The entire original message is not included.]

Yes that's off and I believe all the databases are off too. No authentication or registration is enabled. The sizes of the database are 0 until I call the shutdown command then its about 90k. I have the DBS in Ra drive too. Windows seems pretty good until the 3000 sessions per min mark. Then I just get long pdd and the threads get backed up then no more sessions can be created. Funny thing is I has 10 profiles and the same results. I also tried two consoles with the same results. FreeSWITCH is great. I'd be grateful for help. I've worked on this for 2 years and hate to give up. If any one from consulting@freeswitch.org is willing to help I'd appreciate that. Thanks again, Andre -----Original Message----- From: "Anthony Minessale" <anthony.minessale@gmail.com> Sent: ‎8/‎17/‎2014 10:13 AM To: "Freeswitch-users" <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] high pdd Did you comment out manage-presence ? Also we have no real idea how well FS performs in windows under extreme load. On Aug 17, 2014 5:42 AM, "Andre Demattia" <andretodd@verizon.net> wrote:

Thanks Ken, I'll try the consulting again. Andre ------------------------------ From: Ken Rice <krice@freeswitch.org> Sent: ‎8/‎17/‎2014 1:19 AM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] high pdd

if the last post does not work contact consulting@freeswitch.org for pro help from one of the developers... thats about as far as the list will take you

Ken Sent from my iPad

On Aug 16, 2014, at 15:23, "Andre" <andretodd@verizon.net> wrote:

Going with the idea it’s IO related I did a db_cache status. I just disabled limit in my code by removing it but limit should be HASH not a database however I stills how a database.

Below is the results of the db cache status. I’m not real sure what its saying. I should have all registration and limit disabled and should not be using any sql database from my understanding.

Anyone know what the results mean? I’ve only tested 2 calls on this status.

[The entire original message is not included.]

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

Thanks, I'm using the newest master and the last stable 1.2.23 We are willing to pay to get this fixed. Andre -----Original Message----- From: "Anthony Minessale" <anthony.minessale@gmail.com> Sent: ‎8/‎17/‎2014 12:30 PM To: "Freeswitch-users" <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] high pdd We dont have a lot of tools to troubleshoot windows. You should mention windows in your subject to have a better chance to get the attention of our windows users. You should consider installing debian 7 on the same hardware and perform similar tests to help determine if your issue is o/s specific or not. You also have not indicated the revision of FS. If you have not worked with latest master, you should be also to rule out improvements. On Aug 17, 2014 9:54 AM, "Andre Demattia" <andretodd@verizon.net> wrote:

Yes that's off and I believe all the databases are off too. No authentication or registration is enabled. The sizes of the database are 0 until I call the shutdown command then its about 90k. I have the DBS in Ra drive too.

Windows seems pretty good until the 3000 sessions per min mark. Then I just get long pdd and the threads get backed up then no more sessions can be created.

Funny thing is I has 10 profiles and the same results. I also tried two consoles with the same results.

FreeSWITCH is great.

I'd be grateful for help. I've worked on this for 2 years and hate to give up.

If any one from consulting@freeswitch.org is willing to help I'd appreciate that.

Thanks again, Andre ------------------------------ From: Anthony Minessale <anthony.minessale@gmail.com> Sent: ‎8/‎17/‎2014 10:13 AM To: Freeswitch-users <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] high pdd

Did you comment out manage-presence ? Also we have no real idea how well FS performs in windows under extreme load. On Aug 17, 2014 5:42 AM, "Andre Demattia" <andretodd@verizon.net> wrote:

...

Thanks Ken, I'll try the consulting again. Andre ------------------------------ From: Ken Rice <krice@freeswitch.org> Sent: ‎8/‎17/‎2014 1:19 AM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] high pdd

if the last post does not work contact consulting@freeswitch.org for pro help from one of the developers... thats about as far as the list will take you

Ken Sent from my iPad

On Aug 16, 2014, at 15:23, "Andre" <andretodd@verizon.net> wrote:

Going with the idea it’s IO related I did a db_cache status. I just disabled limit in my code by removing it but limit should be HASH not a database however I stills how a database.

Below is the results of the db cache status. I’m not real sure what its saying. I should have all registration and limit disabled and should not be using any sql database from my understanding.

Anyone know what the results mean? I’ve only tested 2 calls on this status.

[The entire original message is not included.]

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

[The entire original message is not included.]

1.2.X is EOL, 1.4.7 is the current release. We have releases and tags in git that no longer carry the 'stable' moniker starting with 1.4. On Sun, Aug 17, 2014 at 1:39 PM, Andre Demattia <andretodd@verizon.net> wrote:

Thanks, I'm using the newest master and the last stable 1.2.23 We are willing to pay to get this fixed. Andre ------------------------------ From: Anthony Minessale <anthony.minessale@gmail.com> Sent: ‎8/‎17/‎2014 12:30 PM

To: Freeswitch-users <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] high pdd

We dont have a lot of tools to troubleshoot windows. You should mention windows in your subject to have a better chance to get the attention of our windows users.

You should consider installing debian 7 on the same hardware and perform similar tests to help determine if your issue is o/s specific or not.

You also have not indicated the revision of FS. If you have not worked with latest master, you should be also to rule out improvements. On Aug 17, 2014 9:54 AM, "Andre Demattia" <andretodd@verizon.net> wrote:

...

Yes that's off and I believe all the databases are off too. No authentication or registration is enabled. The sizes of the database are 0 until I call the shutdown command then its about 90k. I have the DBS in Ra drive too.

Windows seems pretty good until the 3000 sessions per min mark. Then I just get long pdd and the threads get backed up then no more sessions can be created.

Funny thing is I has 10 profiles and the same results. I also tried two consoles with the same results.

FreeSWITCH is great.

I'd be grateful for help. I've worked on this for 2 years and hate to give up.

If any one from consulting@freeswitch.org is willing to help I'd appreciate that.

Thanks again, Andre ------------------------------ From: Anthony Minessale <anthony.minessale@gmail.com> Sent: ‎8/‎17/‎2014 10:13 AM To: Freeswitch-users <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] high pdd

Did you comment out manage-presence ? Also we have no real idea how well FS performs in windows under extreme load. On Aug 17, 2014 5:42 AM, "Andre Demattia" <andretodd@verizon.net> wrote:

...

Thanks Ken, I'll try the consulting again. Andre ------------------------------ From: Ken Rice <krice@freeswitch.org> Sent: ‎8/‎17/‎2014 1:19 AM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] high pdd

if the last post does not work contact consulting@freeswitch.org for pro help from one of the developers... thats about as far as the list will take you

Ken Sent from my iPad

On Aug 16, 2014, at 15:23, "Andre" <andretodd@verizon.net> wrote:

Going with the idea it’s IO related I did a db_cache status. I just disabled limit in my code by removing it but limit should be HASH not a database however I stills how a database.

Below is the results of the db cache status. I’m not real sure what its saying. I should have all registration and limit disabled and should not be using any sql database from my understanding.

Anyone know what the results mean? I’ve only tested 2 calls on this status.

[The entire original message is not included.]

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users

[The entire original message is not included.]

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- *Brian West* brian@freeswitch.org *Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com *T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest

Try the same thing on Debian 7 if possible. On Wed, Aug 20, 2014 at 12:35 PM, Andre <andretodd@verizon.net> wrote:

Hi, I’m using the master. Should I try 1.4.7?

I’ve done a dump after things started to go bad/ backed up.

I see 8193 threads in _*thread_cond*_timedwait. Looks like its on line 81 of thread_cond.c res = WaitForSingleObject(cond->semaphore, timeout_ms);

The value of timeout_ms is 2016850392.

Not sure if this means anything.

Thanks for the support guys.

Andre

*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Brian West *Sent:* Wednesday, August 20, 2014 11:57 AM

*To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] high pdd

1.2.X is EOL,

1.4.7 is the current release.

We have releases and tags in git that no longer carry the 'stable' moniker starting with 1.4.

On Sun, Aug 17, 2014 at 1:39 PM, Andre Demattia <andretodd@verizon.net> wrote:

Thanks, I'm using the newest master and the last stable 1.2.23 We are willing to pay to get this fixed. Andre ------------------------------

*From: *Anthony Minessale <anthony.minessale@gmail.com> *Sent: *‎8/‎17/‎2014 12:30 PM

*To: *Freeswitch-users <freeswitch-users@lists.freeswitch.org> *Subject: *Re: [Freeswitch-users] high pdd

We dont have a lot of tools to troubleshoot windows. You should mention windows in your subject to have a better chance to get the attention of our windows users.

You should consider installing debian 7 on the same hardware and perform similar tests to help determine if your issue is o/s specific or not.

You also have not indicated the revision of FS. If you have not worked with latest master, you should be also to rule out improvements.

On Aug 17, 2014 9:54 AM, "Andre Demattia" <andretodd@verizon.net> wrote:

Yes that's off and I believe all the databases are off too. No authentication or registration is enabled. The sizes of the database are 0 until I call the shutdown command then its about 90k. I have the DBS in Ra drive too.

Windows seems pretty good until the 3000 sessions per min mark. Then I just get long pdd and the threads get backed up then no more sessions can be created.

Funny thing is I has 10 profiles and the same results. I also tried two consoles with the same results.

FreeSWITCH is great.

I'd be grateful for help. I've worked on this for 2 years and hate to give up.

If any one from consulting@freeswitch.org is willing to help I'd appreciate that.

Thanks again, Andre ------------------------------

*From: *Anthony Minessale <anthony.minessale@gmail.com> *Sent: *‎8/‎17/‎2014 10:13 AM *To: *Freeswitch-users <freeswitch-users@lists.freeswitch.org> *Subject: *Re: [Freeswitch-users] high pdd

Did you comment out manage-presence ? Also we have no real idea how well FS performs in windows under extreme load.

On Aug 17, 2014 5:42 AM, "Andre Demattia" <andretodd@verizon.net> wrote:

Thanks Ken, I'll try the consulting again. Andre ------------------------------

*From: *Ken Rice <krice@freeswitch.org> *Sent: *‎8/‎17/‎2014 1:19 AM *To: *FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject: *Re: [Freeswitch-users] high pdd

if the last post does not work contact consulting@freeswitch.org for pro help from one of the developers... thats about as far as the list will take you

Ken

Sent from my iPad

On Aug 16, 2014, at 15:23, "Andre" <andretodd@verizon.net> wrote:

Going with the idea it’s IO related I did a db_cache status. I just disabled limit in my code by removing it but limit should be HASH not a database however I stills how a database.

Below is the results of the db cache status. I’m not real sure what its saying. I should have all registration and limit disabled and should not be using any sql database from my understanding.

Anyone know what the results mean? I’ve only tested 2 calls on this status.

[The entire original message is not included.]

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users

[The entire original message is not included.]

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

--

*Brian West* brian@freeswitch.org

*Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com

*T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- *Brian West* brian@freeswitch.org *Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com *T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest

It seems it might be stuck in the condition code for Windows. Try this timing patch for Windows, and see if it makes any difference. I've tested in on lower load, but it would be interesting how it performs in this scenario. https://jira.freeswitch.org/secure/attachment/20613/windows-waitable-timer-d... 2014-08-20 21:13 GMT+02:00 Andre <andretodd@verizon.net>:

I will, need to get hardware and figure out how to use mono.

In the meantime, I’ve attached some of the call traces.

*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Brian West *Sent:* Wednesday, August 20, 2014 3:05 PM *To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] high pdd

Try the same thing on Debian 7 if possible.

On Wed, Aug 20, 2014 at 12:35 PM, Andre <andretodd@verizon.net> wrote:

Hi, I’m using the master. Should I try 1.4.7?

I’ve done a dump after things started to go bad/ backed up.

I see 8193 threads in _*thread_cond*_timedwait. Looks like its on line 81 of thread_cond.c res = WaitForSingleObject(cond->semaphore, timeout_ms);

The value of timeout_ms is 2016850392.

Not sure if this means anything.

Thanks for the support guys.

Andre

*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Brian West *Sent:* Wednesday, August 20, 2014 11:57 AM

*To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] high pdd

1.2.X is EOL,

1.4.7 is the current release.

We have releases and tags in git that no longer carry the 'stable' moniker starting with 1.4.

On Sun, Aug 17, 2014 at 1:39 PM, Andre Demattia <andretodd@verizon.net> wrote:

Thanks, I'm using the newest master and the last stable 1.2.23 We are willing to pay to get this fixed. Andre ------------------------------

*From: *Anthony Minessale <anthony.minessale@gmail.com> *Sent: *‎8/‎17/‎2014 12:30 PM

*To: *Freeswitch-users <freeswitch-users@lists.freeswitch.org> *Subject: *Re: [Freeswitch-users] high pdd

We dont have a lot of tools to troubleshoot windows. You should mention windows in your subject to have a better chance to get the attention of our windows users.

You should consider installing debian 7 on the same hardware and perform similar tests to help determine if your issue is o/s specific or not.

You also have not indicated the revision of FS. If you have not worked with latest master, you should be also to rule out improvements.

On Aug 17, 2014 9:54 AM, "Andre Demattia" <andretodd@verizon.net> wrote:

Yes that's off and I believe all the databases are off too. No authentication or registration is enabled. The sizes of the database are 0 until I call the shutdown command then its about 90k. I have the DBS in Ra drive too.

Windows seems pretty good until the 3000 sessions per min mark. Then I just get long pdd and the threads get backed up then no more sessions can be created.

Funny thing is I has 10 profiles and the same results. I also tried two consoles with the same results.

FreeSWITCH is great.

I'd be grateful for help. I've worked on this for 2 years and hate to give up.

If any one from consulting@freeswitch.org is willing to help I'd appreciate that.

Thanks again, Andre ------------------------------

*From: *Anthony Minessale <anthony.minessale@gmail.com> *Sent: *‎8/‎17/‎2014 10:13 AM *To: *Freeswitch-users <freeswitch-users@lists.freeswitch.org> *Subject: *Re: [Freeswitch-users] high pdd

Did you comment out manage-presence ? Also we have no real idea how well FS performs in windows under extreme load.

On Aug 17, 2014 5:42 AM, "Andre Demattia" <andretodd@verizon.net> wrote:

Thanks Ken, I'll try the consulting again. Andre ------------------------------

*From: *Ken Rice <krice@freeswitch.org> *Sent: *‎8/‎17/‎2014 1:19 AM *To: *FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject: *Re: [Freeswitch-users] high pdd

if the last post does not work contact consulting@freeswitch.org for pro help from one of the developers... thats about as far as the list will take you

Ken

Sent from my iPad

On Aug 16, 2014, at 15:23, "Andre" <andretodd@verizon.net> wrote:

Going with the idea it’s IO related I did a db_cache status. I just disabled limit in my code by removing it but limit should be HASH not a database however I stills how a database.

Below is the results of the db cache status. I’m not real sure what its saying. I should have all registration and limit disabled and should not be using any sql database from my understanding.

Anyone know what the results mean? I’ve only tested 2 calls on this status.

[The entire original message is not included.]

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users

[The entire original message is not included.]

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

--

*Brian West* brian@freeswitch.org

*Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com

*T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

--

*Brian West* brian@freeswitch.org

*Twitter: @FreeSWITCH , @briankwest* http://www.freeswitchbook.com http://www.freeswitchcookbook.com

*T:*+19184209001 | *F:*+19184209002 | *M:*+1918424WEST (9378) *iNUM:*+883 5100 1420 9001 | *ISN:*410*543 | *Skype:*briankwest

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

I added the new code but it doesn’t seem to fix it. :/ From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Andre Sent: Wednesday, August 20, 2014 3:37 PM To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] high pdd Thanks I’ll work on it right a way. From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Peter Olsson Sent: Wednesday, August 20, 2014 3:30 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd It seems it might be stuck in the condition code for Windows. Try this timing patch for Windows, and see if it makes any difference. I've tested in on lower load, but it would be interesting how it performs in this scenario. https://jira.freeswitch.org/secure/attachment/20613/windows-waitable-timer-d... 2014-08-20 21:13 GMT+02:00 Andre <andretodd@verizon.net <mailto:andretodd@verizon.net> >: I will, need to get hardware and figure out how to use mono. In the meantime, I’ve attached some of the call traces. From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> ] On Behalf Of Brian West Sent: Wednesday, August 20, 2014 3:05 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd Try the same thing on Debian 7 if possible. On Wed, Aug 20, 2014 at 12:35 PM, Andre <andretodd@verizon.net <mailto:andretodd@verizon.net> > wrote: Hi, I’m using the master. Should I try 1.4.7? I’ve done a dump after things started to go bad/ backed up. I see 8193 threads in _thread_cond_timedwait. Looks like its on line 81 of thread_cond.c res = WaitForSingleObject(cond->semaphore, timeout_ms); The value of timeout_ms is 2016850392 <tel:2016850392> . Not sure if this means anything. Thanks for the support guys. Andre From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> ] On Behalf Of Brian West Sent: Wednesday, August 20, 2014 11:57 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd 1.2.X is EOL, 1.4.7 is the current release. We have releases and tags in git that no longer carry the 'stable' moniker starting with 1.4. On Sun, Aug 17, 2014 at 1:39 PM, Andre Demattia <andretodd@verizon.net <mailto:andretodd@verizon.net> > wrote: Thanks, I'm using the newest master and the last stable 1.2.23 We are willing to pay to get this fixed. Andre _____ From: Anthony Minessale <mailto:anthony.minessale@gmail.com> Sent: ‎8/‎17/‎2014 12:30 PM To: Freeswitch-users <mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] high pdd We dont have a lot of tools to troubleshoot windows. You should mention windows in your subject to have a better chance to get the attention of our windows users. You should consider installing debian 7 on the same hardware and perform similar tests to help determine if your issue is o/s specific or not. You also have not indicated the revision of FS. If you have not worked with latest master, you should be also to rule out improvements. On Aug 17, 2014 9:54 AM, "Andre Demattia" <andretodd@verizon.net <mailto:andretodd@verizon.net> > wrote: Yes that's off and I believe all the databases are off too. No authentication or registration is enabled. The sizes of the database are 0 until I call the shutdown command then its about 90k. I have the DBS in Ra drive too. Windows seems pretty good until the 3000 sessions per min mark. Then I just get long pdd and the threads get backed up then no more sessions can be created. Funny thing is I has 10 profiles and the same results. I also tried two consoles with the same results. FreeSWITCH is great. I'd be grateful for help. I've worked on this for 2 years and hate to give up. If any one from consulting@freeswitch.org <mailto:consulting@freeswitch.org> is willing to help I'd appreciate that. Thanks again, Andre _____ From: Anthony Minessale <mailto:anthony.minessale@gmail.com> Sent: ‎8/‎17/‎2014 10:13 AM To: Freeswitch-users <mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] high pdd Did you comment out manage-presence ? Also we have no real idea how well FS performs in windows under extreme load. On Aug 17, 2014 5:42 AM, "Andre Demattia" <andretodd@verizon.net <mailto:andretodd@verizon.net> > wrote: Thanks Ken, I'll try the consulting again. Andre _____ From: Ken Rice <mailto:krice@freeswitch.org> Sent: ‎8/‎17/‎2014 1:19 AM To: FreeSWITCH Users Help <mailto:freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] high pdd if the last post does not work contact consulting@freeswitch.org <mailto:consulting@freeswitch.org> for pro help from one of the developers... thats about as far as the list will take you Ken Sent from my iPad On Aug 16, 2014, at 15:23, "Andre" <andretodd@verizon.net <mailto:andretodd@verizon.net> > wrote: Going with the idea it’s IO related I did a db_cache status. I just disabled limit in my code by removing it but limit should be HASH not a database however I stills how a database. Below is the results of the db cache status. I’m not real sure what its saying. I should have all registration and limit disabled and should not be using any sql database from my understanding. Anyone know what the results mean? I’ve only tested 2 calls on this status. [The entire original message is not included.] _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users [The entire original message is not included.] _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Brian West brian@freeswitch.org <mailto:brian@freeswitch.org> <http://bkw.org/whmcslogo.png> Twitter: @FreeSWITCH , @briankwest http://www.freeswitchbook.com http://www.freeswitchcookbook.com T:+19184209001 <tel:%2B19184209001> | F:+19184209002 <tel:%2B19184209002> | M:+1918424WEST (9378) iNUM:+883 5100 1420 9001 | ISN:410*543 | Skype:briankwest _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Brian West brian@freeswitch.org <mailto:brian@freeswitch.org> <http://bkw.org/whmcslogo.png> Twitter: @FreeSWITCH , @briankwest http://www.freeswitchbook.com http://www.freeswitchcookbook.com T:+19184209001 <tel:%2B19184209001> | F:+19184209002 <tel:%2B19184209002> | M:+1918424WEST (9378) iNUM:+883 5100 1420 9001 | ISN:410*543 | Skype:briankwest _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

Ok I’ll try that next. I’ll create xml dialplan to see if that makes a difference. One note, when I didn’t load the deck in my managed code my dial plan did return a 503 in 0 – 20 milliseconds. Not sure if that matters since no bridge was ever called. Thanks again, Andre From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Michael Jerris Sent: Wednesday, August 20, 2014 5:02 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd I would test this without any of your managed code at all, to confirm that it is not the bottleneck. On Aug 20, 2014, at 4:28 PM, Andre <andretodd@verizon.net <mailto:andretodd@verizon.net> > wrote: I added the new code but it doesn’t seem to fix it. :/ From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Andre Sent: Wednesday, August 20, 2014 3:37 PM To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] high pdd Thanks I’ll work on it right a way. From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [ <mailto:freeswitch-users-bounces@lists.freeswitch.org> mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Peter Olsson Sent: Wednesday, August 20, 2014 3:30 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd It seems it might be stuck in the condition code for Windows. Try this timing patch for Windows, and see if it makes any difference. I've tested in on lower load, but it would be interesting how it performs in this scenario. <https://jira.freeswitch.org/secure/attachment/20613/windows-waitable-timer-diff.txt> https://jira.freeswitch.org/secure/attachment/20613/windows-waitable-timer-d... 2014-08-20 21:13 GMT+02:00 Andre < <mailto:andretodd@verizon.net> andretodd@verizon.net>: I will, need to get hardware and figure out how to use mono. In the meantime, I’ve attached some of the call traces. <image001.jpg><image002.jpg> <image003.jpg> From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [mailto: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfBrian West Sent: Wednesday, August 20, 2014 3:05 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd Try the same thing on Debian 7 if possible. On Wed, Aug 20, 2014 at 12:35 PM, Andre < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Hi, I’m using the master. Should I try 1.4.7? I’ve done a dump after things started to go bad/ backed up. I see 8193 threads in _thread_cond_timedwait. Looks like its on line 81 of thread_cond.c res = WaitForSingleObject(cond->semaphore, timeout_ms); The value of timeout_ms is <tel:2016850392> 2016850392. Not sure if this means anything. Thanks for the support guys. Andre From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [mailto: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfBrian West Sent: Wednesday, August 20, 2014 11:57 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd 1.2.X is EOL, 1.4.7 is the current release. We have releases and tags in git that no longer carry the 'stable' moniker starting with 1.4. On Sun, Aug 17, 2014 at 1:39 PM, Andre Demattia < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Thanks, I'm using the newest master and the last stable 1.2.23 We are willing to pay to get this fixed. Andre _____ From: <mailto:anthony.minessale@gmail.com> Anthony Minessale Sent: ‎8/‎17/‎2014 12:30 PM To: <mailto:freeswitch-users@lists.freeswitch.org> Freeswitch-users Subject: Re: [Freeswitch-users] high pdd We dont have a lot of tools to troubleshoot windows. You should mention windows in your subject to have a better chance to get the attention of our windows users. You should consider installing debian 7 on the same hardware and perform similar tests to help determine if your issue is o/s specific or not. You also have not indicated the revision of FS. If you have not worked with latest master, you should be also to rule out improvements. On Aug 17, 2014 9:54 AM, "Andre Demattia" < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Yes that's off and I believe all the databases are off too. No authentication or registration is enabled. The sizes of the database are 0 until I call the shutdown command then its about 90k. I have the DBS in Ra drive too. Windows seems pretty good until the 3000 sessions per min mark. Then I just get long pdd and the threads get backed up then no more sessions can be created. Funny thing is I has 10 profiles and the same results. I also tried two consoles with the same results. FreeSWITCH is great. I'd be grateful for help. I've worked on this for 2 years and hate to give up. If any one from <mailto:consulting@freeswitch.org> consulting@freeswitch.org is willing to help I'd appreciate that. Thanks again, Andre _____ From: <mailto:anthony.minessale@gmail.com> Anthony Minessale Sent: ‎8/‎17/‎2014 10:13 AM To: <mailto:freeswitch-users@lists.freeswitch.org> Freeswitch-users Subject: Re: [Freeswitch-users] high pdd Did you comment out manage-presence ? Also we have no real idea how well FS performs in windows under extreme load. On Aug 17, 2014 5:42 AM, "Andre Demattia" < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Thanks Ken, I'll try the consulting again. Andre _____ From: <mailto:krice@freeswitch.org> Ken Rice Sent: ‎8/‎17/‎2014 1:19 AM To: <mailto:freeswitch-users@lists.freeswitch.org> FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd if the last post does not work contact <mailto:consulting@freeswitch.org> consulting@freeswitch.org for pro help from one of the developers... thats about as far as the list will take you Ken Sent from my iPad On Aug 16, 2014, at 15:23, "Andre" < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Going with the idea it’s IO related I did a db_cache status. I just disabled limit in my code by removing it but limit should be HASH not a database however I stills how a database. Below is the results of the db cache status. I’m not real sure what its saying. I should have all registration and limit disabled and should not be using any sql database from my understanding. Anyone know what the results mean? I’ve only tested 2 calls on this status. [The entire original message is not included.] _________________________________________________________________________ Professional FreeSWITCH Consulting Services: <mailto:consulting@freeswitch.org> consulting@freeswitch.org <http://www.freeswitchsolutions.com/> http://www.freeswitchsolutions.com Official FreeSWITCH Sites <http://www.freeswitch.org/> http://www.freeswitch.org <http://confluence.freeswitch.org/> http://confluence.freeswitch.org <http://www.cluecon.com/> http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server <http://www.cudatel.com/> http://www.cudatel.com FreeSWITCH-users mailing list <mailto:FreeSWITCH-users@lists.freeswitch.org> FreeSWITCH-users@lists.freeswitch.org <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://lists.freeswitch.org/mailman/options/freeswitch-users [The entire original message is not included.] _________________________________________________________________________ Professional FreeSWITCH Consulting Services: <mailto:consulting@freeswitch.org> consulting@freeswitch.org <http://www.freeswitchsolutions.com/> http://www.freeswitchsolutions.com Official FreeSWITCH Sites <http://www.freeswitch.org/> http://www.freeswitch.org <http://confluence.freeswitch.org/> http://confluence.freeswitch.org <http://www.cluecon.com/> http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server <http://www.cudatel.com/> http://www.cudatel.com FreeSWITCH-users mailing list <mailto:FreeSWITCH-users@lists.freeswitch.org> FreeSWITCH-users@lists.freeswitch.org <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://lists.freeswitch.org/mailman/options/freeswitch-users <http://www.freeswitch.org/> http://www.freeswitch.org -- Brian West <mailto:brian@freeswitch.org> brian@freeswitch.org <http://bkw.org/whmcslogo.png> Twitter: @FreeSWITCH , @briankwest <http://www.freeswitchbook.com/> http://www.freeswitchbook.com <http://www.freeswitchcookbook.com/> http://www.freeswitchcookbook.com T: <tel:%2B19184209001> +19184209001 | F: <tel:%2B19184209002> +19184209002 | M:+1918424WEST (9378) iNUM:+883 5100 1420 9001 | ISN:410*543 | Skype:briankwest _________________________________________________________________________ Professional FreeSWITCH Consulting Services: <mailto:consulting@freeswitch.org> consulting@freeswitch.org <http://www.freeswitchsolutions.com/> http://www.freeswitchsolutions.com Official FreeSWITCH Sites <http://www.freeswitch.org/> http://www.freeswitch.org <http://confluence.freeswitch.org/> http://confluence.freeswitch.org <http://www.cluecon.com/> http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server <http://www.cudatel.com/> http://www.cudatel.com FreeSWITCH-users mailing list <mailto:FreeSWITCH-users@lists.freeswitch.org> FreeSWITCH-users@lists.freeswitch.org <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://lists.freeswitch.org/mailman/options/freeswitch-users <http://www.freeswitch.org/> http://www.freeswitch.org -- Brian West <mailto:brian@freeswitch.org> brian@freeswitch.org <http://bkw.org/whmcslogo.png> Twitter: @FreeSWITCH , @briankwest <http://www.freeswitchbook.com/> http://www.freeswitchbook.com <http://www.freeswitchcookbook.com/> http://www.freeswitchcookbook.com T: <tel:%2B19184209001> +19184209001 | F: <tel:%2B19184209002> +19184209002 | M:+1918424WEST (9378) iNUM:+883 5100 1420 9001 | ISN:410*543 | Skype:briankwest _________________________________________________________________________ Professional FreeSWITCH Consulting Services: <mailto:consulting@freeswitch.org> consulting@freeswitch.org <http://www.freeswitchsolutions.com/> http://www.freeswitchsolutions.com Official FreeSWITCH Sites <http://www.freeswitch.org/> http://www.freeswitch.org <http://confluence.freeswitch.org/> http://confluence.freeswitch.org <http://www.cluecon.com/> http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server <http://www.cudatel.com/> http://www.cudatel.com FreeSWITCH-users mailing list <mailto:FreeSWITCH-users@lists.freeswitch.org> FreeSWITCH-users@lists.freeswitch.org <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://lists.freeswitch.org/mailman/options/freeswitch-users <http://www.freeswitch.org/> http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

Thanks Anthony, I can provide the Dump for you it’s 8gb unzipped so I’ll compress it. The tool I use is Visual studio but I’m sure there are other tools too. I can run a the profiler tool in visual studio to find the CPU. Also how do I get the modified copy of the APR? Thanks again guys. Andre From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: Wednesday, August 20, 2014 5:26 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd The cond waits are all of your session threads in bypass media mode asleep in the state machine. Its hard to tell with that blurry screen shot what else is going on. Is there a way to produce that data in a way we could browse it? You should be looking for threads in the sofia sip stack and the ones doing originate if you are worried about pdd. Also which threads are costing the most CPU. If conditionals are implemented in windows as waitforsingleobject i have no idea if that's as efficient as it is in linux. You could try forklifting in our modified copy of APR that has been updated to work with their latest release and see if anything has improved in the windows implementations of conditionals. I recall Jeff had to backport in at least one patch for that over the years. On Wed, Aug 20, 2014 at 4:11 PM, Andre <andretodd@verizon.net <mailto:andretodd@verizon.net> > wrote: Ok I’ll try that next. I’ll create xml dialplan to see if that makes a difference. One note, when I didn’t load the deck in my managed code my dial plan did return a 503 in 0 – 20 milliseconds. Not sure if that matters since no bridge was ever called. Thanks again, Andre From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> ] On Behalf Of Michael Jerris Sent: Wednesday, August 20, 2014 5:02 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd I would test this without any of your managed code at all, to confirm that it is not the bottleneck. On Aug 20, 2014, at 4:28 PM, Andre <andretodd@verizon.net <mailto:andretodd@verizon.net> > wrote: I added the new code but it doesn’t seem to fix it. :/ From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Andre Sent: Wednesday, August 20, 2014 3:37 PM To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] high pdd Thanks I’ll work on it right a way. From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [ <mailto:freeswitch-users-bounces@lists.freeswitch.org> mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Peter Olsson Sent: Wednesday, August 20, 2014 3:30 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd It seems it might be stuck in the condition code for Windows. Try this timing patch for Windows, and see if it makes any difference. I've tested in on lower load, but it would be interesting how it performs in this scenario. <https://jira.freeswitch.org/secure/attachment/20613/windows-waitable-timer-diff.txt> https://jira.freeswitch.org/secure/attachment/20613/windows-waitable-timer-d... 2014-08-20 21:13 GMT+02:00 Andre < <mailto:andretodd@verizon.net> andretodd@verizon.net>: I will, need to get hardware and figure out how to use mono. In the meantime, I’ve attached some of the call traces. <image001.jpg><image002.jpg> <image003.jpg> From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [mailto: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfBrian West Sent: Wednesday, August 20, 2014 3:05 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd Try the same thing on Debian 7 if possible. On Wed, Aug 20, 2014 at 12:35 PM, Andre < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Hi, I’m using the master. Should I try 1.4.7? I’ve done a dump after things started to go bad/ backed up. I see 8193 threads in _thread_cond_timedwait. Looks like its on line 81 of thread_cond.c res = WaitForSingleObject(cond->semaphore, timeout_ms); The value of timeout_ms is <tel:2016850392> 2016850392. Not sure if this means anything. Thanks for the support guys. Andre From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [mailto: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfBrian West Sent: Wednesday, August 20, 2014 11:57 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd 1.2.X is EOL, 1.4.7 is the current release. We have releases and tags in git that no longer carry the 'stable' moniker starting with 1.4. On Sun, Aug 17, 2014 at 1:39 PM, Andre Demattia < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Thanks, I'm using the newest master and the last stable 1.2.23 We are willing to pay to get this fixed. Andre _____ From: <mailto:anthony.minessale@gmail.com> Anthony Minessale Sent: ‎8/‎17/‎2014 12:30 PM To: <mailto:freeswitch-users@lists.freeswitch.org> Freeswitch-users Subject: Re: [Freeswitch-users] high pdd We dont have a lot of tools to troubleshoot windows. You should mention windows in your subject to have a better chance to get the attention of our windows users. You should consider installing debian 7 on the same hardware and perform similar tests to help determine if your issue is o/s specific or not. You also have not indicated the revision of FS. If you have not worked with latest master, you should be also to rule out improvements. On Aug 17, 2014 9:54 AM, "Andre Demattia" < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Yes that's off and I believe all the databases are off too. No authentication or registration is enabled. The sizes of the database are 0 until I call the shutdown command then its about 90k. I have the DBS in Ra drive too. Windows seems pretty good until the 3000 sessions per min mark. Then I just get long pdd and the threads get backed up then no more sessions can be created. Funny thing is I has 10 profiles and the same results. I also tried two consoles with the same results. FreeSWITCH is great. I'd be grateful for help. I've worked on this for 2 years and hate to give up. If any one from <mailto:consulting@freeswitch.org> consulting@freeswitch.org is willing to help I'd appreciate that. Thanks again, Andre _____ From: <mailto:anthony.minessale@gmail.com> Anthony Minessale Sent: ‎8/‎17/‎2014 10:13 AM To: <mailto:freeswitch-users@lists.freeswitch.org> Freeswitch-users Subject: Re: [Freeswitch-users] high pdd Did you comment out manage-presence ? Also we have no real idea how well FS performs in windows under extreme load. On Aug 17, 2014 5:42 AM, "Andre Demattia" < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Thanks Ken, I'll try the consulting again. Andre _____ From: <mailto:krice@freeswitch.org> Ken Rice Sent: ‎8/‎17/‎2014 1:19 AM To: <mailto:freeswitch-users@lists.freeswitch.org> FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd if the last post does not work contact <mailto:consulting@freeswitch.org> consulting@freeswitch.org for pro help from one of the developers... thats about as far as the list will take you Ken Sent from my iPad On Aug 16, 2014, at 15:23, "Andre" < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Going with the idea it’s IO related I did a db_cache status. I just disabled limit in my code by removing it but limit should be HASH not a database however I stills how a database. Below is the results of the db cache status. I’m not real sure what its saying. I should have all registration and limit disabled and should not be using any sql database from my understanding. Anyone know what the results mean? I’ve only tested 2 calls on this status. [The entire original message is not included.] _________________________________________________________________________ Professional FreeSWITCH Consulting Services: <mailto:consulting@freeswitch.org> consulting@freeswitch.org <http://www.freeswitchsolutions.com/> http://www.freeswitchsolutions.com Official FreeSWITCH Sites <http://www.freeswitch.org/> http://www.freeswitch.org <http://confluence.freeswitch.org/> http://confluence.freeswitch.org <http://www.cluecon.com/> http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server <http://www.cudatel.com/> http://www.cudatel.com FreeSWITCH-users mailing list <mailto:FreeSWITCH-users@lists.freeswitch.org> FreeSWITCH-users@lists.freeswitch.org <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://lists.freeswitch.org/mailman/options/freeswitch-users [The entire original message is not included.] _________________________________________________________________________ Professional FreeSWITCH Consulting Services: <mailto:consulting@freeswitch.org> consulting@freeswitch.org <http://www.freeswitchsolutions.com/> http://www.freeswitchsolutions.com Official FreeSWITCH Sites <http://www.freeswitch.org/> http://www.freeswitch.org <http://confluence.freeswitch.org/> http://confluence.freeswitch.org <http://www.cluecon.com/> http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server <http://www.cudatel.com/> http://www.cudatel.com FreeSWITCH-users mailing list <mailto:FreeSWITCH-users@lists.freeswitch.org> FreeSWITCH-users@lists.freeswitch.org <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://lists.freeswitch.org/mailman/options/freeswitch-users <http://www.freeswitch.org/> http://www.freeswitch.org -- Brian West <mailto:brian@freeswitch.org> brian@freeswitch.org <http://bkw.org/whmcslogo.png> Twitter: @FreeSWITCH , @briankwest <http://www.freeswitchbook.com/> http://www.freeswitchbook.com <http://www.freeswitchcookbook.com/> http://www.freeswitchcookbook.com T: <tel:%2B19184209001> +19184209001 | F: <tel:%2B19184209002> +19184209002 | M:+1918424WEST (9378) iNUM:+883 5100 1420 9001 | ISN:410*543 | Skype:briankwest _________________________________________________________________________ Professional FreeSWITCH Consulting Services: <mailto:consulting@freeswitch.org> consulting@freeswitch.org <http://www.freeswitchsolutions.com/> http://www.freeswitchsolutions.com Official FreeSWITCH Sites <http://www.freeswitch.org/> http://www.freeswitch.org <http://confluence.freeswitch.org/> http://confluence.freeswitch.org <http://www.cluecon.com/> http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server <http://www.cudatel.com/> http://www.cudatel.com FreeSWITCH-users mailing list <mailto:FreeSWITCH-users@lists.freeswitch.org> FreeSWITCH-users@lists.freeswitch.org <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://lists.freeswitch.org/mailman/options/freeswitch-users <http://www.freeswitch.org/> http://www.freeswitch.org -- Brian West <mailto:brian@freeswitch.org> brian@freeswitch.org <http://bkw.org/whmcslogo.png> Twitter: @FreeSWITCH , @briankwest <http://www.freeswitchbook.com/> http://www.freeswitchbook.com <http://www.freeswitchcookbook.com/> http://www.freeswitchcookbook.com T: <tel:%2B19184209001> +19184209001 | F: <tel:%2B19184209002> +19184209002 | M:+1918424WEST (9378) iNUM:+883 5100 1420 9001 | ISN:410*543 | Skype:briankwest _________________________________________________________________________ Professional FreeSWITCH Consulting Services: <mailto:consulting@freeswitch.org> consulting@freeswitch.org <http://www.freeswitchsolutions.com/> http://www.freeswitchsolutions.com Official FreeSWITCH Sites <http://www.freeswitch.org/> http://www.freeswitch.org <http://confluence.freeswitch.org/> http://confluence.freeswitch.org <http://www.cluecon.com/> http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server <http://www.cudatel.com/> http://www.cudatel.com FreeSWITCH-users mailing list <mailto:FreeSWITCH-users@lists.freeswitch.org> FreeSWITCH-users@lists.freeswitch.org <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://lists.freeswitch.org/mailman/options/freeswitch-users <http://www.freeswitch.org/> http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net <http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900

Hi, I know Andre is trying on UNIX but wanted to share what I found.. I too have the same issue. But I am using Master x64 embedded. After 2600 threads the program just keeps adding a ton threads (very fast) and wireshark shows Lots of re invites. My magic number is 1300 ports(bridge). CPS does not seem to matter but its more ports/Threads. Not sure but when doing a dump I see EnterCriticalSection(&cond->csection); is being pointed to in thread_cond.c static APR_INLINE apr_status_t _thread_cond_timedwait(apr_thread_cond_t *cond, apr_thread_mutex_t *mutex, DWORD timeout_ms ) { DWORD res; apr_status_t rv; unsigned int wake = 0; unsigned long generation; EnterCriticalSection(&cond->csection); cond->num_waiting++; generation = cond->generation; LeaveCriticalSection(&cond->csection); apr_thread_mutex_unlock(mutex); do { res = WaitForSingleObject(cond->semaphore, timeout_ms); EnterCriticalSection(&cond->csection); if (cond->num_wake) { if (cond->generation != generation) { cond->num_wake--; cond->num_waiting--; rv = APR_SUCCESS; break; } else { wake = 1; } } else if (res != WAIT_OBJECT_0) { cond->num_waiting--; rv = APR_TIMEUP; break; } LeaveCriticalSection(&cond->csection); if (wake) { wake = 0; ReleaseSemaphore(cond->semaphore, 1, NULL); } } while (1); LeaveCriticalSection(&cond->csection); apr_thread_mutex_lock(mutex); return rv; } From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: Wednesday, August 20, 2014 6:13 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd https://github.com/traviscross/apr On Wed, Aug 20, 2014 at 4:44 PM, Andre <andretodd@verizon.net <mailto:andretodd@verizon.net> > wrote: Thanks Anthony, I can provide the Dump for you it’s 8gb unzipped so I’ll compress it. The tool I use is Visual studio but I’m sure there are other tools too. I can run a the profiler tool in visual studio to find the CPU. Also how do I get the modified copy of the APR? Thanks again guys. Andre From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> ] On Behalf Of Anthony Minessale Sent: Wednesday, August 20, 2014 5:26 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd The cond waits are all of your session threads in bypass media mode asleep in the state machine. Its hard to tell with that blurry screen shot what else is going on. Is there a way to produce that data in a way we could browse it? You should be looking for threads in the sofia sip stack and the ones doing originate if you are worried about pdd. Also which threads are costing the most CPU. If conditionals are implemented in windows as waitforsingleobject i have no idea if that's as efficient as it is in linux. You could try forklifting in our modified copy of APR that has been updated to work with their latest release and see if anything has improved in the windows implementations of conditionals. I recall Jeff had to backport in at least one patch for that over the years. On Wed, Aug 20, 2014 at 4:11 PM, Andre <andretodd@verizon.net <mailto:andretodd@verizon.net> > wrote: Ok I’ll try that next. I’ll create xml dialplan to see if that makes a difference. One note, when I didn’t load the deck in my managed code my dial plan did return a 503 in 0 – 20 milliseconds. Not sure if that matters since no bridge was ever called. Thanks again, Andre From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> ] On Behalf Of Michael Jerris Sent: Wednesday, August 20, 2014 5:02 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd I would test this without any of your managed code at all, to confirm that it is not the bottleneck. On Aug 20, 2014, at 4:28 PM, Andre <andretodd@verizon.net <mailto:andretodd@verizon.net> > wrote: I added the new code but it doesn’t seem to fix it. :/ From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Andre Sent: Wednesday, August 20, 2014 3:37 PM To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] high pdd Thanks I’ll work on it right a way. From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [ <mailto:freeswitch-users-bounces@lists.freeswitch.org> mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Peter Olsson Sent: Wednesday, August 20, 2014 3:30 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd It seems it might be stuck in the condition code for Windows. Try this timing patch for Windows, and see if it makes any difference. I've tested in on lower load, but it would be interesting how it performs in this scenario. <https://jira.freeswitch.org/secure/attachment/20613/windows-waitable-timer-diff.txt> https://jira.freeswitch.org/secure/attachment/20613/windows-waitable-timer-d... 2014-08-20 21:13 GMT+02:00 Andre < <mailto:andretodd@verizon.net> andretodd@verizon.net>: I will, need to get hardware and figure out how to use mono. In the meantime, I’ve attached some of the call traces. <image001.jpg><image002.jpg> <image003.jpg> From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [mailto: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfBrian West Sent: Wednesday, August 20, 2014 3:05 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd Try the same thing on Debian 7 if possible. On Wed, Aug 20, 2014 at 12:35 PM, Andre < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Hi, I’m using the master. Should I try 1.4.7? I’ve done a dump after things started to go bad/ backed up. I see 8193 threads in _thread_cond_timedwait. Looks like its on line 81 of thread_cond.c res = WaitForSingleObject(cond->semaphore, timeout_ms); The value of timeout_ms is <tel:2016850392> 2016850392. Not sure if this means anything. Thanks for the support guys. Andre From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [mailto: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfBrian West Sent: Wednesday, August 20, 2014 11:57 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd 1.2.X is EOL, 1.4.7 is the current release. We have releases and tags in git that no longer carry the 'stable' moniker starting with 1.4. On Sun, Aug 17, 2014 at 1:39 PM, Andre Demattia < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Thanks, I'm using the newest master and the last stable 1.2.23 We are willing to pay to get this fixed. Andre _____ From: <mailto:anthony.minessale@gmail.com> Anthony Minessale Sent: ‎8/‎17/‎2014 12:30 PM To: <mailto:freeswitch-users@lists.freeswitch.org> Freeswitch-users Subject: Re: [Freeswitch-users] high pdd We dont have a lot of tools to troubleshoot windows. You should mention windows in your subject to have a better chance to get the attention of our windows users. You should consider installing debian 7 on the same hardware and perform similar tests to help determine if your issue is o/s specific or not. You also have not indicated the revision of FS. If you have not worked with latest master, you should be also to rule out improvements. On Aug 17, 2014 9:54 AM, "Andre Demattia" < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Yes that's off and I believe all the databases are off too. No authentication or registration is enabled. The sizes of the database are 0 until I call the shutdown command then its about 90k. I have the DBS in Ra drive too. Windows seems pretty good until the 3000 sessions per min mark. Then I just get long pdd and the threads get backed up then no more sessions can be created. Funny thing is I has 10 profiles and the same results. I also tried two consoles with the same results. FreeSWITCH is great. I'd be grateful for help. I've worked on this for 2 years and hate to give up. If any one from <mailto:consulting@freeswitch.org> consulting@freeswitch.org is willing to help I'd appreciate that. Thanks again, Andre _____ From: <mailto:anthony.minessale@gmail.com> Anthony Minessale Sent: ‎8/‎17/‎2014 10:13 AM To: <mailto:freeswitch-users@lists.freeswitch.org> Freeswitch-users Subject: Re: [Freeswitch-users] high pdd Did you comment out manage-presence ? Also we have no real idea how well FS performs in windows under extreme load. On Aug 17, 2014 5:42 AM, "Andre Demattia" < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Thanks Ken, I'll try the consulting again. Andre _____ From: <mailto:krice@freeswitch.org> Ken Rice Sent: ‎8/‎17/‎2014 1:19 AM To: <mailto:freeswitch-users@lists.freeswitch.org> FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd if the last post does not work contact <mailto:consulting@freeswitch.org> consulting@freeswitch.org for pro help from one of the developers... thats about as far as the list will take you Ken Sent from my iPad On Aug 16, 2014, at 15:23, "Andre" < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Going with the idea it’s IO related I did a db_cache status. I just disabled limit in my code by removing it but limit should be HASH not a database however I stills how a database. Below is the results of the db cache status. I’m not real sure what its saying. I should have all registration and limit disabled and should not be using any sql database from my understanding. Anyone know what the results mean? I’ve only tested 2 calls on this status. [The entire original message is not included.] _________________________________________________________________________ Professional FreeSWITCH Consulting Services: <mailto:consulting@freeswitch.org> consulting@freeswitch.org <http://www.freeswitchsolutions.com/> http://www.freeswitchsolutions.com Official FreeSWITCH Sites <http://www.freeswitch.org/> http://www.freeswitch.org <http://confluence.freeswitch.org/> http://confluence.freeswitch.org <http://www.cluecon.com/> http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server <http://www.cudatel.com/> http://www.cudatel.com FreeSWITCH-users mailing list <mailto:FreeSWITCH-users@lists.freeswitch.org> FreeSWITCH-users@lists.freeswitch.org <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://lists.freeswitch.org/mailman/options/freeswitch-users [The entire original message is not included.] _________________________________________________________________________ Professional FreeSWITCH Consulting Services: <mailto:consulting@freeswitch.org> consulting@freeswitch.org <http://www.freeswitchsolutions.com/> http://www.freeswitchsolutions.com Official FreeSWITCH Sites <http://www.freeswitch.org/> http://www.freeswitch.org <http://confluence.freeswitch.org/> http://confluence.freeswitch.org <http://www.cluecon.com/> http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server <http://www.cudatel.com/> http://www.cudatel.com FreeSWITCH-users mailing list <mailto:FreeSWITCH-users@lists.freeswitch.org> FreeSWITCH-users@lists.freeswitch.org <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://lists.freeswitch.org/mailman/options/freeswitch-users <http://www.freeswitch.org/> http://www.freeswitch.org -- Brian West <mailto:brian@freeswitch.org> brian@freeswitch.org <http://bkw.org/whmcslogo.png> Twitter: @FreeSWITCH , @briankwest <http://www.freeswitchbook.com/> http://www.freeswitchbook.com <http://www.freeswitchcookbook.com/> http://www.freeswitchcookbook.com T: <tel:%2B19184209001> +19184209001 | F: <tel:%2B19184209002> +19184209002 | M:+1918424WEST (9378) iNUM:+883 5100 1420 9001 | ISN:410*543 | Skype:briankwest _________________________________________________________________________ Professional FreeSWITCH Consulting Services: <mailto:consulting@freeswitch.org> consulting@freeswitch.org <http://www.freeswitchsolutions.com/> http://www.freeswitchsolutions.com Official FreeSWITCH Sites <http://www.freeswitch.org/> http://www.freeswitch.org <http://confluence.freeswitch.org/> http://confluence.freeswitch.org <http://www.cluecon.com/> http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server <http://www.cudatel.com/> http://www.cudatel.com FreeSWITCH-users mailing list <mailto:FreeSWITCH-users@lists.freeswitch.org> FreeSWITCH-users@lists.freeswitch.org <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://lists.freeswitch.org/mailman/options/freeswitch-users <http://www.freeswitch.org/> http://www.freeswitch.org -- Brian West <mailto:brian@freeswitch.org> brian@freeswitch.org <http://bkw.org/whmcslogo.png> Twitter: @FreeSWITCH , @briankwest <http://www.freeswitchbook.com/> http://www.freeswitchbook.com <http://www.freeswitchcookbook.com/> http://www.freeswitchcookbook.com T: <tel:%2B19184209001> +19184209001 | F: <tel:%2B19184209002> +19184209002 | M:+1918424WEST (9378) iNUM:+883 5100 1420 9001 | ISN:410*543 | Skype:briankwest _________________________________________________________________________ Professional FreeSWITCH Consulting Services: <mailto:consulting@freeswitch.org> consulting@freeswitch.org <http://www.freeswitchsolutions.com/> http://www.freeswitchsolutions.com Official FreeSWITCH Sites <http://www.freeswitch.org/> http://www.freeswitch.org <http://confluence.freeswitch.org/> http://confluence.freeswitch.org <http://www.cluecon.com/> http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server <http://www.cudatel.com/> http://www.cudatel.com FreeSWITCH-users mailing list <mailto:FreeSWITCH-users@lists.freeswitch.org> FreeSWITCH-users@lists.freeswitch.org <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://lists.freeswitch.org/mailman/options/freeswitch-users <http://www.freeswitch.org/> http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net <http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 <tel:%2B19193869900> _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net <http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900

Thanks I’ll do that in the morning. We still will try Travis APR code but I’m not sure how to compile it. Any hits would be grateful. I’ve read the read me files but I still get errors :/ Also, I’ve installed Debian as suggested and hope to test soon. Andre From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Moishe Grunstein Sent: Wednesday, August 27, 2014 5:22 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd You should put this in a Jira, this thread is getting very complex to track. From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Andrew Sent: Wednesday, August 27, 2014 4:25 PM To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] high pdd Hi, I know Andre is trying on UNIX but wanted to share what I found.. I too have the same issue. But I am using Master x64 embedded. After 2600 threads the program just keeps adding a ton threads (very fast) and wireshark shows Lots of re invites. My magic number is 1300 ports(bridge). CPS does not seem to matter but its more ports/Threads. Not sure but when doing a dump I see EnterCriticalSection(&cond->csection); is being pointed to in thread_cond.c static APR_INLINE apr_status_t _thread_cond_timedwait(apr_thread_cond_t *cond, apr_thread_mutex_t *mutex, DWORD timeout_ms ) { DWORD res; apr_status_t rv; unsigned int wake = 0; unsigned long generation; EnterCriticalSection(&cond->csection); cond->num_waiting++; generation = cond->generation; LeaveCriticalSection(&cond->csection); apr_thread_mutex_unlock(mutex); do { res = WaitForSingleObject(cond->semaphore, timeout_ms); EnterCriticalSection(&cond->csection); if (cond->num_wake) { if (cond->generation != generation) { cond->num_wake--; cond->num_waiting--; rv = APR_SUCCESS; break; } else { wake = 1; } } else if (res != WAIT_OBJECT_0) { cond->num_waiting--; rv = APR_TIMEUP; break; } LeaveCriticalSection(&cond->csection); if (wake) { wake = 0; ReleaseSemaphore(cond->semaphore, 1, NULL); } } while (1); LeaveCriticalSection(&cond->csection); apr_thread_mutex_lock(mutex); return rv; } From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: Wednesday, August 20, 2014 6:13 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd https://github.com/traviscross/apr On Wed, Aug 20, 2014 at 4:44 PM, Andre <andretodd@verizon.net <mailto:andretodd@verizon.net> > wrote: Thanks Anthony, I can provide the Dump for you it’s 8gb unzipped so I’ll compress it. The tool I use is Visual studio but I’m sure there are other tools too. I can run a the profiler tool in visual studio to find the CPU. Also how do I get the modified copy of the APR? Thanks again guys. Andre From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> ] On Behalf Of Anthony Minessale Sent: Wednesday, August 20, 2014 5:26 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd The cond waits are all of your session threads in bypass media mode asleep in the state machine. Its hard to tell with that blurry screen shot what else is going on. Is there a way to produce that data in a way we could browse it? You should be looking for threads in the sofia sip stack and the ones doing originate if you are worried about pdd. Also which threads are costing the most CPU. If conditionals are implemented in windows as waitforsingleobject i have no idea if that's as efficient as it is in linux. You could try forklifting in our modified copy of APR that has been updated to work with their latest release and see if anything has improved in the windows implementations of conditionals. I recall Jeff had to backport in at least one patch for that over the years. On Wed, Aug 20, 2014 at 4:11 PM, Andre <andretodd@verizon.net <mailto:andretodd@verizon.net> > wrote: Ok I’ll try that next. I’ll create xml dialplan to see if that makes a difference. One note, when I didn’t load the deck in my managed code my dial plan did return a 503 in 0 – 20 milliseconds. Not sure if that matters since no bridge was ever called. Thanks again, Andre From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> ] On Behalf Of Michael Jerris Sent: Wednesday, August 20, 2014 5:02 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd I would test this without any of your managed code at all, to confirm that it is not the bottleneck. On Aug 20, 2014, at 4:28 PM, Andre <andretodd@verizon.net <mailto:andretodd@verizon.net> > wrote: I added the new code but it doesn’t seem to fix it. :/ From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Andre Sent: Wednesday, August 20, 2014 3:37 PM To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] high pdd Thanks I’ll work on it right a way. From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [ <mailto:freeswitch-users-bounces@lists.freeswitch.org> mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Peter Olsson Sent: Wednesday, August 20, 2014 3:30 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd It seems it might be stuck in the condition code for Windows. Try this timing patch for Windows, and see if it makes any difference. I've tested in on lower load, but it would be interesting how it performs in this scenario. <https://jira.freeswitch.org/secure/attachment/20613/windows-waitable-timer-diff.txt> https://jira.freeswitch.org/secure/attachment/20613/windows-waitable-timer-d... 2014-08-20 21:13 GMT+02:00 Andre < <mailto:andretodd@verizon.net> andretodd@verizon.net>: I will, need to get hardware and figure out how to use mono. In the meantime, I’ve attached some of the call traces. <image001.jpg><image002.jpg> <image003.jpg> From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [mailto: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfBrian West Sent: Wednesday, August 20, 2014 3:05 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd Try the same thing on Debian 7 if possible. On Wed, Aug 20, 2014 at 12:35 PM, Andre < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Hi, I’m using the master. Should I try 1.4.7? I’ve done a dump after things started to go bad/ backed up. I see 8193 threads in _thread_cond_timedwait. Looks like its on line 81 of thread_cond.c res = WaitForSingleObject(cond->semaphore, timeout_ms); The value of timeout_ms is <tel:2016850392> 2016850392. Not sure if this means anything. Thanks for the support guys. Andre From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [mailto: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfBrian West Sent: Wednesday, August 20, 2014 11:57 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd 1.2.X is EOL, 1.4.7 is the current release. We have releases and tags in git that no longer carry the 'stable' moniker starting with 1.4. On Sun, Aug 17, 2014 at 1:39 PM, Andre Demattia < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Thanks, I'm using the newest master and the last stable 1.2.23 We are willing to pay to get this fixed. Andre _____ From: <mailto:anthony.minessale@gmail.com> Anthony Minessale Sent: ‎8/‎17/‎2014 12:30 PM To: <mailto:freeswitch-users@lists.freeswitch.org> Freeswitch-users Subject: Re: [Freeswitch-users] high pdd We dont have a lot of tools to troubleshoot windows. You should mention windows in your subject to have a better chance to get the attention of our windows users. You should consider installing debian 7 on the same hardware and perform similar tests to help determine if your issue is o/s specific or not. You also have not indicated the revision of FS. If you have not worked with latest master, you should be also to rule out improvements. On Aug 17, 2014 9:54 AM, "Andre Demattia" < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Yes that's off and I believe all the databases are off too. No authentication or registration is enabled. The sizes of the database are 0 until I call the shutdown command then its about 90k. I have the DBS in Ra drive too. Windows seems pretty good until the 3000 sessions per min mark. Then I just get long pdd and the threads get backed up then no more sessions can be created. Funny thing is I has 10 profiles and the same results. I also tried two consoles with the same results. FreeSWITCH is great. I'd be grateful for help. I've worked on this for 2 years and hate to give up. If any one from <mailto:consulting@freeswitch.org> consulting@freeswitch.org is willing to help I'd appreciate that. Thanks again, Andre _____ From: <mailto:anthony.minessale@gmail.com> Anthony Minessale Sent: ‎8/‎17/‎2014 10:13 AM To: <mailto:freeswitch-users@lists.freeswitch.org> Freeswitch-users Subject: Re: [Freeswitch-users] high pdd Did you comment out manage-presence ? Also we have no real idea how well FS performs in windows under extreme load. On Aug 17, 2014 5:42 AM, "Andre Demattia" < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Thanks Ken, I'll try the consulting again. Andre _____ From: <mailto:krice@freeswitch.org> Ken Rice Sent: ‎8/‎17/‎2014 1:19 AM To: <mailto:freeswitch-users@lists.freeswitch.org> FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd if the last post does not work contact <mailto:consulting@freeswitch.org> consulting@freeswitch.org for pro help from one of the developers... thats about as far as the list will take you Ken Sent from my iPad On Aug 16, 2014, at 15:23, "Andre" < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Going with the idea it’s IO related I did a db_cache status. I just disabled limit in my code by removing it but limit should be HASH not a database however I stills how a database. Below is the results of the db cache status. I’m not real sure what its saying. I should have all registration and limit disabled and should not be using any sql database from my understanding. Anyone know what the results mean? I’ve only tested 2 calls on this status. [The entire original message is not included.] _________________________________________________________________________ Professional FreeSWITCH Consulting Services: <mailto:consulting@freeswitch.org> consulting@freeswitch.org <http://www.freeswitchsolutions.com/> http://www.freeswitchsolutions.com Official FreeSWITCH Sites <http://www.freeswitch.org/> http://www.freeswitch.org <http://confluence.freeswitch.org/> http://confluence.freeswitch.org <http://www.cluecon.com/> http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server <http://www.cudatel.com/> http://www.cudatel.com FreeSWITCH-users mailing list <mailto:FreeSWITCH-users@lists.freeswitch.org> FreeSWITCH-users@lists.freeswitch.org <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://lists.freeswitch.org/mailman/options/freeswitch-users [The entire original message is not included.] _________________________________________________________________________ Professional FreeSWITCH Consulting Services: <mailto:consulting@freeswitch.org> consulting@freeswitch.org <http://www.freeswitchsolutions.com/> http://www.freeswitchsolutions.com Official FreeSWITCH Sites <http://www.freeswitch.org/> http://www.freeswitch.org <http://confluence.freeswitch.org/> http://confluence.freeswitch.org <http://www.cluecon.com/> http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server <http://www.cudatel.com/> http://www.cudatel.com FreeSWITCH-users mailing list <mailto:FreeSWITCH-users@lists.freeswitch.org> FreeSWITCH-users@lists.freeswitch.org <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://lists.freeswitch.org/mailman/options/freeswitch-users <http://www.freeswitch.org/> http://www.freeswitch.org -- Brian West <mailto:brian@freeswitch.org> brian@freeswitch.org <http://bkw.org/whmcslogo.png> Twitter: @FreeSWITCH , @briankwest <http://www.freeswitchbook.com/> http://www.freeswitchbook.com <http://www.freeswitchcookbook.com/> http://www.freeswitchcookbook.com T: <tel:%2B19184209001> +19184209001 | F: <tel:%2B19184209002> +19184209002 | M:+1918424WEST (9378) iNUM:+883 5100 1420 9001 | ISN:410*543 | Skype:briankwest _________________________________________________________________________ Professional FreeSWITCH Consulting Services: <mailto:consulting@freeswitch.org> consulting@freeswitch.org <http://www.freeswitchsolutions.com/> http://www.freeswitchsolutions.com Official FreeSWITCH Sites <http://www.freeswitch.org/> http://www.freeswitch.org <http://confluence.freeswitch.org/> http://confluence.freeswitch.org <http://www.cluecon.com/> http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server <http://www.cudatel.com/> http://www.cudatel.com FreeSWITCH-users mailing list <mailto:FreeSWITCH-users@lists.freeswitch.org> FreeSWITCH-users@lists.freeswitch.org <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://lists.freeswitch.org/mailman/options/freeswitch-users <http://www.freeswitch.org/> http://www.freeswitch.org -- Brian West <mailto:brian@freeswitch.org> brian@freeswitch.org <http://bkw.org/whmcslogo.png> Twitter: @FreeSWITCH , @briankwest <http://www.freeswitchbook.com/> http://www.freeswitchbook.com <http://www.freeswitchcookbook.com/> http://www.freeswitchcookbook.com T: <tel:%2B19184209001> +19184209001 | F: <tel:%2B19184209002> +19184209002 | M:+1918424WEST (9378) iNUM:+883 5100 1420 9001 | ISN:410*543 | Skype:briankwest _________________________________________________________________________ Professional FreeSWITCH Consulting Services: <mailto:consulting@freeswitch.org> consulting@freeswitch.org <http://www.freeswitchsolutions.com/> http://www.freeswitchsolutions.com Official FreeSWITCH Sites <http://www.freeswitch.org/> http://www.freeswitch.org <http://confluence.freeswitch.org/> http://confluence.freeswitch.org <http://www.cluecon.com/> http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server <http://www.cudatel.com/> http://www.cudatel.com FreeSWITCH-users mailing list <mailto:FreeSWITCH-users@lists.freeswitch.org> FreeSWITCH-users@lists.freeswitch.org <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://lists.freeswitch.org/mailman/options/freeswitch-users <http://www.freeswitch.org/> http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net <http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 <tel:%2B19193869900> _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net <http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900

Dump Link sent direct to you in email. Andrew From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Dave Kompel Sent: Wednesday, August 27, 2014 6:10 PM To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] high pdd Does anyone still have a dump with the stack trace of those threads in them. The more I think about this, the more it “feels” like it’s not the mutex code in libapr, but what is calling it. Can someone post one level up in the stacktrace? --Dave From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Andre Sent: Wednesday, August 27, 2014 5:34 PM To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] high pdd Thanks I’ll do that in the morning. We still will try Travis APR code but I’m not sure how to compile it. Any hits would be grateful. I’ve read the read me files but I still get errors :/ Also, I’ve installed Debian as suggested and hope to test soon. Andre From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Moishe Grunstein Sent: Wednesday, August 27, 2014 5:22 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd You should put this in a Jira, this thread is getting very complex to track. From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Andrew Sent: Wednesday, August 27, 2014 4:25 PM To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] high pdd Hi, I know Andre is trying on UNIX but wanted to share what I found.. I too have the same issue. But I am using Master x64 embedded. After 2600 threads the program just keeps adding a ton threads (very fast) and wireshark shows Lots of re invites. My magic number is 1300 ports(bridge). CPS does not seem to matter but its more ports/Threads. Not sure but when doing a dump I see EnterCriticalSection(&cond->csection); is being pointed to in thread_cond.c static APR_INLINE apr_status_t _thread_cond_timedwait(apr_thread_cond_t *cond, apr_thread_mutex_t *mutex, DWORD timeout_ms ) { DWORD res; apr_status_t rv; unsigned int wake = 0; unsigned long generation; EnterCriticalSection(&cond->csection); cond->num_waiting++; generation = cond->generation; LeaveCriticalSection(&cond->csection); apr_thread_mutex_unlock(mutex); do { res = WaitForSingleObject(cond->semaphore, timeout_ms); EnterCriticalSection(&cond->csection); if (cond->num_wake) { if (cond->generation != generation) { cond->num_wake--; cond->num_waiting--; rv = APR_SUCCESS; break; } else { wake = 1; } } else if (res != WAIT_OBJECT_0) { cond->num_waiting--; rv = APR_TIMEUP; break; } LeaveCriticalSection(&cond->csection); if (wake) { wake = 0; ReleaseSemaphore(cond->semaphore, 1, NULL); } } while (1); LeaveCriticalSection(&cond->csection); apr_thread_mutex_lock(mutex); return rv; } From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: Wednesday, August 20, 2014 6:13 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd https://github.com/traviscross/apr On Wed, Aug 20, 2014 at 4:44 PM, Andre <andretodd@verizon.net <mailto:andretodd@verizon.net> > wrote: Thanks Anthony, I can provide the Dump for you it’s 8gb unzipped so I’ll compress it. The tool I use is Visual studio but I’m sure there are other tools too. I can run a the profiler tool in visual studio to find the CPU. Also how do I get the modified copy of the APR? Thanks again guys. Andre From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> ] On Behalf Of Anthony Minessale Sent: Wednesday, August 20, 2014 5:26 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd The cond waits are all of your session threads in bypass media mode asleep in the state machine. Its hard to tell with that blurry screen shot what else is going on. Is there a way to produce that data in a way we could browse it? You should be looking for threads in the sofia sip stack and the ones doing originate if you are worried about pdd. Also which threads are costing the most CPU. If conditionals are implemented in windows as waitforsingleobject i have no idea if that's as efficient as it is in linux. You could try forklifting in our modified copy of APR that has been updated to work with their latest release and see if anything has improved in the windows implementations of conditionals. I recall Jeff had to backport in at least one patch for that over the years. On Wed, Aug 20, 2014 at 4:11 PM, Andre <andretodd@verizon.net <mailto:andretodd@verizon.net> > wrote: Ok I’ll try that next. I’ll create xml dialplan to see if that makes a difference. One note, when I didn’t load the deck in my managed code my dial plan did return a 503 in 0 – 20 milliseconds. Not sure if that matters since no bridge was ever called. Thanks again, Andre From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> ] On Behalf Of Michael Jerris Sent: Wednesday, August 20, 2014 5:02 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd I would test this without any of your managed code at all, to confirm that it is not the bottleneck. On Aug 20, 2014, at 4:28 PM, Andre <andretodd@verizon.net <mailto:andretodd@verizon.net> > wrote: I added the new code but it doesn’t seem to fix it. :/ From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Andre Sent: Wednesday, August 20, 2014 3:37 PM To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] high pdd Thanks I’ll work on it right a way. From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [ <mailto:freeswitch-users-bounces@lists.freeswitch.org> mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Peter Olsson Sent: Wednesday, August 20, 2014 3:30 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd It seems it might be stuck in the condition code for Windows. Try this timing patch for Windows, and see if it makes any difference. I've tested in on lower load, but it would be interesting how it performs in this scenario. <https://jira.freeswitch.org/secure/attachment/20613/windows-waitable-timer-diff.txt> https://jira.freeswitch.org/secure/attachment/20613/windows-waitable-timer-d... 2014-08-20 21:13 GMT+02:00 Andre < <mailto:andretodd@verizon.net> andretodd@verizon.net>: I will, need to get hardware and figure out how to use mono. In the meantime, I’ve attached some of the call traces. <image001.jpg><image002.jpg> <image003.jpg> From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [mailto: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfBrian West Sent: Wednesday, August 20, 2014 3:05 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd Try the same thing on Debian 7 if possible. On Wed, Aug 20, 2014 at 12:35 PM, Andre < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Hi, I’m using the master. Should I try 1.4.7? I’ve done a dump after things started to go bad/ backed up. I see 8193 threads in _thread_cond_timedwait. Looks like its on line 81 of thread_cond.c res = WaitForSingleObject(cond->semaphore, timeout_ms); The value of timeout_ms is <tel:2016850392> 2016850392. Not sure if this means anything. Thanks for the support guys. Andre From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [mailto: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfBrian West Sent: Wednesday, August 20, 2014 11:57 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd 1.2.X is EOL, 1.4.7 is the current release. We have releases and tags in git that no longer carry the 'stable' moniker starting with 1.4. On Sun, Aug 17, 2014 at 1:39 PM, Andre Demattia < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Thanks, I'm using the newest master and the last stable 1.2.23 We are willing to pay to get this fixed. Andre _____ From: <mailto:anthony.minessale@gmail.com> Anthony Minessale Sent: ‎8/‎17/‎2014 12:30 PM To: <mailto:freeswitch-users@lists.freeswitch.org> Freeswitch-users Subject: Re: [Freeswitch-users] high pdd We dont have a lot of tools to troubleshoot windows. You should mention windows in your subject to have a better chance to get the attention of our windows users. You should consider installing debian 7 on the same hardware and perform similar tests to help determine if your issue is o/s specific or not. You also have not indicated the revision of FS. If you have not worked with latest master, you should be also to rule out improvements. On Aug 17, 2014 9:54 AM, "Andre Demattia" < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Yes that's off and I believe all the databases are off too. No authentication or registration is enabled. The sizes of the database are 0 until I call the shutdown command then its about 90k. I have the DBS in Ra drive too. Windows seems pretty good until the 3000 sessions per min mark. Then I just get long pdd and the threads get backed up then no more sessions can be created. Funny thing is I has 10 profiles and the same results. I also tried two consoles with the same results. FreeSWITCH is great. I'd be grateful for help. I've worked on this for 2 years and hate to give up. If any one from <mailto:consulting@freeswitch.org> consulting@freeswitch.org is willing to help I'd appreciate that. Thanks again, Andre _____ From: <mailto:anthony.minessale@gmail.com> Anthony Minessale Sent: ‎8/‎17/‎2014 10:13 AM To: <mailto:freeswitch-users@lists.freeswitch.org> Freeswitch-users Subject: Re: [Freeswitch-users] high pdd Did you comment out manage-presence ? Also we have no real idea how well FS performs in windows under extreme load. On Aug 17, 2014 5:42 AM, "Andre Demattia" < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Thanks Ken, I'll try the consulting again. Andre _____ From: <mailto:krice@freeswitch.org> Ken Rice Sent: ‎8/‎17/‎2014 1:19 AM To: <mailto:freeswitch-users@lists.freeswitch.org> FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd if the last post does not work contact <mailto:consulting@freeswitch.org> consulting@freeswitch.org for pro help from one of the developers... thats about as far as the list will take you Ken Sent from my iPad On Aug 16, 2014, at 15:23, "Andre" < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Going with the idea it’s IO related I did a db_cache status. I just disabled limit in my code by removing it but limit should be HASH not a database however I stills how a database. Below is the results of the db cache status. I’m not real sure what its saying. I should have all registration and limit disabled and should not be using any sql database from my understanding. Anyone know what the results mean? I’ve only tested 2 calls on this status. [The entire original message is not included.] _________________________________________________________________________ Professional FreeSWITCH Consulting Services: <mailto:consulting@freeswitch.org> consulting@freeswitch.org <http://www.freeswitchsolutions.com/> http://www.freeswitchsolutions.com Official FreeSWITCH Sites <http://www.freeswitch.org/> http://www.freeswitch.org <http://confluence.freeswitch.org/> http://confluence.freeswitch.org <http://www.cluecon.com/> http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server <http://www.cudatel.com/> http://www.cudatel.com FreeSWITCH-users mailing list <mailto:FreeSWITCH-users@lists.freeswitch.org> FreeSWITCH-users@lists.freeswitch.org <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://lists.freeswitch.org/mailman/options/freeswitch-users [The entire original message is not included.] _________________________________________________________________________ Professional FreeSWITCH Consulting Services: <mailto:consulting@freeswitch.org> consulting@freeswitch.org <http://www.freeswitchsolutions.com/> http://www.freeswitchsolutions.com Official FreeSWITCH Sites <http://www.freeswitch.org/> http://www.freeswitch.org <http://confluence.freeswitch.org/> http://confluence.freeswitch.org <http://www.cluecon.com/> http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server <http://www.cudatel.com/> http://www.cudatel.com FreeSWITCH-users mailing list <mailto:FreeSWITCH-users@lists.freeswitch.org> FreeSWITCH-users@lists.freeswitch.org <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://lists.freeswitch.org/mailman/options/freeswitch-users <http://www.freeswitch.org/> http://www.freeswitch.org -- Brian West <mailto:brian@freeswitch.org> brian@freeswitch.org Twitter: @FreeSWITCH , @briankwest <http://www.freeswitchbook.com/> http://www.freeswitchbook.com <http://www.freeswitchcookbook.com/> http://www.freeswitchcookbook.com T: <tel:%2B19184209001> +19184209001 | F: <tel:%2B19184209002> +19184209002 | M:+1918424WEST (9378) iNUM:+883 5100 1420 9001 | ISN:410*543 | Skype:briankwest _________________________________________________________________________ Professional FreeSWITCH Consulting Services: <mailto:consulting@freeswitch.org> consulting@freeswitch.org <http://www.freeswitchsolutions.com/> http://www.freeswitchsolutions.com Official FreeSWITCH Sites <http://www.freeswitch.org/> http://www.freeswitch.org <http://confluence.freeswitch.org/> http://confluence.freeswitch.org <http://www.cluecon.com/> http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server <http://www.cudatel.com/> http://www.cudatel.com FreeSWITCH-users mailing list <mailto:FreeSWITCH-users@lists.freeswitch.org> FreeSWITCH-users@lists.freeswitch.org <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://lists.freeswitch.org/mailman/options/freeswitch-users <http://www.freeswitch.org/> http://www.freeswitch.org -- Brian West <mailto:brian@freeswitch.org> brian@freeswitch.org Twitter: @FreeSWITCH , @briankwest <http://www.freeswitchbook.com/> http://www.freeswitchbook.com <http://www.freeswitchcookbook.com/> http://www.freeswitchcookbook.com T: <tel:%2B19184209001> +19184209001 | F: <tel:%2B19184209002> +19184209002 | M:+1918424WEST (9378) iNUM:+883 5100 1420 9001 | ISN:410*543 | Skype:briankwest _________________________________________________________________________ Professional FreeSWITCH Consulting Services: <mailto:consulting@freeswitch.org> consulting@freeswitch.org <http://www.freeswitchsolutions.com/> http://www.freeswitchsolutions.com Official FreeSWITCH Sites <http://www.freeswitch.org/> http://www.freeswitch.org <http://confluence.freeswitch.org/> http://confluence.freeswitch.org <http://www.cluecon.com/> http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server <http://www.cudatel.com/> http://www.cudatel.com FreeSWITCH-users mailing list <mailto:FreeSWITCH-users@lists.freeswitch.org> FreeSWITCH-users@lists.freeswitch.org <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: <http://lists.freeswitch.org/mailman/options/freeswitch-users> http://lists.freeswitch.org/mailman/options/freeswitch-users <http://www.freeswitch.org/> http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net <http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900 <tel:%2B19193869900> _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net <http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900

Funny thing is I do think the calls are blocking each other. I have found when embedded hits 2800(1700 ports bridge) threads that no new calls will Come through and they are all blocking. I see the threads go from 2800 to 10k threads in seconds when this happens. I know Andre is not using embedded and is getting double the threads as me but I think we both have the same issue. Attached is the profiler running real live traffic for about 10 min. Using embedded I don’t get passed 17% CPU. All media bypass. Andrew From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Michael Jerris Sent: Thursday, August 28, 2014 11:10 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd Most likely it is neither of these. I'm seriously doubting that the mutexes are causing problems unless you are actually seeing multiple calls blocking each other, which i don't think is happening here. I think you need to be looking at what is using the most cpu. Have you looked at running in the profiler? On Aug 27, 2014, at 6:10 PM, Dave Kompel <drk@drkngs.net <mailto:drk@drkngs.net> > wrote: Does anyone still have a dump with the stack trace of those threads in them. The more I think about this, the more it “feels” like it’s not the mutex code in libapr, but what is calling it. Can someone post one level up in the stacktrace? --Dave From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [ <mailto:freeswitch-users-bounces@lists.freeswitch.org> mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Andre Sent: Wednesday, August 27, 2014 5:34 PM To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] high pdd Thanks I’ll do that in the morning. We still will try Travis APR code but I’m not sure how to compile it. Any hits would be grateful. I’ve read the read me files but I still get errors :/ Also, I’ve installed Debian as suggested and hope to test soon. Andre From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [ <mailto:freeswitch-users-bounces@lists.freeswitch.org> mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfMoishe Grunstein Sent: Wednesday, August 27, 2014 5:22 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd You should put this in a Jira, this thread is getting very complex to track. From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [ <mailto:freeswitch-users-bounces@lists.freeswitch.org> mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfAndrew Sent: Wednesday, August 27, 2014 4:25 PM To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] high pdd Hi, I know Andre is trying on UNIX but wanted to share what I found.. I too have the same issue. But I am using Master x64 embedded. After 2600 threads the program just keeps adding a ton threads (very fast) and wireshark shows Lots of re invites. My magic number is 1300 ports(bridge). CPS does not seem to matter but its more ports/Threads. Not sure but when doing a dump I see EnterCriticalSection(&cond->csection); is being pointed to in thread_cond.c static APR_INLINE apr_status_t _thread_cond_timedwait(apr_thread_cond_t *cond, apr_thread_mutex_t *mutex, DWORD timeout_ms ) { DWORD res; apr_status_t rv; unsigned int wake = 0; unsigned long generation; EnterCriticalSection(&cond->csection); cond->num_waiting++; generation = cond->generation; LeaveCriticalSection(&cond->csection); apr_thread_mutex_unlock(mutex); do { res = WaitForSingleObject(cond->semaphore, timeout_ms); EnterCriticalSection(&cond->csection); if (cond->num_wake) { if (cond->generation != generation) { cond->num_wake--; cond->num_waiting--; rv = APR_SUCCESS; break; } else { wake = 1; } } else if (res != WAIT_OBJECT_0) { cond->num_waiting--; rv = APR_TIMEUP; break; } LeaveCriticalSection(&cond->csection); if (wake) { wake = 0; ReleaseSemaphore(cond->semaphore, 1, NULL); } } while (1); LeaveCriticalSection(&cond->csection); apr_thread_mutex_lock(mutex); return rv; } From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [ <mailto:freeswitch-users-bounces@lists.freeswitch.org> mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfAnthony Minessale Sent: Wednesday, August 20, 2014 6:13 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd <https://github.com/traviscross/apr> https://github.com/traviscross/apr On Wed, Aug 20, 2014 at 4:44 PM, Andre < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Thanks Anthony, I can provide the Dump for you it’s 8gb unzipped so I’ll compress it. The tool I use is Visual studio but I’m sure there are other tools too. I can run a the profiler tool in visual studio to find the CPU. Also how do I get the modified copy of the APR? Thanks again guys. Andre From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [mailto: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfAnthony Minessale Sent: Wednesday, August 20, 2014 5:26 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd The cond waits are all of your session threads in bypass media mode asleep in the state machine. Its hard to tell with that blurry screen shot what else is going on. Is there a way to produce that data in a way we could browse it? You should be looking for threads in the sofia sip stack and the ones doing originate if you are worried about pdd. Also which threads are costing the most CPU. If conditionals are implemented in windows as waitforsingleobject i have no idea if that's as efficient as it is in linux. You could try forklifting in our modified copy of APR that has been updated to work with their latest release and see if anything has improved in the windows implementations of conditionals. I recall Jeff had to backport in at least one patch for that over the years. On Wed, Aug 20, 2014 at 4:11 PM, Andre < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Ok I’ll try that next. I’ll create xml dialplan to see if that makes a difference. One note, when I didn’t load the deck in my managed code my dial plan did return a 503 in 0 – 20 milliseconds. Not sure if that matters since no bridge was ever called. Thanks again, Andre From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [mailto: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfMichael Jerris Sent: Wednesday, August 20, 2014 5:02 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd I would test this without any of your managed code at all, to confirm that it is not the bottleneck. On Aug 20, 2014, at 4:28 PM, Andre < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: I added the new code but it doesn’t seem to fix it. :/ From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [ <mailto:freeswitch-users-bounces@lists.freeswitch.org> mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Andre Sent: Wednesday, August 20, 2014 3:37 PM To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] high pdd Thanks I’ll work on it right a way. From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [ <mailto:freeswitch-users-bounces@lists.freeswitch.org> mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Peter Olsson Sent: Wednesday, August 20, 2014 3:30 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd It seems it might be stuck in the condition code for Windows. Try this timing patch for Windows, and see if it makes any difference. I've tested in on lower load, but it would be interesting how it performs in this scenario. <https://jira.freeswitch.org/secure/attachment/20613/windows-waitable-timer-diff.txt> https://jira.freeswitch.org/secure/attachment/20613/windows-waitable-timer-d... 2014-08-20 21:13 GMT+02:00 Andre < <mailto:andretodd@verizon.net> andretodd@verizon.net>: I will, need to get hardware and figure out how to use mono. In the meantime, I’ve attached some of the call traces. <image001.jpg><image002.jpg> <image003.jpg> From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [mailto: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfBrian West Sent: Wednesday, August 20, 2014 3:05 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd Try the same thing on Debian 7 if possible. On Wed, Aug 20, 2014 at 12:35 PM, Andre < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Hi, I’m using the master. Should I try 1.4.7? I’ve done a dump after things started to go bad/ backed up. I see 8193 threads in _thread_cond_timedwait. Looks like its on line 81 of thread_cond.c res = WaitForSingleObject(cond->semaphore, timeout_ms); The value of timeout_ms is <tel:2016850392> 2016850392. Not sure if this means anything. Thanks for the support guys. Andre From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [mailto: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfBrian West Sent: Wednesday, August 20, 2014 11:57 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd 1.2.X is EOL, 1.4.7 is the current release. We have releases and tags in git that no longer carry the 'stable' moniker starting with 1.4. On Sun, Aug 17, 2014 at 1:39 PM, Andre Demattia < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Thanks, I'm using the newest master and the last stable 1.2.23 We are willing to pay to get this fixed. Andre _____ From: <mailto:anthony.minessale@gmail.com> Anthony Minessale Sent: ‎8/‎17/‎2014 12:30 PM To: <mailto:freeswitch-users@lists.freeswitch.org> Freeswitch-users Subject: Re: [Freeswitch-users] high pdd We dont have a lot of tools to troubleshoot windows. You should mention windows in your subject to have a better chance to get the attention of our windows users. You should consider installing debian 7 on the same hardware and perform similar tests to help determine if your issue is o/s specific or not. You also have not indicated the revision of FS. If you have not worked with latest master, you should be also to rule out improvements. On Aug 17, 2014 9:54 AM, "Andre Demattia" < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Yes that's off and I believe all the databases are off too. No authentication or registration is enabled. The sizes of the database are 0 until I call the shutdown command then its about 90k. I have the DBS in Ra drive too. Windows seems pretty good until the 3000 sessions per min mark. Then I just get long pdd and the threads get backed up then no more sessions can be created. Funny thing is I has 10 profiles and the same results. I also tried two consoles with the same results. FreeSWITCH is great. I'd be grateful for help. I've worked on this for 2 years and hate to give up. If any one from <mailto:consulting@freeswitch.org> consulting@freeswitch.org is willing to help I'd appreciate that. Thanks again, Andre _____ From: <mailto:anthony.minessale@gmail.com> Anthony Minessale Sent: ‎8/‎17/‎2014 10:13 AM To: <mailto:freeswitch-users@lists.freeswitch.org> Freeswitch-users Subject: Re: [Freeswitch-users] high pdd Did you comment out manage-presence ? Also we have no real idea how well FS performs in windows under extreme load. On Aug 17, 2014 5:42 AM, "Andre Demattia" < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Thanks Ken, I'll try the consulting again. Andre _____ From: <mailto:krice@freeswitch.org> Ken Rice Sent: ‎8/‎17/‎2014 1:19 AM To: <mailto:freeswitch-users@lists.freeswitch.org> FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd if the last post does not work contact <mailto:consulting@freeswitch.org> consulting@freeswitch.org for pro help from one of the developers... thats about as far as the list will take you Ken Sent from my iPad On Aug 16, 2014, at 15:23, "Andre" < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Going with the idea it’s IO related I did a db_cache status. I just disabled limit in my code by removing it but limit should be HASH not a database however I stills how a database. Below is the results of the db cache status. I’m not real sure what its saying. I should have all registration and limit disabled and should not be using any sql database from my understanding. Anyone know what the results mean? I’ve only tested 2 calls on this status. [The entire original message is not included.]

What about just producing similar load with static dialplans to rule out user applications? That should be possible to test on both debian and windows as a baseline. On Fri, Aug 29, 2014 at 4:32 PM, Andre <andretodd@verizon.net> wrote:

Thanks all. I put in a ticket and attached the dump images and CPU performance as requested.

I put it under mod_managed but now I think I should of put it under something else because the ticket went to Mike.

https://jira.freeswitch.org/browse/FS-6776

I’m also working on trying Debian as suggested however the mod_managed code needs to be changed. That will take some time.

Thanks everyone for helping.

Andre

*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Michael Jerris *Sent:* Thursday, August 28, 2014 11:10 AM

*To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] high pdd

Most likely it is neither of these. I'm seriously doubting that the mutexes are causing problems unless you are actually seeing multiple calls blocking each other, which i don't think is happening here. I think you need to be looking at what is using the most cpu. Have you looked at running in the profiler?

On Aug 27, 2014, at 6:10 PM, Dave Kompel <drk@drkngs.net> wrote:

Does anyone still have a dump with the stack trace of those threads in them. The more I think about this, the more it “feels” like it’s not the mutex code in libapr, but what is calling it. Can someone post one level up in the stacktrace?

--Dave

*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Andre *Sent:* Wednesday, August 27, 2014 5:34 PM *To:* 'FreeSWITCH Users Help' *Subject:* Re: [Freeswitch-users] high pdd

Thanks I’ll do that in the morning.

We still will try Travis APR code but I’m not sure how to compile it. Any hits would be grateful. I’ve read the read me files but I still get errors :/

Also, I’ve installed Debian as suggested and hope to test soon.

Andre

*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of*Moishe Grunstein *Sent:* Wednesday, August 27, 2014 5:22 PM *To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] high pdd

You should put this in a Jira, this thread is getting very complex to track.

*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of*Andrew *Sent:* Wednesday, August 27, 2014 4:25 PM *To:* 'FreeSWITCH Users Help' *Subject:* Re: [Freeswitch-users] high pdd

Hi,

I know Andre is trying on UNIX but wanted to share what I found..

I too have the same issue. But I am using Master x64 embedded.

After 2600 threads the program just keeps adding a ton threads (very fast) and wireshark shows

Lots of re invites. My magic number is 1300 ports(bridge). CPS does not seem to matter but its more ports/Threads.

Not sure but when doing a dump I see EnterCriticalSection(&cond->csection); is being pointed to in thread_cond.c

static APR_INLINE apr_status_t _thread_cond_timedwait(apr_thread_cond_t *cond,

apr_thread_mutex_t *mutex,

DWORD timeout_ms )

{

DWORD res;

apr_status_t rv;

unsigned int wake = 0;

unsigned long generation;

EnterCriticalSection(&cond->csection);

cond->num_waiting++;

generation = cond->generation;

LeaveCriticalSection(&cond->csection);

apr_thread_mutex_unlock(mutex);

do {

res = WaitForSingleObject(cond->semaphore, timeout_ms);

EnterCriticalSection(&cond->csection);

if (cond->num_wake) {

if (cond->generation != generation) {

cond->num_wake--;

cond->num_waiting--;

rv = APR_SUCCESS;

break;

} else {

wake = 1;

}

}

else if (res != WAIT_OBJECT_0) {

cond->num_waiting--;

rv = APR_TIMEUP;

break;

}

LeaveCriticalSection(&cond->csection);

if (wake) {

wake = 0;

ReleaseSemaphore(cond->semaphore, 1, NULL);

}

} while (1);

LeaveCriticalSection(&cond->csection);

apr_thread_mutex_lock(mutex);

return rv;

}

*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of*Anthony Minessale *Sent:* Wednesday, August 20, 2014 6:13 PM *To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] high pdd

https://github.com/traviscross/apr

On Wed, Aug 20, 2014 at 4:44 PM, Andre <andretodd@verizon.net> wrote:

Thanks Anthony,

I can provide the Dump for you it’s 8gb unzipped so I’ll compress it. The tool I use is Visual studio but I’m sure there are other tools too.

I can run a the profiler tool in visual studio to find the CPU.

Also how do I get the modified copy of the APR?

Thanks again guys.

Andre

*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of*Anthony Minessale *Sent:* Wednesday, August 20, 2014 5:26 PM

*To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] high pdd

The cond waits are all of your session threads in bypass media mode asleep in the state machine.

Its hard to tell with that blurry screen shot what else is going on. Is there a way to produce that data in a way we could browse it?

You should be looking for threads in the sofia sip stack and the ones doing originate if you are worried about pdd. Also which threads are costing the most CPU. If conditionals are implemented in windows as waitforsingleobject i have no idea if that's as efficient as it is in linux.

You could try forklifting in our modified copy of APR that has been updated to work with their latest release and see if anything has improved in the windows implementations of conditionals. I recall Jeff had to backport in at least one patch for that over the years.

On Wed, Aug 20, 2014 at 4:11 PM, Andre <andretodd@verizon.net> wrote:

Ok I’ll try that next. I’ll create xml dialplan to see if that makes a difference.

One note, when I didn’t load the deck in my managed code my dial plan did return a 503 in 0 – 20 milliseconds. Not sure if that matters since no bridge was ever called.

Thanks again,

Andre

*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of*Michael Jerris *Sent:* Wednesday, August 20, 2014 5:02 PM

*To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] high pdd

I would test this without any of your managed code at all, to confirm that it is not the bottleneck.

On Aug 20, 2014, at 4:28 PM, Andre <andretodd@verizon.net> wrote:

I added the new code but it doesn’t seem to fix it. :/

*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Andre *Sent:* Wednesday, August 20, 2014 3:37 PM *To:* 'FreeSWITCH Users Help' *Subject:* Re: [Freeswitch-users] high pdd

Thanks I’ll work on it right a way.

*From:* freeswitch-users-bounces@lists.freeswitch.org [ mailto:freeswitch-users-bounces@lists.freeswitch.org <freeswitch-users-bounces@lists.freeswitch.org>] *On Behalf Of *Peter Olsson *Sent:* Wednesday, August 20, 2014 3:30 PM *To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] high pdd

It seems it might be stuck in the condition code for Windows.

Try this timing patch for Windows, and see if it makes any difference. I've tested in on lower load, but it would be interesting how it performs in this scenario.

https://jira.freeswitch.org/secure/attachment/20613/windows-waitable-timer-d...

2014-08-20 21:13 GMT+02:00 Andre <andretodd@verizon.net>:

I will, need to get hardware and figure out how to use mono.

In the meantime, I’ve attached some of the call traces.

<image001.jpg><image002.jpg>

<image003.jpg>

*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of*Brian West *Sent:* Wednesday, August 20, 2014 3:05 PM *To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] high pdd

Try the same thing on Debian 7 if possible.

On Wed, Aug 20, 2014 at 12:35 PM, Andre <andretodd@verizon.net> wrote:

Hi, I’m using the master. Should I try 1.4.7?

I’ve done a dump after things started to go bad/ backed up.

I see 8193 threads in _*thread_cond*_timedwait. Looks like its on line 81 of thread_cond.c res = WaitForSingleObject(cond->semaphore, timeout_ms);

The value of timeout_ms is 2016850392.

Not sure if this means anything.

Thanks for the support guys.

Andre

*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of*Brian West *Sent:* Wednesday, August 20, 2014 11:57 AM

*To:* FreeSWITCH Users Help *Subject:* Re: [Freeswitch-users] high pdd

1.2.X is EOL,

1.4.7 is the current release.

We have releases and tags in git that no longer carry the 'stable' moniker starting with 1.4.

On Sun, Aug 17, 2014 at 1:39 PM, Andre Demattia <andretodd@verizon.net> wrote:

Thanks, I'm using the newest master and the last stable 1.2.23 We are willing to pay to get this fixed. Andre ------------------------------

*From: *Anthony Minessale <anthony.minessale@gmail.com> *Sent: *‎8/‎17/‎2014 12:30 PM

*To: *Freeswitch-users <freeswitch-users@lists.freeswitch.org> *Subject: *Re: [Freeswitch-users] high pdd

We dont have a lot of tools to troubleshoot windows. You should mention windows in your subject to have a better chance to get the attention of our windows users.

You should consider installing debian 7 on the same hardware and perform similar tests to help determine if your issue is o/s specific or not.

You also have not indicated the revision of FS. If you have not worked with latest master, you should be also to rule out improvements.

On Aug 17, 2014 9:54 AM, "Andre Demattia" <andretodd@verizon.net> wrote:

Yes that's off and I believe all the databases are off too. No authentication or registration is enabled. The sizes of the database are 0 until I call the shutdown command then its about 90k. I have the DBS in Ra drive too.

Windows seems pretty good until the 3000 sessions per min mark. Then I just get long pdd and the threads get backed up then no more sessions can be created.

Funny thing is I has 10 profiles and the same results. I also tried two consoles with the same results.

FreeSWITCH is great.

I'd be grateful for help. I've worked on this for 2 years and hate to give up.

If any one from consulting@freeswitch.org is willing to help I'd appreciate that.

Thanks again, Andre ------------------------------

*From: *Anthony Minessale <anthony.minessale@gmail.com> *Sent: *‎8/‎17/‎2014 10:13 AM *To: *Freeswitch-users <freeswitch-users@lists.freeswitch.org> *Subject: *Re: [Freeswitch-users] high pdd

Did you comment out manage-presence ? Also we have no real idea how well FS performs in windows under extreme load.

On Aug 17, 2014 5:42 AM, "Andre Demattia" <andretodd@verizon.net> wrote:

Thanks Ken, I'll try the consulting again. Andre ------------------------------

*From: *Ken Rice <krice@freeswitch.org> *Sent: *‎8/‎17/‎2014 1:19 AM *To: *FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> *Subject: *Re: [Freeswitch-users] high pdd

if the last post does not work contact consulting@freeswitch.org for pro help from one of the developers... thats about as far as the list will take you

Ken

Sent from my iPad

On Aug 16, 2014, at 15:23, "Andre" <andretodd@verizon.net> wrote:

Going with the idea it’s IO related I did a db_cache status. I just disabled limit in my code by removing it but limit should be HASH not a database however I stills how a database.

Below is the results of the db cache status. I’m not real sure what its saying. I should have all registration and limit disabled and should not be using any sql database from my understanding.

Anyone know what the results mean? I’ve only tested 2 calls on this status.

[The entire original message is not included.]

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900

As suggested I re-ran the test with a simple dial plan (see static.zip at https://jira.freeswitch.org/browse/FS-6776 ) with the same results. The results show the same as mod_managed static dialplan at 150CPS causes 25% CPU in malloc/free -<condition expression="^(4555.+)" field="destination_number"> <action data="hangup_after_bridge=true" application="set"/> <action data="bypass_media=true" application="set"/> <action data="sofia/internal/$1@xxx.xxx.xxx.xxx" application="bridge"/> </condition> </extension> I did try it on Debian and it appeared to have better performance but without VS I wasn’t able to tell what’s going on. Any help? Thanks Andre From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Andre Sent: Friday, August 29, 2014 5:16 PM To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] high pdd Ok that will be my next todo. Thanks for the idea. Andre From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale Sent: Friday, August 29, 2014 4:46 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd What about just producing similar load with static dialplans to rule out user applications? That should be possible to test on both debian and windows as a baseline. On Fri, Aug 29, 2014 at 4:32 PM, Andre <andretodd@verizon.net <mailto:andretodd@verizon.net> > wrote: Thanks all. I put in a ticket and attached the dump images and CPU performance as requested. I put it under mod_managed but now I think I should of put it under something else because the ticket went to Mike. https://jira.freeswitch.org/browse/FS-6776 I’m also working on trying Debian as suggested however the mod_managed code needs to be changed. That will take some time. Thanks everyone for helping. Andre From: freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> [mailto:freeswitch-users-bounces@lists.freeswitch.org <mailto:freeswitch-users-bounces@lists.freeswitch.org> ] On Behalf Of Michael Jerris Sent: Thursday, August 28, 2014 11:10 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd Most likely it is neither of these. I'm seriously doubting that the mutexes are causing problems unless you are actually seeing multiple calls blocking each other, which i don't think is happening here. I think you need to be looking at what is using the most cpu. Have you looked at running in the profiler? On Aug 27, 2014, at 6:10 PM, Dave Kompel <drk@drkngs.net <mailto:drk@drkngs.net> > wrote: Does anyone still have a dump with the stack trace of those threads in them. The more I think about this, the more it “feels” like it’s not the mutex code in libapr, but what is calling it. Can someone post one level up in the stacktrace? --Dave From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [ <mailto:freeswitch-users-bounces@lists.freeswitch.org> mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Andre Sent: Wednesday, August 27, 2014 5:34 PM To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] high pdd Thanks I’ll do that in the morning. We still will try Travis APR code but I’m not sure how to compile it. Any hits would be grateful. I’ve read the read me files but I still get errors :/ Also, I’ve installed Debian as suggested and hope to test soon. Andre From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [ <mailto:freeswitch-users-bounces@lists.freeswitch.org> mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfMoishe Grunstein Sent: Wednesday, August 27, 2014 5:22 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd You should put this in a Jira, this thread is getting very complex to track. From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [ <mailto:freeswitch-users-bounces@lists.freeswitch.org> mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfAndrew Sent: Wednesday, August 27, 2014 4:25 PM To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] high pdd Hi, I know Andre is trying on UNIX but wanted to share what I found.. I too have the same issue. But I am using Master x64 embedded. After 2600 threads the program just keeps adding a ton threads (very fast) and wireshark shows Lots of re invites. My magic number is 1300 ports(bridge). CPS does not seem to matter but its more ports/Threads. Not sure but when doing a dump I see EnterCriticalSection(&cond->csection); is being pointed to in thread_cond.c static APR_INLINE apr_status_t _thread_cond_timedwait(apr_thread_cond_t *cond, apr_thread_mutex_t *mutex, DWORD timeout_ms ) { DWORD res; apr_status_t rv; unsigned int wake = 0; unsigned long generation; EnterCriticalSection(&cond->csection); cond->num_waiting++; generation = cond->generation; LeaveCriticalSection(&cond->csection); apr_thread_mutex_unlock(mutex); do { res = WaitForSingleObject(cond->semaphore, timeout_ms); EnterCriticalSection(&cond->csection); if (cond->num_wake) { if (cond->generation != generation) { cond->num_wake--; cond->num_waiting--; rv = APR_SUCCESS; break; } else { wake = 1; } } else if (res != WAIT_OBJECT_0) { cond->num_waiting--; rv = APR_TIMEUP; break; } LeaveCriticalSection(&cond->csection); if (wake) { wake = 0; ReleaseSemaphore(cond->semaphore, 1, NULL); } } while (1); LeaveCriticalSection(&cond->csection); apr_thread_mutex_lock(mutex); return rv; } From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [ <mailto:freeswitch-users-bounces@lists.freeswitch.org> mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfAnthony Minessale Sent: Wednesday, August 20, 2014 6:13 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd <https://github.com/traviscross/apr> https://github.com/traviscross/apr On Wed, Aug 20, 2014 at 4:44 PM, Andre < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Thanks Anthony, I can provide the Dump for you it’s 8gb unzipped so I’ll compress it. The tool I use is Visual studio but I’m sure there are other tools too. I can run a the profiler tool in visual studio to find the CPU. Also how do I get the modified copy of the APR? Thanks again guys. Andre From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [mailto: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfAnthony Minessale Sent: Wednesday, August 20, 2014 5:26 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd The cond waits are all of your session threads in bypass media mode asleep in the state machine. Its hard to tell with that blurry screen shot what else is going on. Is there a way to produce that data in a way we could browse it? You should be looking for threads in the sofia sip stack and the ones doing originate if you are worried about pdd. Also which threads are costing the most CPU. If conditionals are implemented in windows as waitforsingleobject i have no idea if that's as efficient as it is in linux. You could try forklifting in our modified copy of APR that has been updated to work with their latest release and see if anything has improved in the windows implementations of conditionals. I recall Jeff had to backport in at least one patch for that over the years. On Wed, Aug 20, 2014 at 4:11 PM, Andre < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Ok I’ll try that next. I’ll create xml dialplan to see if that makes a difference. One note, when I didn’t load the deck in my managed code my dial plan did return a 503 in 0 – 20 milliseconds. Not sure if that matters since no bridge was ever called. Thanks again, Andre From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [mailto: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfMichael Jerris Sent: Wednesday, August 20, 2014 5:02 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd I would test this without any of your managed code at all, to confirm that it is not the bottleneck. On Aug 20, 2014, at 4:28 PM, Andre < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: I added the new code but it doesn’t seem to fix it. :/ From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [ <mailto:freeswitch-users-bounces@lists.freeswitch.org> mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Andre Sent: Wednesday, August 20, 2014 3:37 PM To: 'FreeSWITCH Users Help' Subject: Re: [Freeswitch-users] high pdd Thanks I’ll work on it right a way. From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [ <mailto:freeswitch-users-bounces@lists.freeswitch.org> mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Peter Olsson Sent: Wednesday, August 20, 2014 3:30 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd It seems it might be stuck in the condition code for Windows. Try this timing patch for Windows, and see if it makes any difference. I've tested in on lower load, but it would be interesting how it performs in this scenario. <https://jira.freeswitch.org/secure/attachment/20613/windows-waitable-timer-diff.txt> https://jira.freeswitch.org/secure/attachment/20613/windows-waitable-timer-d... 2014-08-20 21:13 GMT+02:00 Andre < <mailto:andretodd@verizon.net> andretodd@verizon.net>: I will, need to get hardware and figure out how to use mono. In the meantime, I’ve attached some of the call traces. <image001.jpg><image002.jpg> <image003.jpg> From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [mailto: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfBrian West Sent: Wednesday, August 20, 2014 3:05 PM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd Try the same thing on Debian 7 if possible. On Wed, Aug 20, 2014 at 12:35 PM, Andre < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Hi, I’m using the master. Should I try 1.4.7? I’ve done a dump after things started to go bad/ backed up. I see 8193 threads in _thread_cond_timedwait. Looks like its on line 81 of thread_cond.c res = WaitForSingleObject(cond->semaphore, timeout_ms); The value of timeout_ms is <tel:2016850392> 2016850392. Not sure if this means anything. Thanks for the support guys. Andre From: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org [mailto: <mailto:freeswitch-users-bounces@lists.freeswitch.org> freeswitch-users-bounces@lists.freeswitch.org] On Behalf OfBrian West Sent: Wednesday, August 20, 2014 11:57 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd 1.2.X is EOL, 1.4.7 is the current release. We have releases and tags in git that no longer carry the 'stable' moniker starting with 1.4. On Sun, Aug 17, 2014 at 1:39 PM, Andre Demattia < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Thanks, I'm using the newest master and the last stable 1.2.23 We are willing to pay to get this fixed. Andre _____ From: <mailto:anthony.minessale@gmail.com> Anthony Minessale Sent: ‎8/‎17/‎2014 12:30 PM To: <mailto:freeswitch-users@lists.freeswitch.org> Freeswitch-users Subject: Re: [Freeswitch-users] high pdd We dont have a lot of tools to troubleshoot windows. You should mention windows in your subject to have a better chance to get the attention of our windows users. You should consider installing debian 7 on the same hardware and perform similar tests to help determine if your issue is o/s specific or not. You also have not indicated the revision of FS. If you have not worked with latest master, you should be also to rule out improvements. On Aug 17, 2014 9:54 AM, "Andre Demattia" < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Yes that's off and I believe all the databases are off too. No authentication or registration is enabled. The sizes of the database are 0 until I call the shutdown command then its about 90k. I have the DBS in Ra drive too. Windows seems pretty good until the 3000 sessions per min mark. Then I just get long pdd and the threads get backed up then no more sessions can be created. Funny thing is I has 10 profiles and the same results. I also tried two consoles with the same results. FreeSWITCH is great. I'd be grateful for help. I've worked on this for 2 years and hate to give up. If any one from <mailto:consulting@freeswitch.org> consulting@freeswitch.org is willing to help I'd appreciate that. Thanks again, Andre _____ From: <mailto:anthony.minessale@gmail.com> Anthony Minessale Sent: ‎8/‎17/‎2014 10:13 AM To: <mailto:freeswitch-users@lists.freeswitch.org> Freeswitch-users Subject: Re: [Freeswitch-users] high pdd Did you comment out manage-presence ? Also we have no real idea how well FS performs in windows under extreme load. On Aug 17, 2014 5:42 AM, "Andre Demattia" < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Thanks Ken, I'll try the consulting again. Andre _____ From: <mailto:krice@freeswitch.org> Ken Rice Sent: ‎8/‎17/‎2014 1:19 AM To: <mailto:freeswitch-users@lists.freeswitch.org> FreeSWITCH Users Help Subject: Re: [Freeswitch-users] high pdd if the last post does not work contact <mailto:consulting@freeswitch.org> consulting@freeswitch.org for pro help from one of the developers... thats about as far as the list will take you Ken Sent from my iPad On Aug 16, 2014, at 15:23, "Andre" < <mailto:andretodd@verizon.net> andretodd@verizon.net> wrote: Going with the idea it’s IO related I did a db_cache status. I just disabled limit in my code by removing it but limit should be HASH not a database however I stills how a database. Below is the results of the db cache status. I’m not real sure what its saying. I should have all registration and limit disabled and should not be using any sql database from my understanding. Anyone know what the results mean? I’ve only tested 2 calls on this status. [The entire original message is not included.] _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net <http://irc.freenode.net> #freeswitch ☞ http://freeswitch.org/g+ ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> ☎ +19193869900

Yes 64 bit. I've even compiled it in visual studio with the optimization. Neither makes a difference. The server has 40 cores and 128 GB memory. Windows server 2012 r2 datacenter. -----Original Message----- From: "Peter Olsson" <peter@olssononline.se> Sent: ‎8/‎17/‎2014 2:30 PM To: "FreeSWITCH Users Help" <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] high pdd Are you using a 64-bit build? 2014-08-17 16:51 GMT+02:00 Andre Demattia <andretodd@verizon.net>:

Yes that's off and I believe all the databases are off too. No authentication or registration is enabled. The sizes of the database are 0 until I call the shutdown command then its about 90k. I have the DBS in Ra drive too.

Windows seems pretty good until the 3000 sessions per min mark. Then I just get long pdd and the threads get backed up then no more sessions can be created.

Funny thing is I has 10 profiles and the same results. I also tried two consoles with the same results.

FreeSWITCH is great.

I'd be grateful for help. I've worked on this for 2 years and hate to give up.

If any one from consulting@freeswitch.org is willing to help I'd appreciate that.

Thanks again, Andre ------------------------------ From: Anthony Minessale <anthony.minessale@gmail.com> Sent: ‎8/‎17/‎2014 10:13 AM To: Freeswitch-users <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] high pdd

Did you comment out manage-presence ? Also we have no real idea how well FS performs in windows under extreme load. On Aug 17, 2014 5:42 AM, "Andre Demattia" <andretodd@verizon.net> wrote:

...

Thanks Ken, I'll try the consulting again. Andre ------------------------------ From: Ken Rice <krice@freeswitch.org> Sent: ‎8/‎17/‎2014 1:19 AM To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] high pdd

if the last post does not work contact consulting@freeswitch.org for pro help from one of the developers... thats about as far as the list will take you

Ken Sent from my iPad

On Aug 16, 2014, at 15:23, "Andre" <andretodd@verizon.net> wrote:

Going with the idea it’s IO related I did a db_cache status. I just disabled limit in my code by removing it but limit should be HASH not a database however I stills how a database.

Below is the results of the db cache status. I’m not real sure what its saying. I should have all registration and limit disabled and should not be using any sql database from my understanding.

Anyone know what the results mean? I’ve only tested 2 calls on this status.

[The entire original message is not included.]

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com

Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com

FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com

FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

[The entire original message is not included.]