Hi, Is there any connection limit perhaps hard coded anywhere in xml_curl module? I am load testing FS with xml_curl to provide config, directory and dial-plan. I observed i start getting HTTP ERROR 0 when i have around 1600 concurrent calls on machine with media. Although i see no sign of system overload either on FS (load average < 1.8) or Apache (load average < 4,75). Both have plenty of RAM and bandwidth free. Is there anyway to get rid of this HTTP ERROR 0? It must be generated by FS internally, since i don't see anything like this error in Apache logs. Below 1,638 concurrent calls @ 50 CPS, i get success rate of 99.999% but as soon as it crosses this mark, success rate drastically drops to 60% and even below (when concurrent calls go beyond 2,000). Here is error console log. ################################################### 2013-06-05 18:09:13.591229 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://172.16.171.61/vm-users.php data: [hostname=vms01§ion=directory&tag_name=domain&key_name=name&key_value=10.1.3.2&Event-Name=GENERAL&Core-UUID=84c917dc-cdde-11e2-9bb2-8b779d6a4e97&FreeSWITCH-Hostname=vms01&FreeSWITCH-Switchname=vms01&FreeSWITCH-IPv4=10.1.3.2&FreeSWITCH-IPv6=%3A%3A1&Event-Date-Local=2013-06-05%2018%3A09%3A03&Event-Date-GMT=Wed,%2005%20Jun%202013%2013%3A09%3A03%20GMT&Event-Date-Timestamp=1370437743591210&Event-Calling-File=mod_voicemail.c&Event-Calling-Function=resolve_id&Event-Calling-Line-Number=1350&Event-Sequence=1776930&action=message-count&key=id&user=989000049598&domain=10.1.3.2] ################################################### Although this setup is more then sufficient for my requirement but i just want to know if i can extend it beyond, since i still have lot of resources free. OS: Debian 6.0.7 x64 Apache: 2.2.16 FreeSWITCH: 1.5.1b+git~20130424T005233Z~66d1755b06 Both FS and Apache are running on separate dedicated machines on same LAN. Thank you. -- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
Is the request going to your server and getting no response? On Jun 5, 2013, at 9:50 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
Hi,
Is there any connection limit perhaps hard coded anywhere in xml_curl module? I am load testing FS with xml_curl to provide config, directory and dial-plan. I observed i start getting HTTP ERROR 0 when i have around 1600 concurrent calls on machine with media. Although i see no sign of system overload either on FS (load average < 1.8) or Apache (load average < 4,75). Both have plenty of RAM and bandwidth free.
Is there anyway to get rid of this HTTP ERROR 0? It must be generated by FS internally, since i don't see anything like this error in Apache logs.
Below 1,638 concurrent calls @ 50 CPS, i get success rate of 99.999% but as soon as it crosses this mark, success rate drastically drops to 60% and even below (when concurrent calls go beyond 2,000). Here is error console log.
################################################### 2013-06-05 18:09:13.591229 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://172.16.171.61/vm-users.php data: [hostname=vms01§ion=directory&tag_name=domain&key_name=name&key_value=10.1.3.2&Event-Name=GENERAL&Core-UUID=84c917dc-cdde-11e2-9bb2-8b779d6a4e97&FreeSWITCH-Hostname=vms01&FreeSWITCH-Switchname=vms01&FreeSWITCH-IPv4=10.1.3.2&FreeSWITCH-IPv6=%3A%3A1&Event-Date-Local=2013-06-05%2018%3A09%3A03&Event-Date-GMT=Wed,%2005%20Jun%202013%2013%3A09%3A03%20GMT&Event-Date-Timestamp=1370437743591210&Event-Calling-File=mod_voicemail.c&Event-Calling-Function=resolve_id&Event-Calling-Line-Number=1350&Event-Sequence=1776930&action=message-count&key=id&user=989000049598&domain=10.1.3.2] ###################################################
Although this setup is more then sufficient for my requirement but i just want to know if i can extend it beyond, since i still have lot of resources free.
OS: Debian 6.0.7 x64 Apache: 2.2.16 FreeSWITCH: 1.5.1b+git~20130424T005233Z~66d1755b06
Both FS and Apache are running on separate dedicated machines on same LAN.
Thank you.
The request does not comes to web server, at least this is what i see in web server logs. I will run a trace and see if it actually is sent out to web server or not. Thank you. On Wed, Jun 5, 2013 at 4:06 PM, Michael Jerris <mike@jerris.com> wrote:
Is the request going to your server and getting no response?
On Jun 5, 2013, at 9:50 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
Hi,
Is there any connection limit perhaps hard coded anywhere in xml_curl module? I am load testing FS with xml_curl to provide config, directory and dial-plan. I observed i start getting HTTP ERROR 0 when i have around 1600 concurrent calls on machine with media. Although i see no sign of system overload either on FS (load average < 1.8) or Apache (load average < 4,75). Both have plenty of RAM and bandwidth free.
Is there anyway to get rid of this HTTP ERROR 0? It must be generated by FS internally, since i don't see anything like this error in Apache logs.
Below 1,638 concurrent calls @ 50 CPS, i get success rate of 99.999% but as soon as it crosses this mark, success rate drastically drops to 60% and even below (when concurrent calls go beyond 2,000). Here is error console log.
################################################### 2013-06-05 18:09:13.591229 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://172.16.171.61/vm-users.php data: [hostname=vms01§ion=directory&tag_name=domain&key_name=name&key_value=10.1.3.2&Event-Name=GENERAL&Core-UUID=84c917dc-cdde-11e2-9bb2-8b779d6a4e97&FreeSWITCH-Hostname=vms01&FreeSWITCH-Switchname=vms01&FreeSWITCH-IPv4=10.1.3.2&FreeSWITCH-IPv6=%3A%3A1&Event-Date-Local=2013-06-05%2018%3A09%3A03&Event-Date-GMT=Wed,%2005%20Jun%202013%2013%3A09%3A03%20GMT&Event-Date-Timestamp=1370437743591210&Event-Calling-File=mod_voicemail.c&Event-Calling-Function=resolve_id&Event-Calling-Line-Number=1350&Event-Sequence=1776930&action=message-count&key=id&user=989000049598&domain=10.1.3.2] ###################################################
Although this setup is more then sufficient for my requirement but i just want to know if i can extend it beyond, since i still have lot of resources free.
OS: Debian 6.0.7 x64 Apache: 2.2.16 FreeSWITCH: 1.5.1b+git~20130424T005233Z~66d1755b06
Both FS and Apache are running on separate dedicated machines on same LAN.
Thank you.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
Did you get to confirm if the request was really going to the server or not? If it is not, then all of the other suggestions above are completely invalid. On Jun 5, 2013, at 10:16 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
The request does not comes to web server, at least this is what i see in web server logs. I will run a trace and see if it actually is sent out to web server or not.
Thank you.
On Wed, Jun 5, 2013 at 4:06 PM, Michael Jerris <mike@jerris.com> wrote: Is the request going to your server and getting no response?
On Jun 5, 2013, at 9:50 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
Hi,
Is there any connection limit perhaps hard coded anywhere in xml_curl module? I am load testing FS with xml_curl to provide config, directory and dial-plan. I observed i start getting HTTP ERROR 0 when i have around 1600 concurrent calls on machine with media. Although i see no sign of system overload either on FS (load average < 1.8) or Apache (load average < 4,75). Both have plenty of RAM and bandwidth free.
Is there anyway to get rid of this HTTP ERROR 0? It must be generated by FS internally, since i don't see anything like this error in Apache logs.
Below 1,638 concurrent calls @ 50 CPS, i get success rate of 99.999% but as soon as it crosses this mark, success rate drastically drops to 60% and even below (when concurrent calls go beyond 2,000). Here is error console log.
################################################### 2013-06-05 18:09:13.591229 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://172.16.171.61/vm-users.php data: [hostname=vms01§ion=directory&tag_name=domain&key_name=name&key_value=10.1.3.2&Event-Name=GENERAL&Core-UUID=84c917dc-cdde-11e2-9bb2-8b779d6a4e97&FreeSWITCH-Hostname=vms01&FreeSWITCH-Switchname=vms01&FreeSWITCH-IPv4=10.1.3.2&FreeSWITCH-IPv6=%3A%3A1&Event-Date-Local=2013-06-05%2018%3A09%3A03&Event-Date-GMT=Wed,%2005%20Jun%202013%2013%3A09%3A03%20GMT&Event-Date-Timestamp=1370437743591210&Event-Calling-File=mod_voicemail.c&Event-Calling-Function=resolve_id&Event-Calling-Line-Number=1350&Event-Sequence=1776930&action=message-count&key=id&user=989000049598&domain=10.1.3.2] ###################################################
Although this setup is more then sufficient for my requirement but i just want to know if i can extend it beyond, since i still have lot of resources free.
OS: Debian 6.0.7 x64 Apache: 2.2.16 FreeSWITCH: 1.5.1b+git~20130424T005233Z~66d1755b06
Both FS and Apache are running on separate dedicated machines on same LAN.
Thank you.
Ah sorry, I had assumed this was already confirmed - good catch Michael. As said above, if you have not confirmed this already, then you will need to do so before moving on. Cal On Wed, Jun 5, 2013 at 7:57 PM, Michael Jerris <mike@jerris.com> wrote:
Did you get to confirm if the request was really going to the server or not? If it is not, then all of the other suggestions above are completely invalid.
On Jun 5, 2013, at 10:16 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
The request does not comes to web server, at least this is what i see in web server logs. I will run a trace and see if it actually is sent out to web server or not.
Thank you.
On Wed, Jun 5, 2013 at 4:06 PM, Michael Jerris <mike@jerris.com> wrote:
Is the request going to your server and getting no response?
On Jun 5, 2013, at 9:50 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
Hi,
Is there any connection limit perhaps hard coded anywhere in xml_curl module? I am load testing FS with xml_curl to provide config, directory and dial-plan. I observed i start getting HTTP ERROR 0 when i have around 1600 concurrent calls on machine with media. Although i see no sign of system overload either on FS (load average < 1.8) or Apache (load average < 4,75). Both have plenty of RAM and bandwidth free.
Is there anyway to get rid of this HTTP ERROR 0? It must be generated by FS internally, since i don't see anything like this error in Apache logs.
Below 1,638 concurrent calls @ 50 CPS, i get success rate of 99.999% but as soon as it crosses this mark, success rate drastically drops to 60% and even below (when concurrent calls go beyond 2,000). Here is error console log.
################################################### 2013-06-05 18:09:13.591229 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://172.16.171.61/vm-users.php data: [hostname=vms01§ion=directory&tag_name=domain&key_name=name&key_value=10.1.3.2&Event-Name=GENERAL&Core-UUID=84c917dc-cdde-11e2-9bb2-8b779d6a4e97&FreeSWITCH-Hostname=vms01&FreeSWITCH-Switchname=vms01&FreeSWITCH-IPv4=10.1.3.2&FreeSWITCH-IPv6=%3A%3A1&Event-Date-Local=2013-06-05%2018%3A09%3A03&Event-Date-GMT=Wed,%2005%20Jun%202013%2013%3A09%3A03%20GMT&Event-Date-Timestamp=1370437743591210&Event-Calling-File=mod_voicemail.c&Event-Calling-Function=resolve_id&Event-Calling-Line-Number=1350&Event-Sequence=1776930&action=message-count&key=id&user=989000049598&domain=10.1.3.2] ###################################################
Although this setup is more then sufficient for my requirement but i just want to know if i can extend it beyond, since i still have lot of resources free.
OS: Debian 6.0.7 x64 Apache: 2.2.16 FreeSWITCH: 1.5.1b+git~20130424T005233Z~66d1755b06
Both FS and Apache are running on separate dedicated machines on same LAN.
Thank you.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I have been trying for about 3 hours now but still not sure what is happening. In SIPp, there are new calls created and old ones hang up at every instant, so i see HTTP Error 0 when calls climb over 1638 and then immediately when some calls hang up everything starts working normal again, then SIPp generates more call and again i get this error. Its pretty difficult to identify if failed hit is in the trace or not. I will debug more tomorrow. Thank you. On Wed, Jun 5, 2013 at 9:04 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
Ah sorry, I had assumed this was already confirmed - good catch Michael.
As said above, if you have not confirmed this already, then you will need to do so before moving on.
Cal
On Wed, Jun 5, 2013 at 7:57 PM, Michael Jerris <mike@jerris.com> wrote:
Did you get to confirm if the request was really going to the server or not? If it is not, then all of the other suggestions above are completely invalid.
On Jun 5, 2013, at 10:16 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
The request does not comes to web server, at least this is what i see in web server logs. I will run a trace and see if it actually is sent out to web server or not.
Thank you.
On Wed, Jun 5, 2013 at 4:06 PM, Michael Jerris <mike@jerris.com> wrote:
Is the request going to your server and getting no response?
On Jun 5, 2013, at 9:50 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
Hi,
Is there any connection limit perhaps hard coded anywhere in xml_curl module? I am load testing FS with xml_curl to provide config, directory and dial-plan. I observed i start getting HTTP ERROR 0 when i have around 1600 concurrent calls on machine with media. Although i see no sign of system overload either on FS (load average < 1.8) or Apache (load average < 4,75). Both have plenty of RAM and bandwidth free.
Is there anyway to get rid of this HTTP ERROR 0? It must be generated by FS internally, since i don't see anything like this error in Apache logs.
Below 1,638 concurrent calls @ 50 CPS, i get success rate of 99.999% but as soon as it crosses this mark, success rate drastically drops to 60% and even below (when concurrent calls go beyond 2,000). Here is error console log.
################################################### 2013-06-05 18:09:13.591229 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://172.16.171.61/vm-users.php data: [hostname=vms01§ion=directory&tag_name=domain&key_name=name&key_value=10.1.3.2&Event-Name=GENERAL&Core-UUID=84c917dc-cdde-11e2-9bb2-8b779d6a4e97&FreeSWITCH-Hostname=vms01&FreeSWITCH-Switchname=vms01&FreeSWITCH-IPv4=10.1.3.2&FreeSWITCH-IPv6=%3A%3A1&Event-Date-Local=2013-06-05%2018%3A09%3A03&Event-Date-GMT=Wed,%2005%20Jun%202013%2013%3A09%3A03%20GMT&Event-Date-Timestamp=1370437743591210&Event-Calling-File=mod_voicemail.c&Event-Calling-Function=resolve_id&Event-Calling-Line-Number=1350&Event-Sequence=1776930&action=message-count&key=id&user=989000049598&domain=10.1.3.2] ###################################################
Although this setup is more then sufficient for my requirement but i just want to know if i can extend it beyond, since i still have lot of resources free.
OS: Debian 6.0.7 x64 Apache: 2.2.16 FreeSWITCH: 1.5.1b+git~20130424T005233Z~66d1755b06
Both FS and Apache are running on separate dedicated machines on same LAN.
Thank you.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
I found this, which may indicate the error is due to CURL timeout. http://jira.freeswitch.org/browse/FS-428?page=com.atlassian.jira.plugin.syst... Thank you. On Wed, Jun 5, 2013 at 11:21 PM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
I have been trying for about 3 hours now but still not sure what is happening. In SIPp, there are new calls created and old ones hang up at every instant, so i see HTTP Error 0 when calls climb over 1638 and then immediately when some calls hang up everything starts working normal again, then SIPp generates more call and again i get this error. Its pretty difficult to identify if failed hit is in the trace or not. I will debug more tomorrow.
Thank you.
On Wed, Jun 5, 2013 at 9:04 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
Ah sorry, I had assumed this was already confirmed - good catch Michael.
As said above, if you have not confirmed this already, then you will need to do so before moving on.
Cal
On Wed, Jun 5, 2013 at 7:57 PM, Michael Jerris <mike@jerris.com> wrote:
Did you get to confirm if the request was really going to the server or not? If it is not, then all of the other suggestions above are completely invalid.
On Jun 5, 2013, at 10:16 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
The request does not comes to web server, at least this is what i see in web server logs. I will run a trace and see if it actually is sent out to web server or not.
Thank you.
On Wed, Jun 5, 2013 at 4:06 PM, Michael Jerris <mike@jerris.com> wrote:
Is the request going to your server and getting no response?
On Jun 5, 2013, at 9:50 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
Hi,
Is there any connection limit perhaps hard coded anywhere in xml_curl module? I am load testing FS with xml_curl to provide config, directory and dial-plan. I observed i start getting HTTP ERROR 0 when i have around 1600 concurrent calls on machine with media. Although i see no sign of system overload either on FS (load average < 1.8) or Apache (load average < 4,75). Both have plenty of RAM and bandwidth free.
Is there anyway to get rid of this HTTP ERROR 0? It must be generated by FS internally, since i don't see anything like this error in Apache logs.
Below 1,638 concurrent calls @ 50 CPS, i get success rate of 99.999% but as soon as it crosses this mark, success rate drastically drops to 60% and even below (when concurrent calls go beyond 2,000). Here is error console log.
################################################### 2013-06-05 18:09:13.591229 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://172.16.171.61/vm-users.php data: [hostname=vms01§ion=directory&tag_name=domain&key_name=name&key_value=10.1.3.2&Event-Name=GENERAL&Core-UUID=84c917dc-cdde-11e2-9bb2-8b779d6a4e97&FreeSWITCH-Hostname=vms01&FreeSWITCH-Switchname=vms01&FreeSWITCH-IPv4=10.1.3.2&FreeSWITCH-IPv6=%3A%3A1&Event-Date-Local=2013-06-05%2018%3A09%3A03&Event-Date-GMT=Wed,%2005%20Jun%202013%2013%3A09%3A03%20GMT&Event-Date-Timestamp=1370437743591210&Event-Calling-File=mod_voicemail.c&Event-Calling-Function=resolve_id&Event-Calling-Line-Number=1350&Event-Sequence=1776930&action=message-count&key=id&user=989000049598&domain=10.1.3.2] ###################################################
Although this setup is more then sufficient for my requirement but i just want to know if i can extend it beyond, since i still have lot of resources free.
OS: Debian 6.0.7 x64 Apache: 2.2.16 FreeSWITCH: 1.5.1b+git~20130424T005233Z~66d1755b06
Both FS and Apache are running on separate dedicated machines on same LAN.
Thank you.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
This is adding a feature to allow timeout to be set in ms instead of seconds. This is not a problem, its a feature, that was added a while ago. Until you can determine if the request is being sent for the one thats failing, your not going to solve this issue. Mike On Jun 5, 2013, at 6:15 PM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
I found this, which may indicate the error is due to CURL timeout.
http://jira.freeswitch.org/browse/FS-428?page=com.atlassian.jira.plugin.syst...
Thank you.
On Wed, Jun 5, 2013 at 11:21 PM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote: I have been trying for about 3 hours now but still not sure what is happening. In SIPp, there are new calls created and old ones hang up at every instant, so i see HTTP Error 0 when calls climb over 1638 and then immediately when some calls hang up everything starts working normal again, then SIPp generates more call and again i get this error. Its pretty difficult to identify if failed hit is in the trace or not. I will debug more tomorrow.
Thank you.
On Wed, Jun 5, 2013 at 9:04 PM, Cal Leeming [Simplicity Media Ltd] <cal.leeming@simplicitymedialtd.co.uk> wrote: Ah sorry, I had assumed this was already confirmed - good catch Michael.
As said above, if you have not confirmed this already, then you will need to do so before moving on.
Cal
On Wed, Jun 5, 2013 at 7:57 PM, Michael Jerris <mike@jerris.com> wrote: Did you get to confirm if the request was really going to the server or not? If it is not, then all of the other suggestions above are completely invalid.
On Jun 5, 2013, at 10:16 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
The request does not comes to web server, at least this is what i see in web server logs. I will run a trace and see if it actually is sent out to web server or not.
Thank you.
On Wed, Jun 5, 2013 at 4:06 PM, Michael Jerris <mike@jerris.com> wrote: Is the request going to your server and getting no response?
On Jun 5, 2013, at 9:50 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
Hi,
Is there any connection limit perhaps hard coded anywhere in xml_curl module? I am load testing FS with xml_curl to provide config, directory and dial-plan. I observed i start getting HTTP ERROR 0 when i have around 1600 concurrent calls on machine with media. Although i see no sign of system overload either on FS (load average < 1.8) or Apache (load average < 4,75). Both have plenty of RAM and bandwidth free.
Is there anyway to get rid of this HTTP ERROR 0? It must be generated by FS internally, since i don't see anything like this error in Apache logs.
Below 1,638 concurrent calls @ 50 CPS, i get success rate of 99.999% but as soon as it crosses this mark, success rate drastically drops to 60% and even below (when concurrent calls go beyond 2,000). Here is error console log.
################################################### 2013-06-05 18:09:13.591229 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://172.16.171.61/vm-users.php data: [hostname=vms01§ion=directory&tag_name=domain&key_name=name&key_value=10.1.3.2&Event-Name=GENERAL&Core-UUID=84c917dc-cdde-11e2-9bb2-8b779d6a4e97&FreeSWITCH-Hostname=vms01&FreeSWITCH-Switchname=vms01&FreeSWITCH-IPv4=10.1.3.2&FreeSWITCH-IPv6=%3A%3A1&Event-Date-Local=2013-06-05%2018%3A09%3A03&Event-Date-GMT=Wed,%2005%20Jun%202013%2013%3A09%3A03%20GMT&Event-Date-Timestamp=1370437743591210&Event-Calling-File=mod_voicemail.c&Event-Calling-Function=resolve_id&Event-Calling-Line-Number=1350&Event-Sequence=1776930&action=message-count&key=id&user=989000049598&domain=10.1.3.2] ###################################################
Although this setup is more then sufficient for my requirement but i just want to know if i can extend it beyond, since i still have lot of resources free.
OS: Debian 6.0.7 x64 Apache: 2.2.16 FreeSWITCH: 1.5.1b+git~20130424T005233Z~66d1755b06
Both FS and Apache are running on separate dedicated machines on same LAN.
Thank you.
Yes, its a feature, i know. I was just looking at how this feature was tested, as mentioned in its logs, 2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 Processing 21653XXXXX->121637XXXXX in context termination 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://192.168.0.2/gateway.php 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml Thank you. On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike@jerris.com> wrote:
This is adding a feature to allow timeout to be set in ms instead of seconds. This is not a problem, its a feature, that was added a while ago. Until you can determine if the request is being sent for the one thats failing, your not going to solve this issue.
Mike
On Jun 5, 2013, at 6:15 PM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
I found this, which may indicate the error is due to CURL timeout.
http://jira.freeswitch.org/browse/FS-428?page=com.atlassian.jira.plugin.syst...
Thank you.
On Wed, Jun 5, 2013 at 11:21 PM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
I have been trying for about 3 hours now but still not sure what is happening. In SIPp, there are new calls created and old ones hang up at every instant, so i see HTTP Error 0 when calls climb over 1638 and then immediately when some calls hang up everything starts working normal again, then SIPp generates more call and again i get this error. Its pretty difficult to identify if failed hit is in the trace or not. I will debug more tomorrow.
Thank you.
On Wed, Jun 5, 2013 at 9:04 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
Ah sorry, I had assumed this was already confirmed - good catch Michael.
As said above, if you have not confirmed this already, then you will need to do so before moving on.
Cal
On Wed, Jun 5, 2013 at 7:57 PM, Michael Jerris <mike@jerris.com> wrote:
Did you get to confirm if the request was really going to the server or not? If it is not, then all of the other suggestions above are completely invalid.
On Jun 5, 2013, at 10:16 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
The request does not comes to web server, at least this is what i see in web server logs. I will run a trace and see if it actually is sent out to web server or not.
Thank you.
On Wed, Jun 5, 2013 at 4:06 PM, Michael Jerris <mike@jerris.com> wrote:
Is the request going to your server and getting no response?
On Jun 5, 2013, at 9:50 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
Hi,
Is there any connection limit perhaps hard coded anywhere in xml_curl module? I am load testing FS with xml_curl to provide config, directory and dial-plan. I observed i start getting HTTP ERROR 0 when i have around 1600 concurrent calls on machine with media. Although i see no sign of system overload either on FS (load average < 1.8) or Apache (load average < 4,75). Both have plenty of RAM and bandwidth free.
Is there anyway to get rid of this HTTP ERROR 0? It must be generated by FS internally, since i don't see anything like this error in Apache logs.
Below 1,638 concurrent calls @ 50 CPS, i get success rate of 99.999% but as soon as it crosses this mark, success rate drastically drops to 60% and even below (when concurrent calls go beyond 2,000). Here is error console log.
################################################### 2013-06-05 18:09:13.591229 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://172.16.171.61/vm-users.php data: [hostname=vms01§ion=directory&tag_name=domain&key_name=name&key_value=10.1.3.2&Event-Name=GENERAL&Core-UUID=84c917dc-cdde-11e2-9bb2-8b779d6a4e97&FreeSWITCH-Hostname=vms01&FreeSWITCH-Switchname=vms01&FreeSWITCH-IPv4=10.1.3.2&FreeSWITCH-IPv6=%3A%3A1&Event-Date-Local=2013-06-05%2018%3A09%3A03&Event-Date-GMT=Wed,%2005%20Jun%202013%2013%3A09%3A03%20GMT&Event-Date-Timestamp=1370437743591210&Event-Calling-File=mod_voicemail.c&Event-Calling-Function=resolve_id&Event-Calling-Line-Number=1350&Event-Sequence=1776930&action=message-count&key=id&user=989000049598&domain=10.1.3.2] ###################################################
Although this setup is more then sufficient for my requirement but i just want to know if i can extend it beyond, since i still have lot of resources free.
OS: Debian 6.0.7 x64 Apache: 2.2.16 FreeSWITCH: 1.5.1b+git~20130424T005233Z~66d1755b06
Both FS and Apache are running on separate dedicated machines on same LAN.
Thank you.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
Guy, after detailed analysis i can confirm it was an Apache issue, and changing Apache configuration changes the concurrent call limit. However, one thing i observed about xml curl is that it makes several request with same parameters per call. Even if we have no calls running on server, as well as as Apache is completely free. I can see FS makes at least 1). 3 hits for dialplan. 2). 2 hits for directory. 3). 4 hits for voicemail. 4). 3 hits for ivr. 5). 2 hits for sofia. and so on.. While ideally only one hit should be sufficient to get all these configurations. What is the point of making all these redundant hits? We have roughly 15 - 20 hits made per call, which is huge and causes significant load on Apache. Both Apache and FS are running on dedicated machines and connected over Gigabit Ethernet with zero packet loss, so i doubt these hits are "re-transmission". Thank you. On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
Yes, its a feature, i know. I was just looking at how this feature was tested, as mentioned in its logs,
2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 Processing 21653XXXXX->121637XXXXX in context termination 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://192.168.0.2/gateway.php 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml
Thank you.
On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike@jerris.com> wrote:
This is adding a feature to allow timeout to be set in ms instead of seconds. This is not a problem, its a feature, that was added a while ago. Until you can determine if the request is being sent for the one thats failing, your not going to solve this issue.
Mike
On Jun 5, 2013, at 6:15 PM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
I found this, which may indicate the error is due to CURL timeout.
http://jira.freeswitch.org/browse/FS-428?page=com.atlassian.jira.plugin.syst...
Thank you.
On Wed, Jun 5, 2013 at 11:21 PM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
I have been trying for about 3 hours now but still not sure what is happening. In SIPp, there are new calls created and old ones hang up at every instant, so i see HTTP Error 0 when calls climb over 1638 and then immediately when some calls hang up everything starts working normal again, then SIPp generates more call and again i get this error. Its pretty difficult to identify if failed hit is in the trace or not. I will debug more tomorrow.
Thank you.
On Wed, Jun 5, 2013 at 9:04 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
Ah sorry, I had assumed this was already confirmed - good catch Michael.
As said above, if you have not confirmed this already, then you will need to do so before moving on.
Cal
On Wed, Jun 5, 2013 at 7:57 PM, Michael Jerris <mike@jerris.com> wrote:
Did you get to confirm if the request was really going to the server or not? If it is not, then all of the other suggestions above are completely invalid.
On Jun 5, 2013, at 10:16 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
The request does not comes to web server, at least this is what i see in web server logs. I will run a trace and see if it actually is sent out to web server or not.
Thank you.
On Wed, Jun 5, 2013 at 4:06 PM, Michael Jerris <mike@jerris.com>wrote:
Is the request going to your server and getting no response?
On Jun 5, 2013, at 9:50 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
Hi,
Is there any connection limit perhaps hard coded anywhere in xml_curl module? I am load testing FS with xml_curl to provide config, directory and dial-plan. I observed i start getting HTTP ERROR 0 when i have around 1600 concurrent calls on machine with media. Although i see no sign of system overload either on FS (load average < 1.8) or Apache (load average < 4,75). Both have plenty of RAM and bandwidth free.
Is there anyway to get rid of this HTTP ERROR 0? It must be generated by FS internally, since i don't see anything like this error in Apache logs.
Below 1,638 concurrent calls @ 50 CPS, i get success rate of 99.999% but as soon as it crosses this mark, success rate drastically drops to 60% and even below (when concurrent calls go beyond 2,000). Here is error console log.
################################################### 2013-06-05 18:09:13.591229 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://172.16.171.61/vm-users.php data: [hostname=vms01§ion=directory&tag_name=domain&key_name=name&key_value=10.1.3.2&Event-Name=GENERAL&Core-UUID=84c917dc-cdde-11e2-9bb2-8b779d6a4e97&FreeSWITCH-Hostname=vms01&FreeSWITCH-Switchname=vms01&FreeSWITCH-IPv4=10.1.3.2&FreeSWITCH-IPv6=%3A%3A1&Event-Date-Local=2013-06-05%2018%3A09%3A03&Event-Date-GMT=Wed,%2005%20Jun%202013%2013%3A09%3A03%20GMT&Event-Date-Timestamp=1370437743591210&Event-Calling-File=mod_voicemail.c&Event-Calling-Function=resolve_id&Event-Calling-Line-Number=1350&Event-Sequence=1776930&action=message-count&key=id&user=989000049598&domain=10.1.3.2] ###################################################
Although this setup is more then sufficient for my requirement but i just want to know if i can extend it beyond, since i still have lot of resources free.
OS: Debian 6.0.7 x64 Apache: 2.2.16 FreeSWITCH: 1.5.1b+git~20130424T005233Z~66d1755b06
Both FS and Apache are running on separate dedicated machines on same LAN.
Thank you.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
I would need to see the specifics of your dialplan, but I suspect you have some transfers or execute extension that are causing it to re-enter dialplan. Are you sure on all the others that they are making the exact same requests? Details here will be necessary to understand exactly what is going on.. Mike On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
Guy, after detailed analysis i can confirm it was an Apache issue, and changing Apache configuration changes the concurrent call limit. However, one thing i observed about xml curl is that it makes several request with same parameters per call. Even if we have no calls running on server, as well as as Apache is completely free. I can see FS makes at least
1). 3 hits for dialplan. 2). 2 hits for directory. 3). 4 hits for voicemail. 4). 3 hits for ivr. 5). 2 hits for sofia. and so on..
While ideally only one hit should be sufficient to get all these configurations. What is the point of making all these redundant hits? We have roughly 15 - 20 hits made per call, which is huge and causes significant load on Apache.
Both Apache and FS are running on dedicated machines and connected over Gigabit Ethernet with zero packet loss, so i doubt these hits are "re-transmission".
Thank you.
On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote: Yes, its a feature, i know. I was just looking at how this feature was tested, as mentioned in its logs,
2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 Processing 21653XXXXX->121637XXXXX in context termination 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://192.168.0.2/gateway.php 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml
Thank you.
On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike@jerris.com> wrote: This is adding a feature to allow timeout to be set in ms instead of seconds. This is not a problem, its a feature, that was added a while ago. Until you can determine if the request is being sent for the one thats failing, your not going to solve this issue.
Mike
You are right. After many improvements and modifications to dail plan, removing unnecessary modules and using Ngnix web server, things are much better now. Except that directory information is fetched 3 times per call and there are overall 4 HTTP hits in a single call (3 for user + 1 for dial plan). Since Monday, we have started analyzing the results of load test, specially looking into packet traces, voicemail recordings and so on. We observed a strange thing, although we were able to test 1536 concurrent calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls per second rate. But there is strange and unexplained delay in starting RTP from FS side. FS plays an IVR in early media state, then answers the call to directly start recording voicemail. The problem is if call rate is under 25 CPS, everything works fine. At 25 CPS there is minor delay in starting RTP in early media state. This delay is negligible after processing few hundred thousand calls but after about 1.2 million calls the delay has creeps up to about 1.2 seconds. After 30 CPS this delay is significantly increases to 2 second after around 400 calls had done. At 50 CPS its ~3 second and after 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in starting playing IVR in early media state, once the IVR starts playing everything works perfectly fine. No call fails, but this delay would be annoying for anyone using the servers with over 30 CPS. Just to let you know, we checked everything, system resources, bandwidth usage, IVR file formats etc. all seems to be fine. We even tested with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing seems to have any impact on the results. I have run of out of possible explanations, so calling you guys for help. Thank you. On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <mike@jerris.com> wrote:
I would need to see the specifics of your dialplan, but I suspect you have some transfers or execute extension that are causing it to re-enter dialplan. Are you sure on all the others that they are making the exact same requests? Details here will be necessary to understand exactly what is going on..
Mike
On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
Guy, after detailed analysis i can confirm it was an Apache issue, and changing Apache configuration changes the concurrent call limit. However, one thing i observed about xml curl is that it makes several request with same parameters per call. Even if we have no calls running on server, as well as as Apache is completely free. I can see FS makes at least
1). 3 hits for dialplan. 2). 2 hits for directory. 3). 4 hits for voicemail. 4). 3 hits for ivr. 5). 2 hits for sofia. and so on..
While ideally only one hit should be sufficient to get all these configurations. What is the point of making all these redundant hits? We have roughly 15 - 20 hits made per call, which is huge and causes significant load on Apache.
Both Apache and FS are running on dedicated machines and connected over Gigabit Ethernet with zero packet loss, so i doubt these hits are "re-transmission".
Thank you.
On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
Yes, its a feature, i know. I was just looking at how this feature was tested, as mentioned in its logs,
2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 Processing 21653XXXXX->121637XXXXX in context termination 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://192.168.0.2/gateway.php 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml
Thank you.
On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike@jerris.com> wrote:
This is adding a feature to allow timeout to be set in ms instead of seconds. This is not a problem, its a feature, that was added a while ago. Until you can determine if the request is being sent for the one thats failing, your not going to solve this issue.
Mike
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
Have you looked at how reliable your system's clock is? http://wiki.freeswitch.org/wiki/Mod_commands#time_test http://wiki.freeswitch.org/wiki/Mod_commands#timer_test<http://wiki.freeswitch.org/wiki/Mod_commands#time_test> -Steve On 12 June 2013 10:21, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
You are right. After many improvements and modifications to dail plan, removing unnecessary modules and using Ngnix web server, things are much better now. Except that directory information is fetched 3 times per call and there are overall 4 HTTP hits in a single call (3 for user + 1 for dial plan).
Since Monday, we have started analyzing the results of load test, specially looking into packet traces, voicemail recordings and so on. We observed a strange thing, although we were able to test 1536 concurrent calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls per second rate. But there is strange and unexplained delay in starting RTP from FS side.
FS plays an IVR in early media state, then answers the call to directly start recording voicemail. The problem is if call rate is under 25 CPS, everything works fine. At 25 CPS there is minor delay in starting RTP in early media state. This delay is negligible after processing few hundred thousand calls but after about 1.2 million calls the delay has creeps up to about 1.2 seconds. After 30 CPS this delay is significantly increases to 2 second after around 400 calls had done. At 50 CPS its ~3 second and after 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in starting playing IVR in early media state, once the IVR starts playing everything works perfectly fine. No call fails, but this delay would be annoying for anyone using the servers with over 30 CPS.
Just to let you know, we checked everything, system resources, bandwidth usage, IVR file formats etc. all seems to be fine. We even tested with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing seems to have any impact on the results.
I have run of out of possible explanations, so calling you guys for help.
Thank you.
On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <mike@jerris.com> wrote:
I would need to see the specifics of your dialplan, but I suspect you have some transfers or execute extension that are causing it to re-enter dialplan. Are you sure on all the others that they are making the exact same requests? Details here will be necessary to understand exactly what is going on..
Mike
On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
Guy, after detailed analysis i can confirm it was an Apache issue, and changing Apache configuration changes the concurrent call limit. However, one thing i observed about xml curl is that it makes several request with same parameters per call. Even if we have no calls running on server, as well as as Apache is completely free. I can see FS makes at least
1). 3 hits for dialplan. 2). 2 hits for directory. 3). 4 hits for voicemail. 4). 3 hits for ivr. 5). 2 hits for sofia. and so on..
While ideally only one hit should be sufficient to get all these configurations. What is the point of making all these redundant hits? We have roughly 15 - 20 hits made per call, which is huge and causes significant load on Apache.
Both Apache and FS are running on dedicated machines and connected over Gigabit Ethernet with zero packet loss, so i doubt these hits are "re-transmission".
Thank you.
On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
Yes, its a feature, i know. I was just looking at how this feature was tested, as mentioned in its logs,
2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 Processing 21653XXXXX->121637XXXXX in context termination 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://192.168.0.2/gateway.php 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml
Thank you.
On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike@jerris.com> wrote:
This is adding a feature to allow timeout to be set in ms instead of seconds. This is not a problem, its a feature, that was added a while ago. Until you can determine if the request is being sent for the one thats failing, your not going to solve this issue.
Mike
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
This sounds quite interesting.. From my own benchmarks, I've seen that IVRs applications can be quite stressful on the server. Have you checked to see if there is a CPU util spike or context switching spike during these tests? Perhaps you are hitting the floor limit of what your CPU is capable of? Do you have CPU graphs? If not, can you enable them and try again? Cal On Wed, Jun 12, 2013 at 10:21 AM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
You are right. After many improvements and modifications to dail plan, removing unnecessary modules and using Ngnix web server, things are much better now. Except that directory information is fetched 3 times per call and there are overall 4 HTTP hits in a single call (3 for user + 1 for dial plan).
Since Monday, we have started analyzing the results of load test, specially looking into packet traces, voicemail recordings and so on. We observed a strange thing, although we were able to test 1536 concurrent calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls per second rate. But there is strange and unexplained delay in starting RTP from FS side.
FS plays an IVR in early media state, then answers the call to directly start recording voicemail. The problem is if call rate is under 25 CPS, everything works fine. At 25 CPS there is minor delay in starting RTP in early media state. This delay is negligible after processing few hundred thousand calls but after about 1.2 million calls the delay has creeps up to about 1.2 seconds. After 30 CPS this delay is significantly increases to 2 second after around 400 calls had done. At 50 CPS its ~3 second and after 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in starting playing IVR in early media state, once the IVR starts playing everything works perfectly fine. No call fails, but this delay would be annoying for anyone using the servers with over 30 CPS.
Just to let you know, we checked everything, system resources, bandwidth usage, IVR file formats etc. all seems to be fine. We even tested with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing seems to have any impact on the results.
I have run of out of possible explanations, so calling you guys for help.
Thank you.
On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <mike@jerris.com> wrote:
I would need to see the specifics of your dialplan, but I suspect you have some transfers or execute extension that are causing it to re-enter dialplan. Are you sure on all the others that they are making the exact same requests? Details here will be necessary to understand exactly what is going on..
Mike
On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
Guy, after detailed analysis i can confirm it was an Apache issue, and changing Apache configuration changes the concurrent call limit. However, one thing i observed about xml curl is that it makes several request with same parameters per call. Even if we have no calls running on server, as well as as Apache is completely free. I can see FS makes at least
1). 3 hits for dialplan. 2). 2 hits for directory. 3). 4 hits for voicemail. 4). 3 hits for ivr. 5). 2 hits for sofia. and so on..
While ideally only one hit should be sufficient to get all these configurations. What is the point of making all these redundant hits? We have roughly 15 - 20 hits made per call, which is huge and causes significant load on Apache.
Both Apache and FS are running on dedicated machines and connected over Gigabit Ethernet with zero packet loss, so i doubt these hits are "re-transmission".
Thank you.
On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
Yes, its a feature, i know. I was just looking at how this feature was tested, as mentioned in its logs,
2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 Processing 21653XXXXX->121637XXXXX in context termination 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://192.168.0.2/gateway.php 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml
Thank you.
On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike@jerris.com> wrote:
This is adding a feature to allow timeout to be set in ms instead of seconds. This is not a problem, its a feature, that was added a while ago. Until you can determine if the request is being sent for the one thats failing, your not going to solve this issue.
Mike
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Steven, i am using soft timer. Should i try another? Please suggest. Cal, no there is nothing unusually changes in RAM or CPU. BTW, i have 24 CPU cores with 16 GB of RAM on each server. Regarding CPU graph, how can i enable it. Thank you. On Wed, Jun 12, 2013 at 1:15 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
This sounds quite interesting.. From my own benchmarks, I've seen that IVRs applications can be quite stressful on the server.
Have you checked to see if there is a CPU util spike or context switching spike during these tests? Perhaps you are hitting the floor limit of what your CPU is capable of?
Do you have CPU graphs? If not, can you enable them and try again?
Cal
On Wed, Jun 12, 2013 at 10:21 AM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
You are right. After many improvements and modifications to dail plan, removing unnecessary modules and using Ngnix web server, things are much better now. Except that directory information is fetched 3 times per call and there are overall 4 HTTP hits in a single call (3 for user + 1 for dial plan).
Since Monday, we have started analyzing the results of load test, specially looking into packet traces, voicemail recordings and so on. We observed a strange thing, although we were able to test 1536 concurrent calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls per second rate. But there is strange and unexplained delay in starting RTP from FS side.
FS plays an IVR in early media state, then answers the call to directly start recording voicemail. The problem is if call rate is under 25 CPS, everything works fine. At 25 CPS there is minor delay in starting RTP in early media state. This delay is negligible after processing few hundred thousand calls but after about 1.2 million calls the delay has creeps up to about 1.2 seconds. After 30 CPS this delay is significantly increases to 2 second after around 400 calls had done. At 50 CPS its ~3 second and after 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in starting playing IVR in early media state, once the IVR starts playing everything works perfectly fine. No call fails, but this delay would be annoying for anyone using the servers with over 30 CPS.
Just to let you know, we checked everything, system resources, bandwidth usage, IVR file formats etc. all seems to be fine. We even tested with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing seems to have any impact on the results.
I have run of out of possible explanations, so calling you guys for help.
Thank you.
On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <mike@jerris.com> wrote:
I would need to see the specifics of your dialplan, but I suspect you have some transfers or execute extension that are causing it to re-enter dialplan. Are you sure on all the others that they are making the exact same requests? Details here will be necessary to understand exactly what is going on..
Mike
On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
Guy, after detailed analysis i can confirm it was an Apache issue, and changing Apache configuration changes the concurrent call limit. However, one thing i observed about xml curl is that it makes several request with same parameters per call. Even if we have no calls running on server, as well as as Apache is completely free. I can see FS makes at least
1). 3 hits for dialplan. 2). 2 hits for directory. 3). 4 hits for voicemail. 4). 3 hits for ivr. 5). 2 hits for sofia. and so on..
While ideally only one hit should be sufficient to get all these configurations. What is the point of making all these redundant hits? We have roughly 15 - 20 hits made per call, which is huge and causes significant load on Apache.
Both Apache and FS are running on dedicated machines and connected over Gigabit Ethernet with zero packet loss, so i doubt these hits are "re-transmission".
Thank you.
On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
Yes, its a feature, i know. I was just looking at how this feature was tested, as mentioned in its logs,
2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 Processing 21653XXXXX->121637XXXXX in context termination 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://192.168.0.2/gateway.php 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml
Thank you.
On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike@jerris.com>wrote:
This is adding a feature to allow timeout to be set in ms instead of seconds. This is not a problem, its a feature, that was added a while ago. Until you can determine if the request is being sent for the one thats failing, your not going to solve this issue.
Mike
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
You could either throw Cacti on the server, or use server density/new relic trial for quick and easy graphing.. or you can just watch `top` and push number 1 whilst running to see the CPU breakdown. Cal On Wed, Jun 12, 2013 at 1:19 PM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
Steven, i am using soft timer. Should i try another? Please suggest.
Cal, no there is nothing unusually changes in RAM or CPU. BTW, i have 24 CPU cores with 16 GB of RAM on each server. Regarding CPU graph, how can i enable it.
Thank you.
On Wed, Jun 12, 2013 at 1:15 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
This sounds quite interesting.. From my own benchmarks, I've seen that IVRs applications can be quite stressful on the server.
Have you checked to see if there is a CPU util spike or context switching spike during these tests? Perhaps you are hitting the floor limit of what your CPU is capable of?
Do you have CPU graphs? If not, can you enable them and try again?
Cal
On Wed, Jun 12, 2013 at 10:21 AM, Muhammad Shahzad <shaheryarkh@gmail.com
wrote:
You are right. After many improvements and modifications to dail plan, removing unnecessary modules and using Ngnix web server, things are much better now. Except that directory information is fetched 3 times per call and there are overall 4 HTTP hits in a single call (3 for user + 1 for dial plan).
Since Monday, we have started analyzing the results of load test, specially looking into packet traces, voicemail recordings and so on. We observed a strange thing, although we were able to test 1536 concurrent calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls per second rate. But there is strange and unexplained delay in starting RTP from FS side.
FS plays an IVR in early media state, then answers the call to directly start recording voicemail. The problem is if call rate is under 25 CPS, everything works fine. At 25 CPS there is minor delay in starting RTP in early media state. This delay is negligible after processing few hundred thousand calls but after about 1.2 million calls the delay has creeps up to about 1.2 seconds. After 30 CPS this delay is significantly increases to 2 second after around 400 calls had done. At 50 CPS its ~3 second and after 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in starting playing IVR in early media state, once the IVR starts playing everything works perfectly fine. No call fails, but this delay would be annoying for anyone using the servers with over 30 CPS.
Just to let you know, we checked everything, system resources, bandwidth usage, IVR file formats etc. all seems to be fine. We even tested with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing seems to have any impact on the results.
I have run of out of possible explanations, so calling you guys for help.
Thank you.
On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <mike@jerris.com> wrote:
I would need to see the specifics of your dialplan, but I suspect you have some transfers or execute extension that are causing it to re-enter dialplan. Are you sure on all the others that they are making the exact same requests? Details here will be necessary to understand exactly what is going on..
Mike
On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
Guy, after detailed analysis i can confirm it was an Apache issue, and changing Apache configuration changes the concurrent call limit. However, one thing i observed about xml curl is that it makes several request with same parameters per call. Even if we have no calls running on server, as well as as Apache is completely free. I can see FS makes at least
1). 3 hits for dialplan. 2). 2 hits for directory. 3). 4 hits for voicemail. 4). 3 hits for ivr. 5). 2 hits for sofia. and so on..
While ideally only one hit should be sufficient to get all these configurations. What is the point of making all these redundant hits? We have roughly 15 - 20 hits made per call, which is huge and causes significant load on Apache.
Both Apache and FS are running on dedicated machines and connected over Gigabit Ethernet with zero packet loss, so i doubt these hits are "re-transmission".
Thank you.
On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad <shaheryarkh@gmail.com
wrote:
Yes, its a feature, i know. I was just looking at how this feature was tested, as mentioned in its logs,
2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 Processing 21653XXXXX->121637XXXXX in context termination 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://192.168.0.2/gateway.php 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml
Thank you.
On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike@jerris.com>wrote:
This is adding a feature to allow timeout to be set in ms instead of seconds. This is not a problem, its a feature, that was added a while ago. Until you can determine if the request is being sent for the one thats failing, your not going to solve this issue.
Mike
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
We are already monitoring using top and htop. Even per core view of top shows no core every goes beyond 40%, average varies between 21 - 33%. Thank you. On Wed, Jun 12, 2013 at 4:02 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
You could either throw Cacti on the server, or use server density/new relic trial for quick and easy graphing.. or you can just watch `top` and push number 1 whilst running to see the CPU breakdown.
Cal
On Wed, Jun 12, 2013 at 1:19 PM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
Steven, i am using soft timer. Should i try another? Please suggest.
Cal, no there is nothing unusually changes in RAM or CPU. BTW, i have 24 CPU cores with 16 GB of RAM on each server. Regarding CPU graph, how can i enable it.
Thank you.
On Wed, Jun 12, 2013 at 1:15 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
This sounds quite interesting.. From my own benchmarks, I've seen that IVRs applications can be quite stressful on the server.
Have you checked to see if there is a CPU util spike or context switching spike during these tests? Perhaps you are hitting the floor limit of what your CPU is capable of?
Do you have CPU graphs? If not, can you enable them and try again?
Cal
On Wed, Jun 12, 2013 at 10:21 AM, Muhammad Shahzad < shaheryarkh@gmail.com> wrote:
You are right. After many improvements and modifications to dail plan, removing unnecessary modules and using Ngnix web server, things are much better now. Except that directory information is fetched 3 times per call and there are overall 4 HTTP hits in a single call (3 for user + 1 for dial plan).
Since Monday, we have started analyzing the results of load test, specially looking into packet traces, voicemail recordings and so on. We observed a strange thing, although we were able to test 1536 concurrent calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls per second rate. But there is strange and unexplained delay in starting RTP from FS side.
FS plays an IVR in early media state, then answers the call to directly start recording voicemail. The problem is if call rate is under 25 CPS, everything works fine. At 25 CPS there is minor delay in starting RTP in early media state. This delay is negligible after processing few hundred thousand calls but after about 1.2 million calls the delay has creeps up to about 1.2 seconds. After 30 CPS this delay is significantly increases to 2 second after around 400 calls had done. At 50 CPS its ~3 second and after 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in starting playing IVR in early media state, once the IVR starts playing everything works perfectly fine. No call fails, but this delay would be annoying for anyone using the servers with over 30 CPS.
Just to let you know, we checked everything, system resources, bandwidth usage, IVR file formats etc. all seems to be fine. We even tested with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing seems to have any impact on the results.
I have run of out of possible explanations, so calling you guys for help.
Thank you.
On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <mike@jerris.com> wrote:
I would need to see the specifics of your dialplan, but I suspect you have some transfers or execute extension that are causing it to re-enter dialplan. Are you sure on all the others that they are making the exact same requests? Details here will be necessary to understand exactly what is going on..
Mike
On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
Guy, after detailed analysis i can confirm it was an Apache issue, and changing Apache configuration changes the concurrent call limit. However, one thing i observed about xml curl is that it makes several request with same parameters per call. Even if we have no calls running on server, as well as as Apache is completely free. I can see FS makes at least
1). 3 hits for dialplan. 2). 2 hits for directory. 3). 4 hits for voicemail. 4). 3 hits for ivr. 5). 2 hits for sofia. and so on..
While ideally only one hit should be sufficient to get all these configurations. What is the point of making all these redundant hits? We have roughly 15 - 20 hits made per call, which is huge and causes significant load on Apache.
Both Apache and FS are running on dedicated machines and connected over Gigabit Ethernet with zero packet loss, so i doubt these hits are "re-transmission".
Thank you.
On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad < shaheryarkh@gmail.com> wrote:
Yes, its a feature, i know. I was just looking at how this feature was tested, as mentioned in its logs,
2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 Processing 21653XXXXX->121637XXXXX in context termination 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://192.168.0.2/gateway.php 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml
Thank you.
On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike@jerris.com>wrote:
> This is adding a feature to allow timeout to be set in ms instead of > seconds. This is not a problem, its a feature, that was added a while ago. > Until you can determine if the request is being sent for the one thats > failing, your not going to solve this issue. > > Mike >
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
Curious, I'm wondering if maybe excessive context switching is causing some issues.. I'm wondering if the latency is actually caused by the overheads on mod_xml_curl. Could you try putting New Relic on your application, and see what sort of response times you get? That will identify which queries are taking a while to process. Cal On Wed, Jun 12, 2013 at 3:14 PM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
We are already monitoring using top and htop. Even per core view of top shows no core every goes beyond 40%, average varies between 21 - 33%.
Thank you.
On Wed, Jun 12, 2013 at 4:02 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
You could either throw Cacti on the server, or use server density/new relic trial for quick and easy graphing.. or you can just watch `top` and push number 1 whilst running to see the CPU breakdown.
Cal
On Wed, Jun 12, 2013 at 1:19 PM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
Steven, i am using soft timer. Should i try another? Please suggest.
Cal, no there is nothing unusually changes in RAM or CPU. BTW, i have 24 CPU cores with 16 GB of RAM on each server. Regarding CPU graph, how can i enable it.
Thank you.
On Wed, Jun 12, 2013 at 1:15 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
This sounds quite interesting.. From my own benchmarks, I've seen that IVRs applications can be quite stressful on the server.
Have you checked to see if there is a CPU util spike or context switching spike during these tests? Perhaps you are hitting the floor limit of what your CPU is capable of?
Do you have CPU graphs? If not, can you enable them and try again?
Cal
On Wed, Jun 12, 2013 at 10:21 AM, Muhammad Shahzad < shaheryarkh@gmail.com> wrote:
You are right. After many improvements and modifications to dail plan, removing unnecessary modules and using Ngnix web server, things are much better now. Except that directory information is fetched 3 times per call and there are overall 4 HTTP hits in a single call (3 for user + 1 for dial plan).
Since Monday, we have started analyzing the results of load test, specially looking into packet traces, voicemail recordings and so on. We observed a strange thing, although we were able to test 1536 concurrent calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls per second rate. But there is strange and unexplained delay in starting RTP from FS side.
FS plays an IVR in early media state, then answers the call to directly start recording voicemail. The problem is if call rate is under 25 CPS, everything works fine. At 25 CPS there is minor delay in starting RTP in early media state. This delay is negligible after processing few hundred thousand calls but after about 1.2 million calls the delay has creeps up to about 1.2 seconds. After 30 CPS this delay is significantly increases to 2 second after around 400 calls had done. At 50 CPS its ~3 second and after 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in starting playing IVR in early media state, once the IVR starts playing everything works perfectly fine. No call fails, but this delay would be annoying for anyone using the servers with over 30 CPS.
Just to let you know, we checked everything, system resources, bandwidth usage, IVR file formats etc. all seems to be fine. We even tested with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing seems to have any impact on the results.
I have run of out of possible explanations, so calling you guys for help.
Thank you.
On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <mike@jerris.com>wrote:
I would need to see the specifics of your dialplan, but I suspect you have some transfers or execute extension that are causing it to re-enter dialplan. Are you sure on all the others that they are making the exact same requests? Details here will be necessary to understand exactly what is going on..
Mike
On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
Guy, after detailed analysis i can confirm it was an Apache issue, and changing Apache configuration changes the concurrent call limit. However, one thing i observed about xml curl is that it makes several request with same parameters per call. Even if we have no calls running on server, as well as as Apache is completely free. I can see FS makes at least
1). 3 hits for dialplan. 2). 2 hits for directory. 3). 4 hits for voicemail. 4). 3 hits for ivr. 5). 2 hits for sofia. and so on..
While ideally only one hit should be sufficient to get all these configurations. What is the point of making all these redundant hits? We have roughly 15 - 20 hits made per call, which is huge and causes significant load on Apache.
Both Apache and FS are running on dedicated machines and connected over Gigabit Ethernet with zero packet loss, so i doubt these hits are "re-transmission".
Thank you.
On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad < shaheryarkh@gmail.com> wrote:
> Yes, its a feature, i know. I was just looking at how this feature > was tested, as mentioned in its logs, > > 2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 Processing > 21653XXXXX->121637XXXXX in context termination > 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP > error 0 trying to fetch http://192.168.0.2/gateway.php > 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML response > is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml > > Thank you. > > > > > On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike@jerris.com>wrote: > >> This is adding a feature to allow timeout to be set in ms instead >> of seconds. This is not a problem, its a feature, that was added a while >> ago. Until you can determine if the request is being sent for the one >> thats failing, your not going to solve this issue. >> >> Mike >> >
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Well, per our observation, we don't see any load or delay from xml curl or web server side. Also the delay is occurring exactly at one location that is after sending 183 Early Media and sending first RTP packet of IVR. When CPS is below 25 this delay is less then 0.20 seconds but after 30 CPS it goes beyond 3.10 seconds. Regarding New Relic, we don't have WAN access on server, since its a closed environment at telco data center. Also we see the behavior remains the same if we add multiple FS serve by same web server. So, i think problem is on FS side rather then web server side. Thank you. On Wed, Jun 12, 2013 at 4:31 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
Curious, I'm wondering if maybe excessive context switching is causing some issues.. I'm wondering if the latency is actually caused by the overheads on mod_xml_curl.
Could you try putting New Relic on your application, and see what sort of response times you get? That will identify which queries are taking a while to process.
Cal
On Wed, Jun 12, 2013 at 3:14 PM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
We are already monitoring using top and htop. Even per core view of top shows no core every goes beyond 40%, average varies between 21 - 33%.
Thank you.
On Wed, Jun 12, 2013 at 4:02 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
You could either throw Cacti on the server, or use server density/new relic trial for quick and easy graphing.. or you can just watch `top` and push number 1 whilst running to see the CPU breakdown.
Cal
On Wed, Jun 12, 2013 at 1:19 PM, Muhammad Shahzad <shaheryarkh@gmail.com
wrote:
Steven, i am using soft timer. Should i try another? Please suggest.
Cal, no there is nothing unusually changes in RAM or CPU. BTW, i have 24 CPU cores with 16 GB of RAM on each server. Regarding CPU graph, how can i enable it.
Thank you.
On Wed, Jun 12, 2013 at 1:15 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
This sounds quite interesting.. From my own benchmarks, I've seen that IVRs applications can be quite stressful on the server.
Have you checked to see if there is a CPU util spike or context switching spike during these tests? Perhaps you are hitting the floor limit of what your CPU is capable of?
Do you have CPU graphs? If not, can you enable them and try again?
Cal
On Wed, Jun 12, 2013 at 10:21 AM, Muhammad Shahzad < shaheryarkh@gmail.com> wrote:
You are right. After many improvements and modifications to dail plan, removing unnecessary modules and using Ngnix web server, things are much better now. Except that directory information is fetched 3 times per call and there are overall 4 HTTP hits in a single call (3 for user + 1 for dial plan).
Since Monday, we have started analyzing the results of load test, specially looking into packet traces, voicemail recordings and so on. We observed a strange thing, although we were able to test 1536 concurrent calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls per second rate. But there is strange and unexplained delay in starting RTP from FS side.
FS plays an IVR in early media state, then answers the call to directly start recording voicemail. The problem is if call rate is under 25 CPS, everything works fine. At 25 CPS there is minor delay in starting RTP in early media state. This delay is negligible after processing few hundred thousand calls but after about 1.2 million calls the delay has creeps up to about 1.2 seconds. After 30 CPS this delay is significantly increases to 2 second after around 400 calls had done. At 50 CPS its ~3 second and after 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in starting playing IVR in early media state, once the IVR starts playing everything works perfectly fine. No call fails, but this delay would be annoying for anyone using the servers with over 30 CPS.
Just to let you know, we checked everything, system resources, bandwidth usage, IVR file formats etc. all seems to be fine. We even tested with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing seems to have any impact on the results.
I have run of out of possible explanations, so calling you guys for help.
Thank you.
On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <mike@jerris.com>wrote:
> I would need to see the specifics of your dialplan, but I suspect > you have some transfers or execute extension that are causing it to > re-enter dialplan. Are you sure on all the others that they are making the > exact same requests? Details here will be necessary to understand exactly > what is going on.. > > Mike > > On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad <shaheryarkh@gmail.com> > wrote: > > Guy, after detailed analysis i can confirm it was an Apache issue, > and changing Apache configuration changes the concurrent call limit. > However, one thing i observed about xml curl is that it makes several > request with same parameters per call. Even if we have no calls running on > server, as well as as Apache is completely free. I can see FS makes at least > > 1). 3 hits for dialplan. > 2). 2 hits for directory. > 3). 4 hits for voicemail. > 4). 3 hits for ivr. > 5). 2 hits for sofia. > and so on.. > > While ideally only one hit should be sufficient to get all these > configurations. What is the point of making all these redundant hits? We > have roughly 15 - 20 hits made per call, which is huge and causes > significant load on Apache. > > Both Apache and FS are running on dedicated machines and connected > over Gigabit Ethernet with zero packet loss, so i doubt these hits are > "re-transmission". > > Thank you. > > > > > On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad < > shaheryarkh@gmail.com> wrote: > >> Yes, its a feature, i know. I was just looking at how this feature >> was tested, as mentioned in its logs, >> >> 2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 Processing >> 21653XXXXX->121637XXXXX in context termination >> 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP >> error 0 trying to fetch http://192.168.0.2/gateway.php >> 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML >> response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml >> >> Thank you. >> >> >> >> >> On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike@jerris.com>wrote: >> >>> This is adding a feature to allow timeout to be set in ms instead >>> of seconds. This is not a problem, its a feature, that was added a while >>> ago. Until you can determine if the request is being sent for the one >>> thats failing, your not going to solve this issue. >>> >>> Mike >>> >> > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: > http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > >
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
Okay, so the next step is to identify what is happening in the code between the 183 and the first RTP packet.. you could do this by either putting in some manual print statements into the code and go for a trial and error, or you could use a profiler or some sort.. I'd advice creating a JIRA ticket on this so that a core dev can assist you, as this is now going outside of my area of expertise. Cal On Wed, Jun 12, 2013 at 3:57 PM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
Well, per our observation, we don't see any load or delay from xml curl or web server side. Also the delay is occurring exactly at one location that is after sending 183 Early Media and sending first RTP packet of IVR. When CPS is below 25 this delay is less then 0.20 seconds but after 30 CPS it goes beyond 3.10 seconds.
Regarding New Relic, we don't have WAN access on server, since its a closed environment at telco data center. Also we see the behavior remains the same if we add multiple FS serve by same web server. So, i think problem is on FS side rather then web server side.
Thank you.
On Wed, Jun 12, 2013 at 4:31 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
Curious, I'm wondering if maybe excessive context switching is causing some issues.. I'm wondering if the latency is actually caused by the overheads on mod_xml_curl.
Could you try putting New Relic on your application, and see what sort of response times you get? That will identify which queries are taking a while to process.
Cal
On Wed, Jun 12, 2013 at 3:14 PM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
We are already monitoring using top and htop. Even per core view of top shows no core every goes beyond 40%, average varies between 21 - 33%.
Thank you.
On Wed, Jun 12, 2013 at 4:02 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
You could either throw Cacti on the server, or use server density/new relic trial for quick and easy graphing.. or you can just watch `top` and push number 1 whilst running to see the CPU breakdown.
Cal
On Wed, Jun 12, 2013 at 1:19 PM, Muhammad Shahzad < shaheryarkh@gmail.com> wrote:
Steven, i am using soft timer. Should i try another? Please suggest.
Cal, no there is nothing unusually changes in RAM or CPU. BTW, i have 24 CPU cores with 16 GB of RAM on each server. Regarding CPU graph, how can i enable it.
Thank you.
On Wed, Jun 12, 2013 at 1:15 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
This sounds quite interesting.. From my own benchmarks, I've seen that IVRs applications can be quite stressful on the server.
Have you checked to see if there is a CPU util spike or context switching spike during these tests? Perhaps you are hitting the floor limit of what your CPU is capable of?
Do you have CPU graphs? If not, can you enable them and try again?
Cal
On Wed, Jun 12, 2013 at 10:21 AM, Muhammad Shahzad < shaheryarkh@gmail.com> wrote:
> You are right. After many improvements and modifications to dail > plan, removing unnecessary modules and using Ngnix web server, things are > much better now. Except that directory information is fetched 3 times per > call and there are overall 4 HTTP hits in a single call (3 for user + 1 for > dial plan). > > Since Monday, we have started analyzing the results of load test, > specially looking into packet traces, voicemail recordings and so on. We > observed a strange thing, although we were able to test 1536 concurrent > calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls > per second rate. But there is strange and unexplained delay in starting RTP > from FS side. > > FS plays an IVR in early media state, then answers the call to > directly start recording voicemail. The problem is if call rate is under 25 > CPS, everything works fine. At 25 CPS there is minor delay in starting RTP > in early media state. This delay is negligible after processing few hundred > thousand calls but after about 1.2 million calls the delay has creeps up to > about 1.2 seconds. After 30 CPS this delay is significantly increases to 2 > second after around 400 calls had done. At 50 CPS its ~3 second and after > 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in > starting playing IVR in early media state, once the IVR starts playing > everything works perfectly fine. No call fails, but this delay would be > annoying for anyone using the servers with over 30 CPS. > > Just to let you know, we checked everything, system resources, > bandwidth usage, IVR file formats etc. all seems to be fine. We even tested > with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing > seems to have any impact on the results. > > I have run of out of possible explanations, so calling you guys for > help. > > Thank you. > > > > > On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <mike@jerris.com>wrote: > >> I would need to see the specifics of your dialplan, but I suspect >> you have some transfers or execute extension that are causing it to >> re-enter dialplan. Are you sure on all the others that they are making the >> exact same requests? Details here will be necessary to understand exactly >> what is going on.. >> >> Mike >> >> On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad <shaheryarkh@gmail.com> >> wrote: >> >> Guy, after detailed analysis i can confirm it was an Apache issue, >> and changing Apache configuration changes the concurrent call limit. >> However, one thing i observed about xml curl is that it makes several >> request with same parameters per call. Even if we have no calls running on >> server, as well as as Apache is completely free. I can see FS makes at least >> >> 1). 3 hits for dialplan. >> 2). 2 hits for directory. >> 3). 4 hits for voicemail. >> 4). 3 hits for ivr. >> 5). 2 hits for sofia. >> and so on.. >> >> While ideally only one hit should be sufficient to get all these >> configurations. What is the point of making all these redundant hits? We >> have roughly 15 - 20 hits made per call, which is huge and causes >> significant load on Apache. >> >> Both Apache and FS are running on dedicated machines and connected >> over Gigabit Ethernet with zero packet loss, so i doubt these hits are >> "re-transmission". >> >> Thank you. >> >> >> >> >> On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad < >> shaheryarkh@gmail.com> wrote: >> >>> Yes, its a feature, i know. I was just looking at how this feature >>> was tested, as mentioned in its logs, >>> >>> 2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 >>> Processing 21653XXXXX->121637XXXXX in context termination >>> 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP >>> error 0 trying to fetch http://192.168.0.2/gateway.php >>> 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML >>> response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml >>> >>> Thank you. >>> >>> >>> >>> >>> On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike@jerris.com>wrote: >>> >>>> This is adding a feature to allow timeout to be set in ms instead >>>> of seconds. This is not a problem, its a feature, that was added a while >>>> ago. Until you can determine if the request is being sent for the one >>>> thats failing, your not going to solve this issue. >>>> >>>> Mike >>>> >>> >> >> >> _________________________________________________________________________ >> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org >> http://www.freeswitchsolutions.com >> >> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >> http://www.cudatel.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://wiki.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE: >> http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >> >> > > > -- > Mit freundlichen Grüßen > Muhammad Shahzad > ----------------------------------- > CISCO Rich Media Communication Specialist (CRMCS) > CISCO Certified Network Associate (CCNA) > Cell: +49 176 99 83 10 85 > MSN: shari_786pk@hotmail.com > Email: shaheryarkh@googlemail.com > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: > http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > >
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
OK. I am already doing trial and error testing on using rtp related variables. I noticed one strange warning upon pre_answer, 2013-06-12 20:30:01.143911 [WARNING] switch_core_media.c:1282 Asynchronous PTIME not supported, changing our end from 0 to 30 Any ideas about it? Could this be cause of all trouble? Just wondering.. Thank you. On Wed, Jun 12, 2013 at 5:17 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
Okay, so the next step is to identify what is happening in the code between the 183 and the first RTP packet.. you could do this by either putting in some manual print statements into the code and go for a trial and error, or you could use a profiler or some sort..
I'd advice creating a JIRA ticket on this so that a core dev can assist you, as this is now going outside of my area of expertise.
Cal
On Wed, Jun 12, 2013 at 3:57 PM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
Well, per our observation, we don't see any load or delay from xml curl or web server side. Also the delay is occurring exactly at one location that is after sending 183 Early Media and sending first RTP packet of IVR. When CPS is below 25 this delay is less then 0.20 seconds but after 30 CPS it goes beyond 3.10 seconds.
Regarding New Relic, we don't have WAN access on server, since its a closed environment at telco data center. Also we see the behavior remains the same if we add multiple FS serve by same web server. So, i think problem is on FS side rather then web server side.
Thank you.
On Wed, Jun 12, 2013 at 4:31 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
Curious, I'm wondering if maybe excessive context switching is causing some issues.. I'm wondering if the latency is actually caused by the overheads on mod_xml_curl.
Could you try putting New Relic on your application, and see what sort of response times you get? That will identify which queries are taking a while to process.
Cal
On Wed, Jun 12, 2013 at 3:14 PM, Muhammad Shahzad <shaheryarkh@gmail.com
wrote:
We are already monitoring using top and htop. Even per core view of top shows no core every goes beyond 40%, average varies between 21 - 33%.
Thank you.
On Wed, Jun 12, 2013 at 4:02 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
You could either throw Cacti on the server, or use server density/new relic trial for quick and easy graphing.. or you can just watch `top` and push number 1 whilst running to see the CPU breakdown.
Cal
On Wed, Jun 12, 2013 at 1:19 PM, Muhammad Shahzad < shaheryarkh@gmail.com> wrote:
Steven, i am using soft timer. Should i try another? Please suggest.
Cal, no there is nothing unusually changes in RAM or CPU. BTW, i have 24 CPU cores with 16 GB of RAM on each server. Regarding CPU graph, how can i enable it.
Thank you.
On Wed, Jun 12, 2013 at 1:15 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
> This sounds quite interesting.. From my own benchmarks, I've seen > that IVRs applications can be quite stressful on the server. > > Have you checked to see if there is a CPU util spike or context > switching spike during these tests? Perhaps you are hitting the floor limit > of what your CPU is capable of? > > Do you have CPU graphs? If not, can you enable them and try again? > > Cal > > > On Wed, Jun 12, 2013 at 10:21 AM, Muhammad Shahzad < > shaheryarkh@gmail.com> wrote: > >> You are right. After many improvements and modifications to dail >> plan, removing unnecessary modules and using Ngnix web server, things are >> much better now. Except that directory information is fetched 3 times per >> call and there are overall 4 HTTP hits in a single call (3 for user + 1 for >> dial plan). >> >> Since Monday, we have started analyzing the results of load test, >> specially looking into packet traces, voicemail recordings and so on. We >> observed a strange thing, although we were able to test 1536 concurrent >> calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls >> per second rate. But there is strange and unexplained delay in starting RTP >> from FS side. >> >> FS plays an IVR in early media state, then answers the call to >> directly start recording voicemail. The problem is if call rate is under 25 >> CPS, everything works fine. At 25 CPS there is minor delay in starting RTP >> in early media state. This delay is negligible after processing few hundred >> thousand calls but after about 1.2 million calls the delay has creeps up to >> about 1.2 seconds. After 30 CPS this delay is significantly increases to 2 >> second after around 400 calls had done. At 50 CPS its ~3 second and after >> 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in >> starting playing IVR in early media state, once the IVR starts playing >> everything works perfectly fine. No call fails, but this delay would be >> annoying for anyone using the servers with over 30 CPS. >> >> Just to let you know, we checked everything, system resources, >> bandwidth usage, IVR file formats etc. all seems to be fine. We even tested >> with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing >> seems to have any impact on the results. >> >> I have run of out of possible explanations, so calling you guys for >> help. >> >> Thank you. >> >> >> >> >> On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <mike@jerris.com>wrote: >> >>> I would need to see the specifics of your dialplan, but I suspect >>> you have some transfers or execute extension that are causing it to >>> re-enter dialplan. Are you sure on all the others that they are making the >>> exact same requests? Details here will be necessary to understand exactly >>> what is going on.. >>> >>> Mike >>> >>> On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad < >>> shaheryarkh@gmail.com> wrote: >>> >>> Guy, after detailed analysis i can confirm it was an Apache issue, >>> and changing Apache configuration changes the concurrent call limit. >>> However, one thing i observed about xml curl is that it makes several >>> request with same parameters per call. Even if we have no calls running on >>> server, as well as as Apache is completely free. I can see FS makes at least >>> >>> 1). 3 hits for dialplan. >>> 2). 2 hits for directory. >>> 3). 4 hits for voicemail. >>> 4). 3 hits for ivr. >>> 5). 2 hits for sofia. >>> and so on.. >>> >>> While ideally only one hit should be sufficient to get all these >>> configurations. What is the point of making all these redundant hits? We >>> have roughly 15 - 20 hits made per call, which is huge and causes >>> significant load on Apache. >>> >>> Both Apache and FS are running on dedicated machines and connected >>> over Gigabit Ethernet with zero packet loss, so i doubt these hits are >>> "re-transmission". >>> >>> Thank you. >>> >>> >>> >>> >>> On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad < >>> shaheryarkh@gmail.com> wrote: >>> >>>> Yes, its a feature, i know. I was just looking at how this >>>> feature was tested, as mentioned in its logs, >>>> >>>> 2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 >>>> Processing 21653XXXXX->121637XXXXX in context termination >>>> 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP >>>> error 0 trying to fetch http://192.168.0.2/gateway.php >>>> 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML >>>> response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml >>>> >>>> Thank you. >>>> >>>> >>>> >>>> >>>> On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike@jerris.com>wrote: >>>> >>>>> This is adding a feature to allow timeout to be set in ms >>>>> instead of seconds. This is not a problem, its a feature, that was added a >>>>> while ago. Until you can determine if the request is being sent for the >>>>> one thats failing, your not going to solve this issue. >>>>> >>>>> Mike >>>>> >>>> >>> >>> >>> _________________________________________________________________________ >>> Professional FreeSWITCH Consulting Services: >>> consulting@freeswitch.org >>> http://www.freeswitchsolutions.com >>> >>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>> http://www.cudatel.com >>> >>> Official FreeSWITCH Sites >>> http://www.freeswitch.org >>> http://wiki.freeswitch.org >>> http://www.cluecon.com >>> >>> FreeSWITCH-users mailing list >>> FreeSWITCH-users@lists.freeswitch.org >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>> UNSUBSCRIBE: >>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>> http://www.freeswitch.org >>> >>> >> >> >> -- >> Mit freundlichen Grüßen >> Muhammad Shahzad >> ----------------------------------- >> CISCO Rich Media Communication Specialist (CRMCS) >> CISCO Certified Network Associate (CCNA) >> Cell: +49 176 99 83 10 85 >> MSN: shari_786pk@hotmail.com >> Email: shaheryarkh@googlemail.com >> >> >> _________________________________________________________________________ >> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org >> http://www.freeswitchsolutions.com >> >> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >> http://www.cudatel.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://wiki.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE: >> http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >> >> > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: > http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > >
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
There is an existing discussion on this; http://lists.freeswitch.org/pipermail/freeswitch-users/2010-December/066697.... Just a guess but, perhaps try a different codec and see what happens? Cal On Wed, Jun 12, 2013 at 4:34 PM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
OK. I am already doing trial and error testing on using rtp related variables. I noticed one strange warning upon pre_answer,
2013-06-12 20:30:01.143911 [WARNING] switch_core_media.c:1282 Asynchronous PTIME not supported, changing our end from 0 to 30
Any ideas about it? Could this be cause of all trouble? Just wondering..
Thank you.
On Wed, Jun 12, 2013 at 5:17 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
Okay, so the next step is to identify what is happening in the code between the 183 and the first RTP packet.. you could do this by either putting in some manual print statements into the code and go for a trial and error, or you could use a profiler or some sort..
I'd advice creating a JIRA ticket on this so that a core dev can assist you, as this is now going outside of my area of expertise.
Cal
On Wed, Jun 12, 2013 at 3:57 PM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
Well, per our observation, we don't see any load or delay from xml curl or web server side. Also the delay is occurring exactly at one location that is after sending 183 Early Media and sending first RTP packet of IVR. When CPS is below 25 this delay is less then 0.20 seconds but after 30 CPS it goes beyond 3.10 seconds.
Regarding New Relic, we don't have WAN access on server, since its a closed environment at telco data center. Also we see the behavior remains the same if we add multiple FS serve by same web server. So, i think problem is on FS side rather then web server side.
Thank you.
On Wed, Jun 12, 2013 at 4:31 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
Curious, I'm wondering if maybe excessive context switching is causing some issues.. I'm wondering if the latency is actually caused by the overheads on mod_xml_curl.
Could you try putting New Relic on your application, and see what sort of response times you get? That will identify which queries are taking a while to process.
Cal
On Wed, Jun 12, 2013 at 3:14 PM, Muhammad Shahzad < shaheryarkh@gmail.com> wrote:
We are already monitoring using top and htop. Even per core view of top shows no core every goes beyond 40%, average varies between 21 - 33%.
Thank you.
On Wed, Jun 12, 2013 at 4:02 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
You could either throw Cacti on the server, or use server density/new relic trial for quick and easy graphing.. or you can just watch `top` and push number 1 whilst running to see the CPU breakdown.
Cal
On Wed, Jun 12, 2013 at 1:19 PM, Muhammad Shahzad < shaheryarkh@gmail.com> wrote:
> Steven, i am using soft timer. Should i try another? Please suggest. > > Cal, no there is nothing unusually changes in RAM or CPU. BTW, i > have 24 CPU cores with 16 GB of RAM on each server. Regarding CPU graph, > how can i enable it. > > Thank you. > > > > On Wed, Jun 12, 2013 at 1:15 PM, Cal Leeming [Simplicity Media Ltd] > <cal.leeming@simplicitymedialtd.co.uk> wrote: > >> This sounds quite interesting.. From my own benchmarks, I've seen >> that IVRs applications can be quite stressful on the server. >> >> Have you checked to see if there is a CPU util spike or context >> switching spike during these tests? Perhaps you are hitting the floor limit >> of what your CPU is capable of? >> >> Do you have CPU graphs? If not, can you enable them and try again? >> >> Cal >> >> >> On Wed, Jun 12, 2013 at 10:21 AM, Muhammad Shahzad < >> shaheryarkh@gmail.com> wrote: >> >>> You are right. After many improvements and modifications to dail >>> plan, removing unnecessary modules and using Ngnix web server, things are >>> much better now. Except that directory information is fetched 3 times per >>> call and there are overall 4 HTTP hits in a single call (3 for user + 1 for >>> dial plan). >>> >>> Since Monday, we have started analyzing the results of load test, >>> specially looking into packet traces, voicemail recordings and so on. We >>> observed a strange thing, although we were able to test 1536 concurrent >>> calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls >>> per second rate. But there is strange and unexplained delay in starting RTP >>> from FS side. >>> >>> FS plays an IVR in early media state, then answers the call to >>> directly start recording voicemail. The problem is if call rate is under 25 >>> CPS, everything works fine. At 25 CPS there is minor delay in starting RTP >>> in early media state. This delay is negligible after processing few hundred >>> thousand calls but after about 1.2 million calls the delay has creeps up to >>> about 1.2 seconds. After 30 CPS this delay is significantly increases to 2 >>> second after around 400 calls had done. At 50 CPS its ~3 second and after >>> 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in >>> starting playing IVR in early media state, once the IVR starts playing >>> everything works perfectly fine. No call fails, but this delay would be >>> annoying for anyone using the servers with over 30 CPS. >>> >>> Just to let you know, we checked everything, system resources, >>> bandwidth usage, IVR file formats etc. all seems to be fine. We even tested >>> with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing >>> seems to have any impact on the results. >>> >>> I have run of out of possible explanations, so calling you guys >>> for help. >>> >>> Thank you. >>> >>> >>> >>> >>> On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <mike@jerris.com>wrote: >>> >>>> I would need to see the specifics of your dialplan, but I suspect >>>> you have some transfers or execute extension that are causing it to >>>> re-enter dialplan. Are you sure on all the others that they are making the >>>> exact same requests? Details here will be necessary to understand exactly >>>> what is going on.. >>>> >>>> Mike >>>> >>>> On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad < >>>> shaheryarkh@gmail.com> wrote: >>>> >>>> Guy, after detailed analysis i can confirm it was an Apache >>>> issue, and changing Apache configuration changes the concurrent call limit. >>>> However, one thing i observed about xml curl is that it makes several >>>> request with same parameters per call. Even if we have no calls running on >>>> server, as well as as Apache is completely free. I can see FS makes at least >>>> >>>> 1). 3 hits for dialplan. >>>> 2). 2 hits for directory. >>>> 3). 4 hits for voicemail. >>>> 4). 3 hits for ivr. >>>> 5). 2 hits for sofia. >>>> and so on.. >>>> >>>> While ideally only one hit should be sufficient to get all these >>>> configurations. What is the point of making all these redundant hits? We >>>> have roughly 15 - 20 hits made per call, which is huge and causes >>>> significant load on Apache. >>>> >>>> Both Apache and FS are running on dedicated machines and >>>> connected over Gigabit Ethernet with zero packet loss, so i doubt these >>>> hits are "re-transmission". >>>> >>>> Thank you. >>>> >>>> >>>> >>>> >>>> On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad < >>>> shaheryarkh@gmail.com> wrote: >>>> >>>>> Yes, its a feature, i know. I was just looking at how this >>>>> feature was tested, as mentioned in its logs, >>>>> >>>>> 2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 >>>>> Processing 21653XXXXX->121637XXXXX in context termination >>>>> 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received >>>>> HTTP error 0 trying to fetch http://192.168.0.2/gateway.php >>>>> 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML >>>>> response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml >>>>> >>>>> Thank you. >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike@jerris.com >>>>> > wrote: >>>>> >>>>>> This is adding a feature to allow timeout to be set in ms >>>>>> instead of seconds. This is not a problem, its a feature, that was added a >>>>>> while ago. Until you can determine if the request is being sent for the >>>>>> one thats failing, your not going to solve this issue. >>>>>> >>>>>> Mike >>>>>> >>>>> >>>> >>>> >>>> _________________________________________________________________________ >>>> Professional FreeSWITCH Consulting Services: >>>> consulting@freeswitch.org >>>> http://www.freeswitchsolutions.com >>>> >>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>>> http://www.cudatel.com >>>> >>>> Official FreeSWITCH Sites >>>> http://www.freeswitch.org >>>> http://wiki.freeswitch.org >>>> http://www.cluecon.com >>>> >>>> FreeSWITCH-users mailing list >>>> FreeSWITCH-users@lists.freeswitch.org >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>> UNSUBSCRIBE: >>>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>>> http://www.freeswitch.org >>>> >>>> >>> >>> >>> -- >>> Mit freundlichen Grüßen >>> Muhammad Shahzad >>> ----------------------------------- >>> CISCO Rich Media Communication Specialist (CRMCS) >>> CISCO Certified Network Associate (CCNA) >>> Cell: +49 176 99 83 10 85 >>> MSN: shari_786pk@hotmail.com >>> Email: shaheryarkh@googlemail.com >>> >>> >>> _________________________________________________________________________ >>> Professional FreeSWITCH Consulting Services: >>> consulting@freeswitch.org >>> http://www.freeswitchsolutions.com >>> >>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>> http://www.cudatel.com >>> >>> Official FreeSWITCH Sites >>> http://www.freeswitch.org >>> http://wiki.freeswitch.org >>> http://www.cluecon.com >>> >>> FreeSWITCH-users mailing list >>> FreeSWITCH-users@lists.freeswitch.org >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>> UNSUBSCRIBE: >>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>> http://www.freeswitch.org >>> >>> >> >> >> _________________________________________________________________________ >> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org >> http://www.freeswitchsolutions.com >> >> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >> http://www.cudatel.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://wiki.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE: >> http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >> >> > > > -- > Mit freundlichen Grüßen > Muhammad Shahzad > ----------------------------------- > CISCO Rich Media Communication Specialist (CRMCS) > CISCO Certified Network Associate (CCNA) > Cell: +49 176 99 83 10 85 > MSN: shari_786pk@hotmail.com > Email: shaheryarkh@googlemail.com > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: > http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > >
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Ok, this "square peg in rouond hole" thread is not getting anything solved. You all need to take a step back, and look at the overall diesign of the applcation you are trying to build from a top down approch. First define what the application has to do, with all of it's requirements, including load, resonse times, what information you need to look up per call, and so on... This seems to me like a backwards approch was taken, as the start of the project, mod_xml_curl was the starting point, and then the application was designed after the method was picked. Isn't this "bass awards"? I know if I wasted my clients time and money by picking tech. before defining the problem, I would have no clients! One thing I can tell you for sure, is that using a design that involves both client and server side synchronous technologies as the starting point, isn’t going to scale out to carrier rate call handling. You can try throwing all sorts of stuff like different web servers, caching proxies in the middle, and a handful of other kludges at it, and all you’re going to get is a big “Cluster F…”, that will be a nightmare to maintain and scale. You may also become a slave to maintaining it, and monitoring your production environment. One other thing to remember about FreeSWITCH, is that mod_xml_curl is not the only way of doing XML config binding to code. There are many modules that use the callback to implement feeding FS with real time XML config on demand. It may be time to re-design your application, rather then trying to build a “better mousetrap” on the external side. We could discuss this on the conference call today, and get input from all the folks on a design, given your requirements --Dave _____ From: Muhammad Shahzad [mailto:shaheryarkh@gmail.com] To: FreeSWITCH Users Help [mailto:freeswitch-users@lists.freeswitch.org] Sent: Wed, 12 Jun 2013 07:57:18 -0700 Subject: Re: [Freeswitch-users] XML_CURL max connection limit Well, per our observation, we don't see any load or delay from xml curl or web server side. Also the delay is occurring exactly at one location that is after sending 183 Early Media and sending first RTP packet of IVR. When CPS is below 25 this delay is less then 0.20 seconds but after 30 CPS it goes beyond 3.10 seconds. Regarding New Relic, we don't have WAN access on server, since its a closed environment at telco data center. Also we see the behavior remains the same if we add multiple FS serve by same web server. So, i think problem is on FS side rather then web server side. Thank you. On Wed, Jun 12, 2013 at 4:31 PM, Cal Leeming [Simplicity Media Ltd] <cal.leeming@simplicitymedialtd.co.uk> wrote: Curious, I'm wondering if maybe excessive context switching is causing some issues.. I'm wondering if the latency is actually caused by the overheads on mod_xml_curl. Could you try putting New Relic on your application, and see what sort of response times you get? That will identify which queries are taking a while to process. Cal On Wed, Jun 12, 2013 at 3:14 PM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote: We are already monitoring using top and htop. Even per core view of top shows no core every goes beyond 40%, average varies between 21 - 33%. Thank you. On Wed, Jun 12, 2013 at 4:02 PM, Cal Leeming [Simplicity Media Ltd] <cal.leeming@simplicitymedialtd.co.uk> wrote: You could either throw Cacti on the server, or use server density/new relic trial for quick and easy graphing.. or you can just watch `top` and push number 1 whilst running to see the CPU breakdown. Cal On Wed, Jun 12, 2013 at 1:19 PM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote: Steven, i am using soft timer. Should i try another? Please suggest. Cal, no there is nothing unusually changes in RAM or CPU. BTW, i have 24 CPU cores with 16 GB of RAM on each server. Regarding CPU graph, how can i enable it. Thank you. On Wed, Jun 12, 2013 at 1:15 PM, Cal Leeming [Simplicity Media Ltd] <cal.leeming@simplicitymedialtd.co.uk> wrote: This sounds quite interesting.. From my own benchmarks, I've seen that IVRs applications can be quite stressful on the server. Have you checked to see if there is a CPU util spike or context switching spike during these tests? Perhaps you are hitting the floor limit of what your CPU is capable of? Do you have CPU graphs? If not, can you enable them and try again? Cal On Wed, Jun 12, 2013 at 10:21 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote: You are right. After many improvements and modifications to dail plan, removing unnecessary modules and using Ngnix web server, things are much better now. Except that directory information is fetched 3 times per call and there are overall 4 HTTP hits in a single call (3 for user + 1 for dial plan). Since Monday, we have started analyzing the results of load test, specially looking into packet traces, voicemail recordings and so on. We observed a strange thing, although we were able to test 1536 concurrent calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls per second rate. But there is strange and unexplained delay in starting RTP from FS side. FS plays an IVR in early media state, then answers the call to directly start recording voicemail. The problem is if call rate is under 25 CPS, everything works fine. At 25 CPS there is minor delay in starting RTP in early media state. This delay is negligible after processing few hundred thousand calls but after about 1.2 million calls the delay has creeps up to about 1.2 seconds. After 30 CPS this delay is significantly increases to 2 second after around 400 calls had done. At 50 CPS its ~3 second and after 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in starting playing IVR in early media state, once the IVR starts playing everything works perfectly fine. No call fails, but this delay would be annoying for anyone using the servers with over 30 CPS. Just to let you know, we checked everything, system resources, bandwidth usage, IVR file formats etc. all seems to be fine. We even tested with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing seems to have any impact on the results. I have run of out of possible explanations, so calling you guys for help. Thank you. On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <mike@jerris.com> wrote: I would need to see the specifics of your dialplan, but I suspect you have some transfers or execute extension that are causing it to re-enter dialplan. Are you sure on all the others that they are making the exact same requests? Details here will be necessary to understand exactly what is going on.. Mike On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote: Guy, after detailed analysis i can confirm it was an Apache issue, and changing Apache configuration changes the concurrent call limit. However, one thing i observed about xml curl is that it makes several request with same parameters per call. Even if we have no calls running on server, as well as as Apache is completely free. I can see FS makes at least 1). 3 hits for dialplan. 2). 2 hits for directory. 3). 4 hits for voicemail. 4). 3 hits for ivr. 5). 2 hits for sofia. and so on.. While ideally only one hit should be sufficient to get all these configurations. What is the point of making all these redundant hits? We have roughly 15 - 20 hits made per call, which is huge and causes significant load on Apache. Both Apache and FS are running on dedicated machines and connected over Gigabit Ethernet with zero packet loss, so i doubt these hits are "re-transmission". Thank you. On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote: Yes, its a feature, i know. I was just looking at how this feature was tested, as mentioned in its logs, 2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 Processing 21653XXXXX->121637XXXXX in context termination 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://192.168.0.2/gateway.php 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml Thank you. On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike@jerris.com> wrote: This is adding a feature to allow timeout to be set in ms instead of seconds. This is not a problem, its a feature, that was added a while ago. Until you can determine if the request is being sent for the one thats failing, your not going to solve this issue. Mike _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Mit freundlichen GrüßenMuhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Mit freundlichen GrüßenMuhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Mit freundlichen GrüßenMuhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Mit freundlichen GrüßenMuhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
Dave, with all due respect, every time the discussion of mod_xml_curl or anything non-windows comes up you immediately throw in a negative note, and it's starting to get a bit old now. Until one of the core devs steps up and says "you shouldn't be using mod_xml_curl" then I don't think it's right to be telling users that this is the wrong approach to be using - especially when we don't even know if it's mod_xml_curl that is causing the problem in this particular case. As mentioned in the last email, the next steps for OP is to identify the root cause of these delays, either via profiling or print statements. OP - I think at this point, you really need to take this over to a JIRA ticket to avoid spamming the mailing list, as the thread is starting to go a bit stale. Once you've created a ticket, let us know in this thread and we'll see about getting a core dev to give it some eyes. If this is an urgent matter, you could also pay for a few hours of FS support which will get the immediate attention of a core dev :) Cal On Wed, Jun 12, 2013 at 5:19 PM, Dave R. Kompel <drk@drkngs.net> wrote:
** Ok, this "square peg in rouond hole" thread is not getting anything solved. You all need to take a step back, and look at the overall diesign of the applcation you are trying to build from a top down approch.
First define what the application has to do, with all of it's requirements, including load, resonse times, what information you need to look up per call, and so on...
This seems to me like a backwards approch was taken, as the start of the project, mod_xml_curl was the starting point, and then the application was designed after the method was picked. Isn't this "bass awards"? I know if I wasted my clients time and money by picking tech. before defining the problem, I would have no clients!
One thing I can tell you for sure, is that using a design that involves both client and server side synchronous technologies as the starting point, isn’t going to scale out to carrier rate call handling. You can try throwing all sorts of stuff like different web servers, caching proxies in the middle, and a handful of other kludges at it, and all you’re going to get is a big “Cluster F…”, that will be a nightmare to maintain and scale. You may also become a slave to maintaining it, and monitoring your production environment.******
One other thing to remember about FreeSWITCH, is that mod_xml_curl is not the only way of doing XML config binding to code. There are many modules that use the callback to implement feeding FS with real time XML config on demand. ****
It may be time to re-design your application, rather then trying to build a “better mousetrap” on the external side. We could discuss this on the conference call today, and get input from all the folks on a design, given your requirements**** --Dave
------------------------------ *From:* Muhammad Shahzad [mailto:shaheryarkh@gmail.com] *To:* FreeSWITCH Users Help [mailto:freeswitch-users@lists.freeswitch.org] *Sent:* Wed, 12 Jun 2013 07:57:18 -0700 *Subject:* Re: [Freeswitch-users] XML_CURL max connection limit
Well, per our observation, we don't see any load or delay from xml curl or web server side. Also the delay is occurring exactly at one location that is after sending 183 Early Media and sending first RTP packet of IVR. When CPS is below 25 this delay is less then 0.20 seconds but after 30 CPS it goes beyond 3.10 seconds.
Regarding New Relic, we don't have WAN access on server, since its a closed environment at telco data center. Also we see the behavior remains the same if we add multiple FS serve by same web server. So, i think problem is on FS side rather then web server side.
Thank you.
On Wed, Jun 12, 2013 at 4:31 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
Curious, I'm wondering if maybe excessive context switching is causing some issues.. I'm wondering if the latency is actually caused by the overheads on mod_xml_curl.
Could you try putting New Relic on your application, and see what sort of response times you get? That will identify which queries are taking a while to process.
Cal
On Wed, Jun 12, 2013 at 3:14 PM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
We are already monitoring using top and htop. Even per core view of top shows no core every goes beyond 40%, average varies between 21 - 33%.
Thank you.
On Wed, Jun 12, 2013 at 4:02 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
You could either throw Cacti on the server, or use server density/new relic trial for quick and easy graphing.. or you can just watch `top` and push number 1 whilst running to see the CPU breakdown.
Cal
On Wed, Jun 12, 2013 at 1:19 PM, Muhammad Shahzad < shaheryarkh@gmail.com> wrote:
Steven, i am using soft timer. Should i try another? Please suggest.
Cal, no there is nothing unusually changes in RAM or CPU. BTW, i have 24 CPU cores with 16 GB of RAM on each server. Regarding CPU graph, how can i enable it.
Thank you.
On Wed, Jun 12, 2013 at 1:15 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
This sounds quite interesting.. From my own benchmarks, I've seen that IVRs applications can be quite stressful on the server.
Have you checked to see if there is a CPU util spike or context switching spike during these tests? Perhaps you are hitting the floor limit of what your CPU is capable of?
Do you have CPU graphs? If not, can you enable them and try again?
Cal
On Wed, Jun 12, 2013 at 10:21 AM, Muhammad Shahzad < shaheryarkh@gmail.com> wrote:
> You are right. After many improvements and modifications to dail > plan, removing unnecessary modules and using Ngnix web server, things are > much better now. Except that directory information is fetched 3 times per > call and there are overall 4 HTTP hits in a single call (3 for user + 1 for > dial plan). > > Since Monday, we have started analyzing the results of load test, > specially looking into packet traces, voicemail recordings and so on. We > observed a strange thing, although we were able to test 1536 concurrent > calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls > per second rate. But there is strange and unexplained delay in starting RTP > from FS side. > > FS plays an IVR in early media state, then answers the call to > directly start recording voicemail. The problem is if call rate is under 25 > CPS, everything works fine. At 25 CPS there is minor delay in starting RTP > in early media state. This delay is negligible after processing few hundred > thousand calls but after about 1.2 million calls the delay has creeps up to > about 1.2 seconds. After 30 CPS this delay is significantly increases to 2 > second after around 400 calls had done. At 50 CPS its ~3 second and after > 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in > starting playing IVR in early media state, once the IVR starts playing > everything works perfectly fine. No call fails, but this delay would be > annoying for anyone using the servers with over 30 CPS. > > Just to let you know, we checked everything, system resources, > bandwidth usage, IVR file formats etc. all seems to be fine. We even tested > with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing > seems to have any impact on the results. > > I have run of out of possible explanations, so calling you guys for > help. > > Thank you. > > > > > On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <mike@jerris.com>wrote: > >> I would need to see the specifics of your dialplan, but I suspect >> you have some transfers or execute extension that are causing it to >> re-enter dialplan. Are you sure on all the others that they are making the >> exact same requests? Details here will be necessary to understand exactly >> what is going on.. >> >> Mike >> >> On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad < >> shaheryarkh@gmail.com> wrote: >> >> Guy, after detailed analysis i can confirm it was an Apache >> issue, and changing Apache configuration changes the concurrent call limit. >> However, one thing i observed about xml curl is that it makes several >> request with same parameters per call. Even if we have no calls running on >> server, as well as as Apache is completely free. I can see FS makes at >> least >> >> 1). 3 hits for dialplan. >> 2). 2 hits for directory. >> 3). 4 hits for voicemail. >> 4). 3 hits for ivr. >> 5). 2 hits for sofia. >> and so on.. >> >> While ideally only one hit should be sufficient to get all these >> configurations. What is the point of making all these redundant hits? We >> have roughly 15 - 20 hits made per call, which is huge and causes >> significant load on Apache. >> >> Both Apache and FS are running on dedicated machines and connected >> over Gigabit Ethernet with zero packet loss, so i doubt these hits are >> "re-transmission". >> >> Thank you. >> >> >> >> >> On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad < >> shaheryarkh@gmail.com> wrote: >> >>> Yes, its a feature, i know. I was just looking at how this feature >>> was tested, as mentioned in its logs, >>> >>> 2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 >>> Processing 21653XXXXX->121637XXXXX in context termination >>> 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP >>> error 0 trying to fetch http://192.168.0.2/gateway.php >>> 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML >>> response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml >>> >>> Thank you. >>> >>> >>> >>> >>> On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike@jerris.com>wrote: >>> >>>> This is adding a feature to allow timeout to be set in ms >>>> instead of seconds. This is not a problem, its a feature, that was added a >>>> while ago. Until you can determine if the request is being sent for the >>>> one thats failing, your not going to solve this issue. >>>> >>>> Mike >>>> >>> >> >> >> _________________________________________________________________________ >> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org >> http://www.freeswitchsolutions.com >> >> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >> http://www.cudatel.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://wiki.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE: >> http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >> >> > > > -- > Mit freundlichen Grüßen > Muhammad Shahzad > ----------------------------------- > CISCO Rich Media Communication Specialist (CRMCS) > CISCO Certified Network Associate (CCNA) > Cell: +49 176 99 83 10 85 > MSN: shari_786pk@hotmail.com > Email: shaheryarkh@googlemail.com > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: > http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > >
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
WTF does any of this have to do w/ windows? _____ From: Cal Leeming [Simplicity Media Ltd] [mailto:cal.leeming@simplicitymedialtd.co.uk] To: FreeSWITCH Users Help [mailto:freeswitch-users@lists.freeswitch.org] Sent: Wed, 12 Jun 2013 09:38:40 -0700 Subject: Re: [Freeswitch-users] XML_CURL max connection limit Dave, with all due respect, every time the discussion of mod_xml_curl or anything non-windows comes up you immediately throw in a negative note, and it's starting to get a bit old now. Until one of the core devs steps up and says "you shouldn't be using mod_xml_curl" then I don't think it's right to be telling users that this is the wrong approach to be using - especially when we don't even know if it's mod_xml_curl that is causing the problem in this particular case. As mentioned in the last email, the next steps for OP is to identify the root cause of these delays, either via profiling or print statements. OP - I think at this point, you really need to take this over to a JIRA ticket to avoid spamming the mailing list, as the thread is starting to go a bit stale. Once you've created a ticket, let us know in this thread and we'll see about getting a core dev to give it some eyes. If this is an urgent matter, you could also pay for a few hours of FS support which will get the immediate attention of a core dev :) Cal On Wed, Jun 12, 2013 at 5:19 PM, Dave R. Kompel <drk@drkngs.net> wrote: Ok, this "square peg in rouond hole" thread is not getting anything solved. You all need to take a step back, and look at the overall diesign of the applcation you are trying to build from a top down approch. First define what the application has to do, with all of it's requirements, including load, resonse times, what information you need to look up per call, and so on... This seems to me like a backwards approch was taken, as the start of the project, mod_xml_curl was the starting point, and then the application was designed after the method was picked. Isn't this "bass awards"? I know if I wasted my clients time and money by picking tech. before defining the problem, I would have no clients! One thing I can tell you for sure, is that using a design that involves both client and server side synchronous technologies as the starting point, isn’t going to scale out to carrier rate call handling. You can try throwing all sorts of stuff like different web servers, caching proxies in the middle, and a handful of other kludges at it, and all you’re going to get is a big “Cluster F…”, that will be a nightmare to maintain and scale. You may also become a slave to maintaining it, and monitoring your production environment. One other thing to remember about FreeSWITCH, is that mod_xml_curl is not the only way of doing XML config binding to code. There are many modules that use the callback to implement feeding FS with real time XML config on demand. It may be time to re-design your application, rather then trying to build a “better mousetrap” on the external side. We could discuss this on the conference call today, and get input from all the folks on a design, given your requirements --Dave _____ From: Muhammad Shahzad [mailto:shaheryarkh@gmail.com] To: FreeSWITCH Users Help [mailto:freeswitch-users@lists.freeswitch.org] Sent: Wed, 12 Jun 2013 07:57:18 -0700 Subject: Re: [Freeswitch-users] XML_CURL max connection limit Well, per our observation, we don't see any load or delay from xml curl or web server side. Also the delay is occurring exactly at one location that is after sending 183 Early Media and sending first RTP packet of IVR. When CPS is below 25 this delay is less then 0.20 seconds but after 30 CPS it goes beyond 3.10 seconds. Regarding New Relic, we don't have WAN access on server, since its a closed environment at telco data center. Also we see the behavior remains the same if we add multiple FS serve by same web server. So, i think problem is on FS side rather then web server side. Thank you. On Wed, Jun 12, 2013 at 4:31 PM, Cal Leeming [Simplicity Media Ltd] <cal.leeming@simplicitymedialtd.co.uk> wrote: Curious, I'm wondering if maybe excessive context switching is causing some issues.. I'm wondering if the latency is actually caused by the overheads on mod_xml_curl. Could you try putting New Relic on your application, and see what sort of response times you get? That will identify which queries are taking a while to process. Cal On Wed, Jun 12, 2013 at 3:14 PM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote: We are already monitoring using top and htop. Even per core view of top shows no core every goes beyond 40%, average varies between 21 - 33%. Thank you. On Wed, Jun 12, 2013 at 4:02 PM, Cal Leeming [Simplicity Media Ltd] <cal.leeming@simplicitymedialtd.co.uk> wrote: You could either throw Cacti on the server, or use server density/new relic trial for quick and easy graphing.. or you can just watch `top` and push number 1 whilst running to see the CPU breakdown. Cal On Wed, Jun 12, 2013 at 1:19 PM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote: Steven, i am using soft timer. Should i try another? Please suggest. Cal, no there is nothing unusually changes in RAM or CPU. BTW, i have 24 CPU cores with 16 GB of RAM on each server. Regarding CPU graph, how can i enable it. Thank you. On Wed, Jun 12, 2013 at 1:15 PM, Cal Leeming [Simplicity Media Ltd] <cal.leeming@simplicitymedialtd.co.uk> wrote: This sounds quite interesting.. From my own benchmarks, I've seen that IVRs applications can be quite stressful on the server. Have you checked to see if there is a CPU util spike or context switching spike during these tests? Perhaps you are hitting the floor limit of what your CPU is capable of? Do you have CPU graphs? If not, can you enable them and try again? Cal On Wed, Jun 12, 2013 at 10:21 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote: You are right. After many improvements and modifications to dail plan, removing unnecessary modules and using Ngnix web server, things are much better now. Except that directory information is fetched 3 times per call and there are overall 4 HTTP hits in a single call (3 for user + 1 for dial plan). Since Monday, we have started analyzing the results of load test, specially looking into packet traces, voicemail recordings and so on. We observed a strange thing, although we were able to test 1536 concurrent calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls per second rate. But there is strange and unexplained delay in starting RTP from FS side. FS plays an IVR in early media state, then answers the call to directly start recording voicemail. The problem is if call rate is under 25 CPS, everything works fine. At 25 CPS there is minor delay in starting RTP in early media state. This delay is negligible after processing few hundred thousand calls but after about 1.2 million calls the delay has creeps up to about 1.2 seconds. After 30 CPS this delay is significantly increases to 2 second after around 400 calls had done. At 50 CPS its ~3 second and after 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in starting playing IVR in early media state, once the IVR starts playing everything works perfectly fine. No call fails, but this delay would be annoying for anyone using the servers with over 30 CPS. Just to let you know, we checked everything, system resources, bandwidth usage, IVR file formats etc. all seems to be fine. We even tested with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing seems to have any impact on the results. I have run of out of possible explanations, so calling you guys for help. Thank you. On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <mike@jerris.com> wrote: I would need to see the specifics of your dialplan, but I suspect you have some transfers or execute extension that are causing it to re-enter dialplan. Are you sure on all the others that they are making the exact same requests? Details here will be necessary to understand exactly what is going on.. Mike On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote: Guy, after detailed analysis i can confirm it was an Apache issue, and changing Apache configuration changes the concurrent call limit. However, one thing i observed about xml curl is that it makes several request with same parameters per call. Even if we have no calls running on server, as well as as Apache is completely free. I can see FS makes at least 1). 3 hits for dialplan. 2). 2 hits for directory. 3). 4 hits for voicemail. 4). 3 hits for ivr. 5). 2 hits for sofia. and so on.. While ideally only one hit should be sufficient to get all these configurations. What is the point of making all these redundant hits? We have roughly 15 - 20 hits made per call, which is huge and causes significant load on Apache. Both Apache and FS are running on dedicated machines and connected over Gigabit Ethernet with zero packet loss, so i doubt these hits are "re-transmission". Thank you. On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote: Yes, its a feature, i know. I was just looking at how this feature was tested, as mentioned in its logs, 2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 Processing 21653XXXXX->121637XXXXX in context termination 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://192.168.0.2/gateway.php 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml Thank you. On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike@jerris.com> wrote: This is adding a feature to allow timeout to be set in ms instead of seconds. This is not a problem, its a feature, that was added a while ago. Until you can determine if the request is being sent for the one thats failing, your not going to solve this issue. Mike _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Mit freundlichen GrüßenMuhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Mit freundlichen GrüßenMuhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Mit freundlichen GrüßenMuhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Mit freundlichen GrüßenMuhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I dont think Dave even mentioned anything about Windows his rant. From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Cal Leeming [Simplicity Media Ltd] Sent: Wednesday, June 12, 2013 9:39 AM To: FreeSWITCH Users Help Subject: Re: [Freeswitch-users] XML_CURL max connection limit Dave, with all due respect, every time the discussion of mod_xml_curl or anything non-windows comes up you immediately throw in a negative note, and it's starting to get a bit old now. Until one of the core devs steps up and says "you shouldn't be using mod_xml_curl" then I don't think it's right to be telling users that this is the wrong approach to be using - especially when we don't even know if it's mod_xml_curl that is causing the problem in this particular case. As mentioned in the last email, the next steps for OP is to identify the root cause of these delays, either via profiling or print statements. OP - I think at this point, you really need to take this over to a JIRA ticket to avoid spamming the mailing list, as the thread is starting to go a bit stale. Once you've created a ticket, let us know in this thread and we'll see about getting a core dev to give it some eyes. If this is an urgent matter, you could also pay for a few hours of FS support which will get the immediate attention of a core dev :) Cal On Wed, Jun 12, 2013 at 5:19 PM, Dave R. Kompel <drk@drkngs.net> wrote: Ok, this "square peg in rouond hole" thread is not getting anything solved. You all need to take a step back, and look at the overall diesign of the applcation you are trying to build from a top down approch. First define what the application has to do, with all of it's requirements, including load, resonse times, what information you need to look up per call, and so on... This seems to me like a backwards approch was taken, as the start of the project, mod_xml_curl was the starting point, and then the application was designed after the method was picked. Isn't this "bass awards"? I know if I wasted my clients time and money by picking tech. before defining the problem, I would have no clients! One thing I can tell you for sure, is that using a design that involves both client and server side synchronous technologies as the starting point, isnt going to scale out to carrier rate call handling. You can try throwing all sorts of stuff like different web servers, caching proxies in the middle, and a handful of other kludges at it, and all youre going to get is a big Cluster F , that will be a nightmare to maintain and scale. You may also become a slave to maintaining it, and monitoring your production environment. One other thing to remember about FreeSWITCH, is that mod_xml_curl is not the only way of doing XML config binding to code. There are many modules that use the callback to implement feeding FS with real time XML config on demand. It may be time to re-design your application, rather then trying to build a better mousetrap on the external side. We could discuss this on the conference call today, and get input from all the folks on a design, given your requirements --Dave _____ From: Muhammad Shahzad [mailto:shaheryarkh@gmail.com] To: FreeSWITCH Users Help [mailto:freeswitch-users@lists.freeswitch.org] Sent: Wed, 12 Jun 2013 07:57:18 -0700 Subject: Re: [Freeswitch-users] XML_CURL max connection limit Well, per our observation, we don't see any load or delay from xml curl or web server side. Also the delay is occurring exactly at one location that is after sending 183 Early Media and sending first RTP packet of IVR. When CPS is below 25 this delay is less then 0.20 seconds but after 30 CPS it goes beyond 3.10 seconds. Regarding New Relic, we don't have WAN access on server, since its a closed environment at telco data center. Also we see the behavior remains the same if we add multiple FS serve by same web server. So, i think problem is on FS side rather then web server side. Thank you. On Wed, Jun 12, 2013 at 4:31 PM, Cal Leeming [Simplicity Media Ltd] <cal.leeming@simplicitymedialtd.co.uk> wrote: Curious, I'm wondering if maybe excessive context switching is causing some issues.. I'm wondering if the latency is actually caused by the overheads on mod_xml_curl. Could you try putting New Relic on your application, and see what sort of response times you get? That will identify which queries are taking a while to process. Cal On Wed, Jun 12, 2013 at 3:14 PM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote: We are already monitoring using top and htop. Even per core view of top shows no core every goes beyond 40%, average varies between 21 - 33%. Thank you. On Wed, Jun 12, 2013 at 4:02 PM, Cal Leeming [Simplicity Media Ltd] <cal.leeming@simplicitymedialtd.co.uk> wrote: You could either throw Cacti on the server, or use server density/new relic trial for quick and easy graphing.. or you can just watch `top` and push number 1 whilst running to see the CPU breakdown. Cal On Wed, Jun 12, 2013 at 1:19 PM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote: Steven, i am using soft timer. Should i try another? Please suggest. Cal, no there is nothing unusually changes in RAM or CPU. BTW, i have 24 CPU cores with 16 GB of RAM on each server. Regarding CPU graph, how can i enable it. Thank you. On Wed, Jun 12, 2013 at 1:15 PM, Cal Leeming [Simplicity Media Ltd] <cal.leeming@simplicitymedialtd.co.uk> wrote: This sounds quite interesting.. From my own benchmarks, I've seen that IVRs applications can be quite stressful on the server. Have you checked to see if there is a CPU util spike or context switching spike during these tests? Perhaps you are hitting the floor limit of what your CPU is capable of? Do you have CPU graphs? If not, can you enable them and try again? Cal On Wed, Jun 12, 2013 at 10:21 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote: You are right. After many improvements and modifications to dail plan, removing unnecessary modules and using Ngnix web server, things are much better now. Except that directory information is fetched 3 times per call and there are overall 4 HTTP hits in a single call (3 for user + 1 for dial plan). Since Monday, we have started analyzing the results of load test, specially looking into packet traces, voicemail recordings and so on. We observed a strange thing, although we were able to test 1536 concurrent calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls per second rate. But there is strange and unexplained delay in starting RTP from FS side. FS plays an IVR in early media state, then answers the call to directly start recording voicemail. The problem is if call rate is under 25 CPS, everything works fine. At 25 CPS there is minor delay in starting RTP in early media state. This delay is negligible after processing few hundred thousand calls but after about 1.2 million calls the delay has creeps up to about 1.2 seconds. After 30 CPS this delay is significantly increases to 2 second after around 400 calls had done. At 50 CPS its ~3 second and after 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in starting playing IVR in early media state, once the IVR starts playing everything works perfectly fine. No call fails, but this delay would be annoying for anyone using the servers with over 30 CPS. Just to let you know, we checked everything, system resources, bandwidth usage, IVR file formats etc. all seems to be fine. We even tested with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing seems to have any impact on the results. I have run of out of possible explanations, so calling you guys for help. Thank you. On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <mike@jerris.com> wrote: I would need to see the specifics of your dialplan, but I suspect you have some transfers or execute extension that are causing it to re-enter dialplan. Are you sure on all the others that they are making the exact same requests? Details here will be necessary to understand exactly what is going on.. Mike On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote: Guy, after detailed analysis i can confirm it was an Apache issue, and changing Apache configuration changes the concurrent call limit. However, one thing i observed about xml curl is that it makes several request with same parameters per call. Even if we have no calls running on server, as well as as Apache is completely free. I can see FS makes at least 1). 3 hits for dialplan. 2). 2 hits for directory. 3). 4 hits for voicemail. 4). 3 hits for ivr. 5). 2 hits for sofia. and so on.. While ideally only one hit should be sufficient to get all these configurations. What is the point of making all these redundant hits? We have roughly 15 - 20 hits made per call, which is huge and causes significant load on Apache. Both Apache and FS are running on dedicated machines and connected over Gigabit Ethernet with zero packet loss, so i doubt these hits are "re-transmission". Thank you. On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote: Yes, its a feature, i know. I was just looking at how this feature was tested, as mentioned in its logs, 2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 Processing 21653XXXXX->121637XXXXX in context termination 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch <http://192.168.0.2/gateway.php> http://192.168.0.2/gateway.php 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml Thank you. On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike@jerris.com> wrote: This is adding a feature to allow timeout to be set in ms instead of seconds. This is not a problem, its a feature, that was added a while ago. Until you can determine if the request is being sent for the one thats failing, your not going to solve this issue. Mike _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/> FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com <http://www.cudatel.com/> Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://wiki.freeswitch.org <http://wiki.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org <http://www.freeswitch.org/> -- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 <tel:%2B49%20176%2099%2083%2010%2085> MSN: <mailto:shari_786pk@hotmail.com> shari_786pk@hotmail.com Email: <mailto:shaheryarkh@googlemail.com> shaheryarkh@googlemail.com _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/> FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com <http://www.cudatel.com/> Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://wiki.freeswitch.org <http://wiki.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org <http://www.freeswitch.org/> _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/> FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com <http://www.cudatel.com/> Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://wiki.freeswitch.org <http://wiki.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org <http://www.freeswitch.org/> -- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 <tel:%2B49%20176%2099%2083%2010%2085> MSN: <mailto:shari_786pk@hotmail.com> shari_786pk@hotmail.com Email: <mailto:shaheryarkh@googlemail.com> shaheryarkh@googlemail.com _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/> FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com <http://www.cudatel.com/> Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://wiki.freeswitch.org <http://wiki.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org <http://www.freeswitch.org/> _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/> FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com <http://www.cudatel.com/> Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://wiki.freeswitch.org <http://wiki.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org <http://www.freeswitch.org/> -- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 <tel:%2B49%20176%2099%2083%2010%2085> MSN: <mailto:shari_786pk@hotmail.com> shari_786pk@hotmail.com Email: <mailto:shaheryarkh@googlemail.com> shaheryarkh@googlemail.com _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/> FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com <http://www.cudatel.com/> Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://wiki.freeswitch.org <http://wiki.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org <http://www.freeswitch.org/> _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/> FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com <http://www.cudatel.com/> Official FreeSWITCH Sites http://www.freeswitch.org <http://www.freeswitch.org/> http://wiki.freeswitch.org <http://wiki.freeswitch.org/> http://www.cluecon.com <http://www.cluecon.com/> FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org <http://www.freeswitch.org/> -- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 <tel:%2B49%20176%2099%2083%2010%2085> MSN: <mailto:shari_786pk@hotmail.com> shari_786pk@hotmail.com Email: <mailto:shaheryarkh@googlemail.com> shaheryarkh@googlemail.com _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Apologies - I was referencing some other discussions of heated debate of windows, but I ended up confusing the conversion! Sorry about that guys! Cal On Wed, Jun 12, 2013 at 5:54 PM, Jeff Leung <jleung@v10networks.ca> wrote:
I don’t think Dave even mentioned anything about Windows his rant.****
** **
*From:* freeswitch-users-bounces@lists.freeswitch.org [mailto: freeswitch-users-bounces@lists.freeswitch.org] *On Behalf Of *Cal Leeming [Simplicity Media Ltd] *Sent:* Wednesday, June 12, 2013 9:39 AM *To:* FreeSWITCH Users Help
*Subject:* Re: [Freeswitch-users] XML_CURL max connection limit****
** **
Dave, with all due respect, every time the discussion of mod_xml_curl or anything non-windows comes up you immediately throw in a negative note, and it's starting to get a bit old now. Until one of the core devs steps up and says "you shouldn't be using mod_xml_curl" then I don't think it's right to be telling users that this is the wrong approach to be using - especially when we don't even know if it's mod_xml_curl that is causing the problem in this particular case. As mentioned in the last email, the next steps for OP is to identify the root cause of these delays, either via profiling or print statements.****
** **
OP - I think at this point, you really need to take this over to a JIRA ticket to avoid spamming the mailing list, as the thread is starting to go a bit stale.****
** **
Once you've created a ticket, let us know in this thread and we'll see about getting a core dev to give it some eyes.****
** **
If this is an urgent matter, you could also pay for a few hours of FS support which will get the immediate attention of a core dev :)****
** **
Cal****
** **
On Wed, Jun 12, 2013 at 5:19 PM, Dave R. Kompel <drk@drkngs.net> wrote:*** *
Ok, this "square peg in rouond hole" thread is not getting anything solved. You all need to take a step back, and look at the overall diesign of the applcation you are trying to build from a top down approch.****
****
First define what the application has to do, with all of it's requirements, including load, resonse times, what information you need to look up per call, and so on...****
****
This seems to me like a backwards approch was taken, as the start of the project, mod_xml_curl was the starting point, and then the application was designed after the method was picked. Isn't this "bass awards"? I know if I wasted my clients time and money by picking tech. before defining the problem, I would have no clients!****
****
One thing I can tell you for sure, is that using a design that involves both client and server side synchronous technologies as the starting point, isn’t going to scale out to carrier rate call handling. You can try throwing all sorts of stuff like different web servers, caching proxies in the middle, and a handful of other kludges at it, and all you’re going to get is a big “Cluster F…”, that will be a nightmare to maintain and scale. You may also become a slave to maintaining it, and monitoring your production environment.****
One other thing to remember about FreeSWITCH, is that mod_xml_curl is not the only way of doing XML config binding to code. There are many modules that use the callback to implement feeding FS with real time XML config on demand. ****
It may be time to re-design your application, rather then trying to build a “better mousetrap” on the external side. We could discuss this on the conference call today, and get input from all the folks on a design, given your requirements****
--Dave****
**** ------------------------------
*From:* Muhammad Shahzad [mailto:shaheryarkh@gmail.com] *To:* FreeSWITCH Users Help [mailto:freeswitch-users@lists.freeswitch.org] *Sent:* Wed, 12 Jun 2013 07:57:18 -0700 *Subject:* Re: [Freeswitch-users] XML_CURL max connection limit****
Well, per our observation, we don't see any load or delay from xml curl or web server side. Also the delay is occurring exactly at one location that is after sending 183 Early Media and sending first RTP packet of IVR. When CPS is below 25 this delay is less then 0.20 seconds but after 30 CPS it goes beyond 3.10 seconds. ****
** **
Regarding New Relic, we don't have WAN access on server, since its a closed environment at telco data center. Also we see the behavior remains the same if we add multiple FS serve by same web server. So, i think problem is on FS side rather then web server side.****
** **
Thank you.****
** **
** **
** **
On Wed, Jun 12, 2013 at 4:31 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:****
Curious, I'm wondering if maybe excessive context switching is causing some issues.. I'm wondering if the latency is actually caused by the overheads on mod_xml_curl. ****
** **
Could you try putting New Relic on your application, and see what sort of response times you get? That will identify which queries are taking a while to process.****
Cal ****
** **
On Wed, Jun 12, 2013 at 3:14 PM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:****
We are already monitoring using top and htop. Even per core view of top shows no core every goes beyond 40%, average varies between 21 - 33%.****
** **
Thank you.****
** **
On Wed, Jun 12, 2013 at 4:02 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:****
You could either throw Cacti on the server, or use server density/new relic trial for quick and easy graphing.. or you can just watch `top` and push number 1 whilst running to see the CPU breakdown. ****
Cal ****
** **
On Wed, Jun 12, 2013 at 1:19 PM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:****
Steven, i am using soft timer. Should i try another? Please suggest. ****
** **
Cal, no there is nothing unusually changes in RAM or CPU. BTW, i have 24 CPU cores with 16 GB of RAM on each server. Regarding CPU graph, how can i enable it.****
** **
Thank you.****
** **
** **
On Wed, Jun 12, 2013 at 1:15 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:****
This sounds quite interesting.. From my own benchmarks, I've seen that IVRs applications can be quite stressful on the server. ****
** **
Have you checked to see if there is a CPU util spike or context switching spike during these tests? Perhaps you are hitting the floor limit of what your CPU is capable of?****
** **
Do you have CPU graphs? If not, can you enable them and try again?****
** **
Cal ****
** **
On Wed, Jun 12, 2013 at 10:21 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:****
You are right. After many improvements and modifications to dail plan, removing unnecessary modules and using Ngnix web server, things are much better now. Except that directory information is fetched 3 times per call and there are overall 4 HTTP hits in a single call (3 for user + 1 for dial plan). ****
** **
Since Monday, we have started analyzing the results of load test, specially looking into packet traces, voicemail recordings and so on. We observed a strange thing, although we were able to test 1536 concurrent calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls per second rate. But there is strange and unexplained delay in starting RTP from FS side.****
** **
FS plays an IVR in early media state, then answers the call to directly start recording voicemail. The problem is if call rate is under 25 CPS, everything works fine. At 25 CPS there is minor delay in starting RTP in early media state. This delay is negligible after processing few hundred thousand calls but after about 1.2 million calls the delay has creeps up to about 1.2 seconds. After 30 CPS this delay is significantly increases to 2 second after around 400 calls had done. At 50 CPS its ~3 second and after 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in starting playing IVR in early media state, once the IVR starts playing everything works perfectly fine. No call fails, but this delay would be annoying for anyone using the servers with over 30 CPS.****
** **
Just to let you know, we checked everything, system resources, bandwidth usage, IVR file formats etc. all seems to be fine. We even tested with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing seems to have any impact on the results.****
** **
I have run of out of possible explanations, so calling you guys for help.* ***
** **
Thank you.****
** **
** **
** **
On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <mike@jerris.com> wrote:*** *
I would need to see the specifics of your dialplan, but I suspect you have some transfers or execute extension that are causing it to re-enter dialplan. Are you sure on all the others that they are making the exact same requests? Details here will be necessary to understand exactly what is going on.. ****
** **
Mike****
** **
On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:****
****
Guy, after detailed analysis i can confirm it was an Apache issue, and changing Apache configuration changes the concurrent call limit. However, one thing i observed about xml curl is that it makes several request with same parameters per call. Even if we have no calls running on server, as well as as Apache is completely free. I can see FS makes at least ****
** **
1). 3 hits for dialplan.****
2). 2 hits for directory.****
3). 4 hits for voicemail.****
4). 3 hits for ivr.****
5). 2 hits for sofia.****
and so on..****
** **
While ideally only one hit should be sufficient to get all these configurations. What is the point of making all these redundant hits? We have roughly 15 - 20 hits made per call, which is huge and causes significant load on Apache.****
** **
Both Apache and FS are running on dedicated machines and connected over Gigabit Ethernet with zero packet loss, so i doubt these hits are "re-transmission".****
** **
Thank you.****
** **
** **
** **
On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:****
Yes, its a feature, i know. I was just looking at how this feature was tested, as mentioned in its logs, ****
** **
2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 Processing 21653XXXXX->121637XXXXX in context termination 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://192.168.0.2/gateway.php 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml****
** **
Thank you.****
** **
** **
** **
On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike@jerris.com> wrote:** **
This is adding a feature to allow timeout to be set in ms instead of seconds. This is not a problem, its a feature, that was added a while ago. Until you can determine if the request is being sent for the one thats failing, your not going to solve this issue. ****
** **
Mike****
** **
** **
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org****
****
** **
-- ****
Mit freundlichen Grüßen****
Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com ****
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org****
** **
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org****
****
** **
-- ****
Mit freundlichen Grüßen****
Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com ****
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org****
** **
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org****
****
** **
-- ****
Mit freundlichen Grüßen****
Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com ****
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org****
** **
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org****
****
** **
-- ****
Mit freundlichen Grüßen****
Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com ****
****
****
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org****
** **
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Well, i am using xml curl, since its the only module i know that offers centralized FS configs management. I know there may be better, faster modules but they don't have benefit that xml curl gives us. As for the system design, client's time and money blah blah, i did mentioned in in my first email that we have achieved our design goals and target capacity requirements etc. So far client is very happy with load test results. But on the technical side, i see over 70% of system resources are free and we can go beyond our achieved capacity. This has benefit for all of us (me, client and FS users), since, client will save cost for expansion (he has 50% growth projection in next 6-9 months), i will have better sizing calculation for future projects, thus lowering cost for future clients and normal FS users will have better understanding of how to scale FS (both horizontally and vertically), i will post results of my test once done with deployment. Neither FS nor web server fail even for 1,536 concurrent calls at rate of 60 CPS, so its great success for both FS and xml curl in my view. Its only the initial delay in playing IVR which is kind of inconvenient from user experience point of view that i am trying to get rid of. Anyways, i am opening up a JIRA ticket now. So core developers may look into it. Thanks to all of you who replied and help me in optimization the setup. Thank you. On Wed, Jun 12, 2013 at 6:38 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
Dave, with all due respect, every time the discussion of mod_xml_curl or anything non-windows comes up you immediately throw in a negative note, and it's starting to get a bit old now. Until one of the core devs steps up and says "you shouldn't be using mod_xml_curl" then I don't think it's right to be telling users that this is the wrong approach to be using - especially when we don't even know if it's mod_xml_curl that is causing the problem in this particular case. As mentioned in the last email, the next steps for OP is to identify the root cause of these delays, either via profiling or print statements.
OP - I think at this point, you really need to take this over to a JIRA ticket to avoid spamming the mailing list, as the thread is starting to go a bit stale.
Once you've created a ticket, let us know in this thread and we'll see about getting a core dev to give it some eyes.
If this is an urgent matter, you could also pay for a few hours of FS support which will get the immediate attention of a core dev :)
Cal
On Wed, Jun 12, 2013 at 5:19 PM, Dave R. Kompel <drk@drkngs.net> wrote:
** Ok, this "square peg in rouond hole" thread is not getting anything solved. You all need to take a step back, and look at the overall diesign of the applcation you are trying to build from a top down approch.
First define what the application has to do, with all of it's requirements, including load, resonse times, what information you need to look up per call, and so on...
This seems to me like a backwards approch was taken, as the start of the project, mod_xml_curl was the starting point, and then the application was designed after the method was picked. Isn't this "bass awards"? I know if I wasted my clients time and money by picking tech. before defining the problem, I would have no clients!
One thing I can tell you for sure, is that using a design that involves both client and server side synchronous technologies as the starting point, isn’t going to scale out to carrier rate call handling. You can try throwing all sorts of stuff like different web servers, caching proxies in the middle, and a handful of other kludges at it, and all you’re going to get is a big “Cluster F…”, that will be a nightmare to maintain and scale. You may also become a slave to maintaining it, and monitoring your production environment.******
One other thing to remember about FreeSWITCH, is that mod_xml_curl is not the only way of doing XML config binding to code. There are many modules that use the callback to implement feeding FS with real time XML config on demand. ****
It may be time to re-design your application, rather then trying to build a “better mousetrap” on the external side. We could discuss this on the conference call today, and get input from all the folks on a design, given your requirements**** --Dave
------------------------------ *From:* Muhammad Shahzad [mailto:shaheryarkh@gmail.com] *To:* FreeSWITCH Users Help [mailto:freeswitch-users@lists.freeswitch.org ] *Sent:* Wed, 12 Jun 2013 07:57:18 -0700 *Subject:* Re: [Freeswitch-users] XML_CURL max connection limit
Well, per our observation, we don't see any load or delay from xml curl or web server side. Also the delay is occurring exactly at one location that is after sending 183 Early Media and sending first RTP packet of IVR. When CPS is below 25 this delay is less then 0.20 seconds but after 30 CPS it goes beyond 3.10 seconds.
Regarding New Relic, we don't have WAN access on server, since its a closed environment at telco data center. Also we see the behavior remains the same if we add multiple FS serve by same web server. So, i think problem is on FS side rather then web server side.
Thank you.
On Wed, Jun 12, 2013 at 4:31 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
Curious, I'm wondering if maybe excessive context switching is causing some issues.. I'm wondering if the latency is actually caused by the overheads on mod_xml_curl.
Could you try putting New Relic on your application, and see what sort of response times you get? That will identify which queries are taking a while to process.
Cal
On Wed, Jun 12, 2013 at 3:14 PM, Muhammad Shahzad <shaheryarkh@gmail.com
wrote:
We are already monitoring using top and htop. Even per core view of top shows no core every goes beyond 40%, average varies between 21 - 33%.
Thank you.
On Wed, Jun 12, 2013 at 4:02 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
You could either throw Cacti on the server, or use server density/new relic trial for quick and easy graphing.. or you can just watch `top` and push number 1 whilst running to see the CPU breakdown.
Cal
On Wed, Jun 12, 2013 at 1:19 PM, Muhammad Shahzad < shaheryarkh@gmail.com> wrote:
Steven, i am using soft timer. Should i try another? Please suggest.
Cal, no there is nothing unusually changes in RAM or CPU. BTW, i have 24 CPU cores with 16 GB of RAM on each server. Regarding CPU graph, how can i enable it.
Thank you.
On Wed, Jun 12, 2013 at 1:15 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
> This sounds quite interesting.. From my own benchmarks, I've seen > that IVRs applications can be quite stressful on the server. > > Have you checked to see if there is a CPU util spike or context > switching spike during these tests? Perhaps you are hitting the floor limit > of what your CPU is capable of? > > Do you have CPU graphs? If not, can you enable them and try again? > > Cal > > > On Wed, Jun 12, 2013 at 10:21 AM, Muhammad Shahzad < > shaheryarkh@gmail.com> wrote: > >> You are right. After many improvements and modifications to dail >> plan, removing unnecessary modules and using Ngnix web server, things are >> much better now. Except that directory information is fetched 3 times per >> call and there are overall 4 HTTP hits in a single call (3 for user + 1 for >> dial plan). >> >> Since Monday, we have started analyzing the results of load test, >> specially looking into packet traces, voicemail recordings and so on. We >> observed a strange thing, although we were able to test 1536 concurrent >> calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls >> per second rate. But there is strange and unexplained delay in starting RTP >> from FS side. >> >> FS plays an IVR in early media state, then answers the call to >> directly start recording voicemail. The problem is if call rate is under 25 >> CPS, everything works fine. At 25 CPS there is minor delay in starting RTP >> in early media state. This delay is negligible after processing few hundred >> thousand calls but after about 1.2 million calls the delay has creeps up to >> about 1.2 seconds. After 30 CPS this delay is significantly increases to 2 >> second after around 400 calls had done. At 50 CPS its ~3 second and after >> 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in >> starting playing IVR in early media state, once the IVR starts playing >> everything works perfectly fine. No call fails, but this delay would be >> annoying for anyone using the servers with over 30 CPS. >> >> Just to let you know, we checked everything, system resources, >> bandwidth usage, IVR file formats etc. all seems to be fine. We even tested >> with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing >> seems to have any impact on the results. >> >> I have run of out of possible explanations, so calling you guys for >> help. >> >> Thank you. >> >> >> >> >> On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <mike@jerris.com>wrote: >> >>> I would need to see the specifics of your dialplan, but I >>> suspect you have some transfers or execute extension that are causing it to >>> re-enter dialplan. Are you sure on all the others that they are making the >>> exact same requests? Details here will be necessary to understand exactly >>> what is going on.. >>> >>> Mike >>> >>> On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad < >>> shaheryarkh@gmail.com> wrote: >>> >>> Guy, after detailed analysis i can confirm it was an Apache >>> issue, and changing Apache configuration changes the concurrent call limit. >>> However, one thing i observed about xml curl is that it makes several >>> request with same parameters per call. Even if we have no calls running on >>> server, as well as as Apache is completely free. I can see FS makes at >>> least >>> >>> 1). 3 hits for dialplan. >>> 2). 2 hits for directory. >>> 3). 4 hits for voicemail. >>> 4). 3 hits for ivr. >>> 5). 2 hits for sofia. >>> and so on.. >>> >>> While ideally only one hit should be sufficient to get all these >>> configurations. What is the point of making all these redundant hits? We >>> have roughly 15 - 20 hits made per call, which is huge and causes >>> significant load on Apache. >>> >>> Both Apache and FS are running on dedicated machines and connected >>> over Gigabit Ethernet with zero packet loss, so i doubt these hits are >>> "re-transmission". >>> >>> Thank you. >>> >>> >>> >>> >>> On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad < >>> shaheryarkh@gmail.com> wrote: >>> >>>> Yes, its a feature, i know. I was just looking at how this >>>> feature was tested, as mentioned in its logs, >>>> >>>> 2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 >>>> Processing 21653XXXXX->121637XXXXX in context termination >>>> 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received HTTP >>>> error 0 trying to fetch http://192.168.0.2/gateway.php >>>> 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML >>>> response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml >>>> >>>> Thank you. >>>> >>>> >>>> >>>> >>>> On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris <mike@jerris.com >>>> > wrote: >>>> >>>>> This is adding a feature to allow timeout to be set in ms >>>>> instead of seconds. This is not a problem, its a feature, that was added a >>>>> while ago. Until you can determine if the request is being sent for the >>>>> one thats failing, your not going to solve this issue. >>>>> >>>>> Mike >>>>> >>>> >>> >>> >>> _________________________________________________________________________ >>> Professional FreeSWITCH Consulting Services: >>> consulting@freeswitch.org >>> http://www.freeswitchsolutions.com >>> >>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>> http://www.cudatel.com >>> >>> Official FreeSWITCH Sites >>> http://www.freeswitch.org >>> http://wiki.freeswitch.org >>> http://www.cluecon.com >>> >>> FreeSWITCH-users mailing list >>> FreeSWITCH-users@lists.freeswitch.org >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>> UNSUBSCRIBE: >>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>> http://www.freeswitch.org >>> >>> >> >> >> -- >> Mit freundlichen Grüßen >> Muhammad Shahzad >> ----------------------------------- >> CISCO Rich Media Communication Specialist (CRMCS) >> CISCO Certified Network Associate (CCNA) >> Cell: +49 176 99 83 10 85 >> MSN: shari_786pk@hotmail.com >> Email: shaheryarkh@googlemail.com >> >> >> _________________________________________________________________________ >> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org >> http://www.freeswitchsolutions.com >> >> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >> http://www.cudatel.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://wiki.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE: >> http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >> >> > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: > http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > >
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
Here is JIRA issue i have just opened, for future reference. http://jira.freeswitch.org/browse/FS-5508 Thank you. On Wed, Jun 12, 2013 at 7:40 PM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
Well, i am using xml curl, since its the only module i know that offers centralized FS configs management. I know there may be better, faster modules but they don't have benefit that xml curl gives us.
As for the system design, client's time and money blah blah, i did mentioned in in my first email that we have achieved our design goals and target capacity requirements etc. So far client is very happy with load test results. But on the technical side, i see over 70% of system resources are free and we can go beyond our achieved capacity. This has benefit for all of us (me, client and FS users), since, client will save cost for expansion (he has 50% growth projection in next 6-9 months), i will have better sizing calculation for future projects, thus lowering cost for future clients and normal FS users will have better understanding of how to scale FS (both horizontally and vertically), i will post results of my test once done with deployment.
Neither FS nor web server fail even for 1,536 concurrent calls at rate of 60 CPS, so its great success for both FS and xml curl in my view. Its only the initial delay in playing IVR which is kind of inconvenient from user experience point of view that i am trying to get rid of.
Anyways, i am opening up a JIRA ticket now. So core developers may look into it. Thanks to all of you who replied and help me in optimization the setup.
Thank you.
On Wed, Jun 12, 2013 at 6:38 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
Dave, with all due respect, every time the discussion of mod_xml_curl or anything non-windows comes up you immediately throw in a negative note, and it's starting to get a bit old now. Until one of the core devs steps up and says "you shouldn't be using mod_xml_curl" then I don't think it's right to be telling users that this is the wrong approach to be using - especially when we don't even know if it's mod_xml_curl that is causing the problem in this particular case. As mentioned in the last email, the next steps for OP is to identify the root cause of these delays, either via profiling or print statements.
OP - I think at this point, you really need to take this over to a JIRA ticket to avoid spamming the mailing list, as the thread is starting to go a bit stale.
Once you've created a ticket, let us know in this thread and we'll see about getting a core dev to give it some eyes.
If this is an urgent matter, you could also pay for a few hours of FS support which will get the immediate attention of a core dev :)
Cal
On Wed, Jun 12, 2013 at 5:19 PM, Dave R. Kompel <drk@drkngs.net> wrote:
** Ok, this "square peg in rouond hole" thread is not getting anything solved. You all need to take a step back, and look at the overall diesign of the applcation you are trying to build from a top down approch.
First define what the application has to do, with all of it's requirements, including load, resonse times, what information you need to look up per call, and so on...
This seems to me like a backwards approch was taken, as the start of the project, mod_xml_curl was the starting point, and then the application was designed after the method was picked. Isn't this "bass awards"? I know if I wasted my clients time and money by picking tech. before defining the problem, I would have no clients!
One thing I can tell you for sure, is that using a design that involves both client and server side synchronous technologies as the starting point, isn’t going to scale out to carrier rate call handling. You can try throwing all sorts of stuff like different web servers, caching proxies in the middle, and a handful of other kludges at it, and all you’re going to get is a big “Cluster F…”, that will be a nightmare to maintain and scale. You may also become a slave to maintaining it, and monitoring your production environment.******
One other thing to remember about FreeSWITCH, is that mod_xml_curl is not the only way of doing XML config binding to code. There are many modules that use the callback to implement feeding FS with real time XML config on demand. ****
It may be time to re-design your application, rather then trying to build a “better mousetrap” on the external side. We could discuss this on the conference call today, and get input from all the folks on a design, given your requirements**** --Dave
------------------------------ *From:* Muhammad Shahzad [mailto:shaheryarkh@gmail.com] *To:* FreeSWITCH Users Help [mailto: freeswitch-users@lists.freeswitch.org] *Sent:* Wed, 12 Jun 2013 07:57:18 -0700 *Subject:* Re: [Freeswitch-users] XML_CURL max connection limit
Well, per our observation, we don't see any load or delay from xml curl or web server side. Also the delay is occurring exactly at one location that is after sending 183 Early Media and sending first RTP packet of IVR. When CPS is below 25 this delay is less then 0.20 seconds but after 30 CPS it goes beyond 3.10 seconds.
Regarding New Relic, we don't have WAN access on server, since its a closed environment at telco data center. Also we see the behavior remains the same if we add multiple FS serve by same web server. So, i think problem is on FS side rather then web server side.
Thank you.
On Wed, Jun 12, 2013 at 4:31 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
Curious, I'm wondering if maybe excessive context switching is causing some issues.. I'm wondering if the latency is actually caused by the overheads on mod_xml_curl.
Could you try putting New Relic on your application, and see what sort of response times you get? That will identify which queries are taking a while to process.
Cal
On Wed, Jun 12, 2013 at 3:14 PM, Muhammad Shahzad < shaheryarkh@gmail.com> wrote:
We are already monitoring using top and htop. Even per core view of top shows no core every goes beyond 40%, average varies between 21 - 33%.
Thank you.
On Wed, Jun 12, 2013 at 4:02 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
You could either throw Cacti on the server, or use server density/new relic trial for quick and easy graphing.. or you can just watch `top` and push number 1 whilst running to see the CPU breakdown.
Cal
On Wed, Jun 12, 2013 at 1:19 PM, Muhammad Shahzad < shaheryarkh@gmail.com> wrote:
> Steven, i am using soft timer. Should i try another? Please suggest. > > Cal, no there is nothing unusually changes in RAM or CPU. BTW, i > have 24 CPU cores with 16 GB of RAM on each server. Regarding CPU graph, > how can i enable it. > > Thank you. > > > > On Wed, Jun 12, 2013 at 1:15 PM, Cal Leeming [Simplicity Media Ltd] > <cal.leeming@simplicitymedialtd.co.uk> wrote: > >> This sounds quite interesting.. From my own benchmarks, I've seen >> that IVRs applications can be quite stressful on the server. >> >> Have you checked to see if there is a CPU util spike or context >> switching spike during these tests? Perhaps you are hitting the floor limit >> of what your CPU is capable of? >> >> Do you have CPU graphs? If not, can you enable them and try again? >> >> Cal >> >> >> On Wed, Jun 12, 2013 at 10:21 AM, Muhammad Shahzad < >> shaheryarkh@gmail.com> wrote: >> >>> You are right. After many improvements and modifications to dail >>> plan, removing unnecessary modules and using Ngnix web server, things are >>> much better now. Except that directory information is fetched 3 times per >>> call and there are overall 4 HTTP hits in a single call (3 for user + 1 for >>> dial plan). >>> >>> Since Monday, we have started analyzing the results of load test, >>> specially looking into packet traces, voicemail recordings and so on. We >>> observed a strange thing, although we were able to test 1536 concurrent >>> calls (the limit we deliberately put in switch.xml.conf) at up to 64 calls >>> per second rate. But there is strange and unexplained delay in starting RTP >>> from FS side. >>> >>> FS plays an IVR in early media state, then answers the call to >>> directly start recording voicemail. The problem is if call rate is under 25 >>> CPS, everything works fine. At 25 CPS there is minor delay in starting RTP >>> in early media state. This delay is negligible after processing few hundred >>> thousand calls but after about 1.2 million calls the delay has creeps up to >>> about 1.2 seconds. After 30 CPS this delay is significantly increases to 2 >>> second after around 400 calls had done. At 50 CPS its ~3 second and after >>> 60 CPS its around 4 seconds. Interesting thing is the delay is ONLY in >>> starting playing IVR in early media state, once the IVR starts playing >>> everything works perfectly fine. No call fails, but this delay would be >>> annoying for anyone using the servers with over 30 CPS. >>> >>> Just to let you know, we checked everything, system resources, >>> bandwidth usage, IVR file formats etc. all seems to be fine. We even tested >>> with various FS start up options, e.g. -rp -nosql -nonat etc. etc. Nothing >>> seems to have any impact on the results. >>> >>> I have run of out of possible explanations, so calling you guys >>> for help. >>> >>> Thank you. >>> >>> >>> >>> >>> On Fri, Jun 7, 2013 at 5:20 PM, Michael Jerris <mike@jerris.com>wrote: >>> >>>> I would need to see the specifics of your dialplan, but I >>>> suspect you have some transfers or execute extension that are causing it to >>>> re-enter dialplan. Are you sure on all the others that they are making the >>>> exact same requests? Details here will be necessary to understand exactly >>>> what is going on.. >>>> >>>> Mike >>>> >>>> On Jun 7, 2013, at 4:07 AM, Muhammad Shahzad < >>>> shaheryarkh@gmail.com> wrote: >>>> >>>> Guy, after detailed analysis i can confirm it was an Apache >>>> issue, and changing Apache configuration changes the concurrent call limit. >>>> However, one thing i observed about xml curl is that it makes several >>>> request with same parameters per call. Even if we have no calls running on >>>> server, as well as as Apache is completely free. I can see FS makes at >>>> least >>>> >>>> 1). 3 hits for dialplan. >>>> 2). 2 hits for directory. >>>> 3). 4 hits for voicemail. >>>> 4). 3 hits for ivr. >>>> 5). 2 hits for sofia. >>>> and so on.. >>>> >>>> While ideally only one hit should be sufficient to get all these >>>> configurations. What is the point of making all these redundant hits? We >>>> have roughly 15 - 20 hits made per call, which is huge and causes >>>> significant load on Apache. >>>> >>>> Both Apache and FS are running on dedicated machines and >>>> connected over Gigabit Ethernet with zero packet loss, so i doubt these >>>> hits are "re-transmission". >>>> >>>> Thank you. >>>> >>>> >>>> >>>> >>>> On Thu, Jun 6, 2013 at 1:24 AM, Muhammad Shahzad < >>>> shaheryarkh@gmail.com> wrote: >>>> >>>>> Yes, its a feature, i know. I was just looking at how this >>>>> feature was tested, as mentioned in its logs, >>>>> >>>>> 2010-01-04 15:01:51.945020 [INFO] mod_dialplan_xml.c:408 >>>>> Processing 21653XXXXX->121637XXXXX in context termination >>>>> 2010-01-04 15:01:52.054466 [ERR] mod_xml_curl.c:310 Received >>>>> HTTP error 0 trying to fetch http://192.168.0.2/gateway.php >>>>> 2010-01-04 15:01:52.054545 [CONSOLE] mod_xml_curl.c:317 XML >>>>> response is in /tmp/6632ebb1-d19f-4d51-a07e-ec2ce0387f83.tmp.xml >>>>> >>>>> Thank you. >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, Jun 6, 2013 at 12:48 AM, Michael Jerris < >>>>> mike@jerris.com> wrote: >>>>> >>>>>> This is adding a feature to allow timeout to be set in ms >>>>>> instead of seconds. This is not a problem, its a feature, that was added a >>>>>> while ago. Until you can determine if the request is being sent for the >>>>>> one thats failing, your not going to solve this issue. >>>>>> >>>>>> Mike >>>>>> >>>>> >>>> >>>> >>>> _________________________________________________________________________ >>>> Professional FreeSWITCH Consulting Services: >>>> consulting@freeswitch.org >>>> http://www.freeswitchsolutions.com >>>> >>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>>> http://www.cudatel.com >>>> >>>> Official FreeSWITCH Sites >>>> http://www.freeswitch.org >>>> http://wiki.freeswitch.org >>>> http://www.cluecon.com >>>> >>>> FreeSWITCH-users mailing list >>>> FreeSWITCH-users@lists.freeswitch.org >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>> UNSUBSCRIBE: >>>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>>> http://www.freeswitch.org >>>> >>>> >>> >>> >>> -- >>> Mit freundlichen Grüßen >>> Muhammad Shahzad >>> ----------------------------------- >>> CISCO Rich Media Communication Specialist (CRMCS) >>> CISCO Certified Network Associate (CCNA) >>> Cell: +49 176 99 83 10 85 >>> MSN: shari_786pk@hotmail.com >>> Email: shaheryarkh@googlemail.com >>> >>> >>> _________________________________________________________________________ >>> Professional FreeSWITCH Consulting Services: >>> consulting@freeswitch.org >>> http://www.freeswitchsolutions.com >>> >>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>> http://www.cudatel.com >>> >>> Official FreeSWITCH Sites >>> http://www.freeswitch.org >>> http://wiki.freeswitch.org >>> http://www.cluecon.com >>> >>> FreeSWITCH-users mailing list >>> FreeSWITCH-users@lists.freeswitch.org >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>> UNSUBSCRIBE: >>> http://lists.freeswitch.org/mailman/options/freeswitch-users >>> http://www.freeswitch.org >>> >>> >> >> >> _________________________________________________________________________ >> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org >> http://www.freeswitchsolutions.com >> >> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >> http://www.cudatel.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://wiki.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE: >> http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >> >> > > > -- > Mit freundlichen Grüßen > Muhammad Shahzad > ----------------------------------- > CISCO Rich Media Communication Specialist (CRMCS) > CISCO Certified Network Associate (CCNA) > Cell: +49 176 99 83 10 85 > MSN: shari_786pk@hotmail.com > Email: shaheryarkh@googlemail.com > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: > http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > >
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
On 06/05/2013 07:36 PM, Michael Jerris wrote:
Is the request going to your server and getting no response?
On Jun 5, 2013, at 9:50 AM, Muhammad Shahzad <shaheryarkh@gmail.com <mailto:shaheryarkh@gmail.com>> wrote:
Hi,
Is there any connection limit perhaps hard coded anywhere in xml_curl module? I am load testing FS with xml_curl to provide config, directory and dial-plan. I observed i start getting HTTP ERROR 0 when i have around 1600 concurrent calls on machine with media. Although i see no sign of system overload either on FS (load average < 1.8) or Apache (load average < 4,75). Both have plenty of RAM and bandwidth free.
Is there anyway to get rid of this HTTP ERROR 0? It must be generated by FS internally, since i don't see anything like this error in Apache logs.
Below 1,638 concurrent calls @ 50 CPS, i get success rate of 99.999% but as soon as it crosses this mark, success rate drastically drops to 60% and even below (when concurrent calls go beyond 2,000). Here is error console log.
################################################### 2013-06-05 18:09:13.591229 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://172.16.171.61/vm-users.php data: [hostname=vms01§ion=directory&tag_name=domain&key_name=name&key_value=10.1.3.2&Event-Name=GENERAL&Core-UUID=84c917dc-cdde-11e2-9bb2-8b779d6a4e97&FreeSWITCH-Hostname=vms01&FreeSWITCH-Switchname=vms01&FreeSWITCH-IPv4=10.1.3.2&FreeSWITCH-IPv6=%3A%3A1&Event-Date-Local=2013-06-05%2018%3A09%3A03&Event-Date-GMT=Wed,%2005%20Jun%202013%2013%3A09%3A03%20GMT&Event-Date-Timestamp=1370437743591210&Event-Calling-File=mod_voicemail.c&Event-Calling-Function=resolve_id&Event-Calling-Line-Number=1350&Event-Sequence=1776930&action=message-count&key=id&user=989000049598&domain=10.1.3.2] ###################################################
Although this setup is more then sufficient for my requirement but i just want to know if i can extend it beyond, since i still have lot of resources free.
OS: Debian 6.0.7 x64 Apache: 2.2.16 FreeSWITCH: 1.5.1b+git~20130424T005233Z~66d1755b06
Both FS and Apache are running on separate dedicated machines on same LAN.
Thank you.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I think you need to do something with Apache configuration. Maybe increase apache child instance/Threads will give you better performance. -- Thanks, Sam
I have already set MaxClients to 2048, but that limit is never reached. Thank you. On Wed, Jun 5, 2013 at 4:17 PM, Samir Doshi <smrdoshi@gmail.com> wrote:
On 06/05/2013 07:36 PM, Michael Jerris wrote:
Is the request going to your server and getting no response?
On Jun 5, 2013, at 9:50 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
Hi,
Is there any connection limit perhaps hard coded anywhere in xml_curl module? I am load testing FS with xml_curl to provide config, directory and dial-plan. I observed i start getting HTTP ERROR 0 when i have around 1600 concurrent calls on machine with media. Although i see no sign of system overload either on FS (load average < 1.8) or Apache (load average < 4,75). Both have plenty of RAM and bandwidth free.
Is there anyway to get rid of this HTTP ERROR 0? It must be generated by FS internally, since i don't see anything like this error in Apache logs.
Below 1,638 concurrent calls @ 50 CPS, i get success rate of 99.999% but as soon as it crosses this mark, success rate drastically drops to 60% and even below (when concurrent calls go beyond 2,000). Here is error console log.
################################################### 2013-06-05 18:09:13.591229 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://172.16.171.61/vm-users.php data: [hostname=vms01§ion=directory&tag_name=domain&key_name=name&key_value=10.1.3.2&Event-Name=GENERAL&Core-UUID=84c917dc-cdde-11e2-9bb2-8b779d6a4e97&FreeSWITCH-Hostname=vms01&FreeSWITCH-Switchname=vms01&FreeSWITCH-IPv4=10.1.3.2&FreeSWITCH-IPv6=%3A%3A1&Event-Date-Local=2013-06-05%2018%3A09%3A03&Event-Date-GMT=Wed,%2005%20Jun%202013%2013%3A09%3A03%20GMT&Event-Date-Timestamp=1370437743591210&Event-Calling-File=mod_voicemail.c&Event-Calling-Function=resolve_id&Event-Calling-Line-Number=1350&Event-Sequence=1776930&action=message-count&key=id&user=989000049598&domain=10.1.3.2] ###################################################
Although this setup is more then sufficient for my requirement but i just want to know if i can extend it beyond, since i still have lot of resources free.
OS: Debian 6.0.7 x64 Apache: 2.2.16 FreeSWITCH: 1.5.1b+git~20130424T005233Z~66d1755b06
Both FS and Apache are running on separate dedicated machines on same LAN.
Thank you.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services:consulting@freeswitch.orghttp://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Serverhttp://www.cudatel.com
Official FreeSWITCH Siteshttp://www.freeswitch.orghttp://wiki.freeswitch.orghttp://www.cluecon.com
FreeSWITCH-users mailing listFreeSWITCH-users@lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.frees...
I think you need to do something with Apache configuration. Maybe increase apache child instance/Threads will give you better performance.
-- Thanks, Sam
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
Hello, This isn't really on topic within FreeSWITCH and is more related to your web application scalability. However, semi high traffic web application platforms is one area I do have a decent amount of experience in, so let me offer some thoughts (I assume you are running PHP based on your log output). I would advise looking into the following; * PHP FPM with NGINX (you'll get better throughput with this than mod_php and apache - look on google for benchmarks) * APC or xcache opcode acceleration (out of the box, this should speed up execution) * Assuming you have a DB, implement a caching library to reduce DB calls (look at memcache, redis if you need atomicity or scaling, or even just xcache) * Implement haproxy/zxtm and balance across multiple web servers (assuming your code isn't vuln to race, and you are able to switch to a db/cache based session backend, rather than disk) * Get new relic on your boxes, identify the bottlenecks, and performance tweak where necessary * If you stay on apache, don't just increase maxclient's... you need to calculate how much resources the box needs to handle that many concurrent requests, and identify the max memory space needed for each request, cpu time etc. * Assuming you use MySQL, switch over to Percona, use InnoDB, and separate out read/writes from the same server (you can do this in master<>master or master<>slave, or any combination you like). This will reduce the innodb thread contention and give you much better throughput Hope this helps Cal On Wed, Jun 5, 2013 at 3:47 PM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
I have already set MaxClients to 2048, but that limit is never reached.
Thank you.
On Wed, Jun 5, 2013 at 4:17 PM, Samir Doshi <smrdoshi@gmail.com> wrote:
On 06/05/2013 07:36 PM, Michael Jerris wrote:
Is the request going to your server and getting no response?
On Jun 5, 2013, at 9:50 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
Hi,
Is there any connection limit perhaps hard coded anywhere in xml_curl module? I am load testing FS with xml_curl to provide config, directory and dial-plan. I observed i start getting HTTP ERROR 0 when i have around 1600 concurrent calls on machine with media. Although i see no sign of system overload either on FS (load average < 1.8) or Apache (load average < 4,75). Both have plenty of RAM and bandwidth free.
Is there anyway to get rid of this HTTP ERROR 0? It must be generated by FS internally, since i don't see anything like this error in Apache logs.
Below 1,638 concurrent calls @ 50 CPS, i get success rate of 99.999% but as soon as it crosses this mark, success rate drastically drops to 60% and even below (when concurrent calls go beyond 2,000). Here is error console log.
################################################### 2013-06-05 18:09:13.591229 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://172.16.171.61/vm-users.php data: [hostname=vms01§ion=directory&tag_name=domain&key_name=name&key_value=10.1.3.2&Event-Name=GENERAL&Core-UUID=84c917dc-cdde-11e2-9bb2-8b779d6a4e97&FreeSWITCH-Hostname=vms01&FreeSWITCH-Switchname=vms01&FreeSWITCH-IPv4=10.1.3.2&FreeSWITCH-IPv6=%3A%3A1&Event-Date-Local=2013-06-05%2018%3A09%3A03&Event-Date-GMT=Wed,%2005%20Jun%202013%2013%3A09%3A03%20GMT&Event-Date-Timestamp=1370437743591210&Event-Calling-File=mod_voicemail.c&Event-Calling-Function=resolve_id&Event-Calling-Line-Number=1350&Event-Sequence=1776930&action=message-count&key=id&user=989000049598&domain=10.1.3.2] ###################################################
Although this setup is more then sufficient for my requirement but i just want to know if i can extend it beyond, since i still have lot of resources free.
OS: Debian 6.0.7 x64 Apache: 2.2.16 FreeSWITCH: 1.5.1b+git~20130424T005233Z~66d1755b06
Both FS and Apache are running on separate dedicated machines on same LAN.
Thank you.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services:consulting@freeswitch.orghttp://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Serverhttp://www.cudatel.com
Official FreeSWITCH Siteshttp://www.freeswitch.orghttp://wiki.freeswitch.orghttp://www.cluecon.com
FreeSWITCH-users mailing listFreeSWITCH-users@lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.frees...
I think you need to do something with Apache configuration. Maybe increase apache child instance/Threads will give you better performance.
-- Thanks, Sam
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Thanks Cal, let me discuss it with my team and see what can be implemented quickly, since we are running short on project deadline. Thank you. On Wed, Jun 5, 2013 at 5:29 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
Hello,
This isn't really on topic within FreeSWITCH and is more related to your web application scalability.
However, semi high traffic web application platforms is one area I do have a decent amount of experience in, so let me offer some thoughts (I assume you are running PHP based on your log output).
I would advise looking into the following;
* PHP FPM with NGINX (you'll get better throughput with this than mod_php and apache - look on google for benchmarks) * APC or xcache opcode acceleration (out of the box, this should speed up execution) * Assuming you have a DB, implement a caching library to reduce DB calls (look at memcache, redis if you need atomicity or scaling, or even just xcache) * Implement haproxy/zxtm and balance across multiple web servers (assuming your code isn't vuln to race, and you are able to switch to a db/cache based session backend, rather than disk) * Get new relic on your boxes, identify the bottlenecks, and performance tweak where necessary * If you stay on apache, don't just increase maxclient's... you need to calculate how much resources the box needs to handle that many concurrent requests, and identify the max memory space needed for each request, cpu time etc. * Assuming you use MySQL, switch over to Percona, use InnoDB, and separate out read/writes from the same server (you can do this in master<>master or master<>slave, or any combination you like). This will reduce the innodb thread contention and give you much better throughput
Hope this helps
Cal
On Wed, Jun 5, 2013 at 3:47 PM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
I have already set MaxClients to 2048, but that limit is never reached.
Thank you.
On Wed, Jun 5, 2013 at 4:17 PM, Samir Doshi <smrdoshi@gmail.com> wrote:
On 06/05/2013 07:36 PM, Michael Jerris wrote:
Is the request going to your server and getting no response?
On Jun 5, 2013, at 9:50 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
Hi,
Is there any connection limit perhaps hard coded anywhere in xml_curl module? I am load testing FS with xml_curl to provide config, directory and dial-plan. I observed i start getting HTTP ERROR 0 when i have around 1600 concurrent calls on machine with media. Although i see no sign of system overload either on FS (load average < 1.8) or Apache (load average < 4,75). Both have plenty of RAM and bandwidth free.
Is there anyway to get rid of this HTTP ERROR 0? It must be generated by FS internally, since i don't see anything like this error in Apache logs.
Below 1,638 concurrent calls @ 50 CPS, i get success rate of 99.999% but as soon as it crosses this mark, success rate drastically drops to 60% and even below (when concurrent calls go beyond 2,000). Here is error console log.
################################################### 2013-06-05 18:09:13.591229 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://172.16.171.61/vm-users.php data: [hostname=vms01§ion=directory&tag_name=domain&key_name=name&key_value=10.1.3.2&Event-Name=GENERAL&Core-UUID=84c917dc-cdde-11e2-9bb2-8b779d6a4e97&FreeSWITCH-Hostname=vms01&FreeSWITCH-Switchname=vms01&FreeSWITCH-IPv4=10.1.3.2&FreeSWITCH-IPv6=%3A%3A1&Event-Date-Local=2013-06-05%2018%3A09%3A03&Event-Date-GMT=Wed,%2005%20Jun%202013%2013%3A09%3A03%20GMT&Event-Date-Timestamp=1370437743591210&Event-Calling-File=mod_voicemail.c&Event-Calling-Function=resolve_id&Event-Calling-Line-Number=1350&Event-Sequence=1776930&action=message-count&key=id&user=989000049598&domain=10.1.3.2] ###################################################
Although this setup is more then sufficient for my requirement but i just want to know if i can extend it beyond, since i still have lot of resources free.
OS: Debian 6.0.7 x64 Apache: 2.2.16 FreeSWITCH: 1.5.1b+git~20130424T005233Z~66d1755b06
Both FS and Apache are running on separate dedicated machines on same LAN.
Thank you.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services:consulting@freeswitch.orghttp://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Serverhttp://www.cudatel.com
Official FreeSWITCH Siteshttp://www.freeswitch.orghttp://wiki.freeswitch.orghttp://www.cluecon.com
FreeSWITCH-users mailing listFreeSWITCH-users@lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.frees...
I think you need to do something with Apache configuration. Maybe increase apache child instance/Threads will give you better performance.
-- Thanks, Sam
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
Also, make sure you don't mix long and short running queries, else you will thrash the innodb thread queue. If you want complete HA, then go for 2x master<>master, and 4x slave (in HA pairs). If you need a ghetto solution, go for 1 master, 1 slave for short queries, 1 slave for long. If you need a really ghetto solution, throw some SSDs on the master and tweak the innodb conf, you *might* get away with it but YMMV. Cal On Wed, Jun 5, 2013 at 4:45 PM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
Thanks Cal, let me discuss it with my team and see what can be implemented quickly, since we are running short on project deadline.
Thank you.
On Wed, Jun 5, 2013 at 5:29 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
Hello,
This isn't really on topic within FreeSWITCH and is more related to your web application scalability.
However, semi high traffic web application platforms is one area I do have a decent amount of experience in, so let me offer some thoughts (I assume you are running PHP based on your log output).
I would advise looking into the following;
* PHP FPM with NGINX (you'll get better throughput with this than mod_php and apache - look on google for benchmarks) * APC or xcache opcode acceleration (out of the box, this should speed up execution) * Assuming you have a DB, implement a caching library to reduce DB calls (look at memcache, redis if you need atomicity or scaling, or even just xcache) * Implement haproxy/zxtm and balance across multiple web servers (assuming your code isn't vuln to race, and you are able to switch to a db/cache based session backend, rather than disk) * Get new relic on your boxes, identify the bottlenecks, and performance tweak where necessary * If you stay on apache, don't just increase maxclient's... you need to calculate how much resources the box needs to handle that many concurrent requests, and identify the max memory space needed for each request, cpu time etc. * Assuming you use MySQL, switch over to Percona, use InnoDB, and separate out read/writes from the same server (you can do this in master<>master or master<>slave, or any combination you like). This will reduce the innodb thread contention and give you much better throughput
Hope this helps
Cal
On Wed, Jun 5, 2013 at 3:47 PM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
I have already set MaxClients to 2048, but that limit is never reached.
Thank you.
On Wed, Jun 5, 2013 at 4:17 PM, Samir Doshi <smrdoshi@gmail.com> wrote:
On 06/05/2013 07:36 PM, Michael Jerris wrote:
Is the request going to your server and getting no response?
On Jun 5, 2013, at 9:50 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
Hi,
Is there any connection limit perhaps hard coded anywhere in xml_curl module? I am load testing FS with xml_curl to provide config, directory and dial-plan. I observed i start getting HTTP ERROR 0 when i have around 1600 concurrent calls on machine with media. Although i see no sign of system overload either on FS (load average < 1.8) or Apache (load average < 4,75). Both have plenty of RAM and bandwidth free.
Is there anyway to get rid of this HTTP ERROR 0? It must be generated by FS internally, since i don't see anything like this error in Apache logs.
Below 1,638 concurrent calls @ 50 CPS, i get success rate of 99.999% but as soon as it crosses this mark, success rate drastically drops to 60% and even below (when concurrent calls go beyond 2,000). Here is error console log.
################################################### 2013-06-05 18:09:13.591229 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://172.16.171.61/vm-users.php data: [hostname=vms01§ion=directory&tag_name=domain&key_name=name&key_value=10.1.3.2&Event-Name=GENERAL&Core-UUID=84c917dc-cdde-11e2-9bb2-8b779d6a4e97&FreeSWITCH-Hostname=vms01&FreeSWITCH-Switchname=vms01&FreeSWITCH-IPv4=10.1.3.2&FreeSWITCH-IPv6=%3A%3A1&Event-Date-Local=2013-06-05%2018%3A09%3A03&Event-Date-GMT=Wed,%2005%20Jun%202013%2013%3A09%3A03%20GMT&Event-Date-Timestamp=1370437743591210&Event-Calling-File=mod_voicemail.c&Event-Calling-Function=resolve_id&Event-Calling-Line-Number=1350&Event-Sequence=1776930&action=message-count&key=id&user=989000049598&domain=10.1.3.2] ###################################################
Although this setup is more then sufficient for my requirement but i just want to know if i can extend it beyond, since i still have lot of resources free.
OS: Debian 6.0.7 x64 Apache: 2.2.16 FreeSWITCH: 1.5.1b+git~20130424T005233Z~66d1755b06
Both FS and Apache are running on separate dedicated machines on same LAN.
Thank you.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services:consulting@freeswitch.orghttp://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Serverhttp://www.cudatel.com
Official FreeSWITCH Siteshttp://www.freeswitch.orghttp://wiki.freeswitch.orghttp://www.cluecon.com
FreeSWITCH-users mailing listFreeSWITCH-users@lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.frees...
I think you need to do something with Apache configuration. Maybe increase apache child instance/Threads will give you better performance.
-- Thanks, Sam
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Oh, sorry one last thing, if your bottleneck is DB, you can be super ghetto and scale up now and worry about scaling out later. Scaling up basically means throwing as much capacity as possible on a single box, i.e. 800GB RAM, RAID with SSDs, dual quad core with HT etc etc. Throw as much memory into the chassis as possible so that the entire innodb buffer pool sits in memory, combined with SSDs, you should be able to get away with that. However as said before, YMMV, because you will be likely encounter other problems along the way - and this is really only a quick fix, you'd need to immediately work on a long term fix. And also - nothing beats doing things properly.. all the above are options to give you an immediate fix to keep production running.. you *must* still take time to properly re-design the architecture to support high throughput. Hope this helps Cal On Wed, Jun 5, 2013 at 5:40 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
Also, make sure you don't mix long and short running queries, else you will thrash the innodb thread queue.
If you want complete HA, then go for 2x master<>master, and 4x slave (in HA pairs).
If you need a ghetto solution, go for 1 master, 1 slave for short queries, 1 slave for long.
If you need a really ghetto solution, throw some SSDs on the master and tweak the innodb conf, you *might* get away with it but YMMV.
Cal
On Wed, Jun 5, 2013 at 4:45 PM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
Thanks Cal, let me discuss it with my team and see what can be implemented quickly, since we are running short on project deadline.
Thank you.
On Wed, Jun 5, 2013 at 5:29 PM, Cal Leeming [Simplicity Media Ltd] < cal.leeming@simplicitymedialtd.co.uk> wrote:
Hello,
This isn't really on topic within FreeSWITCH and is more related to your web application scalability.
However, semi high traffic web application platforms is one area I do have a decent amount of experience in, so let me offer some thoughts (I assume you are running PHP based on your log output).
I would advise looking into the following;
* PHP FPM with NGINX (you'll get better throughput with this than mod_php and apache - look on google for benchmarks) * APC or xcache opcode acceleration (out of the box, this should speed up execution) * Assuming you have a DB, implement a caching library to reduce DB calls (look at memcache, redis if you need atomicity or scaling, or even just xcache) * Implement haproxy/zxtm and balance across multiple web servers (assuming your code isn't vuln to race, and you are able to switch to a db/cache based session backend, rather than disk) * Get new relic on your boxes, identify the bottlenecks, and performance tweak where necessary * If you stay on apache, don't just increase maxclient's... you need to calculate how much resources the box needs to handle that many concurrent requests, and identify the max memory space needed for each request, cpu time etc. * Assuming you use MySQL, switch over to Percona, use InnoDB, and separate out read/writes from the same server (you can do this in master<>master or master<>slave, or any combination you like). This will reduce the innodb thread contention and give you much better throughput
Hope this helps
Cal
On Wed, Jun 5, 2013 at 3:47 PM, Muhammad Shahzad <shaheryarkh@gmail.com>wrote:
I have already set MaxClients to 2048, but that limit is never reached.
Thank you.
On Wed, Jun 5, 2013 at 4:17 PM, Samir Doshi <smrdoshi@gmail.com> wrote:
On 06/05/2013 07:36 PM, Michael Jerris wrote:
Is the request going to your server and getting no response?
On Jun 5, 2013, at 9:50 AM, Muhammad Shahzad <shaheryarkh@gmail.com> wrote:
Hi,
Is there any connection limit perhaps hard coded anywhere in xml_curl module? I am load testing FS with xml_curl to provide config, directory and dial-plan. I observed i start getting HTTP ERROR 0 when i have around 1600 concurrent calls on machine with media. Although i see no sign of system overload either on FS (load average < 1.8) or Apache (load average < 4,75). Both have plenty of RAM and bandwidth free.
Is there anyway to get rid of this HTTP ERROR 0? It must be generated by FS internally, since i don't see anything like this error in Apache logs.
Below 1,638 concurrent calls @ 50 CPS, i get success rate of 99.999% but as soon as it crosses this mark, success rate drastically drops to 60% and even below (when concurrent calls go beyond 2,000). Here is error console log.
################################################### 2013-06-05 18:09:13.591229 [ERR] mod_xml_curl.c:310 Received HTTP error 0 trying to fetch http://172.16.171.61/vm-users.php data: [hostname=vms01§ion=directory&tag_name=domain&key_name=name&key_value=10.1.3.2&Event-Name=GENERAL&Core-UUID=84c917dc-cdde-11e2-9bb2-8b779d6a4e97&FreeSWITCH-Hostname=vms01&FreeSWITCH-Switchname=vms01&FreeSWITCH-IPv4=10.1.3.2&FreeSWITCH-IPv6=%3A%3A1&Event-Date-Local=2013-06-05%2018%3A09%3A03&Event-Date-GMT=Wed,%2005%20Jun%202013%2013%3A09%3A03%20GMT&Event-Date-Timestamp=1370437743591210&Event-Calling-File=mod_voicemail.c&Event-Calling-Function=resolve_id&Event-Calling-Line-Number=1350&Event-Sequence=1776930&action=message-count&key=id&user=989000049598&domain=10.1.3.2] ###################################################
Although this setup is more then sufficient for my requirement but i just want to know if i can extend it beyond, since i still have lot of resources free.
OS: Debian 6.0.7 x64 Apache: 2.2.16 FreeSWITCH: 1.5.1b+git~20130424T005233Z~66d1755b06
Both FS and Apache are running on separate dedicated machines on same LAN.
Thank you.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services:consulting@freeswitch.orghttp://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Serverhttp://www.cudatel.com
Official FreeSWITCH Siteshttp://www.freeswitch.orghttp://wiki.freeswitch.orghttp://www.cluecon.com
FreeSWITCH-users mailing listFreeSWITCH-users@lists.freeswitch.orghttp://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-usershttp://www.frees...
I think you need to do something with Apache configuration. Maybe increase apache child instance/Threads will give you better performance.
-- Thanks, Sam
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Mit freundlichen Grüßen Muhammad Shahzad ----------------------------------- CISCO Rich Media Communication Specialist (CRMCS) CISCO Certified Network Associate (CCNA) Cell: +49 176 99 83 10 85 MSN: shari_786pk@hotmail.com Email: shaheryarkh@googlemail.com
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (7)
-
Cal Leeming [Simplicity Media Ltd]
-
Dave R. Kompel
-
Jeff Leung
-
Michael Jerris
-
Muhammad Shahzad
-
Samir Doshi
-
Steven Ayre