Oh, sorry one last thing, if your bottleneck is DB, you can be super ghetto and scale up now and worry about scaling out later.<div><br></div><div>Scaling up basically means throwing as much capacity as possible on a single box, i.e. 800GB RAM, RAID with SSDs, dual quad core with HT etc etc.</div>
<div><br></div><div>Throw as much memory into the chassis as possible so that the entire innodb buffer pool sits in memory, combined with SSDs, you should be able to get away with that.</div><div><br></div><div>However as said before, YMMV, because you will be likely encounter other problems along the way - and this is really only a quick fix, you'd need to immediately work on a long term fix. </div>
<div><br></div><div>And also - nothing beats doing things properly.. all the above are options to give you an immediate fix to keep production running.. you *must* still take time to properly re-design the architecture to support high throughput.</div>
<div><br></div><div>Hope this helps</div><div><br></div><div>Cal<br><br><div class="gmail_quote">On Wed, Jun 5, 2013 at 5:40 PM, Cal Leeming [Simplicity Media Ltd] <span dir="ltr"><<a href="mailto:cal.leeming@simplicitymedialtd.co.uk" target="_blank">cal.leeming@simplicitymedialtd.co.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Also, make sure you don't mix long and short running queries, else you will thrash the innodb thread queue.<div><br>
</div><div>If you want complete HA, then go for 2x master<>master, and 4x slave (in HA pairs).</div>
<div><br></div><div>If you need a ghetto solution, go for 1 master, 1 slave for short queries, 1 slave for long.</div><div><br>If you need a really ghetto solution, throw some SSDs on the master and tweak the innodb conf, you *might* get away with it but YMMV.</div>
<span class="HOEnZb"><font color="#888888">
<div><br></div></font></span><div><span class="HOEnZb"><font color="#888888">Cal</font></span><div><div class="h5"><br><br><div class="gmail_quote">On Wed, Jun 5, 2013 at 4:45 PM, Muhammad Shahzad <span dir="ltr"><<a href="mailto:shaheryarkh@gmail.com" target="_blank">shaheryarkh@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks Cal, let me discuss it with my team and see what can be implemented quickly, since we are running short on project deadline.<div>
<br></div><div>Thank you.</div><div><br></div></div><div><div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Wed, Jun 5, 2013 at 5:29 PM, Cal Leeming [Simplicity Media Ltd] <span dir="ltr"><<a href="mailto:cal.leeming@simplicitymedialtd.co.uk" target="_blank">cal.leeming@simplicitymedialtd.co.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<div><br></div><div>This isn't really on topic within FreeSWITCH and is more related to your web application scalability.</div>
<div><br></div><div>However, semi high traffic web application platforms is one area I do have a decent amount of experience in, so let me offer some thoughts (I assume you are running PHP based on your log output).</div>
<div><br></div><div>I would advise looking into the following;</div><div><br></div><div>* PHP FPM with NGINX (you'll get better throughput with this than mod_php and apache - look on google for benchmarks)</div><div>
* APC or xcache opcode acceleration (out of the box, this should speed up execution)</div>
<div>* Assuming you have a DB, implement a caching library to reduce DB calls (look at memcache, redis if you need atomicity or scaling, or even just xcache)</div><div>* Implement haproxy/zxtm and balance across multiple web servers (assuming your code isn't vuln to race, and you are able to switch to a db/cache based session backend, rather than disk)</div>
<div>* Get new relic on your boxes, identify the bottlenecks, and performance tweak where necessary</div><div>* If you stay on apache, don't just increase maxclient's... you need to calculate how much resources the box needs to handle that many concurrent requests, and identify the max memory space needed for each request, cpu time etc.</div>
<div>* Assuming you use MySQL, switch over to Percona, use InnoDB, and separate out read/writes from the same server (you can do this in master<>master or master<>slave, or any combination you like). This will reduce the innodb thread contention and give you much better throughput</div>
<div><br></div><div>Hope this helps</div><span><font color="#888888"><div><br></div><div>Cal</div></font></span><div><div><div><br><div class="gmail_quote">On Wed, Jun 5, 2013 at 3:47 PM, Muhammad Shahzad <span dir="ltr"><<a href="mailto:shaheryarkh@gmail.com" target="_blank">shaheryarkh@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I have already set MaxClients to 2048, but that limit is never reached.<div><br></div><div>Thank you.</div>
</div><div class="gmail_extra"><div><div><br><br><div class="gmail_quote">On Wed, Jun 5, 2013 at 4:17 PM, Samir Doshi <span dir="ltr"><<a href="mailto:smrdoshi@gmail.com" target="_blank">smrdoshi@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><div><div>
<div>On 06/05/2013 07:36 PM, Michael Jerris
wrote:<br>
</div>
</div></div><blockquote type="cite"><div><div>
Is the request going to your server and getting no response?
<div><br>
<div>
<div>On Jun 5, 2013, at 9:50 AM, Muhammad Shahzad <<a href="mailto:shaheryarkh@gmail.com" target="_blank">shaheryarkh@gmail.com</a>>
wrote:</div>
<br>
<blockquote type="cite">
<div dir="ltr">Hi,
<div><br>
</div>
<div>Is there any connection limit perhaps hard coded
anywhere in xml_curl module? I am load testing FS with
xml_curl to provide config, directory and dial-plan. I
observed i start getting HTTP ERROR 0 when i have around
1600 concurrent calls on machine with media. Although i
see no sign of system overload either on FS (load
average < 1.8) or Apache (load average < 4,75).
Both have plenty of RAM and bandwidth free.</div>
<div><br>
</div>
<div>Is there anyway to get rid of this HTTP ERROR 0? It
must be generated by FS internally, since i don't see
anything like this error in Apache logs.</div>
<div><br>
</div>
<div>Below 1,638 concurrent calls @ 50 CPS, i get success
rate of 99.999% but as soon as it crosses this mark,
success rate drastically drops to 60% and even below
(when concurrent calls go beyond 2,000). Here is error
console log.<br clear="all">
<div><br>
</div>
<div>###################################################</div>
<div>
<div>2013-06-05 18:09:13.591229 [ERR]
mod_xml_curl.c:310 Received HTTP error 0 trying to
fetch <a href="http://172.16.171.61/vm-users.php" target="_blank">http://172.16.171.61/vm-users.php</a></div>
<div>data:
[hostname=vms01&section=directory&tag_name=domain&key_name=name&key_value=10.1.3.2&Event-Name=GENERAL&Core-UUID=84c917dc-cdde-11e2-9bb2-8b779d6a4e97&FreeSWITCH-Hostname=vms01&FreeSWITCH-Switchname=vms01&FreeSWITCH-IPv4=10.1.3.2&FreeSWITCH-IPv6=%3A%3A1&Event-Date-Local=2013-06-05%2018%3A09%3A03&Event-Date-GMT=Wed,%2005%20Jun%202013%2013%3A09%3A03%20GMT&Event-Date-Timestamp=1370437743591210&Event-Calling-File=mod_voicemail.c&Event-Calling-Function=resolve_id&Event-Calling-Line-Number=1350&Event-Sequence=1776930&action=message-count&key=id&user=989000049598&domain=10.1.3.2]</div>
</div>
<div>###################################################<br>
</div>
<div><br>
</div>
<div>Although this setup is more then
sufficient for my requirement but i just want to know
if i can extend it beyond, since i still have lot of
resources free. <br>
</div>
<div><br>
</div>
<div>OS: Debian 6.0.7 x64</div>
<div>Apache: 2.2.16</div>
<div>FreeSWITCH: 1.5.1b+git~20130424T005233Z~66d1755b06</div>
<div><br>
</div>
<div>Both FS and Apache are running on separate
dedicated machines on same LAN.</div>
<div><br>
</div>
<div>Thank you.</div>
<div><br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
</div></div><div><pre>_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a>
Official FreeSWITCH Sites
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a>
FreeSWITCH-users mailing list
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a>
</pre>
</div></blockquote>
<br>
I think you need to do something with Apache configuration. Maybe
increase apache child instance/Threads will give you better
performance. <br><span><font color="#888888">
<pre cols="72">--
Thanks,
Sam</pre>
</font></span></div>
<br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div></div></div><div>-- <br><div><span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">Mit freundlichen Grüßen</span></div>
<span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">Muhammad Shahzad</span><br style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">------------------------------</span><span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">-----</span><br style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">CISCO Rich Media Communication Specialist (CRMCS)</span><br style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">CISCO Certified Network Associate (CCNA)</span><br style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">Cell: <a href="tel:%2B49%20176%2099%2083%2010%2085" value="+4917699831085" target="_blank">+49 176 99 83 10 85</a></span><br style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">MSN: </span><a href="mailto:shari_786pk@hotmail.com" style="color:rgb(17,85,204);font-size:13px;font-family:arial,sans-serif" target="_blank">shari_786pk@hotmail.com</a><br style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">Email: </span><a href="mailto:shaheryarkh@googlemail.com" style="color:rgb(17,85,204);font-size:13px;font-family:arial,sans-serif" target="_blank">shaheryarkh@googlemail.com</a>
</div></div>
<br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br></blockquote></div><br></div>
</div></div><br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div><span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">Mit freundlichen Grüßen</span></div>
<span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">Muhammad Shahzad</span><br style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">------------------------------</span><span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">-----</span><br style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">CISCO Rich Media Communication Specialist (CRMCS)</span><br style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">CISCO Certified Network Associate (CCNA)</span><br style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">Cell: <a href="tel:%2B49%20176%2099%2083%2010%2085" value="+4917699831085" target="_blank">+49 176 99 83 10 85</a></span><br style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">MSN: </span><a href="mailto:shari_786pk@hotmail.com" style="color:rgb(17,85,204);font-size:13px;font-family:arial,sans-serif" target="_blank">shari_786pk@hotmail.com</a><br style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">
<span style="color:rgb(136,136,136);font-size:13px;font-family:arial,sans-serif">Email: </span><a href="mailto:shaheryarkh@googlemail.com" style="color:rgb(17,85,204);font-size:13px;font-family:arial,sans-serif" target="_blank">shaheryarkh@googlemail.com</a>
</div>
</div></div><br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br></blockquote></div><br></div></div></div>
</blockquote></div><br></div>