<div dir="ltr">Ultimately it turned out to be one costly query, I had to fix that and did not see this issue since then. (running 500 calls).<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Nov 14, 2015 at 8:18 PM, John Nash <span dir="ltr"><<a href="mailto:john.nash778@gmail.com" target="_blank">john.nash778@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">one more test observation. I specified readtimeout to 10 seconds and on database server allocated only 50 connections limit to the user which is used in freeswitch and let the issue replicate and found that there is no query failure because of timeout, however dbh is unable to get new connections after a while.<br><br><br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Nov 14, 2015 at 5:44 AM, Michael Giagnocavo <span dir="ltr"><<a href="mailto:mgg@giagnocavo.net" target="_blank">mgg@giagnocavo.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="blue" vlink="purple" lang="EN-US">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Just consider your failure case: What do you do when the HTTP endpoint or DB is down? In most cases, if CDRs are really important, you’ll write to disk then queue
them up and insert them when the service is up again. Which makes ya wonder why bother supporting two paths in the first place? Just always write the CDRs to disk, then pick them up. Note that you’ll wanna patch xml-cdr as it just writes to the file in-place,
so you don’t know when it’s done. (It should instead write to a tmp file then move when finished.) Then queue them whatever way you like (with an actual message queue, or just with a loop that tries to insert the cdr and delete the file.)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">If you feel good enough you can always keep a local service up alongside FS, you might run a local DB or message queue and go direct from FS to that.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">-Michael<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a> [mailto:<a href="mailto:freeswitch-users-bounces@lists.freeswitch.org" target="_blank">freeswitch-users-bounces@lists.freeswitch.org</a>]
<b>On Behalf Of </b>John Nash<br>
<b>Sent:</b> Friday, November 13, 2015 12:49 PM<span><br>
<b>To:</b> FreeSWITCH Users Help <<a href="mailto:freeswitch-users@lists.freeswitch.org" target="_blank">freeswitch-users@lists.freeswitch.org</a>><br>
<b>Subject:</b> Re: [Freeswitch-users] Situation with freeswitch dbh + odbc (mysql) in lua<u></u><u></u></span></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Thank you very much for the comments. Yes it had to be the long running queries and the way lua handles connections, it creates a new connection if existing pool is all in use. Apart from dialplan lua to play
balance, I am also logging CDR in database in reporting hook (lua scripts again) which now I am thinking of scrapping , will XML-curl or XML-CDR be a better option to post CDR into remote database?</p><div><div><br>
<br>
Memcache is an excellent suggestion but I do not see any command in Lua api documentation on how to use it. I will do more searching.<u></u><u></u></div></div><p></p>
</div><div><div>
<p class="MsoNormal" style="margin-bottom:12.0pt">I also want to avoid calls stuck as CS_HIBERNATE (which i think is because in hangup hook I am using lua script to log CDR and in case DB is non responsive however I am using luarun so theoritically it should
not stuck right?)<u></u><u></u></p>
</div></div></div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div><div><div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Fri, Nov 13, 2015 at 7:55 PM, Luis Daniel Lucio Quiroz <<a href="mailto:luis.daniel.lucio@gmail.com" target="_blank">luis.daniel.lucio@gmail.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p>If they take too long go for indexing and memcached<u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">Le 13 nov. 2015 8:37 AM, "John Nash" <<a href="mailto:john.nash778@gmail.com" target="_blank">john.nash778@gmail.com</a>> a écrit :<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<p class="MsoNormal">Thank you for sharing ideas ..<u></u><u></u></p>
</div>
<p class="MsoNormal">1- I have made sure that dbh is released even in cases of query error.<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">2- In mysql processlist I see lot of connections but all in sleep mode.<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">My stored procedure is complicated but it has all select statements and I run with least possible locking isolation. But I am not really sure on if with time or because of some other application stored procedure
is taking long to to execute hence each new request creating more dbh handles. I am also using readtimeout as 10 seconds in odbc.ini so that in case stored procedure takes time, i get error and then release handle.<br>
<br>
<u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Fri, Nov 13, 2015 at 6:49 PM, Florent Krieg <<a href="mailto:flokrrr@gmail.com" target="_blank">flokrrr@gmail.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">Hello John,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">If you are under heavy load, could this be a db issue? Like the db engine taking long to return the data, hence queries are getting queued one after the other and finally reach a few hundreds of it?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Or you are right and your Lua script sometimes doesn't release the handler as it just shall... To check this, you could for instance log a debug message before and after doing your db query, and check after a few minutes if you have the
equivalent of 'before' and 'after' query messages... If more before than after, then something is happening.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">You could do a 'show processlist;' on the MySQL cli too to understand what are exactly the few hundreds of open connections.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Also, is your stored procedure complicated? I'm not DBA but if it's too tricky, you could imagine to move the load on the Lua processing instead of MySQL. I have the same use cases on my environment and I made sure my MySQL queries are
dumb as possible and I take the benefit of Lua to process the data returned.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I'm just dropping some ideas that I'm thinking about now. Nothing very precise though (sorry...).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Florent<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">2015-11-13 12:49 GMT+01:00 John Nash <<a href="mailto:john.nash778@gmail.com" target="_blank">john.nash778@gmail.com</a>>:<u></u><u></u></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">I am calling a lua script from dialplan which does following steps ...<u></u><u></u></p>
</div>
<p class="MsoNormal">1- Gets freeswitch dbh handle (odbc mysql)<u></u><u></u></p>
</div>
<p class="MsoNormal">2- Runs a mysql stored procedure which returns balance remaining amount for the subscriber who made call.<u></u><u></u></p>
</div>
<p class="MsoNormal">3- Save the amount in session variable and released dbh handle<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">4- Play sound files as per output received and saved in step 3<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Now upto 200 calls and first 2-3 hours of start all work fine. I can see in Database that script uses 32 connections approx at a given time (I think connection pooling works) but suddenly connections start increasing
and go upto 300+ and because of that database stops responding and i have to kill freeswitch to make production DB responsive again.<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt">I am not sure where and what I should check. Can someone give me pointers on what I can check?..Or I am doing something fundamentally wrong.<u></u><u></u></p>
</div>
<p class="MsoNormal">Can there be such a case where lua scripts hang and never release connection?<u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
<br>
<br>
<br>
<u></u><u></u></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<p class="MsoNormal">_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><u></u><u></u></p>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal"><br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div></div>
</div>
<br>_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org" target="_blank">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" rel="noreferrer" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://confluence.freeswitch.org" rel="noreferrer" target="_blank">http://confluence.freeswitch.org</a><br>
<a href="http://www.cluecon.com" rel="noreferrer" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-users mailing list<br>
<a href="mailto:FreeSWITCH-users@lists.freeswitch.org" target="_blank">FreeSWITCH-users@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-users</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-users" rel="noreferrer" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-users</a><br>
<a href="http://www.freeswitch.org" rel="noreferrer" target="_blank">http://www.freeswitch.org</a><br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>