<div dir="ltr"><div>Raised <a href="http://jira.freeswitch.org/browse/FS-5036">http://jira.freeswitch.org/browse/FS-5036</a> - thanks - David</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jan 17, 2013 at 9:36 PM, David Brazier <span dir="ltr"><<a href="mailto:davidjbrazier@gmail.com" target="_blank">davidjbrazier@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">François<br>
<br>
Thanks for the suggestion, will try re-working with execute_extension.<br>
<br>
David<br>
<br>
<br>
From: François Delawarde<br>
Sent: 17/01/2013 11:26<br>
To: <a href="mailto:freeswitch-dev@lists.freeswitch.org">freeswitch-dev@lists.freeswitch.org</a><br>
Subject: Re: [Freeswitch-dev] transfer_history performance<br>
<div class="HOEnZb"><div class="h5">Hi David,<br>
<br>
The "transfer" app is quite heavy to be chained 1000s of times as it<br>
does many things to keep track of what happened (handle max-fw, dup<br>
caller profile, history var, ...). You might end-up with other issues<br>
trying to scale, maybe disk bottleneck writing huge xml-cdr files, ...<br>
<br>
What about making your application use "execute_extension" instead? Note<br>
that I still vote yes for your optimization ideas!<br>
<br>
François.<br>
<br>
<br>
On Thu, 2013-01-17 at 10:48 +0000, David Brazier wrote:<br>
> Hi again<br>
><br>
> A bit more information on reproducing it: each call using<br>
> test_transfer will fully occupy one CPU, so on a 4 core machine you<br>
> need to start 4 of these calls to tie it up, then audio on other calls<br>
> is broken or completely stopped, CLI stops responding etc. As a<br>
> workaround I have commented out from switch_ivr.c line 1808 near end<br>
> of switch_ivr_session_transfer:<br>
><br>
> /*switch_channel_add_variable_var_check(channel,<br>
> SWITCH_TRANSFER_HISTORY_VARIABLE, new_profile->transfer_source,<br>
> SWITCH_FALSE, SWITCH_STACK_PUSH);*/<br>
><br>
> Of course that means transfer_history is no longer populated, but it<br>
> does remove the performance problem, the test_transfer calls now run<br>
> through so fast I had to start the loop off at 9999 and although they<br>
> do use a bit of CPU (it is a tight loop with no sleep/IO etc after<br>
> all) 4 simultaneously do not tie up the server and other calls<br>
> continue fine. I can think of a few more complete solutions:<br>
> 1. Improve the performance of appending to variables with big<br>
> values - I think this is ultimately done in<br>
> switch_event_base_add_header in switch_event.c, though that<br>
> looks like quite a complex function. In the end, repeated<br>
> appending to long strings is never going to be efficient.<br>
> 2. Maintain the transfer history as a stack/list/array structure<br>
> and either render it to the variable string whenever it is<br>
> changed (still expensive?) or generate the string only on<br>
> demand - though I'm not sure how that would work with info,<br>
> uuid_dump and verbose events in event sockets etc.<br>
> 3. Allow channels to opt out of having transfer_history tracked,<br>
> perhaps by setting it to "suppress" or "false", this could be<br>
> checked before the line above. Would put the onus on the<br>
> dialplan to know if it was going to do lots of transfers, but<br>
> it would be backward-compatible with anyone who needs this<br>
> variable. Out of interest why was it added?<br>
> Once I have managed to log in to Jira I will raise all this there.<br>
><br>
> Cheers<br>
><br>
> David<br>
><br>
><br>
> On Thu, Jan 17, 2013 at 12:57 AM, David Brazier<br>
> <<a href="mailto:davidjbrazier@gmail.com">davidjbrazier@gmail.com</a>> wrote:<br>
> Hi<br>
><br>
> Our application makes use of long-running calls with a lot of<br>
> diaplan transfers to bridge them to other calls, put them on<br>
> hold etc. We've noticed a slow down in performance (high CPU,<br>
> audio dropped, delayed) after several hours. I can reproduce<br>
> it with some dialplan that does 1000 transfers:<br>
><br>
> <extension name="test_transfer"><br>
> <condition field="destination_number"<br>
> expression="^test_transfer$"><br>
> <action application="set" data="max_forwards=9999"/><br>
> <action application="transfer" data="test_transfer_999"/><br>
> </condition><br>
> </extension><br>
> <extension name="test_transfer_0" continue="false"><br>
> <condition field="destination_number"<br>
> expression="^test_transfer_0$"><br>
> <action application="hangup"/><br>
> </condition><br>
> </extension><br>
> <extension name="test_transfer_n"><br>
> <condition field="destination_number"<br>
> expression="^test_transfer_(\d+)$"><br>
> <!--<action application="info"/>--><br>
> <action application="transfer" data="test_transfer_${expr<br>
> $1-1}"/><br>
> </condition><br>
> </extension><br>
><br>
> and then<br>
><br>
> originate user/1000 test_transfer XML blah<br>
><br>
> On a Windows laptop (git head) it slows down to more than 1<br>
> sec per iteration after it gets past 800, on a bigger Linux<br>
> server (FS 1.2.5.1) it gets a bit futher but not much. On a<br>
> server with an old version (git-1086cba 2011-05-23 22-51-43<br>
> -0500) it zips through to the end with no slow down. If you<br>
> put the info call in you can see that<br>
> variable_transfer_history is growing, and I think this is the<br>
> source of the problem - this variable is missing from the<br>
> older version but the newer ones are apparently appending to<br>
> it on each iteration which becomes a very expensive<br>
> operation.<br>
><br>
> I'll raise a Jira. Any suggestion for a workaround? (Apart<br>
> from rewriting our dialplan!)<br>
><br>
> David<br>
><br>
><br>
><br>
><br>
><br>
><br>
> _________________________________________________________________________<br>
> Professional FreeSWITCH Consulting Services:<br>
> <a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
> <a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
><br>
> FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
> <a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
><br>
> Official FreeSWITCH Sites<br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
> <a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
> <a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
><br>
> FreeSWITCH-dev mailing list<br>
> <a href="mailto:FreeSWITCH-dev@lists.freeswitch.org">FreeSWITCH-dev@lists.freeswitch.org</a><br>
> <a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev</a><br>
> UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-dev" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-dev</a><br>
> <a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<br>
<br>
<br>
_________________________________________________________________________<br>
Professional FreeSWITCH Consulting Services:<br>
<a href="mailto:consulting@freeswitch.org">consulting@freeswitch.org</a><br>
<a href="http://www.freeswitchsolutions.com" target="_blank">http://www.freeswitchsolutions.com</a><br>
<br>
FreeSWITCH-powered IP PBX: The CudaTel Communication Server<br>
<a href="http://www.cudatel.com" target="_blank">http://www.cudatel.com</a><br>
<br>
Official FreeSWITCH Sites<br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
<a href="http://wiki.freeswitch.org" target="_blank">http://wiki.freeswitch.org</a><br>
<a href="http://www.cluecon.com" target="_blank">http://www.cluecon.com</a><br>
<br>
FreeSWITCH-dev mailing list<br>
<a href="mailto:FreeSWITCH-dev@lists.freeswitch.org">FreeSWITCH-dev@lists.freeswitch.org</a><br>
<a href="http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev" target="_blank">http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev</a><br>
UNSUBSCRIBE:<a href="http://lists.freeswitch.org/mailman/options/freeswitch-dev" target="_blank">http://lists.freeswitch.org/mailman/options/freeswitch-dev</a><br>
<a href="http://www.freeswitch.org" target="_blank">http://www.freeswitch.org</a><br>
</div></div></blockquote></div><br></div>