[Freeswitch-dev] transfer_history performance

[David Brazier]

Raised http://jira.freeswitch.org/browse/FS-5036 - thanks - David


On Thu, Jan 17, 2013 at 9:36 PM, David Brazier <davidjbrazier at gmail.com>wrote:

> François
>
> Thanks for the suggestion, will try re-working with execute_extension.
>
> David
>
>
> From: François Delawarde
> Sent: 17/01/2013 11:26
> To: freeswitch-dev at lists.freeswitch.org
> Subject: Re: [Freeswitch-dev] transfer_history performance
> Hi David,
>
> The "transfer" app is quite heavy to be chained 1000s of times as it
> does many things to keep track of what happened (handle max-fw, dup
> caller profile, history var, ...). You might end-up with other issues
> trying to scale, maybe disk bottleneck writing huge xml-cdr files, ...
>
> What about making your application use "execute_extension" instead? Note
> that I still vote yes for your optimization ideas!
>
> François.
>
>
> On Thu, 2013-01-17 at 10:48 +0000, David Brazier wrote:
> > Hi again
> >
> > A bit more information on reproducing it: each call using
> > test_transfer will fully occupy one CPU, so on a 4 core machine you
> > need to start 4 of these calls to tie it up, then audio on other calls
> > is broken or completely stopped, CLI stops responding etc.  As a
> > workaround I have commented out from switch_ivr.c line 1808 near end
> > of switch_ivr_session_transfer:
> >
> > /*switch_channel_add_variable_var_check(channel,
> > SWITCH_TRANSFER_HISTORY_VARIABLE, new_profile->transfer_source,
> > SWITCH_FALSE, SWITCH_STACK_PUSH);*/
> >
> > Of course that means transfer_history is no longer populated, but it
> > does remove the performance problem, the test_transfer calls now run
> > through so fast I had to start the loop off at 9999 and although they
> > do use a bit of CPU (it is a tight loop with no sleep/IO etc after
> > all) 4 simultaneously do not tie up the server and other calls
> > continue fine.  I can think of a few more complete solutions:
> >      1. Improve the performance of appending to variables with big
> >         values - I think this is ultimately done in
> >         switch_event_base_add_header in switch_event.c, though that
> >         looks like quite a complex function.  In the end, repeated
> >         appending to long strings is never going to be efficient.
> >      2. Maintain the transfer history as a stack/list/array structure
> >         and either render it to the variable string whenever it is
> >         changed (still expensive?) or generate the string only on
> >         demand - though I'm not sure how that would work with info,
> >         uuid_dump and verbose events in event sockets etc.
> >      3. Allow channels to opt out of having transfer_history tracked,
> >         perhaps by setting it to "suppress" or "false", this could be
> >         checked before the line above.  Would put the onus on the
> >         dialplan to know if it was going to do lots of transfers, but
> >         it would be backward-compatible with anyone who needs this
> >         variable.  Out of interest why was it added?
> > Once I have managed to log in to Jira I will raise all this there.
> >
> > Cheers
> >
> > David
> >
> >
> > On Thu, Jan 17, 2013 at 12:57 AM, David Brazier
> > <davidjbrazier at gmail.com> wrote:
> >         Hi
> >
> >         Our application makes use of long-running calls with a lot of
> >         diaplan transfers to bridge them to other calls, put them on
> >         hold etc.  We've noticed a slow down in performance (high CPU,
> >         audio dropped, delayed) after several hours.  I can reproduce
> >         it with some dialplan that does 1000 transfers:
> >
> >         <extension name="test_transfer">
> >           <condition field="destination_number"
> >         expression="^test_transfer$">
> >             <action application="set" data="max_forwards=9999"/>
> >             <action application="transfer" data="test_transfer_999"/>
> >           </condition>
> >         </extension>
> >         <extension name="test_transfer_0" continue="false">
> >           <condition field="destination_number"
> >         expression="^test_transfer_0$">
> >             <action application="hangup"/>
> >           </condition>
> >         </extension>
> >         <extension name="test_transfer_n">
> >           <condition field="destination_number"
> >         expression="^test_transfer_(\d+)$">
> >             <!--<action application="info"/>-->
> >             <action application="transfer" data="test_transfer_${expr
> >         $1-1}"/>
> >           </condition>
> >         </extension>
> >
> >         and then
> >
> >         originate user/1000 test_transfer XML blah
> >
> >         On a Windows laptop (git head) it slows down to more than 1
> >         sec per iteration after it gets past 800, on a bigger Linux
> >         server (FS 1.2.5.1) it gets a bit futher but not much.  On a
> >         server with an old version (git-1086cba 2011-05-23 22-51-43
> >         -0500) it zips through to the end with no slow down.  If you
> >         put the info call in you can see that
> >         variable_transfer_history is growing, and I think this is the
> >         source of the problem - this variable is missing from the
> >         older version but the newer ones are apparently appending to
> >         it on each iteration which becomes a very expensive
> >         operation.
> >
> >         I'll raise a Jira.  Any suggestion for a workaround? (Apart
> >         from rewriting our dialplan!)
> >
> >         David
> >
> >
> >
> >
> >
> >
> > _________________________________________________________________________
> > Professional FreeSWITCH Consulting Services:
> > consulting at freeswitch.org
> > http://www.freeswitchsolutions.com
> >
> > 
> > 
> >
> > Official FreeSWITCH Sites
> > http://www.freeswitch.org
> > http://wiki.freeswitch.org
> > http://www.cluecon.com
> >
> > FreeSWITCH-dev mailing list
> > FreeSWITCH-dev at lists.freeswitch.org
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev
> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev
> > http://www.freeswitch.org
>
>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-dev mailing list
> FreeSWITCH-dev at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev
> http://www.freeswitch.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-dev/attachments/20130118/5c0cb4fd/attachment.html