[Freeswitch-dev] transfer_history performance

[David Brazier]

Hi again

A bit more information on reproducing it: each call using test_transfer
will fully occupy one CPU, so on a 4 core machine you need to start 4 of
these calls to tie it up, then audio on other calls is broken or completely
stopped, CLI stops responding etc.  As a workaround I have commented out
from switch_ivr.c line 1808 near end of switch_ivr_session_transfer:

/*switch_channel_add_variable_var_check(channel,
SWITCH_TRANSFER_HISTORY_VARIABLE, new_profile->transfer_source,
SWITCH_FALSE, SWITCH_STACK_PUSH);*/
Of course that means transfer_history is no longer populated, but it does
remove the performance problem, the test_transfer calls now run through so
fast I had to start the loop off at 9999 and although they do use a bit of
CPU (it is a tight loop with no sleep/IO etc after all) 4 simultaneously do
not tie up the server and other calls continue fine.  I can think of a few
more complete solutions:

   1. Improve the performance of appending to variables with big values - I
   think this is ultimately done in switch_event_base_add_header in
   switch_event.c, though that looks like quite a complex function.  In the
   end, repeated appending to long strings is never going to be efficient.
   2. Maintain the transfer history as a stack/list/array structure and
   either render it to the variable string whenever it is changed (still
   expensive?) or generate the string only on demand - though I'm not sure how
   that would work with info, uuid_dump and verbose events in event sockets
   etc.
   3. Allow channels to opt out of having transfer_history tracked, perhaps
   by setting it to "suppress" or "false", this could be checked before the
   line above.  Would put the onus on the dialplan to know if it was going to
   do lots of transfers, but it would be backward-compatible with anyone who
   needs this variable.  Out of interest why was it added?

Once I have managed to log in to Jira I will raise all this there.

Cheers

David


On Thu, Jan 17, 2013 at 12:57 AM, David Brazier <davidjbrazier at gmail.com>wrote:

> Hi
>
> Our application makes use of long-running calls with a lot of diaplan
> transfers to bridge them to other calls, put them on hold etc.  We've
> noticed a slow down in performance (high CPU, audio dropped, delayed) after
> several hours.  I can reproduce it with some dialplan that does 1000
> transfers:
>
> <extension name="test_transfer">
>   <condition field="destination_number" expression="^test_transfer$">
>     <action application="set" data="max_forwards=9999"/>
>     <action application="transfer" data="test_transfer_999"/>
>   </condition>
> </extension>
> <extension name="test_transfer_0" continue="false">
>   <condition field="destination_number" expression="^test_transfer_0$">
>     <action application="hangup"/>
>   </condition>
> </extension>
> <extension name="test_transfer_n">
>   <condition field="destination_number" expression="^test_transfer_(\d+)$">
>     <!--<action application="info"/>-->
>     <action application="transfer" data="test_transfer_${expr $1-1}"/>
>   </condition>
> </extension>
> and then
>
> originate user/1000 test_transfer XML blah
>
> On a Windows laptop (git head) it slows down to more than 1 sec per
> iteration after it gets past 800, on a bigger Linux server (FS 1.2.5.1) it
> gets a bit futher but not much.  On a server with an old version
> (git-1086cba 2011-05-23 22-51-43 -0500) it zips through to the end with no
> slow down.  If you put the info call in you can see that
> variable_transfer_history is growing, and I think this is the source of the
> problem - this variable is missing from the older version but the newer
> ones are apparently appending to it on each iteration which becomes a very
> expensive operation.
>
> I'll raise a Jira.  Any suggestion for a workaround? (Apart from rewriting
> our dialplan!)
>
> David
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-dev/attachments/20130117/a47773cf/attachment.html