Hi again

A bit more information on reproducing it: each call using test_transfer will fully occupy one CPU, so on a 4 core machine you need to start 4 of these calls to tie it up, then audio on other calls is broken or completely stopped, CLI stops responding etc. As a workaround I have commented out from switch_ivr.c line 1808 near end of switch_ivr_session_transfer:

/*switch_channel_add_variable_var_check(channel, SWITCH_TRANSFER_HISTORY_VARIABLE, new_profile->transfer_source, SWITCH_FALSE, SWITCH_STACK_PUSH);*/

Of course that means transfer_history is no longer populated, but it does remove the performance problem, the test_transfer calls now run through so fast I had to start the loop off at 9999 and although they do use a bit of CPU (it is a tight loop with no sleep/IO etc after all) 4 simultaneously do not tie up the server and other calls continue fine. I can think of a few more complete solutions:

Improve the performance of appending to variables with big values - I think this is ultimately done in switch_event_base_add_header in switch_event.c, though that looks like quite a complex function. In the end, repeated appending to long strings is never going to be efficient.
Maintain the transfer history as a stack/list/array structure and either render it to the variable string whenever it is changed (still expensive?) or generate the string only on demand - though I'm not sure how that would work with info, uuid_dump and verbose events in event sockets etc.
Allow channels to opt out of having transfer_history tracked, perhaps by setting it to "suppress" or "false", this could be checked before the line above. Would put the onus on the dialplan to know if it was going to do lots of transfers, but it would be backward-compatible with anyone who needs this variable. Out of interest why was it added?

Once I have managed to log in to Jira I will raise all this there.

Cheers

David

On Thu, Jan 17, 2013 at 12:57 AM, David Brazier <davidjbrazier@gmail.com> wrote:

Hi

Our application makes use of long-running calls with a lot of diaplan transfers to bridge them to other calls, put them on hold etc. We've noticed a slow down in performance (high CPU, audio dropped, delayed) after several hours. I can reproduce it with some dialplan that does 1000 transfers:

<extension name="test_transfer">
<condition field="destination_number" expression="^test_transfer$">
    <action application="set" data="max_forwards=9999"/>
    <action application="transfer" data="test_transfer_999"/>
</condition>
</extension>
<extension name="test_transfer_0" continue="false">
<condition field="destination_number" expression="^test_transfer_0$">
    <action application="hangup"/>
</condition>
</extension>
<extension name="test_transfer_n">
<condition field="destination_number" expression="^test_transfer_(\d+)$">
    
    <action application="transfer" data="test_transfer_${expr $1-1}"/>
</condition>
</extension>
and then

originate user/1000 test_transfer XML blah

On a Windows laptop (git head) it slows down to more than 1 sec per iteration after it gets past 800, on a bigger Linux server (FS 1.2.5.1) it gets a bit futher but not much. On a server with an old version (git-1086cba 2011-05-23 22-51-43 -0500) it zips through to the end with no slow down. If you put the info call in you can see that variable_transfer_history is growing, and I think this is the source of the problem - this variable is missing from the older version but the newer ones are apparently appending to it on each iteration which becomes a very expensive operation.

I'll raise a Jira. Any suggestion for a workaround? (Apart from rewriting our dialplan!)

David