Re: [Freeswitch-dev] transfer_history performance
François Thanks for the suggestion, will try re-working with execute_extension. David From: François Delawarde Sent: 17/01/2013 11:26 To: freeswitch-dev@lists.freeswitch.org Subject: Re: [Freeswitch-dev] transfer_history performance Hi David, The "transfer" app is quite heavy to be chained 1000s of times as it does many things to keep track of what happened (handle max-fw, dup caller profile, history var, ...). You might end-up with other issues trying to scale, maybe disk bottleneck writing huge xml-cdr files, ... What about making your application use "execute_extension" instead? Note that I still vote yes for your optimization ideas! François. On Thu, 2013-01-17 at 10:48 +0000, David Brazier wrote:
Hi again
A bit more information on reproducing it: each call using test_transfer will fully occupy one CPU, so on a 4 core machine you need to start 4 of these calls to tie it up, then audio on other calls is broken or completely stopped, CLI stops responding etc. As a workaround I have commented out from switch_ivr.c line 1808 near end of switch_ivr_session_transfer:
/*switch_channel_add_variable_var_check(channel, SWITCH_TRANSFER_HISTORY_VARIABLE, new_profile->transfer_source, SWITCH_FALSE, SWITCH_STACK_PUSH);*/
Of course that means transfer_history is no longer populated, but it does remove the performance problem, the test_transfer calls now run through so fast I had to start the loop off at 9999 and although they do use a bit of CPU (it is a tight loop with no sleep/IO etc after all) 4 simultaneously do not tie up the server and other calls continue fine. I can think of a few more complete solutions: 1. Improve the performance of appending to variables with big values - I think this is ultimately done in switch_event_base_add_header in switch_event.c, though that looks like quite a complex function. In the end, repeated appending to long strings is never going to be efficient. 2. Maintain the transfer history as a stack/list/array structure and either render it to the variable string whenever it is changed (still expensive?) or generate the string only on demand - though I'm not sure how that would work with info, uuid_dump and verbose events in event sockets etc. 3. Allow channels to opt out of having transfer_history tracked, perhaps by setting it to "suppress" or "false", this could be checked before the line above. Would put the onus on the dialplan to know if it was going to do lots of transfers, but it would be backward-compatible with anyone who needs this variable. Out of interest why was it added? Once I have managed to log in to Jira I will raise all this there.
Cheers
David
On Thu, Jan 17, 2013 at 12:57 AM, David Brazier <davidjbrazier@gmail.com> wrote: Hi
Our application makes use of long-running calls with a lot of diaplan transfers to bridge them to other calls, put them on hold etc. We've noticed a slow down in performance (high CPU, audio dropped, delayed) after several hours. I can reproduce it with some dialplan that does 1000 transfers:
<extension name="test_transfer"> <condition field="destination_number" expression="^test_transfer$"> <action application="set" data="max_forwards=9999"/> <action application="transfer" data="test_transfer_999"/> </condition> </extension> <extension name="test_transfer_0" continue="false"> <condition field="destination_number" expression="^test_transfer_0$"> <action application="hangup"/> </condition> </extension> <extension name="test_transfer_n"> <condition field="destination_number" expression="^test_transfer_(\d+)$"> <!--<action application="info"/>--> <action application="transfer" data="test_transfer_${expr $1-1}"/> </condition> </extension>
and then
originate user/1000 test_transfer XML blah
On a Windows laptop (git head) it slows down to more than 1 sec per iteration after it gets past 800, on a bigger Linux server (FS 1.2.5.1) it gets a bit futher but not much. On a server with an old version (git-1086cba 2011-05-23 22-51-43 -0500) it zips through to the end with no slow down. If you put the info call in you can see that variable_transfer_history is growing, and I think this is the source of the problem - this variable is missing from the older version but the newer ones are apparently appending to it on each iteration which becomes a very expensive operation.
I'll raise a Jira. Any suggestion for a workaround? (Apart from rewriting our dialplan!)
David
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
Raised http://jira.freeswitch.org/browse/FS-5036 - thanks - David On Thu, Jan 17, 2013 at 9:36 PM, David Brazier <davidjbrazier@gmail.com>wrote:
François
Thanks for the suggestion, will try re-working with execute_extension.
David
From: François Delawarde Sent: 17/01/2013 11:26 To: freeswitch-dev@lists.freeswitch.org Subject: Re: [Freeswitch-dev] transfer_history performance Hi David,
The "transfer" app is quite heavy to be chained 1000s of times as it does many things to keep track of what happened (handle max-fw, dup caller profile, history var, ...). You might end-up with other issues trying to scale, maybe disk bottleneck writing huge xml-cdr files, ...
What about making your application use "execute_extension" instead? Note that I still vote yes for your optimization ideas!
François.
On Thu, 2013-01-17 at 10:48 +0000, David Brazier wrote:
Hi again
A bit more information on reproducing it: each call using test_transfer will fully occupy one CPU, so on a 4 core machine you need to start 4 of these calls to tie it up, then audio on other calls is broken or completely stopped, CLI stops responding etc. As a workaround I have commented out from switch_ivr.c line 1808 near end of switch_ivr_session_transfer:
/*switch_channel_add_variable_var_check(channel, SWITCH_TRANSFER_HISTORY_VARIABLE, new_profile->transfer_source, SWITCH_FALSE, SWITCH_STACK_PUSH);*/
Of course that means transfer_history is no longer populated, but it does remove the performance problem, the test_transfer calls now run through so fast I had to start the loop off at 9999 and although they do use a bit of CPU (it is a tight loop with no sleep/IO etc after all) 4 simultaneously do not tie up the server and other calls continue fine. I can think of a few more complete solutions: 1. Improve the performance of appending to variables with big values - I think this is ultimately done in switch_event_base_add_header in switch_event.c, though that looks like quite a complex function. In the end, repeated appending to long strings is never going to be efficient. 2. Maintain the transfer history as a stack/list/array structure and either render it to the variable string whenever it is changed (still expensive?) or generate the string only on demand - though I'm not sure how that would work with info, uuid_dump and verbose events in event sockets etc. 3. Allow channels to opt out of having transfer_history tracked, perhaps by setting it to "suppress" or "false", this could be checked before the line above. Would put the onus on the dialplan to know if it was going to do lots of transfers, but it would be backward-compatible with anyone who needs this variable. Out of interest why was it added? Once I have managed to log in to Jira I will raise all this there.
Cheers
David
On Thu, Jan 17, 2013 at 12:57 AM, David Brazier <davidjbrazier@gmail.com> wrote: Hi
Our application makes use of long-running calls with a lot of diaplan transfers to bridge them to other calls, put them on hold etc. We've noticed a slow down in performance (high CPU, audio dropped, delayed) after several hours. I can reproduce it with some dialplan that does 1000 transfers:
<extension name="test_transfer"> <condition field="destination_number" expression="^test_transfer$"> <action application="set" data="max_forwards=9999"/> <action application="transfer" data="test_transfer_999"/> </condition> </extension> <extension name="test_transfer_0" continue="false"> <condition field="destination_number" expression="^test_transfer_0$"> <action application="hangup"/> </condition> </extension> <extension name="test_transfer_n"> <condition field="destination_number" expression="^test_transfer_(\d+)$"> <!--<action application="info"/>--> <action application="transfer" data="test_transfer_${expr $1-1}"/> </condition> </extension>
and then
originate user/1000 test_transfer XML blah
On a Windows laptop (git head) it slows down to more than 1 sec per iteration after it gets past 800, on a bigger Linux server (FS 1.2.5.1) it gets a bit futher but not much. On a server with an old version (git-1086cba 2011-05-23 22-51-43 -0500) it zips through to the end with no slow down. If you put the info call in you can see that variable_transfer_history is growing, and I think this is the source of the problem - this variable is missing from the older version but the newer ones are apparently appending to it on each iteration which becomes a very expensive operation.
I'll raise a Jira. Any suggestion for a workaround? (Apart from rewriting our dialplan!)
David
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-dev mailing list FreeSWITCH-dev@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-dev UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-dev http://www.freeswitch.org
participants (1)
-
David Brazier