[Freeswitch-users] Strange Performance when using as SBC

rod kawarod at laposte.net
Mon Feb 2 04:36:44 PST 2009


Some update:
    - I switched log level to "err"
    - I'm now using proxy-media
    - and I erased the directive answer in the dialplan (useless and 
seems that it consumes lots of CPU, don't know why)

the dialplan now looks like this:
 <extension name="hold_music_bridge">
      <condition field="destination_number" expression="^9999$">
        <action application="set" data="proxy_media=true"/>
        <action application="bridge" 
data="sofia/external/9999 at 10.10.20.100"/>
        </condition>
    </extension>

instead of
 <extension name="hold_music_bridge">
      <condition field="destination_number" expression="^9999$">
        <action application="answer"/>
        <action application="bridge" 
data="sofia/external/9999 at 10.10.20.100"/>
        </condition>
    </extension>

The box is now able to bridge 300 calls with 20-30% of free CPU. I will 
run a long term test.

I see this error in the log and don't understand exactly if somebody 
could help (I'm running latest trunk 11592M):
2009-02-02 13:29:54 [ERR] switch_core_io.c:117 
switch_core_session_read_frame() sofia/external/9998 at 10.10.20.100 has no 
read codec.

regards,
rodrigue

rod wrote:
> Hi Ken, Jay,
>
> thanks for pointing to proxy media, I will test.
>
> Ken, you are right, I was brain damaged (a stupid mistake) when setting 
> INFO cause this kind of level could be very verbose. I'm switching to 
> CRIT or ERR.
>
> Thanks guys,
> rod.
>
> thanks for
>
> Ken Rice wrote:
>   
>> If you don't have to transcode, using proxy media mode will still save you
>> some CPU time. This is 1/2 way between bypass media and the default media
>> interactive mode. The other draw back to this mode is if you are using FS to
>> clean up RTP and DTMF you loose those functions but they are not needed in
>> most use cases.
>>
>> As far as the log level goes, I found that once I had things stable setting
>> the loglevel to helped a good deal... Info is probably a bit too high of a
>> loglevel I would probably go for CRIT or ERR (2 or 1 respectively) if you
>> insist on leaving logging turned on... On a busy system these can and will
>> generate a good deal of activity (and disk IO if using mod_logfile)
>>
>> Ken
>>
>>
>>   
>>     
>>> From: rod <kawarod at laposte.net>
>>> Reply-To: <freeswitch-users at lists.freeswitch.org>
>>> Date: Mon, 02 Feb 2009 11:36:35 +0400
>>> To: <freeswitch-users at lists.freeswitch.org>
>>> Subject: Re: [Freeswitch-users] Strange Performance when using as SBC
>>>
>>> Hi Ken,
>>>
>>> 1) I'd like to use FS to hide topology, so bypass media is not possible
>>> 2) done
>>> 3) done
>>> 4) not used
>>> 5) i'm using this ins switch.xml -> <param name="loglevel"
>>> value="info"/>, if you think an other log level is more suitable.
>>>
>>> Regarding logging, I can see in console and in the freeswitch.log that
>>> there is still a lot of NOTICE logging, see below:
>>> 2009-02-02 08:33:56 [NOTICE] switch_core_session.c:960
>>> switch_core_session_thread() Session 8721
>>> (sofia/internal/sipp at 10.10.10.1:5060) Ended
>>> 2009-02-02 08:33:56 [NOTICE] switch_core_session.c:962
>>> switch_core_session_thread() Close Channel
>>> sofia/internal/sipp at 10.10.10.1:5060 [CS_HANGUP]
>>> 2009-02-02 08:33:56 [NOTICE] switch_core_session.c:960
>>> switch_core_session_thread() Session 8722
>>> (sofia/external/9998 at 10.10.20.100) Ended
>>> 2009-02-02 08:33:56 [NOTICE] switch_core_session.c:962
>>> switch_core_session_thread() Close Channel
>>> sofia/external/9998 at 10.10.20.100 [CS_HANGUP]
>>> 2009-02-02 08:33:56 [NOTICE] sofia.c:3164 sofia_handle_sip_i_state()
>>> Channel [sofia/external/9998 at 10.10.20.100] has been answered
>>> 2009-02-02 08:33:56 [WARNING] mod_sofia.c:740 sofia_read_frame()
>>> Changing codec ptime to 30. I bet you have a linksys/sipura =D
>>>
>>> Do you have any idea where I can switch off this kind of logging. I
>>> thought it should be in /dialplan/internal.xml, but I see that in
>>> internal.xml -> <param name="debug" value="0"/>
>>>
>>> thanks a lot for your suggestion.
>>>
>>> regards,
>>> rod
>>>
>>> Ken Rice wrote:
>>>     
>>>       
>>>> Dont forget there are several things you can do to increase performance...
>>>>
>>>> 1) where possible use bypass media or media proxy modes
>>>> 2) mount freeswitch/db as a ram drive (if you are using voicemail with
>>>> the internal FS DBs you'll need a way to make this persistant across
>>>> reboots)
>>>> 3) see the wiki for setting reasonable ulimits
>>>> 4) (this is my oppinion others may vary) dont use mod_cdr_csv
>>>> 5) turn off (or reduce logging) in switch.conf.xml
>>>>
>>>> all of these thing can greatly improve performance.
>>>>
>>>> On Mon, Feb 2, 2009 at 1:04 AM, rod <kawarod at laposte.net
>>>> <mailto:kawarod at laposte.net>> wrote:
>>>>
>>>>     Thanks Anthony,
>>>>
>>>>     the setup is like this:
>>>>
>>>>     sipp server ---- FS 1 ---- FS2
>>>>
>>>>     FS1 is the AMD CPU that has only one extension in dialplan that
>>>>     bridges
>>>>     9999 to FS2. 9999 is the first extension in FS2 dialplan that
>>>>     plays moh,
>>>>     FS2 has no CPU pbm.
>>>>
>>>>     FS1 is maxing out at 60 bridged calls without your option -hp.
>>>>
>>>>     Using -hp, I'm now able to bridge 200 concurrent calls (a great
>>>>     improvement) and the system is still reactive. CPU load is high
>>>>     but not
>>>>     100% and as the system responds well, I think that doesn't matter. The
>>>>     2GB of memory are completely consumed (top command shows 700MB for FS
>>>>     process).
>>>>
>>>>     I understand that FS1 server is not the best hardware platform,
>>>>     and I'm
>>>>     waiting for new 4 cores server for testing.
>>>>     I will update those numbers when testing with the new hardware.
>>>>
>>>>     regards,
>>>>     rod.
>>>>
>>>>     Anthony Minessale wrote:
>>>>       
>>>>         
>>>>> Which of the 2 machines has the load issue? You said it was one box
>>>>> calling the other.
>>>>>
>>>>> You have 2 major things against you, single CPU and AMD, but you
>>>>> should at least be able to get in the vicinity of 800-1000 calls
>>>>>         
>>>>>           
>>>>     on a
>>>>       
>>>>         
>>>>> box like that.
>>>>>
>>>>> Are you calling the default 9999?  It's not really an appropriate
>>>>> extension for load testing.
>>>>> On the terminating box you should set up a manual extension that is
>>>>> the first one in the dial plan
>>>>> to play a wav file from preferably a ram disk or /tmp
>>>>>
>>>>> If you do plan on using this in production accept nothing less
>>>>>         
>>>>>           
>>>>     than a
>>>>       
>>>>         
>>>>> multi-core intel machine with at least 4 cores, the more cores the
>>>>> better because that parallel processing is where FS gets it's
>>>>>         
>>>>>           
>>>>     atvantage.
>>>>       
>>>>         
>>>>> On Fri, Jan 30, 2009 at 5:56 AM, rod <kawarod at laposte.net
>>>>>         
>>>>>           
>>>>     <mailto:kawarod at laposte.net>
>>>>       
>>>>         
>>>>> <mailto:kawarod at laposte.net <mailto:kawarod at laposte.net>>> wrote:
>>>>>
>>>>>     Dear list,
>>>>>
>>>>>     I've been playing with freeswitch for some time (2 months)
>>>>>         
>>>>>           
>>>>     and the
>>>>       
>>>>         
>>>>>     fact
>>>>>     is that I'm very pleased with the functionnalities of this
>>>>>         
>>>>>           
>>>>     software.
>>>>       
>>>>         
>>>>>     I'd like to use FS as a SBC handling media and I'm doing some
>>>>>     tests with
>>>>>     sipp to load the machine but I'm unable to bridge more than
>>>>>         
>>>>>           
>>>>     60 calls
>>>>       
>>>>         
>>>>>     without seeing the CPU being loaded at 100%. I'm sure
>>>>>         
>>>>>           
>>>>     something is
>>>>       
>>>>         
>>>>>     going
>>>>>     wrong with my setup but I'm unable to see what.
>>>>>
>>>>>     The test machine has the following specs:
>>>>>       Athlon XP 3500+ with 2GB of memory (I know this is not a
>>>>>         
>>>>>           
>>>>     high end
>>>>       
>>>>         
>>>>>     machine :p)
>>>>>
>>>>>     Freeswitch:/opt/freeswitch/log# cat /proc/cpuinfo
>>>>>     processor       : 0
>>>>>     vendor_id       : AuthenticAMD
>>>>>     cpu family      : 15
>>>>>     model           : 95
>>>>>     model name      : AMD Athlon(tm) 64 Processor 3500+
>>>>>     stepping        : 2
>>>>>     cpu MHz         : 2199.973
>>>>>     cache size      : 512 KB
>>>>>     fpu             : yes
>>>>>     fpu_exception   : yes
>>>>>     cpuid level     : 1
>>>>>     wp              : yes
>>>>>     flags           : fpu vme de pse tsc msr pae mce cx8 apic
>>>>>         
>>>>>           
>>>>     sep mtrr pge
>>>>       
>>>>         
>>>>>     mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext
>>>>>     fxsr_opt
>>>>>     rdtscp lm 3dnowext 3dnow up rep_good pni cx16 lahf_lm svm
>>>>>         
>>>>>           
>>>>     extapic
>>>>       
>>>>         
>>>>>     cr8_legacy
>>>>>     bogomips        : 4402.97
>>>>>     TLB size        : 1024 4K pages
>>>>>     clflush size    : 64
>>>>>     cache_alignment : 64
>>>>>     address sizes   : 40 bits physical, 48 bits virtual
>>>>>     power management: ts fid vid ttp tm stc
>>>>>
>>>>>     I installed FS on a fresh debian 64:
>>>>>       Linux Freeswitch 2.6.26-1-amd64 #1 SMP Sat Jan 10 17:57:00
>>>>>         
>>>>>           
>>>>     UTC 2009
>>>>       
>>>>         
>>>>>     x86_64 GNU/Linux
>>>>>
>>>>>     I set the ulimit parameters like those on the website:
>>>>>       freeswitch at internal> ...
>>>>>     Freeswitch:/opt/free-svn/bin# ulimit -a
>>>>>     core file size          (blocks, -c) unlimited
>>>>>     data seg size           (kbytes, -d) unlimited
>>>>>     scheduling priority             (-e) 0
>>>>>     file size               (blocks, -f) unlimited
>>>>>     pending signals                 (-i) unlimited
>>>>>     max locked memory       (kbytes, -l) unlimited
>>>>>     max memory size         (kbytes, -m) unlimited
>>>>>     open files                      (-n) 999999
>>>>>     pipe size            (512 bytes, -p) 8
>>>>>     POSIX message queues     (bytes, -q) unlimited
>>>>>     real-time priority              (-r) 0
>>>>>     stack size              (kbytes, -s) 244
>>>>>     cpu time               (seconds, -t) unlimited
>>>>>     max user processes              (-u) unlimited
>>>>>     virtual memory          (kbytes, -v) unlimited
>>>>>     file locks                      (-x) unlimited
>>>>>
>>>>>
>>>>>     My network setup is the following:
>>>>>
>>>>>       SIPP machine (10.10.10.1/24)----------------vlan
>>>>>         
>>>>>           
>>>>     <http://10.10.10.1/24%29----------------vlan>
>>>>       
>>>>         
>>>>>     <http://10.10.10.1/24%29----------------vlan> 55
>>>>>     ----------(10.10.10.254/24 <http://10.10.10.254/24>
>>>>>         
>>>>>           
>>>>     <http://10.10.10.254/24>) FS
>>>>       
>>>>         
>>>>>     (10.10.20.254/24)--------------
>>>>>         
>>>>>           
>>>>     <http://10.10.20.254/24%29-------------->
>>>>       
>>>>         
>>>>>     <http://10.10.20.254/24%29--------------> vlan56
>>>>>     -------------------(10.10.20.100/24 <http://10.10.20.100/24>
>>>>>         
>>>>>           
>>>>     <http://10.10.20.100/24>)
>>>>       
>>>>         
>>>>>     OTHER STOCK FS
>>>>>
>>>>>
>>>>>     I launched sipp with:
>>>>>       sipp -sn uac_pcap -s 9999 -r 10 -l 80 -d 60000 -mi
>>>>>         
>>>>>           
>>>>     10.10.10.1 -i
>>>>       
>>>>         
>>>>>     10.10.10.1 -mp 25000 10.10.10.254:5060
>>>>>         
>>>>>           
>>>>     <http://10.10.10.254:5060> <http://10.10.10.254:5060>
>>>>       
>>>>         
>>>>>     The dialplan on FS is very simple:
>>>>>     <?xml version="1.0" encoding="utf-8"?>
>>>>>     <!-- http://wiki.freeswitch.org/wiki/Dialplan_XML -->
>>>>>     <include>
>>>>>      <context name="default">
>>>>>
>>>>>        <extension name="hold_music_bridge">
>>>>>         <condition field="destination_number" expression="^9999$">
>>>>>           <action application="answer"/>
>>>>>           <action application="bridge"
>>>>>     data="sofia/external/9999 at 10.10.20.100
>>>>>         
>>>>>           
>>>>     <mailto:9999 at 10.10.20.100> <mailto:9999 at 10.10.20.100
>>>>     <mailto:9999 at 10.10.20.100>>"/>
>>>>       
>>>>         
>>>>>           </condition>
>>>>>       </extension>
>>>>>       </context>
>>>>>
>>>>>     </include>
>>>>>
>>>>>     FreeSWITCH Version 1.0.trunk (11560M) Started.
>>>>>     Crash Protection [Disabled]
>>>>>     Max Sessions[1000]
>>>>>     Session Rate[100]
>>>>>     SQL [Enabled]
>>>>>
>>>>>
>>>>>     The test is very simple: sipp dial 9999 that matches in my
>>>>>         
>>>>>           
>>>>     FS dialplan
>>>>       
>>>>         
>>>>>     and this is bridged to an other FS machine playing music on
>>>>>         
>>>>>           
>>>>     hold.
>>>>       
>>>>         
>>>>>     When I launch "top"  I see after 30 to 40 s that FS consumes all
>>>>>     the CPU
>>>>>     ressources (with a mean of 50-60 % before), with 80 calls.
>>>>>     When I set 70 calls, I have to wait 70-80 s before seeing
>>>>>         
>>>>>           
>>>>     the same
>>>>       
>>>>         
>>>>>     issue.
>>>>>
>>>>>     Presence is set to false on the 2 profile.
>>>>>
>>>>>     I have the same issue with FS 1.0.2 that' s why I tried FS
>>>>>         
>>>>>           
>>>>     11560.
>>>>       
>>>>         
>>>>>     When I use the FS machine as a router to test the packet per
>>>>>         
>>>>>           
>>>>     second
>>>>       
>>>>         
>>>>>     performance, I'm reaching 100Mbps with 8000pps in each
>>>>>         
>>>>>           
>>>>     direction (from
>>>>       
>>>>         
>>>>>     vlan 55 to vlan56) with less than 12% CPU. So that I don't think
>>>>>     there's
>>>>>     an issue with the network.
>>>>>
>>>>>     Here is an "mpstat -P ALL 1" to show you what's happening
>>>>>         
>>>>>           
>>>>     suddenly
>>>>       
>>>>         
>>>>>     with
>>>>>     70 bridge calls:
>>>>>     12:31:26     CPU   %user   %nice    %sys %iowait    %irq   %soft
>>>>>     %steal   %idle    intr/s
>>>>>     12:31:27     all    3,00    0,00    3,00    0,00    1,00    4,00
>>>>>     0,00   89,00   6241,00
>>>>>     12:31:27       0    3,00    0,00    3,00    0,00    1,00    4,00
>>>>>     0,00   89,00   6241,00
>>>>>
>>>>>     12:31:27     CPU   %user   %nice    %sys %iowait    %irq   %soft
>>>>>     %steal   %idle    intr/s
>>>>>     12:31:28     all   14,14    0,00   56,57    0,00    2,02    5,05
>>>>>     0,00   22,22   6035,35
>>>>>     12:31:28       0   14,14    0,00   56,57    0,00    2,02    5,05
>>>>>     0,00   22,22   6035,35
>>>>>
>>>>>     12:31:28     CPU   %user   %nice    %sys %iowait    %irq   %soft
>>>>>     %steal   %idle    intr/s
>>>>>     12:31:29     all   24,75    0,00   67,33    0,00    0,99    6,93
>>>>>     0,00    0,00   5483,17
>>>>>     12:31:29       0   24,75    0,00   67,33    0,00    0,99    6,93
>>>>>     0,00    0,00   5483,17
>>>>>
>>>>>
>>>>>     The CPU is going from 89% idle to 0% in less than 2 seconds.
>>>>>
>>>>>     I know that I don't have to expect too much from this kind of
>>>>>     hardware,
>>>>>     but it seems strange that the CPU power vanished so suddenly.
>>>>>
>>>>>     Thanks a lot for the guys that have read this long mail :p
>>>>>
>>>>>     kind regards,
>>>>>     rod
>>>>>
>>>>>
>>>>>     _______________________________________________
>>>>>     Freeswitch-users mailing list
>>>>>     Freeswitch-users at lists.freeswitch.org
>>>>>         
>>>>>           
>>>>     <mailto:Freeswitch-users at lists.freeswitch.org>
>>>>       
>>>>         
>>>>>     <mailto:Freeswitch-users at lists.freeswitch.org
>>>>>         
>>>>>           
>>>>     <mailto:Freeswitch-users at lists.freeswitch.org>>
>>>>       
>>>>         
>>>>>     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>    
>>>>>         
>>>>>           
>>>>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>       
>>>>         
>>>>>     http://www.freeswitch.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Anthony Minessale II
>>>>>
>>>>> FreeSWITCH http://www.freeswitch.org/
>>>>> ClueCon http://www.cluecon.com/
>>>>>
>>>>> AIM: anthm
>>>>> MSN:anthony_minessale at hotmail.com
>>>>>         
>>>>>           
>>>>     <mailto:MSN%3Aanthony_minessale at hotmail.com>
>>>>       
>>>>         
>>>>> <mailto:MSN%3Aanthony_minessale at hotmail.com
>>>>>         
>>>>>           
>>>>     <mailto:MSN%253Aanthony_minessale at hotmail.com>>
>>>>       
>>>>         
>>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>>>>         
>>>>>           
>>>>     <mailto:PAYPAL%3Aanthony.minessale at gmail.com>
>>>>       
>>>>         
>>>>> <mailto:PAYPAL%3Aanthony.minessale at gmail.com
>>>>>         
>>>>>           
>>>>     <mailto:PAYPAL%253Aanthony.minessale at gmail.com>>
>>>>       
>>>>         
>>>>> IRC: irc.freenode.net <http://irc.freenode.net>
>>>>>         
>>>>>           
>>>>     <http://irc.freenode.net> #freeswitch
>>>>       
>>>>         
>>>>> FreeSWITCH Developer Conference
>>>>> sip:888 at conference.freeswitch.org
>>>>>         
>>>>>           
>>>>     <mailto:sip%3A888 at conference.freeswitch.org>
>>>>       
>>>>         
>>>>> <mailto:sip%3A888 at conference.freeswitch.org
>>>>>         
>>>>>           
>>>>     <mailto:sip%253A888 at conference.freeswitch.org>>
>>>>       
>>>>         
>>>>> iax:guest at conference.freeswitch.org/888
>>>>>         
>>>>>           
>>>>     <http://iax:guest@conference.freeswitch.org/888>
>>>>       
>>>>         
>>>>> <http://iax:guest@conference.freeswitch.org/888>
>>>>> googletalk:conf+888 at conference.freeswitch.org
>>>>>         
>>>>>           
>>>>     <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org>
>>>>       
>>>>         
>>>>> <mailto:googletalk%3Aconf%2B888 at conference.freeswitch.org
>>>>>         
>>>>>           
>>>>     <mailto:googletalk%253Aconf%252B888 at conference.freeswitch.org>>
>>>>       
>>>>         
>>>>> pstn:213-799-1400
>>>>>
>>>>>         
>>>>>           
>>>>     ------------------------------------------------------------------------
>>>>       
>>>>         
>>>>> _______________________________________________
>>>>> Freeswitch-users mailing list
>>>>> Freeswitch-users at lists.freeswitch.org
>>>>>         
>>>>>           
>>>>     <mailto:Freeswitch-users at lists.freeswitch.org>
>>>>       
>>>>         
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>
>>>>>         
>>>>>           
>>>>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>       
>>>>         
>>>>> http://www.freeswitch.org
>>>>>
>>>>>         
>>>>>           
>>>>     _______________________________________________
>>>>     Freeswitch-users mailing list
>>>>     Freeswitch-users at lists.freeswitch.org
>>>>     <mailto:Freeswitch-users at lists.freeswitch.org>
>>>>     http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>     UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>     http://www.freeswitch.org
>>>>
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> _______________________________________________
>>>> Freeswitch-users mailing list
>>>> Freeswitch-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>   
>>>>       
>>>>         
>>> _______________________________________________
>>> Freeswitch-users mailing list
>>> Freeswitch-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>     
>>>       
>> _______________________________________________
>> Freeswitch-users mailing list
>> Freeswitch-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>>   
>>     
>
> _______________________________________________
> Freeswitch-users mailing list
> Freeswitch-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
>   




More information about the FreeSWITCH-users mailing list