[Freeswitch-users] [SOLVED] High cps load causes weird cpu and memory starvation. Need suggestions on how to debug.

bratner bratner ratner2 at gmail.com
Sun Mar 10 16:17:39 MSK 2013


List, Dmitry

Everything looks fine @180CPS and I can conclude that running FS with -np
together with switching the kernel to ubuntu's low latency image
solved the problem. The mem leak remains but this I can handle by selective
restarts.

Thanks!
Boris Ratner


On Sun, Mar 10, 2013 at 12:35 PM, bratner bratner <ratner2 at gmail.com> wrote:

> Dmitry, Hi!
>
> Running with -np at 180CPS for 3000sec now (over 500k calls). I already
> passed by far the amount of calls i was able to do at this CPS previously.
> I can see that all FS threads are the same priority and there are no RT
> threads.
> Context switches per ser are rising slowly. If i can make a million calls
> it is good enough for me.
> There is a small mem leak but that is not what have me worried because i
> can monitor it and restart FS when necessary.
> In my previous tests when CS reached closer to 60k the spiral down began.
>
> You think that FS RT threads slowly starve another important task?
>
> Holding my fingers crossed.
>
> Thanks!
> Boris Ratner.
>
>
>
> On Sun, Mar 10, 2013 at 2:02 AM, Dmitry Lysenko <dvl36.ripe.nick at gmail.com
> > wrote:
>
>> Boris, did you try to test load forcing freeswitch to run with normal
>> priority? (-np)
>> It seems that I have workaround, but don't sure exactly that your cpu
>> load issue has the same root as mine. My system setup is uncommon
>> (arm,128mb of RAM,RT kernel,mod_gsmopen), so I can't test it myself.
>>
>>
>> 2013/3/10 bratner bratner <ratner2 at gmail.com>
>>
>>> List, Steve
>>>
>>> I will clarify what i'm asking here before I take Anothny's suggestion
>>> and join a "computer tuning" club as a way to "move forward".
>>> http://media.bestofmicro.com/gerbilpc-tuning-pc,S-L-252453-13.jpg
>>>
>>> What is there to read on this subject? Links, textbook names -
>>> everything is appreciated.
>>> What are the tools that show useful data and what i can do with FS to
>>> make the work easier? Compile with some flags to get more info on running
>>> threads?
>>>
>>> Thanks,
>>> Boris Ratner.
>>>
>>>
>>> On Sat, Mar 9, 2013 at 12:51 AM, Steven Ayre <steveayre at gmail.com>wrote:
>>>
>>>> After stopping the load FS still hogs 22.1% of memory.
>>>>>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>>>
>>>>>
>>>>> 15995 freeswit  -2 -10 4677m 873m 5028 S    0 22.1 755:28.65
>>>>> freeswitch
>>>>
>>>>
>>>> Until you test with the version you're building from master I would
>>>> ignore the memory usage since you're running a version with known memory
>>>> leaks.
>>>>
>>>> -Steve
>>>>
>>>>
>>>>
>>>>
>>>> On 8 March 2013 18:15, bratner bratner <ratner2 at gmail.com> wrote:
>>>> > Here is sipp output and additional numbers for a test I ran with
>>>> -nosql
>>>> > param.
>>>> >
>>>> > The test ran 180CPS for ~3500seconds and the rest with 210cps.
>>>> >
>>>> > Trouble (as in higher system cpu% ) started to appear around
>>>> 8591seconds
>>>> > into the test.
>>>> > As you can see below the problem started just before 9124sec into the
>>>> test
>>>> > 210cps 5sec calls
>>>> > should not give you a lot more then 1050 concurrent calls.
>>>> >
>>>> > ------------------------------ Scenario Screen -------- [1-9]: Change
>>>> Screen
>>>> > --
>>>> >   Call-rate(length)   Port   Total-time  Total-calls  Remote-host
>>>> > 210.0(5000 ms)/1.000s   5061    9157.32 s      1834024
>>>> > 192.96.201.164:5060(UDP)
>>>> >
>>>> >   0 new calls during 0.000 s period      0 ms scheduler resolution
>>>> >   0 calls (limit 2000)                   Peak was 2000 calls, after
>>>> 9124 s
>>>> >   0 Running, 4640 Paused, 0 Woken up
>>>> >   20 dead call msg (discarded)           0 out-of-call msg
>>>> (discarded)
>>>> >   1 open sockets
>>>> >
>>>> >                                  Messages  Retrans   Timeout
>>>> > Unexpected-Msg
>>>> >       INVITE ---------->         1834024   74        0
>>>>
>>>> >          100 <----------         1834024   0         0         0
>>>>
>>>> >          180 <----------         1834024   0         0         0
>>>>
>>>> >          183 <----------         0         0         0         0
>>>>
>>>> >          500 <----------         0         0         0         0
>>>>
>>>> >          502 <----------         0         0         0         0
>>>>
>>>> >          503 <----------         0         0         0         0
>>>>
>>>> >          408 <----------         0         0         0         0
>>>>
>>>> >          480 <----------         0         0         0         0
>>>>
>>>> >          200 <----------  E-RTD1 1834024   81        0         0
>>>>
>>>> >
>>>> >          ACK ---------->         1834024   81
>>>>
>>>> >        Pause [   5000ms]         1834024                       0
>>>>
>>>> >          BYE ---------->         1834024   7646      0
>>>>
>>>> >          503 <----------         0         0         0         0
>>>>
>>>> >          200 <----------         1834024   0         0         0
>>>>
>>>> >
>>>> > ------------------------------ Test Terminated
>>>> > --------------------------------
>>>> >
>>>> >
>>>> > ----------------------------- Statistics Screen ------- [1-9]: Change
>>>> Screen
>>>> > --
>>>> >   Start Time             | 2013-03-08    15:22:18:204
>>>>  1362756138.204833
>>>> >   Last Reset Time        | 2013-03-08    17:54:55:535
>>>>  1362765295.535214
>>>> >   Current Time           | 2013-03-08    17:54:55:535
>>>>  1362765295.535437
>>>> >
>>>> -------------------------+---------------------------+--------------------------
>>>> >   Counter Name           | Periodic value            | Cumulative
>>>> value
>>>> >
>>>> -------------------------+---------------------------+--------------------------
>>>> >   Elapsed Time           | 00:00:00:000              | 02:32:37:330
>>>>
>>>> >   Call Rate              |    0.000 cps              |  200.279 cps
>>>>
>>>> >
>>>> -------------------------+---------------------------+--------------------------
>>>> >   Incoming call created  |        0                  |        0
>>>>
>>>> >   OutGoing call created  |        0                  |  1834024
>>>>
>>>> >   Total Call created     |                           |  1834024
>>>>
>>>> >   Current Call           |        0                  |
>>>>
>>>> >
>>>> -------------------------+---------------------------+--------------------------
>>>> >   Successful call        |        0                  |  1834024
>>>>
>>>> >   Failed call            |        0                  |        0
>>>>
>>>> >
>>>> -------------------------+---------------------------+--------------------------
>>>> >   Response Time 1        | 00:00:00:000              | 00:00:00:149
>>>>
>>>> >   Call Length            | 00:00:00:000              | 00:00:05:158
>>>>
>>>> > ------------------------------ Test Terminated
>>>> > --------------------------------
>>>> >
>>>> >
>>>> > After stopping the load FS still hogs 22.1% of memory.
>>>> >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>>
>>>>
>>>> > 15995 freeswit  -2 -10 4677m 873m 5028 S    0 22.1 755:28.65
>>>> freeswitch
>>>>
>>>> >
>>>> >
>>>> > The symptoms of the crash are the same, just now with higher CPS and
>>>> takes
>>>> > more time (more calls ) before crashing.
>>>> >
>>>> > I will appreciate any suggestion.
>>>> >
>>>> > Regards,
>>>> > Boris Ratner.
>>>> >
>>>> >
>>>> >
>>>> > On Fri, Mar 8, 2013 at 6:22 PM, bratner bratner <ratner2 at gmail.com>
>>>> wrote:
>>>> >>
>>>> >> The original test was done on git master at the date mentioned. The
>>>> sqlite
>>>> >> core.db file was on /run/shm which is a tmpfs on unbuntu 12.04.
>>>> >> I will be recompiling from git master and test running with -nosql.
>>>> >>
>>>> >> Testing my existing setup with -nosql seems more stable now running
>>>> at
>>>> >> 210CPS for some time (500k calls already passed) with ~35% idle cpu.
>>>> >> But the free mem is slowly going down. I will let it run untill the
>>>> kernel
>>>> >> will kill it to see how many calls it can handle.
>>>> >>
>>>> >> During my tests i did not run FS with RT priority but according to
>>>> htop
>>>> >> some of the threads are scheduled as RT.
>>>> >> My setup is doing bypass-media , thus FS handling only call
>>>> establishment
>>>> >> and teardown on both legs.
>>>> >>
>>>> >> cat /proc/<FS pid>/status
>>>> >>
>>>> >> Name:   freeswitch
>>>> >> State:  S (sleeping)
>>>> >> Tgid:   15995
>>>> >> Pid:    15995
>>>> >> PPid:   1
>>>> >> TracerPid:      0
>>>> >> Uid:    999     999     999     999
>>>> >> Gid:    999     999     999     999
>>>> >> FDSize: 64
>>>> >> Groups:
>>>> >> VmPeak:  5002808 kB
>>>> >> VmSize:  5002088 kB
>>>> >> VmLck:         0 kB
>>>> >> VmPin:         0 kB
>>>> >> VmHWM:    625900 kB
>>>> >> VmRSS:    624156 kB  <-- this is going up
>>>> >> VmData:  4855788 kB
>>>> >> VmStk:       136 kB
>>>> >> VmExe:        20 kB
>>>> >> VmLib:     18288 kB
>>>> >> VmPTE:      2352 kB
>>>> >> VmSwap:        0 kB
>>>> >> Threads:        1866
>>>> >> SigQ:   0/18446744073709551615
>>>> >> SigPnd: 0000000000000000
>>>> >> ShdPnd: 0000000000000000
>>>> >> SigBlk: 0000000000000000
>>>> >> SigIgn: 0000000010003006
>>>> >> SigCgt: 0000000180014209
>>>> >> CapInh: 0000000000000000
>>>> >> CapPrm: 0000000000000000
>>>> >> CapEff: 0000000000000000
>>>> >> CapBnd: ffffffffffffffff
>>>> >> Cpus_allowed:   ffffff
>>>> >> Cpus_allowed_list:      0-23
>>>> >> Mems_allowed:   00000000,00000003
>>>> >> Mems_allowed_list:      0-1
>>>> >> voluntary_ctxt_switches:        1803
>>>> >> nonvoluntary_ctxt_switches:     23
>>>> >>
>>>> >>
>>>> >> output of 'top -H' at 180CPS
>>>> >>
>>>> >>
>>>> >> top - 15:27:00 up 2 days,  5:32,  5 users,  load average: 8.19,
>>>> 91.07,
>>>> >> 65.03
>>>> >> Tasks: 2066 total,   3 running, 2063 sleeping,   0 stopped,   0
>>>> zombie
>>>> >> Cpu(s): 50.1%us,  3.9%sy,  0.0%ni, 45.9%id,  0.0%wa,  0.0%hi,
>>>>  0.2%si,
>>>> >> 0.0%st
>>>> >> Mem:   4038512k total,  2282260k used,  1756252k free,   114112k
>>>> buffers
>>>> >> Swap:        0k total,        0k used,        0k free,  1165868k
>>>> cached
>>>> >>
>>>> >>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>>
>>>>
>>>> >> 16000 freeswit  RT -10 4885m 594m 4964 R   69 15.1   3:10.26
>>>> freeswitch
>>>>
>>>> >> 16009 freeswit  RT -10 4885m 594m 4964 S   33 15.1   1:26.20
>>>> freeswitch
>>>>
>>>> >> 16008 freeswit  RT -10 4885m 594m 4964 S   28 15.1   1:17.30
>>>> freeswitch
>>>>
>>>> >> 16007 freeswit  RT -10 4885m 594m 4964 S    4 15.1   0:10.80
>>>> freeswitch
>>>>
>>>> >> 16004 freeswit  RT -10 4885m 594m 4964 S    2 15.1   0:06.63
>>>> freeswitch
>>>>
>>>> >> 19171 root      20   0 18988 2948  944 R    2  0.1   0:00.64 top
>>>>
>>>>
>>>> >> 18735 freeswit  -2 -10 4885m 594m 4964 S    1 15.1   0:00.29
>>>> freeswitch
>>>>
>>>> >> 16003 freeswit  -2 -10 4885m 594m 4964 S    1 15.1   0:01.61
>>>> freeswitch
>>>>
>>>> >> 16690 freeswit  -2 -10 4885m 594m 4964 S    1 15.1   0:00.42
>>>> freeswitch
>>>>
>>>> >> 16730 freeswit  -2 -10 4885m 594m 4964 S    1 15.1   0:00.42
>>>> freeswitch
>>>>
>>>> >> 16750 freeswit  -2 -10 4885m 594m 4964 S    1 15.1   0:00.45
>>>> freeswitch
>>>>
>>>> >> 16764 freeswit  -2 -10 4885m 594m 4964 S    1 15.1   0:00.44
>>>> freeswitch
>>>>
>>>> >> <more of the above>
>>>> >> ....
>>>> >> ....
>>>> >>
>>>> >>
>>>> >> Thanks to all of you ,
>>>> >> Boris Ratner.
>>>> >>
>>>> >> On Fri, Mar 8, 2013 at 4:22 AM, Dmitry Lysenko <
>>>> dvl36.ripe.nick at gmail.com>
>>>> >> wrote:
>>>> >>>
>>>> >>> I can't reproduce such cps load on my ARMv5TE system. )
>>>> >>> bratner, please give us 'top -H'. I guess freeswitch running at
>>>> realtime
>>>> >>> priority.
>>>> >>>
>>>> >>>
>>>> >>> 2013/3/8 Ken Rice <krice at freeswitch.org>
>>>> >>>>
>>>> >>>> Sqlite is probably getting hammered... Trust me... Mount the fs db
>>>> dir
>>>> >>>> as tmpfs or use the –nosql flag when starting freeswitch
>>>> >>>>
>>>> >>>> I routinely run dialer traffic at much higher CPS then that
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> On 3/7/13 7:58 PM, "Dmitry Lysenko" <dvl36.ripe.nick at gmail.com>
>>>> wrote:
>>>> >>>>
>>>> >>>> bi, bo and wa field is low, so it seems that is not disk subsystem.
>>>> >>>>
>>>> >>>>
>>>> >>>> 2013/3/8 Ken Rice <krice at freeswitch.org>
>>>> >>>>
>>>> >>>> You are probably hammering the disk subsystem... Keep in mind that
>>>> FS
>>>> >>>> uses multiple sqlite databases by default... Mount the fs db dir
>>>> as tmpfs
>>>> >>>> and try again
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> On 3/7/13 7:35 PM, "Dmitry Lysenko" <dvl36.ripe.nick at gmail.com
>>>> >>>> <http://dvl36.ripe.nick@gmail.com> > wrote:
>>>> >>>>
>>>> >>>> Hm... But what about huge interrupt and context switching  number?
>>>> >>>>
>>>> >>>>
>>>> >>>> ________________________________
>>>> >>>>
>>>> >>>>
>>>> _________________________________________________________________________
>>>> >>>> Professional FreeSWITCH Consulting Services:
>>>> >>>> consulting at freeswitch.org
>>>> >>>> http://www.freeswitchsolutions.com
>>>> >>>>
>>>> >>>> 
>>>> >>>> 
>>>> >>>>
>>>> >>>> Official FreeSWITCH Sites
>>>> >>>> http://www.freeswitch.org
>>>> >>>> http://wiki.freeswitch.org
>>>> >>>> http://www.cluecon.com
>>>> >>>>
>>>> >>>> FreeSWITCH-users mailing list
>>>> >>>> FreeSWITCH-users at lists.freeswitch.org
>>>> >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> >>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> >>>> http://www.freeswitch.org
>>>> >>>>
>>>> >>>>
>>>> >>>> --
>>>> >>>> Ken
>>>> >>>> http://www.FreeSWITCH.org
>>>> >>>> http://www.ClueCon.com
>>>> >>>> http://www.OSTAG.org
>>>> >>>> irc.freenode.net #freeswitch
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> _________________________________________________________________________
>>>> >>>> Professional FreeSWITCH Consulting Services:
>>>> >>>> consulting at freeswitch.org
>>>> >>>> http://www.freeswitchsolutions.com
>>>> >>>>
>>>> >>>> 
>>>> >>>> 
>>>> >>>>
>>>> >>>> Official FreeSWITCH Sites
>>>> >>>> http://www.freeswitch.org
>>>> >>>> http://wiki.freeswitch.org
>>>> >>>> http://www.cluecon.com
>>>> >>>>
>>>> >>>> FreeSWITCH-users mailing list
>>>> >>>> FreeSWITCH-users at lists.freeswitch.org
>>>> >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> >>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> >>>> http://www.freeswitch.org
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> _________________________________________________________________________
>>>> >>> Professional FreeSWITCH Consulting Services:
>>>> >>> consulting at freeswitch.org
>>>> >>> http://www.freeswitchsolutions.com
>>>> >>>
>>>> >>> 
>>>> >>> 
>>>> >>>
>>>> >>> Official FreeSWITCH Sites
>>>> >>> http://www.freeswitch.org
>>>> >>> http://wiki.freeswitch.org
>>>> >>> http://www.cluecon.com
>>>> >>>
>>>> >>> FreeSWITCH-users mailing list
>>>> >>> FreeSWITCH-users at lists.freeswitch.org
>>>> >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> >>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> >>> http://www.freeswitch.org
>>>> >>>
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> _________________________________________________________________________
>>>> > Professional FreeSWITCH Consulting Services:
>>>> > consulting at freeswitch.org
>>>> > http://www.freeswitchsolutions.com
>>>> >
>>>> > 
>>>> > 
>>>> >
>>>> > Official FreeSWITCH Sites
>>>> > http://www.freeswitch.org
>>>> > http://wiki.freeswitch.org
>>>> > http://www.cluecon.com
>>>> >
>>>> > FreeSWITCH-users mailing list
>>>> > FreeSWITCH-users at lists.freeswitch.org
>>>> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> > UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> > http://www.freeswitch.org
>>>> >
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>> 
>>>> 
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://wiki.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>> 
>>> 
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>>
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20130310/0be817ef/attachment-0001.html 


Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users mailing list