[Freeswitch-users] High cps load causes weird cpu and memory starvation. Need suggestions on how to debug.

Fri Mar 8 19:22:07 MSK 2013

The original test was done on git master at the date mentioned. The sqlite
core.db file was on /run/shm which is a tmpfs on unbuntu 12.04.
I will be recompiling from git master and test running with -nosql.

Testing my existing setup with -nosql seems more stable now running at
210CPS for some time (500k calls already passed) with ~35% idle cpu.
But the free mem is slowly going down. I will let it run untill the kernel
will kill it to see how many calls it can handle.

During my tests i did not run FS with RT priority but according to htop
some of the threads are scheduled as RT.
My setup is doing bypass-media , thus FS handling only call establishment
and teardown on both legs.

cat /proc/<FS pid>/status

Name:   freeswitch
State:  S (sleeping)
Tgid:   15995
Pid:    15995
PPid:   1
TracerPid:      0
Uid:    999     999     999     999
Gid:    999     999     999     999
FDSize: 64
Groups:
VmPeak:  5002808 kB
VmSize:  5002088 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:    625900 kB
VmRSS:    624156 kB  <-- this is going up
VmData:  4855788 kB
VmStk:       136 kB
VmExe:        20 kB
VmLib:     18288 kB
VmPTE:      2352 kB
VmSwap:        0 kB
Threads:        1866
SigQ:   0/18446744073709551615
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000010003006
SigCgt: 0000000180014209
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: ffffffffffffffff
Cpus_allowed:   ffffff
Cpus_allowed_list:      0-23
Mems_allowed:   00000000,00000003
Mems_allowed_list:      0-1
voluntary_ctxt_switches:        1803
nonvoluntary_ctxt_switches:     23

output of 'top -H' at 180CPS

top - 15:27:00 up 2 days,  5:32,  5 users,  load average: 8.19, 91.07, 65.03
Tasks: 2066 total,   3 running, 2063 sleeping,   0 stopped,   0 zombie
Cpu(s): 50.1%us,  3.9%sy,  0.0%ni, 45.9%id,  0.0%wa,  0.0%hi,  0.2%si,
0.0%st
Mem:   4038512k total,  2282260k used,  1756252k free,   114112k buffers
Swap:        0k total,        0k used,        0k free,  1165868k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+
COMMAND

16000 freeswit  RT -10 4885m 594m 4964 R   69 15.1   3:10.26
freeswitch

16009 freeswit  RT -10 4885m 594m 4964 S   33 15.1   1:26.20
freeswitch

16008 freeswit  RT -10 4885m 594m 4964 S   28 15.1   1:17.30
freeswitch

16007 freeswit  RT -10 4885m 594m 4964 S    4 15.1   0:10.80
freeswitch

16004 freeswit  RT -10 4885m 594m 4964 S    2 15.1   0:06.63
freeswitch

19171 root      20   0 18988 2948  944 R    2  0.1   0:00.64
top

18735 freeswit  -2 -10 4885m 594m 4964 S    1 15.1   0:00.29
freeswitch

16003 freeswit  -2 -10 4885m 594m 4964 S    1 15.1   0:01.61
freeswitch

16690 freeswit  -2 -10 4885m 594m 4964 S    1 15.1   0:00.42
freeswitch

16730 freeswit  -2 -10 4885m 594m 4964 S    1 15.1   0:00.42
freeswitch

16750 freeswit  -2 -10 4885m 594m 4964 S    1 15.1   0:00.45
freeswitch

16764 freeswit  -2 -10 4885m 594m 4964 S    1 15.1   0:00.44
freeswitch

<more of the above>
....
....

Thanks to all of you ,
Boris Ratner.

On Fri, Mar 8, 2013 at 4:22 AM, Dmitry Lysenko <dvl36.ripe.nick at gmail.com>wrote:

> I can't reproduce such cps load on my ARMv5TE system. )
> bratner, please give us 'top -H'. I guess freeswitch running at realtime
> priority.
>
>
> 2013/3/8 Ken Rice <krice at freeswitch.org>
>
>>  Sqlite is probably getting hammered... Trust me... Mount the fs db dir
>> as tmpfs or use the –nosql flag when starting freeswitch
>>
>> I routinely run dialer traffic at much higher CPS then that
>>
>>
>>
>> On 3/7/13 7:58 PM, "Dmitry Lysenko" <dvl36.ripe.nick at gmail.com> wrote:
>>
>> bi, bo and wa field is low, so it seems that is not disk subsystem.
>>
>>
>> 2013/3/8 Ken Rice <krice at freeswitch.org>
>>
>> You are probably hammering the disk subsystem... Keep in mind that FS
>> uses multiple sqlite databases by default... Mount the fs db dir as tmpfs
>> and try again
>>
>>
>>
>> On 3/7/13 7:35 PM, "Dmitry Lysenko" <dvl36.ripe.nick at gmail.com <
>> http://dvl36.ripe.nick@gmail.com> > wrote:
>>
>> Hm... But what about huge interrupt and context switching  number?
>>
>>
>> ------------------------------
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>> --
>> Ken
>> *http://www.FreeSWITCH.org
>> http://www.ClueCon.com
>> http://www.OSTAG.org
>> *irc.freenode.net #freeswitch
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>> 
>> 
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> 
> 
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20130308/22d0487f/attachment.html 