After stopping the load FS still hogs 22.1% of memory. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 freeswitch
Until you test with the version you're building from master I would ignore the memory usage since you're running a version with known memory leaks. -Steve On 8 March 2013 18:15, bratner bratner <ratner2@gmail.com> wrote:
Here is sipp output and additional numbers for a test I ran with -nosql param.
The test ran 180CPS for ~3500seconds and the rest with 210cps.
Trouble (as in higher system cpu% ) started to appear around 8591seconds into the test. As you can see below the problem started just before 9124sec into the test 210cps 5sec calls should not give you a lot more then 1050 concurrent calls.
------------------------------ Scenario Screen -------- [1-9]: Change Screen -- Call-rate(length) Port Total-time Total-calls Remote-host 210.0(5000 ms)/1.000s 5061 9157.32 s 1834024 192.96.201.164:5060(UDP)
0 new calls during 0.000 s period 0 ms scheduler resolution 0 calls (limit 2000) Peak was 2000 calls, after 9124 s 0 Running, 4640 Paused, 0 Woken up 20 dead call msg (discarded) 0 out-of-call msg (discarded)
1 open sockets
Messages Retrans Timeout Unexpected-Msg INVITE ----------> 1834024 74 0 100 <---------- 1834024 0 0 0 180 <---------- 1834024 0 0 0 183 <---------- 0 0 0 0 500 <---------- 0 0 0 0 502 <---------- 0 0 0 0 503 <---------- 0 0 0 0 408 <---------- 0 0 0 0 480 <---------- 0 0 0 0 200 <---------- E-RTD1 1834024 81 0 0
ACK ----------> 1834024 81 Pause [ 5000ms] 1834024 0 BYE ----------> 1834024 7646 0 503 <---------- 0 0 0 0 200 <---------- 1834024 0 0 0
------------------------------ Test Terminated --------------------------------
----------------------------- Statistics Screen ------- [1-9]: Change Screen -- Start Time | 2013-03-08 15:22:18:204 1362756138.204833 Last Reset Time | 2013-03-08 17:54:55:535 1362765295.535214 Current Time | 2013-03-08 17:54:55:535 1362765295.535437
-------------------------+---------------------------+--------------------------
Counter Name | Periodic value | Cumulative value
-------------------------+---------------------------+--------------------------
Elapsed Time | 00:00:00:000 | 02:32:37:330
Call Rate | 0.000 cps | 200.279 cps
-------------------------+---------------------------+--------------------------
Incoming call created | 0 | 0
OutGoing call created | 0 | 1834024
Total Call created | | 1834024
Current Call | 0 |
-------------------------+---------------------------+--------------------------
Successful call | 0 | 1834024
Failed call | 0 | 0
-------------------------+---------------------------+--------------------------
Response Time 1 | 00:00:00:000 | 00:00:00:149
Call Length | 00:00:00:000 | 00:00:05:158
------------------------------ Test Terminated --------------------------------
After stopping the load FS still hogs 22.1% of memory. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 freeswitch
The symptoms of the crash are the same, just now with higher CPS and takes more time (more calls ) before crashing.
I will appreciate any suggestion.
Regards, Boris Ratner.
On Fri, Mar 8, 2013 at 6:22 PM, bratner bratner <ratner2@gmail.com> wrote:
The original test was done on git master at the date mentioned. The
sqlite
core.db file was on /run/shm which is a tmpfs on unbuntu 12.04. I will be recompiling from git master and test running with -nosql.
Testing my existing setup with -nosql seems more stable now running at 210CPS for some time (500k calls already passed) with ~35% idle cpu. But the free mem is slowly going down. I will let it run untill the kernel will kill it to see how many calls it can handle.
During my tests i did not run FS with RT priority but according to htop some of the threads are scheduled as RT. My setup is doing bypass-media , thus FS handling only call establishment and teardown on both legs.
cat /proc/<FS pid>/status
Name: freeswitch State: S (sleeping) Tgid: 15995 Pid: 15995 PPid: 1 TracerPid: 0 Uid: 999 999 999 999 Gid: 999 999 999 999 FDSize: 64 Groups: VmPeak: 5002808 kB VmSize: 5002088 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 625900 kB VmRSS: 624156 kB <-- this is going up VmData: 4855788 kB VmStk: 136 kB VmExe: 20 kB VmLib: 18288 kB VmPTE: 2352 kB VmSwap: 0 kB Threads: 1866 SigQ: 0/18446744073709551615 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000010003006 SigCgt: 0000000180014209 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: ffffffffffffffff Cpus_allowed: ffffff Cpus_allowed_list: 0-23 Mems_allowed: 00000000,00000003 Mems_allowed_list: 0-1 voluntary_ctxt_switches: 1803 nonvoluntary_ctxt_switches: 23
output of 'top -H' at 180CPS
top - 15:27:00 up 2 days, 5:32, 5 users, load average: 8.19, 91.07, 65.03 Tasks: 2066 total, 3 running, 2063 sleeping, 0 stopped, 0 zombie Cpu(s): 50.1%us, 3.9%sy, 0.0%ni, 45.9%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 4038512k total, 2282260k used, 1756252k free, 114112k buffers Swap: 0k total, 0k used, 0k free, 1165868k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16000 freeswit RT -10 4885m 594m 4964 R 69 15.1 3:10.26 freeswitch
16009 freeswit RT -10 4885m 594m 4964 S 33 15.1 1:26.20 freeswitch
16008 freeswit RT -10 4885m 594m 4964 S 28 15.1 1:17.30 freeswitch
16007 freeswit RT -10 4885m 594m 4964 S 4 15.1 0:10.80 freeswitch
16004 freeswit RT -10 4885m 594m 4964 S 2 15.1 0:06.63 freeswitch
19171 root 20 0 18988 2948 944 R 2 0.1 0:00.64 top
18735 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.29 freeswitch
16003 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:01.61 freeswitch
16690 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
16730 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
16750 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.45 freeswitch
16764 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.44 freeswitch
<more of the above> .... ....
Thanks to all of you , Boris Ratner.
On Fri, Mar 8, 2013 at 4:22 AM, Dmitry Lysenko <dvl36.ripe.nick@gmail.com
wrote:
I can't reproduce such cps load on my ARMv5TE system. ) bratner, please give us 'top -H'. I guess freeswitch running at realtime priority.
2013/3/8 Ken Rice <krice@freeswitch.org>
Sqlite is probably getting hammered... Trust me... Mount the fs db dir as tmpfs or use the –nosql flag when starting freeswitch
I routinely run dialer traffic at much higher CPS then that
On 3/7/13 7:58 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com> wrote:
bi, bo and wa field is low, so it seems that is not disk subsystem.
2013/3/8 Ken Rice <krice@freeswitch.org>
You are probably hammering the disk subsystem... Keep in mind that FS uses multiple sqlite databases by default... Mount the fs db dir as
tmpfs
and try again
On 3/7/13 7:35 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com <http://dvl36.ripe.nick@gmail.com> > wrote:
Hm... But what about huge interrupt and context switching number?
________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org irc.freenode.net #freeswitch
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org