High cps load causes weird cpu and memory starvation. Need suggestions on how to debug.
Hi! Please assist me to find a way to get some insight into this problem and file a proper bug report. Looks like 2 different problems, one is load related (like a race of some sort) the other is a mem leak. I'm running FS debs compiled on ubuntu 12.04 (-n20130227T114536Z-1~presice+ 1git). The machine is ubuntu 12.04 server , CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 4GB RAM and no swap. 3.1G RAM free after FS is started. I disabled HT in BIOS to get proper CPU load readings from top and vmstat (no HT also allows for higher cps for some reason...). The FS setup has 2 SIP profiles , internal/external (functioning as incoming/outgoing ), XML Dialplan that runs a lua script. Calls come in to profile 'internal', authed by ip acl , go to the dialplan and sent through profile 'external' to a uri string generated by the lua script. I bombard the machine with high call rate of short calls ,120-180cps , 5sec. Everything stays fine for some time , the higher the load the faster the problems appear. When i pressure the machine with 150cps and up , it works fine at first with 20%-30% idle CPU. After some time passes (10-30minutes) system cpu usage jumps from 3%-7% to around 50%, CPU is 95% busy and vmstat reports hundereds and sometimes thousands of runnable tasks. If i stop sending calls at this moment, wait for the CPU load to wind down and start calling again then the CPU load saturates right away. Even if i lower the CPS rate. restarting freeswitch at this point brings the system to the original state and it will take some time to get saturated again. Another symptom is that during the test the available free mem is slowly allocated to freeswitch until it reaches 13.1Gb virtual image and then the kernel kills it. Here are some numbers: Running 120 calls per second , 5second calls: Starts like this: This is very good from my POV. procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 5 0 0 3255876 35444 96580 0 0 9 2 66 90 1 1 98 0 4 0 0 3253080 35444 96652 0 0 0 0 16048 20727 36 3 61 0 4 0 0 3247780 35444 96684 0 0 0 0 16521 21048 36 4 60 0 4 0 0 3242860 35444 96784 0 0 0 0 15695 19825 36 3 62 0 1 0 0 3237980 35452 96768 0 0 0 24 16072 19851 36 3 60 0 4 0 0 3233692 35452 96812 0 0 0 0 16299 20980 35 4 61 0 8 0 0 3230280 35452 96928 0 0 0 0 15034 18965 36 3 61 0 2 0 0 3227764 35452 96896 0 0 0 0 13956 20621 37 3 60 0 After some (long) time it started having trouble and eat lots of free ram (note that the idle cpu% and system cpu% jumped up but there are still over 1G free mem available) : procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 22 0 0 1108588 100304 2034628 0 0 0 0 145350 183629 37 53 10 0 5 0 0 1108224 100304 2034668 0 0 0 0 151937 167249 38 57 5 0 11 0 0 1108268 100304 2034904 0 0 0 0 149931 176143 37 58 5 0 7 0 0 1108244 100312 2034752 0 0 0 32 138612 183583 40 47 14 0 3 0 0 1108056 100312 2034792 0 0 0 0 111446 162733 42 29 29 0 4 0 0 1108064 100312 2034836 0 0 0 0 112568 164187 41 30 29 0 14 0 0 1108040 100312 2034880 0 0 0 0 133372 174676 39 45 17 0 10 0 0 1107468 100312 2034920 0 0 0 0 157371 184888 37 58 5 0 Short time after that is got into really serious trouble (3000 runnable tasks!? heading toward memory exhaustion): procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 3336 0 0 146580 9248 1457016 0 0 0 0 306642 291819 28 67 5 0 1180 0 0 145060 9220 1430212 0 0 248 848 294654 285273 27 68 5 0 1028 0 0 148616 8996 1366844 0 0 0 0 421614 385342 32 63 4 0 1026 0 0 150964 8880 1298832 0 0 0 28 462825 416480 31 65 4 0 1338 0 0 146260 8688 1242440 0 0 0 0 442247 391139 27 69 5 0 1945 0 0 143148 8680 1197180 0 0 0 0 364361 329958 24 71 5 0 1381 0 0 154668 8596 1137912 0 0 0 0 359982 323543 23 72 5 0 After UAC completed transactions 5198242/ UAS completed transactions 5512619 calls FS crashed and it looked like this: procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 3 2 0 92972 568 20492 0 0 156 8 117314 130220 7 40 38 15 <-- back in the day i would kill for 92Mb free mem. 10 5 0 3473328 208 9848 0 0 24344 48 104557 89011 5 62 15 17 <- Now the kernel kills FS for taking the rest of it. 8 0 0 3482436 340 14408 0 0 3208 0 2582 2046 0 95 4 1 9 0 0 3504792 348 14344 0 0 0 48 2312 1836 0 93 7 0 12 0 0 3532184 348 14348 0 0 0 0 2387 1984 0 94 6 0 6 0 0 3561792 348 14876 0 0 528 0 2523 2178 0 93 6 0 11 0 0 3592816 348 14876 0 0 0 0 2684 2248 0 95 5 0 3 0 0 3615424 348 14876 0 0 0 0 2820 2494 0 95 5 0 7 0 0 3633488 348 14876 0 0 0 0 2990 2805 0 94 6 0 7 0 0 3654916 348 14876 0 0 0 0 3350 3222 0 96 4 0 11 0 0 3680708 348 14876 0 0 0 0 3869 4018 0 94 6 0 0 0 0 3773532 476 16876 0 0 2344 0 6598 9501 0 74 24 2 0 0 0 3773836 476 16912 0 0 0 0 191 129 0 0 100 0 <- FS is not running any more, 100% idle CPU , lots of free ram 0 0 0 3773856 476 17108 0 0 0 0 91 63 0 0 100 0 dmesg of the crash is this: [48674.576126] Out of memory: Kill process 31552 (freeswitch) score 674 or sacrifice child [48674.578188] Killed process 31552 (freeswitch) total-vm:10177480kB, anon-rss:2710092kB, file-rss:0kB fs_log switches to the dark side : (there was no disk space problem of course, i have no idea at which time point in the spiral down this popped on the console ) 2013-03-06 23:27:22.923640 [ERR] switch_core_sqldb.c:579 NATIVE SQL ERR [database or disk is full] BEGIN EXCLUSIVE 2013-03-06 23:27:22.923640 [CRIT] switch_core_sqldb.c:1679 ERROR [database or disk is full] 2013-03-06 23:27:22.923640 [ERR] switch_core_sqldb.c:579 NATIVE SQL ERR [cannot commit - no transaction is active] COMMIT 2013-03-06 23:27:22.963609 [CRIT] switch_core_session.c:1674 Thread Failure! 2013-03-06 23:27:22.963609 [CRIT] switch_core_session.c:1634 LUKE: I'm hit, but not bad. 2013-03-06 23:27:22.963609 [CRIT] switch_core_session.c:1635 LUKE'S VOICE: Artoo, see what you can do with it. Hang on back there.... Green laserfire moves past the beeping little robot as his head turns. After a few beeps and a twist of his mechanical arm, Artoo reduces the max sessions to 13897 thus, saving the switch from certain doom. and thats all , FS is dead. Runing the test at 180 cps will significantly hasten the death. It starts normal with ~40% idle cpu then after circa 160k calls it starts to spiral: procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 6 0 0 2901484 63204 161132 0 0 0 0 20586 33073 54 5 41 0 5 0 0 2901468 63212 160988 0 0 0 12 25359 41915 53 6 42 0 4 0 0 2901220 63212 161064 0 0 0 0 19650 31211 55 5 40 0 7 0 0 2901080 63212 161124 0 0 0 0 21148 33462 54 5 41 0 2 0 0 2900676 63212 161188 0 0 0 0 21767 35568 54 5 42 0 5 0 0 2900468 63212 161256 0 0 0 0 19398 30073 54 5 41 0 2 0 0 2900088 63220 161304 0 0 0 12 21315 34563 54 5 42 0 3 0 0 2900180 63220 161484 0 0 0 0 19858 31561 42 4 54 0 <-- this should not happen the load is constant and idle cpu% should stay at ~40% 3 0 0 2899768 63220 161520 0 0 0 0 29907 46883 39 4 57 0 11 0 0 2900056 63220 161504 0 0 0 0 41730 63440 41 5 54 0 9 0 0 2899536 63220 161768 0 0 0 0 48876 68159 40 5 54 0 6 0 0 2899512 63228 161620 0 0 0 12 59988 80421 41 6 53 0 49 0 0 2876072 63228 162004 0 0 0 0 193384 241453 57 23 21 0 <-- here we go, starting the memory hogging cpu eating spiral 40 0 0 2830720 63228 162080 0 0 0 0 308948 356178 51 49 0 0 156 0 0 2776576 63228 161984 0 0 0 0 301514 325415 48 48 5 0 178 0 0 2727044 63228 162260 0 0 0 0 290168 306158 44 51 5 0 70 0 0 2683252 63236 161924 0 0 0 48 291792 302632 37 58 5 0 5 0 0 2618432 63236 162204 0 0 0 0 316170 302886 35 59 7 0 2 0 0 2548092 63236 162364 0 0 0 0 340707 319105 38 56 5 0 2 0 0 2460528 63236 162464 0 0 0 1596 386953 348210 40 54 6 0 Nothing on fs warn/err log at this point until memory is all full. Please give me some ideas on how to get some useful info for a bug report. Kind Regards, Boris Ratner.
Upgrade to the latest version. There are some memory leaks that have been found and fixed in 1.2.7. -Steve On 7 March 2013 11:09, bratner bratner <ratner2@gmail.com> wrote:
Hi!
Please assist me to find a way to get some insight into this problem and file a proper bug report. Looks like 2 different problems, one is load related (like a race of some sort) the other is a mem leak.
I'm running FS debs compiled on ubuntu 12.04 (-n20130227T114536Z-1~presice+ 1git). The machine is ubuntu 12.04 server , CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 4GB RAM and no swap. 3.1G RAM free after FS is started. I disabled HT in BIOS to get proper CPU load readings from top and vmstat (no HT also allows for higher cps for some reason...).
The FS setup has 2 SIP profiles , internal/external (functioning as incoming/outgoing ), XML Dialplan that runs a lua script. Calls come in to profile 'internal', authed by ip acl , go to the dialplan and sent through profile 'external' to a uri string generated by the lua script.
I bombard the machine with high call rate of short calls ,120-180cps , 5sec. Everything stays fine for some time , the higher the load the faster the problems appear. When i pressure the machine with 150cps and up , it works fine at first with 20%-30% idle CPU. After some time passes (10-30minutes) system cpu usage jumps from 3%-7% to around 50%, CPU is 95% busy and vmstat reports hundereds and sometimes thousands of runnable tasks. If i stop sending calls at this moment, wait for the CPU load to wind down and start calling again then the CPU load saturates right away. Even if i lower the CPS rate. restarting freeswitch at this point brings the system to the original state and it will take some time to get saturated again.
Another symptom is that during the test the available free mem is slowly allocated to freeswitch until it reaches 13.1Gb virtual image and then the kernel kills it.
Here are some numbers:
Running 120 calls per second , 5second calls:
Starts like this: This is very good from my POV.
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 5 0 0 3255876 35444 96580 0 0 9 2 66 90 1 1 98 0 4 0 0 3253080 35444 96652 0 0 0 0 16048 20727 36 3 61 0 4 0 0 3247780 35444 96684 0 0 0 0 16521 21048 36 4 60 0 4 0 0 3242860 35444 96784 0 0 0 0 15695 19825 36 3 62 0 1 0 0 3237980 35452 96768 0 0 0 24 16072 19851 36 3 60 0 4 0 0 3233692 35452 96812 0 0 0 0 16299 20980 35 4 61 0 8 0 0 3230280 35452 96928 0 0 0 0 15034 18965 36 3 61 0 2 0 0 3227764 35452 96896 0 0 0 0 13956 20621 37 3 60 0
After some (long) time it started having trouble and eat lots of free ram (note that the idle cpu% and system cpu% jumped up but there are still over 1G free mem available) :
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 22 0 0 1108588 100304 2034628 0 0 0 0 145350 183629 37 53 10 0 5 0 0 1108224 100304 2034668 0 0 0 0 151937 167249 38 57 5 0 11 0 0 1108268 100304 2034904 0 0 0 0 149931 176143 37 58 5 0 7 0 0 1108244 100312 2034752 0 0 0 32 138612 183583 40 47 14 0 3 0 0 1108056 100312 2034792 0 0 0 0 111446 162733 42 29 29 0 4 0 0 1108064 100312 2034836 0 0 0 0 112568 164187 41 30 29 0 14 0 0 1108040 100312 2034880 0 0 0 0 133372 174676 39 45 17 0 10 0 0 1107468 100312 2034920 0 0 0 0 157371 184888 37 58 5 0
Short time after that is got into really serious trouble (3000 runnable tasks!? heading toward memory exhaustion):
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa
3336 0 0 146580 9248 1457016 0 0 0 0 306642 291819 28 67 5 0 1180 0 0 145060 9220 1430212 0 0 248 848 294654 285273 27 68 5 0 1028 0 0 148616 8996 1366844 0 0 0 0 421614 385342 32 63 4 0 1026 0 0 150964 8880 1298832 0 0 0 28 462825 416480 31 65 4 0 1338 0 0 146260 8688 1242440 0 0 0 0 442247 391139 27 69 5 0 1945 0 0 143148 8680 1197180 0 0 0 0 364361 329958 24 71 5 0 1381 0 0 154668 8596 1137912 0 0 0 0 359982 323543 23 72 5 0
After UAC completed transactions 5198242/ UAS completed transactions 5512619 calls FS crashed and it looked like this:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa
3 2 0 92972 568 20492 0 0 156 8 117314 130220 7 40 38 15 <-- back in the day i would kill for 92Mb free mem. 10 5 0 3473328 208 9848 0 0 24344 48 104557 89011 5 62 15 17 <- Now the kernel kills FS for taking the rest of it. 8 0 0 3482436 340 14408 0 0 3208 0 2582 2046 0 95 4 1 9 0 0 3504792 348 14344 0 0 0 48 2312 1836 0 93 7 0 12 0 0 3532184 348 14348 0 0 0 0 2387 1984 0 94 6 0 6 0 0 3561792 348 14876 0 0 528 0 2523 2178 0 93 6 0 11 0 0 3592816 348 14876 0 0 0 0 2684 2248 0 95 5 0 3 0 0 3615424 348 14876 0 0 0 0 2820 2494 0 95 5 0 7 0 0 3633488 348 14876 0 0 0 0 2990 2805 0 94 6 0 7 0 0 3654916 348 14876 0 0 0 0 3350 3222 0 96 4 0 11 0 0 3680708 348 14876 0 0 0 0 3869 4018 0 94 6 0 0 0 0 3773532 476 16876 0 0 2344 0 6598 9501 0 74 24 2 0 0 0 3773836 476 16912 0 0 0 0 191 129 0 0 100 0 <- FS is not running any more, 100% idle CPU , lots of free ram 0 0 0 3773856 476 17108 0 0 0 0 91 63 0 0 100 0
dmesg of the crash is this:
[48674.576126] Out of memory: Kill process 31552 (freeswitch) score 674 or sacrifice child [48674.578188] Killed process 31552 (freeswitch) total-vm:10177480kB, anon-rss:2710092kB, file-rss:0kB
fs_log switches to the dark side : (there was no disk space problem of course, i have no idea at which time point in the spiral down this popped on the console )
2013-03-06 23:27:22.923640 [ERR] switch_core_sqldb.c:579 NATIVE SQL ERR [database or disk is full] BEGIN EXCLUSIVE 2013-03-06 23:27:22.923640 [CRIT] switch_core_sqldb.c:1679 ERROR [database or disk is full] 2013-03-06 23:27:22.923640 [ERR] switch_core_sqldb.c:579 NATIVE SQL ERR [cannot commit - no transaction is active] COMMIT 2013-03-06 23:27:22.963609 [CRIT] switch_core_session.c:1674 Thread Failure! 2013-03-06 23:27:22.963609 [CRIT] switch_core_session.c:1634 LUKE: I'm hit, but not bad. 2013-03-06 23:27:22.963609 [CRIT] switch_core_session.c:1635 LUKE'S VOICE: Artoo, see what you can do with it. Hang on back there.... Green laserfire moves past the beeping little robot as his head turns. After a few beeps and a twist of his mechanical arm, Artoo reduces the max sessions to 13897 thus, saving the switch from certain doom.
and thats all , FS is dead.
Runing the test at 180 cps will significantly hasten the death. It starts normal with ~40% idle cpu then after circa 160k calls it starts to spiral: procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 6 0 0 2901484 63204 161132 0 0 0 0 20586 33073 54 5 41 0 5 0 0 2901468 63212 160988 0 0 0 12 25359 41915 53 6 42 0 4 0 0 2901220 63212 161064 0 0 0 0 19650 31211 55 5 40 0 7 0 0 2901080 63212 161124 0 0 0 0 21148 33462 54 5 41 0 2 0 0 2900676 63212 161188 0 0 0 0 21767 35568 54 5 42 0 5 0 0 2900468 63212 161256 0 0 0 0 19398 30073 54 5 41 0 2 0 0 2900088 63220 161304 0 0 0 12 21315 34563 54 5 42 0 3 0 0 2900180 63220 161484 0 0 0 0 19858 31561 42 4 54 0 <-- this should not happen the load is constant and idle cpu% should stay at ~40% 3 0 0 2899768 63220 161520 0 0 0 0 29907 46883 39 4 57 0 11 0 0 2900056 63220 161504 0 0 0 0 41730 63440 41 5 54 0 9 0 0 2899536 63220 161768 0 0 0 0 48876 68159 40 5 54 0 6 0 0 2899512 63228 161620 0 0 0 12 59988 80421 41 6 53 0 49 0 0 2876072 63228 162004 0 0 0 0 193384 241453 57 23 21 0 <-- here we go, starting the memory hogging cpu eating spiral 40 0 0 2830720 63228 162080 0 0 0 0 308948 356178 51 49 0 0 156 0 0 2776576 63228 161984 0 0 0 0 301514 325415 48 48 5 0 178 0 0 2727044 63228 162260 0 0 0 0 290168 306158 44 51 5 0 70 0 0 2683252 63236 161924 0 0 0 48 291792 302632 37 58 5 0 5 0 0 2618432 63236 162204 0 0 0 0 316170 302886 35 59 7 0 2 0 0 2548092 63236 162364 0 0 0 0 340707 319105 38 56 5 0 2 0 0 2460528 63236 162464 0 0 0 1596 386953 348210 40 54 6 0
Nothing on fs warn/err log at this point until memory is all full.
Please give me some ideas on how to get some useful info for a bug report.
Kind Regards, Boris Ratner.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Hm... But what about huge interrupt and context switching number? 2013/3/7 Steven Ayre <steveayre@gmail.com>
Upgrade to the latest version.
There are some memory leaks that have been found and fixed in 1.2.7.
-Steve
On 7 March 2013 11:09, bratner bratner <ratner2@gmail.com> wrote:
Hi!
Please assist me to find a way to get some insight into this problem and file a proper bug report. Looks like 2 different problems, one is load related (like a race of some sort) the other is a mem leak.
I'm running FS debs compiled on ubuntu 12.04 (-n20130227T114536Z-1~presice+ 1git). The machine is ubuntu 12.04 server , CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 4GB RAM and no swap. 3.1G RAM free after FS is started. I disabled HT in BIOS to get proper CPU load readings from top and vmstat (no HT also allows for higher cps for some reason...).
The FS setup has 2 SIP profiles , internal/external (functioning as incoming/outgoing ), XML Dialplan that runs a lua script. Calls come in to profile 'internal', authed by ip acl , go to the dialplan and sent through profile 'external' to a uri string generated by the lua script.
I bombard the machine with high call rate of short calls ,120-180cps , 5sec. Everything stays fine for some time , the higher the load the faster the problems appear. When i pressure the machine with 150cps and up , it works fine at first with 20%-30% idle CPU. After some time passes (10-30minutes) system cpu usage jumps from 3%-7% to around 50%, CPU is 95% busy and vmstat reports hundereds and sometimes thousands of runnable tasks. If i stop sending calls at this moment, wait for the CPU load to wind down and start calling again then the CPU load saturates right away. Even if i lower the CPS rate. restarting freeswitch at this point brings the system to the original state and it will take some time to get saturated again.
Another symptom is that during the test the available free mem is slowly allocated to freeswitch until it reaches 13.1Gb virtual image and then the kernel kills it.
Here are some numbers:
Running 120 calls per second , 5second calls:
Starts like this: This is very good from my POV.
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 5 0 0 3255876 35444 96580 0 0 9 2 66 90 1 1 98 0 4 0 0 3253080 35444 96652 0 0 0 0 16048 20727 36 3 61 0 4 0 0 3247780 35444 96684 0 0 0 0 16521 21048 36 4 60 0 4 0 0 3242860 35444 96784 0 0 0 0 15695 19825 36 3 62 0 1 0 0 3237980 35452 96768 0 0 0 24 16072 19851 36 3 60 0 4 0 0 3233692 35452 96812 0 0 0 0 16299 20980 35 4 61 0 8 0 0 3230280 35452 96928 0 0 0 0 15034 18965 36 3 61 0 2 0 0 3227764 35452 96896 0 0 0 0 13956 20621 37 3 60 0
After some (long) time it started having trouble and eat lots of free ram (note that the idle cpu% and system cpu% jumped up but there are still over 1G free mem available) :
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 22 0 0 1108588 100304 2034628 0 0 0 0 145350 183629 37 53 10 0 5 0 0 1108224 100304 2034668 0 0 0 0 151937 167249 38 57 5 0 11 0 0 1108268 100304 2034904 0 0 0 0 149931 176143 37 58 5 0 7 0 0 1108244 100312 2034752 0 0 0 32 138612 183583 40 47 14 0 3 0 0 1108056 100312 2034792 0 0 0 0 111446 162733 42 29 29 0 4 0 0 1108064 100312 2034836 0 0 0 0 112568 164187 41 30 29 0 14 0 0 1108040 100312 2034880 0 0 0 0 133372 174676 39 45 17 0 10 0 0 1107468 100312 2034920 0 0 0 0 157371 184888 37 58 5 0
Short time after that is got into really serious trouble (3000 runnable tasks!? heading toward memory exhaustion):
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa
3336 0 0 146580 9248 1457016 0 0 0 0 306642 291819 28 67 5 0 1180 0 0 145060 9220 1430212 0 0 248 848 294654 285273 27 68 5 0 1028 0 0 148616 8996 1366844 0 0 0 0 421614 385342 32 63 4 0 1026 0 0 150964 8880 1298832 0 0 0 28 462825 416480 31 65 4 0 1338 0 0 146260 8688 1242440 0 0 0 0 442247 391139 27 69 5 0 1945 0 0 143148 8680 1197180 0 0 0 0 364361 329958 24 71 5 0 1381 0 0 154668 8596 1137912 0 0 0 0 359982 323543 23 72 5 0
After UAC completed transactions 5198242/ UAS completed transactions 5512619 calls FS crashed and it looked like this:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa
3 2 0 92972 568 20492 0 0 156 8 117314 130220 7 40 38 15 <-- back in the day i would kill for 92Mb free mem. 10 5 0 3473328 208 9848 0 0 24344 48 104557 89011 5 62 15 17 <- Now the kernel kills FS for taking the rest of it. 8 0 0 3482436 340 14408 0 0 3208 0 2582 2046 0 95 4 1 9 0 0 3504792 348 14344 0 0 0 48 2312 1836 0 93 7 0 12 0 0 3532184 348 14348 0 0 0 0 2387 1984 0 94 6 0 6 0 0 3561792 348 14876 0 0 528 0 2523 2178 0 93 6 0 11 0 0 3592816 348 14876 0 0 0 0 2684 2248 0 95 5 0 3 0 0 3615424 348 14876 0 0 0 0 2820 2494 0 95 5 0 7 0 0 3633488 348 14876 0 0 0 0 2990 2805 0 94 6 0 7 0 0 3654916 348 14876 0 0 0 0 3350 3222 0 96 4 0 11 0 0 3680708 348 14876 0 0 0 0 3869 4018 0 94 6 0 0 0 0 3773532 476 16876 0 0 2344 0 6598 9501 0 74 24 2 0 0 0 3773836 476 16912 0 0 0 0 191 129 0 0 100 0 <- FS is not running any more, 100% idle CPU , lots of free ram 0 0 0 3773856 476 17108 0 0 0 0 91 63 0 0 100 0
dmesg of the crash is this:
[48674.576126] Out of memory: Kill process 31552 (freeswitch) score 674 or sacrifice child [48674.578188] Killed process 31552 (freeswitch) total-vm:10177480kB, anon-rss:2710092kB, file-rss:0kB
fs_log switches to the dark side : (there was no disk space problem of course, i have no idea at which time point in the spiral down this popped on the console )
2013-03-06 23:27:22.923640 [ERR] switch_core_sqldb.c:579 NATIVE SQL ERR [database or disk is full] BEGIN EXCLUSIVE 2013-03-06 23:27:22.923640 [CRIT] switch_core_sqldb.c:1679 ERROR [database or disk is full] 2013-03-06 23:27:22.923640 [ERR] switch_core_sqldb.c:579 NATIVE SQL ERR [cannot commit - no transaction is active] COMMIT 2013-03-06 23:27:22.963609 [CRIT] switch_core_session.c:1674 Thread Failure! 2013-03-06 23:27:22.963609 [CRIT] switch_core_session.c:1634 LUKE: I'm hit, but not bad. 2013-03-06 23:27:22.963609 [CRIT] switch_core_session.c:1635 LUKE'S VOICE: Artoo, see what you can do with it. Hang on back there.... Green laserfire moves past the beeping little robot as his head turns. After a few beeps and a twist of his mechanical arm, Artoo reduces the max sessions to 13897 thus, saving the switch from certain doom.
and thats all , FS is dead.
Runing the test at 180 cps will significantly hasten the death. It starts normal with ~40% idle cpu then after circa 160k calls it starts to spiral: procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 6 0 0 2901484 63204 161132 0 0 0 0 20586 33073 54 5 41 0 5 0 0 2901468 63212 160988 0 0 0 12 25359 41915 53 6 42 0 4 0 0 2901220 63212 161064 0 0 0 0 19650 31211 55 5 40 0 7 0 0 2901080 63212 161124 0 0 0 0 21148 33462 54 5 41 0 2 0 0 2900676 63212 161188 0 0 0 0 21767 35568 54 5 42 0 5 0 0 2900468 63212 161256 0 0 0 0 19398 30073 54 5 41 0 2 0 0 2900088 63220 161304 0 0 0 12 21315 34563 54 5 42 0 3 0 0 2900180 63220 161484 0 0 0 0 19858 31561 42 4 54 0 <-- this should not happen the load is constant and idle cpu% should stay at ~40% 3 0 0 2899768 63220 161520 0 0 0 0 29907 46883 39 4 57 0 11 0 0 2900056 63220 161504 0 0 0 0 41730 63440 41 5 54 0 9 0 0 2899536 63220 161768 0 0 0 0 48876 68159 40 5 54 0 6 0 0 2899512 63228 161620 0 0 0 12 59988 80421 41 6 53 0 49 0 0 2876072 63228 162004 0 0 0 0 193384 241453 57 23 21 0 <-- here we go, starting the memory hogging cpu eating spiral 40 0 0 2830720 63228 162080 0 0 0 0 308948 356178 51 49 0 0 156 0 0 2776576 63228 161984 0 0 0 0 301514 325415 48 48 5 0 178 0 0 2727044 63228 162260 0 0 0 0 290168 306158 44 51 5 0 70 0 0 2683252 63236 161924 0 0 0 48 291792 302632 37 58 5 0 5 0 0 2618432 63236 162204 0 0 0 0 316170 302886 35 59 7 0 2 0 0 2548092 63236 162364 0 0 0 0 340707 319105 38 56 5 0 2 0 0 2460528 63236 162464 0 0 0 1596 386953 348210 40 54 6 0
Nothing on fs warn/err log at this point until memory is all full.
Please give me some ideas on how to get some useful info for a bug report.
Kind Regards, Boris Ratner.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
You are probably hammering the disk subsystem... Keep in mind that FS uses multiple sqlite databases by default... Mount the fs db dir as tmpfs and try again On 3/7/13 7:35 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com> wrote:
Hm... But what about huge interrupt and context switching number?
2013/3/7 Steven Ayre <steveayre@gmail.com>
Upgrade to the latest version.
There are some memory leaks that have been found and fixed in 1.2.7.
-Steve
On 7 March 2013 11:09, bratner bratner <ratner2@gmail.com> wrote:
Hi!
Please assist me to find a way to get some insight into this problem and file a proper bug report. Looks like 2 different problems, one is load related (like a race of some sort) the other is a mem leak.
I'm running FS debs compiled on ubuntu 12.04 (-n20130227T114536Z-1~presice+ 1git). The machine is ubuntu 12.04 server , CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz, 4GB RAM and no swap. 3.1G RAM free after FS is started. I disabled HT in BIOS to get proper CPU load readings from top and vmstat (no HT also allows for higher cps for some reason...).
The FS setup has 2 SIP profiles , internal/external (functioning as incoming/outgoing ), XML Dialplan that runs a lua script. Calls come in to profile 'internal', authed by ip acl , go to the dialplan and sent through profile 'external' to a uri string generated by the lua script.
I bombard the machine with high call rate of short calls ,120-180cps , 5sec. Everything stays fine for some time , the higher the load the faster the problems appear. When i pressure the machine with 150cps and up , it works fine at first with 20%-30% idle CPU. After some time passes (10-30minutes) system cpu usage jumps from 3%-7% to around 50%, CPU is 95% busy and vmstat reports hundereds and sometimes thousands of runnable tasks. If i stop sending calls at this moment, wait for the CPU load to wind down and start calling again then the CPU load saturates right away. Even if i lower the CPS rate. restarting freeswitch at this point brings the system to the original state and it will take some time to get saturated again.
Another symptom is that during the test the available free mem is slowly allocated to freeswitch until it reaches 13.1Gb virtual image and then the kernel kills it.
Here are some numbers:
Running 120 calls per second , 5second calls:
Starts like this: This is very good from my POV.
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 5 0 0 3255876 35444 96580 0 0 9 2 66 90 1 1 98 0 4 0 0 3253080 35444 96652 0 0 0 0 16048 20727 36 3 61 0 4 0 0 3247780 35444 96684 0 0 0 0 16521 21048 36 4 60 0 4 0 0 3242860 35444 96784 0 0 0 0 15695 19825 36 3 62 0 1 0 0 3237980 35452 96768 0 0 0 24 16072 19851 36 3 60 0 4 0 0 3233692 35452 96812 0 0 0 0 16299 20980 35 4 61 0 8 0 0 3230280 35452 96928 0 0 0 0 15034 18965 36 3 61 0 2 0 0 3227764 35452 96896 0 0 0 0 13956 20621 37 3 60 0
After some (long) time it started having trouble and eat lots of free ram (note that the idle cpu% and system cpu% jumped up but there are still over 1G free mem available) :
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 22 0 0 1108588 100304 2034628 0 0 0 0 145350 183629 37 53 10 0 5 0 0 1108224 100304 2034668 0 0 0 0 151937 167249 38 57 5 0 11 0 0 1108268 100304 2034904 0 0 0 0 149931 176143 37 58 5 0 7 0 0 1108244 100312 2034752 0 0 0 32 138612 183583 40 47 14 0 3 0 0 1108056 100312 2034792 0 0 0 0 111446 162733 42 29 29 0 4 0 0 1108064 100312 2034836 0 0 0 0 112568 164187 41 30 29 0 14 0 0 1108040 100312 2034880 0 0 0 0 133372 174676 39 45 17 0 10 0 0 1107468 100312 2034920 0 0 0 0 157371 184888 37 58 5 0
Short time after that is got into really serious trouble (3000 runnable tasks!? heading toward memory exhaustion):
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa
3336 0 0 146580 9248 1457016 0 0 0 0 306642 291819 28 67 5 0 1180 0 0 145060 9220 1430212 0 0 248 848 294654 285273 27 68 5 0 1028 0 0 148616 8996 1366844 0 0 0 0 421614 385342 32 63 4 0 1026 0 0 150964 8880 1298832 0 0 0 28 462825 416480 31 65 4 0 1338 0 0 146260 8688 1242440 0 0 0 0 442247 391139 27 69 5 0 1945 0 0 143148 8680 1197180 0 0 0 0 364361 329958 24 71 5 0 1381 0 0 154668 8596 1137912 0 0 0 0 359982 323543 23 72 5 0
After UAC completed transactions 5198242/ UAS completed transactions 5512619 calls FS crashed and it looked like this:
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa
3 2 0 92972 568 20492 0 0 156 8 117314 130220 7 40 38 15 <-- back in the day i would kill for 92Mb free mem. 10 5 0 3473328 208 9848 0 0 24344 48 104557 89011 5 62 15 17 <- Now the kernel kills FS for taking the rest of it. 8 0 0 3482436 340 14408 0 0 3208 0 2582 2046 0 95 4 1 9 0 0 3504792 348 14344 0 0 0 48 2312 1836 0 93 7 0 12 0 0 3532184 348 14348 0 0 0 0 2387 1984 0 94 6 0 6 0 0 3561792 348 14876 0 0 528 0 2523 2178 0 93 6 0 11 0 0 3592816 348 14876 0 0 0 0 2684 2248 0 95 5 0 3 0 0 3615424 348 14876 0 0 0 0 2820 2494 0 95 5 0 7 0 0 3633488 348 14876 0 0 0 0 2990 2805 0 94 6 0 7 0 0 3654916 348 14876 0 0 0 0 3350 3222 0 96 4 0 11 0 0 3680708 348 14876 0 0 0 0 3869 4018 0 94 6 0 0 0 0 3773532 476 16876 0 0 2344 0 6598 9501 0 74 24 2 0 0 0 3773836 476 16912 0 0 0 0 191 129 0 0 100 0 <- FS is not running any more, 100% idle CPU , lots of free ram 0 0 0 3773856 476 17108 0 0 0 0 91 63 0 0 100 0 <tel:91%C2%A0%C2%A0%2063%C2%A0%200%C2%A0%200%20100%C2%A0%200>
dmesg of the crash is this:
[48674.576126] Out of memory: Kill process 31552 (freeswitch) score 674 or sacrifice child [48674.578188] Killed process 31552 (freeswitch) total-vm:10177480kB, anon-rss:2710092kB, file-rss:0kB
fs_log switches to the dark side : (there was no disk space problem of course, i have no idea at which time point in the spiral down this popped on the console )
2013-03-06 23:27:22.923640 [ERR] switch_core_sqldb.c:579 NATIVE SQL ERR [database or disk is full] BEGIN EXCLUSIVE 2013-03-06 23:27:22.923640 [CRIT] switch_core_sqldb.c:1679 ERROR [database or disk is full] 2013-03-06 23:27:22.923640 [ERR] switch_core_sqldb.c:579 NATIVE SQL ERR [cannot commit - no transaction is active] COMMIT 2013-03-06 23:27:22.963609 [CRIT] switch_core_session.c:1674 Thread Failure! 2013-03-06 23:27:22.963609 [CRIT] switch_core_session.c:1634 LUKE: I'm hit, but not bad. 2013-03-06 23:27:22.963609 [CRIT] switch_core_session.c:1635 LUKE'S VOICE: Artoo, see what you can do with it. Hang on back there.... Green laserfire moves past the beeping little robot as his head turns. After a few beeps and a twist of his mechanical arm, Artoo reduces the max sessions to 13897 thus, saving the switch from certain doom.
and thats all , FS is dead.
Runing the test at 180 cps will significantly hasten the death. It starts normal with ~40% idle cpu then after circa 160k calls it starts to spiral: procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 6 0 0 2901484 63204 161132 0 0 0 0 20586 33073 54 5 41 0 5 0 0 2901468 63212 160988 0 0 0 12 25359 41915 53 6 42 0 4 0 0 2901220 63212 161064 0 0 0 0 19650 31211 55 5 40 0 7 0 0 2901080 63212 161124 0 0 0 0 21148 33462 54 5 41 0 2 0 0 2900676 63212 161188 0 0 0 0 21767 35568 54 5 42 0 5 0 0 2900468 63212 161256 0 0 0 0 19398 30073 54 5 41 0 2 0 0 2900088 63220 161304 0 0 0 12 21315 34563 54 5 42 0 3 0 0 2900180 63220 161484 0 0 0 0 19858 31561 42 4 54 0 <-- this should not happen the load is constant and idle cpu% should stay at ~40% 3 0 0 2899768 63220 161520 0 0 0 0 29907 46883 39 4 57 0 11 0 0 2900056 63220 161504 0 0 0 0 41730 63440 41 5 54 0 9 0 0 2899536 63220 161768 0 0 0 0 48876 68159 40 5 54 0 6 0 0 2899512 63228 161620 0 0 0 12 59988 80421 41 6 53 0 49 0 0 2876072 63228 162004 0 0 0 0 193384 241453 57 23 21 0 <-- here we go, starting the memory hogging cpu eating spiral 40 0 0 2830720 63228 162080 0 0 0 0 308948 356178 51 49 0 0 156 0 0 2776576 63228 161984 0 0 0 0 301514 325415 48 48 5 0 178 0 0 2727044 63228 162260 0 0 0 0 290168 306158 44 51 5 0 70 0 0 2683252 63236 161924 0 0 0 48 291792 302632 37 58 5 0 5 0 0 2618432 63236 162204 0 0 0 0 316170 302886 35 59 7 0 2 0 0 2548092 63236 162364 0 0 0 0 340707 319105 38 56 5 0 2 0 0 2460528 63236 162464 0 0 0 1596 386953 348210 40 54 6 0
Nothing on fs warn/err log at this point until memory is all full.
Please give me some ideas on how to get some useful info for a bug report.
Kind Regards, Boris Ratner.
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org irc.freenode.net #freeswitch
bi, bo and wa field is low, so it seems that is not disk subsystem. 2013/3/8 Ken Rice <krice@freeswitch.org>
You are probably hammering the disk subsystem... Keep in mind that FS uses multiple sqlite databases by default... Mount the fs db dir as tmpfs and try again
On 3/7/13 7:35 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com> wrote:
Hm... But what about huge interrupt and context switching number?
Sqlite is probably getting hammered... Trust me... Mount the fs db dir as tmpfs or use the nosql flag when starting freeswitch I routinely run dialer traffic at much higher CPS then that On 3/7/13 7:58 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com> wrote:
bi, bo and wa field is low, so it seems that is not disk subsystem.
2013/3/8 Ken Rice <krice@freeswitch.org>
You are probably hammering the disk subsystem... Keep in mind that FS uses multiple sqlite databases by default... Mount the fs db dir as tmpfs and try again
On 3/7/13 7:35 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com <http://dvl36.ripe.nick@gmail.com> > wrote:
Hm... But what about huge interrupt and context switching number?
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org irc.freenode.net #freeswitch
I can't reproduce such cps load on my ARMv5TE system. ) bratner, please give us 'top -H'. I guess freeswitch running at realtime priority. 2013/3/8 Ken Rice <krice@freeswitch.org>
Sqlite is probably getting hammered... Trust me... Mount the fs db dir as tmpfs or use the –nosql flag when starting freeswitch
I routinely run dialer traffic at much higher CPS then that
On 3/7/13 7:58 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com> wrote:
bi, bo and wa field is low, so it seems that is not disk subsystem.
2013/3/8 Ken Rice <krice@freeswitch.org>
You are probably hammering the disk subsystem... Keep in mind that FS uses multiple sqlite databases by default... Mount the fs db dir as tmpfs and try again
On 3/7/13 7:35 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com < http://dvl36.ripe.nick@gmail.com> > wrote:
Hm... But what about huge interrupt and context switching number?
------------------------------ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken *http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org *irc.freenode.net #freeswitch
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
The original test was done on git master at the date mentioned. The sqlite core.db file was on /run/shm which is a tmpfs on unbuntu 12.04. I will be recompiling from git master and test running with -nosql. Testing my existing setup with -nosql seems more stable now running at 210CPS for some time (500k calls already passed) with ~35% idle cpu. But the free mem is slowly going down. I will let it run untill the kernel will kill it to see how many calls it can handle. During my tests i did not run FS with RT priority but according to htop some of the threads are scheduled as RT. My setup is doing bypass-media , thus FS handling only call establishment and teardown on both legs. cat /proc/<FS pid>/status Name: freeswitch State: S (sleeping) Tgid: 15995 Pid: 15995 PPid: 1 TracerPid: 0 Uid: 999 999 999 999 Gid: 999 999 999 999 FDSize: 64 Groups: VmPeak: 5002808 kB VmSize: 5002088 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 625900 kB VmRSS: 624156 kB <-- this is going up VmData: 4855788 kB VmStk: 136 kB VmExe: 20 kB VmLib: 18288 kB VmPTE: 2352 kB VmSwap: 0 kB Threads: 1866 SigQ: 0/18446744073709551615 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000010003006 SigCgt: 0000000180014209 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: ffffffffffffffff Cpus_allowed: ffffff Cpus_allowed_list: 0-23 Mems_allowed: 00000000,00000003 Mems_allowed_list: 0-1 voluntary_ctxt_switches: 1803 nonvoluntary_ctxt_switches: 23 output of 'top -H' at 180CPS top - 15:27:00 up 2 days, 5:32, 5 users, load average: 8.19, 91.07, 65.03 Tasks: 2066 total, 3 running, 2063 sleeping, 0 stopped, 0 zombie Cpu(s): 50.1%us, 3.9%sy, 0.0%ni, 45.9%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 4038512k total, 2282260k used, 1756252k free, 114112k buffers Swap: 0k total, 0k used, 0k free, 1165868k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16000 freeswit RT -10 4885m 594m 4964 R 69 15.1 3:10.26 freeswitch 16009 freeswit RT -10 4885m 594m 4964 S 33 15.1 1:26.20 freeswitch 16008 freeswit RT -10 4885m 594m 4964 S 28 15.1 1:17.30 freeswitch 16007 freeswit RT -10 4885m 594m 4964 S 4 15.1 0:10.80 freeswitch 16004 freeswit RT -10 4885m 594m 4964 S 2 15.1 0:06.63 freeswitch 19171 root 20 0 18988 2948 944 R 2 0.1 0:00.64 top 18735 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.29 freeswitch 16003 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:01.61 freeswitch 16690 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch 16730 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch 16750 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.45 freeswitch 16764 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.44 freeswitch <more of the above> .... .... Thanks to all of you , Boris Ratner. On Fri, Mar 8, 2013 at 4:22 AM, Dmitry Lysenko <dvl36.ripe.nick@gmail.com>wrote:
I can't reproduce such cps load on my ARMv5TE system. ) bratner, please give us 'top -H'. I guess freeswitch running at realtime priority.
2013/3/8 Ken Rice <krice@freeswitch.org>
Sqlite is probably getting hammered... Trust me... Mount the fs db dir as tmpfs or use the –nosql flag when starting freeswitch
I routinely run dialer traffic at much higher CPS then that
On 3/7/13 7:58 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com> wrote:
bi, bo and wa field is low, so it seems that is not disk subsystem.
2013/3/8 Ken Rice <krice@freeswitch.org>
You are probably hammering the disk subsystem... Keep in mind that FS uses multiple sqlite databases by default... Mount the fs db dir as tmpfs and try again
On 3/7/13 7:35 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com < http://dvl36.ripe.nick@gmail.com> > wrote:
Hm... But what about huge interrupt and context switching number?
------------------------------ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken *http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org *irc.freenode.net #freeswitch
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Here is sipp output and additional numbers for a test I ran with -nosql param. The test ran 180CPS for ~3500seconds and the rest with 210cps. Trouble (as in higher system cpu% ) started to appear around 8591seconds into the test. As you can see below the problem started just before 9124sec into the test 210cps 5sec calls should not give you a lot more then 1050 concurrent calls. ------------------------------ Scenario Screen -------- [1-9]: Change Screen -- Call-rate(length) Port Total-time Total-calls Remote-host 210.0(5000 ms)/1.000s 5061 9157.32 s 1834024 192.96.201.164:5060 (UDP) 0 new calls during 0.000 s period 0 ms scheduler resolution 0 calls (limit 2000) Peak was 2000 calls, after 9124 s 0 Running, 4640 Paused, 0 Woken up 20 dead call msg (discarded) 0 out-of-call msg (discarded) 1 open sockets Messages Retrans Timeout Unexpected-Msg INVITE ----------> 1834024 74 0 100 <---------- 1834024 0 0 0 180 <---------- 1834024 0 0 0 183 <---------- 0 0 0 0 500 <---------- 0 0 0 0 502 <---------- 0 0 0 0 503 <---------- 0 0 0 0 408 <---------- 0 0 0 0 480 <---------- 0 0 0 0 200 <---------- E-RTD1 1834024 81 0 0 ACK ----------> 1834024 81 Pause [ 5000ms] 1834024 0 BYE ----------> 1834024 7646 0 503 <---------- 0 0 0 0 200 <---------- 1834024 0 0 0 ------------------------------ Test Terminated -------------------------------- ----------------------------- Statistics Screen ------- [1-9]: Change Screen -- Start Time | 2013-03-08 15:22:18:204 1362756138.204833 Last Reset Time | 2013-03-08 17:54:55:535 1362765295.535214 Current Time | 2013-03-08 17:54:55:535 1362765295.535437 -------------------------+---------------------------+-------------------------- Counter Name | Periodic value | Cumulative value -------------------------+---------------------------+-------------------------- Elapsed Time | 00:00:00:000 | 02:32:37:330 Call Rate | 0.000 cps | 200.279 cps -------------------------+---------------------------+-------------------------- Incoming call created | 0 | 0 OutGoing call created | 0 | 1834024 Total Call created | | 1834024 Current Call | 0 | -------------------------+---------------------------+-------------------------- Successful call | 0 | 1834024 Failed call | 0 | 0 -------------------------+---------------------------+-------------------------- Response Time 1 | 00:00:00:000 | 00:00:00:149 Call Length | 00:00:00:000 | 00:00:05:158 ------------------------------ Test Terminated -------------------------------- After stopping the load FS still hogs 22.1% of memory. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 freeswitch The symptoms of the crash are the same, just now with higher CPS and takes more time (more calls ) before crashing. I will appreciate any suggestion. Regards, Boris Ratner. On Fri, Mar 8, 2013 at 6:22 PM, bratner bratner <ratner2@gmail.com> wrote:
The original test was done on git master at the date mentioned. The sqlite core.db file was on /run/shm which is a tmpfs on unbuntu 12.04. I will be recompiling from git master and test running with -nosql.
Testing my existing setup with -nosql seems more stable now running at 210CPS for some time (500k calls already passed) with ~35% idle cpu. But the free mem is slowly going down. I will let it run untill the kernel will kill it to see how many calls it can handle.
During my tests i did not run FS with RT priority but according to htop some of the threads are scheduled as RT. My setup is doing bypass-media , thus FS handling only call establishment and teardown on both legs.
cat /proc/<FS pid>/status
Name: freeswitch State: S (sleeping) Tgid: 15995 Pid: 15995 PPid: 1 TracerPid: 0 Uid: 999 999 999 999 Gid: 999 999 999 999 FDSize: 64 Groups: VmPeak: 5002808 kB VmSize: 5002088 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 625900 kB VmRSS: 624156 kB <-- this is going up VmData: 4855788 kB VmStk: 136 kB VmExe: 20 kB VmLib: 18288 kB VmPTE: 2352 kB VmSwap: 0 kB Threads: 1866 SigQ: 0/18446744073709551615 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000010003006 SigCgt: 0000000180014209 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: ffffffffffffffff Cpus_allowed: ffffff Cpus_allowed_list: 0-23 Mems_allowed: 00000000,00000003 Mems_allowed_list: 0-1 voluntary_ctxt_switches: 1803 nonvoluntary_ctxt_switches: 23
output of 'top -H' at 180CPS
top - 15:27:00 up 2 days, 5:32, 5 users, load average: 8.19, 91.07, 65.03 Tasks: 2066 total, 3 running, 2063 sleeping, 0 stopped, 0 zombie Cpu(s): 50.1%us, 3.9%sy, 0.0%ni, 45.9%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 4038512k total, 2282260k used, 1756252k free, 114112k buffers Swap: 0k total, 0k used, 0k free, 1165868k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16000 freeswit RT -10 4885m 594m 4964 R 69 15.1 3:10.26 freeswitch
16009 freeswit RT -10 4885m 594m 4964 S 33 15.1 1:26.20 freeswitch
16008 freeswit RT -10 4885m 594m 4964 S 28 15.1 1:17.30 freeswitch
16007 freeswit RT -10 4885m 594m 4964 S 4 15.1 0:10.80 freeswitch
16004 freeswit RT -10 4885m 594m 4964 S 2 15.1 0:06.63 freeswitch
19171 root 20 0 18988 2948 944 R 2 0.1 0:00.64 top
18735 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.29 freeswitch
16003 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:01.61 freeswitch
16690 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
16730 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
16750 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.45 freeswitch
16764 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.44 freeswitch
<more of the above> .... ....
Thanks to all of you , Boris Ratner.
On Fri, Mar 8, 2013 at 4:22 AM, Dmitry Lysenko <dvl36.ripe.nick@gmail.com>wrote:
I can't reproduce such cps load on my ARMv5TE system. ) bratner, please give us 'top -H'. I guess freeswitch running at realtime priority.
2013/3/8 Ken Rice <krice@freeswitch.org>
Sqlite is probably getting hammered... Trust me... Mount the fs db dir as tmpfs or use the –nosql flag when starting freeswitch
I routinely run dialer traffic at much higher CPS then that
On 3/7/13 7:58 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com> wrote:
bi, bo and wa field is low, so it seems that is not disk subsystem.
2013/3/8 Ken Rice <krice@freeswitch.org>
You are probably hammering the disk subsystem... Keep in mind that FS uses multiple sqlite databases by default... Mount the fs db dir as tmpfs and try again
On 3/7/13 7:35 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com < http://dvl36.ripe.nick@gmail.com> > wrote:
Hm... But what about huge interrupt and context switching number?
------------------------------ _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken *http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org *irc.freenode.net #freeswitch
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
FS runs itself as RT priority for the threads that are critical to its operation. We do not typically entertain load testing questions here because its subjective to the environment and requires a lot of knowledge and computer tuning skills and we have lost a lot of time over the years addressing this kind of topic. Typically when you find the limits your machine can handle, its best to set the params designed to protect the process from getting overloaded. like the min-cpu-idle and max-sessions etc. The call flow you are using and 100 other variables factor into what you can get as a max. We do not endorse or quote performance numbers. One hint, try a bigger ptime such as 40ms during testing to reduce the load on the scheduler. Another is get a nice 12 core box if you want insane call volume. The more cpu, the more concurrent context switches you can endure. What you are experiencing is just the tip of the iceberg on the realm of performance of user-space low latency media. There is a wealth of information collected on this topic and you will see a lot of the challenges as you move forward. On Fri, Mar 8, 2013 at 12:15 PM, bratner bratner <ratner2@gmail.com> wrote:
Here is sipp output and additional numbers for a test I ran with -nosql param.
The test ran 180CPS for ~3500seconds and the rest with 210cps.
Trouble (as in higher system cpu% ) started to appear around 8591seconds into the test. As you can see below the problem started just before 9124sec into the test 210cps 5sec calls should not give you a lot more then 1050 concurrent calls.
------------------------------ Scenario Screen -------- [1-9]: Change Screen -- Call-rate(length) Port Total-time Total-calls Remote-host 210.0(5000 ms)/1.000s 5061 9157.32 s 1834024 192.96.201.164 :5060(UDP)
0 new calls during 0.000 s period 0 ms scheduler resolution 0 calls (limit 2000) Peak was 2000 calls, after 9124 s 0 Running, 4640 Paused, 0 Woken up 20 dead call msg (discarded) 0 out-of-call msg (discarded) 1 open sockets
Messages Retrans Timeout Unexpected-Msg INVITE ----------> 1834024 74 0 100 <---------- 1834024 0 0 0 180 <---------- 1834024 0 0 0 183 <---------- 0 0 0 0 500 <---------- 0 0 0 0 502 <---------- 0 0 0 0 503 <---------- 0 0 0 0 408 <---------- 0 0 0 0 480 <---------- 0 0 0 0 200 <---------- E-RTD1 1834024 81 0 0
ACK ----------> 1834024 81 Pause [ 5000ms] 1834024 0 BYE ----------> 1834024 7646 0 503 <---------- 0 0 0 0 200 <---------- 1834024 0 0 0
------------------------------ Test Terminated --------------------------------
----------------------------- Statistics Screen ------- [1-9]: Change Screen -- Start Time | 2013-03-08 15:22:18:204 1362756138.204833 Last Reset Time | 2013-03-08 17:54:55:535 1362765295.535214 Current Time | 2013-03-08 17:54:55:535 1362765295.535437
-------------------------+---------------------------+-------------------------- Counter Name | Periodic value | Cumulative value
-------------------------+---------------------------+-------------------------- Elapsed Time | 00:00:00:000 | 02:32:37:330 Call Rate | 0.000 cps | 200.279 cps
-------------------------+---------------------------+-------------------------- Incoming call created | 0 | 0 OutGoing call created | 0 | 1834024 Total Call created | | 1834024 Current Call | 0 |
-------------------------+---------------------------+-------------------------- Successful call | 0 | 1834024 Failed call | 0 | 0
-------------------------+---------------------------+-------------------------- Response Time 1 | 00:00:00:000 | 00:00:00:149 Call Length | 00:00:00:000 | 00:00:05:158 ------------------------------ Test Terminated --------------------------------
After stopping the load FS still hogs 22.1% of memory. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 freeswitch
The symptoms of the crash are the same, just now with higher CPS and takes more time (more calls ) before crashing.
I will appreciate any suggestion.
Regards, Boris Ratner.
On Fri, Mar 8, 2013 at 6:22 PM, bratner bratner <ratner2@gmail.com> wrote:
The original test was done on git master at the date mentioned. The sqlite core.db file was on /run/shm which is a tmpfs on unbuntu 12.04. I will be recompiling from git master and test running with -nosql.
Testing my existing setup with -nosql seems more stable now running at 210CPS for some time (500k calls already passed) with ~35% idle cpu. But the free mem is slowly going down. I will let it run untill the kernel will kill it to see how many calls it can handle.
During my tests i did not run FS with RT priority but according to htop some of the threads are scheduled as RT. My setup is doing bypass-media , thus FS handling only call establishment and teardown on both legs.
cat /proc/<FS pid>/status
Name: freeswitch State: S (sleeping) Tgid: 15995 Pid: 15995 PPid: 1 TracerPid: 0 Uid: 999 999 999 999 Gid: 999 999 999 999 FDSize: 64 Groups: VmPeak: 5002808 kB VmSize: 5002088 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 625900 kB VmRSS: 624156 kB <-- this is going up VmData: 4855788 kB VmStk: 136 kB VmExe: 20 kB VmLib: 18288 kB VmPTE: 2352 kB VmSwap: 0 kB Threads: 1866 SigQ: 0/18446744073709551615 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000010003006 SigCgt: 0000000180014209 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: ffffffffffffffff Cpus_allowed: ffffff Cpus_allowed_list: 0-23 Mems_allowed: 00000000,00000003 Mems_allowed_list: 0-1 voluntary_ctxt_switches: 1803 nonvoluntary_ctxt_switches: 23
output of 'top -H' at 180CPS
top - 15:27:00 up 2 days, 5:32, 5 users, load average: 8.19, 91.07, 65.03 Tasks: 2066 total, 3 running, 2063 sleeping, 0 stopped, 0 zombie Cpu(s): 50.1%us, 3.9%sy, 0.0%ni, 45.9%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 4038512k total, 2282260k used, 1756252k free, 114112k buffers Swap: 0k total, 0k used, 0k free, 1165868k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16000 freeswit RT -10 4885m 594m 4964 R 69 15.1 3:10.26 freeswitch
16009 freeswit RT -10 4885m 594m 4964 S 33 15.1 1:26.20 freeswitch
16008 freeswit RT -10 4885m 594m 4964 S 28 15.1 1:17.30 freeswitch
16007 freeswit RT -10 4885m 594m 4964 S 4 15.1 0:10.80 freeswitch
16004 freeswit RT -10 4885m 594m 4964 S 2 15.1 0:06.63 freeswitch
19171 root 20 0 18988 2948 944 R 2 0.1 0:00.64 top
18735 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.29 freeswitch
16003 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:01.61 freeswitch
16690 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
16730 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
16750 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.45 freeswitch
16764 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.44 freeswitch
<more of the above> .... ....
Thanks to all of you , Boris Ratner.
On Fri, Mar 8, 2013 at 4:22 AM, Dmitry Lysenko <dvl36.ripe.nick@gmail.com
wrote:
I can't reproduce such cps load on my ARMv5TE system. ) bratner, please give us 'top -H'. I guess freeswitch running at realtime priority.
2013/3/8 Ken Rice <krice@freeswitch.org>
Sqlite is probably getting hammered... Trust me... Mount the fs db dir as tmpfs or use the –nosql flag when starting freeswitch
I routinely run dialer traffic at much higher CPS then that
On 3/7/13 7:58 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com> wrote:
bi, bo and wa field is low, so it seems that is not disk subsystem.
2013/3/8 Ken Rice <krice@freeswitch.org>
You are probably hammering the disk subsystem... Keep in mind that FS uses multiple sqlite databases by default... Mount the fs db dir as tmpfs and try again
On 3/7/13 7:35 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com < http://dvl36.ripe.nick@gmail.com> > wrote:
Hm... But what about huge interrupt and context switching number?
------------------------------
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken *http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org *irc.freenode.net #freeswitch
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Please try to reproduce with -np switch. Thanks. 2013/3/8 bratner bratner <ratner2@gmail.com>
Here is sipp output and additional numbers for a test I ran with -nosql param.
The test ran 180CPS for ~3500seconds and the rest with 210cps.
Trouble (as in higher system cpu% ) started to appear around 8591seconds into the test. As you can see below the problem started just before 9124sec into the test 210cps 5sec calls should not give you a lot more then 1050 concurrent calls.
------------------------------ Scenario Screen -------- [1-9]: Change Screen -- Call-rate(length) Port Total-time Total-calls Remote-host 210.0(5000 ms)/1.000s 5061 9157.32 s 1834024 192.96.201.164:5060 (UDP)
0 new calls during 0.000 s period 0 ms scheduler resolution 0 calls (limit 2000) Peak was 2000 calls, after 9124 s 0 Running, 4640 Paused, 0 Woken up 20 dead call msg (discarded) 0 out-of-call msg (discarded) 1 open sockets
Messages Retrans Timeout Unexpected-Msg INVITE ----------> 1834024 74 0 100 <---------- 1834024 0 0 0 180 <---------- 1834024 0 0 0 183 <---------- 0 0 0 0 500 <---------- 0 0 0 0 502 <---------- 0 0 0 0 503 <---------- 0 0 0 0 408 <---------- 0 0 0 0 480 <---------- 0 0 0 0 200 <---------- E-RTD1 1834024 81 0 0
ACK ----------> 1834024 81 Pause [ 5000ms] 1834024 0 BYE ----------> 1834024 7646 0 503 <---------- 0 0 0 0 200 <---------- 1834024 0 0 0
------------------------------ Test Terminated --------------------------------
----------------------------- Statistics Screen ------- [1-9]: Change Screen -- Start Time | 2013-03-08 15:22:18:204 1362756138.204833 Last Reset Time | 2013-03-08 17:54:55:535 1362765295.535214 Current Time | 2013-03-08 17:54:55:535 1362765295.535437
-------------------------+---------------------------+-------------------------- Counter Name | Periodic value | Cumulative value
-------------------------+---------------------------+-------------------------- Elapsed Time | 00:00:00:000 | 02:32:37:330 Call Rate | 0.000 cps | 200.279 cps
-------------------------+---------------------------+-------------------------- Incoming call created | 0 | 0 OutGoing call created | 0 | 1834024 Total Call created | | 1834024 Current Call | 0 |
-------------------------+---------------------------+-------------------------- Successful call | 0 | 1834024 Failed call | 0 | 0
-------------------------+---------------------------+-------------------------- Response Time 1 | 00:00:00:000 | 00:00:00:149 Call Length | 00:00:00:000 | 00:00:05:158 ------------------------------ Test Terminated --------------------------------
After stopping the load FS still hogs 22.1% of memory. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 freeswitch
The symptoms of the crash are the same, just now with higher CPS and takes more time (more calls ) before crashing.
I will appreciate any suggestion.
Regards, Boris Ratner.
On Fri, Mar 8, 2013 at 6:22 PM, bratner bratner <ratner2@gmail.com> wrote:
The original test was done on git master at the date mentioned. The sqlite core.db file was on /run/shm which is a tmpfs on unbuntu 12.04. I will be recompiling from git master and test running with -nosql.
Testing my existing setup with -nosql seems more stable now running at 210CPS for some time (500k calls already passed) with ~35% idle cpu. But the free mem is slowly going down. I will let it run untill the kernel will kill it to see how many calls it can handle.
During my tests i did not run FS with RT priority but according to htop some of the threads are scheduled as RT. My setup is doing bypass-media , thus FS handling only call establishment and teardown on both legs.
cat /proc/<FS pid>/status
Name: freeswitch State: S (sleeping) Tgid: 15995 Pid: 15995 PPid: 1 TracerPid: 0 Uid: 999 999 999 999 Gid: 999 999 999 999 FDSize: 64 Groups: VmPeak: 5002808 kB VmSize: 5002088 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 625900 kB VmRSS: 624156 kB <-- this is going up VmData: 4855788 kB VmStk: 136 kB VmExe: 20 kB VmLib: 18288 kB VmPTE: 2352 kB VmSwap: 0 kB Threads: 1866 SigQ: 0/18446744073709551615 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000010003006 SigCgt: 0000000180014209 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: ffffffffffffffff Cpus_allowed: ffffff Cpus_allowed_list: 0-23 Mems_allowed: 00000000,00000003 Mems_allowed_list: 0-1 voluntary_ctxt_switches: 1803 nonvoluntary_ctxt_switches: 23
output of 'top -H' at 180CPS
top - 15:27:00 up 2 days, 5:32, 5 users, load average: 8.19, 91.07, 65.03 Tasks: 2066 total, 3 running, 2063 sleeping, 0 stopped, 0 zombie Cpu(s): 50.1%us, 3.9%sy, 0.0%ni, 45.9%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 4038512k total, 2282260k used, 1756252k free, 114112k buffers Swap: 0k total, 0k used, 0k free, 1165868k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16000 freeswit RT -10 4885m 594m 4964 R 69 15.1 3:10.26 freeswitch
16009 freeswit RT -10 4885m 594m 4964 S 33 15.1 1:26.20 freeswitch
16008 freeswit RT -10 4885m 594m 4964 S 28 15.1 1:17.30 freeswitch
16007 freeswit RT -10 4885m 594m 4964 S 4 15.1 0:10.80 freeswitch
16004 freeswit RT -10 4885m 594m 4964 S 2 15.1 0:06.63 freeswitch
19171 root 20 0 18988 2948 944 R 2 0.1 0:00.64 top
18735 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.29 freeswitch
16003 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:01.61 freeswitch
16690 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
16730 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
16750 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.45 freeswitch
16764 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.44 freeswitch
<more of the above> .... ....
Thanks to all of you , Boris Ratner.
On Fri, Mar 8, 2013 at 4:22 AM, Dmitry Lysenko <dvl36.ripe.nick@gmail.com
wrote:
I can't reproduce such cps load on my ARMv5TE system. ) bratner, please give us 'top -H'. I guess freeswitch running at realtime priority.
2013/3/8 Ken Rice <krice@freeswitch.org>
Sqlite is probably getting hammered... Trust me... Mount the fs db dir as tmpfs or use the –nosql flag when starting freeswitch
I routinely run dialer traffic at much higher CPS then that
On 3/7/13 7:58 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com> wrote:
bi, bo and wa field is low, so it seems that is not disk subsystem.
2013/3/8 Ken Rice <krice@freeswitch.org>
You are probably hammering the disk subsystem... Keep in mind that FS uses multiple sqlite databases by default... Mount the fs db dir as tmpfs and try again
On 3/7/13 7:35 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com < http://dvl36.ripe.nick@gmail.com> > wrote:
Hm... But what about huge interrupt and context switching number?
------------------------------
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken *http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org *irc.freenode.net #freeswitch
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
After stopping the load FS still hogs 22.1% of memory. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 freeswitch
Until you test with the version you're building from master I would ignore the memory usage since you're running a version with known memory leaks. -Steve On 8 March 2013 18:15, bratner bratner <ratner2@gmail.com> wrote:
Here is sipp output and additional numbers for a test I ran with -nosql param.
The test ran 180CPS for ~3500seconds and the rest with 210cps.
Trouble (as in higher system cpu% ) started to appear around 8591seconds into the test. As you can see below the problem started just before 9124sec into the test 210cps 5sec calls should not give you a lot more then 1050 concurrent calls.
------------------------------ Scenario Screen -------- [1-9]: Change Screen -- Call-rate(length) Port Total-time Total-calls Remote-host 210.0(5000 ms)/1.000s 5061 9157.32 s 1834024 192.96.201.164:5060(UDP)
0 new calls during 0.000 s period 0 ms scheduler resolution 0 calls (limit 2000) Peak was 2000 calls, after 9124 s 0 Running, 4640 Paused, 0 Woken up 20 dead call msg (discarded) 0 out-of-call msg (discarded)
1 open sockets
Messages Retrans Timeout Unexpected-Msg INVITE ----------> 1834024 74 0 100 <---------- 1834024 0 0 0 180 <---------- 1834024 0 0 0 183 <---------- 0 0 0 0 500 <---------- 0 0 0 0 502 <---------- 0 0 0 0 503 <---------- 0 0 0 0 408 <---------- 0 0 0 0 480 <---------- 0 0 0 0 200 <---------- E-RTD1 1834024 81 0 0
ACK ----------> 1834024 81 Pause [ 5000ms] 1834024 0 BYE ----------> 1834024 7646 0 503 <---------- 0 0 0 0 200 <---------- 1834024 0 0 0
------------------------------ Test Terminated --------------------------------
----------------------------- Statistics Screen ------- [1-9]: Change Screen -- Start Time | 2013-03-08 15:22:18:204 1362756138.204833 Last Reset Time | 2013-03-08 17:54:55:535 1362765295.535214 Current Time | 2013-03-08 17:54:55:535 1362765295.535437
-------------------------+---------------------------+--------------------------
Counter Name | Periodic value | Cumulative value
-------------------------+---------------------------+--------------------------
Elapsed Time | 00:00:00:000 | 02:32:37:330
Call Rate | 0.000 cps | 200.279 cps
-------------------------+---------------------------+--------------------------
Incoming call created | 0 | 0
OutGoing call created | 0 | 1834024
Total Call created | | 1834024
Current Call | 0 |
-------------------------+---------------------------+--------------------------
Successful call | 0 | 1834024
Failed call | 0 | 0
-------------------------+---------------------------+--------------------------
Response Time 1 | 00:00:00:000 | 00:00:00:149
Call Length | 00:00:00:000 | 00:00:05:158
------------------------------ Test Terminated --------------------------------
After stopping the load FS still hogs 22.1% of memory. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 freeswitch
The symptoms of the crash are the same, just now with higher CPS and takes more time (more calls ) before crashing.
I will appreciate any suggestion.
Regards, Boris Ratner.
On Fri, Mar 8, 2013 at 6:22 PM, bratner bratner <ratner2@gmail.com> wrote:
The original test was done on git master at the date mentioned. The
sqlite
core.db file was on /run/shm which is a tmpfs on unbuntu 12.04. I will be recompiling from git master and test running with -nosql.
Testing my existing setup with -nosql seems more stable now running at 210CPS for some time (500k calls already passed) with ~35% idle cpu. But the free mem is slowly going down. I will let it run untill the kernel will kill it to see how many calls it can handle.
During my tests i did not run FS with RT priority but according to htop some of the threads are scheduled as RT. My setup is doing bypass-media , thus FS handling only call establishment and teardown on both legs.
cat /proc/<FS pid>/status
Name: freeswitch State: S (sleeping) Tgid: 15995 Pid: 15995 PPid: 1 TracerPid: 0 Uid: 999 999 999 999 Gid: 999 999 999 999 FDSize: 64 Groups: VmPeak: 5002808 kB VmSize: 5002088 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 625900 kB VmRSS: 624156 kB <-- this is going up VmData: 4855788 kB VmStk: 136 kB VmExe: 20 kB VmLib: 18288 kB VmPTE: 2352 kB VmSwap: 0 kB Threads: 1866 SigQ: 0/18446744073709551615 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000010003006 SigCgt: 0000000180014209 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: ffffffffffffffff Cpus_allowed: ffffff Cpus_allowed_list: 0-23 Mems_allowed: 00000000,00000003 Mems_allowed_list: 0-1 voluntary_ctxt_switches: 1803 nonvoluntary_ctxt_switches: 23
output of 'top -H' at 180CPS
top - 15:27:00 up 2 days, 5:32, 5 users, load average: 8.19, 91.07, 65.03 Tasks: 2066 total, 3 running, 2063 sleeping, 0 stopped, 0 zombie Cpu(s): 50.1%us, 3.9%sy, 0.0%ni, 45.9%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 4038512k total, 2282260k used, 1756252k free, 114112k buffers Swap: 0k total, 0k used, 0k free, 1165868k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16000 freeswit RT -10 4885m 594m 4964 R 69 15.1 3:10.26 freeswitch
16009 freeswit RT -10 4885m 594m 4964 S 33 15.1 1:26.20 freeswitch
16008 freeswit RT -10 4885m 594m 4964 S 28 15.1 1:17.30 freeswitch
16007 freeswit RT -10 4885m 594m 4964 S 4 15.1 0:10.80 freeswitch
16004 freeswit RT -10 4885m 594m 4964 S 2 15.1 0:06.63 freeswitch
19171 root 20 0 18988 2948 944 R 2 0.1 0:00.64 top
18735 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.29 freeswitch
16003 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:01.61 freeswitch
16690 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
16730 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
16750 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.45 freeswitch
16764 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.44 freeswitch
<more of the above> .... ....
Thanks to all of you , Boris Ratner.
On Fri, Mar 8, 2013 at 4:22 AM, Dmitry Lysenko <dvl36.ripe.nick@gmail.com
wrote:
I can't reproduce such cps load on my ARMv5TE system. ) bratner, please give us 'top -H'. I guess freeswitch running at realtime priority.
2013/3/8 Ken Rice <krice@freeswitch.org>
Sqlite is probably getting hammered... Trust me... Mount the fs db dir as tmpfs or use the –nosql flag when starting freeswitch
I routinely run dialer traffic at much higher CPS then that
On 3/7/13 7:58 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com> wrote:
bi, bo and wa field is low, so it seems that is not disk subsystem.
2013/3/8 Ken Rice <krice@freeswitch.org>
You are probably hammering the disk subsystem... Keep in mind that FS uses multiple sqlite databases by default... Mount the fs db dir as
tmpfs
and try again
On 3/7/13 7:35 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com <http://dvl36.ripe.nick@gmail.com> > wrote:
Hm... But what about huge interrupt and context switching number?
________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org irc.freenode.net #freeswitch
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
List, Steve I will clarify what i'm asking here before I take Anothny's suggestion and join a "computer tuning" club as a way to "move forward". http://media.bestofmicro.com/gerbilpc-tuning-pc,S-L-252453-13.jpg What is there to read on this subject? Links, textbook names - everything is appreciated. What are the tools that show useful data and what i can do with FS to make the work easier? Compile with some flags to get more info on running threads? Thanks, Boris Ratner. On Sat, Mar 9, 2013 at 12:51 AM, Steven Ayre <steveayre@gmail.com> wrote:
After stopping the load FS still hogs 22.1% of memory.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 freeswitch
Until you test with the version you're building from master I would ignore the memory usage since you're running a version with known memory leaks.
-Steve
On 8 March 2013 18:15, bratner bratner <ratner2@gmail.com> wrote:
Here is sipp output and additional numbers for a test I ran with -nosql param.
The test ran 180CPS for ~3500seconds and the rest with 210cps.
Trouble (as in higher system cpu% ) started to appear around 8591seconds into the test. As you can see below the problem started just before 9124sec into the test 210cps 5sec calls should not give you a lot more then 1050 concurrent calls.
------------------------------ Scenario Screen -------- [1-9]: Change Screen -- Call-rate(length) Port Total-time Total-calls Remote-host 210.0(5000 ms)/1.000s 5061 9157.32 s 1834024 192.96.201.164:5060(UDP)
0 new calls during 0.000 s period 0 ms scheduler resolution 0 calls (limit 2000) Peak was 2000 calls, after 9124 s 0 Running, 4640 Paused, 0 Woken up 20 dead call msg (discarded) 0 out-of-call msg (discarded)
1 open sockets
Messages Retrans Timeout Unexpected-Msg INVITE ----------> 1834024 74 0 100 <---------- 1834024 0 0 0 180 <---------- 1834024 0 0 0 183 <---------- 0 0 0 0 500 <---------- 0 0 0 0 502 <---------- 0 0 0 0 503 <---------- 0 0 0 0 408 <---------- 0 0 0 0 480 <---------- 0 0 0 0 200 <---------- E-RTD1 1834024 81 0 0
ACK ----------> 1834024 81 Pause [ 5000ms] 1834024 0 BYE ----------> 1834024 7646 0 503 <---------- 0 0 0 0 200 <---------- 1834024 0 0 0
------------------------------ Test Terminated --------------------------------
----------------------------- Statistics Screen ------- [1-9]: Change Screen -- Start Time | 2013-03-08 15:22:18:204 1362756138.204833 Last Reset Time | 2013-03-08 17:54:55:535 1362765295.535214 Current Time | 2013-03-08 17:54:55:535 1362765295.535437
-------------------------+---------------------------+--------------------------
Counter Name | Periodic value | Cumulative value
-------------------------+---------------------------+--------------------------
Elapsed Time | 00:00:00:000 | 02:32:37:330
Call Rate | 0.000 cps | 200.279 cps
-------------------------+---------------------------+--------------------------
Incoming call created | 0 | 0
OutGoing call created | 0 | 1834024
Total Call created | | 1834024
Current Call | 0 |
-------------------------+---------------------------+--------------------------
Successful call | 0 | 1834024
Failed call | 0 | 0
-------------------------+---------------------------+--------------------------
Response Time 1 | 00:00:00:000 | 00:00:00:149
Call Length | 00:00:00:000 | 00:00:05:158
------------------------------ Test Terminated --------------------------------
After stopping the load FS still hogs 22.1% of memory. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 freeswitch
The symptoms of the crash are the same, just now with higher CPS and
takes
more time (more calls ) before crashing.
I will appreciate any suggestion.
Regards, Boris Ratner.
On Fri, Mar 8, 2013 at 6:22 PM, bratner bratner <ratner2@gmail.com> wrote:
The original test was done on git master at the date mentioned. The
sqlite
core.db file was on /run/shm which is a tmpfs on unbuntu 12.04. I will be recompiling from git master and test running with -nosql.
Testing my existing setup with -nosql seems more stable now running at 210CPS for some time (500k calls already passed) with ~35% idle cpu. But the free mem is slowly going down. I will let it run untill the kernel will kill it to see how many calls it can handle.
During my tests i did not run FS with RT priority but according to htop some of the threads are scheduled as RT. My setup is doing bypass-media , thus FS handling only call establishment and teardown on both legs.
cat /proc/<FS pid>/status
Name: freeswitch State: S (sleeping) Tgid: 15995 Pid: 15995 PPid: 1 TracerPid: 0 Uid: 999 999 999 999 Gid: 999 999 999 999 FDSize: 64 Groups: VmPeak: 5002808 kB VmSize: 5002088 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 625900 kB VmRSS: 624156 kB <-- this is going up VmData: 4855788 kB VmStk: 136 kB VmExe: 20 kB VmLib: 18288 kB VmPTE: 2352 kB VmSwap: 0 kB Threads: 1866 SigQ: 0/18446744073709551615 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000010003006 SigCgt: 0000000180014209 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: ffffffffffffffff Cpus_allowed: ffffff Cpus_allowed_list: 0-23 Mems_allowed: 00000000,00000003 Mems_allowed_list: 0-1 voluntary_ctxt_switches: 1803 nonvoluntary_ctxt_switches: 23
output of 'top -H' at 180CPS
top - 15:27:00 up 2 days, 5:32, 5 users, load average: 8.19, 91.07, 65.03 Tasks: 2066 total, 3 running, 2063 sleeping, 0 stopped, 0 zombie Cpu(s): 50.1%us, 3.9%sy, 0.0%ni, 45.9%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 4038512k total, 2282260k used, 1756252k free, 114112k buffers Swap: 0k total, 0k used, 0k free, 1165868k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16000 freeswit RT -10 4885m 594m 4964 R 69 15.1 3:10.26 freeswitch
16009 freeswit RT -10 4885m 594m 4964 S 33 15.1 1:26.20 freeswitch
16008 freeswit RT -10 4885m 594m 4964 S 28 15.1 1:17.30 freeswitch
16007 freeswit RT -10 4885m 594m 4964 S 4 15.1 0:10.80 freeswitch
16004 freeswit RT -10 4885m 594m 4964 S 2 15.1 0:06.63 freeswitch
19171 root 20 0 18988 2948 944 R 2 0.1 0:00.64 top
18735 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.29 freeswitch
16003 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:01.61 freeswitch
16690 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
16730 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
16750 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.45 freeswitch
16764 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.44 freeswitch
<more of the above> .... ....
Thanks to all of you , Boris Ratner.
On Fri, Mar 8, 2013 at 4:22 AM, Dmitry Lysenko < dvl36.ripe.nick@gmail.com> wrote:
I can't reproduce such cps load on my ARMv5TE system. ) bratner, please give us 'top -H'. I guess freeswitch running at
realtime
priority.
2013/3/8 Ken Rice <krice@freeswitch.org>
Sqlite is probably getting hammered... Trust me... Mount the fs db dir as tmpfs or use the –nosql flag when starting freeswitch
I routinely run dialer traffic at much higher CPS then that
On 3/7/13 7:58 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com>
wrote:
bi, bo and wa field is low, so it seems that is not disk subsystem.
2013/3/8 Ken Rice <krice@freeswitch.org>
You are probably hammering the disk subsystem... Keep in mind that FS uses multiple sqlite databases by default... Mount the fs db dir as
tmpfs
and try again
On 3/7/13 7:35 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com <http://dvl36.ripe.nick@gmail.com> > wrote:
Hm... But what about huge interrupt and context switching number?
________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org irc.freenode.net #freeswitch
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Boris, did you try to test load forcing freeswitch to run with normal priority? (-np) It seems that I have workaround, but don't sure exactly that your cpu load issue has the same root as mine. My system setup is uncommon (arm,128mb of RAM,RT kernel,mod_gsmopen), so I can't test it myself. 2013/3/10 bratner bratner <ratner2@gmail.com>
List, Steve
I will clarify what i'm asking here before I take Anothny's suggestion and join a "computer tuning" club as a way to "move forward". http://media.bestofmicro.com/gerbilpc-tuning-pc,S-L-252453-13.jpg
What is there to read on this subject? Links, textbook names - everything is appreciated. What are the tools that show useful data and what i can do with FS to make the work easier? Compile with some flags to get more info on running threads?
Thanks, Boris Ratner.
On Sat, Mar 9, 2013 at 12:51 AM, Steven Ayre <steveayre@gmail.com> wrote:
After stopping the load FS still hogs 22.1% of memory.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 freeswitch
Until you test with the version you're building from master I would ignore the memory usage since you're running a version with known memory leaks.
-Steve
On 8 March 2013 18:15, bratner bratner <ratner2@gmail.com> wrote:
Here is sipp output and additional numbers for a test I ran with -nosql param.
The test ran 180CPS for ~3500seconds and the rest with 210cps.
Trouble (as in higher system cpu% ) started to appear around 8591seconds into the test. As you can see below the problem started just before 9124sec into the test 210cps 5sec calls should not give you a lot more then 1050 concurrent calls.
------------------------------ Scenario Screen -------- [1-9]: Change Screen -- Call-rate(length) Port Total-time Total-calls Remote-host 210.0(5000 ms)/1.000s 5061 9157.32 s 1834024 192.96.201.164:5060(UDP)
0 new calls during 0.000 s period 0 ms scheduler resolution 0 calls (limit 2000) Peak was 2000 calls, after 9124 s 0 Running, 4640 Paused, 0 Woken up 20 dead call msg (discarded) 0 out-of-call msg (discarded)
1 open sockets
Messages Retrans Timeout Unexpected-Msg INVITE ----------> 1834024 74 0 100 <---------- 1834024 0 0 0 180 <---------- 1834024 0 0 0 183 <---------- 0 0 0 0 500 <---------- 0 0 0 0 502 <---------- 0 0 0 0 503 <---------- 0 0 0 0 408 <---------- 0 0 0 0 480 <---------- 0 0 0 0 200 <---------- E-RTD1 1834024 81 0 0
ACK ----------> 1834024 81 Pause [ 5000ms] 1834024 0 BYE ----------> 1834024 7646 0 503 <---------- 0 0 0 0 200 <---------- 1834024 0 0 0
------------------------------ Test Terminated --------------------------------
----------------------------- Statistics Screen ------- [1-9]: Change Screen -- Start Time | 2013-03-08 15:22:18:204 1362756138.204833 Last Reset Time | 2013-03-08 17:54:55:535 1362765295.535214 Current Time | 2013-03-08 17:54:55:535 1362765295.535437
-------------------------+---------------------------+--------------------------
Counter Name | Periodic value | Cumulative value
-------------------------+---------------------------+--------------------------
Elapsed Time | 00:00:00:000 | 02:32:37:330
Call Rate | 0.000 cps | 200.279 cps
-------------------------+---------------------------+--------------------------
Incoming call created | 0 | 0
OutGoing call created | 0 | 1834024
Total Call created | | 1834024
Current Call | 0 |
-------------------------+---------------------------+--------------------------
Successful call | 0 | 1834024
Failed call | 0 | 0
-------------------------+---------------------------+--------------------------
Response Time 1 | 00:00:00:000 | 00:00:00:149
Call Length | 00:00:00:000 | 00:00:05:158
------------------------------ Test Terminated --------------------------------
After stopping the load FS still hogs 22.1% of memory. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 freeswitch
The symptoms of the crash are the same, just now with higher CPS and
takes
more time (more calls ) before crashing.
I will appreciate any suggestion.
Regards, Boris Ratner.
On Fri, Mar 8, 2013 at 6:22 PM, bratner bratner <ratner2@gmail.com> wrote:
The original test was done on git master at the date mentioned. The
sqlite
core.db file was on /run/shm which is a tmpfs on unbuntu 12.04. I will be recompiling from git master and test running with -nosql.
Testing my existing setup with -nosql seems more stable now running at 210CPS for some time (500k calls already passed) with ~35% idle cpu. But the free mem is slowly going down. I will let it run untill the kernel will kill it to see how many calls it can handle.
During my tests i did not run FS with RT priority but according to htop some of the threads are scheduled as RT. My setup is doing bypass-media , thus FS handling only call establishment and teardown on both legs.
cat /proc/<FS pid>/status
Name: freeswitch State: S (sleeping) Tgid: 15995 Pid: 15995 PPid: 1 TracerPid: 0 Uid: 999 999 999 999 Gid: 999 999 999 999 FDSize: 64 Groups: VmPeak: 5002808 kB VmSize: 5002088 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 625900 kB VmRSS: 624156 kB <-- this is going up VmData: 4855788 kB VmStk: 136 kB VmExe: 20 kB VmLib: 18288 kB VmPTE: 2352 kB VmSwap: 0 kB Threads: 1866 SigQ: 0/18446744073709551615 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000010003006 SigCgt: 0000000180014209 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: ffffffffffffffff Cpus_allowed: ffffff Cpus_allowed_list: 0-23 Mems_allowed: 00000000,00000003 Mems_allowed_list: 0-1 voluntary_ctxt_switches: 1803 nonvoluntary_ctxt_switches: 23
output of 'top -H' at 180CPS
top - 15:27:00 up 2 days, 5:32, 5 users, load average: 8.19, 91.07, 65.03 Tasks: 2066 total, 3 running, 2063 sleeping, 0 stopped, 0 zombie Cpu(s): 50.1%us, 3.9%sy, 0.0%ni, 45.9%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 4038512k total, 2282260k used, 1756252k free, 114112k buffers Swap: 0k total, 0k used, 0k free, 1165868k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16000 freeswit RT -10 4885m 594m 4964 R 69 15.1 3:10.26 freeswitch
16009 freeswit RT -10 4885m 594m 4964 S 33 15.1 1:26.20 freeswitch
16008 freeswit RT -10 4885m 594m 4964 S 28 15.1 1:17.30 freeswitch
16007 freeswit RT -10 4885m 594m 4964 S 4 15.1 0:10.80 freeswitch
16004 freeswit RT -10 4885m 594m 4964 S 2 15.1 0:06.63 freeswitch
19171 root 20 0 18988 2948 944 R 2 0.1 0:00.64 top
18735 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.29 freeswitch
16003 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:01.61 freeswitch
16690 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
16730 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
16750 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.45 freeswitch
16764 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.44 freeswitch
<more of the above> .... ....
Thanks to all of you , Boris Ratner.
On Fri, Mar 8, 2013 at 4:22 AM, Dmitry Lysenko < dvl36.ripe.nick@gmail.com> wrote:
I can't reproduce such cps load on my ARMv5TE system. ) bratner, please give us 'top -H'. I guess freeswitch running at
realtime
priority.
2013/3/8 Ken Rice <krice@freeswitch.org>
Sqlite is probably getting hammered... Trust me... Mount the fs db
dir
as tmpfs or use the –nosql flag when starting freeswitch
I routinely run dialer traffic at much higher CPS then that
On 3/7/13 7:58 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com> wrote:
bi, bo and wa field is low, so it seems that is not disk subsystem.
2013/3/8 Ken Rice <krice@freeswitch.org>
You are probably hammering the disk subsystem... Keep in mind that FS uses multiple sqlite databases by default... Mount the fs db dir as tmpfs and try again
On 3/7/13 7:35 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com <http://dvl36.ripe.nick@gmail.com> > wrote:
Hm... But what about huge interrupt and context switching number?
________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org irc.freenode.net #freeswitch
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Dmitry, Hi! Running with -np at 180CPS for 3000sec now (over 500k calls). I already passed by far the amount of calls i was able to do at this CPS previously. I can see that all FS threads are the same priority and there are no RT threads. Context switches per ser are rising slowly. If i can make a million calls it is good enough for me. There is a small mem leak but that is not what have me worried because i can monitor it and restart FS when necessary. In my previous tests when CS reached closer to 60k the spiral down began. You think that FS RT threads slowly starve another important task? Holding my fingers crossed. Thanks! Boris Ratner. On Sun, Mar 10, 2013 at 2:02 AM, Dmitry Lysenko <dvl36.ripe.nick@gmail.com>wrote:
Boris, did you try to test load forcing freeswitch to run with normal priority? (-np) It seems that I have workaround, but don't sure exactly that your cpu load issue has the same root as mine. My system setup is uncommon (arm,128mb of RAM,RT kernel,mod_gsmopen), so I can't test it myself.
2013/3/10 bratner bratner <ratner2@gmail.com>
List, Steve
I will clarify what i'm asking here before I take Anothny's suggestion and join a "computer tuning" club as a way to "move forward". http://media.bestofmicro.com/gerbilpc-tuning-pc,S-L-252453-13.jpg
What is there to read on this subject? Links, textbook names - everything is appreciated. What are the tools that show useful data and what i can do with FS to make the work easier? Compile with some flags to get more info on running threads?
Thanks, Boris Ratner.
On Sat, Mar 9, 2013 at 12:51 AM, Steven Ayre <steveayre@gmail.com> wrote:
After stopping the load FS still hogs 22.1% of memory.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 freeswitch
Until you test with the version you're building from master I would ignore the memory usage since you're running a version with known memory leaks.
-Steve
On 8 March 2013 18:15, bratner bratner <ratner2@gmail.com> wrote:
Here is sipp output and additional numbers for a test I ran with -nosql param.
The test ran 180CPS for ~3500seconds and the rest with 210cps.
Trouble (as in higher system cpu% ) started to appear around 8591seconds into the test. As you can see below the problem started just before 9124sec into the test 210cps 5sec calls should not give you a lot more then 1050 concurrent calls.
------------------------------ Scenario Screen -------- [1-9]: Change Screen -- Call-rate(length) Port Total-time Total-calls Remote-host 210.0(5000 ms)/1.000s 5061 9157.32 s 1834024 192.96.201.164:5060(UDP)
0 new calls during 0.000 s period 0 ms scheduler resolution 0 calls (limit 2000) Peak was 2000 calls, after 9124 s 0 Running, 4640 Paused, 0 Woken up 20 dead call msg (discarded) 0 out-of-call msg (discarded)
1 open sockets
Messages Retrans Timeout Unexpected-Msg INVITE ----------> 1834024 74 0 100 <---------- 1834024 0 0 0 180 <---------- 1834024 0 0 0 183 <---------- 0 0 0 0 500 <---------- 0 0 0 0 502 <---------- 0 0 0 0 503 <---------- 0 0 0 0 408 <---------- 0 0 0 0 480 <---------- 0 0 0 0 200 <---------- E-RTD1 1834024 81 0 0
ACK ----------> 1834024 81 Pause [ 5000ms] 1834024 0 BYE ----------> 1834024 7646 0 503 <---------- 0 0 0 0 200 <---------- 1834024 0 0 0
------------------------------ Test Terminated --------------------------------
----------------------------- Statistics Screen ------- [1-9]: Change Screen -- Start Time | 2013-03-08 15:22:18:204 1362756138.204833 Last Reset Time | 2013-03-08 17:54:55:535 1362765295.535214 Current Time | 2013-03-08 17:54:55:535 1362765295.535437
-------------------------+---------------------------+--------------------------
Counter Name | Periodic value | Cumulative value
-------------------------+---------------------------+--------------------------
Elapsed Time | 00:00:00:000 | 02:32:37:330
Call Rate | 0.000 cps | 200.279 cps
-------------------------+---------------------------+--------------------------
Incoming call created | 0 | 0
OutGoing call created | 0 | 1834024
Total Call created | | 1834024
Current Call | 0 |
-------------------------+---------------------------+--------------------------
Successful call | 0 | 1834024
Failed call | 0 | 0
-------------------------+---------------------------+--------------------------
Response Time 1 | 00:00:00:000 | 00:00:00:149
Call Length | 00:00:00:000 | 00:00:05:158
------------------------------ Test Terminated --------------------------------
After stopping the load FS still hogs 22.1% of memory. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 freeswitch
The symptoms of the crash are the same, just now with higher CPS and
takes
more time (more calls ) before crashing.
I will appreciate any suggestion.
Regards, Boris Ratner.
On Fri, Mar 8, 2013 at 6:22 PM, bratner bratner <ratner2@gmail.com> wrote:
The original test was done on git master at the date mentioned. The
sqlite
core.db file was on /run/shm which is a tmpfs on unbuntu 12.04. I will be recompiling from git master and test running with -nosql.
Testing my existing setup with -nosql seems more stable now running at 210CPS for some time (500k calls already passed) with ~35% idle cpu. But the free mem is slowly going down. I will let it run untill the kernel will kill it to see how many calls it can handle.
During my tests i did not run FS with RT priority but according to htop some of the threads are scheduled as RT. My setup is doing bypass-media , thus FS handling only call establishment and teardown on both legs.
cat /proc/<FS pid>/status
Name: freeswitch State: S (sleeping) Tgid: 15995 Pid: 15995 PPid: 1 TracerPid: 0 Uid: 999 999 999 999 Gid: 999 999 999 999 FDSize: 64 Groups: VmPeak: 5002808 kB VmSize: 5002088 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 625900 kB VmRSS: 624156 kB <-- this is going up VmData: 4855788 kB VmStk: 136 kB VmExe: 20 kB VmLib: 18288 kB VmPTE: 2352 kB VmSwap: 0 kB Threads: 1866 SigQ: 0/18446744073709551615 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000010003006 SigCgt: 0000000180014209 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: ffffffffffffffff Cpus_allowed: ffffff Cpus_allowed_list: 0-23 Mems_allowed: 00000000,00000003 Mems_allowed_list: 0-1 voluntary_ctxt_switches: 1803 nonvoluntary_ctxt_switches: 23
output of 'top -H' at 180CPS
top - 15:27:00 up 2 days, 5:32, 5 users, load average: 8.19, 91.07, 65.03 Tasks: 2066 total, 3 running, 2063 sleeping, 0 stopped, 0 zombie Cpu(s): 50.1%us, 3.9%sy, 0.0%ni, 45.9%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 4038512k total, 2282260k used, 1756252k free, 114112k buffers Swap: 0k total, 0k used, 0k free, 1165868k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16000 freeswit RT -10 4885m 594m 4964 R 69 15.1 3:10.26 freeswitch
16009 freeswit RT -10 4885m 594m 4964 S 33 15.1 1:26.20 freeswitch
16008 freeswit RT -10 4885m 594m 4964 S 28 15.1 1:17.30 freeswitch
16007 freeswit RT -10 4885m 594m 4964 S 4 15.1 0:10.80 freeswitch
16004 freeswit RT -10 4885m 594m 4964 S 2 15.1 0:06.63 freeswitch
19171 root 20 0 18988 2948 944 R 2 0.1 0:00.64 top
18735 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.29 freeswitch
16003 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:01.61 freeswitch
16690 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
16730 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
16750 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.45 freeswitch
16764 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.44 freeswitch
<more of the above> .... ....
Thanks to all of you , Boris Ratner.
On Fri, Mar 8, 2013 at 4:22 AM, Dmitry Lysenko < dvl36.ripe.nick@gmail.com> wrote:
I can't reproduce such cps load on my ARMv5TE system. ) bratner, please give us 'top -H'. I guess freeswitch running at
realtime
priority.
2013/3/8 Ken Rice <krice@freeswitch.org> > > Sqlite is probably getting hammered... Trust me... Mount the fs db dir > as tmpfs or use the –nosql flag when starting freeswitch > > I routinely run dialer traffic at much higher CPS then that > > > > On 3/7/13 7:58 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com> wrote: > > bi, bo and wa field is low, so it seems that is not disk subsystem. > > > 2013/3/8 Ken Rice <krice@freeswitch.org> > > You are probably hammering the disk subsystem... Keep in mind that FS > uses multiple sqlite databases by default... Mount the fs db dir as tmpfs > and try again > > > > On 3/7/13 7:35 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com > <http://dvl36.ripe.nick@gmail.com> > wrote: > > Hm... But what about huge interrupt and context switching number? > > > ________________________________ > >
> Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org > > > -- > Ken > http://www.FreeSWITCH.org > http://www.ClueCon.com > http://www.OSTAG.org > irc.freenode.net #freeswitch > > >
> Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org >
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I am not amused by your comment. If you want to mock my advise then stop asking for help. Someday you will learn that ddos with sipp is not the same as real traffic but this thread is now closed. On Mar 10, 2013 5:40 AM, "bratner bratner" <ratner2@gmail.com> wrote:
Dmitry, Hi!
Running with -np at 180CPS for 3000sec now (over 500k calls). I already passed by far the amount of calls i was able to do at this CPS previously. I can see that all FS threads are the same priority and there are no RT threads. Context switches per ser are rising slowly. If i can make a million calls it is good enough for me. There is a small mem leak but that is not what have me worried because i can monitor it and restart FS when necessary. In my previous tests when CS reached closer to 60k the spiral down began.
You think that FS RT threads slowly starve another important task?
Holding my fingers crossed.
Thanks! Boris Ratner.
On Sun, Mar 10, 2013 at 2:02 AM, Dmitry Lysenko <dvl36.ripe.nick@gmail.com
wrote:
Boris, did you try to test load forcing freeswitch to run with normal priority? (-np) It seems that I have workaround, but don't sure exactly that your cpu load issue has the same root as mine. My system setup is uncommon (arm,128mb of RAM,RT kernel,mod_gsmopen), so I can't test it myself.
2013/3/10 bratner bratner <ratner2@gmail.com>
List, Steve
I will clarify what i'm asking here before I take Anothny's suggestion and join a "computer tuning" club as a way to "move forward". http://media.bestofmicro.com/gerbilpc-tuning-pc,S-L-252453-13.jpg
What is there to read on this subject? Links, textbook names - everything is appreciated. What are the tools that show useful data and what i can do with FS to make the work easier? Compile with some flags to get more info on running threads?
Thanks, Boris Ratner.
On Sat, Mar 9, 2013 at 12:51 AM, Steven Ayre <steveayre@gmail.com>wrote:
After stopping the load FS still hogs 22.1% of memory.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 freeswitch
Until you test with the version you're building from master I would ignore the memory usage since you're running a version with known memory leaks.
-Steve
On 8 March 2013 18:15, bratner bratner <ratner2@gmail.com> wrote:
Here is sipp output and additional numbers for a test I ran with -nosql param.
The test ran 180CPS for ~3500seconds and the rest with 210cps.
Trouble (as in higher system cpu% ) started to appear around 8591seconds into the test. As you can see below the problem started just before 9124sec into the test 210cps 5sec calls should not give you a lot more then 1050 concurrent calls.
------------------------------ Scenario Screen -------- [1-9]: Change Screen -- Call-rate(length) Port Total-time Total-calls Remote-host 210.0(5000 ms)/1.000s 5061 9157.32 s 1834024 192.96.201.164:5060(UDP)
0 new calls during 0.000 s period 0 ms scheduler resolution 0 calls (limit 2000) Peak was 2000 calls, after 9124 s 0 Running, 4640 Paused, 0 Woken up 20 dead call msg (discarded) 0 out-of-call msg (discarded) 1 open sockets
Messages Retrans Timeout Unexpected-Msg INVITE ----------> 1834024 74 0
100 <---------- 1834024 0 0 0
180 <---------- 1834024 0 0 0
183 <---------- 0 0 0 0
500 <---------- 0 0 0 0
502 <---------- 0 0 0 0
503 <---------- 0 0 0 0
408 <---------- 0 0 0 0
480 <---------- 0 0 0 0
200 <---------- E-RTD1 1834024 81 0 0
ACK ----------> 1834024 81
Pause [ 5000ms] 1834024 0
BYE ----------> 1834024 7646 0
503 <---------- 0 0 0 0
200 <---------- 1834024 0 0 0
------------------------------ Test Terminated --------------------------------
----------------------------- Statistics Screen ------- [1-9]: Change
Screen
-- Start Time | 2013-03-08 15:22:18:204 1362756138.204833 Last Reset Time | 2013-03-08 17:54:55:535 1362765295.535214 Current Time | 2013-03-08 17:54:55:535 1362765295.535437
-------------------------+---------------------------+--------------------------
Counter Name | Periodic value | Cumulative value
-------------------------+---------------------------+--------------------------
Elapsed Time | 00:00:00:000 | 02:32:37:330
Call Rate | 0.000 cps | 200.279 cps
-------------------------+---------------------------+--------------------------
Incoming call created | 0 | 0
OutGoing call created | 0 | 1834024
Total Call created | | 1834024
Current Call | 0 |
-------------------------+---------------------------+--------------------------
Successful call | 0 | 1834024
Failed call | 0 | 0
-------------------------+---------------------------+--------------------------
Response Time 1 | 00:00:00:000 | 00:00:00:149
Call Length | 00:00:00:000 | 00:00:05:158
------------------------------ Test Terminated --------------------------------
After stopping the load FS still hogs 22.1% of memory. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 freeswitch
The symptoms of the crash are the same, just now with higher CPS and
takes
more time (more calls ) before crashing.
I will appreciate any suggestion.
Regards, Boris Ratner.
On Fri, Mar 8, 2013 at 6:22 PM, bratner bratner <ratner2@gmail.com> wrote:
The original test was done on git master at the date mentioned. The
sqlite
core.db file was on /run/shm which is a tmpfs on unbuntu 12.04. I will be recompiling from git master and test running with -nosql.
Testing my existing setup with -nosql seems more stable now running at 210CPS for some time (500k calls already passed) with ~35% idle cpu. But the free mem is slowly going down. I will let it run untill the kernel will kill it to see how many calls it can handle.
During my tests i did not run FS with RT priority but according to htop some of the threads are scheduled as RT. My setup is doing bypass-media , thus FS handling only call establishment and teardown on both legs.
cat /proc/<FS pid>/status
Name: freeswitch State: S (sleeping) Tgid: 15995 Pid: 15995 PPid: 1 TracerPid: 0 Uid: 999 999 999 999 Gid: 999 999 999 999 FDSize: 64 Groups: VmPeak: 5002808 kB VmSize: 5002088 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 625900 kB VmRSS: 624156 kB <-- this is going up VmData: 4855788 kB VmStk: 136 kB VmExe: 20 kB VmLib: 18288 kB VmPTE: 2352 kB VmSwap: 0 kB Threads: 1866 SigQ: 0/18446744073709551615 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000010003006 SigCgt: 0000000180014209 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: ffffffffffffffff Cpus_allowed: ffffff Cpus_allowed_list: 0-23 Mems_allowed: 00000000,00000003 Mems_allowed_list: 0-1 voluntary_ctxt_switches: 1803 nonvoluntary_ctxt_switches: 23
output of 'top -H' at 180CPS
top - 15:27:00 up 2 days, 5:32, 5 users, load average: 8.19, 91.07, 65.03 Tasks: 2066 total, 3 running, 2063 sleeping, 0 stopped, 0 zombie Cpu(s): 50.1%us, 3.9%sy, 0.0%ni, 45.9%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 4038512k total, 2282260k used, 1756252k free, 114112k buffers Swap: 0k total, 0k used, 0k free, 1165868k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16000 freeswit RT -10 4885m 594m 4964 R 69 15.1 3:10.26 freeswitch
16009 freeswit RT -10 4885m 594m 4964 S 33 15.1 1:26.20 freeswitch
16008 freeswit RT -10 4885m 594m 4964 S 28 15.1 1:17.30 freeswitch
16007 freeswit RT -10 4885m 594m 4964 S 4 15.1 0:10.80 freeswitch
16004 freeswit RT -10 4885m 594m 4964 S 2 15.1 0:06.63 freeswitch
19171 root 20 0 18988 2948 944 R 2 0.1 0:00.64 top
18735 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.29 freeswitch
16003 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:01.61 freeswitch
16690 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
16730 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
16750 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.45 freeswitch
16764 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.44 freeswitch
<more of the above> .... ....
Thanks to all of you , Boris Ratner.
On Fri, Mar 8, 2013 at 4:22 AM, Dmitry Lysenko < dvl36.ripe.nick@gmail.com> wrote: > > I can't reproduce such cps load on my ARMv5TE system. ) > bratner, please give us 'top -H'. I guess freeswitch running at realtime > priority. > > > 2013/3/8 Ken Rice <krice@freeswitch.org> >> >> Sqlite is probably getting hammered... Trust me... Mount the fs db dir >> as tmpfs or use the –nosql flag when starting freeswitch >> >> I routinely run dialer traffic at much higher CPS then that >> >> >> >> On 3/7/13 7:58 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com> wrote: >> >> bi, bo and wa field is low, so it seems that is not disk subsystem. >> >> >> 2013/3/8 Ken Rice <krice@freeswitch.org> >> >> You are probably hammering the disk subsystem... Keep in mind that FS >> uses multiple sqlite databases by default... Mount the fs db dir as tmpfs >> and try again >> >> >> >> On 3/7/13 7:35 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com >> <http://dvl36.ripe.nick@gmail.com> > wrote: >> >> Hm... But what about huge interrupt and context switching number? >> >> >> ________________________________ >> >>
>> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org >> http://www.freeswitchsolutions.com >> >> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >> http://www.cudatel.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://wiki.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >> >> >> -- >> Ken >> http://www.FreeSWITCH.org >> http://www.ClueCon.com >> http://www.OSTAG.org >> irc.freenode.net #freeswitch >> >> >>
>> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org >> http://www.freeswitchsolutions.com >> >> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >> http://www.cudatel.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://wiki.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >> > > >
> Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org >
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
List, Dmitry Everything looks fine @180CPS and I can conclude that running FS with -np together with switching the kernel to ubuntu's low latency image solved the problem. The mem leak remains but this I can handle by selective restarts. Thanks! Boris Ratner On Sun, Mar 10, 2013 at 12:35 PM, bratner bratner <ratner2@gmail.com> wrote:
Dmitry, Hi!
Running with -np at 180CPS for 3000sec now (over 500k calls). I already passed by far the amount of calls i was able to do at this CPS previously. I can see that all FS threads are the same priority and there are no RT threads. Context switches per ser are rising slowly. If i can make a million calls it is good enough for me. There is a small mem leak but that is not what have me worried because i can monitor it and restart FS when necessary. In my previous tests when CS reached closer to 60k the spiral down began.
You think that FS RT threads slowly starve another important task?
Holding my fingers crossed.
Thanks! Boris Ratner.
On Sun, Mar 10, 2013 at 2:02 AM, Dmitry Lysenko <dvl36.ripe.nick@gmail.com
wrote:
Boris, did you try to test load forcing freeswitch to run with normal priority? (-np) It seems that I have workaround, but don't sure exactly that your cpu load issue has the same root as mine. My system setup is uncommon (arm,128mb of RAM,RT kernel,mod_gsmopen), so I can't test it myself.
2013/3/10 bratner bratner <ratner2@gmail.com>
List, Steve
I will clarify what i'm asking here before I take Anothny's suggestion and join a "computer tuning" club as a way to "move forward". http://media.bestofmicro.com/gerbilpc-tuning-pc,S-L-252453-13.jpg
What is there to read on this subject? Links, textbook names - everything is appreciated. What are the tools that show useful data and what i can do with FS to make the work easier? Compile with some flags to get more info on running threads?
Thanks, Boris Ratner.
On Sat, Mar 9, 2013 at 12:51 AM, Steven Ayre <steveayre@gmail.com>wrote:
After stopping the load FS still hogs 22.1% of memory.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 freeswitch
Until you test with the version you're building from master I would ignore the memory usage since you're running a version with known memory leaks.
-Steve
On 8 March 2013 18:15, bratner bratner <ratner2@gmail.com> wrote:
Here is sipp output and additional numbers for a test I ran with -nosql param.
The test ran 180CPS for ~3500seconds and the rest with 210cps.
Trouble (as in higher system cpu% ) started to appear around 8591seconds into the test. As you can see below the problem started just before 9124sec into the test 210cps 5sec calls should not give you a lot more then 1050 concurrent calls.
------------------------------ Scenario Screen -------- [1-9]: Change Screen -- Call-rate(length) Port Total-time Total-calls Remote-host 210.0(5000 ms)/1.000s 5061 9157.32 s 1834024 192.96.201.164:5060(UDP)
0 new calls during 0.000 s period 0 ms scheduler resolution 0 calls (limit 2000) Peak was 2000 calls, after 9124 s 0 Running, 4640 Paused, 0 Woken up 20 dead call msg (discarded) 0 out-of-call msg (discarded) 1 open sockets
Messages Retrans Timeout Unexpected-Msg INVITE ----------> 1834024 74 0
100 <---------- 1834024 0 0 0
180 <---------- 1834024 0 0 0
183 <---------- 0 0 0 0
500 <---------- 0 0 0 0
502 <---------- 0 0 0 0
503 <---------- 0 0 0 0
408 <---------- 0 0 0 0
480 <---------- 0 0 0 0
200 <---------- E-RTD1 1834024 81 0 0
ACK ----------> 1834024 81
Pause [ 5000ms] 1834024 0
BYE ----------> 1834024 7646 0
503 <---------- 0 0 0 0
200 <---------- 1834024 0 0 0
------------------------------ Test Terminated --------------------------------
----------------------------- Statistics Screen ------- [1-9]: Change
Screen
-- Start Time | 2013-03-08 15:22:18:204 1362756138.204833 Last Reset Time | 2013-03-08 17:54:55:535 1362765295.535214 Current Time | 2013-03-08 17:54:55:535 1362765295.535437
-------------------------+---------------------------+--------------------------
Counter Name | Periodic value | Cumulative value
-------------------------+---------------------------+--------------------------
Elapsed Time | 00:00:00:000 | 02:32:37:330
Call Rate | 0.000 cps | 200.279 cps
-------------------------+---------------------------+--------------------------
Incoming call created | 0 | 0
OutGoing call created | 0 | 1834024
Total Call created | | 1834024
Current Call | 0 |
-------------------------+---------------------------+--------------------------
Successful call | 0 | 1834024
Failed call | 0 | 0
-------------------------+---------------------------+--------------------------
Response Time 1 | 00:00:00:000 | 00:00:00:149
Call Length | 00:00:00:000 | 00:00:05:158
------------------------------ Test Terminated --------------------------------
After stopping the load FS still hogs 22.1% of memory. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 freeswitch
The symptoms of the crash are the same, just now with higher CPS and
takes
more time (more calls ) before crashing.
I will appreciate any suggestion.
Regards, Boris Ratner.
On Fri, Mar 8, 2013 at 6:22 PM, bratner bratner <ratner2@gmail.com> wrote:
The original test was done on git master at the date mentioned. The
sqlite
core.db file was on /run/shm which is a tmpfs on unbuntu 12.04. I will be recompiling from git master and test running with -nosql.
Testing my existing setup with -nosql seems more stable now running at 210CPS for some time (500k calls already passed) with ~35% idle cpu. But the free mem is slowly going down. I will let it run untill the kernel will kill it to see how many calls it can handle.
During my tests i did not run FS with RT priority but according to htop some of the threads are scheduled as RT. My setup is doing bypass-media , thus FS handling only call establishment and teardown on both legs.
cat /proc/<FS pid>/status
Name: freeswitch State: S (sleeping) Tgid: 15995 Pid: 15995 PPid: 1 TracerPid: 0 Uid: 999 999 999 999 Gid: 999 999 999 999 FDSize: 64 Groups: VmPeak: 5002808 kB VmSize: 5002088 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 625900 kB VmRSS: 624156 kB <-- this is going up VmData: 4855788 kB VmStk: 136 kB VmExe: 20 kB VmLib: 18288 kB VmPTE: 2352 kB VmSwap: 0 kB Threads: 1866 SigQ: 0/18446744073709551615 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000010003006 SigCgt: 0000000180014209 CapInh: 0000000000000000 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: ffffffffffffffff Cpus_allowed: ffffff Cpus_allowed_list: 0-23 Mems_allowed: 00000000,00000003 Mems_allowed_list: 0-1 voluntary_ctxt_switches: 1803 nonvoluntary_ctxt_switches: 23
output of 'top -H' at 180CPS
top - 15:27:00 up 2 days, 5:32, 5 users, load average: 8.19, 91.07, 65.03 Tasks: 2066 total, 3 running, 2063 sleeping, 0 stopped, 0 zombie Cpu(s): 50.1%us, 3.9%sy, 0.0%ni, 45.9%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 4038512k total, 2282260k used, 1756252k free, 114112k buffers Swap: 0k total, 0k used, 0k free, 1165868k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16000 freeswit RT -10 4885m 594m 4964 R 69 15.1 3:10.26 freeswitch
16009 freeswit RT -10 4885m 594m 4964 S 33 15.1 1:26.20 freeswitch
16008 freeswit RT -10 4885m 594m 4964 S 28 15.1 1:17.30 freeswitch
16007 freeswit RT -10 4885m 594m 4964 S 4 15.1 0:10.80 freeswitch
16004 freeswit RT -10 4885m 594m 4964 S 2 15.1 0:06.63 freeswitch
19171 root 20 0 18988 2948 944 R 2 0.1 0:00.64 top
18735 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.29 freeswitch
16003 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:01.61 freeswitch
16690 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
16730 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
16750 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.45 freeswitch
16764 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.44 freeswitch
<more of the above> .... ....
Thanks to all of you , Boris Ratner.
On Fri, Mar 8, 2013 at 4:22 AM, Dmitry Lysenko < dvl36.ripe.nick@gmail.com> wrote: > > I can't reproduce such cps load on my ARMv5TE system. ) > bratner, please give us 'top -H'. I guess freeswitch running at realtime > priority. > > > 2013/3/8 Ken Rice <krice@freeswitch.org> >> >> Sqlite is probably getting hammered... Trust me... Mount the fs db dir >> as tmpfs or use the –nosql flag when starting freeswitch >> >> I routinely run dialer traffic at much higher CPS then that >> >> >> >> On 3/7/13 7:58 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com> wrote: >> >> bi, bo and wa field is low, so it seems that is not disk subsystem. >> >> >> 2013/3/8 Ken Rice <krice@freeswitch.org> >> >> You are probably hammering the disk subsystem... Keep in mind that FS >> uses multiple sqlite databases by default... Mount the fs db dir as tmpfs >> and try again >> >> >> >> On 3/7/13 7:35 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com >> <http://dvl36.ripe.nick@gmail.com> > wrote: >> >> Hm... But what about huge interrupt and context switching number? >> >> >> ________________________________ >> >>
>> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org >> http://www.freeswitchsolutions.com >> >> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >> http://www.cudatel.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://wiki.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >> >> >> -- >> Ken >> http://www.FreeSWITCH.org >> http://www.ClueCon.com >> http://www.OSTAG.org >> irc.freenode.net #freeswitch >> >> >>
>> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org >> http://www.freeswitchsolutions.com >> >> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >> http://www.cudatel.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://wiki.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >> > > >
> Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org >
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
It seems that we found bug (racing condition?) in sofia lib code. It related to setting of highest possible (99) thread realtime priority in libs/sofia-sip/libsofia-sip-ua/su/su_pthread_port.c This issue can happen only when freeswitch is running with realtime priority (defaults in multi-cpu configuration in the latest git). If someone interesting, give me know. Thanks. 2013/3/10 bratner bratner <ratner2@gmail.com>
List, Dmitry
Everything looks fine @180CPS and I can conclude that running FS with -np together with switching the kernel to ubuntu's low latency image solved the problem. The mem leak remains but this I can handle by selective restarts.
Thanks! Boris Ratner
On Sun, Mar 10, 2013 at 12:35 PM, bratner bratner <ratner2@gmail.com>wrote:
Dmitry, Hi!
Running with -np at 180CPS for 3000sec now (over 500k calls). I already passed by far the amount of calls i was able to do at this CPS previously. I can see that all FS threads are the same priority and there are no RT threads. Context switches per ser are rising slowly. If i can make a million calls it is good enough for me. There is a small mem leak but that is not what have me worried because i can monitor it and restart FS when necessary. In my previous tests when CS reached closer to 60k the spiral down began.
You think that FS RT threads slowly starve another important task?
Holding my fingers crossed.
Thanks! Boris Ratner.
On Sun, Mar 10, 2013 at 2:02 AM, Dmitry Lysenko < dvl36.ripe.nick@gmail.com> wrote:
Boris, did you try to test load forcing freeswitch to run with normal priority? (-np) It seems that I have workaround, but don't sure exactly that your cpu load issue has the same root as mine. My system setup is uncommon (arm,128mb of RAM,RT kernel,mod_gsmopen), so I can't test it myself.
2013/3/10 bratner bratner <ratner2@gmail.com>
List, Steve
I will clarify what i'm asking here before I take Anothny's suggestion and join a "computer tuning" club as a way to "move forward". http://media.bestofmicro.com/gerbilpc-tuning-pc,S-L-252453-13.jpg
What is there to read on this subject? Links, textbook names - everything is appreciated. What are the tools that show useful data and what i can do with FS to make the work easier? Compile with some flags to get more info on running threads?
Thanks, Boris Ratner.
On Sat, Mar 9, 2013 at 12:51 AM, Steven Ayre <steveayre@gmail.com>wrote:
After stopping the load FS still hogs 22.1% of memory.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 freeswitch
Until you test with the version you're building from master I would ignore the memory usage since you're running a version with known memory leaks.
-Steve
On 8 March 2013 18:15, bratner bratner <ratner2@gmail.com> wrote:
Here is sipp output and additional numbers for a test I ran with -nosql param.
The test ran 180CPS for ~3500seconds and the rest with 210cps.
Trouble (as in higher system cpu% ) started to appear around 8591seconds into the test. As you can see below the problem started just before 9124sec into the test 210cps 5sec calls should not give you a lot more then 1050 concurrent calls.
------------------------------ Scenario Screen -------- [1-9]: Change Screen -- Call-rate(length) Port Total-time Total-calls Remote-host 210.0(5000 ms)/1.000s 5061 9157.32 s 1834024 192.96.201.164:5060(UDP)
0 new calls during 0.000 s period 0 ms scheduler resolution 0 calls (limit 2000) Peak was 2000 calls, after 9124 s 0 Running, 4640 Paused, 0 Woken up 20 dead call msg (discarded) 0 out-of-call msg (discarded) 1 open sockets
Messages Retrans Timeout Unexpected-Msg INVITE ----------> 1834024 74 0
100 <---------- 1834024 0 0 0
180 <---------- 1834024 0 0 0
183 <---------- 0 0 0 0
500 <---------- 0 0 0 0
502 <---------- 0 0 0 0
503 <---------- 0 0 0 0
408 <---------- 0 0 0 0
480 <---------- 0 0 0 0
200 <---------- E-RTD1 1834024 81 0 0
ACK ----------> 1834024 81
Pause [ 5000ms] 1834024 0
BYE ----------> 1834024 7646 0
503 <---------- 0 0 0 0
200 <---------- 1834024 0 0 0
------------------------------ Test Terminated --------------------------------
----------------------------- Statistics Screen ------- [1-9]:
Change Screen
-- Start Time | 2013-03-08 15:22:18:204 1362756138.204833 Last Reset Time | 2013-03-08 17:54:55:535 1362765295.535214 Current Time | 2013-03-08 17:54:55:535 1362765295.535437
-------------------------+---------------------------+--------------------------
Counter Name | Periodic value | Cumulative value
-------------------------+---------------------------+--------------------------
Elapsed Time | 00:00:00:000 | 02:32:37:330
Call Rate | 0.000 cps | 200.279 cps
-------------------------+---------------------------+--------------------------
Incoming call created | 0 | 0
OutGoing call created | 0 | 1834024
Total Call created | | 1834024
Current Call | 0 |
-------------------------+---------------------------+--------------------------
Successful call | 0 | 1834024
Failed call | 0 | 0
-------------------------+---------------------------+--------------------------
Response Time 1 | 00:00:00:000 | 00:00:00:149
Call Length | 00:00:00:000 | 00:00:05:158
------------------------------ Test Terminated --------------------------------
After stopping the load FS still hogs 22.1% of memory. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 freeswitch
The symptoms of the crash are the same, just now with higher CPS and
takes
more time (more calls ) before crashing.
I will appreciate any suggestion.
Regards, Boris Ratner.
On Fri, Mar 8, 2013 at 6:22 PM, bratner bratner <ratner2@gmail.com> wrote: > > The original test was done on git master at the date mentioned. The sqlite > core.db file was on /run/shm which is a tmpfs on unbuntu 12.04. > I will be recompiling from git master and test running with -nosql. > > Testing my existing setup with -nosql seems more stable now running at > 210CPS for some time (500k calls already passed) with ~35% idle cpu. > But the free mem is slowly going down. I will let it run untill the kernel > will kill it to see how many calls it can handle. > > During my tests i did not run FS with RT priority but according to htop > some of the threads are scheduled as RT. > My setup is doing bypass-media , thus FS handling only call establishment > and teardown on both legs. > > cat /proc/<FS pid>/status > > Name: freeswitch > State: S (sleeping) > Tgid: 15995 > Pid: 15995 > PPid: 1 > TracerPid: 0 > Uid: 999 999 999 999 > Gid: 999 999 999 999 > FDSize: 64 > Groups: > VmPeak: 5002808 kB > VmSize: 5002088 kB > VmLck: 0 kB > VmPin: 0 kB > VmHWM: 625900 kB > VmRSS: 624156 kB <-- this is going up > VmData: 4855788 kB > VmStk: 136 kB > VmExe: 20 kB > VmLib: 18288 kB > VmPTE: 2352 kB > VmSwap: 0 kB > Threads: 1866 > SigQ: 0/18446744073709551615 > SigPnd: 0000000000000000 > ShdPnd: 0000000000000000 > SigBlk: 0000000000000000 > SigIgn: 0000000010003006 > SigCgt: 0000000180014209 > CapInh: 0000000000000000 > CapPrm: 0000000000000000 > CapEff: 0000000000000000 > CapBnd: ffffffffffffffff > Cpus_allowed: ffffff > Cpus_allowed_list: 0-23 > Mems_allowed: 00000000,00000003 > Mems_allowed_list: 0-1 > voluntary_ctxt_switches: 1803 > nonvoluntary_ctxt_switches: 23 > > > output of 'top -H' at 180CPS > > > top - 15:27:00 up 2 days, 5:32, 5 users, load average: 8.19, 91.07, > 65.03 > Tasks: 2066 total, 3 running, 2063 sleeping, 0 stopped, 0 zombie > Cpu(s): 50.1%us, 3.9%sy, 0.0%ni, 45.9%id, 0.0%wa, 0.0%hi, 0.2%si, > 0.0%st > Mem: 4038512k total, 2282260k used, 1756252k free, 114112k buffers > Swap: 0k total, 0k used, 0k free, 1165868k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 16000 freeswit RT -10 4885m 594m 4964 R 69 15.1 3:10.26 freeswitch
> 16009 freeswit RT -10 4885m 594m 4964 S 33 15.1 1:26.20 freeswitch
> 16008 freeswit RT -10 4885m 594m 4964 S 28 15.1 1:17.30 freeswitch
> 16007 freeswit RT -10 4885m 594m 4964 S 4 15.1 0:10.80 freeswitch
> 16004 freeswit RT -10 4885m 594m 4964 S 2 15.1 0:06.63 freeswitch
> 19171 root 20 0 18988 2948 944 R 2 0.1 0:00.64 top
> 18735 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.29 freeswitch
> 16003 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:01.61 freeswitch
> 16690 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
> 16730 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
> 16750 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.45 freeswitch
> 16764 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.44 freeswitch
> <more of the above> > .... > .... > > > Thanks to all of you , > Boris Ratner. > > On Fri, Mar 8, 2013 at 4:22 AM, Dmitry Lysenko < dvl36.ripe.nick@gmail.com> > wrote: >> >> I can't reproduce such cps load on my ARMv5TE system. ) >> bratner, please give us 'top -H'. I guess freeswitch running at realtime >> priority. >> >> >> 2013/3/8 Ken Rice <krice@freeswitch.org> >>> >>> Sqlite is probably getting hammered... Trust me... Mount the fs db dir >>> as tmpfs or use the –nosql flag when starting freeswitch >>> >>> I routinely run dialer traffic at much higher CPS then that >>> >>> >>> >>> On 3/7/13 7:58 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com> wrote: >>> >>> bi, bo and wa field is low, so it seems that is not disk subsystem. >>> >>> >>> 2013/3/8 Ken Rice <krice@freeswitch.org> >>> >>> You are probably hammering the disk subsystem... Keep in mind that FS >>> uses multiple sqlite databases by default... Mount the fs db dir as tmpfs >>> and try again >>> >>> >>> >>> On 3/7/13 7:35 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com >>> <http://dvl36.ripe.nick@gmail.com> > wrote: >>> >>> Hm... But what about huge interrupt and context switching number? >>> >>> >>> ________________________________ >>> >>>
>>> Professional FreeSWITCH Consulting Services: >>> consulting@freeswitch.org >>> http://www.freeswitchsolutions.com >>> >>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>> http://www.cudatel.com >>> >>> Official FreeSWITCH Sites >>> http://www.freeswitch.org >>> http://wiki.freeswitch.org >>> http://www.cluecon.com >>> >>> FreeSWITCH-users mailing list >>> FreeSWITCH-users@lists.freeswitch.org >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>> UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users >>> http://www.freeswitch.org >>> >>> >>> -- >>> Ken >>> http://www.FreeSWITCH.org >>> http://www.ClueCon.com >>> http://www.OSTAG.org >>> irc.freenode.net #freeswitch >>> >>> >>>
>>> Professional FreeSWITCH Consulting Services: >>> consulting@freeswitch.org >>> http://www.freeswitchsolutions.com >>> >>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>> http://www.cudatel.com >>> >>> Official FreeSWITCH Sites >>> http://www.freeswitch.org >>> http://wiki.freeswitch.org >>> http://www.cluecon.com >>> >>> FreeSWITCH-users mailing list >>> FreeSWITCH-users@lists.freeswitch.org >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>> UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users >>> http://www.freeswitch.org >>> >> >> >>
>> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org >> http://www.freeswitchsolutions.com >> >> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >> http://www.cudatel.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://wiki.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >> >
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
When you run in rt it dies trying to keep up with your ddos. In np, it falls behind and queues up all the work, aka memory leak. On Mar 10, 2013 9:08 AM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com> wrote:
It seems that we found bug (racing condition?) in sofia lib code. It related to setting of highest possible (99) thread realtime priority in libs/sofia-sip/libsofia-sip-ua/su/su_pthread_port.c This issue can happen only when freeswitch is running with realtime priority (defaults in multi-cpu configuration in the latest git). If someone interesting, give me know. Thanks.
2013/3/10 bratner bratner <ratner2@gmail.com>
List, Dmitry
Everything looks fine @180CPS and I can conclude that running FS with -np together with switching the kernel to ubuntu's low latency image solved the problem. The mem leak remains but this I can handle by selective restarts.
Thanks! Boris Ratner
On Sun, Mar 10, 2013 at 12:35 PM, bratner bratner <ratner2@gmail.com>wrote:
Dmitry, Hi!
Running with -np at 180CPS for 3000sec now (over 500k calls). I already passed by far the amount of calls i was able to do at this CPS previously. I can see that all FS threads are the same priority and there are no RT threads. Context switches per ser are rising slowly. If i can make a million calls it is good enough for me. There is a small mem leak but that is not what have me worried because i can monitor it and restart FS when necessary. In my previous tests when CS reached closer to 60k the spiral down began.
You think that FS RT threads slowly starve another important task?
Holding my fingers crossed.
Thanks! Boris Ratner.
On Sun, Mar 10, 2013 at 2:02 AM, Dmitry Lysenko < dvl36.ripe.nick@gmail.com> wrote:
Boris, did you try to test load forcing freeswitch to run with normal priority? (-np) It seems that I have workaround, but don't sure exactly that your cpu load issue has the same root as mine. My system setup is uncommon (arm,128mb of RAM,RT kernel,mod_gsmopen), so I can't test it myself.
2013/3/10 bratner bratner <ratner2@gmail.com>
List, Steve
I will clarify what i'm asking here before I take Anothny's suggestion and join a "computer tuning" club as a way to "move forward". http://media.bestofmicro.com/gerbilpc-tuning-pc,S-L-252453-13.jpg
What is there to read on this subject? Links, textbook names - everything is appreciated. What are the tools that show useful data and what i can do with FS to make the work easier? Compile with some flags to get more info on running threads?
Thanks, Boris Ratner.
On Sat, Mar 9, 2013 at 12:51 AM, Steven Ayre <steveayre@gmail.com>wrote:
After stopping the load FS still hogs 22.1% of memory. > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > > 15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 > freeswitch
Until you test with the version you're building from master I would ignore the memory usage since you're running a version with known memory leaks.
-Steve
On 8 March 2013 18:15, bratner bratner <ratner2@gmail.com> wrote: > Here is sipp output and additional numbers for a test I ran with -nosql > param. > > The test ran 180CPS for ~3500seconds and the rest with 210cps. > > Trouble (as in higher system cpu% ) started to appear around 8591seconds > into the test. > As you can see below the problem started just before 9124sec into the test > 210cps 5sec calls > should not give you a lot more then 1050 concurrent calls. > > ------------------------------ Scenario Screen -------- [1-9]: Change Screen > -- > Call-rate(length) Port Total-time Total-calls Remote-host > 210.0(5000 ms)/1.000s 5061 9157.32 s 1834024 > 192.96.201.164:5060(UDP) > > 0 new calls during 0.000 s period 0 ms scheduler resolution > 0 calls (limit 2000) Peak was 2000 calls, after 9124 s > 0 Running, 4640 Paused, 0 Woken up > 20 dead call msg (discarded) 0 out-of-call msg (discarded) > 1 open sockets > > Messages Retrans Timeout > Unexpected-Msg > INVITE ----------> 1834024 74 0
> 100 <---------- 1834024 0 0 0
> 180 <---------- 1834024 0 0 0
> 183 <---------- 0 0 0 0
> 500 <---------- 0 0 0 0
> 502 <---------- 0 0 0 0
> 503 <---------- 0 0 0 0
> 408 <---------- 0 0 0 0
> 480 <---------- 0 0 0 0
> 200 <---------- E-RTD1 1834024 81 0 0
> > ACK ----------> 1834024 81
> Pause [ 5000ms] 1834024 0
> BYE ----------> 1834024 7646 0
> 503 <---------- 0 0 0 0
> 200 <---------- 1834024 0 0 0
> > ------------------------------ Test Terminated > -------------------------------- > > > ----------------------------- Statistics Screen ------- [1-9]: Change Screen > -- > Start Time | 2013-03-08 15:22:18:204 1362756138.204833 > Last Reset Time | 2013-03-08 17:54:55:535 1362765295.535214 > Current Time | 2013-03-08 17:54:55:535 1362765295.535437 > -------------------------+---------------------------+-------------------------- > Counter Name | Periodic value | Cumulative value > -------------------------+---------------------------+-------------------------- > Elapsed Time | 00:00:00:000 | 02:32:37:330
> Call Rate | 0.000 cps | 200.279 cps
> -------------------------+---------------------------+-------------------------- > Incoming call created | 0 | 0
> OutGoing call created | 0 | 1834024
> Total Call created | | 1834024
> Current Call | 0 |
> -------------------------+---------------------------+-------------------------- > Successful call | 0 | 1834024
> Failed call | 0 | 0
> -------------------------+---------------------------+-------------------------- > Response Time 1 | 00:00:00:000 | 00:00:00:149
> Call Length | 00:00:00:000 | 00:00:05:158
> ------------------------------ Test Terminated > -------------------------------- > > > After stopping the load FS still hogs 22.1% of memory. > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 15995 freeswit -2 -10 4677m 873m 5028 S 0 22.1 755:28.65 freeswitch
> > > The symptoms of the crash are the same, just now with higher CPS and takes > more time (more calls ) before crashing. > > I will appreciate any suggestion. > > Regards, > Boris Ratner. > > > > On Fri, Mar 8, 2013 at 6:22 PM, bratner bratner <ratner2@gmail.com> wrote: >> >> The original test was done on git master at the date mentioned. The sqlite >> core.db file was on /run/shm which is a tmpfs on unbuntu 12.04. >> I will be recompiling from git master and test running with -nosql. >> >> Testing my existing setup with -nosql seems more stable now running at >> 210CPS for some time (500k calls already passed) with ~35% idle cpu. >> But the free mem is slowly going down. I will let it run untill the kernel >> will kill it to see how many calls it can handle. >> >> During my tests i did not run FS with RT priority but according to htop >> some of the threads are scheduled as RT. >> My setup is doing bypass-media , thus FS handling only call establishment >> and teardown on both legs. >> >> cat /proc/<FS pid>/status >> >> Name: freeswitch >> State: S (sleeping) >> Tgid: 15995 >> Pid: 15995 >> PPid: 1 >> TracerPid: 0 >> Uid: 999 999 999 999 >> Gid: 999 999 999 999 >> FDSize: 64 >> Groups: >> VmPeak: 5002808 kB >> VmSize: 5002088 kB >> VmLck: 0 kB >> VmPin: 0 kB >> VmHWM: 625900 kB >> VmRSS: 624156 kB <-- this is going up >> VmData: 4855788 kB >> VmStk: 136 kB >> VmExe: 20 kB >> VmLib: 18288 kB >> VmPTE: 2352 kB >> VmSwap: 0 kB >> Threads: 1866 >> SigQ: 0/18446744073709551615 >> SigPnd: 0000000000000000 >> ShdPnd: 0000000000000000 >> SigBlk: 0000000000000000 >> SigIgn: 0000000010003006 >> SigCgt: 0000000180014209 >> CapInh: 0000000000000000 >> CapPrm: 0000000000000000 >> CapEff: 0000000000000000 >> CapBnd: ffffffffffffffff >> Cpus_allowed: ffffff >> Cpus_allowed_list: 0-23 >> Mems_allowed: 00000000,00000003 >> Mems_allowed_list: 0-1 >> voluntary_ctxt_switches: 1803 >> nonvoluntary_ctxt_switches: 23 >> >> >> output of 'top -H' at 180CPS >> >> >> top - 15:27:00 up 2 days, 5:32, 5 users, load average: 8.19, 91.07, >> 65.03 >> Tasks: 2066 total, 3 running, 2063 sleeping, 0 stopped, 0 zombie >> Cpu(s): 50.1%us, 3.9%sy, 0.0%ni, 45.9%id, 0.0%wa, 0.0%hi, 0.2%si, >> 0.0%st >> Mem: 4038512k total, 2282260k used, 1756252k free, 114112k buffers >> Swap: 0k total, 0k used, 0k free, 1165868k cached >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 16000 freeswit RT -10 4885m 594m 4964 R 69 15.1 3:10.26 freeswitch
>> 16009 freeswit RT -10 4885m 594m 4964 S 33 15.1 1:26.20 freeswitch
>> 16008 freeswit RT -10 4885m 594m 4964 S 28 15.1 1:17.30 freeswitch
>> 16007 freeswit RT -10 4885m 594m 4964 S 4 15.1 0:10.80 freeswitch
>> 16004 freeswit RT -10 4885m 594m 4964 S 2 15.1 0:06.63 freeswitch
>> 19171 root 20 0 18988 2948 944 R 2 0.1 0:00.64 top
>> 18735 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.29 freeswitch
>> 16003 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:01.61 freeswitch
>> 16690 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
>> 16730 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.42 freeswitch
>> 16750 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.45 freeswitch
>> 16764 freeswit -2 -10 4885m 594m 4964 S 1 15.1 0:00.44 freeswitch
>> <more of the above> >> .... >> .... >> >> >> Thanks to all of you , >> Boris Ratner. >> >> On Fri, Mar 8, 2013 at 4:22 AM, Dmitry Lysenko < dvl36.ripe.nick@gmail.com> >> wrote: >>> >>> I can't reproduce such cps load on my ARMv5TE system. ) >>> bratner, please give us 'top -H'. I guess freeswitch running at realtime >>> priority. >>> >>> >>> 2013/3/8 Ken Rice <krice@freeswitch.org> >>>> >>>> Sqlite is probably getting hammered... Trust me... Mount the fs db dir >>>> as tmpfs or use the –nosql flag when starting freeswitch >>>> >>>> I routinely run dialer traffic at much higher CPS then that >>>> >>>> >>>> >>>> On 3/7/13 7:58 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com> wrote: >>>> >>>> bi, bo and wa field is low, so it seems that is not disk subsystem. >>>> >>>> >>>> 2013/3/8 Ken Rice <krice@freeswitch.org> >>>> >>>> You are probably hammering the disk subsystem... Keep in mind that FS >>>> uses multiple sqlite databases by default... Mount the fs db dir as tmpfs >>>> and try again >>>> >>>> >>>> >>>> On 3/7/13 7:35 PM, "Dmitry Lysenko" <dvl36.ripe.nick@gmail.com >>>> <http://dvl36.ripe.nick@gmail.com> > wrote: >>>> >>>> Hm... But what about huge interrupt and context switching number? >>>> >>>> >>>> ________________________________ >>>> >>>> _________________________________________________________________________ >>>> Professional FreeSWITCH Consulting Services: >>>> consulting@freeswitch.org >>>> http://www.freeswitchsolutions.com >>>> >>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>>> http://www.cudatel.com >>>> >>>> Official FreeSWITCH Sites >>>> http://www.freeswitch.org >>>> http://wiki.freeswitch.org >>>> http://www.cluecon.com >>>> >>>> FreeSWITCH-users mailing list >>>> FreeSWITCH-users@lists.freeswitch.org >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>> UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users >>>> http://www.freeswitch.org >>>> >>>> >>>> -- >>>> Ken >>>> http://www.FreeSWITCH.org >>>> http://www.ClueCon.com >>>> http://www.OSTAG.org >>>> irc.freenode.net #freeswitch >>>> >>>> >>>> _________________________________________________________________________ >>>> Professional FreeSWITCH Consulting Services: >>>> consulting@freeswitch.org >>>> http://www.freeswitchsolutions.com >>>> >>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>>> http://www.cudatel.com >>>> >>>> Official FreeSWITCH Sites >>>> http://www.freeswitch.org >>>> http://wiki.freeswitch.org >>>> http://www.cluecon.com >>>> >>>> FreeSWITCH-users mailing list >>>> FreeSWITCH-users@lists.freeswitch.org >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>> UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users >>>> http://www.freeswitch.org >>>> >>> >>> >>> _________________________________________________________________________ >>> Professional FreeSWITCH Consulting Services: >>> consulting@freeswitch.org >>> http://www.freeswitchsolutions.com >>> >>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>> http://www.cudatel.com >>> >>> Official FreeSWITCH Sites >>> http://www.freeswitch.org >>> http://wiki.freeswitch.org >>> http://www.cluecon.com >>> >>> FreeSWITCH-users mailing list >>> FreeSWITCH-users@lists.freeswitch.org >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>> UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users >>> http://www.freeswitch.org >>> >> > > > _________________________________________________________________________ > Professional FreeSWITCH Consulting Services: > consulting@freeswitch.org > http://www.freeswitchsolutions.com > > FreeSWITCH-powered IP PBX: The CudaTel Communication Server > http://www.cudatel.com > > Official FreeSWITCH Sites > http://www.freeswitch.org > http://wiki.freeswitch.org > http://www.cluecon.com > > FreeSWITCH-users mailing list > FreeSWITCH-users@lists.freeswitch.org > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users > UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users > http://www.freeswitch.org >
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (5)
-
Anthony Minessale
-
bratner bratner
-
Dmitry Lysenko
-
Ken Rice
-
Steven Ayre