Can you run FS under valgrind and see if its really leaking? As Brian mentioned Earlier, Memory pooling does not give up memory... I run freeswitch at 1000s of concurrent channels... You might not be able to do 5cps under valgrind (this is due to the extensive memory checks valgrind does) but you should be able with in a minute or two determine exactly what (if anything) is actually leaking
From: Sangwoo Jin <swjinjin@volans.kr> Reply-To: <freeswitch-users@lists.freeswitch.org> Date: Thu, 31 Jul 2008 16:39:22 +0900 To: <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] Memory Leaks
I don't have gotten the same result in testing MOH calls with 15 CPS. The memory usage of freeswitch on testing doesn't grow at some point. But, The memory usage of freeswitch on testing bridged calls with 5CPS and 1 second duration was growing endlessly. I have watched The memory usage of freeswitch becomes 1G bytes.
I attached the scripts of my test. Firstly, execute the scripts like uas_10xx.sh in each shells, and execute call.sh. Then, the following call flows is rotated at 5 CPS. 1000 -> freeswitch -> 1011 1000 -> freeswitch -> 1012 1000 -> freeswitch -> 1013 1000 -> freeswitch -> 1014 1000 -> freeswitch -> 1015 1000 -> freeswitch -> 1016 1000 -> freeswitch -> 1017 1000 -> freeswitch -> 1018 1000 -> freeswitch -> 1019
And, please inform me where I can find the memory pre-allocation codes on freeswitch's starting?
Sangwoo Jin.
-----Original Message----- From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch- users-bounces@lists.freeswitch.org] On Behalf Of Michael Jerris Sent: Thursday, July 31, 2008 3:45 PM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] Memory Leaks
Can you detail the exact callflow, config files and testing methods you used to produce this issue? I suspect that this may be a real issue but we ran sipp testing 2 days ago so your scenario must be hitting an edge case we were not seeing. That being said we have made no efforts to get freeswitch to run in so little memory, in fact we have quite a few things we allocate at switch start that could very well ammount to over 128mb even without calls. You will likely not get it to work reliably with so little memory without some significant modifications.
Mike
On Jul 31, 2008, at 12:07 AM, "Sangwoo Jin" <swjinjin@volans.kr> wrote:
I want to run freeswitch on low memory machine with 128M ram and no swap partition. On testing freeswitch on it, I have always seen the out of memory error. Could I control the memory pools?
-----Original Message----- From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch- users-bounces@lists.freeswitch.org] On Behalf Of Brian West Sent: Thursday, July 31, 2008 11:38 AM To: freeswitch-users@lists.freeswitch.org Subject: Re: [Freeswitch-users] Memory Leaks
Try running 500,000 calls at 100-200 cps with a duration of 1 second. You'll always see something about lost memory on shut down and you can't fully trust that output due to the usage of memory pools. You'll see a running FreeSWITCH hit a high water mark and level off on memory usage. I suspect that if you do this test with that many calls you'll see the same usage at some point or it will be more pronounced. 100 or 1000 calls isn't enough to really tell what is going on.
/b
On Jul 30, 2008, at 9:25 PM, Sangwoo Jin wrote:
I can't use JIRA because I'm waiting for confirmation of creating account.
Now, I tested again with the current svn trunk. The lost memory of 1000 calls is 10 times more than 100 calls.
The following logs are my filtered valgrind's output. I have removed the lost blocks log which may not have to be free.
Could you check this problem? Sangwoo Jin
= = = = = = === =================================================================== 100 calls' log = = = = = = === =================================================================== ==29384== 284 bytes in 1 blocks are still reachable in loss record 25 of 60 ==29384== at 0x4021C2E: calloc (vg_replace_malloc.c:397) ==29384== by 0x7D98B43: su_hash_alloc (su_alloc.c:390) ==29384== by 0x7D98CDE: su_home_new (su_alloc.c:562) ==29384== by 0x7D06390: msg_create (msg.c:61) ==29384== by 0x7D2490A: nta_msg_create (nta.c:2831) ==29384== by 0x7D4336A: nua_client_request_template (nua_stack.c: 2361) ==29384== by 0x7D42C8E: nua_client_init_request (nua_stack.c: 2223) ==29384== by 0x7D4256D: nua_client_create (nua_stack.c:2043) ==29384== by 0x7D55125: nua_stack_invite (nua_session.c:701) ==29384== by 0x7D3EA92: nua_stack_signal (nua_stack.c:592) ==29384== by 0x7DA2FA9: su_base_port_execute_msgs (su_base_port.c: 276) ==29384== by 0x7DA2D50: su_base_port_getmsgs (su_base_port.c:198) ==29384== ==29384==
==29384== ==29384== 504 bytes in 1 blocks are possibly lost in loss record 30 of 60 ==29384== at 0x4021C2E: calloc (vg_replace_malloc.c:397) ==29384== by 0x7D98CC1: su_home_new (su_alloc.c:559) ==29384== by 0x7D06390: msg_create (msg.c:61) ==29384== by 0x7D2490A: nta_msg_create (nta.c:2831) ==29384== by 0x7D4336A: nua_client_request_template (nua_stack.c: 2361) ==29384== by 0x7D42C8E: nua_client_init_request (nua_stack.c: 2223) ==29384== by 0x7D4256D: nua_client_create (nua_stack.c:2043) ==29384== by 0x7D55125: nua_stack_invite (nua_session.c:701) ==29384== by 0x7D3EA92: nua_stack_signal (nua_stack.c:592) ==29384== by 0x7DA2FA9: su_base_port_execute_msgs (su_base_port.c: 276) ==29384== by 0x7DA2D50: su_base_port_getmsgs (su_base_port.c:198) ==29384== by 0x7DA3084: su_base_port_run (su_base_port.c:331) ==29384== ==29384== ==29384== 671 bytes in 8 blocks are possibly lost in loss record 32 of 60 ==29384== at 0x4022B08: malloc (vg_replace_malloc.c:207) ==29384== by 0x7D989A2: sub_alloc (su_alloc.c:490) ==29384== by 0x7D9929E: su_alloc (su_alloc.c:771) ==29384== by 0x7D04E15: msg_header_alloc (msg_parser.c:2309) ==29384== by 0x7D0273D: header_parse (msg_parser.c:1114) ==29384== by 0x7D025E3: extract_header (msg_parser.c:1071) ==29384== by 0x7D02315: msg_extract_header (msg_parser.c:1008) ==29384== by 0x7D05BD2: msg_header_parse_str (msg_parser.c:2801) ==29384== by 0x7D05B1E: msg_header_add_str (msg_parser.c:2764) ==29384== by 0x7D7B534: sip_add_tagis (sip_tag_class.c:260) ==29384== by 0x7D42EB4: nua_client_init_request (nua_stack.c: 2278) ==29384== by 0x7D4256D: nua_client_create (nua_stack.c:2043) ==29384== ==29384== ==29384== 1,072 bytes in 2 blocks are possibly lost in loss record 37 of 60 ==29384== at 0x4021C2E: calloc (vg_replace_malloc.c:397) ==29384== by 0x7D9ABB4: su_zalloc (su_alloc.c:1437) ==29384== by 0x7DA0A84: su_msg_new (su_root.c:901) ==29384== by 0x7D3E4CB: nua_signal (nua_stack.c:495) ==29384== by 0x7D46FF7: nua_invite (nua.c:633) ==29384== by 0x7CEB0DE: sofia_glue_do_invite (sofia_glue.c:1305) ==29384== by 0x7CCA773: sofia_on_init (mod_sofia.c:99) ==29384== by 0x40A630D: switch_core_session_run (switch_core_state_machine.c:415) ==29384== by 0x40A52F4: switch_core_session_thread (switch_core_session.c:787) ==29384== by 0x40FC695: dummy_worker (thread.c:138) ==29384== by 0x41EB191: start_thread (in /lib/ libpthread-2.6.1.so) ==29384== by 0x42F502D: clone (in /lib/libc-2.6.1.so) ==29384== ==29384== ==29384== 1,295 bytes in 21 blocks are still reachable in loss record 38 of 60 ==29384== at 0x4022B08: malloc (vg_replace_malloc.c:207) ==29384== by 0x7D989A2: sub_alloc (su_alloc.c:490) ==29384== by 0x7D9929E: su_alloc (su_alloc.c:771) ==29384== by 0x7D04E15: msg_header_alloc (msg_parser.c:2309) ==29384== by 0x7D11E7A: msg_header_make (msg_header_make.c:86) ==29384== by 0x7D05A73: msg_header_add_make (msg_parser.c:2729) ==29384== by 0x7D7B503: sip_add_tagis (sip_tag_class.c:256) ==29384== by 0x7D433D8: nua_client_request_template (nua_stack.c: 2377) ==29384== by 0x7D42C8E: nua_client_init_request (nua_stack.c: 2223) ==29384== by 0x7D4256D: nua_client_create (nua_stack.c:2043) ==29384== by 0x7D55125: nua_stack_invite (nua_session.c:701) ==29384== by 0x7D3EA92: nua_stack_signal (nua_stack.c:592) ==29384== ==29384== ==29384== 28,116 bytes in 99 blocks are indirectly lost in loss record 55 of 60 ==29384== at 0x4021C2E: calloc (vg_replace_malloc.c:397) ==29384== by 0x7D98B43: su_hash_alloc (su_alloc.c:390) ==29384== by 0x7D98CDE: su_home_new (su_alloc.c:562) ==29384== by 0x7D06390: msg_create (msg.c:61) ==29384== by 0x7D2490A: nta_msg_create (nta.c:2831) ==29384== by 0x7D4336A: nua_client_request_template (nua_stack.c: 2361) ==29384== by 0x7D42C8E: nua_client_init_request (nua_stack.c: 2223) ==29384== by 0x7D4256D: nua_client_create (nua_stack.c:2043) ==29384== by 0x7D55125: nua_stack_invite (nua_session.c:701) ==29384== by 0x7D3EA92: nua_stack_signal (nua_stack.c:592) ==29384== by 0x7DA2FA9: su_base_port_execute_msgs (su_base_port.c: 276) ==29384== by 0x7DA2D50: su_base_port_getmsgs (su_base_port.c:198) ==29384== ==29384== ==29384== 52,528 bytes in 98 blocks are definitely lost in loss record 56 of 60 ==29384== at 0x4021C2E: calloc (vg_replace_malloc.c:397) ==29384== by 0x7D9ABB4: su_zalloc (su_alloc.c:1437) ==29384== by 0x7DA0A84: su_msg_new (su_root.c:901) ==29384== by 0x7D3E4CB: nua_signal (nua_stack.c:495) ==29384== by 0x7D46FF7: nua_invite (nua.c:633) ==29384== by 0x7CEB0DE: sofia_glue_do_invite (sofia_glue.c:1305) ==29384== by 0x7CCA773: sofia_on_init (mod_sofia.c:99) ==29384== by 0x40A630D: switch_core_session_run (switch_core_state_machine.c:415) ==29384== by 0x40A52F4: switch_core_session_thread (switch_core_session.c:787) ==29384== by 0x40FC695: dummy_worker (thread.c:138) ==29384== by 0x41EB191: start_thread (in /lib/ libpthread-2.6.1.so) ==29384== by 0x42F502D: clone (in /lib/libc-2.6.1.so) ==29384== ==29384== ==29384== 57,034 bytes in 871 blocks are indirectly lost in loss record 57 of 60 ==29384== at 0x4022B08: malloc (vg_replace_malloc.c:207) ==29384== by 0x7D989A2: sub_alloc (su_alloc.c:490) ==29384== by 0x7D9929E: su_alloc (su_alloc.c:771) ==29384== by 0x7D04E15: msg_header_alloc (msg_parser.c:2309) ==29384== by 0x7D11E7A: msg_header_make (msg_header_make.c:86) ==29384== by 0x7D05A73: msg_header_add_make (msg_parser.c:2729) ==29384== by 0x7D7B503: sip_add_tagis (sip_tag_class.c:256) ==29384== by 0x7D42EB4: nua_client_init_request (nua_stack.c: 2278) ==29384== by 0x7D4256D: nua_client_create (nua_stack.c:2043) ==29384== by 0x7D55125: nua_stack_invite (nua_session.c:701) ==29384== by 0x7D3EA92: nua_stack_signal (nua_stack.c:592) ==29384== by 0x7DA2FA9: su_base_port_execute_msgs (su_base_port.c: 276) ==29384== ==29384== ==29384== 135,046 (49,896 direct, 85,150 indirect) bytes in 99 blocks are definitely lost in loss record 58 of 60 ==29384== at 0x4021C2E: calloc (vg_replace_malloc.c:397) ==29384== by 0x7D98CC1: su_home_new (su_alloc.c:559) ==29384== by 0x7D06390: msg_create (msg.c:61) ==29384== by 0x7D2490A: nta_msg_create (nta.c:2831) ==29384== by 0x7D4336A: nua_client_request_template (nua_stack.c: 2361) ==29384== by 0x7D42C8E: nua_client_init_request (nua_stack.c: 2223) ==29384== by 0x7D4256D: nua_client_create (nua_stack.c:2043) ==29384== by 0x7D55125: nua_stack_invite (nua_session.c:701) ==29384== by 0x7D3EA92: nua_stack_signal (nua_stack.c:592) ==29384== by 0x7DA2FA9: su_base_port_execute_msgs (su_base_port.c: 276) ==29384== by 0x7DA2D50: su_base_port_getmsgs (su_base_port.c:198) ==29384== by 0x7DA3084: su_base_port_run (su_base_port.c:331) ==29384== ==29384== LEAK SUMMARY: ==29384== definitely lost: 108,035 bytes in 303 blocks. ==29384== indirectly lost: 85,177 bytes in 972 blocks. ==29384== possibly lost: 2,535 bytes in 13 blocks. ==29384== still reachable: 16,325,250 bytes in 524 blocks. ==29384== suppressed: 0 bytes in 0 blocks.
= = = = = = === =================================================================== 1000 calls' log = = = = = = === =================================================================== ==29612== ==29612== 3,024 bytes in 6 blocks are still reachable in loss record 39 of 64 ==29612== at 0x4021C2E: calloc (vg_replace_malloc.c:397) ==29612== by 0x7D98CC1: su_home_new (su_alloc.c:559) ==29612== by 0x7D06390: msg_create (msg.c:61) ==29612== by 0x7D2490A: nta_msg_create (nta.c:2831) ==29612== by 0x7D4336A: nua_client_request_template (nua_stack.c: 2361) ==29612== by 0x7D42C8E: nua_client_init_request (nua_stack.c: 2223) ==29612== by 0x7D4256D: nua_client_create (nua_stack.c:2043) ==29612== by 0x7D55125: nua_stack_invite (nua_session.c:701) ==29612== by 0x7D3EA92: nua_stack_signal (nua_stack.c:592) ==29612== by 0x7DA2FA9: su_base_port_execute_msgs (su_base_port.c: 276) ==29612== by 0x7DA2D50: su_base_port_getmsgs (su_base_port.c:198) ==29612== by 0x7DA3084: su_base_port_run (su_base_port.c:331) ==29612== ==29612== ==29612== 3,692 bytes in 13 blocks are possibly lost in loss record 43 of 64 ==29612== at 0x4021C2E: calloc (vg_replace_malloc.c:397) ==29612== by 0x7D98B43: su_hash_alloc (su_alloc.c:390) ==29612== by 0x7D98CDE: su_home_new (su_alloc.c:562) ==29612== by 0x7D06390: msg_create (msg.c:61) ==29612== by 0x7D2490A: nta_msg_create (nta.c:2831) ==29612== by 0x7D4336A: nua_client_request_template (nua_stack.c: 2361) ==29612== by 0x7D42C8E: nua_client_init_request (nua_stack.c: 2223) ==29612== by 0x7D4256D: nua_client_create (nua_stack.c:2043) ==29612== by 0x7D55125: nua_stack_invite (nua_session.c:701) ==29612== by 0x7D3EA92: nua_stack_signal (nua_stack.c:592) ==29612== by 0x7DA2FA9: su_base_port_execute_msgs (su_base_port.c: 276) ==29612== by 0x7DA2D50: su_base_port_getmsgs (su_base_port.c:198) ==29612== ==29612== ==29612== 4,322 bytes in 55 blocks are possibly lost in loss record 47 of 64 ==29612== at 0x4022B08: malloc (vg_replace_malloc.c:207) ==29612== by 0x7D989A2: sub_alloc (su_alloc.c:490) ==29612== by 0x7D9929E: su_alloc (su_alloc.c:771) ==29612== by 0x7D9CAA2: su_strdup (su_strdup.c:55) ==29612== by 0x7D05AF3: msg_header_add_str (msg_parser.c:2759) ==29612== by 0x7D7B534: sip_add_tagis (sip_tag_class.c:260) ==29612== by 0x7D42EB4: nua_client_init_request (nua_stack.c: 2278) ==29612== by 0x7D4256D: nua_client_create (nua_stack.c:2043) ==29612== by 0x7D55125: nua_stack_invite (nua_session.c:701) ==29612== by 0x7D3EA92: nua_stack_signal (nua_stack.c:592) ==29612== by 0x7DA2FA9: su_base_port_execute_msgs (su_base_port.c: 276) ==29612== by 0x7DA2D50: su_base_port_getmsgs (su_base_port.c:198) ==29612== ==29612== ==29612== 5,360 bytes in 10 blocks are still reachable in loss record 49 of 64 ==29612== at 0x4021C2E: calloc (vg_replace_malloc.c:397) ==29612== by 0x7D9ABB4: su_zalloc (su_alloc.c:1437) ==29612== by 0x7DA0A84: su_msg_new (su_root.c:901) ==29612== by 0x7D3E4CB: nua_signal (nua_stack.c:495) ==29612== by 0x7D46FF7: nua_invite (nua.c:633) ==29612== by 0x7CEB0DE: sofia_glue_do_invite (sofia_glue.c:1305) ==29612== by 0x7CCA773: sofia_on_init (mod_sofia.c:99) ==29612== by 0x40A630D: switch_core_session_run (switch_core_state_machine.c:415) ==29612== by 0x40A52F4: switch_core_session_thread (switch_core_session.c:787) ==29612== by 0x40FC695: dummy_worker (thread.c:138) ==29612== by 0x41EB191: start_thread (in /lib/ libpthread-2.6.1.so) ==29612== by 0x42F502D: clone (in /lib/libc-2.6.1.so) ==29612== ==29612== ==29612== 5,360 bytes in 10 blocks are possibly lost in loss record 50 of 64 ==29612== at 0x4021C2E: calloc (vg_replace_malloc.c:397)