[Freeswitch-users] Memory Leaks

Thu Jul 31 01:04:43 PDT 2008

Can you run FS under valgrind and see if its really leaking?

As Brian mentioned Earlier, Memory pooling does not give up memory... I run
freeswitch at 1000s of concurrent channels...

You might not be able to do 5cps under valgrind (this is due to the
extensive memory checks valgrind does) but you should be able with in a
minute or two determine exactly what (if anything) is actually leaking

> From: Sangwoo Jin <swjinjin at volans.kr>
> Reply-To: <freeswitch-users at lists.freeswitch.org>
> Date: Thu, 31 Jul 2008 16:39:22 +0900
> To: <freeswitch-users at lists.freeswitch.org>
> Subject: Re: [Freeswitch-users] Memory Leaks
> 
> I don't have gotten the same result in testing MOH calls with 15 CPS.
> The memory usage of freeswitch on testing doesn't grow at some point.
> But, The memory usage of freeswitch on testing bridged calls with 5CPS and 1
> second duration was growing endlessly. I have watched The memory usage of
> freeswitch becomes 1G bytes.
> 
> I attached the scripts of my test.
> Firstly, execute the scripts like uas_10xx.sh in each shells,
> and execute call.sh.
> Then, the following call flows is rotated at 5 CPS.
> 1000 -> freeswitch -> 1011
> 1000 -> freeswitch -> 1012
> 1000 -> freeswitch -> 1013
> 1000 -> freeswitch -> 1014
> 1000 -> freeswitch -> 1015
> 1000 -> freeswitch -> 1016
> 1000 -> freeswitch -> 1017
> 1000 -> freeswitch -> 1018
> 1000 -> freeswitch -> 1019
> 
> And, please inform me where I can find the memory pre-allocation codes on
> freeswitch's starting?
> 
> Sangwoo Jin.
>> -----Original Message-----
>> From: freeswitch-users-bounces at lists.freeswitch.org [mailto:freeswitch-
>> users-bounces at lists.freeswitch.org] On Behalf Of Michael Jerris
>> Sent: Thursday, July 31, 2008 3:45 PM
>> To: freeswitch-users at lists.freeswitch.org
>> Subject: Re: [Freeswitch-users] Memory Leaks
>> 
>> Can you detail the exact callflow, config files and testing methods
>> you used to produce this issue?  I suspect that this may be a real
>> issue but we ran sipp testing 2 days ago so your scenario must be
>> hitting an edge case we were not seeing.  That being said we have made
>> no efforts to get freeswitch to run in so little memory, in fact we
>> have quite a few things we allocate at switch start that could very
>> well ammount to over 128mb even without calls.  You will likely not
>> get it to work reliably with so little memory without some significant
>> modifications.
>> 
>> Mike
>> 
>> On Jul 31, 2008, at 12:07 AM, "Sangwoo Jin" <swjinjin at volans.kr> wrote:
>> 
>>> I want to run freeswitch on low memory machine with 128M ram and no
>>> swap
>>> partition.
>>> On testing freeswitch on it, I have always seen the out of memory
>>> error.
>>> Could I control the memory pools?
>>> 
>>>> -----Original Message-----
>>>> From: freeswitch-users-bounces at lists.freeswitch.org [mailto:freeswitch-
>>>> users-bounces at lists.freeswitch.org] On Behalf Of Brian West
>>>> Sent: Thursday, July 31, 2008 11:38 AM
>>>> To: freeswitch-users at lists.freeswitch.org
>>>> Subject: Re: [Freeswitch-users] Memory Leaks
>>>> 
>>>> Try running 500,000 calls at 100-200 cps with a duration of 1 second.
>>>> You'll always see something about lost memory on shut down and you
>>>> can't fully trust that output due to the usage of memory pools.
>>>> You'll see a running FreeSWITCH hit a high water mark and level off
>>>> on
>>>> memory usage.  I suspect that if you do this test with that many
>>>> calls
>>>> you'll see the same usage at some point or it will be more
>>>> pronounced.  100 or 1000 calls isn't enough to really tell what is
>>>> going on.
>>>> 
>>>> /b
>>>> 
>>>> On Jul 30, 2008, at 9:25 PM, Sangwoo Jin wrote:
>>>> 
>>>>> 
>>>>> I can't use JIRA because I'm waiting for confirmation of creating
>>>>> account.
>>>>> 
>>>>> Now, I tested again with the current svn trunk.
>>>>> The lost memory of 1000 calls is 10 times more than 100 calls.
>>>>> 
>>>>> The following logs are my filtered valgrind's output.
>>>>> I have removed the lost blocks log which may not have to be free.
>>>>> 
>>>>> Could you check this problem?
>>>>> Sangwoo Jin
>>>>> 
>>>>> =
>>>>> =
>>>>> =
>>>>> =
>>>>> =
>>>>> =
>>>>> ===
>>>>> ===================================================================
>>>>> 100 calls' log
>>>>> =
>>>>> =
>>>>> =
>>>>> =
>>>>> =
>>>>> =
>>>>> ===
>>>>> ===================================================================
>>>>> ==29384== 284 bytes in 1 blocks are still reachable in loss record
>>>>> 25 of 60
>>>>> ==29384==    at 0x4021C2E: calloc (vg_replace_malloc.c:397)
>>>>> ==29384==    by 0x7D98B43: su_hash_alloc (su_alloc.c:390)
>>>>> ==29384==    by 0x7D98CDE: su_home_new (su_alloc.c:562)
>>>>> ==29384==    by 0x7D06390: msg_create (msg.c:61)
>>>>> ==29384==    by 0x7D2490A: nta_msg_create (nta.c:2831)
>>>>> ==29384==    by 0x7D4336A: nua_client_request_template (nua_stack.c:
>>>>> 2361)
>>>>> ==29384==    by 0x7D42C8E: nua_client_init_request (nua_stack.c:
>>>>> 2223)
>>>>> ==29384==    by 0x7D4256D: nua_client_create (nua_stack.c:2043)
>>>>> ==29384==    by 0x7D55125: nua_stack_invite (nua_session.c:701)
>>>>> ==29384==    by 0x7D3EA92: nua_stack_signal (nua_stack.c:592)
>>>>> ==29384==    by 0x7DA2FA9: su_base_port_execute_msgs
>>>>> (su_base_port.c:
>>>>> 276)
>>>>> ==29384==    by 0x7DA2D50: su_base_port_getmsgs (su_base_port.c:198)
>>>>> ==29384==
>>>>> ==29384==
>>>>> 
>>>>> ==29384==
>>>>> ==29384== 504 bytes in 1 blocks are possibly lost in loss record 30
>>>>> of 60
>>>>> ==29384==    at 0x4021C2E: calloc (vg_replace_malloc.c:397)
>>>>> ==29384==    by 0x7D98CC1: su_home_new (su_alloc.c:559)
>>>>> ==29384==    by 0x7D06390: msg_create (msg.c:61)
>>>>> ==29384==    by 0x7D2490A: nta_msg_create (nta.c:2831)
>>>>> ==29384==    by 0x7D4336A: nua_client_request_template (nua_stack.c:
>>>>> 2361)
>>>>> ==29384==    by 0x7D42C8E: nua_client_init_request (nua_stack.c:
>>>>> 2223)
>>>>> ==29384==    by 0x7D4256D: nua_client_create (nua_stack.c:2043)
>>>>> ==29384==    by 0x7D55125: nua_stack_invite (nua_session.c:701)
>>>>> ==29384==    by 0x7D3EA92: nua_stack_signal (nua_stack.c:592)
>>>>> ==29384==    by 0x7DA2FA9: su_base_port_execute_msgs
>>>>> (su_base_port.c:
>>>>> 276)
>>>>> ==29384==    by 0x7DA2D50: su_base_port_getmsgs (su_base_port.c:198)
>>>>> ==29384==    by 0x7DA3084: su_base_port_run (su_base_port.c:331)
>>>>> ==29384==
>>>>> ==29384==
>>>>> ==29384== 671 bytes in 8 blocks are possibly lost in loss record 32
>>>>> of 60
>>>>> ==29384==    at 0x4022B08: malloc (vg_replace_malloc.c:207)
>>>>> ==29384==    by 0x7D989A2: sub_alloc (su_alloc.c:490)
>>>>> ==29384==    by 0x7D9929E: su_alloc (su_alloc.c:771)
>>>>> ==29384==    by 0x7D04E15: msg_header_alloc (msg_parser.c:2309)
>>>>> ==29384==    by 0x7D0273D: header_parse (msg_parser.c:1114)
>>>>> ==29384==    by 0x7D025E3: extract_header (msg_parser.c:1071)
>>>>> ==29384==    by 0x7D02315: msg_extract_header (msg_parser.c:1008)
>>>>> ==29384==    by 0x7D05BD2: msg_header_parse_str (msg_parser.c:2801)
>>>>> ==29384==    by 0x7D05B1E: msg_header_add_str (msg_parser.c:2764)
>>>>> ==29384==    by 0x7D7B534: sip_add_tagis (sip_tag_class.c:260)
>>>>> ==29384==    by 0x7D42EB4: nua_client_init_request (nua_stack.c:
>>>>> 2278)
>>>>> ==29384==    by 0x7D4256D: nua_client_create (nua_stack.c:2043)
>>>>> ==29384==
>>>>> ==29384==
>>>>> ==29384== 1,072 bytes in 2 blocks are possibly lost in loss record
>>>>> 37 of 60
>>>>> ==29384==    at 0x4021C2E: calloc (vg_replace_malloc.c:397)
>>>>> ==29384==    by 0x7D9ABB4: su_zalloc (su_alloc.c:1437)
>>>>> ==29384==    by 0x7DA0A84: su_msg_new (su_root.c:901)
>>>>> ==29384==    by 0x7D3E4CB: nua_signal (nua_stack.c:495)
>>>>> ==29384==    by 0x7D46FF7: nua_invite (nua.c:633)
>>>>> ==29384==    by 0x7CEB0DE: sofia_glue_do_invite (sofia_glue.c:1305)
>>>>> ==29384==    by 0x7CCA773: sofia_on_init (mod_sofia.c:99)
>>>>> ==29384==    by 0x40A630D: switch_core_session_run
>>>>> (switch_core_state_machine.c:415)
>>>>> ==29384==    by 0x40A52F4: switch_core_session_thread
>>>>> (switch_core_session.c:787)
>>>>> ==29384==    by 0x40FC695: dummy_worker (thread.c:138)
>>>>> ==29384==    by 0x41EB191: start_thread (in /lib/
>>>>> libpthread-2.6.1.so)
>>>>> ==29384==    by 0x42F502D: clone (in /lib/libc-2.6.1.so)
>>>>> ==29384==
>>>>> ==29384==
>>>>> ==29384== 1,295 bytes in 21 blocks are still reachable in loss
>>>>> record 38 of
>>>>> 60
>>>>> ==29384==    at 0x4022B08: malloc (vg_replace_malloc.c:207)
>>>>> ==29384==    by 0x7D989A2: sub_alloc (su_alloc.c:490)
>>>>> ==29384==    by 0x7D9929E: su_alloc (su_alloc.c:771)
>>>>> ==29384==    by 0x7D04E15: msg_header_alloc (msg_parser.c:2309)
>>>>> ==29384==    by 0x7D11E7A: msg_header_make (msg_header_make.c:86)
>>>>> ==29384==    by 0x7D05A73: msg_header_add_make (msg_parser.c:2729)
>>>>> ==29384==    by 0x7D7B503: sip_add_tagis (sip_tag_class.c:256)
>>>>> ==29384==    by 0x7D433D8: nua_client_request_template (nua_stack.c:
>>>>> 2377)
>>>>> ==29384==    by 0x7D42C8E: nua_client_init_request (nua_stack.c:
>>>>> 2223)
>>>>> ==29384==    by 0x7D4256D: nua_client_create (nua_stack.c:2043)
>>>>> ==29384==    by 0x7D55125: nua_stack_invite (nua_session.c:701)
>>>>> ==29384==    by 0x7D3EA92: nua_stack_signal (nua_stack.c:592)
>>>>> ==29384==
>>>>> ==29384==
>>>>> ==29384== 28,116 bytes in 99 blocks are indirectly lost in loss
>>>>> record 55 of
>>>>> 60
>>>>> ==29384==    at 0x4021C2E: calloc (vg_replace_malloc.c:397)
>>>>> ==29384==    by 0x7D98B43: su_hash_alloc (su_alloc.c:390)
>>>>> ==29384==    by 0x7D98CDE: su_home_new (su_alloc.c:562)
>>>>> ==29384==    by 0x7D06390: msg_create (msg.c:61)
>>>>> ==29384==    by 0x7D2490A: nta_msg_create (nta.c:2831)
>>>>> ==29384==    by 0x7D4336A: nua_client_request_template (nua_stack.c:
>>>>> 2361)
>>>>> ==29384==    by 0x7D42C8E: nua_client_init_request (nua_stack.c:
>>>>> 2223)
>>>>> ==29384==    by 0x7D4256D: nua_client_create (nua_stack.c:2043)
>>>>> ==29384==    by 0x7D55125: nua_stack_invite (nua_session.c:701)
>>>>> ==29384==    by 0x7D3EA92: nua_stack_signal (nua_stack.c:592)
>>>>> ==29384==    by 0x7DA2FA9: su_base_port_execute_msgs
>>>>> (su_base_port.c:
>>>>> 276)
>>>>> ==29384==    by 0x7DA2D50: su_base_port_getmsgs (su_base_port.c:198)
>>>>> ==29384==
>>>>> ==29384==
>>>>> ==29384== 52,528 bytes in 98 blocks are definitely lost in loss
>>>>> record 56 of
>>>>> 60
>>>>> ==29384==    at 0x4021C2E: calloc (vg_replace_malloc.c:397)
>>>>> ==29384==    by 0x7D9ABB4: su_zalloc (su_alloc.c:1437)
>>>>> ==29384==    by 0x7DA0A84: su_msg_new (su_root.c:901)
>>>>> ==29384==    by 0x7D3E4CB: nua_signal (nua_stack.c:495)
>>>>> ==29384==    by 0x7D46FF7: nua_invite (nua.c:633)
>>>>> ==29384==    by 0x7CEB0DE: sofia_glue_do_invite (sofia_glue.c:1305)
>>>>> ==29384==    by 0x7CCA773: sofia_on_init (mod_sofia.c:99)
>>>>> ==29384==    by 0x40A630D: switch_core_session_run
>>>>> (switch_core_state_machine.c:415)
>>>>> ==29384==    by 0x40A52F4: switch_core_session_thread
>>>>> (switch_core_session.c:787)
>>>>> ==29384==    by 0x40FC695: dummy_worker (thread.c:138)
>>>>> ==29384==    by 0x41EB191: start_thread (in /lib/
>>>>> libpthread-2.6.1.so)
>>>>> ==29384==    by 0x42F502D: clone (in /lib/libc-2.6.1.so)
>>>>> ==29384==
>>>>> ==29384==
>>>>> ==29384== 57,034 bytes in 871 blocks are indirectly lost in loss
>>>>> record 57
>>>>> of 60
>>>>> ==29384==    at 0x4022B08: malloc (vg_replace_malloc.c:207)
>>>>> ==29384==    by 0x7D989A2: sub_alloc (su_alloc.c:490)
>>>>> ==29384==    by 0x7D9929E: su_alloc (su_alloc.c:771)
>>>>> ==29384==    by 0x7D04E15: msg_header_alloc (msg_parser.c:2309)
>>>>> ==29384==    by 0x7D11E7A: msg_header_make (msg_header_make.c:86)
>>>>> ==29384==    by 0x7D05A73: msg_header_add_make (msg_parser.c:2729)
>>>>> ==29384==    by 0x7D7B503: sip_add_tagis (sip_tag_class.c:256)
>>>>> ==29384==    by 0x7D42EB4: nua_client_init_request (nua_stack.c:
>>>>> 2278)
>>>>> ==29384==    by 0x7D4256D: nua_client_create (nua_stack.c:2043)
>>>>> ==29384==    by 0x7D55125: nua_stack_invite (nua_session.c:701)
>>>>> ==29384==    by 0x7D3EA92: nua_stack_signal (nua_stack.c:592)
>>>>> ==29384==    by 0x7DA2FA9: su_base_port_execute_msgs
>>>>> (su_base_port.c:
>>>>> 276)
>>>>> ==29384==
>>>>> ==29384==
>>>>> ==29384== 135,046 (49,896 direct, 85,150 indirect) bytes in 99
>>>>> blocks are
>>>>> definitely lost in loss record 58 of 60
>>>>> ==29384==    at 0x4021C2E: calloc (vg_replace_malloc.c:397)
>>>>> ==29384==    by 0x7D98CC1: su_home_new (su_alloc.c:559)
>>>>> ==29384==    by 0x7D06390: msg_create (msg.c:61)
>>>>> ==29384==    by 0x7D2490A: nta_msg_create (nta.c:2831)
>>>>> ==29384==    by 0x7D4336A: nua_client_request_template (nua_stack.c:
>>>>> 2361)
>>>>> ==29384==    by 0x7D42C8E: nua_client_init_request (nua_stack.c:
>>>>> 2223)
>>>>> ==29384==    by 0x7D4256D: nua_client_create (nua_stack.c:2043)
>>>>> ==29384==    by 0x7D55125: nua_stack_invite (nua_session.c:701)
>>>>> ==29384==    by 0x7D3EA92: nua_stack_signal (nua_stack.c:592)
>>>>> ==29384==    by 0x7DA2FA9: su_base_port_execute_msgs
>>>>> (su_base_port.c:
>>>>> 276)
>>>>> ==29384==    by 0x7DA2D50: su_base_port_getmsgs (su_base_port.c:198)
>>>>> ==29384==    by 0x7DA3084: su_base_port_run (su_base_port.c:331)
>>>>> ==29384==
>>>>> ==29384== LEAK SUMMARY:
>>>>> ==29384==    definitely lost: 108,035 bytes in 303 blocks.
>>>>> ==29384==    indirectly lost: 85,177 bytes in 972 blocks.
>>>>> ==29384==      possibly lost: 2,535 bytes in 13 blocks.
>>>>> ==29384==    still reachable: 16,325,250 bytes in 524 blocks.
>>>>> ==29384==         suppressed: 0 bytes in 0 blocks.
>>>>> 
>>>>> =
>>>>> =
>>>>> =
>>>>> =
>>>>> =
>>>>> =
>>>>> ===
>>>>> ===================================================================
>>>>> 1000 calls' log
>>>>> =
>>>>> =
>>>>> =
>>>>> =
>>>>> =
>>>>> =
>>>>> ===
>>>>> ===================================================================
>>>>> ==29612==
>>>>> ==29612== 3,024 bytes in 6 blocks are still reachable in loss record
>>>>> 39 of
>>>>> 64
>>>>> ==29612==    at 0x4021C2E: calloc (vg_replace_malloc.c:397)
>>>>> ==29612==    by 0x7D98CC1: su_home_new (su_alloc.c:559)
>>>>> ==29612==    by 0x7D06390: msg_create (msg.c:61)
>>>>> ==29612==    by 0x7D2490A: nta_msg_create (nta.c:2831)
>>>>> ==29612==    by 0x7D4336A: nua_client_request_template (nua_stack.c:
>>>>> 2361)
>>>>> ==29612==    by 0x7D42C8E: nua_client_init_request (nua_stack.c:
>>>>> 2223)
>>>>> ==29612==    by 0x7D4256D: nua_client_create (nua_stack.c:2043)
>>>>> ==29612==    by 0x7D55125: nua_stack_invite (nua_session.c:701)
>>>>> ==29612==    by 0x7D3EA92: nua_stack_signal (nua_stack.c:592)
>>>>> ==29612==    by 0x7DA2FA9: su_base_port_execute_msgs
>>>>> (su_base_port.c:
>>>>> 276)
>>>>> ==29612==    by 0x7DA2D50: su_base_port_getmsgs (su_base_port.c:198)
>>>>> ==29612==    by 0x7DA3084: su_base_port_run (su_base_port.c:331)
>>>>> ==29612==
>>>>> ==29612==
>>>>> ==29612== 3,692 bytes in 13 blocks are possibly lost in loss record
>>>>> 43 of 64
>>>>> ==29612==    at 0x4021C2E: calloc (vg_replace_malloc.c:397)
>>>>> ==29612==    by 0x7D98B43: su_hash_alloc (su_alloc.c:390)
>>>>> ==29612==    by 0x7D98CDE: su_home_new (su_alloc.c:562)
>>>>> ==29612==    by 0x7D06390: msg_create (msg.c:61)
>>>>> ==29612==    by 0x7D2490A: nta_msg_create (nta.c:2831)
>>>>> ==29612==    by 0x7D4336A: nua_client_request_template (nua_stack.c:
>>>>> 2361)
>>>>> ==29612==    by 0x7D42C8E: nua_client_init_request (nua_stack.c:
>>>>> 2223)
>>>>> ==29612==    by 0x7D4256D: nua_client_create (nua_stack.c:2043)
>>>>> ==29612==    by 0x7D55125: nua_stack_invite (nua_session.c:701)
>>>>> ==29612==    by 0x7D3EA92: nua_stack_signal (nua_stack.c:592)
>>>>> ==29612==    by 0x7DA2FA9: su_base_port_execute_msgs
>>>>> (su_base_port.c:
>>>>> 276)
>>>>> ==29612==    by 0x7DA2D50: su_base_port_getmsgs (su_base_port.c:198)
>>>>> ==29612==
>>>>> ==29612==
>>>>> ==29612== 4,322 bytes in 55 blocks are possibly lost in loss record
>>>>> 47 of 64
>>>>> ==29612==    at 0x4022B08: malloc (vg_replace_malloc.c:207)
>>>>> ==29612==    by 0x7D989A2: sub_alloc (su_alloc.c:490)
>>>>> ==29612==    by 0x7D9929E: su_alloc (su_alloc.c:771)
>>>>> ==29612==    by 0x7D9CAA2: su_strdup (su_strdup.c:55)
>>>>> ==29612==    by 0x7D05AF3: msg_header_add_str (msg_parser.c:2759)
>>>>> ==29612==    by 0x7D7B534: sip_add_tagis (sip_tag_class.c:260)
>>>>> ==29612==    by 0x7D42EB4: nua_client_init_request (nua_stack.c:
>>>>> 2278)
>>>>> ==29612==    by 0x7D4256D: nua_client_create (nua_stack.c:2043)
>>>>> ==29612==    by 0x7D55125: nua_stack_invite (nua_session.c:701)
>>>>> ==29612==    by 0x7D3EA92: nua_stack_signal (nua_stack.c:592)
>>>>> ==29612==    by 0x7DA2FA9: su_base_port_execute_msgs
>>>>> (su_base_port.c:
>>>>> 276)
>>>>> ==29612==    by 0x7DA2D50: su_base_port_getmsgs (su_base_port.c:198)
>>>>> ==29612==
>>>>> ==29612==
>>>>> ==29612== 5,360 bytes in 10 blocks are still reachable in loss
>>>>> record 49 of
>>>>> 64
>>>>> ==29612==    at 0x4021C2E: calloc (vg_replace_malloc.c:397)
>>>>> ==29612==    by 0x7D9ABB4: su_zalloc (su_alloc.c:1437)
>>>>> ==29612==    by 0x7DA0A84: su_msg_new (su_root.c:901)
>>>>> ==29612==    by 0x7D3E4CB: nua_signal (nua_stack.c:495)
>>>>> ==29612==    by 0x7D46FF7: nua_invite (nua.c:633)
>>>>> ==29612==    by 0x7CEB0DE: sofia_glue_do_invite (sofia_glue.c:1305)
>>>>> ==29612==    by 0x7CCA773: sofia_on_init (mod_sofia.c:99)
>>>>> ==29612==    by 0x40A630D: switch_core_session_run
>>>>> (switch_core_state_machine.c:415)
>>>>> ==29612==    by 0x40A52F4: switch_core_session_thread
>>>>> (switch_core_session.c:787)
>>>>> ==29612==    by 0x40FC695: dummy_worker (thread.c:138)
>>>>> ==29612==    by 0x41EB191: start_thread (in /lib/
>>>>> libpthread-2.6.1.so)
>>>>> ==29612==    by 0x42F502D: clone (in /lib/libc-2.6.1.so)
>>>>> ==29612==
>>>>> ==29612==
>>>>> ==29612== 5,360 bytes in 10 blocks are possibly lost in loss record
>>>>> 50 of 64
>>>>> ==29612==    at 0x4021C2E: calloc (vg_replace_malloc.c:397)
>>>>> 