[Freeswitch-users] Memory leak when using ESL
gggyniidt
gggyniidt at gmail.com
Sat Nov 16 07:10:55 MSK 2013
Sorry,I reply this late.
In these days I did some test for esl client app.My code like this:
#include <stdio.h>
#include <stdlib.h>
#include <esl.h>
static void mycallback(esl_socket_t server_sock, esl_socket_t client_sock,
struct sockaddr_in *addr)
{
esl_handle_t handle = {{0}};
int done = 0;
esl_status_t status;
time_t exp = 0;
esl_attach_handle(&handle, client_sock, addr);
esl_log(ESL_LOG_INFO, "Connected! %d\n", handle.sock);
esl_filter(&handle, "unique-id", esl_event_get_header(handle.info_event,
"caller-unique-id"));
esl_events(&handle, ESL_EVENT_TYPE_PLAIN, "SESSION_HEARTBEAT CHANNEL_ANSWER
CHANNEL_ORIGINATE CHANNEL_PROGRESS CHANNEL_HANGUP "
"CHANNEL_BRIDGE CHANNEL_UNBRIDGE CHANNEL_OUTGOING CHANNEL_EXECUTE
CHANNEL_EXECUTE_COMPLETE DTMF CUSTOM conference::maintenance");
esl_send_recv(&handle, "linger");
esl_execute(&handle, "set", "recfile=/usr/local/rec/test.gsm", NULL);
esl_send_recv(&handle,"api sofia status profile internal reg
6100010030100\n\n");
char *b=esl_event_get_body(handle.last_sr_event);
printf("%s\r\n", b);
esl_send_recv(&handle,"api sofia status profile internal reg
6100010030200\n\n");
b=esl_event_get_body(handle.last_sr_event);
printf("%s\r\n", b);
while((status = esl_recv_timed(&handle, 1000)) != ESL_FAIL) {
if (done) {
if (time(NULL) >= exp) {
break;
}
} else if (status == ESL_SUCCESS) {
const char *type = esl_event_get_header(handle.last_event, "content-type");
if (type && !strcasecmp(type, "text/disconnect-notice")) {
const char *dispo = esl_event_get_header(handle.last_event,
"content-disposition");
esl_log(ESL_LOG_INFO, "Got a disconnection notice dispostion: [%s]\n",
dispo ? dispo : "");
if (dispo && !strcmp(dispo, "linger")) {
done = 1;
esl_log(ESL_LOG_INFO, "Waiting 5 seconds for any remaining events.\n");
exp = time(NULL) + 5;
}
}
}
}
esl_log(ESL_LOG_INFO, "Disconnected! %d\n", handle.sock);
esl_disconnect(&handle);
}
int main(){
esl_global_set_default_logger(0);
esl_listen_threaded("localhost", 8084, mycallback, 100000);
return 0;
}
I made 10 calls from a rtmp client to a eyeBeam client. sometimes there was
no lake, but sometimes there was still a leak .
The valgrind full log as follows
valgrind --tool=memcheck --leak-check=full ./wtdpl
==11071== Memcheck, a memory error detector
==11071== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==11071== Using Valgrind-3.9.0 and LibVEX; rerun with -h for copyright info
==11071== Command: ./wtdpl
==11071==
Registrations:
=================================================================================================
Call-ID: Y2VkZjY4MzY4OTFmZDMxZjJkY2FkNmQ1OWZkNjg5YTU.
User: 6100010030100 at 192.168.1.200
Contact: "abc"
<sip:6100010030100 at 192.168.1.6:2964;rinstance=239a1e88dd886ca0>
Agent: eyeBeam release 1011d stamp 40820
Status: Registered(UDP)(unknown) EXP(2013-11-16 12:31:53)
EXPSECS(2321)
Host: test.u1u2u3.com
IP: 192.168.1.6
Port: 2964
Auth-User: 6100010030100
Auth-Realm: 192.168.1.200
MWI-Account: 6100010030100 at 192.168.1.200
Total items returned: 1
=================================================================================================
Registrations:
=================================================================================================
Total items returned: 0
=================================================================================================
Registrations:
=================================================================================================
Call-ID: Y2VkZjY4MzY4OTFmZDMxZjJkY2FkNmQ1OWZkNjg5YTU.
User: 6100010030100 at 192.168.1.200
Contact: "abc"
<sip:6100010030100 at 192.168.1.6:2964;rinstance=239a1e88dd886ca0>
Agent: eyeBeam release 1011d stamp 40820
Status: Registered(UDP)(unknown) EXP(2013-11-16 12:31:53)
EXPSECS(2320)
Host: test.u1u2u3.com
IP: 192.168.1.6
Port: 2964
Auth-User: 6100010030100
Auth-Realm: 192.168.1.200
MWI-Account: 6100010030100 at 192.168.1.200
Total items returned: 1
=================================================================================================
Registrations:
=================================================================================================
Total items returned: 0
=================================================================================================
Registrations:
=================================================================================================
Call-ID: Y2VkZjY4MzY4OTFmZDMxZjJkY2FkNmQ1OWZkNjg5YTU.
User: 6100010030100 at 192.168.1.200
Contact: "abc"
<sip:6100010030100 at 192.168.1.6:2964;rinstance=239a1e88dd886ca0>
Agent: eyeBeam release 1011d stamp 40820
Status: Registered(UDP)(unknown) EXP(2013-11-16 12:31:53)
EXPSECS(2319)
Host: test.u1u2u3.com
IP: 192.168.1.6
Port: 2964
Auth-User: 6100010030100
Auth-Realm: 192.168.1.200
MWI-Account: 6100010030100 at 192.168.1.200
Total items returned: 1
=================================================================================================
Registrations:
=================================================================================================
Total items returned: 0
=================================================================================================
Registrations:
=================================================================================================
Call-ID: Y2VkZjY4MzY4OTFmZDMxZjJkY2FkNmQ1OWZkNjg5YTU.
User: 6100010030100 at 192.168.1.200
Contact: "abc"
<sip:6100010030100 at 192.168.1.6:2964;rinstance=239a1e88dd886ca0>
Agent: eyeBeam release 1011d stamp 40820
Status: Registered(UDP)(unknown) EXP(2013-11-16 12:31:53)
EXPSECS(2318)
Host: test.u1u2u3.com
IP: 192.168.1.6
Port: 2964
Auth-User: 6100010030100
Auth-Realm: 192.168.1.200
MWI-Account: 6100010030100 at 192.168.1.200
Total items returned: 1
=================================================================================================
Registrations:
=================================================================================================
Total items returned: 0
=================================================================================================
Registrations:
=================================================================================================
Call-ID: Y2VkZjY4MzY4OTFmZDMxZjJkY2FkNmQ1OWZkNjg5YTU.
User: 6100010030100 at 192.168.1.200
Contact: "abc"
<sip:6100010030100 at 192.168.1.6:2964;rinstance=239a1e88dd886ca0>
Agent: eyeBeam release 1011d stamp 40820
Status: Registered(UDP)(unknown) EXP(2013-11-16 12:31:53)
EXPSECS(2316)
Host: test.u1u2u3.com
IP: 192.168.1.6
Port: 2964
Auth-User: 6100010030100
Auth-Realm: 192.168.1.200
MWI-Account: 6100010030100 at 192.168.1.200
Total items returned: 1
=================================================================================================
Registrations:
=================================================================================================
Total items returned: 0
=================================================================================================
Registrations:
=================================================================================================
Call-ID: Y2VkZjY4MzY4OTFmZDMxZjJkY2FkNmQ1OWZkNjg5YTU.
User: 6100010030100 at 192.168.1.200
Contact: "abc"
<sip:6100010030100 at 192.168.1.6:2964;rinstance=239a1e88dd886ca0>
Agent: eyeBeam release 1011d stamp 40820
Status: Registered(UDP)(unknown) EXP(2013-11-16 12:31:53)
EXPSECS(2315)
Host: test.u1u2u3.com
IP: 192.168.1.6
Port: 2964
Auth-User: 6100010030100
Auth-Realm: 192.168.1.200
MWI-Account: 6100010030100 at 192.168.1.200
Total items returned: 1
=================================================================================================
Registrations:
=================================================================================================
Total items returned: 0
=================================================================================================
Registrations:
=================================================================================================
Call-ID: Y2VkZjY4MzY4OTFmZDMxZjJkY2FkNmQ1OWZkNjg5YTU.
User: 6100010030100 at 192.168.1.200
Contact: "abc"
<sip:6100010030100 at 192.168.1.6:2964;rinstance=239a1e88dd886ca0>
Agent: eyeBeam release 1011d stamp 40820
Status: Registered(UDP)(unknown) EXP(2013-11-16 12:31:53)
EXPSECS(2314)
Host: test.u1u2u3.com
IP: 192.168.1.6
Port: 2964
Auth-User: 6100010030100
Auth-Realm: 192.168.1.200
MWI-Account: 6100010030100 at 192.168.1.200
Total items returned: 1
=================================================================================================
Registrations:
=================================================================================================
Total items returned: 0
=================================================================================================
Registrations:
=================================================================================================
Call-ID: Y2VkZjY4MzY4OTFmZDMxZjJkY2FkNmQ1OWZkNjg5YTU.
User: 6100010030100 at 192.168.1.200
Contact: "abc"
<sip:6100010030100 at 192.168.1.6:2964;rinstance=239a1e88dd886ca0>
Agent: eyeBeam release 1011d stamp 40820
Status: Registered(UDP)(unknown) EXP(2013-11-16 12:31:53)
EXPSECS(2313)
Host: test.u1u2u3.com
IP: 192.168.1.6
Port: 2964
Auth-User: 6100010030100
Auth-Realm: 192.168.1.200
MWI-Account: 6100010030100 at 192.168.1.200
Total items returned: 1
=================================================================================================
Registrations:
=================================================================================================
Total items returned: 0
=================================================================================================
Registrations:
=================================================================================================
Call-ID: Y2VkZjY4MzY4OTFmZDMxZjJkY2FkNmQ1OWZkNjg5YTU.
User: 6100010030100 at 192.168.1.200
Contact: "abc"
<sip:6100010030100 at 192.168.1.6:2964;rinstance=239a1e88dd886ca0>
Agent: eyeBeam release 1011d stamp 40820
Status: Registered(UDP)(unknown) EXP(2013-11-16 12:31:53)
EXPSECS(2312)
Host: test.u1u2u3.com
IP: 192.168.1.6
Port: 2964
Auth-User: 6100010030100
Auth-Realm: 192.168.1.200
MWI-Account: 6100010030100 at 192.168.1.200
Total items returned: 1
=================================================================================================
Registrations:
=================================================================================================
Total items returned: 0
=================================================================================================
Registrations:
=================================================================================================
Call-ID: Y2VkZjY4MzY4OTFmZDMxZjJkY2FkNmQ1OWZkNjg5YTU.
User: 6100010030100 at 192.168.1.200
Contact: "abc"
<sip:6100010030100 at 192.168.1.6:2964;rinstance=239a1e88dd886ca0>
Agent: eyeBeam release 1011d stamp 40820
Status: Registered(UDP)(unknown) EXP(2013-11-16 12:31:53)
EXPSECS(2310)
Host: test.u1u2u3.com
IP: 192.168.1.6
Port: 2964
Auth-User: 6100010030100
Auth-Realm: 192.168.1.200
MWI-Account: 6100010030100 at 192.168.1.200
Total items returned: 1
=================================================================================================
Registrations:
=================================================================================================
Total items returned: 0
=================================================================================================
^C==11071==
==11071== HEAP SUMMARY:
==11071== in use at exit: 26,309,844 bytes in 2,639 blocks
==11071== total heap usage: 14,394 allocs, 11,755 frees, 66,216,694 bytes
allocated
==11071==
==11071== 1,152 bytes in 4 blocks are possibly lost in loss record 41 of 51
==11071== at 0x4A0580F: calloc (vg_replace_malloc.c:618)
==11071== by 0x34C9E11892: _dl_allocate_tls (in /lib64/ld-2.12.so)
==11071== by 0x34CAA07068: pthread_create@@GLIBC_2.2.5 (in /lib64/
libpthread-2.12.so)
==11071== by 0x405CFC: esl_thread_create_detached_ex
(esl_threadmutex.c:118)
==11071== by 0x402410: esl_listen_threaded (esl.c:771)
==11071== by 0x401CD6: main (wtdpl.cpp:58)
==11071==
==11071== 4,008 (88 direct, 3,920 indirect) bytes in 1 blocks are
definitely lost in loss record 43 of 51
==11071== at 0x4A06AAA: malloc (vg_replace_malloc.c:291)
==11071== by 0x4056B1: esl_event_create_subclass (esl_event.c:179)
==11071== by 0x402B33: esl_recv_event (esl.c:1245)
==11071== by 0x4033E5: esl_send_recv_timed (esl.c:1517)
==11071== by 0x401AE1: mycallback(int, int, sockaddr_in*) (wtdpl.cpp:28)
==11071== by 0x40209F: client_thread (esl.c:654)
==11071== by 0x405D37: thread_launch (esl_threadmutex.c:74)
==11071== by 0x34CAA07850: start_thread (in /lib64/libpthread-2.12.so)
==11071== by 0x34CA6E890C: clone (in /lib64/libc-2.12.so)
==11071==
==11071== LEAK SUMMARY:
==11071== definitely lost: 88 bytes in 1 blocks
==11071== indirectly lost: 3,920 bytes in 10 blocks
==11071== possibly lost: 1,152 bytes in 4 blocks
==11071== still reachable: 26,304,684 bytes in 2,624 blocks
==11071== suppressed: 0 bytes in 0 blocks
==11071== Reachable blocks (those to which a pointer was found) are not
shown.
==11071== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==11071==
==11071== For counts of detected and suppressed errors, rerun with: -v
==11071== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 4 from 4)
Looking forward to your help
2013/11/14 Anthony Minessale <anthony.minessale at gmail.com>
> I dug into ESL for a while.
>
> I ran into some issues that I corrected but the only thing that I can see
> that would show up as a memory leak looking like what you show there is if
> you are doing a forking server and you are calling fork after you create or
> attach the handle.
>
> Make sure you fork at the beginning or you will duplicate the handle and
> all the memory its using and it will never be destroyed.
>
> If you do "make" in esl dir you will have a default app "testserver_fork"
> you can run and point to it with socket app to port 8040.
> Running it on valgrind with full leak detection gives me.
>
>
> ==11807== Memcheck, a memory error detector
> ==11807== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
> ==11807== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
> ==11807== Command: ./testserver
> ==11807== Parent PID: 28177
> ==11807==
> ==11820==
> ==11820== HEAP SUMMARY:
> ==11820== in use at exit: 0 bytes in 0 blocks
> ==11820== total heap usage: 5,555 allocs, 5,555 frees, 7,293,931 bytes
> allocated
> ==11820==
> ==11820== All heap blocks were freed -- no leaks are possible
> ==11820==
> ==11820== For counts of detected and suppressed errors, rerun with: -v
> ==11820== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 4 from 4)
> ==11807==
> ==11807== HEAP SUMMARY:
> ==11807== in use at exit: 0 bytes in 0 blocks
> ==11807== total heap usage: 0 allocs, 0 frees, 0 bytes allocated
> ==11807==
> ==11807== All heap blocks were freed -- no leaks are possible
> ==11807==
> ==11807== For counts of detected and suppressed errors, rerun with: -v
> ==11807== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 4 from 4)
>
>
>
>
>
> On Wed, Nov 13, 2013 at 7:21 AM, Dmitry Sytchev <kbdfck at gmail.com> wrote:
>
>> Please create a JIRA ticket with your information!
>>
>>
>> 2013/11/13 gggyniidt <gggyniidt at gmail.com>
>>
>>> Thanks for Dmitry and Anthony's help, But my problem is not like Dmitry
>>> said.
>>>
>>> My program is a esl client. The leak is from esl client, not freeswitch.
>>>
>>> I add the Anthony's patch code into esl.c,but it doesn't work. Each call
>>> will cause the 20-40KB of memory growth.
>>>
>>> I make 3 calls, the valgrind outputs like this
>>>
>>> ==14459== 90,642 (264 direct, 90,378 indirect) bytes in 3 blocks are
>>> definitely lost in loss record 403 of 408
>>> ==14459== at 0x4A06AAA: malloc (vg_replace_malloc.c:291)
>>>
>>> ==14459== by 0x414524: esl_event_create_subclass (esl_event.c:179)
>>>
>>> ==14459== by 0x412054: esl_recv_event (esl.c:1245)
>>>
>>> ==14459== by 0x411C38: esl_recv_event_timed (esl.c:1137)
>>>
>>> ==14459== by 0x412E22: esl_send_recv_timed (esl.c:1521)
>>>
>>> ==14459== by 0x410C06: esl_execute (esl.c:555)
>>>
>>> ==14459== by 0x4087A8: CRBT::Route() (CRBT.cpp:42)
>>>
>>> ==14459== by 0x409FE6: DPL::route() (DPL.cpp:286)
>>>
>>> ==14459== by 0x40FBE0: mycallback(int, int, sockaddr_in*)
>>> (listener.cpp:47)
>>> ==14459== by 0x410FED: client_thread (esl.c:654)
>>>
>>> ==14459== by 0x41A334: thread_launch (esl_threadmutex.c:74)
>>>
>>> ==14459== by 0x34CAA07850: start_thread (in /lib64/libpthread-2.12.so)
>>>
>>> ==14459==
>>>
>>> ==14459== LEAK SUMMARY:
>>> ==14459== definitely lost: 264 bytes in 3 blocks
>>> ==14459== indirectly lost: 90,378 bytes in 228 blocks
>>> ==14459== possibly lost: 550,082 bytes in 51 blocks
>>> ==14459== still reachable: 832,938 bytes in 840 blocks
>>> ==14459== suppressed: 0 bytes in 0 blocks
>>> ==14459== Reachable blocks (those to which a pointer was found) are not
>>> shown.
>>> ==14459== To see them, rerun with: --leak-check=full
>>> --show-leak-kinds=all
>>>
>>> I checked my code and no leak found. The leak may be in esl.c.
>>>
>>>
>>>
>>> 2013/11/13 Anthony Minessale <anthony.minessale at gmail.com>
>>>
>>>> You are showing valgrind traces of ESL client code but now you are
>>>> saying your FS process has memory growth not the ESL app?
>>>> You should be doing this on jira and you should have more details about
>>>> the env.
>>>> This mailing list is not for bug reports.
>>>>
>>>> I did push a patch for what I think might be the problem you showed on
>>>> esl lib.
>>>>
>>>> FS can use up to a gig of memory without giving it back. To make sure
>>>> you have a real memory leak you need to have it never peak but continue to
>>>> grow.
>>>>
>>>>
>>>>
>>>>
>>>> On Tue, Nov 12, 2013 at 11:17 AM, Dmitry Sytchev <kbdfck at gmail.com>wrote:
>>>>
>>>>> I can confirm significant Freeswitch process memory growth over time
>>>>> when ESL is used intensively, from 100MB in normal case to 700-900MB after
>>>>> few hours of high esl load, though I have no valgrind check results, as I
>>>>> can't do tests at production server.
>>>>> With less than 300 channels in peak we have 900MB FS process size,
>>>>> never shrinking back :(
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2013/11/12 gggyniidt <gggyniidt at gmail.com>
>>>>>
>>>>>> Hello
>>>>>> Using ESL found a memory leak problem
>>>>>> valgrind --tool=memcheck --leak-check=full ./myprog
>>>>>> output as follows
>>>>>>
>>>>>> ==6872== 29,986 (88 direct, 29,898 indirect) bytes in 1 blocks are
>>>>>> definitely lost in loss record 401 of 408
>>>>>> ==6872== at 0x4A06AAA: malloc (vg_replace_malloc.c:291)
>>>>>> ==6872== by 0x413941: esl_event_create_subclass (esl_event.c:179)
>>>>>>
>>>>>> ==6872== by 0x410DC3: esl_recv_event (esl.c:1245)
>>>>>>
>>>>>> ==6872== by 0x411675: esl_send_recv_timed (esl.c:1517)
>>>>>>
>>>>>> ==6872== by 0x411C1B: esl_execute (esl.c:555)
>>>>>>
>>>>>> ==6872== by 0x408758: CRBT::Route() (CRBT.cpp:42)
>>>>>>
>>>>>> ==6872== by 0x40A02F: DPL::route() (DPL.cpp:234)
>>>>>>
>>>>>> ==6872== by 0x40FCA8: mycallback(int, int, sockaddr_in*)
>>>>>> (listener.cpp:47)
>>>>>> ==6872== by 0x41032F: client_thread (esl.c:654)
>>>>>>
>>>>>> ==6872== by 0x413FC7: thread_launch (esl_threadmutex.c:74)
>>>>>>
>>>>>> ==6872== by 0x34CAA07850: start_thread (in /lib64/
>>>>>> libpthread-2.12.so)
>>>>>>
>>>>>> ==6872== by 0x34CA6E890C: clone (in /lib64/libc-2.12.so)
>>>>>>
>>>>>>
>>>>>> ==6872== LEAK SUMMARY:
>>>>>> ==6872== definitely lost: 88 bytes in 1 blocks
>>>>>> ==6872== indirectly lost: 29,898 bytes in 76 blocks
>>>>>> ==6872== possibly lost: 550,082 bytes in 51 blocks
>>>>>> ==6872== still reachable: 832,937 bytes in 840 blocks
>>>>>> ==6872== suppressed: 0 bytes in 0 blocks
>>>>>> ==6872== Reachable blocks (those to which a pointer was found) are
>>>>>> not shown.
>>>>>> ==6872== To see them, rerun with: --leak-check=full
>>>>>> --show-leak-kinds=all
>>>>>>
>>>>>>
>>>>>>
>>>>>> Part of my code is as follows
>>>>>>
>>>>>> bool CRBT::Route() {
>>>>>> string data = "ringback=file_string://";
>>>>>> data.append(FileName);
>>>>>>
>>>>>> esl_execute(handle, "set", data.c_str(), NULL);
>>>>>> return true;
>>>>>> }
>>>>>>
>>>>>> The OS version is Centos x86_64 6.3
>>>>>> The Freeswitch version is 1.2.13
>>>>>>
>>>>>> Thanks for your help.
>>>>>>
>>>>>>
>>>>>> _________________________________________________________________________
>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>> consulting at freeswitch.org
>>>>>> http://www.freeswitchsolutions.com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Official FreeSWITCH Sites
>>>>>> http://www.freeswitch.org
>>>>>> http://wiki.freeswitch.org
>>>>>> http://www.cluecon.com
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>> UNSUBSCRIBE:
>>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>>
>>>>> Dmitry Sytchev,
>>>>> IT Engineer
>>>>>
>>>>>
>>>>> _________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org
>>>>> http://www.freeswitchsolutions.com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org
>>>>> http://wiki.freeswitch.org
>>>>> http://www.cluecon.com
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:
>>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Anthony Minessale II
>>>>
>>>> FreeSWITCH http://www.freeswitch.org/
>>>> ClueCon http://www.cluecon.com/
>>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>>
>>>> AIM: anthm
>>>> MSN:anthony_minessale at hotmail.com
>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>>> IRC: irc.freenode.net #freeswitch
>>>>
>>>> FreeSWITCH Developer Conference
>>>> sip:888 at conference.freeswitch.org
>>>> googletalk:conf+888 at conference.freeswitch.org
>>>> pstn:+19193869900
>>>>
>>>>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org
>>>> http://www.freeswitchsolutions.com
>>>>
>>>>
>>>>
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org
>>>> http://wiki.freeswitch.org
>>>> http://www.cluecon.com
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:
>>>> http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>>
>>>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org
>>> http://www.freeswitchsolutions.com
>>>
>>>
>>>
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org
>>> http://wiki.freeswitch.org
>>> http://www.cluecon.com
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>>
>>>
>>
>>
>> --
>> Best regards,
>>
>> Dmitry Sytchev,
>> IT Engineer
>>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org
>> http://www.freeswitchsolutions.com
>>
>>
>>
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org
>> http://wiki.freeswitch.org
>> http://www.cluecon.com
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>>
>>
>
>
> --
> Anthony Minessale II
>
> FreeSWITCH http://www.freeswitch.org/
> ClueCon http://www.cluecon.com/
> Twitter: http://twitter.com/FreeSWITCH_wire
>
> AIM: anthm
> MSN:anthony_minessale at hotmail.com
> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
> IRC: irc.freenode.net #freeswitch
>
> FreeSWITCH Developer Conference
> sip:888 at conference.freeswitch.org
> googletalk:conf+888 at conference.freeswitch.org
> pstn:+19193869900
>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
>
>
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20131116/364f4434/attachment-0001.html
Join us at ClueCon 2013 Aug 6-8, 2013
More information about the FreeSWITCH-users
mailing list