[Freeswitch-users] registration fails after several hours - FS problem?
Anthony Minessale
anthony.minessale at gmail.com
Sat Oct 30 12:34:23 PDT 2010
ok so there must be a problem in MAC, have you compared the results on
win32 or linux?
some other people are reporting mem issues on mac lets find it.
http://valgrind.org/downloads/valgrind-3.6.0.tar.bz2
build and install valgrind
run fs in vg mode
valgrind --tool=memcheck --log-file=vg.log --leak-check=full
--leak-resolution=high --show-reachable=yes /path/to/lt-freeswitch -vg
do something that eats mem
exit and get me the vg.log
On Sat, Oct 30, 2010 at 11:37 AM, Mario G <mario_fs at mgtech.com> wrote:
> Setting sofia loglevel to 9 made Terminal app memory go from 20meg to eventually using all memory and locking the machine up. I spent all night trying to get a good trace. I need to get one when the failure occurs early before the lockup. Will be working all w/e on this. Thanks for you help.
>
> I got this with only 1 gateway defined but did not catch it early. Notice how there is no SIP trace activity:
> 2010-10-30 09:30:15.046900 [WARNING] sofia_reg.c:387 mvvyl Failed Registration, setting retry to 15 seconds.
> nta: timer K fired, terminate NOTIFY (3892042)
> outgoing_reclaim_all(0x0, 0x0, 0x102309cb0)
> nta_outgoing_timer: 0/0 resent, 0/0 tout, 1/1 term, 1/1 free
> nta: timer not set
> 2010-10-30 09:30:31.926382 [NOTICE] sofia_reg.c:342 Registering mvvyl
> nua: nua_register: entering
> nua(0x1003c3750): sent signal r_register
> nua(0x1003c3750): recv signal r_register
> 2010-10-30 09:30:47.685212 [WARNING] sofia_reg.c:387 mvvyl Failed Registration, setting retry to 15 seconds.
> 2010-10-30 09:31:03.478941 [NOTICE] sofia_reg.c:342 Registering mvvyl
> nua: nua_register: entering
> nua(0x1003c3750): sent signal r_register
> nua(0x1003c3750): recv signal r_register
> 2010-10-30 09:31:19.219346 [WARNING] sofia_reg.c:387 mvvyl Failed Registration, setting retry to 15 seconds.
> 2010-10-30 09:31:36.208448 [NOTICE] sofia_reg.c:342 Registering mvvyl
> nua: nua_register: entering
> nua(0x1003c3750): sent signal r_register
> nua(0x1003c3750): recv signal r_register
>
> Mario
>
>>
>> On Oct 29, 2010, at 5:43 PM, Anthony Minessale wrote:
>>
>>> also raise loglevel to debug
>>> console loglevel debug
>>>
>>>
>>> On Fri, Oct 29, 2010 at 7:29 PM, Mario G <mario_fs at mgtech.com> wrote:
>>>> Will so, BTW, I mentioned below one is dns and another uses IP, I tested that theory using an IP does not help. I also updated to todays git version and no help there. Will post when the trace is done.
>>>>
>>>> On Oct 29, 2010, at 5:17 PM, Anthony Minessale wrote:
>>>>
>>>>> can you repeat that trace with sofia debug on
>>>>> sofia loglevel all 9
>>>>>
>>>>> Are you doing DNS by any chance in the gateway "proxy" param?
>>>>> you could try filling in the register-proxy param in your gateway to
>>>>> sip:<ip> <-- not dns but ip that dns resolves to
>>>>> <param name="register-proxy" value="sip:1.2.3.4"/>
>>>>>
>>>>> I'm just guessing but Its possible some bad dns query could be throwing FS off.
>>>>> so this test would force all the packets to the exact IP of your host
>>>>> instead of looking it up.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Oct 29, 2010 at 6:30 PM, Mario G <mario_fs at mgtech.com> wrote:
>>>>>> Oh Boy: Not only is everything set to the static route but I turned
>>>>>> off/disconnected the dynamic DSL line so I only had 1 static line to the
>>>>>> router. The second wan is set in the router off. Even turned off router uPnP
>>>>>> even though I am using -nonat. Guess what.... I still have the problem. Look
>>>>>> like when FS says is going to retry it actually does not.
>>>>>> Here is a short one: http://pastebin.freeswitch.org/14359 - I caught one
>>>>>> right after I started FS, failure occurred in minutes (lucky). Look at the
>>>>>> bottom of the trace, you see SIP trace activity and then when it fails no
>>>>>> SIP trace activity. Could this possibly be a FS bug? (I am a mainframe
>>>>>> assembler systems programmer and I might think so if there were error retry
>>>>>> messages but nothing showing in one of my traces...)
>>>>>> Notes:
>>>>>> idone is gateway 1
>>>>>> idtwo is gateway 2 I had to trace both because it was impossible to figure
>>>>>> out which one would fail first. Ran several times but kept missing the right
>>>>>> one.
>>>>>> I use a url for one gateway and ip for another but it makes no difference
>>>>>> since both eventually fail.
>>>>>> 10. is local lan
>>>>>> 210. is external ip
>>>>>> 216. is itsp
>>>>>>
>>>>>> Here is a longer one from earlier http://pastebin.freeswitch.org/14357
>>>>>> Notes:
>>>>>> A call was received and hung up for idtwo - beginning of trace
>>>>>> 11 minutes later idtwo failed - see last line of trace
>>>>>>
>>>>>> Thank you very much!
>>>>>> Mario
>>>>>>
>>>>>> On Oct 29, 2010, at 12:11 PM, Anthony Minessale wrote:
>>>>>>
>>>>>> stun-enabled must be true in your profile XML to see what you pasted.
>>>>>>
>>>>>> Get me a sip trace of this from when it works until when it fails
>>>>>>
>>>>>> only enable the sip trace on the profile with the gateway to reduce traffic
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 29, 2010 at 1:56 PM, Mario G <mario_fs at mgtech.com> wrote:
>>>>>>
>>>>>> Oh my... looks it was not NAT after all? Please help! I changed to all
>>>>>> profiles to static per instructions below and still have the problem:
>>>>>>
>>>>>> 2010-10-29 11:15:18.536446 [NOTICE] sofia_reg.c:342 Registering uuid1
>>>>>>
>>>>>> 2010-10-29 11:15:34.313150 [WARNING] sofia_reg.c:387 uuid1 Failed
>>>>>> Registration, setting retry to 15 seconds.
>>>>>>
>>>>>> sofia global siptrace on did not show any activity for this gateway in or
>>>>>> out, others were fine but eventually fail. I setup static:
>>>>>>
>>>>>> 1. set the params ext-sip-ip and ext-rtp-ip to my external static IP
>>>>>>
>>>>>> 2. map the sip ports (5060-5080) and all of the rtp ports (16384-32767) to
>>>>>> FS lan addr.
>>>>>>
>>>>>> 3. set sip-ip and rtp-ip to the lan addr of FS
>>>>>>
>>>>>> 4. start FS with -nonat
>>>>>>
>>>>>> I don't know what to try next. BTW, the sofia status for the profiles shows
>>>>>> stun enabled but I did not set it up anywhere:
>>>>>>
>>>>>> Name uuid1
>>>>>>
>>>>>> Domain Name N/A
>>>>>>
>>>>>> Auto-NAT false
>>>>>>
>>>>>> DBName sofia_reg_mvvyl
>>>>>>
>>>>>> Pres Hosts
>>>>>>
>>>>>> Dialplan XML
>>>>>>
>>>>>> Context public
>>>>>>
>>>>>> Challenge Realm auto_to
>>>>>>
>>>>>> RTP-IP 10.x.x.20
>>>>>>
>>>>>> Ext-RTP-IP 210.x.x.100
>>>>>>
>>>>>> SIP-IP 10.x.x.20
>>>>>>
>>>>>> Ext-SIP-IP 210.x.x.100
>>>>>>
>>>>>> URL sip:mod_sofia at 210.x.x.100:5068
>>>>>>
>>>>>> BIND-URL sip:mod_sofia at 210.x.x.100:5068;maddr=10.x.x.20
>>>>>>
>>>>>> HOLD-MUSIC local_stream://moh
>>>>>>
>>>>>> OUTBOUND-PROXY N/A
>>>>>>
>>>>>> CODECS IN G7221 at 32000h,G7221 at 16000h,G722,PCMU,PCMA,GSM
>>>>>>
>>>>>> CODECS OUT PCMU,PCMA,GSM
>>>>>>
>>>>>> TEL-EVENT 101
>>>>>>
>>>>>> DTMF-MODE rfc2833
>>>>>>
>>>>>> CNG 13
>>>>>>
>>>>>> SESSION-TO 0
>>>>>>
>>>>>> MAX-DIALOG 0
>>>>>>
>>>>>> NOMEDIA false
>>>>>>
>>>>>> LATE-NEG false
>>>>>>
>>>>>> PROXY-MEDIA false
>>>>>>
>>>>>> AGGRESSIVENAT false
>>>>>>
>>>>>> STUN-ENABLED true
>>>>>>
>>>>>> STUN-AUTO-DISABLE false
>>>>>>
>>>>>> CALLS-IN 0
>>>>>>
>>>>>> FAILED-CALLS-IN 0
>>>>>>
>>>>>> CALLS-OUT 0
>>>>>>
>>>>>> FAILED-CALLS-OUT 0
>>>>>>
>>>>>>
>>>>>> On Oct 27, 2010, at 10:04 AM, Anthony Minessale wrote:
>>>>>>
>>>>>> if you map it or not, a scanner would penetrate it.
>>>>>>
>>>>>> There are lot of sip scanners out there now, you just need to beware of
>>>>>> them.
>>>>>>
>>>>>>
>>>>>> On Wed, Oct 27, 2010 at 11:50 AM, Mario G <mario_fs at mgtech.com> wrote:
>>>>>>
>>>>>> Thanks so much! I am sure many others will find this info invaluable. I will
>>>>>> try the static route again but have one question: When I started with FS I
>>>>>> found a "sip scanner" in FS and someone on this group said not to use port
>>>>>> mapping since it was a security risk. Is that true?
>>>>>>
>>>>>> On Oct 27, 2010, at 9:10 AM, Anthony Minessale wrote:
>>>>>>
>>>>>> you are completely guessing at things.
>>>>>>
>>>>>> I want you to understand that the only reason you are having problems
>>>>>>
>>>>>> with this is because you don't understand how it works enough to know
>>>>>>
>>>>>> what you are doing 100%
>>>>>>
>>>>>> Its a given that the pnp stuff is only for your dynamic IP.
>>>>>>
>>>>>> aggressive-nat-detection and sip-force-expires are all related to
>>>>>>
>>>>>> inbound calls when the things who are registering to you may be behind
>>>>>>
>>>>>> nat.
>>>>>>
>>>>>> You need to learn the difference between which nat tools are
>>>>>>
>>>>>> *) designed for your FS to run behind nat
>>>>>>
>>>>>> *) designed for FS to run public and accept connections from devices behind
>>>>>> nat.
>>>>>>
>>>>>> If you have a static IP, you don't need the pnp stuff so -nonat is fine
>>>>>>
>>>>>> What you need to do is set
>>>>>>
>>>>>> 1) set the params ext-sip-ip and ext-rtp-ip to your external static IP
>>>>>>
>>>>>> 2) map the sip ports and all of the rtp ports from your static IP to FS lan
>>>>>> addr
>>>>>>
>>>>>> 3) set sip-ip and rtp-ip to the lan addr you forwarded through.
>>>>>>
>>>>>>
>>>>>> If you don't do this: your outbound registration will use NAT to your
>>>>>>
>>>>>> provider and if there is no activity for the expire time on your NAT
>>>>>>
>>>>>> mapping the reverse port mapping from your provider back to you is
>>>>>>
>>>>>> lost. This is why you set your register expires to a very low number,
>>>>>>
>>>>>> (you need to make sure the provider does not turn the expires back up
>>>>>>
>>>>>> in the reply because it will beat your choice *see sip trace) if this
>>>>>>
>>>>>> is the case then you need the "ping" option set to 30, to continuously
>>>>>>
>>>>>> send an options to your provider.
>>>>>>
>>>>>> The static mapping is obviously the better, easier and more reliable
>>>>>> solution.
>>>>>>
>>>>>> So I want you to understand that the only way to keep a nat mapped
>>>>>>
>>>>>> port alive is to continuously send traffic, all the other methods that
>>>>>>
>>>>>> you are mentioning are to detect that phones registered to your are
>>>>>>
>>>>>> behind nat, I gave you that force-expires option before because your
>>>>>>
>>>>>> trace was full of inbound reg so I thought that is what you wanted
>>>>>>
>>>>>> help with.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Oct 27, 2010 at 10:43 AM, Mario G <mario_fs at mgtech.com> wrote:
>>>>>>
>>>>>> I should mention that I did not have this problem with an SPA9000 PBX
>>>>>>
>>>>>> (asterisk based) for over two years so FS may be pickier about upnp and/or
>>>>>>
>>>>>> nat, or just better at it exposing a problem in the router.
>>>>>>
>>>>>> I made different changes to the gateways to test different things. One
>>>>>>
>>>>>> failed after 17 hours, the other two stayed up. What did not work:
>>>>>>
>>>>>> added <variable name="sip-force-expires" value="30"/> to the directory
>>>>>>
>>>>>> entries as suggested.
>>>>>>
>>>>>> set the gateway expire times to 30 seconds.
>>>>>>
>>>>>> What worked (could be coincidental) for the two gateways that stayed up:
>>>>>>
>>>>>> I Added <param name="aggressive-nat-detection" value="true"/>
>>>>>>
>>>>>> I originally setup FS to use the static ip by setting external sip/rtp to
>>>>>>
>>>>>> just the static ip (no autonat:) and ran with -nonat but I could not get
>>>>>>
>>>>>> incoming calls. The only way it worked was to use autonat:1.2.3.4. The
>>>>>>
>>>>>> router has 1 static public address and 1 dynamic external IP, this is the
>>>>>>
>>>>>> root of the problem, upnp only tells FS about the dynamic ip Will keep this
>>>>>>
>>>>>> thread up-to-date for anyone who may be in the same boat someday. Thanks
>>>>>>
>>>>>> again for looking at the trace.
>>>>>>
>>>>>> Mario
>>>>>>
>>>>>> You should be setting the req freq to a low number on the outbound gateways
>>>>>>
>>>>>> The examples you showed had a series of inbound reg
>>>>>>
>>>>>> also set expire-seconds to 30 in your gateway xml
>>>>>>
>>>>>>
>>>>>> The problem is if you are not constantly sending traffic to the box
>>>>>>
>>>>>> the nat mapping will go away.
>>>>>>
>>>>>> If you are in production you should be using a static ip with a static
>>>>>>
>>>>>> mapping, any trouble you are having is your own fault for playing with
>>>>>>
>>>>>> fire. The best we can do is tell you how to keep it contained.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 26, 2010 at 12:34 PM, Mario G <mario_fs at mgtech.com> wrote:
>>>>>>
>>>>>> I made the change. I had no idea the settings for the inside phones effected
>>>>>>
>>>>>> nat for the outside sip accounts. I was looking into aggressive-nat-
>>>>>>
>>>>>> detection since the internal profile status always shows the right external
>>>>>>
>>>>>> static IP but the nat_ap status always shows the dynamic ip. Crossing
>>>>>>
>>>>>> fingers/etc since this problem is 85% of time (weeks!) into FS changeover.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Mario
>>>>>>
>>>>>> On Oct 26, 2010, at 10:15 AM, Anthony Minessale wrote:
>>>>>>
>>>>>> add
>>>>>>
>>>>>> <variable name="sip-force-expires" value="30"/>
>>>>>>
>>>>>> to the <variables> section of your <user>
>>>>>>
>>>>>> you have it at 600 and the nat mapping is timing out while the 600
>>>>>>
>>>>>> seconds is ticking away
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 26, 2010 at 12:01 PM, Mario G <mario_fs at mgtech.com> wrote:
>>>>>>
>>>>>> From the TSP:
>>>>>>
>>>>>> "I have enabled the SIP trace on your account. We are not currently seeing
>>>>>>
>>>>>> any registration attempts to your account within the last 15 minutes. Please
>>>>>>
>>>>>> restart FreeSwitch so that registration attempts begin again. Thank you. ".
>>>>>>
>>>>>> So FS is not getting past router.
>>>>>>
>>>>>> On Oct 26, 2010, at 9:09 AM, Mario G wrote:
>>>>>>
>>>>>> I ran the global trace during the problem and it is
>>>>>>
>>>>>> at http://pastebin.freeswitch.org/14324 . You can find "rnktel", "acctone",
>>>>>>
>>>>>> "accttwo", "acct3". The trace includes phones since it was global. I am
>>>>>>
>>>>>> using:
>>>>>>
>>>>>> <param name="ext-rtp-ip" value="autonat:my-static.ip"/>
>>>>>>
>>>>>> <param name="ext-sip-ip" value="autonat:my-static.ip"/>
>>>>>>
>>>>>> I tried dumping nat and removing the autonat: above and using -nonat but
>>>>>>
>>>>>> that did not work, registration proceeded but no calls inbound.
>>>>>>
>>>>>> On Oct 25, 2010, at 4:11 PM, Mario G wrote:
>>>>>>
>>>>>> Whoops, I am using an IP address for at least one gateway so that is not the
>>>>>>
>>>>>> problem:
>>>>>>
>>>>>> They look outbound to me and I am using dns for 2 and an IP for one so that
>>>>>>
>>>>>> is not the issue. I was able to get FS to clear this up by doing "nat_map
>>>>>>
>>>>>> reinit" which is why I think this is a nat problem. I will do the trace you
>>>>>>
>>>>>> mentioned. I will plug an ip address into one of the gateways to see what
>>>>>>
>>>>>> happens, they all fail at once. Thanks for responding!
>>>>>>
>>>>>> Mario
>>>>>>
>>>>>> On Oct 25, 2010, at 3:26 PM, Mario wrote:
>>>>>>
>>>>>> I really need help on this as I have weeks into this problem. I thought I
>>>>>>
>>>>>> had it nailed but I guess not. After 5.5 hours I get:
>>>>>>
>>>>>> 2010-10-25 15:05:43.407272 [WARNING] sofia_reg.c:387 mguuid Failed
>>>>>>
>>>>>> Registration, setting retry to 15 seconds.
>>>>>>
>>>>>> 2010-10-25 15:05:49.557478 [NOTICE] sofia_reg.c:342 Registering mvuuid
>>>>>>
>>>>>> 2010-10-25 15:05:59.206273 [NOTICE] sofia_reg.c:342 Registering mguuid
>>>>>>
>>>>>> 2010-10-25 15:06:04.923157 [WARNING] sofia_reg.c:387 mynum777 Failed
>>>>>>
>>>>>> Registration, setting retry to 30 seconds.
>>>>>>
>>>>>> 2010-10-25 15:06:05.358321 [WARNING] sofia_reg.c:387 mvuuid Failed
>>>>>>
>>>>>> Registration, setting retry to 15 seconds.
>>>>>>
>>>>>> 2010-10-25 15:06:16.125060 [WARNING] sofia_reg.c:387 mguuid Failed
>>>>>>
>>>>>> Registration, setting retry to 15 seconds.
>>>>>>
>>>>>> 2010-10-25 15:06:21.151240 [NOTICE] sofia_reg.c:342 Registering mvuuid
>>>>>>
>>>>>> 2010-10-25 15:06:33.060421 [NOTICE] sofia_reg.c:342 Registering mguuid
>>>>>>
>>>>>> 2010-10-25 15:06:35.392655 [NOTICE] sofia_reg.c:342 Registering mynum777
>>>>>>
>>>>>> and no way to make/get calls until I restart FS. I did this:
>>>>>>
>>>>>> 1. log 7
>>>>>>
>>>>>> 2. sofia profile xxxx siptrace on for each profile/gateway
>>>>>>
>>>>>> 3. restarted router
>>>>>>
>>>>>> All three did not solve the problem. The trace and log produced no
>>>>>>
>>>>>> additional lines which is why I am wondering if FS has a problem since the
>>>>>>
>>>>>> trace shows no SIP activity.
>>>>>>
>>>>>> 3 gateways with 2 ITSPs
>>>>>>
>>>>>> 2 DSL/WAN lines, 1 static and 1 dynamic
>>>>>>
>>>>>> I am using autonat:1.2.3.4 in internal and external profiles. 1.2.3.4 is the
>>>>>>
>>>>>> external static ip.
>>>>>>
>>>>>> sofia status profile ... has the right ext ip
>>>>>>
>>>>>> nat_map status shows the dynamic (wrong) IP
>>>>>>
>>>>>> I tried starting with -nonat but that was worse
>>>>>>
>>>>>> the only way to fix is restart FS.
>>>>>>
>>>>>> I read the wiki on external nat, auto_nat and everything else many times.
>>>>>>
>>>>>> Thanks Mario
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>>
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>>
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>> _______________________________________________
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>>
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>>
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Anthony Minessale II
>>>>>>
>>>>>> FreeSWITCH http://www.freeswitch.org/
>>>>>>
>>>>>> ClueCon http://www.cluecon.com/
>>>>>>
>>>>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>>>>
>>>>>> AIM: anthm
>>>>>>
>>>>>> MSN:anthony_minessale at hotmail.com
>>>>>>
>>>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>>>>>
>>>>>> IRC: irc.freenode.net #freeswitch
>>>>>>
>>>>>> FreeSWITCH Developer Conference
>>>>>>
>>>>>> sip:888 at conference.freeswitch.org
>>>>>>
>>>>>> googletalk:conf+888 at conference.freeswitch.org
>>>>>>
>>>>>> pstn:+19193869900
>>>>>>
>>>>>> _______________________________________________
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>>
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>>
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Anthony Minessale II
>>>>>>
>>>>>> FreeSWITCH http://www.freeswitch.org/
>>>>>>
>>>>>> ClueCon http://www.cluecon.com/
>>>>>>
>>>>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>>>>
>>>>>> AIM: anthm
>>>>>>
>>>>>> MSN:anthony_minessale at hotmail.com
>>>>>>
>>>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>>>>>
>>>>>> IRC: irc.freenode.net #freeswitch
>>>>>>
>>>>>> FreeSWITCH Developer Conference
>>>>>>
>>>>>> sip:888 at conference.freeswitch.org
>>>>>>
>>>>>> googletalk:conf+888 at conference.freeswitch.org
>>>>>>
>>>>>> pstn:+19193869900
>>>>>>
>>>>>> _______________________________________________
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>>
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>>
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>>
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Anthony Minessale II
>>>>>>
>>>>>> FreeSWITCH http://www.freeswitch.org/
>>>>>>
>>>>>> ClueCon http://www.cluecon.com/
>>>>>>
>>>>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>>>>
>>>>>> AIM: anthm
>>>>>>
>>>>>> MSN:anthony_minessale at hotmail.com
>>>>>>
>>>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>>>>>
>>>>>> IRC: irc.freenode.net #freeswitch
>>>>>>
>>>>>> FreeSWITCH Developer Conference
>>>>>>
>>>>>> sip:888 at conference.freeswitch.org
>>>>>>
>>>>>> googletalk:conf+888 at conference.freeswitch.org
>>>>>>
>>>>>> pstn:+19193869900
>>>>>>
>>>>>> _______________________________________________
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>>
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>>
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Anthony Minessale II
>>>>>>
>>>>>> FreeSWITCH http://www.freeswitch.org/
>>>>>>
>>>>>> ClueCon http://www.cluecon.com/
>>>>>>
>>>>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>>>>
>>>>>> AIM: anthm
>>>>>>
>>>>>> MSN:anthony_minessale at hotmail.com
>>>>>>
>>>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>>>>>
>>>>>> IRC: irc.freenode.net #freeswitch
>>>>>>
>>>>>> FreeSWITCH Developer Conference
>>>>>>
>>>>>> sip:888 at conference.freeswitch.org
>>>>>>
>>>>>> googletalk:conf+888 at conference.freeswitch.org
>>>>>>
>>>>>> pstn:+19193869900
>>>>>>
>>>>>> _______________________________________________
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>>
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>>
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Anthony Minessale II
>>>>>>
>>>>>> FreeSWITCH http://www.freeswitch.org/
>>>>>>
>>>>>> ClueCon http://www.cluecon.com/
>>>>>>
>>>>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>>>>
>>>>>> AIM: anthm
>>>>>>
>>>>>> MSN:anthony_minessale at hotmail.com
>>>>>>
>>>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>>>>>
>>>>>> IRC: irc.freenode.net #freeswitch
>>>>>>
>>>>>> FreeSWITCH Developer Conference
>>>>>>
>>>>>> sip:888 at conference.freeswitch.org
>>>>>>
>>>>>> googletalk:conf+888 at conference.freeswitch.org
>>>>>>
>>>>>> pstn:+19193869900
>>>>>>
>>>>>> _______________________________________________
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>>
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>>
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>>
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>>
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> FreeSWITCH-users mailing list
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> FreeSWITCH-users mailing list
>>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>>> http://www.freeswitch.org
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Anthony Minessale II
>>>>>
>>>>> FreeSWITCH http://www.freeswitch.org/
>>>>> ClueCon http://www.cluecon.com/
>>>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>>>
>>>>> AIM: anthm
>>>>> MSN:anthony_minessale at hotmail.com
>>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>>>> IRC: irc.freenode.net #freeswitch
>>>>>
>>>>> FreeSWITCH Developer Conference
>>>>> sip:888 at conference.freeswitch.org
>>>>> googletalk:conf+888 at conference.freeswitch.org
>>>>> pstn:+19193869900
>>>>>
>>>>> _______________________________________________
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org
>>>>
>>>>
>>>> _______________________________________________
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org
>>>>
>>>
>>>
>>>
>>> --
>>> Anthony Minessale II
>>>
>>> FreeSWITCH http://www.freeswitch.org/
>>> ClueCon http://www.cluecon.com/
>>> Twitter: http://twitter.com/FreeSWITCH_wire
>>>
>>> AIM: anthm
>>> MSN:anthony_minessale at hotmail.com
>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
>>> IRC: irc.freenode.net #freeswitch
>>>
>>> FreeSWITCH Developer Conference
>>> sip:888 at conference.freeswitch.org
>>> googletalk:conf+888 at conference.freeswitch.org
>>> pstn:+19193869900
>>>
>>> _______________________________________________
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org
>>
>>
>> _______________________________________________
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
>> http://www.freeswitch.org
>
>
> _______________________________________________
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
>
--
Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/
ClueCon http://www.cluecon.com/
Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm
MSN:anthony_minessale at hotmail.com
GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference
sip:888 at conference.freeswitch.org
googletalk:conf+888 at conference.freeswitch.org
pstn:+19193869900
More information about the FreeSWITCH-users
mailing list