[Freeswitch-users] B-Leg / A-Leg Race Condition

Mike Burlingame mike.burlingame at me.com
Fri Sep 7 21:23:31 MSD 2012 

I figured I would update to the latest git due to in the past I have seen "make sure your running the latest git" the odd thing is no core dump was created on the last crash.

I will post the backtrace if a core dump is generated after the next crash


On Sep 7, 2012, at 10:16 AM, Anthony Minessale <anthony.minessale at gmail.com> wrote:

> FYI: You need to get the backtrace before you update or the core file becomes useless.
> It dumps core by default so you should have one but if you updated, its tool late unless you can checkout the exact previous rev.
> 
> 
> 
> On Fri, Sep 7, 2012 at 12:00 PM, Mike Burlingame <mike.burlingame at me.com> wrote:
> after putting the server in production and  loading it up it just randomly crashed I am updating to the latest git will try to capture a core dump if one is created on exit nothing in the logs point to anything it was handling calls and just exited.
> 
> However on a positive note the switch sip_wait_for_aleg_ack=true seems to do as expected with all my tests.
> 
> 
> 
> On Sep 6, 2012, at 12:50 PM, Mike Burlingame <mike.burlingame at me.com> wrote:
> 
> > After about 20K test calls this seems to have addressed the issue - I will keep running my test's for today and put this box in a production environment tomorrow to validate it still holds up with load. I will report back after that is completed.
> >
> > Thanks
> >
> > On Sep 5, 2012, at 12:28 PM, Mike Burlingame <mike.burlingame at me.com> wrote:
> >
> >> Looks much much better Thank you -- Now to conduct more testing
> >>
> >> 0.000000 A-Leg -> FreeSwitch SIP/SDP Request: INVITE sip:+13605551212 at A-Leg:5060, with session description
> >> 0.000639 FreeSwitch -> A-Leg SIP Status: 100 Trying
> >> 0.051096 FreeSwitch -> B-Leg SIP/SDP Request: INVITE sip:13605551212 at B-Leg, with session description
> >> 0.051351 B-Leg -> FreeSwitch SIP Status: 100 Giving a try
> >> 0.286387 B-Leg -> FreeSwitch SIP Status: 180 Ringing
> >> 0.288216 FreeSwitch -> A-Leg SIP Status: 180 Ringing
> >> 0.475452 B-Leg -> FreeSwitch SIP Status: 180 Ringing
> >> 5.311144 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session description
> >> 5.313775 FreeSwitch -> A-Leg SIP/SDP Status: 200 OK, with session description
> >> 5.385087 A-Leg -> FreeSwitch SIP Request: ACK sip:+13605551212 at FreeSwitch:5070;transport=udp
> >> 5.385796 FreeSwitch -> B-Leg SIP Request: ACK sip:13605551212 at B-Leg
> >> 12.027026 A-Leg -> FreeSwitch SIP Request: BYE sip:+13605551212 at FreeSwitch:5070;transport=udp
> >> 12.029232 FreeSwitch -> A-Leg SIP Status: 200 OK
> >> 12.030707 FreeSwitch -> B-Leg SIP Request: BYE sip:13605551212 at B-Leg
> >> 12.033888 B-Leg -> FreeSwitch SIP Status: 200 OK
> >>
> >>
> >>
> >> On Sep 5, 2012, at 12:01 PM, Anthony Minessale <anthony.minessale at gmail.com> wrote:
> >>
> >>> ok,
> >>>
> >>> update one more time, if it still does not work just go right to jira
> >>> with the latest (not before today's changes)
> >>>
> >>>
> >>> On Wed, Sep 5, 2012 at 1:37 PM, Mike Burlingame <mike.burlingame at me.com> wrote:
> >>>> The change seems to have broken the ability for the call to connect - would you like me to open a jira up with the current log files or before the change was made today?
> >>>>
> >>>> commit d45db898339e1b2212f5befff1af714abcec034f
> >>>> Author: Anthony Minessale <anthm at freeswitch.org>
> >>>> Date:   Wed Sep 5 13:11:32 2012 -0500
> >>>>
> >>>> 0.000000 A-LEG -> FreeSwitch SIP/SDP Request: INVITE sip:+13605551212 at A-LEG:5060, with session description
> >>>> 0.002715 FreeSwitch -> A-LEG SIP Status: 100 Trying
> >>>> 0.062694 FreeSwitch -> B-Leg SIP/SDP Request: INVITE sip:13605551212 at B-Leg, with session description
> >>>> 0.062976 B-Leg -> FreeSwitch SIP Status: 100 Giving a try
> >>>> 0.238255 B-Leg -> FreeSwitch SIP Status: 180 Ringing
> >>>> 0.239830 FreeSwitch -> A-LEG SIP Status: 180 Ringing
> >>>> 0.456828 B-Leg -> FreeSwitch SIP Status: 180 Ringing
> >>>> 4.642993 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session description
> >>>> 5.637738 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session description
> >>>> 5.638091 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session description
> >>>> 6.037804 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session description
> >>>> 6.839818 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session description
> >>>> 8.438750 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session description
> >>>> 11.638797 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session description
> >>>> 24.925970 FreeSwitch -> A-LEG SIP Status: 480 Temporarily Unavailable
> >>>> 24.926257 A-LEG -> FreeSwitch SIP Request: ACK sip:+13605551212 at A-LEG:5060
> >>>> 24.926321 FreeSwitch -> B-Leg SIP Request: ACK sip:13605551212 at B-Leg
> >>>> 24.926580 FreeSwitch -> B-Leg SIP Request: BYE sip:13605551212 at B-Leg
> >>>> 24.927029 B-Leg -> FreeSwitch SIP Status: 481 Call leg/transaction does not exist
> >>>> 27.078016 A-LEG -> FreeSwitch SIP/SDP Request: INVITE sip:+13605551212 at A-LEG:5060, with session description
> >>>> 27.078467 FreeSwitch -> A-LEG SIP Status: 100 Trying
> >>>> 27.123233 FreeSwitch -> B-Leg SIP/SDP Request: INVITE sip:13605551212 at B-Leg, with session description
> >>>> 27.123445 B-Leg -> FreeSwitch SIP Status: 100 Giving a try
> >>>> 27.315802 B-Leg -> FreeSwitch SIP Status: 180 Ringing
> >>>> 27.317391 FreeSwitch -> A-LEG SIP Status: 180 Ringing
> >>>> 27.529142 B-Leg -> FreeSwitch SIP Status: 180 Ringing
> >>>> 31.520118 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session description
> >>>> 31.769831 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session description
> >>>> 31.920832 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session description
> >>>> 32.319816 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session description
> >>>> 33.120808 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session description
> >>>> 34.720813 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session description
> >>>> 37.920852 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session description
> >>>> 49.362647 A-LEG -> FreeSwitch SIP Request: CANCEL sip:+13605551212 at A-LEG:5060
> >>>> 49.362952 FreeSwitch -> A-LEG SIP Status: 200 OK
> >>>> 49.363196 FreeSwitch -> A-LEG SIP Status: 487 Request Terminated
> >>>> 49.363307 A-LEG -> FreeSwitch SIP Request: ACK sip:+13605551212 at A-LEG:5060
> >>>> 49.365975 FreeSwitch -> B-Leg SIP Request: ACK sip:13605551212 at B-Leg
> >>>> 49.366171 FreeSwitch -> B-Leg SIP Request: BYE sip:13605551212 at B-Leg
> >>>> 49.366522 B-Leg -> FreeSwitch SIP Status: 481 Call leg/transaction does not exist
> >>>>
> >>>>
> >>>> On Sep 5, 2012, at 11:10 AM, Anthony Minessale <anthony.minessale at gmail.com> wrote:
> >>>>
> >>>>> update and try again, if it still doesn't work open a jira
> >>>>>
> >>>>>
> >>>>> On Wed, Sep 5, 2012 at 12:18 PM, Mike Burlingame <mike.burlingame at me.com> wrote:
> >>>>>> as promised here is the update testing and enabling
> >>>>>> {sip_wait_for_aleg_ack=true} causes a log delay before the audio starts to
> >>>>>> flow to the B-Leg endpoint with multi 200 OK's being sent to FS from the
> >>>>>> B-Leg as well as an abnormally long delay in getting an ACK from the A-Leg
> >>>>>>
> >>>>>> I can open a JIRA case on this and provide the console log file / PCAP's ect
> >>>>>> if that would help
> >>>>>>
> >>>>>>
> >>>>>> Call Flow with out {sip_wait_for_aleg_ack=true}
> >>>>>> 0.000000 A-LEG -> FreeSwitch SIP/SDP Request: INVITE
> >>>>>> sip:+13605551212 at A-LEG:5060, with session description
> >>>>>> 0.000652 FreeSwitch -> A-LEG SIP Status: 100 Trying
> >>>>>> 0.042837 FreeSwitch -> B-Leg SIP/SDP Request: INVITE
> >>>>>> sip:13605551212 at B-Leg, with session description
> >>>>>> 0.043059 B-Leg -> FreeSwitch SIP Status: 100 Giving a try
> >>>>>> 0.290947 B-Leg -> FreeSwitch SIP Status: 180 Ringing
> >>>>>> 0.292890 FreeSwitch -> A-LEG SIP Status: 180 Ringing
> >>>>>> 0.490220 B-Leg -> FreeSwitch SIP Status: 180 Ringing
> >>>>>> 4.481038 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session
> >>>>>> description
> >>>>>> 4.482310 FreeSwitch -> B-Leg SIP Request: ACK sip:13605551212 at B-Leg
> >>>>>> 4.483474 FreeSwitch -> A-LEG SIP/SDP Status: 200 OK, with session
> >>>>>> description
> >>>>>> 4.533691 A-LEG -> FreeSwitch SIP Request: ACK
> >>>>>> sip:+13605551212 at FreeSwitch:5070;transport=udp
> >>>>>> 19.953061 A-LEG -> FreeSwitch SIP Request: BYE
> >>>>>> sip:+13605551212 at FreeSwitch:5070;transport=udp
> >>>>>> 19.954592 FreeSwitch -> A-LEG SIP Status: 200 OK
> >>>>>> 19.955454 FreeSwitch -> B-Leg SIP Request: BYE sip:13605551212 at B-Leg
> >>>>>> 19.956014 B-Leg -> FreeSwitch SIP Status: 200 OK
> >>>>>>
> >>>>>> Call Flow with {sip_wait_for_aleg_ack=true} enabled
> >>>>>> 0.000000 A-LEG -> FreeSwitch SIP/SDP Request: INVITE
> >>>>>> sip:+13605551212 at A-LEG:5060, with session description
> >>>>>> 0.000651 FreeSwitch -> A-LEG SIP Status: 100 Trying
> >>>>>> 0.039455 FreeSwitch -> B-Leg SIP/SDP Request: INVITE
> >>>>>> sip:13605551212 at B-Leg, with session description
> >>>>>> 0.039709 B-Leg -> FreeSwitch SIP Status: 100 Giving a try
> >>>>>> 0.244269 B-Leg -> FreeSwitch SIP Status: 180 Ringing
> >>>>>> 0.245607 FreeSwitch -> A-LEG SIP Status: 180 Ringing
> >>>>>> 0.363325 B-Leg -> FreeSwitch SIP Status: 180 Ringing
> >>>>>> 4.718173 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session
> >>>>>> description
> >>>>>> 4.918915 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session
> >>>>>> description
> >>>>>> 5.117917 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session
> >>>>>> description
> >>>>>> 5.518902 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session
> >>>>>> description
> >>>>>> 6.318053 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session
> >>>>>> description
> >>>>>> 7.917921 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session
> >>>>>> description
> >>>>>> 11.118917 B-Leg -> FreeSwitch SIP/SDP Status: 200 OK, with session
> >>>>>> description
> >>>>>> 15.102917 FreeSwitch -> B-Leg SIP Request: ACK sip:13605551212 at B-Leg
> >>>>>> 15.104488 FreeSwitch -> A-LEG SIP/SDP Status: 200 OK, with session
> >>>>>> description
> >>>>>> 15.173265 A-LEG -> FreeSwitch SIP Request: ACK
> >>>>>> sip:+13605551212 at FreeSwitch:5070;transport=udp
> >>>>>> 29.566504 A-LEG -> FreeSwitch SIP Request: BYE
> >>>>>> sip:+13605551212 at FreeSwitch:5070;transport=udp
> >>>>>> 29.568902 FreeSwitch -> A-LEG SIP Status: 200 OK
> >>>>>> 29.570603 FreeSwitch -> B-Leg SIP Request: BYE sip:13605551212 at B-Leg
> >>>>>> 29.571240 B-Leg -> FreeSwitch SIP Status: 200 OK
> >>>>>>
> >>>>>> On Aug 31, 2012, at 1:52 PM, Mike Burlingame <mike.burlingame at me.com> wrote:
> >>>>>>
> >>>>>> No worries I will be out this weekend for the long weekend I will work on
> >>>>>> getting the test box upgraded and a test case setup on Tuesday I will report
> >>>>>> back the results mid to late next week and provided everything works as I
> >>>>>> hope it will I will be happy to pay the Wiki tax :)
> >>>>>>
> >>>>>>
> >>>>>> On Aug 31, 2012, at 1:42 PM, Michael Collins <msc at freeswitch.org> wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Aug 31, 2012 at 12:59 PM, Mike Burlingame <mike.burlingame at me.com>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> Cool I will nail that up on my test box and see if that works
> >>>>>>>
> >>>>>> Please report back on whether it works or not and then be prepared to pay
> >>>>>> the wiki tax. :) I'll be glad to assist with getting this documented
> >>>>>> although I think you're in the best position to give that documentation some
> >>>>>> real-world context.
> >>>>>>
> >>>>>> -MC
> >>>>>> _________________________________________________________________________
> >>>>>> Professional FreeSWITCH Consulting Services:
> >>>>>> consulting at freeswitch.org
> >>>>>> http://www.freeswitchsolutions.com
> >>>>>>
> >>>>>> 
> >>>>>> 
> >>>>>>
> >>>>>> Official FreeSWITCH Sites
> >>>>>> http://www.freeswitch.org
> >>>>>> http://wiki.freeswitch.org
> >>>>>> http://www.cluecon.com
> >>>>>>
> >>>>>> FreeSWITCH-users mailing list
> >>>>>> FreeSWITCH-users at lists.freeswitch.org
> >>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>>>> http://www.freeswitch.org
> >>>>>>
> >>>>>>
> >>>>>> _________________________________________________________________________
> >>>>>> Professional FreeSWITCH Consulting Services:
> >>>>>> consulting at freeswitch.org
> >>>>>> http://www.freeswitchsolutions.com
> >>>>>>
> >>>>>> 
> >>>>>> 
> >>>>>>
> >>>>>> Official FreeSWITCH Sites
> >>>>>> http://www.freeswitch.org
> >>>>>> http://wiki.freeswitch.org
> >>>>>> http://www.cluecon.com
> >>>>>>
> >>>>>> FreeSWITCH-users mailing list
> >>>>>> FreeSWITCH-users at lists.freeswitch.org
> >>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>>>> http://www.freeswitch.org
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> _________________________________________________________________________
> >>>>>> Professional FreeSWITCH Consulting Services:
> >>>>>> consulting at freeswitch.org
> >>>>>> http://www.freeswitchsolutions.com
> >>>>>>
> >>>>>> 
> >>>>>> 
> >>>>>>
> >>>>>> Official FreeSWITCH Sites
> >>>>>> http://www.freeswitch.org
> >>>>>> http://wiki.freeswitch.org
> >>>>>> http://www.cluecon.com
> >>>>>>
> >>>>>> FreeSWITCH-users mailing list
> >>>>>> FreeSWITCH-users at lists.freeswitch.org
> >>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>>>> http://www.freeswitch.org
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Anthony Minessale II
> >>>>>
> >>>>> FreeSWITCH http://www.freeswitch.org/
> >>>>> ClueCon http://www.cluecon.com/
> >>>>> Twitter: http://twitter.com/FreeSWITCH_wire
> >>>>>
> >>>>> AIM: anthm
> >>>>> MSN:anthony_minessale at hotmail.com
> >>>>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
> >>>>> IRC: irc.freenode.net #freeswitch
> >>>>>
> >>>>> FreeSWITCH Developer Conference
> >>>>> sip:888 at conference.freeswitch.org
> >>>>> googletalk:conf+888 at conference.freeswitch.org
> >>>>> pstn:+19193869900
> >>>>>
> >>>>> _________________________________________________________________________
> >>>>> Professional FreeSWITCH Consulting Services:
> >>>>> consulting at freeswitch.org
> >>>>> http://www.freeswitchsolutions.com
> >>>>>
> >>>>> 
> >>>>> 
> >>>>>
> >>>>> Official FreeSWITCH Sites
> >>>>> http://www.freeswitch.org
> >>>>> http://wiki.freeswitch.org
> >>>>> http://www.cluecon.com
> >>>>>
> >>>>> FreeSWITCH-users mailing list
> >>>>> FreeSWITCH-users at lists.freeswitch.org
> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>>> http://www.freeswitch.org
> >>>>
> >>>>
> >>>> _________________________________________________________________________
> >>>> Professional FreeSWITCH Consulting Services:
> >>>> consulting at freeswitch.org
> >>>> http://www.freeswitchsolutions.com
> >>>>
> >>>> 
> >>>> 
> >>>>
> >>>> Official FreeSWITCH Sites
> >>>> http://www.freeswitch.org
> >>>> http://wiki.freeswitch.org
> >>>> http://www.cluecon.com
> >>>>
> >>>> FreeSWITCH-users mailing list
> >>>> FreeSWITCH-users at lists.freeswitch.org
> >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>>> http://www.freeswitch.org
> >>>
> >>>
> >>>
> >>> --
> >>> Anthony Minessale II
> >>>
> >>> FreeSWITCH http://www.freeswitch.org/
> >>> ClueCon http://www.cluecon.com/
> >>> Twitter: http://twitter.com/FreeSWITCH_wire
> >>>
> >>> AIM: anthm
> >>> MSN:anthony_minessale at hotmail.com
> >>> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
> >>> IRC: irc.freenode.net #freeswitch
> >>>
> >>> FreeSWITCH Developer Conference
> >>> sip:888 at conference.freeswitch.org
> >>> googletalk:conf+888 at conference.freeswitch.org
> >>> pstn:+19193869900
> >>>
> >>> _________________________________________________________________________
> >>> Professional FreeSWITCH Consulting Services:
> >>> consulting at freeswitch.org
> >>> http://www.freeswitchsolutions.com
> >>>
> >>> 
> >>> 
> >>>
> >>> Official FreeSWITCH Sites
> >>> http://www.freeswitch.org
> >>> http://wiki.freeswitch.org
> >>> http://www.cluecon.com
> >>>
> >>> FreeSWITCH-users mailing list
> >>> FreeSWITCH-users at lists.freeswitch.org
> >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> >>> http://www.freeswitch.org
> >>
> >>
> >> _________________________________________________________________________
> >> Professional FreeSWITCH Consulting Services:
> >> consulting at freeswitch.org
> >> http://www.freeswitchsolutions.com
> >>
> >> 
> >> 
> >>
> >> Official FreeSWITCH Sites
> >> http://www.freeswitch.org
> >> http://wiki.freeswitch.org
> >> http://www.cluecon.com
> >>
> >> FreeSWITCH-users mailing list
> >> FreeSWITCH-users at lists.freeswitch.org
> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> >> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> >> http://www.freeswitch.org
> >
> >
> > _________________________________________________________________________
> > Professional FreeSWITCH Consulting Services:
> > consulting at freeswitch.org
> > http://www.freeswitchsolutions.com
> >
> > 
> > 
> >
> > Official FreeSWITCH Sites
> > http://www.freeswitch.org
> > http://wiki.freeswitch.org
> > http://www.cluecon.com
> >
> > FreeSWITCH-users mailing list
> > FreeSWITCH-users at lists.freeswitch.org
> > http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> > UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> > http://www.freeswitch.org
> 
> 
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
> 
> 
> 
> 
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
> 
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
> 
> 
> 
> -- 
> Anthony Minessale II
> 
> FreeSWITCH http://www.freeswitch.org/
> ClueCon http://www.cluecon.com/
> Twitter: http://twitter.com/FreeSWITCH_wire
> 
> AIM: anthm
> MSN:anthony_minessale at hotmail.com
> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
> IRC: irc.freenode.net #freeswitch
> 
> FreeSWITCH Developer Conference
> sip:888 at conference.freeswitch.org
> googletalk:conf+888 at conference.freeswitch.org
> pstn:+19193869900
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
> 
> 
> 
> 
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
> 
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20120907/4d899e1e/attachment-0001.html

Join us at ClueCon 2011 Aug 9-11, 2011
More information about the FreeSWITCH-users mailing list