[Freeswitch-users] Security vs compatibility / NAT etc
Rick Jarvis
rick.jarvis at magicmail.mooo.com
Mon Jul 4 23:14:05 MSD 2016
I was using a Yealink T22P, which is end of life now, so just switched over to a T48P. It’s ok if the older phones don’t work with it, as long as the new ones do! :)
So the T48 works fine with STUN disabled. But, it also works fine with the proxy disabled (and I’ve no idea why that should even work... Maybe the phone is keeping the connection open to FS anyway?!).
> On 4 Jul 2016, at 19:37, Colin Morelli <colin.morelli at gmail.com> wrote:
>
> Rick,
>
> At this point most likely you need the handset to advertise explicit outbound support for Kamailio to act appropriately. Handsets/softphones generally don't do this by default, but need to be configured to do such (the RFC was originally released in 2009 IIRC so it likely requires at least a fairly recent handset). You'll want to make sure that the device is setup using TCP as its transport and that "SIP outbound" (or similarly named features) are enabled. You can see if it's being used by making sure the request has "Supported: path, outbound" and a "sip.instance" parameter in the Contact URL.
>
> If the handset doesn't explicitly support outbound behavior, you may be able to still get it working by adding modparam("outbound", "force_outbound_flag", 1) in your kamailio config after the loadmodules. This should work as long as the device doesn't close its TCP connection after registering.
>
> Hope we didn't drag you through this just to find out the device doesn't support this behavior!
>
> Best,
> Colin
>
> On Mon, Jul 4, 2016 at 2:28 PM Rick Jarvis <rick.jarvis at magicmail.mooo.com <mailto:rick.jarvis at magicmail.mooo.com>> wrote:
> That’s amazing, really getting somewhere now! So I’m now routing calls through the kamailo and onto FS fine, but I’m guessing I’m not quite there, as it only works if STUN is enabled on the handset... so would I be right in thinking it’s not completely doing the ‘sip outbound’ bit?
>
>> On 4 Jul 2016, at 17:22, Colin Morelli <colin.morelli at gmail.com <mailto:colin.morelli at gmail.com>> wrote:
>>
>> Awesome! Glad that things are working well.
>>
>> Now, we're almost there with the outbound support. You'll notice in the script that you're using now that Kamailio will be record-routing itself on INVITE/SUBSCRIBE requests, and it'll be adding itself to the path on REGISTER requests (check that first route {} block and you'll see what I'm referring to). This is 90% of the way there. The only remaining piece is to get Kamailio to correctly handle persistent outbound-style TCP connections and create flow tokens. This is where the outbound module comes in.
>>
>> You'll want to add loadmodule "outbound.so" to your config file for kamailio. Make sure this appears before the path.so and rr.so loadmodule calls. In order for Kamailio to handle outbound correctly it needs those modules to bind to the outbound module. With that done, Kamailio will include flow tokens in the record-route and path headers (you can see this in a pcap if you're interested). Now, you'll want to add a Flow-Timer header to inform your client of how often it should be sending keepalive pings. Unfortunately, at least last I checked, Freeswitch doesn't properly add the "Require: outbound" header to a response for a REGISTER request that includes "Supported: outbound" and a Contact with an ;ob parameter - however - it does properly follow the path provided in the register which is all that we really need. So this means we need to instead base the choice on Flow-Timer on whether or not the client advertises support for outbound. This may not work in all cases but I can't think of a case where it wouldn't off the top of my head.
>>
>> So, let's add the following block to your onreply_route (this handler already exists in the config file I linked to - we just need to add to it). Also, you may adjust the 30 below to whatever ping interval you want to use between the client and Kamailio:
>>
>> if ($rm == "REGISTER" && $rs >= 200 && $rs <= 299) {
>> remove_hf("Flow-Timer");
>> if ($(hdr(Supported)[*]) =~ "outbound") {
>> insert_hf("Flow-Timer: 30\r\n", "Call-ID");
>> }
>> }
>>
>> That should work, thought admittedly you might need to play around with other ways of detecting if the client is trying to use an outbound connection (I'm using Kamailio as a registrar in my case which does properly add the Require: header, so the check there is easy). If it doesn't work, another way of checking that should work is to check if the "Contact:" header contains the "ob" param.
>>
>> Anyway, that should technically be all you need to make this work. But, it's generally based to let Freeswitch know if it's calling out to a client that no longer exists. For example, you may have a 30 minute registration timeline but the socket may be closed after 5 minutes. When FS makes a call out to Kamailio to forward to an outbound client, Kamailio should respond with a "430 Flow Failed" response so Freeswitch knows to cleanup that (now invalid) registration.
>>
>> Check out the edge-proxy example on this page http://www.kamailio.org/docs/modules/4.3.x/modules/outbound.html#idp17551232 <http://www.kamailio.org/docs/modules/4.3.x/modules/outbound.html#idp17551232> to see how you can add that functionality (you'll want to look at the failure_route and t_on_failure calls).
>>
>> Note - at some point it might make more sense to remove registrations from FS and free it up to just handle media, but this should work for now.
>>
>> Best,
>> Colin
>>
>> On Mon, Jul 4, 2016 at 11:54 AM Rick Jarvis <rick.jarvis at magicmail.mooo.com <mailto:rick.jarvis at magicmail.mooo.com>> wrote:
>> I’m on Kamailio v 4.2.0…
>>
>> Great, that’s working now, restarts fine and I can register using the proxy! Also if I do a sofia status reg, it shows the IP for the registration as the Kamailio IP, so I guess the signalling is working ok? Is it just the outbound/path support it needs now?
>>
>>
>>> On 4 Jul 2016, at 16:23, Colin Morelli <colin.morelli at gmail.com <mailto:colin.morelli at gmail.com>> wrote:
>>>
>>> Hey Rick,
>>>
>>> Are you using the exact config file posted on the wiki? That looks like it's probably a bit old, for a much earlier version of Kamailio. I'm assuming you're using something in the 4.x range, which might require some adjustments.
>>>
>>> First, I'd suggest trying to use a more up-to-date config file. The one over here looks decent: https://blog.voipxswitch.com/2015/03/27/kamailio-basic-sip-proxy-all-requests-setup/ <https://blog.voipxswitch.com/2015/03/27/kamailio-basic-sip-proxy-all-requests-setup/>
>>>
>>> Note that the proxy script there is a bit more complex than what you've got now, but it should be pretty straightforward to figure out. Additionally, it includes log statements that'll show up without needing to enable the very verbose Kamailio debug logs. They should help you figure out what's going on. Also, that proxy script doesn't yet add the outbound/path support that you're looking for - but we can get to that point once we get traffic just getting to Freeswitch (it's simple to add).
>>>
>>> Next, try to get a SIP trace and see what the traffic looks like. You can capture TCP traffic with (assuming eth0 is the interface that the traffic is on): tcpdump -s 0 -i eth0 -w /tmp/sip.pcap port 5060. Getting a packet capture and looking through it will help you identify what Kamailio is doing with the call.
>>>
>>> Start with that and let's see where we get. From there we'll probably have a better idea of the next steps.
>>>
>>> Best,
>>> Colin
>>>
>>> On Mon, Jul 4, 2016 at 11:05 AM Rick Jarvis <rick.jarvis at magicmail.mooo.com <mailto:rick.jarvis at magicmail.mooo.com>> wrote:
>>> Still struggling with this config trying to get Kamailio to act as an outbound proxy for NAT’d connections.
>>>
>>>>> https://wiki.freeswitch.org/wiki/Kamailio_basic_setup_as_proxy_for_FreeSWITCH <https://wiki.freeswitch.org/wiki/Kamailio_basic_setup_as_proxy_for_FreeSWITCH>
>>> I’ve changed the IPs to mine of course, and I’ve even simplified by not using private IPs anywhere. Enabling debugging doesn’t give me anything I understand. The IP of FreeSWITCH isn’t showing anywhere in the log, which suggests to me that it’s not even getting as far as passing the SIP request to FS…? The x.x.x.x below is the IP of the kamailio box...
>>>
>>> Help! This is driving me mad….
>>>
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14188]: DEBUG: <core> [parser/parse_via.c:1284]: parse_via_param(): Found param type 235, <rport> = <5076>; state=16
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14187]: DEBUG: <core> [parser/parse_via.c:1284]: parse_via_param(): Found param type 232, <branch> = <z9hG4bK-5aed5dc1317bd4b9202a5f65f33730f7>; state=6
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14188]: DEBUG: <core> [parser/parse_via.c:2672]: parse_via(): end of header reached, state=5
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14187]: DEBUG: <core> [parser/parse_via.c:1284]: parse_via_param(): Found param type 235, <rport> = <5076>; state=16
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14188]: DEBUG: <core> [parser/msg_parser.c:513]: parse_headers(): parse_headers: Via found, flags=62
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14187]: DEBUG: <core> [parser/parse_via.c:2672]: parse_via(): end of header reached, state=5
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14188]: DEBUG: <core> [parser/msg_parser.c:526]: parse_headers(): parse_headers: this is the second via
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14187]: DEBUG: <core> [parser/msg_parser.c:513]: parse_headers(): parse_headers: Via found, flags=ffffffffffffffff
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14188]: DEBUG: <core> [parser/parse_addr_spec.c:898]: parse_addr_spec(): end of header reached, state=10
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14187]: DEBUG: <core> [parser/msg_parser.c:526]: parse_headers(): parse_headers: this is the second via
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14188]: DEBUG: <core> [parser/msg_parser.c:190]: get_hdr_field(): DEBUG: get_hdr_field: <To> [61]; uri=[sip:*910000972592892325 at x.x.x.x] <>
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14187]: DEBUG: <core> [parser/parse_addr_spec.c:176]: parse_to_param(): DEBUG: add_param: tag=73pNXe7gUUXpr
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14188]: DEBUG: <core> [parser/msg_parser.c:192]: get_hdr_field(): DEBUG: to body [*910000972592892325 <sip:*910000972592892325 at x.x.x.x <>>#015#012]
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14187]: DEBUG: <core> [parser/parse_addr_spec.c:898]: parse_addr_spec(): end of header reached, state=29
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14188]: DEBUG: <core> [parser/msg_parser.c:170]: get_hdr_field(): get_hdr_field: cseq <CSeq>: <1> <INVITE>
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14187]: DEBUG: <core> [parser/msg_parser.c:190]: get_hdr_field(): DEBUG: get_hdr_field: <To> [79]; uri=[sip:*910000972592892325 at x.x.x.x] <>
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14188]: DEBUG: tm [t_lookup.c:949]: t_reply_matching(): DEBUG: t_reply_matching: hash 21995 label 0 branch 0
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14187]: DEBUG: <core> [parser/msg_parser.c:192]: get_hdr_field(): DEBUG: to body [*910000972592892325 <sip:*910000972592892325 at x.x.x.x <>>]
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14188]: DEBUG: tm [t_lookup.c:1004]: t_reply_matching(): DEBUG: t_reply_matching: reply matched (T=0x7f3ab64ecbe0)!
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14187]: DEBUG: <core> [parser/msg_parser.c:170]: get_hdr_field(): get_hdr_field: cseq <CSeq>: <1> <INVITE>
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14188]: DEBUG: tm [t_lookup.c:1141]: t_check_msg(): DEBUG: t_check_msg: msg id=2 global id=2 T end=0x7f3ab64ecbe0
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14187]: DEBUG: <core> [parser/msg_parser.c:204]: get_hdr_field(): DEBUG: get_hdr_body : content_length=0
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14188]: DEBUG: tm [t_reply.c:2210]: reply_received(): DEBUG: reply_received: org. status uas=100, uac[0]=0 local=0 is_invite=1)
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14187]: DEBUG: <core> [parser/msg_parser.c:106]: get_hdr_field(): found end of header
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14188]: DEBUG: tm [t_reply.c:1304]: t_should_relay_response(): ->>>>>>>>> T_code=100, new_code=100
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14187]: DEBUG: tm [t_lookup.c:949]: t_reply_matching(): DEBUG: t_reply_matching: hash 21995 label 0 branch 0
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14188]: DEBUG: tm [t_reply.c:1822]: relay_reply(): DEBUG: relay_reply: branch=0, save=0, relay=-1 icode=0
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14187]: DEBUG: tm [t_lookup.c:1004]: t_reply_matching(): DEBUG: t_reply_matching: reply matched (T=0x7f3ab64ecbe0)!
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14188]: DEBUG: <core> [usr_avp.c:643]: destroy_avp_list(): DEBUG:destroy_avp_list: destroying list (nil)
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14187]: DEBUG: tm [t_lookup.c:1141]: t_check_msg(): DEBUG: t_check_msg: msg id=2 global id=2 T end=0x7f3ab64ecbe0
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14188]: DEBUG: <core> [usr_avp.c:643]: destroy_avp_list(): DEBUG:destroy_avp_list: destroying list (nil)
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14187]: DEBUG: tm [t_reply.c:2210]: reply_received(): DEBUG: reply_received: org. status uas=100, uac[0]=100 local=0 is_invite=1)
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14188]: DEBUG: <core> [usr_avp.c:643]: destroy_avp_list(): DEBUG:destroy_avp_list: destroying list (nil)
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14187]: DEBUG: tm [t_reply.c:1304]: t_should_relay_response(): ->>>>>>>>> T_code=100, new_code=407
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14188]: DEBUG: <core> [usr_avp.c:643]: destroy_avp_list(): DEBUG:destroy_avp_list: destroying list (nil)
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14187]: DEBUG: tm [t_reply.c:1822]: relay_reply(): DEBUG: relay_reply: branch=0, save=0, relay=0 icode=0
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14188]: DEBUG: <core> [usr_avp.c:643]: destroy_avp_list(): DEBUG:destroy_avp_list: destroying list (nil)
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14187]: DEBUG: <core> [msg_translator.c:2266]: generate_res_buf_from_sip_res(): old size: 976, new size: 890
>>> Jul 4 10:53:40 proxy /usr/sbin/kamailio[14188]: DEBUG: <core> [usr_avp.c:643]: destroy_avp_list(): DEBUG:destroy_avp_list: destroying list (nil)
>>>> On 2 Jul 2016, at 20:08, Rick Jarvis <rick.jarvis at magicmail.mooo.com <mailto:rick.jarvis at magicmail.mooo.com>> wrote:
>>>>
>>>> Thanks Colin, really appreciate the help!
>>>>
>>>> So should the SIP_DOMAIN in kamctlrc be the IP of the kamailio server… or the FS server?
>>>>
>>>> And should the handset be set to use the FS public IP as the SIP server, and the kamailio server as the proxy?
>>>>
>>>> And should there be something else listed as the @ realm in the register name?!
>>>>
>>>> I am determined to get my head around proxying!
>>>>
>>>>> On 2 Jul 2016, at 19:35, Colin Morelli <colin.morelli at gmail.com <mailto:colin.morelli at gmail.com>> wrote:
>>>>>
>>>>> Rick,
>>>>>
>>>>> FS has four variables you'll need to be concerned with. sip-ip, rtp-ip, ext-sip-ip, and ext-rtp-ip.
>>>>>
>>>>> The sip/rtp-ip variables tell Freeswitch which IP address it should bind to for SIP and RTP traffic, respectively. The ext-*-ip variables tell Freeswitch what IP addresses it should advertise that it listens to, for SIP and RTP, respectively. In cases where FS is behind a NAT of some type, these may be different. If your public IP is directly attached to an interface on the FS box, they may very well be the same.
>>>>>
>>>>> In your case, you likely want sip-ip and ext-sip-ip set to the private IP address for the FS instances. Similarly, you want rtp-ip and ext-rtp-ip set to the public IP address for the FS instance. (I believe if you don't set ext-*-ip variables at all they just default to the same as their non-ext counterparts).
>>>>>
>>>>> With this configuration, FS will listen to SIP traffic over the private interface (from your Kamailio proxy). When it needs to construct 200 OKs or INVITEs, it will advertise itself as listening to media on its own public interface (the value of the rtp-ip variable).
>>>>>
>>>>> As far as not seeing anything in syslog for the registration, Kamailio is fairly light on logging by default (unless you set debug=4), and the script that you referenced doesn't contain any xlog statements so you probably won't see anything. I'd instrument the script with xlog() calls throughout and see if it gets you anything. When in doubt you can always tcpdump as well to see if you're getting the requests/responses you hope to get.
>>>>>
>>>>> Best,
>>>>> Colin
>>>>>
>>>>> On Sat, Jul 2, 2016 at 2:29 PM Rick Jarvis <rick.jarvis at magicmail.mooo.com <mailto:rick.jarvis at magicmail.mooo.com>> wrote:
>>>>> Ok, so first step is to get Kamailio working as a proxy. TLS will come in time….
>>>>>
>>>>> I’ve used this config file https://wiki.freeswitch.org/wiki/Kamailio_basic_setup_as_proxy_for_FreeSWITCH <https://wiki.freeswitch.org/wiki/Kamailio_basic_setup_as_proxy_for_FreeSWITCH>
>>>>> Kamailio server and FS server both have public and private interfaces
>>>>> Kamailio server is listening on its public IP
>>>>> FS is listening (profile internal) on its private IP
>>>>> Kamailio is set to rewrite to FS server’s private IP
>>>>>
>>>>> What I don’t quite get, having not used a proxy in this way before, is what happens next. If FS needs to advertise its public IPs for the RTP, how does this happen?
>>>>>
>>>>> So far I’m not seeing anything at all in Syslog for the incoming registration.
>>>>>
>>>>> Any help appreciated!
>>>>>
>>>>>
>>>>>> On 30 Jun 2016, at 14:39, Colin Morelli <colin.morelli at gmail.com <mailto:colin.morelli at gmail.com>> wrote:
>>>>>>
>>>>>
>>>>>> Rick,
>>>>>>
>>>>>> (Sorry for the long email, hopefully it's helpful)
>>>>>>
>>>>>> It sounds like you're mostly concerned with FS initiating calls to handsets behind NAT, is that correct?
>>>>>>
>>>>>> If so, what you probably want is SIP outbound (RFC 5626). It's the best way to avoid NAT issues with clients. Under this model, clients keep a persistent connection open to the server. The server is responsible for using that connection to deliver INVITEs to the client, thus avoiding the need to ever open its own connection.
>>>>>>
>>>>>> In my (relatively limited) experience with FS, it was able to act like a SIP outbound server, but it doesn't directly advertise it and supporting SIP outbound is really outside of the core scope of what FS does. So, in my setup, I use Kamailio to provide the SIP outbound support. A brief description of my setup (which seems to work fine with clients behind NAT)
>>>>>>
>>>>>> Kamailio edge proxy cluster (provides SIP outbound support to clients, allows public SIP traffic)
>>>>>> Kamailio proxy + registrar (only allows SIP traffic from inside the local network, provides registration support)
>>>>>> Freeswitch (only allows SIP traffic from inside the local network, has a public IP address and open firewall for RTP traffic).
>>>>>>
>>>>>> So, a registration from a client hits the Kamailio edge proxy, which parks the socket connection and sends it on to the second Kamailio proxy/registrar. When FS needs to make outbound calls to clients, it hits the Kamailio proxy/registrar, which forwards it to the edge proxy that has an existing connection the client and uses it to deliver the invite (this is all handled by Kamailio with it's outbound, path, registrar, and usrloc modules).
>>>>>>
>>>>>> Note your setup might not require the use of two layers of proxies before FS. In my case, I keep registrations off of FS so it's only handling calls. If you have registrations in FS, you can likely just have a Kamailio edge proxy for advertising SIP outbound support, and have it proxy all traffic into FS.
>>>>>>
>>>>>> With this setup, FS will receive SIP traffic from Kamailio, and advertise (in the SDP) its public IP address for RTP media (which needs to be allowed through the firewall). Freeswitch will then open what it refers to as an auto-adjust window for the RTP media. In other words, FS will assume that the first address/port to send RTP media to the RTP port configured for a call is the remote client for that call. As a result, FS is able to cope with clients behind NAT on the media side as well. I believe this feature is enabled by default, but you may have to enable it - you'd have to check the docs on this one.
>>>>>>
>>>>>> With those two pieces combined you should be able to get past any NAT issues without the need for STUN/TURN. Unless you bypass media on FS, in which case you're going to need those.
>>>>>>
>>>>>> Hopefully that helps you out a bit.
>>>>>>
>>>>>> Best,
>>>>>> Colin
>>>>>>
>>>>>> On Thu, Jun 30, 2016 at 8:52 AM Rick Jarvis <rick.jarvis at magicmail.mooo.com <mailto:rick.jarvis at magicmail.mooo.com>> wrote:
>>>>>> I’d be interested to hear what different people use to provide some level of security for remote end-users such as homeworkers, and to get round NAT issues.
>>>>>>
>>>>>> We currently use OpenVPN, as this is built into the firmware of Yealink handsets (it’s a great feature, I’m not sure why more handset manufacturers don’t do this?!). The pros are that not only is it secure, but it also removes any problems with NAT for the RTP streams.
>>>>>>
>>>>>> The downsides are that it is complicated (and downright frustrating sometimes) to set up, and there are additional things to consider such as the server configuration and overheads.
>>>>>>
>>>>>> TLS/SSL with SRTP is another option, but my understanding of this is that it can cause NAT problems, with FreeSWITCH trying to initiate control channels back to the phone for inbound calls. In fact, I’ve always had problems with getting phones to work when behind NAT anyway, even without SSL/TLS. STUN can be used to ascertain the IP, but how do you handle situations where multiple handsets are behind NAT - you can’t open all RTP ports to all handsets at once?!!
>>>>>>
>>>>>> Would be very interested to hear thoughts and methods on these points.
>>>>>>
>>>>>> Thanks
>>>>>> R
>>>>>> _________________________________________________________________________
>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>>>>> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
>>>>>>
>>>>>> Official FreeSWITCH Sites
>>>>>> http://www.freeswitch.org <http://www.freeswitch.org/>
>>>>>> http://confluence.freeswitch.org <http://confluence.freeswitch.org/>
>>>>>> http://www.cluecon.com <http://www.cluecon.com/>
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>>>>> http://www.freeswitch.org <http://www.freeswitch.org/>_________________________________________________________________________
>>>>>> Professional FreeSWITCH Consulting Services:
>>>>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>>>>> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
>>>>>>
>>>>>> Official FreeSWITCH Sites
>>>>>> http://www.freeswitch.org <http://www.freeswitch.org/>
>>>>>> http://confluence.freeswitch.org <http://confluence.freeswitch.org/>
>>>>>> http://www.cluecon.com <http://www.cluecon.com/>
>>>>>>
>>>>>> FreeSWITCH-users mailing list
>>>>>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>>>>> http://www.freeswitch.org <http://www.freeswitch.org/>_________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>>>> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org <http://www.freeswitch.org/>
>>>>> http://confluence.freeswitch.org <http://confluence.freeswitch.org/>
>>>>> http://www.cluecon.com <http://www.cluecon.com/>
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>>>> http://www.freeswitch.org <http://www.freeswitch.org/>_________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services:
>>>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>>>> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
>>>>>
>>>>> Official FreeSWITCH Sites
>>>>> http://www.freeswitch.org <http://www.freeswitch.org/>
>>>>> http://confluence.freeswitch.org <http://confluence.freeswitch.org/>
>>>>> http://www.cluecon.com <http://www.cluecon.com/>
>>>>>
>>>>> FreeSWITCH-users mailing list
>>>>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>>>> http://www.freeswitch.org <http://www.freeswitch.org/>
>>>> _________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services:
>>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>>> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
>>>>
>>>> Official FreeSWITCH Sites
>>>> http://www.freeswitch.org <http://www.freeswitch.org/>
>>>> http://confluence.freeswitch.org <http://confluence.freeswitch.org/>
>>>> http://www.cluecon.com <http://www.cluecon.com/>
>>>>
>>>> FreeSWITCH-users mailing list
>>>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>>> http://www.freeswitch.org <http://www.freeswitch.org/>
>>> _________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org <http://www.freeswitch.org/>
>>> http://confluence.freeswitch.org <http://confluence.freeswitch.org/>
>>> http://www.cluecon.com <http://www.cluecon.com/>
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>> http://www.freeswitch.org <http://www.freeswitch.org/>_________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services:
>>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>>> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
>>>
>>> Official FreeSWITCH Sites
>>> http://www.freeswitch.org <http://www.freeswitch.org/>
>>> http://confluence.freeswitch.org <http://confluence.freeswitch.org/>
>>> http://www.cluecon.com <http://www.cluecon.com/>
>>>
>>> FreeSWITCH-users mailing list
>>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>>> http://www.freeswitch.org <http://www.freeswitch.org/>
>> _________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org <http://www.freeswitch.org/>
>> http://confluence.freeswitch.org <http://confluence.freeswitch.org/>
>> http://www.cluecon.com <http://www.cluecon.com/>
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>> http://www.freeswitch.org <http://www.freeswitch.org/>_________________________________________________________________________
>> Professional FreeSWITCH Consulting Services:
>> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
>> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
>>
>> Official FreeSWITCH Sites
>> http://www.freeswitch.org <http://www.freeswitch.org/>
>> http://confluence.freeswitch.org <http://confluence.freeswitch.org/>
>> http://www.cluecon.com <http://www.cluecon.com/>
>>
>> FreeSWITCH-users mailing list
>> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users>
>> http://www.freeswitch.org <http://www.freeswitch.org/>
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org <mailto:consulting at freeswitch.org>
> http://www.freeswitchsolutions.com <http://www.freeswitchsolutions.com/>
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org <http://www.freeswitch.org/>
> http://confluence.freeswitch.org <http://confluence.freeswitch.org/>
> http://www.cluecon.com <http://www.cluecon.com/>
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org <mailto:FreeSWITCH-users at lists.freeswitch.org>
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users <http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users <http://lists.freeswitch.org/mailman/options/freeswitch-users>
> http://www.freeswitch.org <http://www.freeswitch.org/>_________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
>
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://confluence.freeswitch.org
> http://www.cluecon.com
>
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20160704/ab058151/attachment-0001.html
Join us at ClueCon 2016 Aug 8-12, 2016
More information about the FreeSWITCH-users
mailing list