Hey all, I've previously been using mod_httapi with no problems - so I'm not quite at the point of calling this a FS bug yet (although the behavior is certainly unexpected). I recently set up a t2.nano instance in AWS - which are 512m memory. The purpose of this instance is just to test FS configurations. Generally speaking it works fine, with one exception: Relatively frequently (throwing out a number of 15% here which may not be right but it's in the right ball park), the response that FS receives from my endpoint is cut off at some point throughout the string. When it happens, the last character that FS receives exactly matches the final character of a single TCP packet. However, after running tcpdump on the machine I do see the full response coming back (split across two packets). Now, I had not seen this problem until launching the t2.nano instance. Three things have changed since then: 1. FS 1.6.7 -> 1.6.9 (though I don't think this matters, as I don't see any relevant httapi changes) 2. Moved into a docker container 3. Reduction in instance specs (2 cores + 2GB ram down to 1 core + 512mb RAM). I should note it's also possible that the dependencies (such as libcurl) have changed, though I searched through libcurls bug list and couldn't find anything related to this. Because of item #3 my first assumption was a memory allocation issue, though FS reports nothing about this in the logs and when it happens I can see that the system still has 100+MB free. I thought it would help to post here first to see if the general consensus is "there's no way FS will run correctly on 512mb of ram" or something along those lines. But please let me know if I should make a bug for this. Best, Colin
Is the content length on the response consistent with the bytes in both packets together? is there any delay between the 2 packets. I might have to see the pcap to really comment. On Mon, Jul 11, 2016 at 4:13 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Hey all,
I've previously been using mod_httapi with no problems - so I'm not quite at the point of calling this a FS bug yet (although the behavior is certainly unexpected).
I recently set up a t2.nano instance in AWS - which are 512m memory. The purpose of this instance is just to test FS configurations. Generally speaking it works fine, with one exception:
Relatively frequently (throwing out a number of 15% here which may not be right but it's in the right ball park), the response that FS receives from my endpoint is cut off at some point throughout the string. When it happens, the last character that FS receives exactly matches the final character of a single TCP packet. However, after running tcpdump on the machine I do see the full response coming back (split across two packets).
Now, I had not seen this problem until launching the t2.nano instance. Three things have changed since then:
1. FS 1.6.7 -> 1.6.9 (though I don't think this matters, as I don't see any relevant httapi changes) 2. Moved into a docker container 3. Reduction in instance specs (2 cores + 2GB ram down to 1 core + 512mb RAM).
I should note it's also possible that the dependencies (such as libcurl) have changed, though I searched through libcurls bug list and couldn't find anything related to this.
Because of item #3 my first assumption was a memory allocation issue, though FS reports nothing about this in the logs and when it happens I can see that the system still has 100+MB free.
I thought it would help to post here first to see if the general consensus is "there's no way FS will run correctly on 512mb of ram" or something along those lines. But please let me know if I should make a bug for this.
Best, Colin
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
Yeah, the content-length appears to be correct. I didn't look at the delay between the packets, but I can check on that (is there a low default read timeout in httapi?) What ends up happening is that FS receives a partial XML response, processes all of the complete instructions, and then re-requests the same URL again. Sometimes on the second time it works. In the worst case, I have seen FS loop over the request 6 times before eventually it receives a full result. Also, for what it's worth, I build FS off of master for a different issue I was trying to fix, and consequently ran it outside of docker (on Ubuntu Xenial) and the same issue happened. So, that rules out anything specific to that FS version and/or docker. Most likely curl issues also (unless Xenial and the libcurl in the jessie container both had the bug). I'll try to grab a pcap and send it your way. On Tue, Jul 12, 2016 at 10:55 AM Anthony Minessale < anthony.minessale@gmail.com> wrote:
Is the content length on the response consistent with the bytes in both packets together? is there any delay between the 2 packets. I might have to see the pcap to really comment.
On Mon, Jul 11, 2016 at 4:13 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Hey all,
I've previously been using mod_httapi with no problems - so I'm not quite at the point of calling this a FS bug yet (although the behavior is certainly unexpected).
I recently set up a t2.nano instance in AWS - which are 512m memory. The purpose of this instance is just to test FS configurations. Generally speaking it works fine, with one exception:
Relatively frequently (throwing out a number of 15% here which may not be right but it's in the right ball park), the response that FS receives from my endpoint is cut off at some point throughout the string. When it happens, the last character that FS receives exactly matches the final character of a single TCP packet. However, after running tcpdump on the machine I do see the full response coming back (split across two packets).
Now, I had not seen this problem until launching the t2.nano instance. Three things have changed since then:
1. FS 1.6.7 -> 1.6.9 (though I don't think this matters, as I don't see any relevant httapi changes) 2. Moved into a docker container 3. Reduction in instance specs (2 cores + 2GB ram down to 1 core + 512mb RAM).
I should note it's also possible that the dependencies (such as libcurl) have changed, though I searched through libcurls bug list and couldn't find anything related to this.
Because of item #3 my first assumption was a memory allocation issue, though FS reports nothing about this in the logs and when it happens I can see that the system still has 100+MB free.
I thought it would help to post here first to see if the general consensus is "there's no way FS will run correctly on 512mb of ram" or something along those lines. But please let me know if I should make a bug for this.
Best, Colin
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
https://freeswitch.org/jira/browse/FS-9345 Try master On Tue, Jul 12, 2016 at 10:00 AM, Colin Morelli <colin.morelli@gmail.com> wrote:
Yeah, the content-length appears to be correct. I didn't look at the delay between the packets, but I can check on that (is there a low default read timeout in httapi?)
What ends up happening is that FS receives a partial XML response, processes all of the complete instructions, and then re-requests the same URL again. Sometimes on the second time it works. In the worst case, I have seen FS loop over the request 6 times before eventually it receives a full result.
Also, for what it's worth, I build FS off of master for a different issue I was trying to fix, and consequently ran it outside of docker (on Ubuntu Xenial) and the same issue happened. So, that rules out anything specific to that FS version and/or docker. Most likely curl issues also (unless Xenial and the libcurl in the jessie container both had the bug).
I'll try to grab a pcap and send it your way.
On Tue, Jul 12, 2016 at 10:55 AM Anthony Minessale < anthony.minessale@gmail.com> wrote:
Is the content length on the response consistent with the bytes in both packets together? is there any delay between the 2 packets. I might have to see the pcap to really comment.
On Mon, Jul 11, 2016 at 4:13 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Hey all,
I've previously been using mod_httapi with no problems - so I'm not quite at the point of calling this a FS bug yet (although the behavior is certainly unexpected).
I recently set up a t2.nano instance in AWS - which are 512m memory. The purpose of this instance is just to test FS configurations. Generally speaking it works fine, with one exception:
Relatively frequently (throwing out a number of 15% here which may not be right but it's in the right ball park), the response that FS receives from my endpoint is cut off at some point throughout the string. When it happens, the last character that FS receives exactly matches the final character of a single TCP packet. However, after running tcpdump on the machine I do see the full response coming back (split across two packets).
Now, I had not seen this problem until launching the t2.nano instance. Three things have changed since then:
1. FS 1.6.7 -> 1.6.9 (though I don't think this matters, as I don't see any relevant httapi changes) 2. Moved into a docker container 3. Reduction in instance specs (2 cores + 2GB ram down to 1 core + 512mb RAM).
I should note it's also possible that the dependencies (such as libcurl) have changed, though I searched through libcurls bug list and couldn't find anything related to this.
Because of item #3 my first assumption was a memory allocation issue, though FS reports nothing about this in the logs and when it happens I can see that the system still has 100+MB free.
I thought it would help to post here first to see if the general consensus is "there's no way FS will run correctly on 512mb of ram" or something along those lines. But please let me know if I should make a bug for this.
Best, Colin
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
Anthony, Just tried a number of test calls on master and am not seeing the issue anymore. When I was going through the httapi source I saw that and thought "this might be an issue" but for some reason I convinced myself that httapi was using a different callback. In any case, this must be some kind of absolute record-breaking response time! Thanks for the awesome response. Best, Colin On Tue, Jul 12, 2016 at 11:58 AM Anthony Minessale < anthony.minessale@gmail.com> wrote:
https://freeswitch.org/jira/browse/FS-9345
Try master
On Tue, Jul 12, 2016 at 10:00 AM, Colin Morelli <colin.morelli@gmail.com> wrote:
Yeah, the content-length appears to be correct. I didn't look at the delay between the packets, but I can check on that (is there a low default read timeout in httapi?)
What ends up happening is that FS receives a partial XML response, processes all of the complete instructions, and then re-requests the same URL again. Sometimes on the second time it works. In the worst case, I have seen FS loop over the request 6 times before eventually it receives a full result.
Also, for what it's worth, I build FS off of master for a different issue I was trying to fix, and consequently ran it outside of docker (on Ubuntu Xenial) and the same issue happened. So, that rules out anything specific to that FS version and/or docker. Most likely curl issues also (unless Xenial and the libcurl in the jessie container both had the bug).
I'll try to grab a pcap and send it your way.
On Tue, Jul 12, 2016 at 10:55 AM Anthony Minessale < anthony.minessale@gmail.com> wrote:
Is the content length on the response consistent with the bytes in both packets together? is there any delay between the 2 packets. I might have to see the pcap to really comment.
On Mon, Jul 11, 2016 at 4:13 PM, Colin Morelli <colin.morelli@gmail.com> wrote:
Hey all,
I've previously been using mod_httapi with no problems - so I'm not quite at the point of calling this a FS bug yet (although the behavior is certainly unexpected).
I recently set up a t2.nano instance in AWS - which are 512m memory. The purpose of this instance is just to test FS configurations. Generally speaking it works fine, with one exception:
Relatively frequently (throwing out a number of 15% here which may not be right but it's in the right ball park), the response that FS receives from my endpoint is cut off at some point throughout the string. When it happens, the last character that FS receives exactly matches the final character of a single TCP packet. However, after running tcpdump on the machine I do see the full response coming back (split across two packets).
Now, I had not seen this problem until launching the t2.nano instance. Three things have changed since then:
1. FS 1.6.7 -> 1.6.9 (though I don't think this matters, as I don't see any relevant httapi changes) 2. Moved into a docker container 3. Reduction in instance specs (2 cores + 2GB ram down to 1 core + 512mb RAM).
I should note it's also possible that the dependencies (such as libcurl) have changed, though I searched through libcurls bug list and couldn't find anything related to this.
Because of item #3 my first assumption was a memory allocation issue, though FS reports nothing about this in the logs and when it happens I can see that the system still has 100+MB free.
I thought it would help to post here first to see if the general consensus is "there's no way FS will run correctly on 512mb of ram" or something along those lines. But please let me know if I should make a bug for this.
Best, Colin
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Sure no problem. You should come see us at ClueCon next month! On Tue, Jul 12, 2016 at 11:05 AM, Colin Morelli <colin.morelli@gmail.com> wrote:
Anthony,
Just tried a number of test calls on master and am not seeing the issue anymore. When I was going through the httapi source I saw that and thought "this might be an issue" but for some reason I convinced myself that httapi was using a different callback.
In any case, this must be some kind of absolute record-breaking response time!
Thanks for the awesome response.
Best, Colin
On Tue, Jul 12, 2016 at 11:58 AM Anthony Minessale < anthony.minessale@gmail.com> wrote:
https://freeswitch.org/jira/browse/FS-9345
Try master
On Tue, Jul 12, 2016 at 10:00 AM, Colin Morelli <colin.morelli@gmail.com> wrote:
Yeah, the content-length appears to be correct. I didn't look at the delay between the packets, but I can check on that (is there a low default read timeout in httapi?)
What ends up happening is that FS receives a partial XML response, processes all of the complete instructions, and then re-requests the same URL again. Sometimes on the second time it works. In the worst case, I have seen FS loop over the request 6 times before eventually it receives a full result.
Also, for what it's worth, I build FS off of master for a different issue I was trying to fix, and consequently ran it outside of docker (on Ubuntu Xenial) and the same issue happened. So, that rules out anything specific to that FS version and/or docker. Most likely curl issues also (unless Xenial and the libcurl in the jessie container both had the bug).
I'll try to grab a pcap and send it your way.
On Tue, Jul 12, 2016 at 10:55 AM Anthony Minessale < anthony.minessale@gmail.com> wrote:
Is the content length on the response consistent with the bytes in both packets together? is there any delay between the 2 packets. I might have to see the pcap to really comment.
On Mon, Jul 11, 2016 at 4:13 PM, Colin Morelli <colin.morelli@gmail.com
wrote:
Hey all,
I've previously been using mod_httapi with no problems - so I'm not quite at the point of calling this a FS bug yet (although the behavior is certainly unexpected).
I recently set up a t2.nano instance in AWS - which are 512m memory. The purpose of this instance is just to test FS configurations. Generally speaking it works fine, with one exception:
Relatively frequently (throwing out a number of 15% here which may not be right but it's in the right ball park), the response that FS receives from my endpoint is cut off at some point throughout the string. When it happens, the last character that FS receives exactly matches the final character of a single TCP packet. However, after running tcpdump on the machine I do see the full response coming back (split across two packets).
Now, I had not seen this problem until launching the t2.nano instance. Three things have changed since then:
1. FS 1.6.7 -> 1.6.9 (though I don't think this matters, as I don't see any relevant httapi changes) 2. Moved into a docker container 3. Reduction in instance specs (2 cores + 2GB ram down to 1 core + 512mb RAM).
I should note it's also possible that the dependencies (such as libcurl) have changed, though I searched through libcurls bug list and couldn't find anything related to this.
Because of item #3 my first assumption was a memory allocation issue, though FS reports nothing about this in the logs and when it happens I can see that the system still has 100+MB free.
I thought it would help to post here first to see if the general consensus is "there's no way FS will run correctly on 512mb of ram" or something along those lines. But please let me know if I should make a bug for this.
Best, Colin
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬
☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>*
ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900
https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
participants (2)
-
Anthony Minessale
-
Colin Morelli