Hi All, I wanted to share my findings with everyone on the list in case anyone runs into the same issue I had as it what a nightmare to find. A special thanks must be given to FreeSWITCH Consulting, especially Eliot who assisted in debugging and giving me some fresh ideas to look at. For some time now I have struggled to upgrade my servers to 1.2 stable from 1.0.head (git-e3698c2 2012-03-23 01-21-55 +0100). The issue I always faced was is that after I upgraded the servers it would run happily for the next 5-10 hours but for no reason FreeSWITCH would become unresponsive and stop accepting registrations. If you ran command like show channels, status etc the command would just freeze and there would be no response. Typically the freeze would occur when the servers were practically idle, looking at the logs you would see: [WARNING] sofia_reg.c:1524 SIP auth challenge (REGISTER) on sofia profile 'internal' for [1050@domain.com] from ip x.x.x.x [WARNING] sofia.c:1639 MSG Thread 2 Started [WARNING] sofia.c:1639 MSG Thread 1 Started In attempt to resolve it I tried new hardware, changed from Centos to Debian, MySQL to Postgres, nothing I seemed to do fixed the issue. I started to run some network analysis on the server and what I noticed is about 20 minutes before the server freeze the traffic would spike from 100kbps to around 400kbps, on closer inspection I noticed that the 300kbps spike was actually between the server and a single IP on port 5060. Analysing the packets (tcpdump -s0 -A port 5060) I could see that one registration was sending SUBSCRIBE events at the rate of about 300-400 per second and FreeSWITCH was sending a NOTIFY back for each SUBSCRIBE, once the server froze the traffic would stop which is what made it hard to find as you had to be looking at the network traffic before the freeze. Once I knew the user causing the increased traffic flow I was able to work out that the user agent causing this was Zoiper (rev. 11137). It would appear Zoiper would get into a loop with the server and keep sending SUBSCRIBE packets for the message waiting indicator. I have spoken to Zoiper who have taken the issue on board and informed me it does not happen on Zoiper 3, but does happen on Zoiper 2, however they plan to fix it in the next release of Zoiper 2. They suggested for the moment to disable the MWI Subscribe option by doing "Go to the preferences -> select your sip account -> enable "show advanced options" andchange the mwi subscribe drop down to "disabled" The strange thing with all this is that the same Zoiper client never had a problem on version 1.0.head (git-e3698c2 2012-03-23 01-21-55 +0100) but as soon as you upgraded to 1.2 or even the current git versions you would find that FreeSWITCH and Zoiper would get into a SUBSCRIBE/NOTIFY Loop. I hope this information is of some help to someone else in the future and is able to ensure they do not have to go through the same nightmare I did. If anyone has any suggestions on ways to limit the SUBSCRIBE/NOTIFY responses so that you can avoid this happening in the future I would be very interested to hear, even if you have any other feedback that might have assisted in finding this issue quicker I would be very interested. Regards, Ashley Breeden URL Networks Pty Ltd WWW: http://www.url.net.au Twitter: @urlnetworks
I checked the code and found the problem. Zoiper was not happy to get a NOTIFY with Expires: 0 after sending a SUBSCRIBE with Expires: 0 I added a patch to the code to skip responding with such a NOTIFY when zoiper is present. This is not a real fix but its better than having everyone's box die. On Thu, Sep 12, 2013 at 9:40 AM, Ashley Breeden <ash@url.net.au> wrote:
Hi All,
I wanted to share my findings with everyone on the list in case anyone runs into the same issue I had as it what a nightmare to find. A special thanks must be given to FreeSWITCH Consulting, especially Eliot who assisted in debugging and giving me some fresh ideas to look at.
For some time now I have struggled to upgrade my servers to 1.2 stable from 1.0.head (git-e3698c2 2012-03-23 01-21-55 +0100). The issue I always faced was is that after I upgraded the servers it would run happily for the next 5-10 hours but for no reason FreeSWITCH would become unresponsive and stop accepting registrations. If you ran command like show channels, status etc the command would just freeze and there would be no response.
Typically the freeze would occur when the servers were practically idle, looking at the logs you would see:
[WARNING] sofia_reg.c:1524 SIP auth challenge (REGISTER) on sofia profile 'internal' for [1050@domain.com] from ip x.x.x.x [WARNING] sofia.c:1639 MSG Thread 2 Started [WARNING] sofia.c:1639 MSG Thread 1 Started
In attempt to resolve it I tried new hardware, changed from Centos to Debian, MySQL to Postgres, nothing I seemed to do fixed the issue.
I started to run some network analysis on the server and what I noticed is about 20 minutes before the server freeze the traffic would spike from 100kbps to around 400kbps, on closer inspection I noticed that the 300kbps spike was actually between the server and a single IP on port 5060. Analysing the packets (tcpdump -s0 -A port 5060) I could see that one registration was sending SUBSCRIBE events at the rate of about 300-400 per second and FreeSWITCH was sending a NOTIFY back for each SUBSCRIBE, once the server froze the traffic would stop which is what made it hard to find as you had to be looking at the network traffic before the freeze. Once I knew the user causing the increased traffic flow I was able to work out that the user agent causing this was Zoiper (rev. 11137). It would appear Zoiper would get into a loop with the server and keep sending SUBSCRIBE packets for the message waiting indicator.
I have spoken to Zoiper who have taken the issue on board and informed me it does not happen on Zoiper 3, but does happen on Zoiper 2, however they plan to fix it in the next release of Zoiper 2. They suggested for the moment to disable the MWI Subscribe option by doing "Go to the preferences -> select your sip account -> enable "show advanced options" andchange the mwi subscribe drop down to "disabled"
The strange thing with all this is that the same Zoiper client never had a problem on version 1.0.head (git-e3698c2 2012-03-23 01-21-55 +0100) but as soon as you upgraded to 1.2 or even the current git versions you would find that FreeSWITCH and Zoiper would get into a SUBSCRIBE/NOTIFY Loop.
I hope this information is of some help to someone else in the future and is able to ensure they do not have to go through the same nightmare I did.
If anyone has any suggestions on ways to limit the SUBSCRIBE/NOTIFY responses so that you can avoid this happening in the future I would be very interested to hear, even if you have any other feedback that might have assisted in finding this issue quicker I would be very interested.
Regards,
*Ashley Breeden* URL Networks Pty Ltd *WWW:* http://www.url.net.au *Twitter: *@urlnetworks
* *
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Hi Anthony, Thanks for doing that so quickly, greatly appreciated. Is it possible to have this patch added to the next 1.2 stable release? Ash. On 13/09/2013, at 1:23 AM, Anthony Minessale <anthony.minessale@gmail.com> wrote:
I checked the code and found the problem. Zoiper was not happy to get a NOTIFY with Expires: 0 after sending a SUBSCRIBE with Expires: 0 I added a patch to the code to skip responding with such a NOTIFY when zoiper is present.
This is not a real fix but its better than having everyone's box die.
On Thu, Sep 12, 2013 at 9:40 AM, Ashley Breeden <ash@url.net.au> wrote: Hi All,
I wanted to share my findings with everyone on the list in case anyone runs into the same issue I had as it what a nightmare to find. A special thanks must be given to FreeSWITCH Consulting, especially Eliot who assisted in debugging and giving me some fresh ideas to look at.
For some time now I have struggled to upgrade my servers to 1.2 stable from 1.0.head (git-e3698c2 2012-03-23 01-21-55 +0100). The issue I always faced was is that after I upgraded the servers it would run happily for the next 5-10 hours but for no reason FreeSWITCH would become unresponsive and stop accepting registrations. If you ran command like show channels, status etc the command would just freeze and there would be no response.
Typically the freeze would occur when the servers were practically idle, looking at the logs you would see:
[WARNING] sofia_reg.c:1524 SIP auth challenge (REGISTER) on sofia profile 'internal' for [1050@domain.com] from ip x.x.x.x [WARNING] sofia.c:1639 MSG Thread 2 Started [WARNING] sofia.c:1639 MSG Thread 1 Started
In attempt to resolve it I tried new hardware, changed from Centos to Debian, MySQL to Postgres, nothing I seemed to do fixed the issue.
I started to run some network analysis on the server and what I noticed is about 20 minutes before the server freeze the traffic would spike from 100kbps to around 400kbps, on closer inspection I noticed that the 300kbps spike was actually between the server and a single IP on port 5060. Analysing the packets (tcpdump -s0 -A port 5060) I could see that one registration was sending SUBSCRIBE events at the rate of about 300-400 per second and FreeSWITCH was sending a NOTIFY back for each SUBSCRIBE, once the server froze the traffic would stop which is what made it hard to find as you had to be looking at the network traffic before the freeze. Once I knew the user causing the increased traffic flow I was able to work out that the user agent causing this was Zoiper (rev. 11137). It would appear Zoiper would get into a loop with the server and keep sending SUBSCRIBE packets for the message waiting indicator.
I have spoken to Zoiper who have taken the issue on board and informed me it does not happen on Zoiper 3, but does happen on Zoiper 2, however they plan to fix it in the next release of Zoiper 2. They suggested for the moment to disable the MWI Subscribe option by doing "Go to the preferences -> select your sip account -> enable "show advanced options" andchange the mwi subscribe drop down to "disabled"
The strange thing with all this is that the same Zoiper client never had a problem on version 1.0.head (git-e3698c2 2012-03-23 01-21-55 +0100) but as soon as you upgraded to 1.2 or even the current git versions you would find that FreeSWITCH and Zoiper would get into a SUBSCRIBE/NOTIFY Loop.
I hope this information is of some help to someone else in the future and is able to ensure they do not have to go through the same nightmare I did.
If anyone has any suggestions on ways to limit the SUBSCRIBE/NOTIFY responses so that you can avoid this happening in the future I would be very interested to hear, even if you have any other feedback that might have assisted in finding this issue quicker I would be very interested.
Regards,
Ashley Breeden URL Networks Pty Ltd WWW: http://www.url.net.au Twitter: @urlnetworks
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900 _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
yes On Thu, Sep 12, 2013 at 9:50 PM, Ashley Breeden <ash@url.net.au> wrote:
Hi Anthony,
Thanks for doing that so quickly, greatly appreciated.
Is it possible to have this patch added to the next 1.2 stable release?
Ash.
On 13/09/2013, at 1:23 AM, Anthony Minessale <anthony.minessale@gmail.com> wrote:
I checked the code and found the problem. Zoiper was not happy to get a NOTIFY with Expires: 0 after sending a SUBSCRIBE with Expires: 0 I added a patch to the code to skip responding with such a NOTIFY when zoiper is present.
This is not a real fix but its better than having everyone's box die.
On Thu, Sep 12, 2013 at 9:40 AM, Ashley Breeden <ash@url.net.au> wrote:
Hi All,
I wanted to share my findings with everyone on the list in case anyone runs into the same issue I had as it what a nightmare to find. A special thanks must be given to FreeSWITCH Consulting, especially Eliot who assisted in debugging and giving me some fresh ideas to look at.
For some time now I have struggled to upgrade my servers to 1.2 stable from 1.0.head (git-e3698c2 2012-03-23 01-21-55 +0100). The issue I always faced was is that after I upgraded the servers it would run happily for the next 5-10 hours but for no reason FreeSWITCH would become unresponsive and stop accepting registrations. If you ran command like show channels, status etc the command would just freeze and there would be no response.
Typically the freeze would occur when the servers were practically idle, looking at the logs you would see:
[WARNING] sofia_reg.c:1524 SIP auth challenge (REGISTER) on sofia profile 'internal' for [1050@domain.com] from ip x.x.x.x [WARNING] sofia.c:1639 MSG Thread 2 Started [WARNING] sofia.c:1639 MSG Thread 1 Started
In attempt to resolve it I tried new hardware, changed from Centos to Debian, MySQL to Postgres, nothing I seemed to do fixed the issue.
I started to run some network analysis on the server and what I noticed is about 20 minutes before the server freeze the traffic would spike from 100kbps to around 400kbps, on closer inspection I noticed that the 300kbps spike was actually between the server and a single IP on port 5060. Analysing the packets (tcpdump -s0 -A port 5060) I could see that one registration was sending SUBSCRIBE events at the rate of about 300-400 per second and FreeSWITCH was sending a NOTIFY back for each SUBSCRIBE, once the server froze the traffic would stop which is what made it hard to find as you had to be looking at the network traffic before the freeze. Once I knew the user causing the increased traffic flow I was able to work out that the user agent causing this was Zoiper (rev. 11137). It would appear Zoiper would get into a loop with the server and keep sending SUBSCRIBE packets for the message waiting indicator.
I have spoken to Zoiper who have taken the issue on board and informed me it does not happen on Zoiper 3, but does happen on Zoiper 2, however they plan to fix it in the next release of Zoiper 2. They suggested for the moment to disable the MWI Subscribe option by doing "Go to the preferences -> select your sip account -> enable "show advanced options" andchange the mwi subscribe drop down to "disabled"
The strange thing with all this is that the same Zoiper client never had a problem on version 1.0.head (git-e3698c2 2012-03-23 01-21-55 +0100) but as soon as you upgraded to 1.2 or even the current git versions you would find that FreeSWITCH and Zoiper would get into a SUBSCRIBE/NOTIFY Loop.
I hope this information is of some help to someone else in the future and is able to ensure they do not have to go through the same nightmare I did.
If anyone has any suggestions on ways to limit the SUBSCRIBE/NOTIFY responses so that you can avoid this happening in the future I would be very interested to hear, even if you have any other feedback that might have assisted in finding this issue quicker I would be very interested.
Regards,
*Ashley Breeden* URL Networks Pty Ltd *WWW:* http://www.url.net.au *Twitter: *@urlnetworks
* *
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900 _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
Hi, Thanks a lot for this bug reporting. We need more bug/problems solved report like this. It is very interesting and important to find help when we try to solve similar bugs and find people who report a clear and factual solution to a problem and not only bug or asking help with big thread with a lot a suggestion... Thanks a lot and a nice work to solve this..!! :). Regards, 2013/9/13 Anthony Minessale <anthony.minessale@gmail.com>
yes
On Thu, Sep 12, 2013 at 9:50 PM, Ashley Breeden <ash@url.net.au> wrote:
Hi Anthony,
Thanks for doing that so quickly, greatly appreciated.
Is it possible to have this patch added to the next 1.2 stable release?
Ash.
On 13/09/2013, at 1:23 AM, Anthony Minessale <anthony.minessale@gmail.com> wrote:
I checked the code and found the problem. Zoiper was not happy to get a NOTIFY with Expires: 0 after sending a SUBSCRIBE with Expires: 0 I added a patch to the code to skip responding with such a NOTIFY when zoiper is present.
This is not a real fix but its better than having everyone's box die.
On Thu, Sep 12, 2013 at 9:40 AM, Ashley Breeden <ash@url.net.au> wrote:
Hi All,
I wanted to share my findings with everyone on the list in case anyone runs into the same issue I had as it what a nightmare to find. A special thanks must be given to FreeSWITCH Consulting, especially Eliot who assisted in debugging and giving me some fresh ideas to look at.
For some time now I have struggled to upgrade my servers to 1.2 stable from 1.0.head (git-e3698c2 2012-03-23 01-21-55 +0100). The issue I always faced was is that after I upgraded the servers it would run happily for the next 5-10 hours but for no reason FreeSWITCH would become unresponsive and stop accepting registrations. If you ran command like show channels, status etc the command would just freeze and there would be no response.
Typically the freeze would occur when the servers were practically idle, looking at the logs you would see:
[WARNING] sofia_reg.c:1524 SIP auth challenge (REGISTER) on sofia profile 'internal' for [1050@domain.com] from ip x.x.x.x [WARNING] sofia.c:1639 MSG Thread 2 Started [WARNING] sofia.c:1639 MSG Thread 1 Started
In attempt to resolve it I tried new hardware, changed from Centos to Debian, MySQL to Postgres, nothing I seemed to do fixed the issue.
I started to run some network analysis on the server and what I noticed is about 20 minutes before the server freeze the traffic would spike from 100kbps to around 400kbps, on closer inspection I noticed that the 300kbps spike was actually between the server and a single IP on port 5060. Analysing the packets (tcpdump -s0 -A port 5060) I could see that one registration was sending SUBSCRIBE events at the rate of about 300-400 per second and FreeSWITCH was sending a NOTIFY back for each SUBSCRIBE, once the server froze the traffic would stop which is what made it hard to find as you had to be looking at the network traffic before the freeze. Once I knew the user causing the increased traffic flow I was able to work out that the user agent causing this was Zoiper (rev. 11137). It would appear Zoiper would get into a loop with the server and keep sending SUBSCRIBE packets for the message waiting indicator.
I have spoken to Zoiper who have taken the issue on board and informed me it does not happen on Zoiper 3, but does happen on Zoiper 2, however they plan to fix it in the next release of Zoiper 2. They suggested for the moment to disable the MWI Subscribe option by doing "Go to the preferences -> select your sip account -> enable "show advanced options" andchange the mwi subscribe drop down to "disabled"
The strange thing with all this is that the same Zoiper client never had a problem on version 1.0.head (git-e3698c2 2012-03-23 01-21-55 +0100) but as soon as you upgraded to 1.2 or even the current git versions you would find that FreeSWITCH and Zoiper would get into a SUBSCRIBE/NOTIFY Loop.
I hope this information is of some help to someone else in the future and is able to ensure they do not have to go through the same nightmare I did.
If anyone has any suggestions on ways to limit the SUBSCRIBE/NOTIFY responses so that you can avoid this happening in the future I would be very interested to hear, even if you have any other feedback that might have assisted in finding this issue quicker I would be very interested.
Regards,
*Ashley Breeden* URL Networks Pty Ltd *WWW:* http://www.url.net.au *Twitter: *@urlnetworks
* *
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900 _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (3)
-
Anthony Minessale
-
Ashley Breeden
-
Regis M