Sofia freezing for 5 minutes then starting again
Hello, I'm having a bit of an odd problem. Intermittently, often every 2-3 days or so, Freeswitch stops replying to SIP for about 5 minutes. I can't verify if it's EXACTLY 5 minutes, but it seems to be pretty close. During this time, no new registrations or invites can happen, but existing calls stay connected for at least a minute or two. In the logs, you can see calls slowly hanging up with "NORMAL_CLEARING". In 5 minutes, everything starts up again with no word about it at all in the logs. When calls resume, I notice that the number of sessions returned by the status command is one higher than the actual number sessions returned by show channels, or by looking in the database. Every time this happens, the discrepancy increases by one. The interruption happens on all SIP profiles, but calls originated from the socket API still work, insofar as they return with PROGRESS_TIMEOUT since the profiles are still running, but stuck. We are using ODBC/MySQL for the core database, and the database server only runs this database and some basic PHP/xml-curl stuff. We have 416 endpoints registered, and usually sit at about 30 sessions during the day. This never happens at night, only during busier times, but not necessarily busy hour. I'm running on FreeBSD 8.2-RELEASE AMD 64(2 XEON cores, 4G ram) Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500) Yes, I know it's old and I'm trying to upgrade, but I'm still having some problems getting all my phones to work properly with 1.2 stable. This is a production system, so I can't just blindly put out the newest release. Mostly, I need to buy myself some time so that I can get the kinks worked out of the latest version and then upgrade the production box. I'm grateful for any insights as to what could be happening, even if a solution is just a temporary workaround. Thanks! -Tim
Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
As you've already acknowledged it's a very old version. It's possible that your issue has already been found and fixed, but if it hasn't then the code will have changed significantly since then and you'd really need to reproduce it on the latest code for it to be investigated. As some general thoughts though, are you able to spot it happening while it's happening or only afterwards? If you're able to get on the system during one of those times look at what else is happening. Is the load average/cpu usage/io high? Perhaps something's running that's blocking all access or causing very high IO. What DB backend are you using for Sofia? Is it possible that that's hanging for a moment? For example if you're running a backup on the DB that blocks all writes to the DB while Sofia is trying to update the DB that perhaps would cause this. Try running a SIP OPTIONS ping your your sofia profile from the localhost during that time, which should exclude it being any issue on the ethernet. -Steve On 22 November 2012 19:31, Tim St. Pierre <fs-list@communicatefreely.net>wrote:
Hello,
I'm having a bit of an odd problem.
Intermittently, often every 2-3 days or so, Freeswitch stops replying to SIP for about 5 minutes. I can't verify if it's EXACTLY 5 minutes, but it seems to be pretty close.
During this time, no new registrations or invites can happen, but existing calls stay connected for at least a minute or two. In the logs, you can see calls slowly hanging up with "NORMAL_CLEARING". In 5 minutes, everything starts up again with no word about it at all in the logs.
When calls resume, I notice that the number of sessions returned by the status command is one higher than the actual number sessions returned by show channels, or by looking in the database. Every time this happens, the discrepancy increases by one.
The interruption happens on all SIP profiles, but calls originated from the socket API still work, insofar as they return with PROGRESS_TIMEOUT since the profiles are still running, but stuck.
We are using ODBC/MySQL for the core database, and the database server only runs this database and some basic PHP/xml-curl stuff.
We have 416 endpoints registered, and usually sit at about 30 sessions during the day.
This never happens at night, only during busier times, but not necessarily busy hour.
I'm running on FreeBSD 8.2-RELEASE AMD 64(2 XEON cores, 4G ram)
Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
Yes, I know it's old and I'm trying to upgrade, but I'm still having some problems getting all my phones to work properly with 1.2 stable. This is a production system, so I can't just blindly put out the newest release. Mostly, I need to buy myself some time so that I can get the kinks worked out of the latest version and then upgrade the production box.
I'm grateful for any insights as to what could be happening, even if a solution is just a temporary workaround.
Thanks!
-Tim
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Hi Steven, Thanks for the suggestions. I'm hoping once I get the upgrade done it will all go away. I have watched it happen at least once. I was on the phone at the time. Console activity more or less stopped, except for a few calls hanging up. The console remains responsive, and my call wasn't dropped for at least a minute or two (media timeout?). I was able to run sofia status and other commands that use the database, so I'm assuming that the connection was still working. All our media is runs through the box, so I think things are fine on the Ethernet level. I do see higher load averages - maybe 3-4, but that's the only obvious indication. It's not taking CPU beyond 10% or so. We are using MySQL as the core DB and also as the DB backend for each sofia profile. This is connecting through ODBC of course. If I can get the other kinks worked out, then I will try 1.2 stable in production and we'll see how it goes. -Tim Steven Ayre wrote:
Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
As you've already acknowledged it's a very old version.
It's possible that your issue has already been found and fixed, but if it hasn't then the code will have changed significantly since then and you'd really need to reproduce it on the latest code for it to be investigated.
As some general thoughts though, are you able to spot it happening while it's happening or only afterwards?
If you're able to get on the system during one of those times look at what else is happening. Is the load average/cpu usage/io high? Perhaps something's running that's blocking all access or causing very high IO.
What DB backend are you using for Sofia? Is it possible that that's hanging for a moment? For example if you're running a backup on the DB that blocks all writes to the DB while Sofia is trying to update the DB that perhaps would cause this.
Try running a SIP OPTIONS ping your your sofia profile from the localhost during that time, which should exclude it being any issue on the ethernet.
-Steve
On 22 November 2012 19:31, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>> wrote:
Hello,
I'm having a bit of an odd problem.
Intermittently, often every 2-3 days or so, Freeswitch stops replying to SIP for about 5 minutes. I can't verify if it's EXACTLY 5 minutes, but it seems to be pretty close.
During this time, no new registrations or invites can happen, but existing calls stay connected for at least a minute or two. In the logs, you can see calls slowly hanging up with "NORMAL_CLEARING". In 5 minutes, everything starts up again with no word about it at all in the logs.
When calls resume, I notice that the number of sessions returned by the status command is one higher than the actual number sessions returned by show channels, or by looking in the database. Every time this happens, the discrepancy increases by one.
The interruption happens on all SIP profiles, but calls originated from the socket API still work, insofar as they return with PROGRESS_TIMEOUT since the profiles are still running, but stuck.
We are using ODBC/MySQL for the core database, and the database server only runs this database and some basic PHP/xml-curl stuff.
We have 416 endpoints registered, and usually sit at about 30 sessions during the day.
This never happens at night, only during busier times, but not necessarily busy hour.
I'm running on FreeBSD 8.2-RELEASE AMD 64(2 XEON cores, 4G ram)
Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
Yes, I know it's old and I'm trying to upgrade, but I'm still having some problems getting all my phones to work properly with 1.2 stable. This is a production system, so I can't just blindly put out the newest release. Mostly, I need to buy myself some time so that I can get the kinks worked out of the latest version and then upgrade the production box.
I'm grateful for any insights as to what could be happening, even if a solution is just a temporary workaround.
Thanks!
-Tim
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
------------------------------------------------------------------------
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Any kind of DB backup running? Or any long-running queries (innotop is great for highlighting queries that've been running a while, including on non-innodb tables). A global read lock, or queries waiting for a lock could block a db update from the sofia profile thread but still allow read-only queries (sofia status) to run. -Steve On 23 November 2012 15:32, Tim St. Pierre <fs-list@communicatefreely.net> wrote:
Hi Steven,
Thanks for the suggestions. I'm hoping once I get the upgrade done it will all go away. I have watched it happen at least once. I was on the phone at the time. Console activity more or less stopped, except for a few calls hanging up. The console remains responsive, and my call wasn't dropped for at least a minute or two (media timeout?). I was able to run sofia status and other commands that use the database, so I'm assuming that the connection was still working. All our media is runs through the box, so I think things are fine on the Ethernet level. I do see higher load averages - maybe 3-4, but that's the only obvious indication. It's not taking CPU beyond 10% or so.
We are using MySQL as the core DB and also as the DB backend for each sofia profile. This is connecting through ODBC of course.
If I can get the other kinks worked out, then I will try 1.2 stable in production and we'll see how it goes.
-Tim
Steven Ayre wrote:
Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
As you've already acknowledged it's a very old version.
It's possible that your issue has already been found and fixed, but if it hasn't then the code will have changed significantly since then and you'd really need to reproduce it on the latest code for it to be investigated.
As some general thoughts though, are you able to spot it happening while it's happening or only afterwards?
If you're able to get on the system during one of those times look at what else is happening. Is the load average/cpu usage/io high? Perhaps something's running that's blocking all access or causing very high IO.
What DB backend are you using for Sofia? Is it possible that that's hanging for a moment? For example if you're running a backup on the DB that blocks all writes to the DB while Sofia is trying to update the DB that perhaps would cause this.
Try running a SIP OPTIONS ping your your sofia profile from the localhost during that time, which should exclude it being any issue on the ethernet.
-Steve
On 22 November 2012 19:31, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>> wrote:
Hello,
I'm having a bit of an odd problem.
Intermittently, often every 2-3 days or so, Freeswitch stops replying to SIP for about 5 minutes. I can't verify if it's EXACTLY 5 minutes, but it seems to be pretty close.
During this time, no new registrations or invites can happen, but existing calls stay connected for at least a minute or two. In the logs, you can see calls slowly hanging up with "NORMAL_CLEARING". In 5 minutes, everything starts up again with no word about it at all in the logs.
When calls resume, I notice that the number of sessions returned by the status command is one higher than the actual number sessions returned by show channels, or by looking in the database. Every time this happens, the discrepancy increases by one.
The interruption happens on all SIP profiles, but calls originated from the socket API still work, insofar as they return with PROGRESS_TIMEOUT since the profiles are still running, but stuck.
We are using ODBC/MySQL for the core database, and the database server only runs this database and some basic PHP/xml-curl stuff.
We have 416 endpoints registered, and usually sit at about 30 sessions during the day.
This never happens at night, only during busier times, but not necessarily busy hour.
I'm running on FreeBSD 8.2-RELEASE AMD 64(2 XEON cores, 4G ram)
Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
Yes, I know it's old and I'm trying to upgrade, but I'm still having some problems getting all my phones to work properly with 1.2 stable. This is a production system, so I can't just blindly put out the newest release. Mostly, I need to buy myself some time so that I can get the kinks worked out of the latest version and then upgrade the production box.
I'm grateful for any insights as to what could be happening, even if a solution is just a temporary workaround.
Thanks!
-Tim
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
------------------------------------------------------------------------
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
you mentioned that you use xml_curl, if your web server hangs it may hang sofia, iirc sofia is running a single thread and it will wait for the xml_curl response before continuing to the next request. On 11/23/2012 1:13 PM, Steven Ayre wrote:
Any kind of DB backup running? Or any long-running queries (innotop is great for highlighting queries that've been running a while, including on non-innodb tables).
A global read lock, or queries waiting for a lock could block a db update from the sofia profile thread but still allow read-only queries (sofia status) to run.
-Steve
On 23 November 2012 15:32, Tim St. Pierre <fs-list@communicatefreely.net> wrote:
Hi Steven,
Thanks for the suggestions. I'm hoping once I get the upgrade done it will all go away. I have watched it happen at least once. I was on the phone at the time. Console activity more or less stopped, except for a few calls hanging up. The console remains responsive, and my call wasn't dropped for at least a minute or two (media timeout?). I was able to run sofia status and other commands that use the database, so I'm assuming that the connection was still working. All our media is runs through the box, so I think things are fine on the Ethernet level. I do see higher load averages - maybe 3-4, but that's the only obvious indication. It's not taking CPU beyond 10% or so.
We are using MySQL as the core DB and also as the DB backend for each sofia profile. This is connecting through ODBC of course.
If I can get the other kinks worked out, then I will try 1.2 stable in production and we'll see how it goes.
-Tim
Steven Ayre wrote:
Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
As you've already acknowledged it's a very old version.
It's possible that your issue has already been found and fixed, but if it hasn't then the code will have changed significantly since then and you'd really need to reproduce it on the latest code for it to be investigated.
As some general thoughts though, are you able to spot it happening while it's happening or only afterwards?
If you're able to get on the system during one of those times look at what else is happening. Is the load average/cpu usage/io high? Perhaps something's running that's blocking all access or causing very high IO.
What DB backend are you using for Sofia? Is it possible that that's hanging for a moment? For example if you're running a backup on the DB that blocks all writes to the DB while Sofia is trying to update the DB that perhaps would cause this.
Try running a SIP OPTIONS ping your your sofia profile from the localhost during that time, which should exclude it being any issue on the ethernet.
-Steve
On 22 November 2012 19:31, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>> wrote:
Hello,
I'm having a bit of an odd problem.
Intermittently, often every 2-3 days or so, Freeswitch stops replying to SIP for about 5 minutes. I can't verify if it's EXACTLY 5 minutes, but it seems to be pretty close.
During this time, no new registrations or invites can happen, but existing calls stay connected for at least a minute or two. In the logs, you can see calls slowly hanging up with "NORMAL_CLEARING". In 5 minutes, everything starts up again with no word about it at all in the logs.
When calls resume, I notice that the number of sessions returned by the status command is one higher than the actual number sessions returned by show channels, or by looking in the database. Every time this happens, the discrepancy increases by one.
The interruption happens on all SIP profiles, but calls originated from the socket API still work, insofar as they return with PROGRESS_TIMEOUT since the profiles are still running, but stuck.
We are using ODBC/MySQL for the core database, and the database server only runs this database and some basic PHP/xml-curl stuff.
We have 416 endpoints registered, and usually sit at about 30 sessions during the day.
This never happens at night, only during busier times, but not necessarily busy hour.
I'm running on FreeBSD 8.2-RELEASE AMD 64(2 XEON cores, 4G ram)
Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
Yes, I know it's old and I'm trying to upgrade, but I'm still having some problems getting all my phones to work properly with 1.2 stable. This is a production system, so I can't just blindly put out the newest release. Mostly, I need to buy myself some time so that I can get the kinks worked out of the latest version and then upgrade the production box.
I'm grateful for any insights as to what could be happening, even if a solution is just a temporary workaround.
Thanks!
-Tim
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
------------------------------------------------------------------------
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Sofia is not single threaded except for in one spot deep in libsofia,
From there, messages are handed off to a number of message queues for FS core to handle as needed...
Check to see if anything that fs is depending on is blocking on info retrieval like the databases or other areas... K On 11/25/12 9:51 AM, "Abaci" <abaci64@gmail.com> wrote:
you mentioned that you use xml_curl, if your web server hangs it may hang sofia, iirc sofia is running a single thread and it will wait for the xml_curl response before continuing to the next request.
On 11/23/2012 1:13 PM, Steven Ayre wrote:
Any kind of DB backup running? Or any long-running queries (innotop is great for highlighting queries that've been running a while, including on non-innodb tables).
A global read lock, or queries waiting for a lock could block a db update from the sofia profile thread but still allow read-only queries (sofia status) to run.
-Steve
On 23 November 2012 15:32, Tim St. Pierre <fs-list@communicatefreely.net> wrote:
Hi Steven,
Thanks for the suggestions. I'm hoping once I get the upgrade done it will all go away. I have watched it happen at least once. I was on the phone at the time. Console activity more or less stopped, except for a few calls hanging up. The console remains responsive, and my call wasn't dropped for at least a minute or two (media timeout?). I was able to run sofia status and other commands that use the database, so I'm assuming that the connection was still working. All our media is runs through the box, so I think things are fine on the Ethernet level. I do see higher load averages - maybe 3-4, but that's the only obvious indication. It's not taking CPU beyond 10% or so.
We are using MySQL as the core DB and also as the DB backend for each sofia profile. This is connecting through ODBC of course.
If I can get the other kinks worked out, then I will try 1.2 stable in production and we'll see how it goes.
-Tim
Steven Ayre wrote:
Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
As you've already acknowledged it's a very old version.
It's possible that your issue has already been found and fixed, but if it hasn't then the code will have changed significantly since then and you'd really need to reproduce it on the latest code for it to be investigated.
As some general thoughts though, are you able to spot it happening while it's happening or only afterwards?
If you're able to get on the system during one of those times look at what else is happening. Is the load average/cpu usage/io high? Perhaps something's running that's blocking all access or causing very high IO.
What DB backend are you using for Sofia? Is it possible that that's hanging for a moment? For example if you're running a backup on the DB that blocks all writes to the DB while Sofia is trying to update the DB that perhaps would cause this.
Try running a SIP OPTIONS ping your your sofia profile from the localhost during that time, which should exclude it being any issue on the ethernet.
-Steve
On 22 November 2012 19:31, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>> wrote:
Hello,
I'm having a bit of an odd problem.
Intermittently, often every 2-3 days or so, Freeswitch stops replying to SIP for about 5 minutes. I can't verify if it's EXACTLY 5 minutes, but it seems to be pretty close.
During this time, no new registrations or invites can happen, but existing calls stay connected for at least a minute or two. In the logs, you can see calls slowly hanging up with "NORMAL_CLEARING". In 5 minutes, everything starts up again with no word about it at all in the logs.
When calls resume, I notice that the number of sessions returned by the status command is one higher than the actual number sessions returned by show channels, or by looking in the database. Every time this happens, the discrepancy increases by one.
The interruption happens on all SIP profiles, but calls originated from the socket API still work, insofar as they return with PROGRESS_TIMEOUT since the profiles are still running, but stuck.
We are using ODBC/MySQL for the core database, and the database server only runs this database and some basic PHP/xml-curl stuff.
We have 416 endpoints registered, and usually sit at about 30 sessions during the day.
This never happens at night, only during busier times, but not necessarily busy hour.
I'm running on FreeBSD 8.2-RELEASE AMD 64(2 XEON cores, 4G ram)
Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
Yes, I know it's old and I'm trying to upgrade, but I'm still having some problems getting all my phones to work properly with 1.2 stable. This is a production system, so I can't just blindly put out the newest release. Mostly, I need to buy myself some time so that I can get the kinks worked out of the latest version and then upgrade the production box.
I'm grateful for any insights as to what could be happening, even if a solution is just a temporary workaround.
Thanks!
-Tim
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
------------------------------------------------------------------------
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org irc.freenode.net #freeswitch
I just remember reading somewhere on a blog something about a similar issue and this was the cause. Just found that post http://blog.godson.in/2011/06/pitfalls-to-avoid-while-using.html not sure that's the issue. either way it's a good idea to set the timeout option. On 11/25/2012 11:32 AM, Ken Rice wrote:
Sofia is not single threaded except for in one spot deep in libsofia,
From there, messages are handed off to a number of message queues for FS core to handle as needed...
Check to see if anything that fs is depending on is blocking on info retrieval like the databases or other areas...
K
On 11/25/12 9:51 AM, "Abaci" <abaci64@gmail.com> wrote:
you mentioned that you use xml_curl, if your web server hangs it may hang sofia, iirc sofia is running a single thread and it will wait for the xml_curl response before continuing to the next request.
On 11/23/2012 1:13 PM, Steven Ayre wrote:
Any kind of DB backup running? Or any long-running queries (innotop is great for highlighting queries that've been running a while, including on non-innodb tables).
A global read lock, or queries waiting for a lock could block a db update from the sofia profile thread but still allow read-only queries (sofia status) to run.
-Steve
On 23 November 2012 15:32, Tim St. Pierre <fs-list@communicatefreely.net> wrote:
Hi Steven,
Thanks for the suggestions. I'm hoping once I get the upgrade done it will all go away. I have watched it happen at least once. I was on the phone at the time. Console activity more or less stopped, except for a few calls hanging up. The console remains responsive, and my call wasn't dropped for at least a minute or two (media timeout?). I was able to run sofia status and other commands that use the database, so I'm assuming that the connection was still working. All our media is runs through the box, so I think things are fine on the Ethernet level. I do see higher load averages - maybe 3-4, but that's the only obvious indication. It's not taking CPU beyond 10% or so.
We are using MySQL as the core DB and also as the DB backend for each sofia profile. This is connecting through ODBC of course.
If I can get the other kinks worked out, then I will try 1.2 stable in production and we'll see how it goes.
-Tim
Steven Ayre wrote:
Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
As you've already acknowledged it's a very old version.
It's possible that your issue has already been found and fixed, but if it hasn't then the code will have changed significantly since then and you'd really need to reproduce it on the latest code for it to be investigated.
As some general thoughts though, are you able to spot it happening while it's happening or only afterwards?
If you're able to get on the system during one of those times look at what else is happening. Is the load average/cpu usage/io high? Perhaps something's running that's blocking all access or causing very high IO.
What DB backend are you using for Sofia? Is it possible that that's hanging for a moment? For example if you're running a backup on the DB that blocks all writes to the DB while Sofia is trying to update the DB that perhaps would cause this.
Try running a SIP OPTIONS ping your your sofia profile from the localhost during that time, which should exclude it being any issue on the ethernet.
-Steve
On 22 November 2012 19:31, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>> wrote:
Hello,
I'm having a bit of an odd problem.
Intermittently, often every 2-3 days or so, Freeswitch stops replying to SIP for about 5 minutes. I can't verify if it's EXACTLY 5 minutes, but it seems to be pretty close.
During this time, no new registrations or invites can happen, but existing calls stay connected for at least a minute or two. In the logs, you can see calls slowly hanging up with "NORMAL_CLEARING". In 5 minutes, everything starts up again with no word about it at all in the logs.
When calls resume, I notice that the number of sessions returned by the status command is one higher than the actual number sessions returned by show channels, or by looking in the database. Every time this happens, the discrepancy increases by one.
The interruption happens on all SIP profiles, but calls originated from the socket API still work, insofar as they return with PROGRESS_TIMEOUT since the profiles are still running, but stuck.
We are using ODBC/MySQL for the core database, and the database server only runs this database and some basic PHP/xml-curl stuff.
We have 416 endpoints registered, and usually sit at about 30 sessions during the day.
This never happens at night, only during busier times, but not necessarily busy hour.
I'm running on FreeBSD 8.2-RELEASE AMD 64(2 XEON cores, 4G ram)
Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
Yes, I know it's old and I'm trying to upgrade, but I'm still having some problems getting all my phones to work properly with 1.2 stable. This is a production system, so I can't just blindly put out the newest release. Mostly, I need to buy myself some time so that I can get the kinks worked out of the latest version and then upgrade the production box.
I'm grateful for any insights as to what could be happening, even if a solution is just a temporary workaround.
Thanks!
-Tim
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
------------------------------------------------------------------------
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
See also http://jira.freeswitch.org/browse/FS-3328 On 11/25/2012 1:46 PM, Abaci wrote:
I just remember reading somewhere on a blog something about a similar issue and this was the cause. Just found that post http://blog.godson.in/2011/06/pitfalls-to-avoid-while-using.html not sure that's the issue. either way it's a good idea to set the timeout option.
On 11/25/2012 11:32 AM, Ken Rice wrote:
Sofia is not single threaded except for in one spot deep in libsofia,
From there, messages are handed off to a number of message queues for FS core to handle as needed...
Check to see if anything that fs is depending on is blocking on info retrieval like the databases or other areas...
K
On 11/25/12 9:51 AM, "Abaci" <abaci64@gmail.com> wrote:
you mentioned that you use xml_curl, if your web server hangs it may hang sofia, iirc sofia is running a single thread and it will wait for the xml_curl response before continuing to the next request.
On 11/23/2012 1:13 PM, Steven Ayre wrote:
Any kind of DB backup running? Or any long-running queries (innotop is great for highlighting queries that've been running a while, including on non-innodb tables).
A global read lock, or queries waiting for a lock could block a db update from the sofia profile thread but still allow read-only queries (sofia status) to run.
-Steve
On 23 November 2012 15:32, Tim St. Pierre <fs-list@communicatefreely.net> wrote:
Hi Steven,
Thanks for the suggestions. I'm hoping once I get the upgrade done it will all go away. I have watched it happen at least once. I was on the phone at the time. Console activity more or less stopped, except for a few calls hanging up. The console remains responsive, and my call wasn't dropped for at least a minute or two (media timeout?). I was able to run sofia status and other commands that use the database, so I'm assuming that the connection was still working. All our media is runs through the box, so I think things are fine on the Ethernet level. I do see higher load averages - maybe 3-4, but that's the only obvious indication. It's not taking CPU beyond 10% or so.
We are using MySQL as the core DB and also as the DB backend for each sofia profile. This is connecting through ODBC of course.
If I can get the other kinks worked out, then I will try 1.2 stable in production and we'll see how it goes.
-Tim
Steven Ayre wrote:
Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
As you've already acknowledged it's a very old version.
It's possible that your issue has already been found and fixed, but if it hasn't then the code will have changed significantly since then and you'd really need to reproduce it on the latest code for it to be investigated.
As some general thoughts though, are you able to spot it happening while it's happening or only afterwards?
If you're able to get on the system during one of those times look at what else is happening. Is the load average/cpu usage/io high? Perhaps something's running that's blocking all access or causing very high IO.
What DB backend are you using for Sofia? Is it possible that that's hanging for a moment? For example if you're running a backup on the DB that blocks all writes to the DB while Sofia is trying to update the DB that perhaps would cause this.
Try running a SIP OPTIONS ping your your sofia profile from the localhost during that time, which should exclude it being any issue on the ethernet.
-Steve
On 22 November 2012 19:31, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>> wrote:
Hello,
I'm having a bit of an odd problem.
Intermittently, often every 2-3 days or so, Freeswitch stops replying to SIP for about 5 minutes. I can't verify if it's EXACTLY 5 minutes, but it seems to be pretty close.
During this time, no new registrations or invites can happen, but existing calls stay connected for at least a minute or two. In the logs, you can see calls slowly hanging up with "NORMAL_CLEARING". In 5 minutes, everything starts up again with no word about it at all in the logs.
When calls resume, I notice that the number of sessions returned by the status command is one higher than the actual number sessions returned by show channels, or by looking in the database. Every time this happens, the discrepancy increases by one.
The interruption happens on all SIP profiles, but calls originated from the socket API still work, insofar as they return with PROGRESS_TIMEOUT since the profiles are still running, but stuck.
We are using ODBC/MySQL for the core database, and the database server only runs this database and some basic PHP/xml-curl stuff.
We have 416 endpoints registered, and usually sit at about 30 sessions during the day.
This never happens at night, only during busier times, but not necessarily busy hour.
I'm running on FreeBSD 8.2-RELEASE AMD 64(2 XEON cores, 4G ram)
Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
Yes, I know it's old and I'm trying to upgrade, but I'm still having some problems getting all my phones to work properly with 1.2 stable. This is a production system, so I can't just blindly put out the newest release. Mostly, I need to buy myself some time so that I can get the kinks worked out of the latest version and then upgrade the production box.
I'm grateful for any insights as to what could be happening, even if a solution is just a temporary workaround.
Thanks!
-Tim
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
------------------------------------------------------------------------
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
Gcore it when its stuck. Then you can bt. There will be no real help if the problem is that its older code. We don't have the resources to support old versions. You may want to test out the stable branch.... On Nov 25, 2012 12:31 PM, "Ken Rice" <krice@freeswitch.org> wrote:
Sofia is not single threaded except for in one spot deep in libsofia,
From there, messages are handed off to a number of message queues for FS core to handle as needed...
Check to see if anything that fs is depending on is blocking on info retrieval like the databases or other areas...
K
On 11/25/12 9:51 AM, "Abaci" <abaci64@gmail.com> wrote:
you mentioned that you use xml_curl, if your web server hangs it may hang sofia, iirc sofia is running a single thread and it will wait for the xml_curl response before continuing to the next request.
On 11/23/2012 1:13 PM, Steven Ayre wrote:
Any kind of DB backup running? Or any long-running queries (innotop is great for highlighting queries that've been running a while, including on non-innodb tables).
A global read lock, or queries waiting for a lock could block a db update from the sofia profile thread but still allow read-only queries (sofia status) to run.
-Steve
On 23 November 2012 15:32, Tim St. Pierre < fs-list@communicatefreely.net> wrote:
Hi Steven,
Thanks for the suggestions. I'm hoping once I get the upgrade done it will all go away. I have watched it happen at least once. I was on the phone at the time. Console activity more or less stopped, except for a few calls hanging up. The console remains responsive, and my call wasn't dropped for at least a minute or two (media timeout?). I was able to run sofia status and other commands that use the database, so I'm assuming that the connection was still working. All our media is runs through the box, so I think things are fine on the Ethernet level. I do see higher load averages - maybe 3-4, but that's the only obvious indication. It's not taking CPU beyond 10% or so.
We are using MySQL as the core DB and also as the DB backend for each sofia profile. This is connecting through ODBC of course.
If I can get the other kinks worked out, then I will try 1.2 stable in production and we'll see how it goes.
-Tim
Steven Ayre wrote:
Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
As you've already acknowledged it's a very old version.
It's possible that your issue has already been found and fixed, but if it hasn't then the code will have changed significantly since then and you'd really need to reproduce it on the latest code for it to be investigated.
As some general thoughts though, are you able to spot it happening
while
it's happening or only afterwards?
If you're able to get on the system during one of those times look at what else is happening. Is the load average/cpu usage/io high? Perhaps something's running that's blocking all access or causing very high IO.
What DB backend are you using for Sofia? Is it possible that that's hanging for a moment? For example if you're running a backup on the DB that blocks all writes to the DB while Sofia is trying to update the DB that perhaps would cause this.
Try running a SIP OPTIONS ping your your sofia profile from the localhost during that time, which should exclude it being any issue on the ethernet.
-Steve
On 22 November 2012 19:31, Tim St. Pierre < fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>> wrote:
Hello,
I'm having a bit of an odd problem.
Intermittently, often every 2-3 days or so, Freeswitch stops replying to SIP for about 5 minutes. I can't verify if it's EXACTLY 5 minutes, but it seems to be pretty close.
During this time, no new registrations or invites can happen, but existing calls stay connected for at least a minute or two. In the logs, you can see calls slowly hanging up with "NORMAL_CLEARING". In 5 minutes, everything starts up again with no word about it at all in the logs.
When calls resume, I notice that the number of sessions returned by the status command is one higher than the actual number sessions returned by show channels, or by looking in the database. Every time this happens, the discrepancy increases by one.
The interruption happens on all SIP profiles, but calls originated from the socket API still work, insofar as they return with PROGRESS_TIMEOUT since the profiles are still running, but stuck.
We are using ODBC/MySQL for the core database, and the database server only runs this database and some basic PHP/xml-curl stuff.
We have 416 endpoints registered, and usually sit at about 30 sessions during the day.
This never happens at night, only during busier times, but not necessarily busy hour.
I'm running on FreeBSD 8.2-RELEASE AMD 64(2 XEON cores, 4G ram)
Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
Yes, I know it's old and I'm trying to upgrade, but I'm still having some problems getting all my phones to work properly with 1.2 stable. This is a production system, so I can't just blindly put out the newest release. Mostly, I need to buy myself some time so that I can get the kinks worked out of the latest version and then upgrade the production box.
I'm grateful for any insights as to what could be happening, even if a solution is just a temporary workaround.
Thanks!
-Tim
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Ken http://www.FreeSWITCH.org http://www.ClueCon.com http://www.OSTAG.org irc.freenode.net #freeswitch
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Thanks for all the input everyone! I just had it happen again, this time at night when there were only 2 sessions active! I was in doing some other work, so I managed to see it happen and found a few interesting things. CPU load next to nothing, load average around 0.12. I used iSQL to test ODBC connectivity, and I could read and write to the freeswitch database using the same DSN that freeswitch is using. I could delete a SIP registration from the API Since I had things on a higher debug level, I saw this: freeswitch@stefan> 2012-11-26 21:21:51.914082 [CONSOLE] sofia.c:1144 MSG Thread Started As soon as that came up, everything started working again. What's that about? -Tim Ken Rice wrote:
Sofia is not single threaded except for in one spot deep in libsofia,
From there, messages are handed off to a number of message queues for FS core to handle as needed...
Check to see if anything that fs is depending on is blocking on info retrieval like the databases or other areas...
K
On 11/25/12 9:51 AM, "Abaci" <abaci64@gmail.com> wrote:
you mentioned that you use xml_curl, if your web server hangs it may hang sofia, iirc sofia is running a single thread and it will wait for the xml_curl response before continuing to the next request.
On 11/23/2012 1:13 PM, Steven Ayre wrote:
Any kind of DB backup running? Or any long-running queries (innotop is great for highlighting queries that've been running a while, including on non-innodb tables).
A global read lock, or queries waiting for a lock could block a db update from the sofia profile thread but still allow read-only queries (sofia status) to run.
-Steve
On 23 November 2012 15:32, Tim St. Pierre <fs-list@communicatefreely.net> wrote:
Hi Steven,
Thanks for the suggestions. I'm hoping once I get the upgrade done it will all go away. I have watched it happen at least once. I was on the phone at the time. Console activity more or less stopped, except for a few calls hanging up. The console remains responsive, and my call wasn't dropped for at least a minute or two (media timeout?). I was able to run sofia status and other commands that use the database, so I'm assuming that the connection was still working. All our media is runs through the box, so I think things are fine on the Ethernet level. I do see higher load averages - maybe 3-4, but that's the only obvious indication. It's not taking CPU beyond 10% or so.
We are using MySQL as the core DB and also as the DB backend for each sofia profile. This is connecting through ODBC of course.
If I can get the other kinks worked out, then I will try 1.2 stable in production and we'll see how it goes.
-Tim
Steven Ayre wrote:
Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
As you've already acknowledged it's a very old version.
It's possible that your issue has already been found and fixed, but if it hasn't then the code will have changed significantly since then and you'd really need to reproduce it on the latest code for it to be investigated.
As some general thoughts though, are you able to spot it happening while it's happening or only afterwards?
If you're able to get on the system during one of those times look at what else is happening. Is the load average/cpu usage/io high? Perhaps something's running that's blocking all access or causing very high IO.
What DB backend are you using for Sofia? Is it possible that that's hanging for a moment? For example if you're running a backup on the DB that blocks all writes to the DB while Sofia is trying to update the DB that perhaps would cause this.
Try running a SIP OPTIONS ping your your sofia profile from the localhost during that time, which should exclude it being any issue on the ethernet.
-Steve
On 22 November 2012 19:31, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>> wrote:
Hello,
I'm having a bit of an odd problem.
Intermittently, often every 2-3 days or so, Freeswitch stops replying to SIP for about 5 minutes. I can't verify if it's EXACTLY 5 minutes, but it seems to be pretty close.
During this time, no new registrations or invites can happen, but existing calls stay connected for at least a minute or two. In the logs, you can see calls slowly hanging up with "NORMAL_CLEARING". In 5 minutes, everything starts up again with no word about it at all in the logs.
When calls resume, I notice that the number of sessions returned by the status command is one higher than the actual number sessions returned by show channels, or by looking in the database. Every time this happens, the discrepancy increases by one.
The interruption happens on all SIP profiles, but calls originated from the socket API still work, insofar as they return with PROGRESS_TIMEOUT since the profiles are still running, but stuck.
We are using ODBC/MySQL for the core database, and the database server only runs this database and some basic PHP/xml-curl stuff.
We have 416 endpoints registered, and usually sit at about 30 sessions during the day.
This never happens at night, only during busier times, but not necessarily busy hour.
I'm running on FreeBSD 8.2-RELEASE AMD 64(2 XEON cores, 4G ram)
Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
Yes, I know it's old and I'm trying to upgrade, but I'm still having some problems getting all my phones to work properly with 1.2 stable. This is a production system, so I can't just blindly put out the newest release. Mostly, I need to buy myself some time so that I can get the kinks worked out of the latest version and then upgrade the production box.
I'm grateful for any insights as to what could be happening, even if a solution is just a temporary workaround.
Thanks!
-Tim
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
------------------------------------------------------------------------
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
too bad you did not gcore it like I suggested, otherwise who knows... That's just the message it prints when the message parsing thread goes up. On Mon, Nov 26, 2012 at 8:25 PM, Tim St. Pierre < fs-list@communicatefreely.net> wrote:
Thanks for all the input everyone!
I just had it happen again, this time at night when there were only 2 sessions active!
I was in doing some other work, so I managed to see it happen and found a few interesting things.
CPU load next to nothing, load average around 0.12. I used iSQL to test ODBC connectivity, and I could read and write to the freeswitch database using the same DSN that freeswitch is using.
I could delete a SIP registration from the API
Since I had things on a higher debug level, I saw this:
freeswitch@stefan> 2012-11-26 21:21:51.914082 [CONSOLE] sofia.c:1144 MSG Thread Started
As soon as that came up, everything started working again.
What's that about?
-Tim
Ken Rice wrote:
Sofia is not single threaded except for in one spot deep in libsofia,
From there, messages are handed off to a number of message queues for FS core to handle as needed...
Check to see if anything that fs is depending on is blocking on info retrieval like the databases or other areas...
K
On 11/25/12 9:51 AM, "Abaci" <abaci64@gmail.com> wrote:
you mentioned that you use xml_curl, if your web server hangs it may hang sofia, iirc sofia is running a single thread and it will wait for the xml_curl response before continuing to the next request.
On 11/23/2012 1:13 PM, Steven Ayre wrote:
Any kind of DB backup running? Or any long-running queries (innotop is great for highlighting queries that've been running a while, including on non-innodb tables).
A global read lock, or queries waiting for a lock could block a db update from the sofia profile thread but still allow read-only queries (sofia status) to run.
-Steve
On 23 November 2012 15:32, Tim St. Pierre < fs-list@communicatefreely.net> wrote:
Hi Steven,
Thanks for the suggestions. I'm hoping once I get the upgrade done it will all go away. I have watched it happen at least once. I was on the phone at the time. Console activity more or less stopped, except for a few calls hanging up. The console remains responsive, and my call wasn't dropped for at least a minute or two (media timeout?). I was able to run sofia status and other commands that use the database, so I'm assuming that the connection was still working. All our media is runs through the box, so I think things are fine on the Ethernet level. I do see higher load averages - maybe 3-4, but that's the only obvious indication. It's not taking CPU beyond 10% or so.
We are using MySQL as the core DB and also as the DB backend for each sofia profile. This is connecting through ODBC of course.
If I can get the other kinks worked out, then I will try 1.2 stable in production and we'll see how it goes.
-Tim
Steven Ayre wrote:
Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
As you've already acknowledged it's a very old version.
It's possible that your issue has already been found and fixed, but
if
it hasn't then the code will have changed significantly since then and you'd really need to reproduce it on the latest code for it to be investigated.
As some general thoughts though, are you able to spot it happening while it's happening or only afterwards?
If you're able to get on the system during one of those times look at what else is happening. Is the load average/cpu usage/io high? Perhaps something's running that's blocking all access or causing very high IO.
What DB backend are you using for Sofia? Is it possible that that's hanging for a moment? For example if you're running a backup on the DB that blocks all writes to the DB while Sofia is trying to update the DB that perhaps would cause this.
Try running a SIP OPTIONS ping your your sofia profile from the localhost during that time, which should exclude it being any issue on the ethernet.
-Steve
On 22 November 2012 19:31, Tim St. Pierre < fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>> wrote:
Hello,
I'm having a bit of an odd problem.
Intermittently, often every 2-3 days or so, Freeswitch stops replying to SIP for about 5 minutes. I can't verify if it's EXACTLY 5 minutes, but it seems to be pretty close.
During this time, no new registrations or invites can happen, but existing calls stay connected for at least a minute or two. In the logs, you can see calls slowly hanging up with "NORMAL_CLEARING". In 5 minutes, everything starts up again with no word about it at all in the logs.
When calls resume, I notice that the number of sessions returned by the status command is one higher than the actual number sessions returned by show channels, or by looking in the database. Every time this happens, the discrepancy increases by one.
The interruption happens on all SIP profiles, but calls originated from the socket API still work, insofar as they return with PROGRESS_TIMEOUT since the profiles are still running, but stuck.
We are using ODBC/MySQL for the core database, and the database server only runs this database and some basic PHP/xml-curl stuff.
We have 416 endpoints registered, and usually sit at about 30 sessions during the day.
This never happens at night, only during busier times, but not necessarily busy hour.
I'm running on FreeBSD 8.2-RELEASE AMD 64(2 XEON cores, 4G ram)
Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)
Yes, I know it's old and I'm trying to upgrade, but I'm still having some problems getting all my phones to work properly with 1.2 stable. This is a production system, so I can't just blindly put out the newest release. Mostly, I need to buy myself some time so that I can get the kinks worked out of the latest version and then upgrade the production box.
I'm grateful for any insights as to what could be happening, even if a solution is just a temporary workaround.
Thanks!
-Tim
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
It's not a matter of not taking your suggestion, it's an issue of not having had the time yet to learn how gcore works and put it in place on a production system. It's on my "to-do" list, but I thought I would at least update the thread, hoping maybe that message meant something to someone. Anthony Minessale wrote:
too bad you did not gcore it like I suggested, otherwise who knows... That's just the message it prints when the message parsing thread goes up.
On Mon, Nov 26, 2012 at 8:25 PM, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>> wrote:
Thanks for all the input everyone!
I just had it happen again, this time at night when there were only 2 sessions active!
I was in doing some other work, so I managed to see it happen and found a few interesting things.
CPU load next to nothing, load average around 0.12. I used iSQL to test ODBC connectivity, and I could read and write to the freeswitch database using the same DSN that freeswitch is using.
I could delete a SIP registration from the API
Since I had things on a higher debug level, I saw this:
freeswitch@stefan> 2012-11-26 21:21:51.914082 [CONSOLE] sofia.c:1144 MSG Thread Started
As soon as that came up, everything started working again.
What's that about?
-Tim
Ken Rice wrote: > Sofia is not single threaded except for in one spot deep in libsofia, >>From there, messages are handed off to a number of message queues for FS > core to handle as needed... > > Check to see if anything that fs is depending on is blocking on info > retrieval like the databases or other areas... > > K > > On 11/25/12 9:51 AM, "Abaci" <abaci64@gmail.com <mailto:abaci64@gmail.com>> wrote: > >> you mentioned that you use xml_curl, if your web server hangs it may >> hang sofia, iirc sofia is running a single thread and it will wait for >> the xml_curl response before continuing to the next request. >> >> On 11/23/2012 1:13 PM, Steven Ayre wrote: >>> Any kind of DB backup running? Or any long-running queries (innotop is >>> great for highlighting queries that've been running a while, including >>> on non-innodb tables). >>> >>> A global read lock, or queries waiting for a lock could block a db >>> update from the sofia profile thread but still allow read-only queries >>> (sofia status) to run. >>> >>> -Steve >>> >>> >>> >>> On 23 November 2012 15:32, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>> >>> wrote: >>>> Hi Steven, >>>> >>>> Thanks for the suggestions. I'm hoping once I get the upgrade done it will >>>> all go away. >>>> I have watched it happen at least once. I was on the phone at the time. >>>> Console activity >>>> more or less stopped, except for a few calls hanging up. The console >>>> remains responsive, >>>> and my call wasn't dropped for at least a minute or two (media timeout?). I >>>> was able to >>>> run sofia status and other commands that use the database, so I'm assuming >>>> that the >>>> connection was still working. All our media is runs through the box, so I >>>> think things >>>> are fine on the Ethernet level. I do see higher load averages - maybe 3-4, >>>> but that's the >>>> only obvious indication. It's not taking CPU beyond 10% or so. >>>> >>>> We are using MySQL as the core DB and also as the DB backend for each sofia >>>> profile. This >>>> is connecting through ODBC of course. >>>> >>>> If I can get the other kinks worked out, then I will try 1.2 stable in >>>> production and >>>> we'll see how it goes. >>>> >>>> -Tim >>>> >>>> Steven Ayre wrote: >>>>> Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500) >>>>> >>>>> >>>>> As you've already acknowledged it's a very old version. >>>>> >>>>> It's possible that your issue has already been found and fixed, but if >>>>> it hasn't then the code will have changed significantly since then and >>>>> you'd really need to reproduce it on the latest code for it to be >>>>> investigated. >>>>> >>>>> >>>>> As some general thoughts though, are you able to spot it happening while >>>>> it's happening or only afterwards? >>>>> >>>>> If you're able to get on the system during one of those times look at >>>>> what else is happening. Is the load average/cpu usage/io high? Perhaps >>>>> something's running that's blocking all access or causing very high IO. >>>>> >>>>> What DB backend are you using for Sofia? Is it possible that that's >>>>> hanging for a moment? For example if you're running a backup on the DB >>>>> that blocks all writes to the DB while Sofia is trying to update the DB >>>>> that perhaps would cause this. >>>>> >>>>> Try running a SIP OPTIONS ping your your sofia profile from the >>>>> localhost during that time, which should exclude it being any issue on >>>>> the ethernet. >>>>> >>>>> -Steve >>>>> >>>>> >>>>> >>>>> >>>>> On 22 November 2012 19:31, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net> >>>>> <mailto:fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>>> wrote: >>>>> >>>>> Hello, >>>>> >>>>> I'm having a bit of an odd problem. >>>>> >>>>> Intermittently, often every 2-3 days or so, Freeswitch stops >>>>> replying to SIP for about 5 >>>>> minutes. I can't verify if it's EXACTLY 5 minutes, but it seems to >>>>> be pretty close. >>>>> >>>>> During this time, no new registrations or invites can happen, but >>>>> existing calls stay >>>>> connected for at least a minute or two. In the logs, you can see >>>>> calls slowly hanging up >>>>> with "NORMAL_CLEARING". In 5 minutes, everything starts up again >>>>> with no word about it at >>>>> all in the logs. >>>>> >>>>> When calls resume, I notice that the number of sessions returned by >>>>> the status command is >>>>> one higher than the actual number sessions returned by show >>>>> channels, or by looking in the >>>>> database. Every time this happens, the discrepancy increases by one. >>>>> >>>>> The interruption happens on all SIP profiles, but calls originated >>>>> from the socket API >>>>> still work, insofar as they return with PROGRESS_TIMEOUT since the >>>>> profiles are still >>>>> running, but stuck. >>>>> >>>>> We are using ODBC/MySQL for the core database, and the database >>>>> server only runs this >>>>> database and some basic PHP/xml-curl stuff. >>>>> >>>>> We have 416 endpoints registered, and usually sit at about 30 >>>>> sessions during the day. >>>>> >>>>> This never happens at night, only during busier times, but not >>>>> necessarily busy hour. >>>>> >>>>> I'm running on FreeBSD 8.2-RELEASE AMD 64(2 XEON cores, 4G ram) >>>>> >>>>> Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500) >>>>> >>>>> Yes, I know it's old and I'm trying to upgrade, but I'm still having >>>>> some problems getting >>>>> all my phones to work properly with 1.2 stable. This is a >>>>> production system, so I can't >>>>> just blindly put out the newest release. Mostly, I need to buy >>>>> myself some time so that I >>>>> can get the kinks worked out of the latest version and then upgrade >>>>> the production box. >>>>> >>>>> I'm grateful for any insights as to what could be happening, even if >>>>> a solution is just a >>>>> temporary workaround. >>>>> >>>>> Thanks! >>>>> >>>>> -Tim >>>>> >>>>> >>>>> _________________________________________________________________________ >>>>> Professional FreeSWITCH Consulting Services: >>>>> consulting@freeswitch.org <mailto:consulting@freeswitch.org> <mailto:consulting@freeswitch.org <mailto:consulting@freeswitch.org>> >>>>> http://www.freeswitchsolutions.com >>>>> >>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>>>> http://www.cudatel.com >>>>> >>>>> Official FreeSWITCH Sites >>>>> http://www.freeswitch.org >>>>> http://wiki.freeswitch.org >>>>> http://www.cluecon.com >>>>> >>>>> FreeSWITCH-users mailing list >>>>> FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> >>>>> <mailto:FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org>> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>>> >>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users >>>>> http://www.freeswitch.org >>>>> >>>>> >>>>> >>>>> ------------------------------------------------------------------------ >>>>> >>>>> _________________________________________________________________________ >>>>> Professional FreeSWITCH Consulting Services: >>>>> consulting@freeswitch.org <mailto:consulting@freeswitch.org> >>>>> http://www.freeswitchsolutions.com >>>>> >>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>>>> http://www.cudatel.com >>>>> >>>>> Official FreeSWITCH Sites >>>>> http://www.freeswitch.org >>>>> http://wiki.freeswitch.org >>>>> http://www.cluecon.com >>>>> >>>>> FreeSWITCH-users mailing list >>>>> FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users >>>>> http://www.freeswitch.org >>>> _________________________________________________________________________ >>>> Professional FreeSWITCH Consulting Services: >>>> consulting@freeswitch.org <mailto:consulting@freeswitch.org> >>>> http://www.freeswitchsolutions.com >>>> >>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>>> http://www.cudatel.com >>>> >>>> Official FreeSWITCH Sites >>>> http://www.freeswitch.org >>>> http://wiki.freeswitch.org >>>> http://www.cluecon.com >>>> >>>> FreeSWITCH-users mailing list >>>> FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users >>>> http://www.freeswitch.org >>> _________________________________________________________________________ >>> Professional FreeSWITCH Consulting Services: >>> consulting@freeswitch.org <mailto:consulting@freeswitch.org> >>> http://www.freeswitchsolutions.com >>> >>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>> http://www.cudatel.com >>> >>> Official FreeSWITCH Sites >>> http://www.freeswitch.org >>> http://wiki.freeswitch.org >>> http://www.cluecon.com >>> >>> FreeSWITCH-users mailing list >>> FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users >>> http://www.freeswitch.org >> >> _________________________________________________________________________ >> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org <mailto:consulting@freeswitch.org> >> http://www.freeswitchsolutions.com >> >> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >> http://www.cudatel.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://wiki.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users >> http://www.freeswitch.org >
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com <mailto:MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com <mailto:PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> googletalk:conf+888@conference.freeswitch.org <mailto:googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900
------------------------------------------------------------------------
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
cliff notes version would be gcore `cat /usr/local/freeswitch/log/freeswitch.pid` Take the resulting core file and follow the backtrace instructions at http://wiki.freeswitch.org/wiki/Debugging_Freeswitch#Getting_a_Backtrace On Thu, Nov 29, 2012 at 12:09 PM, Tim St. Pierre < fs-list@communicatefreely.net> wrote:
It's not a matter of not taking your suggestion, it's an issue of not having had the time yet to learn how gcore works and put it in place on a production system. It's on my "to-do" list, but I thought I would at least update the thread, hoping maybe that message meant something to someone.
Anthony Minessale wrote:
too bad you did not gcore it like I suggested, otherwise who knows... That's just the message it prints when the message parsing thread goes up.
On Mon, Nov 26, 2012 at 8:25 PM, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>> wrote:
Thanks for all the input everyone!
I just had it happen again, this time at night when there were only 2 sessions active!
I was in doing some other work, so I managed to see it happen and found a few interesting things.
CPU load next to nothing, load average around 0.12. I used iSQL to test ODBC connectivity, and I could read and write to the freeswitch database using the same DSN that freeswitch is using.
I could delete a SIP registration from the API
Since I had things on a higher debug level, I saw this:
freeswitch@stefan> 2012-11-26 21:21:51.914082 [CONSOLE] sofia.c:1144 MSG Thread Started
As soon as that came up, everything started working again.
What's that about?
-Tim
Ken Rice wrote: > Sofia is not single threaded except for in one spot deep in libsofia, >>From there, messages are handed off to a number of message queues for FS > core to handle as needed... > > Check to see if anything that fs is depending on is blocking on info > retrieval like the databases or other areas... > > K > > On 11/25/12 9:51 AM, "Abaci" <abaci64@gmail.com <mailto:abaci64@gmail.com>> wrote: > >> you mentioned that you use xml_curl, if your web server hangs it may >> hang sofia, iirc sofia is running a single thread and it will wait for >> the xml_curl response before continuing to the next request. >> >> On 11/23/2012 1:13 PM, Steven Ayre wrote: >>> Any kind of DB backup running? Or any long-running queries (innotop is >>> great for highlighting queries that've been running a while, including >>> on non-innodb tables). >>> >>> A global read lock, or queries waiting for a lock could block a db >>> update from the sofia profile thread but still allow read-only queries >>> (sofia status) to run. >>> >>> -Steve >>> >>> >>> >>> On 23 November 2012 15:32, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net
>>> wrote: >>>> Hi Steven, >>>> >>>> Thanks for the suggestions. I'm hoping once I get the upgrade done it will >>>> all go away. >>>> I have watched it happen at least once. I was on the phone at the time. >>>> Console activity >>>> more or less stopped, except for a few calls hanging up. The console >>>> remains responsive, >>>> and my call wasn't dropped for at least a minute or two (media timeout?). I >>>> was able to >>>> run sofia status and other commands that use the database, so I'm assuming >>>> that the >>>> connection was still working. All our media is runs through the box, so I >>>> think things >>>> are fine on the Ethernet level. I do see higher load averages - maybe 3-4, >>>> but that's the >>>> only obvious indication. It's not taking CPU beyond 10% or so. >>>> >>>> We are using MySQL as the core DB and also as the DB backend for each sofia >>>> profile. This >>>> is connecting through ODBC of course. >>>> >>>> If I can get the other kinks worked out, then I will try 1.2 stable in >>>> production and >>>> we'll see how it goes. >>>> >>>> -Tim >>>> >>>> Steven Ayre wrote: >>>>> Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500) >>>>> >>>>> >>>>> As you've already acknowledged it's a very old version. >>>>> >>>>> It's possible that your issue has already been found and fixed, but if >>>>> it hasn't then the code will have changed significantly since then and >>>>> you'd really need to reproduce it on the latest code for it to be >>>>> investigated. >>>>> >>>>> >>>>> As some general thoughts though, are you able to spot it happening while >>>>> it's happening or only afterwards? >>>>> >>>>> If you're able to get on the system during one of those times look at >>>>> what else is happening. Is the load average/cpu usage/io high? Perhaps >>>>> something's running that's blocking all access or causing very high IO. >>>>> >>>>> What DB backend are you using for Sofia? Is it possible that that's >>>>> hanging for a moment? For example if you're running a backup on the DB >>>>> that blocks all writes to the DB while Sofia is trying to update the DB >>>>> that perhaps would cause this. >>>>> >>>>> Try running a SIP OPTIONS ping your your sofia profile from the >>>>> localhost during that time, which should exclude it being any issue on >>>>> the ethernet. >>>>> >>>>> -Steve >>>>> >>>>> >>>>> >>>>> >>>>> On 22 November 2012 19:31, Tim St. Pierre <fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net
>>>>> <mailto:fs-list@communicatefreely.net <mailto:fs-list@communicatefreely.net>>> wrote: >>>>> >>>>> Hello, >>>>> >>>>> I'm having a bit of an odd problem. >>>>> >>>>> Intermittently, often every 2-3 days or so, Freeswitch stops >>>>> replying to SIP for about 5 >>>>> minutes. I can't verify if it's EXACTLY 5 minutes, but it seems to >>>>> be pretty close. >>>>> >>>>> During this time, no new registrations or invites can happen, but >>>>> existing calls stay >>>>> connected for at least a minute or two. In the logs, you can see >>>>> calls slowly hanging up >>>>> with "NORMAL_CLEARING". In 5 minutes, everything starts up again >>>>> with no word about it at >>>>> all in the logs. >>>>> >>>>> When calls resume, I notice that the number of sessions returned by >>>>> the status command is >>>>> one higher than the actual number sessions returned by show >>>>> channels, or by looking in the >>>>> database. Every time this happens, the discrepancy increases by one. >>>>> >>>>> The interruption happens on all SIP profiles, but calls originated >>>>> from the socket API >>>>> still work, insofar as they return with PROGRESS_TIMEOUT since the >>>>> profiles are still >>>>> running, but stuck. >>>>> >>>>> We are using ODBC/MySQL for the core database, and the database >>>>> server only runs this >>>>> database and some basic PHP/xml-curl stuff. >>>>> >>>>> We have 416 endpoints registered, and usually sit at about 30 >>>>> sessions during the day. >>>>> >>>>> This never happens at night, only during busier times, but not >>>>> necessarily busy hour. >>>>> >>>>> I'm running on FreeBSD 8.2-RELEASE AMD 64(2 XEON cores, 4G ram) >>>>> >>>>> Freeswitch is 1.0.head (git-7531fed 2011-08-17 11-27-20 -0500) >>>>> >>>>> Yes, I know it's old and I'm trying to upgrade, but I'm still having >>>>> some problems getting >>>>> all my phones to work properly with 1.2 stable. This is a >>>>> production system, so I can't >>>>> just blindly put out the newest release. Mostly, I need to buy >>>>> myself some time so that I >>>>> can get the kinks worked out of the latest version and then upgrade >>>>> the production box. >>>>> >>>>> I'm grateful for any insights as to what could be happening, even if >>>>> a solution is just a >>>>> temporary workaround. >>>>> >>>>> Thanks! >>>>> >>>>> -Tim >>>>> >>>>> >>>>>
_________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services: >>>>> consulting@freeswitch.org <mailto:consulting@freeswitch.org> <mailto:consulting@freeswitch.org <mailto:consulting@freeswitch.org>> >>>>> http://www.freeswitchsolutions.com >>>>> >>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication
Server
>>>>> http://www.cudatel.com >>>>> >>>>> Official FreeSWITCH Sites >>>>> http://www.freeswitch.org >>>>> http://wiki.freeswitch.org >>>>> http://www.cluecon.com >>>>> >>>>> FreeSWITCH-users mailing list >>>>> FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> >>>>> <mailto:FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org>> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>>> >>>>> UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org >>>>> >>>>> >>>>> >>>>>
------------------------------------------------------------------------
>>>>> >>>>>
_________________________________________________________________________
>>>>> Professional FreeSWITCH Consulting Services: >>>>> consulting@freeswitch.org <mailto:consulting@freeswitch.org> >>>>> http://www.freeswitchsolutions.com >>>>> >>>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>>>> http://www.cudatel.com >>>>> >>>>> Official FreeSWITCH Sites >>>>> http://www.freeswitch.org >>>>> http://wiki.freeswitch.org >>>>> http://www.cluecon.com >>>>> >>>>> FreeSWITCH-users mailing list >>>>> FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> >>>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>>> UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>>> http://www.freeswitch.org >>>>
_________________________________________________________________________
>>>> Professional FreeSWITCH Consulting Services: >>>> consulting@freeswitch.org <mailto:consulting@freeswitch.org> >>>> http://www.freeswitchsolutions.com >>>> >>>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>>> http://www.cudatel.com >>>> >>>> Official FreeSWITCH Sites >>>> http://www.freeswitch.org >>>> http://wiki.freeswitch.org >>>> http://www.cluecon.com >>>> >>>> FreeSWITCH-users mailing list >>>> FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> >>>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>>> UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
>>>> http://www.freeswitch.org >>>
_________________________________________________________________________
>>> Professional FreeSWITCH Consulting Services: >>> consulting@freeswitch.org <mailto:consulting@freeswitch.org> >>> http://www.freeswitchsolutions.com >>> >>> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >>> http://www.cudatel.com >>> >>> Official FreeSWITCH Sites >>> http://www.freeswitch.org >>> http://wiki.freeswitch.org >>> http://www.cluecon.com >>> >>> FreeSWITCH-users mailing list >>> FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> >>> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >>> UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
>>> http://www.freeswitch.org >> >>
_________________________________________________________________________
>> Professional FreeSWITCH Consulting Services: >> consulting@freeswitch.org <mailto:consulting@freeswitch.org> >> http://www.freeswitchsolutions.com >> >> FreeSWITCH-powered IP PBX: The CudaTel Communication Server >> http://www.cudatel.com >> >> Official FreeSWITCH Sites >> http://www.freeswitch.org >> http://wiki.freeswitch.org >> http://www.cluecon.com >> >> FreeSWITCH-users mailing list >> FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> >> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users >> UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org <mailto:consulting@freeswitch.org> http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org <mailto:FreeSWITCH-users@lists.freeswitch.org> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
-- Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire
AIM: anthm MSN:anthony_minessale@hotmail.com <mailto:MSN%3Aanthony_minessale@hotmail.com> GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com <mailto:PAYPAL%3Aanthony.minessale@gmail.com> IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
FreeSWITCH Developer Conference sip:888@conference.freeswitch.org <mailto:sip%3A888@conference.freeswitch.org> googletalk:conf+888@conference.freeswitch.org <mailto:googletalk%3Aconf%2B888@conference.freeswitch.org> pstn:+19193869900
------------------------------------------------------------------------
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_minessale@hotmail.com GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com IRC: irc.freenode.net #freeswitch FreeSWITCH Developer Conference sip:888@conference.freeswitch.org googletalk:conf+888@conference.freeswitch.org pstn:+19193869900
participants (5)
-
Abaci
-
Anthony Minessale
-
Ken Rice
-
Steven Ayre
-
Tim St. Pierre