I've been getting this recently: 2012-06-05 10:31:44.579059 [ERR] mod_commands.c:4063 Error connecting 2012-06-05 10:31:44.619059 [ERR] sofia_glue.c:6464 Error connecting 2012-06-05 10:31:44.619059 [ERR] sofia_glue.c:6504 Error Opening DB 2012-06-05 10:31:44.619059 [WARNING] sofia_glue.c:6464 Max handles 50 exceeded, blocking.... Is there a workaround? I see runtime.max_db_handles = 50; in switch_core.c, so I assume there's no setting I can adjust to help. Would switching core to use ODBC solve the issue? Or something simpler? Regards, Cliff
On Tue, 2012-06-05 at 10:36 -0700, Cliff Wells wrote:
I've been getting this recently:
2012-06-05 10:31:44.579059 [ERR] mod_commands.c:4063 Error connecting 2012-06-05 10:31:44.619059 [ERR] sofia_glue.c:6464 Error connecting 2012-06-05 10:31:44.619059 [ERR] sofia_glue.c:6504 Error Opening DB 2012-06-05 10:31:44.619059 [WARNING] sofia_glue.c:6464 Max handles 50 exceeded, blocking....
Is there a workaround? I see runtime.max_db_handles = 50; in switch_core.c, so I assume there's no setting I can adjust to help. Would switching core to use ODBC solve the issue? Or something simpler?
... and in classic fashion, two minutes after posting I find the answer in autoload_configs/switch.conf.xml. Sorry for the noise. Cliff
I'm not in front of a server right now but check in /usr/local/freeswitch/conf/autoload_confocal/switch.conf.xml and see if there's a variable along the lines of Max DB handles. If it just refers to odbc stuff you have your answer. Before you go much further, what was your call volume? It could have thrown that error because something is locking the DB, so there could be something wrong with it. Brian Foster Endigo Computer LLC Sent from a mobile device. On Jun 5, 2012 1:36 PM, "Cliff Wells" <cliff@develix.com> wrote:
I've been getting this recently:
2012-06-05 10:31:44.579059 [ERR] mod_commands.c:4063 Error connecting 2012-06-05 10:31:44.619059 [ERR] sofia_glue.c:6464 Error connecting 2012-06-05 10:31:44.619059 [ERR] sofia_glue.c:6504 Error Opening DB 2012-06-05 10:31:44.619059 [WARNING] sofia_glue.c:6464 Max handles 50 exceeded, blocking....
Is there a workaround? I see runtime.max_db_handles = 50; in switch_core.c, so I assume there's no setting I can adjust to help. Would switching core to use ODBC solve the issue? Or something simpler?
Regards, Cliff
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
On Tue, 2012-06-05 at 13:51 -0400, Brian Foster wrote:
I'm not in front of a server right now but check in /usr/local/freeswitch/conf/autoload_confocal/switch.conf.xml and see if there's a variable along the lines of Max DB handles. If it just refers to odbc stuff you have your answer.
There was just such a setting. Thanks.
Before you go much further, what was your call volume? It could have thrown that error because something is locking the DB, so there could be something wrong with it.
When total channels exceeds 400ish (~65% inbound, ~35% outbound), system load suddenly spikes (under 350 or so, it's usually under 2 or 3, when we hit 400+, it will jump as high as 80). At that point the system appears to be queueing calls as the volume will sometimes shoot up to as high as 900, even though that's not really possible (hard limit due to T1 count), so it seems that it might be blocking on some resource. Also caught this error: 2012-06-05 11:23:21.419535 [CRIT] switch_core_session.c:1861 Throttle Error! 1342 2012-06-05 11:23:21.520601 [CRIT] switch_time.c:957 Over Session Rate of 30! So I adjusted that upwards as well. 16 cores Xeon 3GHz, 16GB RAM, 6 drive SAS RAID 10 Scientific Linux 6.1 Freeswitch git pull from yesterday. Regards, Cliff
If you're writing the channels to the DB, you can either skip it with -nosql or you can ramcache the SQL DBs. http://wiki.freeswitch.org/wiki/Performance_testing_and_configurations#FreeS... -Avi On Tue, Jun 5, 2012 at 9:24 PM, Cliff Wells <cliff@develix.com> wrote:
On Tue, 2012-06-05 at 13:51 -0400, Brian Foster wrote:
I'm not in front of a server right now but check in /usr/local/freeswitch/conf/autoload_confocal/switch.conf.xml and see if there's a variable along the lines of Max DB handles. If it just refers to odbc stuff you have your answer.
There was just such a setting. Thanks.
Before you go much further, what was your call volume? It could have thrown that error because something is locking the DB, so there could be something wrong with it.
When total channels exceeds 400ish (~65% inbound, ~35% outbound), system load suddenly spikes (under 350 or so, it's usually under 2 or 3, when we hit 400+, it will jump as high as 80). At that point the system appears to be queueing calls as the volume will sometimes shoot up to as high as 900, even though that's not really possible (hard limit due to T1 count), so it seems that it might be blocking on some resource.
Also caught this error:
2012-06-05 11:23:21.419535 [CRIT] switch_core_session.c:1861 Throttle Error! 1342 2012-06-05 11:23:21.520601 [CRIT] switch_time.c:957 Over Session Rate of 30!
So I adjusted that upwards as well.
16 cores Xeon 3GHz, 16GB RAM, 6 drive SAS RAID 10 Scientific Linux 6.1 Freeswitch git pull from yesterday.
Regards, Cliff
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Looks like one of our profiles was getting flooded with too many CPS (300+). Added more profiles and it seems okay now (although won't know for sure until peak hours tomorrow). Thanks for all the tips. Cliff On Tue, 2012-06-05 at 21:35 +0300, Avi Marcus wrote:
If you're writing the channels to the DB, you can either skip it with -nosql or you can ramcache the SQL DBs. http://wiki.freeswitch.org/wiki/Performance_testing_and_configurations#FreeS...
-Avi
On Tue, Jun 5, 2012 at 9:24 PM, Cliff Wells <cliff@develix.com> wrote: On Tue, 2012-06-05 at 13:51 -0400, Brian Foster wrote: > I'm not in front of a server right now but check > in /usr/local/freeswitch/conf/autoload_confocal/switch.conf.xml and > see if there's a variable along the lines of Max DB handles. If it > just refers to odbc stuff you have your answer.
There was just such a setting. Thanks.
> > Before you go much further, what was your call volume? It could have > thrown that error because something is locking the DB, so there could > be something wrong with it. >
When total channels exceeds 400ish (~65% inbound, ~35% outbound), system load suddenly spikes (under 350 or so, it's usually under 2 or 3, when we hit 400+, it will jump as high as 80). At that point the system appears to be queueing calls as the volume will sometimes shoot up to as high as 900, even though that's not really possible (hard limit due to T1 count), so it seems that it might be blocking on some resource.
Also caught this error:
2012-06-05 11:23:21.419535 [CRIT] switch_core_session.c:1861 Throttle Error! 1342 2012-06-05 11:23:21.520601 [CRIT] switch_time.c:957 Over Session Rate of 30!
So I adjusted that upwards as well.
16 cores Xeon 3GHz, 16GB RAM, 6 drive SAS RAID 10 Scientific Linux 6.1 Freeswitch git pull from yesterday.
Regards, Cliff
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
If you're running that kind of call volume, you might consider taking Avi's advice and go for a ramcache setup. I think your testing really the limits of what you can handle with the DB being on a disk. Maybe not, it depends on what all you are doing with the DB. I would look into that possibility though, it may give you better results. Brian Foster Endigo Computer LLC Sent from a mobile device. On Jun 5, 2012 6:23 PM, "Cliff Wells" <cliff@develix.com> wrote:
Looks like one of our profiles was getting flooded with too many CPS (300+). Added more profiles and it seems okay now (although won't know for sure until peak hours tomorrow).
Thanks for all the tips.
Cliff
On Tue, 2012-06-05 at 21:35 +0300, Avi Marcus wrote:
If you're writing the channels to the DB, you can either skip it with -nosql or you can ramcache the SQL DBs.
http://wiki.freeswitch.org/wiki/Performance_testing_and_configurations#FreeS...
-Avi
On Tue, Jun 5, 2012 at 9:24 PM, Cliff Wells <cliff@develix.com> wrote: On Tue, 2012-06-05 at 13:51 -0400, Brian Foster wrote: > I'm not in front of a server right now but check > in /usr/local/freeswitch/conf/autoload_confocal/switch.conf.xml
and
> see if there's a variable along the lines of Max DB handles. If it > just refers to odbc stuff you have your answer.
There was just such a setting. Thanks.
> > Before you go much further, what was your call volume? It could have > thrown that error because something is locking the DB, so there could > be something wrong with it. >
When total channels exceeds 400ish (~65% inbound, ~35% outbound), system load suddenly spikes (under 350 or so, it's usually under 2 or 3, when we hit 400+, it will jump as high as 80). At that point the system appears to be queueing calls as the volume will sometimes shoot up to as high as 900, even though that's not really possible (hard limit due to T1 count), so it seems that it might be blocking on some resource.
Also caught this error:
2012-06-05 11:23:21.419535 [CRIT] switch_core_session.c:1861 Throttle Error! 1342 2012-06-05 11:23:21.520601 [CRIT] switch_time.c:957 Over Session Rate of 30!
So I adjusted that upwards as well.
16 cores Xeon 3GHz, 16GB RAM, 6 drive SAS RAID 10 Scientific Linux 6.1 Freeswitch git pull from yesterday.
Regards, Cliff
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:
http://lists.freeswitch.org/mailman/options/freeswitch-users
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
FreeSWITCH-powered IP PBX: The CudaTel Communication Server http://www.cudatel.com
Official FreeSWITCH Sites http://www.freeswitch.org http://wiki.freeswitch.org http://www.cluecon.com
Join Us At ClueCon - Aug 7-9, 2012
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
On Tue, 2012-06-05 at 20:10 -0400, Brian Foster wrote:
If you're running that kind of call volume, you might consider taking Avi's advice and go for a ramcache setup. I think your testing really the limits of what you can handle with the DB being on a disk. Maybe not, it depends on what all you are doing with the DB. I would look into that possibility though, it may give you better results.
I did that as well. Regards, Cliff
participants (3)
-
Avi Marcus
-
Brian Foster
-
Cliff Wells