[Freeswitch-users] High CPU usage with only 30 channels?

[Tim St. Pierre]

Hello,

I'm having some scaling problems with our Freeswitch system.  The 
machine running Freeswitch seems to be working very hard during the day, 
and we don't have that big a load.  I need to be able to double our 
subscriber base, and I don't really want to go to a cluster just yet.  
Here's what top -CP displays during busy hour:

tim at stefan%top -PC
last pid: 75968;  load averages:  4.64,  2.92,  
1.74                                                                                              
up 10+08:54:14  11:17:25
129 processes: 1 running, 128 sleeping
CPU 0:  4.5% user,  0.0% nice,  1.9% system,  0.0% interrupt, 93.6% idle
CPU 1:  3.8% user,  0.0% nice,  2.6% system,  0.0% interrupt, 93.6% idle
Mem: 632M Active, 2093M Inact, 322M Wired, 83M Cache, 384M Buf, 785M Free
Swap: 4096M Total, 36K Used, 4096M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    CPU COMMAND
58521 freeswitch   98  74  r30   749M   648M ucond   0 528:54  0.10% 
freeswitch
  983 root          1  44    0 10824K  2272K select  1   0:40  0.00% ntpd

As you can see, the load average is pretty high, at least for something 
not that big.  FS often has up to about 110 threads going.  At night, 
this will settle right down to a 0.05 load average.  CPU load never gets 
that high, but the machine still seems busy.

Here's some other details:

40 channels, nearly all of them in CS_EXECUTE or CS_EXCHANGE_MEDIA  We 
see only a few CPS, maybe 10 if it's really busy, but most calls are 
longer.  Nearly all our customers are regular office users.

There are 340 phones registered.  About 300 of those are behind NAT 
getting an options-ping every 30s.  They re-register every 5 minutes, so 
we see around 1 registration attempt per second, although it can be a 
little bursty.

Each phone has at least 5 BLF subscriptions,   There are 1156 rows in 
the sip_subscriptions table.

All our trunks are SIP.  Trunk side is g.711u and endpoints can do 
either g.711u or g.722

We use ODBC in the core - as well as for sofia, voicemail, and anything 
else that can use it.  Directory and dialplan are done using CURL and 
handled by another server.  The database is also on another server.

Our MySQL system sits at about 100 Queries per second during the day , 
down to about 80 qps at night.  This system started to choke until I 
changed a lot of the tables to the MEMORY engine and installed SSDs as 
the primary storage devices.  Now the database is pretty happy - about 
0.5 load average at the moment.  It was up to 2 with 60% CPU going to 
mysqld before I went to memory tables.

I keep hearing about FS systems handling thousands of users on one box.  
I'm nowhere near that and it seems to be maxed out.

This system  has a dual core Xeon 5100 at 2.6 GHz and 4GB of ram.  
Memory doesn't seem to be the issue, as there is still some unused.   
Operating system is FreeBSD 8.0-RELEASE (amd64)  Freeswitch is Version 
1.0.head (git-7531fed 2011-08-17 11-27-20 -0500)  I'm trying to update, 
but newer version break functionality, so I have to find a release that 
work before I can upgrade.

What should  I look for to remedy this, or is this considered normal and 
I just need more power?  I could easily upgrade this box and install a 
pair of quad core chips.

Any other information that would be helpful?