[Freeswitch-users] CPS per mod_sofia profile limited?

[Robin Vleij]

Hi guys,

Doing a bit of testing / benchmarking with FS 1.04 (and 1.05 SVN). Found
out some interesting things, I think.

The setup is like this:

SIPP Client -> FS -> SIPP Server

The dialplan is as simple as it gets:

<extension name="test-out">
  <condition field="network_addr" expression="^130\.244\.190\.150$"/>
  <condition field="destination_number" expression="^(\d+)$">
    <action application="set" data="proxy_media=true"/>
    <action application="set" data="hangup_after_bridge=true"/>
    <action application="bridge"
data="{sip_contact_user=transit}sofia/gateway/${distributor(gwg1)}/$1|sofia/gateway/${distributor(gwg2)}/$1"/>
  </condition>
</extension>

For the rest it's running CSV cdr's, commented out all modules I'm not
using, etc etc all that I could find on the wiki and the Interwebs.
Hardware / OS: from the shelve quadcore Xeon, debian 64-bit, 12GB
memory. SIPP is running a 500ms RTP pcap and the other side echos back.

I had a few test setups then:

1: FS SVN, 1 sofia profile where the gateways were configured and the
server_IP:5060 was used.

2. FS SVN, 2 sofia profiles where the gateways where in an seperate
profile (server_IP:5070) and the "customer facing" side was the original
profile.

3. FS 1.04, same as above

4. FS 1.04, 4 sofia profiles, distributor to spread load over 2 incoming
and 2 outgoing profiles.

Now the interesting thing was that under 1 I could go up, almost without
any CPU load, to 50cps. As soon as I went over this, calls where handled
 slower and "ongoing" calls would pile up untill it became really slow.
CPU load went to 100% on the FS process (both user and system time).
Lots of interupts and context switches. No throughput anymore untill I
lower and wait till the "buffer" is empty and FS is keeping up again.

Under 2, I was able to increase the CPS to about 100 with the same effect.

3 then went much better, I was able to increase CPS to about 200 cps and
response times in SIPP went up slighty untill it just hits some kind of
limit and calls are handled slower.

4 is pretty cool. Here I can run 2 sipp clients both doing 150cps to the
gateways that are spread on the distribution module, so I spread traffic
over 2 profiles. With 300 cps in total, FS is keeping up and I have 30%
idle CPU. However, increasing to over 300cps gives problems again, even
though I have idle CPU left!

All in all, I have a feeling that a single sip profile can't run more
than a certain limit untill it gets into some problem. Depending on if
I'm running SVN or 1.04 that limit seems to be 50cps or a bit higher.
After that limit it starts piling up "ongoing" calls, by taking time to
handle them and when that limit gets too high it's too late. All in all
really fine, I just set the system wide limit to a little under that
"threshold". But when I'm running just UNDER the threshold it's not CPU
that's a problem. Theoretically I should be able to run (based on the
CPU usage at 300cps) about 400cps.

When running at 300 I get SOME failed calls and I see

"switch_core_state_machine.c:525 a9a60636-0cea-11df-85a1-09c991f2afc5
Timeout waiting for next instruction in CS_NEW!"

in the console.

I didn't find much on how people do high cps setups and it feels a bit
like a "friday afternoon solution" to run multiple sofia profiles on the
same machine in order to max out the system.

Maybe I'm missing something and I know it's not an exact science this,
but I'm not sure "all is OK" because I'm not slowly getting to a 100%
cpu (or disk / network) usage, I hit some kind of limit after which
stuff goes wrong.

Anyone any input!

/Robin