[Freeswitch-users] freeswitch suddenly very slow

[devel at omninet.eu]

Hello,
I am facing a very strange problem: my fs server suddenly performs very
very slow.
Here a description of the system:
Debian 7, odbc, mysql is on another server, xml_curl for the users

Suddenly the server stops responding to registration requests or answers
with a very big delay. Sometimes I see there a message “503 System Busy”.
When this behavior starts, the active calls can continue, without any
issue. The problem is with new requests of any type.
Once, during this phase, I’ve tried to start manually (cli) the watchdog.
It crashed the fs session immediately. I thought I found the problem and
next time I started the watchdog while the system was running well. The
watchdog did not recognize the problem when fs was running again very
slow.
Sofia siptrace shows the registration attempts. Almost no answers are shown.
Tracing on the network side shows the registration attempts arriving, but
usually there is no response from the server. No other activity towards
mysql.

Restarting freeswitch can solve the problem for a few minutes, sometimes
for several hours, but it suddenly happens again. Lately, even after the
restart the speed is not coming back to normal.
This happened also during the night, with almost no calls and only
registration requests coming in. In the morning there was no user
registered. We restarted fs and the registrations started over.

I’ve tried to restart also Apache, MySQL, without noticing any difference.
Also a reboot of the server did not help.

A workaround now is to increase the re-registration timer for the
registration. Now I can see that sometimes the response to the
registration attempt is immediately, sometimes there is a bigger delay,
but enough for the client to stay registered. With some clients the
registration has a bigger delay and they get un-registered.
It looks like suddenly we have a  bottleneck, but I can’t find it.

I forgot to mention, that the server hardware has plenty of resources (24
GB RAM, 4 cores, cpu running very low).

In the past I have seen similar behavior, when we had some attacks from
“bad guys”, trying to place a lot of calls at the same time… We can’t see
such an attack now.

Does anybody have an idea into which direction to continue investigating?

Thank you
A