Seem to be attacked, CPU rate 95%, stop working, no log output.
Hi, I am using FreeSWITCH Version 1.4.18+git~20150312T185523Z~4eed221b69~64bit (git 4eed221 2015-03-12 18:55:23Z 64bit) on ubuntu 14.04, Linode VPS. It has been working fine for about 2 months. But recently it stopped working with CPU rate above 95% for several times. SIP client got 408 timeout error. I have to login system to kill the process and start over again. Then it back to normal for a period. Checked the log file, it seem to be a sudden stop. Nothing special before the stop and completely no log starting from the issue. May I ask how I should handle this issue? Thanks. Regards, Eric
here are iptables rules which do a simple rate limiting for SIP messages, probably this helps: https://txlab.wordpress.com/2013/06/29/protecting-a-vpbx-from-dos-attacks/ On Sat, Jun 13, 2015 at 1:46 AM, Eric Ni <xyangni@gmail.com> wrote:
Hi,
I am using FreeSWITCH Version 1.4.18+git~20150312T185523Z~4eed221b69~64bit (git 4eed221 2015-03-12 18:55:23Z 64bit) on ubuntu 14.04, Linode VPS. It has been working fine for about 2 months. But recently it stopped working with CPU rate above 95% for several times. SIP client got 408 timeout error. I have to login system to kill the process and start over again. Then it back to normal for a period. Checked the log file, it seem to be a sudden stop. Nothing special before the stop and completely no log starting from the issue. May I ask how I should handle this issue? Thanks.
Regards, Eric
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Thanks. I have tried this. But today it happen again. When I killed freeswith process and restart it, everything back to normal. So it does seem to be a DOS attack now. Or restarting fs won't make any difference. On Sat, Jun 13, 2015 at 1:04 AM, Stanislav Sinyagin <ssinyagin@gmail.com> wrote:
here are iptables rules which do a simple rate limiting for SIP messages, probably this helps: https://txlab.wordpress.com/2013/06/29/protecting-a-vpbx-from-dos-attacks/
On Sat, Jun 13, 2015 at 1:46 AM, Eric Ni <xyangni@gmail.com> wrote:
Hi,
I am using FreeSWITCH Version 1.4.18+git~20150312T185523Z~4eed221b69~64bit (git 4eed221 2015-03-12 18:55:23Z 64bit) on ubuntu 14.04, Linode VPS. It has been working fine for about 2 months. But recently it stopped working with CPU rate above 95% for several times. SIP client got 408 timeout error. I have to login system to kill the process and start over again. Then it back to normal for a period. Checked the log file, it seem to be a sudden stop. Nothing special before the stop and completely no log starting from the issue. May I ask how I should handle this issue? Thanks.
Regards, Eric
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
But do you see an excessive amount of SIP messages coming to your server? On Jun 26, 2015 6:22 PM, "Eric Ni" <xyangni@gmail.com> wrote:
Thanks. I have tried this. But today it happen again. When I killed freeswith process and restart it, everything back to normal. So it does seem to be a DOS attack now. Or restarting fs won't make any difference.
On Sat, Jun 13, 2015 at 1:04 AM, Stanislav Sinyagin <ssinyagin@gmail.com> wrote:
here are iptables rules which do a simple rate limiting for SIP messages, probably this helps: https://txlab.wordpress.com/2013/06/29/protecting-a-vpbx-from-dos-attacks/
On Sat, Jun 13, 2015 at 1:46 AM, Eric Ni <xyangni@gmail.com> wrote:
Hi,
I am using FreeSWITCH Version 1.4.18+git~20150312T185523Z~4eed221b69~64bit (git 4eed221 2015-03-12 18:55:23Z 64bit) on ubuntu 14.04, Linode VPS. It has been working fine for about 2 months. But recently it stopped working with CPU rate above 95% for several times. SIP client got 408 timeout error. I have to login system to kill the process and start over again. Then it back to normal for a period. Checked the log file, it seem to be a sudden stop. Nothing special before the stop and completely no log starting from the issue. May I ask how I should handle this issue? Thanks.
Regards, Eric
_________________________________________________________________________
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Could be less sinister. If you are bridging calls and the remote ip stops responding, fs can get backed up with calls, show channels will take forever and even trying to tab out commands will take some time. You can confirm if this is the case by opening core.db in sqlite3 and issue a command something like "select * from channels". If it is a back log, you can delete the channels from sqlite (delete from ... Where ..), you also need to delete from calls. I'm not 100% is its safe to do so though, but it's worked for me in the past. Bridging to a gateway rather than ip will help if this is the issue. The other thing I've seen cause what you describe is a bunged up ip tables (lots and lots of ruled from automated blocking) Have a look at iptables -L -n (I think that's right), and if lots of old rules, try restarting iptables. If it is a dos attack then as many have said, fail2ban is the way to go - though you should have that installed anyway. The simplest way to check if you are under attack is to install wireshark and issue: tethereal port 5060. Thanks Oz
On 26 Jun 2015, at 18:31, Stanislav Sinyagin <ssinyagin@gmail.com> wrote:
But do you see an excessive amount of SIP messages coming to your server?
On Jun 26, 2015 6:22 PM, "Eric Ni" <xyangni@gmail.com> wrote: Thanks. I have tried this. But today it happen again. When I killed freeswith process and restart it, everything back to normal. So it does seem to be a DOS attack now. Or restarting fs won't make any difference.
On Sat, Jun 13, 2015 at 1:04 AM, Stanislav Sinyagin <ssinyagin@gmail.com> wrote: here are iptables rules which do a simple rate limiting for SIP messages, probably this helps: https://txlab.wordpress.com/2013/06/29/protecting-a-vpbx-from-dos-attacks/
On Sat, Jun 13, 2015 at 1:46 AM, Eric Ni <xyangni@gmail.com> wrote:
Hi,
I am using FreeSWITCH Version 1.4.18+git~20150312T185523Z~4eed221b69~64bit (git 4eed221 2015-03-12 18:55:23Z 64bit) on ubuntu 14.04, Linode VPS. It has been working fine for about 2 months. But recently it stopped working with CPU rate above 95% for several times. SIP client got 408 timeout error. I have to login system to kill the process and start over again. Then it back to normal for a period. Checked the log file, it seem to be a sudden stop. Nothing special before the stop and completely no log starting from the issue. May I ask how I should handle this issue? Thanks.
Regards, Eric
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
In answer to a query on silently expiring fS from Eric Ni ... On 26 Jun 2015, at 19:56, Oz Mortimer <omortimer@gmail.com> wrote:
If it is a dos attack then as many have said, fail2ban is the way to go - though you should have that installed anyway. The simplest way to check if you are under attack is to install wireshark and issue: tethereal port 5060.
Most of the attacks I see seem to do DNS queries rather than blindly pinging 5060 => assume that the irritating bots know what port you're publishing in DNS as in use by SIP. And, of course, tethereal/tcpdump/... port <the port on which your server listens> all the best, Lawrence My permanent ipfw block list is now about 200-ish IP address ranges, all gleaned from fail2ban (don't leave home without it!)
Unless targeted, all attacks will start from an ip. There are endless combinations on domains. It's a data market; someone will be scanning subdomains for open ports and then sell that list on to someone that is interested in http/MySQL/sql/sip an so on. The http guys will do a reverse ip scan to find domains / subdomains. The sip/VoIP guys will try many tools, sipvicious being the most used, probably. One once in, the data will then "probably" be sold again to the real nasty guys. As Eric said, assume that they know everything before even turning your switch on. A high profile colo, will have several subnets, it wouldn't take long to scan every ip they host for services that are open. Back to the question in hand though, I don't think this is a dos-I'd imagine if it was, his server would be struggling too (and his access)-not just fs. Thanks Oz
On 26 Jun 2015, at 20:32, Lawrence Conroy <lconroy@insensate.co.uk> wrote:
In answer to a query on silently expiring fS from Eric Ni ...
On 26 Jun 2015, at 19:56, Oz Mortimer <omortimer@gmail.com> wrote: If it is a dos attack then as many have said, fail2ban is the way to go - though you should have that installed anyway. The simplest way to check if you are under attack is to install wireshark and issue: tethereal port 5060.
Most of the attacks I see seem to do DNS queries rather than blindly pinging 5060 => assume that the irritating bots know what port you're publishing in DNS as in use by SIP. And, of course, tethereal/tcpdump/... port <the port on which your server listens>
all the best, Lawrence
My permanent ipfw block list is now about 200-ish IP address ranges, all gleaned from fail2ban (don't leave home without it!)
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
You can configure fail2ban daemon to block attacks to FS server. On Sat, Jun 13, 2015, 02:48 Eric Ni <xyangni@gmail.com> wrote:
Hi,
I am using FreeSWITCH Version 1.4.18+git~20150312T185523Z~4eed221b69~64bit (git 4eed221 2015-03-12 18:55:23Z 64bit) on ubuntu 14.04, Linode VPS. It has been working fine for about 2 months. But recently it stopped working with CPU rate above 95% for several times. SIP client got 408 timeout error. I have to login system to kill the process and start over again. Then it back to normal for a period. Checked the log file, it seem to be a sudden stop. Nothing special before the stop and completely no log starting from the issue. May I ask how I should handle this issue? Thanks.
Regards, Eric _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
I can't see anything as the log stopped. As to those been logged, no sign of excessive sip message. Statistics from the Linode VPS also show a normal traffic volume. It look like a fs problem, but I don't know how to debug or report it as no log and no core dump. By the way fail2ban is already configured. On Fri, Jun 26, 2015 at 7:15 PM, Sergey Safarov <s.safarov@gmail.com> wrote:
You can configure fail2ban daemon to block attacks to FS server.
On Sat, Jun 13, 2015, 02:48 Eric Ni <xyangni@gmail.com> wrote:
Hi,
I am using FreeSWITCH Version 1.4.18+git~20150312T185523Z~4eed221b69~64bit (git 4eed221 2015-03-12 18:55:23Z 64bit) on ubuntu 14.04, Linode VPS. It has been working fine for about 2 months. But recently it stopped working with CPU rate above 95% for several times. SIP client got 408 timeout error. I have to login system to kill the process and start over again. Then it back to normal for a period. Checked the log file, it seem to be a sudden stop. Nothing special before the stop and completely no log starting from the issue. May I ask how I should handle this issue? Thanks.
Regards, Eric _________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (5)
-
Eric Ni
-
Lawrence Conroy
-
Oz Mortimer
-
Sergey Safarov
-
Stanislav Sinyagin