INVITE DoS Prevention
Hi, We run hosted Freeswitch instances in VMs with the internal profile on port 5060 connecting to clients mostly behind NAT and then the external profile connecting to our proxies only. Protecting the external profile its straightforward.. we only allow traffic to/from our proxies at the firewall level. But protecting the internal profile seems to be a bit more difficult because the UACs could be theoretically anywhere on the network. I'm currently using Fail2Ban to prevent brute force registration and INVITEs on auth failures, e.g.: failregex = \[WARNING\] sofia_reg.c:\d+ SIP auth failure \(REGISTER\) on sofia profile \'\w+\' for \[.*\] from ip <HOST> \[WARNING\] sofia_reg.c:\d+ SIP auth failure \(INVITE\) on sofia profile \'\w+\' for \[.*\] from ip <HOST> My question is, since its part of a normal SIP dialog to challenge the INVITE, is there any way to prevent a possible DoS from just sheer volume of incoming INVITEs on an Internet facing server automatically. I.e., If you block the logged challenge, you'd block all legitimate INVITEs and registrations. Since its UDP traffic I couldn't come up with a way to do it automatically at the iptables level. i.e. number of concurrent connections. Is there some option to just not respond if a client is sending a number of requests over a certain threshold? It might not stop them from sending the traffic but pretty soon they'd get the idea that it wasn't going to go anywhere. My concern is say there are 50 Freeswitch instances on a box (albeit 8 core, 32GB ram, 8 15K raid 10 storage) and someone starts sending thousands of rouge INVITEs to every VM on a physical box that the CPU load from just challenging the incoming INVITEs would create a DoS. We the logs regularly to try to catch people doing this sort of thing and drop them at a router upstream of the core network, but I'd like to have it happen without human intervention. Have I completely over thought this and am missing something obvious? Thanks, Spencer
Fail2Ban ... This is block an IP with too many failed attempts from something like SipVicious pretty quickly On 2/20/11 11:07 PM, "Spencer Thomason" <spencer@5ninesolutions.com> wrote:
Hi, We run hosted Freeswitch instances in VMs with the internal profile on port 5060 connecting to clients mostly behind NAT and then the external profile connecting to our proxies only. Protecting the external profile its straightforward.. we only allow traffic to/from our proxies at the firewall level. But protecting the internal profile seems to be a bit more difficult because the UACs could be theoretically anywhere on the network.
I'm currently using Fail2Ban to prevent brute force registration and INVITEs on auth failures, e.g.: failregex = \[WARNING\] sofia_reg.c:\d+ SIP auth failure \(REGISTER\) on sofia profile \'\w+\' for \[.*\] from ip <HOST> \[WARNING\] sofia_reg.c:\d+ SIP auth failure \(INVITE\) on sofia profile \'\w+\' for \[.*\] from ip <HOST>
My question is, since its part of a normal SIP dialog to challenge the INVITE, is there any way to prevent a possible DoS from just sheer volume of incoming INVITEs on an Internet facing server automatically. I.e., If you block the logged challenge, you'd block all legitimate INVITEs and registrations. Since its UDP traffic I couldn't come up with a way to do it automatically at the iptables level. i.e. number of concurrent connections. Is there some option to just not respond if a client is sending a number of requests over a certain threshold? It might not stop them from sending the traffic but pretty soon they'd get the idea that it wasn't going to go anywhere. My concern is say there are 50 Freeswitch instances on a box (albeit 8 core, 32GB ram, 8 15K raid 10 storage) and someone starts sending thousands of rouge INVITEs to every VM on a physical box that the CPU load from just challenging the incoming INVITEs would create a DoS. We the logs regularly to try to catch people doing this sort of thing and drop them at a router upstream of the core network, but I'd like to have it happen without human intervention. Have I completely over thought this and am missing something obvious?
Thanks, Spencer
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Yes, that works great if they respond to the challenge with a failed auth. But the scenario I'm trying to prevent is if they just send the INVITE and never respond to the challenge. Fail2Ban will not work as every endpoint will initially send an INVITE and receive a challenge. Legit calls will then respond correctly and not be logged as a SIP auth failure but every call that is challenged will show up as SIP auth challenge in the logs so there is no regex to differentiate between legit an non legit traffic. Spencer On Feb 20, 2011, at 10:39 PM, Ken Rice wrote:
Fail2Ban ... This is block an IP with too many failed attempts from something like SipVicious pretty quickly
On 2/20/11 11:07 PM, "Spencer Thomason" <spencer@5ninesolutions.com> wrote:
Hi, We run hosted Freeswitch instances in VMs with the internal profile on port 5060 connecting to clients mostly behind NAT and then the external profile connecting to our proxies only. Protecting the external profile its straightforward.. we only allow traffic to/from our proxies at the firewall level. But protecting the internal profile seems to be a bit more difficult because the UACs could be theoretically anywhere on the network.
I'm currently using Fail2Ban to prevent brute force registration and INVITEs on auth failures, e.g.: failregex = \[WARNING\] sofia_reg.c:\d+ SIP auth failure \(REGISTER\) on sofia profile \'\w+\' for \[.*\] from ip <HOST> \[WARNING\] sofia_reg.c:\d+ SIP auth failure \(INVITE\) on sofia profile \'\w+\' for \[.*\] from ip <HOST>
My question is, since its part of a normal SIP dialog to challenge the INVITE, is there any way to prevent a possible DoS from just sheer volume of incoming INVITEs on an Internet facing server automatically. I.e., If you block the logged challenge, you'd block all legitimate INVITEs and registrations. Since its UDP traffic I couldn't come up with a way to do it automatically at the iptables level. i.e. number of concurrent connections. Is there some option to just not respond if a client is sending a number of requests over a certain threshold? It might not stop them from sending the traffic but pretty soon they'd get the idea that it wasn't going to go anywhere. My concern is say there are 50 Freeswitch instances on a box (albeit 8 core, 32GB ram, 8 15K raid 10 storage) and someone starts sending thousands of rouge INVITEs to every VM on a physical box that the CPU load from just challenging the incoming INVITEs would create a DoS. We the logs regularly to try to catch people doing this sort of thing and drop them at a router upstream of the core network, but I'd like to have it happen without human intervention. Have I completely over thought this and am missing something obvious?
Thanks, Spencer
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Howdy mmm I just had a quick look at mod_sofia , and unfortunatly it looks like the channel is hungup after the challenge is sent. But my guess is that there HAS To be a list or similar somewhere that tracks endpoints waiting for challenge responses. Ill see if can track it down and submit a patch, because I too would like to see this at least logged. as for rate-limiting responses you can have iptables drop packets over X number of invites per sec ... ( But they are dropped silently at the Firewall - Kristians SIP Ratelimiter has this in it ) or you can use mod_limit to allow X number of invites per sec also, you would want to tell FS to hit the dialplan before codec negotiation though to do that nice and early in the piece. Jay On Mon, Feb 21, 2011 at 5:10 PM, Spencer Thomason < spencer@5ninesolutions.com> wrote:
Yes, that works great if they respond to the challenge with a failed auth. But the scenario I'm trying to prevent is if they just send the INVITE and never respond to the challenge. Fail2Ban will not work as every endpoint will initially send an INVITE and receive a challenge. Legit calls will then respond correctly and not be logged as a SIP auth failure but every call that is challenged will show up as SIP auth challenge in the logs so there is no regex to differentiate between legit an non legit traffic.
Spencer
On Feb 20, 2011, at 10:39 PM, Ken Rice wrote:
Fail2Ban ... This is block an IP with too many failed attempts from something like SipVicious pretty quickly
On 2/20/11 11:07 PM, "Spencer Thomason" <spencer@5ninesolutions.com> wrote:
Hi, We run hosted Freeswitch instances in VMs with the internal profile on port 5060 connecting to clients mostly behind NAT and then the external profile connecting to our proxies only. Protecting the external profile its straightforward.. we only allow traffic to/from our proxies at the firewall level. But protecting the internal profile seems to be a bit more difficult because the UACs could be theoretically anywhere on the network.
I'm currently using Fail2Ban to prevent brute force registration and INVITEs on auth failures, e.g.: failregex = \[WARNING\] sofia_reg.c:\d+ SIP auth failure \(REGISTER\) on sofia profile \'\w+\' for \[.*\] from ip <HOST> \[WARNING\] sofia_reg.c:\d+ SIP auth failure \(INVITE\) on sofia profile \'\w+\' for \[.*\] from ip <HOST>
My question is, since its part of a normal SIP dialog to challenge the INVITE, is there any way to prevent a possible DoS from just sheer volume of incoming INVITEs on an Internet facing server automatically. I.e., If you block the logged challenge, you'd block all legitimate INVITEs and registrations. Since its UDP traffic I couldn't come up with a way to do it automatically at the iptables level. i.e. number of concurrent connections. Is there some option to just not respond if a client is sending a number of requests over a certain threshold? It might not stop them from sending the traffic but pretty soon they'd get the idea that it wasn't going to go anywhere. My concern is say there are 50 Freeswitch instances on a box (albeit 8 core, 32GB ram, 8 15K raid 10 storage) and someone starts sending thousands of rouge INVITEs to every VM on a physical box that the CPU load from just challenging the incoming INVITEs would create a DoS. We the logs regularly to try to catch people doing this sort of thing and drop them at a router upstream of the core network, but I'd like to have it happen without human intervention. Have I completely over thought this and am missing something obvious?
Thanks, Spencer
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Sincerely Jay
Thanks for the info! Yes it would be great if Freeswitch logged this info somewhere but this firewall script was was I was trying to accomplish. :-) I was trying to do this with the minimal amount of overhead and I think that might be the ticket. The cool thing about having Freeswitch log it and then pick it up with Fail2ban would be that you could define the limit per user/IP. Say for instance you are using Freeswitch as an SBC and if the invite matches a user or an IP that is known, that limit could be much higher than an unknown IP or user. That would keep you from having to define a limit that is overly "safe" to prevent blocking real traffic especially if you are talking about a lot of traffic. Thanks again, Spencer On Feb 20, 2011, at 11:22 PM, jay binks wrote:
Howdy
mmm I just had a quick look at mod_sofia , and unfortunatly it looks like the channel is hungup after the challenge is sent. But my guess is that there HAS To be a list or similar somewhere that tracks endpoints waiting for challenge responses.
Ill see if can track it down and submit a patch, because I too would like to see this at least logged.
as for rate-limiting responses you can have iptables drop packets over X number of invites per sec ... ( But they are dropped silently at the Firewall - Kristians SIP Ratelimiter has this in it )
or you can use mod_limit to allow X number of invites per sec also, you would want to tell FS to hit the dialplan before codec negotiation though to do that nice and early in the piece.
Jay
On Mon, Feb 21, 2011 at 5:10 PM, Spencer Thomason <spencer@5ninesolutions.com
wrote: Yes, that works great if they respond to the challenge with a failed auth. But the scenario I'm trying to prevent is if they just send the INVITE and never respond to the challenge. Fail2Ban will not work as every endpoint will initially send an INVITE and receive a challenge. Legit calls will then respond correctly and not be logged as a SIP auth failure but every call that is challenged will show up as SIP auth challenge in the logs so there is no regex to differentiate between legit an non legit traffic.
Spencer
On Feb 20, 2011, at 10:39 PM, Ken Rice wrote:
Fail2Ban ... This is block an IP with too many failed attempts from something like SipVicious pretty quickly
On 2/20/11 11:07 PM, "Spencer Thomason" <spencer@5ninesolutions.com> wrote:
Hi, We run hosted Freeswitch instances in VMs with the internal profile on port 5060 connecting to clients mostly behind NAT and then the external profile connecting to our proxies only. Protecting the external profile its straightforward.. we only allow traffic to/ from our proxies at the firewall level. But protecting the internal profile seems to be a bit more difficult because the UACs could be theoretically anywhere on the network.
I'm currently using Fail2Ban to prevent brute force registration and INVITEs on auth failures, e.g.: failregex = \[WARNING\] sofia_reg.c:\d+ SIP auth failure \ (REGISTER\) on sofia profile \'\w+\' for \[.*\] from ip <HOST> \[WARNING\] sofia_reg.c:\d+ SIP auth failure \(INVITE\) on sofia profile \'\w+\' for \[.*\] from ip <HOST>
My question is, since its part of a normal SIP dialog to challenge the INVITE, is there any way to prevent a possible DoS from just sheer volume of incoming INVITEs on an Internet facing server automatically. I.e., If you block the logged challenge, you'd block all legitimate INVITEs and registrations. Since its UDP traffic I couldn't come up with a way to do it automatically at the iptables level. i.e. number of concurrent connections. Is there some option to just not respond if a client is sending a number of requests over a certain threshold? It might not stop them from sending the traffic but pretty soon they'd get the idea that it wasn't going to go anywhere. My concern is say there are 50 Freeswitch instances on a box (albeit 8 core, 32GB ram, 8 15K raid 10 storage) and someone starts sending thousands of rouge INVITEs to every VM on a physical box that the CPU load from just challenging the incoming INVITEs would create a DoS. We the logs regularly to try to catch people doing this sort of thing and drop them at a router upstream of the core network, but I'd like to have it happen without human intervention. Have I completely over thought this and am missing something obvious?
Thanks, Spencer
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Sincerely
Jay
BTW.. here is a link :-) http://etel.wiki.oreilly.com/wiki/index.php/SIP_DoS/DDoS_Mitigation On Feb 21, 2011, at 12:04 AM, Spencer Thomason wrote:
Thanks for the info! Yes it would be great if Freeswitch logged this info somewhere but this firewall script was was I was trying to accomplish. :-) I was trying to do this with the minimal amount of overhead and I think that might be the ticket. The cool thing about having Freeswitch log it and then pick it up with Fail2ban would be that you could define the limit per user/IP. Say for instance you are using Freeswitch as an SBC and if the invite matches a user or an IP that is known, that limit could be much higher than an unknown IP or user. That would keep you from having to define a limit that is overly "safe" to prevent blocking real traffic especially if you are talking about a lot of traffic.
Thanks again, Spencer
On Feb 20, 2011, at 11:22 PM, jay binks wrote:
Howdy
mmm I just had a quick look at mod_sofia , and unfortunatly it looks like the channel is hungup after the challenge is sent. But my guess is that there HAS To be a list or similar somewhere that tracks endpoints waiting for challenge responses.
Ill see if can track it down and submit a patch, because I too would like to see this at least logged.
as for rate-limiting responses you can have iptables drop packets over X number of invites per sec ... ( But they are dropped silently at the Firewall - Kristians SIP Ratelimiter has this in it )
or you can use mod_limit to allow X number of invites per sec also, you would want to tell FS to hit the dialplan before codec negotiation though to do that nice and early in the piece.
Jay
On Mon, Feb 21, 2011 at 5:10 PM, Spencer Thomason <spencer@5ninesolutions.com
wrote: Yes, that works great if they respond to the challenge with a failed auth. But the scenario I'm trying to prevent is if they just send the INVITE and never respond to the challenge. Fail2Ban will not work as every endpoint will initially send an INVITE and receive a challenge. Legit calls will then respond correctly and not be logged as a SIP auth failure but every call that is challenged will show up as SIP auth challenge in the logs so there is no regex to differentiate between legit an non legit traffic.
Spencer
On Feb 20, 2011, at 10:39 PM, Ken Rice wrote:
Fail2Ban ... This is block an IP with too many failed attempts from something like SipVicious pretty quickly
On 2/20/11 11:07 PM, "Spencer Thomason" <spencer@5ninesolutions.com> wrote:
Hi, We run hosted Freeswitch instances in VMs with the internal profile on port 5060 connecting to clients mostly behind NAT and then the external profile connecting to our proxies only. Protecting the external profile its straightforward.. we only allow traffic to/ from our proxies at the firewall level. But protecting the internal profile seems to be a bit more difficult because the UACs could be theoretically anywhere on the network.
I'm currently using Fail2Ban to prevent brute force registration and INVITEs on auth failures, e.g.: failregex = \[WARNING\] sofia_reg.c:\d+ SIP auth failure \ (REGISTER\) on sofia profile \'\w+\' for \[.*\] from ip <HOST> \[WARNING\] sofia_reg.c:\d+ SIP auth failure \(INVITE \) on sofia profile \'\w+\' for \[.*\] from ip <HOST>
My question is, since its part of a normal SIP dialog to challenge the INVITE, is there any way to prevent a possible DoS from just sheer volume of incoming INVITEs on an Internet facing server automatically. I.e., If you block the logged challenge, you'd block all legitimate INVITEs and registrations. Since its UDP traffic I couldn't come up with a way to do it automatically at the iptables level. i.e. number of concurrent connections. Is there some option to just not respond if a client is sending a number of requests over a certain threshold? It might not stop them from sending the traffic but pretty soon they'd get the idea that it wasn't going to go anywhere. My concern is say there are 50 Freeswitch instances on a box (albeit 8 core, 32GB ram, 8 15K raid 10 storage) and someone starts sending thousands of rouge INVITEs to every VM on a physical box that the CPU load from just challenging the incoming INVITEs would create a DoS. We the logs regularly to try to catch people doing this sort of thing and drop them at a router upstream of the core network, but I'd like to have it happen without human intervention. Have I completely over thought this and am missing something obvious?
Thanks, Spencer
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Sincerely
Jay
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
jay binks wrote:
as for rate-limiting responses you can have iptables drop packets over X number of invites per sec ... Just a thought. Perhaps, we should contemplate to add a feature on FS to set maximum of invites/sec/host. When the invites max out, add some sleep to slow down the response to the requested host. This will probably slow down the bot, especially if the bot is trying to hit a lot of FS servers out there.
----- FreeSWITCH hosted on a Seagate DockStar with OpenWRT. -- View this message in context: http://freeswitch-users.2379917.n2.nabble.com/INVITE-DoS-Prevention-tp604761... Sent from the freeswitch-users mailing list archive at Nabble.com.
That's be within the Sofia stack and it had to acknowledge the INVITE with a 100 Trying otherwise the INVITE either resends or gives up the same as a packet drop. Sleeping would mean keeping the INVITE in memory for longer while starting the session before accepting/rejecting it, increasing memory usage under a DOS attack and therefore making FS fall over faster. It'd also increase the complexity of the code starting up a session while it puts new invites aside and schedules them to be processed shortly afterwards. You couldn't just do a sleep as it'd probably lock Sofia up. Doesn't seem worth it. Iptables is a much better way of handling it. You can rate limit per-host. -Steve On 23 February 2011 15:10, mazilo <Nabble@slickdeals.endjunk.com> wrote:
jay binks wrote:
as for rate-limiting responses you can have iptables drop packets over X number of invites per sec ... Just a thought. Perhaps, we should contemplate to add a feature on FS to set maximum of invites/sec/host. When the invites max out, add some sleep to slow down the response to the requested host. This will probably slow down the bot, especially if the bot is trying to hit a lot of FS servers out there.
----- FreeSWITCH hosted on a Seagate DockStar with OpenWRT. -- View this message in context: http://freeswitch-users.2379917.n2.nabble.com/INVITE-DoS-Prevention-tp604761... Sent from the freeswitch-users mailing list archive at Nabble.com.
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Steven Ayre wrote:
That's be within the Sofia stack and it had to acknowledge the INVITE with a 100 Trying otherwise the INVITE either resends or gives up the same as a packet drop. Sleeping would mean keeping the INVITE in memory for longer while starting the session before accepting/rejecting it, increasing memory usage under a DOS attack and therefore making FS fall over faster. It'd also increase the complexity of the code starting up a session while it puts new invites aside and schedules them to be processed shortly afterwards. You couldn't just do a sleep as it'd probably lock Sofia up. Doesn't seem worth it.
When in sleep mode, check the new INVITEs against the same host. If so, then send them to sleep, too. If invites in sleep reaches Y amount, we can flag the host to ignore further requests and release/junk all the sleeping INVITEs. A database needs be setup to keep a tab of all rogue hosts. ----- FreeSWITCH hosted on a Seagate DockStar with OpenWRT. -- View this message in context: http://freeswitch-users.2379917.n2.nabble.com/INVITE-DoS-Prevention-tp604761... Sent from the freeswitch-users mailing list archive at Nabble.com.
I would change sip auth failure to challenge and then have sufficient times to only block if there are too many challenges in a certain time. I am not even sure the failure works any more in recent gits. Spencer Thomason <spencer@5ninesolutions.com> wrote:
Yes, that works great if they respond to the challenge with a failed auth. But the scenario I'm trying to prevent is if they just send the INVITE and never respond to the challenge. Fail2Ban will not work as every endpoint will initially send an INVITE and receive a challenge. Legit calls will then respond correctly and not be logged as a SIP auth failure but every call that is challenged will show up as SIP auth challenge in the logs so there is no regex to differentiate between legit an non legit traffic.
Spencer
On Feb 20, 2011, at 10:39 PM, Ken Rice wrote:
Fail2Ban ... This is block an IP with too many failed attempts from something like SipVicious pretty quickly
On 2/20/11 11:07 PM, "Spencer Thomason" <spencer@5ninesolutions.com> wrote:
Hi, We run hosted Freeswitch instances in VMs with the internal profile on port 5060 connecting to clients mostly behind NAT and then the external profile connecting to our proxies only. Protecting the external profile its straightforward.. we only allow traffic to/from our proxies at the firewall level. But protecting the internal profile seems to be a bit more difficult because the UACs could be theoretically anywhere on the network.
I'm currently using Fail2Ban to prevent brute force registration and INVITEs on auth failures, e.g.: failregex = \[WARNING\] sofia_reg.c:\d+ SIP auth failure \(REGISTER\) on sofia profile \'\w+\' for \[.*\] from ip <HOST> \[WARNING\] sofia_reg.c:\d+ SIP auth failure \(INVITE\) on sofia profile \'\w+\' for \[.*\] from ip <HOST>
My question is, since its part of a normal SIP dialog to challenge the INVITE, is there any way to prevent a possible DoS from just sheer volume of incoming INVITEs on an Internet facing server automatically. I.e., If you block the logged challenge, you'd block all legitimate INVITEs and registrations. Since its UDP traffic I couldn't come up with a way to do it automatically at the iptables level. i.e. number of concurrent connections. Is there some option to just not respond if a client is sending a number of requests over a certain threshold? It might not stop them from sending the traffic but pretty soon they'd get the idea that it wasn't going to go anywhere. My concern is say there are 50 Freeswitch instances on a box (albeit 8 core, 32GB ram, 8 15K raid 10 storage) and someone starts sending thousands of rouge INVITEs to every VM on a physical box that the CPU load from just challenging the incoming INVITEs would create a DoS. We the logs regularly to try to catch people doing this sort of thing and drop them at a router upstream of the core network, but I'd like to have it happen without human intervention. Have I completely over thought this and am missing something obvious?
Thanks, Spencer
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Your life is like a penny. You're going to lose it. The question is: How do you spend it? John Covici covici@ccs.covici.com
Auth failure has been working fine for me I the last 2 weeks. What makes you this it's not been working ?? On 21/02/2011, at 7:31 PM, covici@ccs.covici.com wrote:
I would change sip auth failure to challenge and then have sufficient times to only block if there are too many challenges in a certain time. I am not even sure the failure works any more in recent gits.
Spencer Thomason <spencer@5ninesolutions.com> wrote:
Yes, that works great if they respond to the challenge with a failed auth. But the scenario I'm trying to prevent is if they just send the INVITE and never respond to the challenge. Fail2Ban will not work as every endpoint will initially send an INVITE and receive a challenge. Legit calls will then respond correctly and not be logged as a SIP auth failure but every call that is challenged will show up as SIP auth challenge in the logs so there is no regex to differentiate between legit an non legit traffic.
Spencer
On Feb 20, 2011, at 10:39 PM, Ken Rice wrote:
Fail2Ban ... This is block an IP with too many failed attempts from something like SipVicious pretty quickly
On 2/20/11 11:07 PM, "Spencer Thomason" <spencer@5ninesolutions.com> wrote:
Hi, We run hosted Freeswitch instances in VMs with the internal profile on port 5060 connecting to clients mostly behind NAT and then the external profile connecting to our proxies only. Protecting the external profile its straightforward.. we only allow traffic to/from our proxies at the firewall level. But protecting the internal profile seems to be a bit more difficult because the UACs could be theoretically anywhere on the network.
I'm currently using Fail2Ban to prevent brute force registration and INVITEs on auth failures, e.g.: failregex = \[WARNING\] sofia_reg.c:\d+ SIP auth failure \(REGISTER\) on sofia profile \'\w+\' for \[.*\] from ip <HOST> \[WARNING\] sofia_reg.c:\d+ SIP auth failure \(INVITE\) on sofia profile \'\w+\' for \[.*\] from ip <HOST>
My question is, since its part of a normal SIP dialog to challenge the INVITE, is there any way to prevent a possible DoS from just sheer volume of incoming INVITEs on an Internet facing server automatically. I.e., If you block the logged challenge, you'd block all legitimate INVITEs and registrations. Since its UDP traffic I couldn't come up with a way to do it automatically at the iptables level. i.e. number of concurrent connections. Is there some option to just not respond if a client is sending a number of requests over a certain threshold? It might not stop them from sending the traffic but pretty soon they'd get the idea that it wasn't going to go anywhere. My concern is say there are 50 Freeswitch instances on a box (albeit 8 core, 32GB ram, 8 15K raid 10 storage) and someone starts sending thousands of rouge INVITEs to every VM on a physical box that the CPU load from just challenging the incoming INVITEs would create a DoS. We the logs regularly to try to catch people doing this sort of thing and drop them at a router upstream of the core network, but I'd like to have it happen without human intervention. Have I completely over thought this and am missing something obvious?
Thanks, Spencer
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Your life is like a penny. You're going to lose it. The question is: How do you spend it?
John Covici covici@ccs.covici.com
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
After tinkering with it, I think that might be the best way. The iptables method is cool but I'd like to have more dynamic control and with Fail2Ban looking at the challenges you could specifically ignore certain high traffic IPs and block others. What would be very cool is if instead of logging every challenge, a log entry was written if there was a high number from a specific IP, then you could decide what to do about it with fail2ban, similar to the pike module for opensips does. On Feb 21, 2011, at 1:31 AM, covici@ccs.covici.com wrote:
I would change sip auth failure to challenge and then have sufficient times to only block if there are too many challenges in a certain time. I am not even sure the failure works any more in recent gits.
Spencer Thomason <spencer@5ninesolutions.com> wrote:
Yes, that works great if they respond to the challenge with a failed auth. But the scenario I'm trying to prevent is if they just send the INVITE and never respond to the challenge. Fail2Ban will not work as every endpoint will initially send an INVITE and receive a challenge. Legit calls will then respond correctly and not be logged as a SIP auth failure but every call that is challenged will show up as SIP auth challenge in the logs so there is no regex to differentiate between legit an non legit traffic.
Spencer
On Feb 20, 2011, at 10:39 PM, Ken Rice wrote:
Fail2Ban ... This is block an IP with too many failed attempts from something like SipVicious pretty quickly
On 2/20/11 11:07 PM, "Spencer Thomason" <spencer@5ninesolutions.com> wrote:
Hi, We run hosted Freeswitch instances in VMs with the internal profile on port 5060 connecting to clients mostly behind NAT and then the external profile connecting to our proxies only. Protecting the external profile its straightforward.. we only allow traffic to/ from our proxies at the firewall level. But protecting the internal profile seems to be a bit more difficult because the UACs could be theoretically anywhere on the network.
I'm currently using Fail2Ban to prevent brute force registration and INVITEs on auth failures, e.g.: failregex = \[WARNING\] sofia_reg.c:\d+ SIP auth failure \ (REGISTER\) on sofia profile \'\w+\' for \[.*\] from ip <HOST> \[WARNING\] sofia_reg.c:\d+ SIP auth failure \(INVITE\) on sofia profile \'\w+\' for \[.*\] from ip <HOST>
My question is, since its part of a normal SIP dialog to challenge the INVITE, is there any way to prevent a possible DoS from just sheer volume of incoming INVITEs on an Internet facing server automatically. I.e., If you block the logged challenge, you'd block all legitimate INVITEs and registrations. Since its UDP traffic I couldn't come up with a way to do it automatically at the iptables level. i.e. number of concurrent connections. Is there some option to just not respond if a client is sending a number of requests over a certain threshold? It might not stop them from sending the traffic but pretty soon they'd get the idea that it wasn't going to go anywhere. My concern is say there are 50 Freeswitch instances on a box (albeit 8 core, 32GB ram, 8 15K raid 10 storage) and someone starts sending thousands of rouge INVITEs to every VM on a physical box that the CPU load from just challenging the incoming INVITEs would create a DoS. We the logs regularly to try to catch people doing this sort of thing and drop them at a router upstream of the core network, but I'd like to have it happen without human intervention. Have I completely over thought this and am missing something obvious?
Thanks, Spencer
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Your life is like a penny. You're going to lose it. The question is: How do you spend it?
John Covici covici@ccs.covici.com
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Just an idea for the other developers. Wouldn't be cool to have and event thrown by each module that does authentication so that other application modules are able to listen in those events and do whatever with it? Some people might even like to have that running an eel daemon and throw some snmp traps based on that? Just an idea that might be completely ridiculous or might raise some interest. Events are pretty cheap so there wouldn't be a lot of overhead involved. Regards, João Mesquita On 21/02/2011, at 07:28, Spencer Thomason <spencer@5ninesolutions.com> wrote:
After tinkering with it, I think that might be the best way. The iptables method is cool but I'd like to have more dynamic control and with Fail2Ban looking at the challenges you could specifically ignore certain high traffic IPs and block others. What would be very cool is if instead of logging every challenge, a log entry was written if there was a high number from a specific IP, then you could decide what to do about it with fail2ban, similar to the pike module for opensips does.
On Feb 21, 2011, at 1:31 AM, covici@ccs.covici.com wrote:
I would change sip auth failure to challenge and then have sufficient times to only block if there are too many challenges in a certain time. I am not even sure the failure works any more in recent gits.
Spencer Thomason <spencer@5ninesolutions.com> wrote:
Yes, that works great if they respond to the challenge with a failed auth. But the scenario I'm trying to prevent is if they just send the INVITE and never respond to the challenge. Fail2Ban will not work as every endpoint will initially send an INVITE and receive a challenge. Legit calls will then respond correctly and not be logged as a SIP auth failure but every call that is challenged will show up as SIP auth challenge in the logs so there is no regex to differentiate between legit an non legit traffic.
Spencer
On Feb 20, 2011, at 10:39 PM, Ken Rice wrote:
Fail2Ban ... This is block an IP with too many failed attempts from something like SipVicious pretty quickly
On 2/20/11 11:07 PM, "Spencer Thomason" <spencer@5ninesolutions.com> wrote:
Hi, We run hosted Freeswitch instances in VMs with the internal profile on port 5060 connecting to clients mostly behind NAT and then the external profile connecting to our proxies only. Protecting the external profile its straightforward.. we only allow traffic to/ from our proxies at the firewall level. But protecting the internal profile seems to be a bit more difficult because the UACs could be theoretically anywhere on the network.
I'm currently using Fail2Ban to prevent brute force registration and INVITEs on auth failures, e.g.: failregex = \[WARNING\] sofia_reg.c:\d+ SIP auth failure \ (REGISTER\) on sofia profile \'\w+\' for \[.*\] from ip <HOST> \[WARNING\] sofia_reg.c:\d+ SIP auth failure \(INVITE\) on sofia profile \'\w+\' for \[.*\] from ip <HOST>
My question is, since its part of a normal SIP dialog to challenge the INVITE, is there any way to prevent a possible DoS from just sheer volume of incoming INVITEs on an Internet facing server automatically. I.e., If you block the logged challenge, you'd block all legitimate INVITEs and registrations. Since its UDP traffic I couldn't come up with a way to do it automatically at the iptables level. i.e. number of concurrent connections. Is there some option to just not respond if a client is sending a number of requests over a certain threshold? It might not stop them from sending the traffic but pretty soon they'd get the idea that it wasn't going to go anywhere. My concern is say there are 50 Freeswitch instances on a box (albeit 8 core, 32GB ram, 8 15K raid 10 storage) and someone starts sending thousands of rouge INVITEs to every VM on a physical box that the CPU load from just challenging the incoming INVITEs would create a DoS. We the logs regularly to try to catch people doing this sort of thing and drop them at a router upstream of the core network, but I'd like to have it happen without human intervention. Have I completely over thought this and am missing something obvious?
Thanks, Spencer
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Your life is like a penny. You're going to lose it. The question is: How do you spend it?
John Covici covici@ccs.covici.com
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
could you not just modify your fail2ban regex and set the threshold for Register Auths in fail2ban ? ( you would not want to do it on invite auths, because it will include GOOD auths for calls ) I still like the idea of loggin sip_authentication timeouts .. I might play with that a little today. J On Mon, Feb 21, 2011 at 8:28 PM, Spencer Thomason < spencer@5ninesolutions.com> wrote:
After tinkering with it, I think that might be the best way. The iptables method is cool but I'd like to have more dynamic control and with Fail2Ban looking at the challenges you could specifically ignore certain high traffic IPs and block others. What would be very cool is if instead of logging every challenge, a log entry was written if there was a high number from a specific IP, then you could decide what to do about it with fail2ban, similar to the pike module for opensips does.
On Feb 21, 2011, at 1:31 AM, covici@ccs.covici.com wrote:
I would change sip auth failure to challenge and then have sufficient times to only block if there are too many challenges in a certain time. I am not even sure the failure works any more in recent gits.
Spencer Thomason <spencer@5ninesolutions.com> wrote:
Yes, that works great if they respond to the challenge with a failed auth. But the scenario I'm trying to prevent is if they just send the INVITE and never respond to the challenge. Fail2Ban will not work as every endpoint will initially send an INVITE and receive a challenge. Legit calls will then respond correctly and not be logged as a SIP auth failure but every call that is challenged will show up as SIP auth challenge in the logs so there is no regex to differentiate between legit an non legit traffic.
Spencer
On Feb 20, 2011, at 10:39 PM, Ken Rice wrote:
Fail2Ban ... This is block an IP with too many failed attempts from something like SipVicious pretty quickly
On 2/20/11 11:07 PM, "Spencer Thomason" <spencer@5ninesolutions.com> wrote:
Hi, We run hosted Freeswitch instances in VMs with the internal profile on port 5060 connecting to clients mostly behind NAT and then the external profile connecting to our proxies only. Protecting the external profile its straightforward.. we only allow traffic to/ from our proxies at the firewall level. But protecting the internal profile seems to be a bit more difficult because the UACs could be theoretically anywhere on the network.
I'm currently using Fail2Ban to prevent brute force registration and INVITEs on auth failures, e.g.: failregex = \[WARNING\] sofia_reg.c:\d+ SIP auth failure \ (REGISTER\) on sofia profile \'\w+\' for \[.*\] from ip <HOST> \[WARNING\] sofia_reg.c:\d+ SIP auth failure \(INVITE\) on sofia profile \'\w+\' for \[.*\] from ip <HOST>
My question is, since its part of a normal SIP dialog to challenge the INVITE, is there any way to prevent a possible DoS from just sheer volume of incoming INVITEs on an Internet facing server automatically. I.e., If you block the logged challenge, you'd block all legitimate INVITEs and registrations. Since its UDP traffic I couldn't come up with a way to do it automatically at the iptables level. i.e. number of concurrent connections. Is there some option to just not respond if a client is sending a number of requests over a certain threshold? It might not stop them from sending the traffic but pretty soon they'd get the idea that it wasn't going to go anywhere. My concern is say there are 50 Freeswitch instances on a box (albeit 8 core, 32GB ram, 8 15K raid 10 storage) and someone starts sending thousands of rouge INVITEs to every VM on a physical box that the CPU load from just challenging the incoming INVITEs would create a DoS. We the logs regularly to try to catch people doing this sort of thing and drop them at a router upstream of the core network, but I'd like to have it happen without human intervention. Have I completely over thought this and am missing something obvious?
Thanks, Spencer
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Your life is like a penny. You're going to lose it. The question is: How do you spend it?
John Covici covici@ccs.covici.com
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Sincerely Jay
Yes, and that works well. Initially I was trying to guard against a bunch of INVITEs without auth replys. For the time being I've set up a separate fail2ban filter that looks at the invite challenges and only blocks someone if its extremely high in a short period of time. I still have to figure out how many of these INVITEs without auths the systems can handle because when you've got several instances going just the disk activity from the logs becomes problematic if there is a massive spike. Logging the auth timeouts seems to be the ideal solution because you could then drop the timeouts without affecting legit traffic. Spencer On Feb 21, 2011, at 2:33 PM, jay binks wrote:
could you not just modify your fail2ban regex and set the threshold for Register Auths in fail2ban ? ( you would not want to do it on invite auths, because it will include GOOD auths for calls )
I still like the idea of loggin sip_authentication timeouts .. I might play with that a little today.
J
On Mon, Feb 21, 2011 at 8:28 PM, Spencer Thomason <spencer@5ninesolutions.com
wrote: After tinkering with it, I think that might be the best way. The iptables method is cool but I'd like to have more dynamic control and with Fail2Ban looking at the challenges you could specifically ignore certain high traffic IPs and block others. What would be very cool is if instead of logging every challenge, a log entry was written if there was a high number from a specific IP, then you could decide what to do about it with fail2ban, similar to the pike module for opensips does.
On Feb 21, 2011, at 1:31 AM, covici@ccs.covici.com wrote:
I would change sip auth failure to challenge and then have sufficient times to only block if there are too many challenges in a certain time. I am not even sure the failure works any more in recent gits.
Spencer Thomason <spencer@5ninesolutions.com> wrote:
Yes, that works great if they respond to the challenge with a failed auth. But the scenario I'm trying to prevent is if they just send the INVITE and never respond to the challenge. Fail2Ban will not work as every endpoint will initially send an INVITE and receive a challenge. Legit calls will then respond correctly and not be logged as a SIP auth failure but every call that is challenged will show up as SIP auth challenge in the logs so there is no regex to differentiate between legit an non legit traffic.
Spencer
On Feb 20, 2011, at 10:39 PM, Ken Rice wrote:
Fail2Ban ... This is block an IP with too many failed attempts from something like SipVicious pretty quickly
On 2/20/11 11:07 PM, "Spencer Thomason" <spencer@5ninesolutions.com
wrote:
Hi, We run hosted Freeswitch instances in VMs with the internal profile on port 5060 connecting to clients mostly behind NAT and then the external profile connecting to our proxies only. Protecting the external profile its straightforward.. we only allow traffic to/ from our proxies at the firewall level. But protecting the internal profile seems to be a bit more difficult because the UACs could be theoretically anywhere on the network.
I'm currently using Fail2Ban to prevent brute force registration and INVITEs on auth failures, e.g.: failregex = \[WARNING\] sofia_reg.c:\d+ SIP auth failure \ (REGISTER\) on sofia profile \'\w+\' for \[.*\] from ip <HOST> \[WARNING\] sofia_reg.c:\d+ SIP auth failure \(INVITE \) on sofia profile \'\w+\' for \[.*\] from ip <HOST>
My question is, since its part of a normal SIP dialog to challenge the INVITE, is there any way to prevent a possible DoS from just sheer volume of incoming INVITEs on an Internet facing server automatically. I.e., If you block the logged challenge, you'd block all legitimate INVITEs and registrations. Since its UDP traffic I couldn't come up with a way to do it automatically at the iptables level. i.e. number of concurrent connections. Is there some option to just not respond if a client is sending a number of requests over a certain threshold? It might not stop them from sending the traffic but pretty soon they'd get the idea that it wasn't going to go anywhere. My concern is say there are 50 Freeswitch instances on a box (albeit 8 core, 32GB ram, 8 15K raid 10 storage) and someone starts sending thousands of rouge INVITEs to every VM on a physical box that the CPU load from just challenging the incoming INVITEs would create a DoS. We the logs regularly to try to catch people doing this sort of thing and drop them at a router upstream of the core network, but I'd like to have it happen without human intervention. Have I completely over thought this and am missing something obvious?
Thanks, Spencer
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Your life is like a penny. You're going to lose it. The question is: How do you spend it?
John Covici covici@ccs.covici.com
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Sincerely
Jay _______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
maybe put your logs on TmpFS also then ? I assume you have your DB on TmpFS already ? Jay On Tue, Feb 22, 2011 at 8:55 AM, Spencer Thomason < spencer@5ninesolutions.com> wrote:
Yes, and that works well. Initially I was trying to guard against a bunch of INVITEs without auth replys. For the time being I've set up a separate fail2ban filter that looks at the invite challenges and only blocks someone if its extremely high in a short period of time. I still have to figure out how many of these INVITEs without auths the systems can handle because when you've got several instances going just the disk activity from the logs becomes problematic if there is a massive spike. Logging the auth timeouts seems to be the ideal solution because you could then drop the timeouts without affecting legit traffic.
Spencer
On Feb 21, 2011, at 2:33 PM, jay binks wrote:
could you not just modify your fail2ban regex and set the threshold for Register Auths in fail2ban ? ( you would not want to do it on invite auths, because it will include GOOD auths for calls )
I still like the idea of loggin sip_authentication timeouts .. I might play with that a little today.
J
On Mon, Feb 21, 2011 at 8:28 PM, Spencer Thomason < spencer@5ninesolutions.com> wrote:
After tinkering with it, I think that might be the best way. The iptables method is cool but I'd like to have more dynamic control and with Fail2Ban looking at the challenges you could specifically ignore certain high traffic IPs and block others. What would be very cool is if instead of logging every challenge, a log entry was written if there was a high number from a specific IP, then you could decide what to do about it with fail2ban, similar to the pike module for opensips does.
On Feb 21, 2011, at 1:31 AM, covici@ccs.covici.com wrote:
I would change sip auth failure to challenge and then have sufficient times to only block if there are too many challenges in a certain time. I am not even sure the failure works any more in recent gits.
Spencer Thomason <spencer@5ninesolutions.com> wrote:
Yes, that works great if they respond to the challenge with a failed auth. But the scenario I'm trying to prevent is if they just send the INVITE and never respond to the challenge. Fail2Ban will not work as every endpoint will initially send an INVITE and receive a challenge. Legit calls will then respond correctly and not be logged as a SIP auth failure but every call that is challenged will show up as SIP auth challenge in the logs so there is no regex to differentiate between legit an non legit traffic.
Spencer
On Feb 20, 2011, at 10:39 PM, Ken Rice wrote:
Fail2Ban ... This is block an IP with too many failed attempts from something like SipVicious pretty quickly
On 2/20/11 11:07 PM, "Spencer Thomason" <spencer@5ninesolutions.com> wrote:
Hi, We run hosted Freeswitch instances in VMs with the internal profile on port 5060 connecting to clients mostly behind NAT and then the external profile connecting to our proxies only. Protecting the external profile its straightforward.. we only allow traffic to/ from our proxies at the firewall level. But protecting the internal profile seems to be a bit more difficult because the UACs could be theoretically anywhere on the network.
I'm currently using Fail2Ban to prevent brute force registration and INVITEs on auth failures, e.g.: failregex = \[WARNING\] sofia_reg.c:\d+ SIP auth failure \ (REGISTER\) on sofia profile \'\w+\' for \[.*\] from ip <HOST> \[WARNING\] sofia_reg.c:\d+ SIP auth failure \(INVITE\) on sofia profile \'\w+\' for \[.*\] from ip <HOST>
My question is, since its part of a normal SIP dialog to challenge the INVITE, is there any way to prevent a possible DoS from just sheer volume of incoming INVITEs on an Internet facing server automatically. I.e., If you block the logged challenge, you'd block all legitimate INVITEs and registrations. Since its UDP traffic I couldn't come up with a way to do it automatically at the iptables level. i.e. number of concurrent connections. Is there some option to just not respond if a client is sending a number of requests over a certain threshold? It might not stop them from sending the traffic but pretty soon they'd get the idea that it wasn't going to go anywhere. My concern is say there are 50 Freeswitch instances on a box (albeit 8 core, 32GB ram, 8 15K raid 10 storage) and someone starts sending thousands of rouge INVITEs to every VM on a physical box that the CPU load from just challenging the incoming INVITEs would create a DoS. We the logs regularly to try to catch people doing this sort of thing and drop them at a router upstream of the core network, but I'd like to have it happen without human intervention. Have I completely over thought this and am missing something obvious?
Thanks, Spencer
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Your life is like a penny. You're going to lose it. The question is: How do you spend it?
John Covici covici@ccs.covici.com
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE: http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Sincerely
Jay _______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- Sincerely Jay
We run a somewhat similar sounding setup. We wrote some code that grabs the /32 from the fail2ban instance running on each VM, and automatically puts that /32 into a BGP blackhole sink which stops traffic to the entire network for that /32. You could look to do something similar, either BGP blackholing it or if you have a single upstream, use something like expect to insert a firewall rule ?? With that said, it could be nice for some of this to exist in FS itself (as you say, slowing down responses over certain thresholds). Brent On Mon, Feb 21, 2011 at 3:07 PM, Spencer Thomason < spencer@5ninesolutions.com> wrote:
Hi, We run hosted Freeswitch instances in VMs with the internal profile on port 5060 connecting to clients mostly behind NAT and then the external profile connecting to our proxies only. Protecting the external profile its straightforward.. we only allow traffic to/from our proxies at the firewall level. But protecting the internal profile seems to be a bit more difficult because the UACs could be theoretically anywhere on the network.
I'm currently using Fail2Ban to prevent brute force registration and INVITEs on auth failures, e.g.: failregex = \[WARNING\] sofia_reg.c:\d+ SIP auth failure \(REGISTER\) on sofia profile \'\w+\' for \[.*\] from ip <HOST> \[WARNING\] sofia_reg.c:\d+ SIP auth failure \(INVITE\) on sofia profile \'\w+\' for \[.*\] from ip <HOST>
My question is, since its part of a normal SIP dialog to challenge the INVITE, is there any way to prevent a possible DoS from just sheer volume of incoming INVITEs on an Internet facing server automatically. I.e., If you block the logged challenge, you'd block all legitimate INVITEs and registrations. Since its UDP traffic I couldn't come up with a way to do it automatically at the iptables level. i.e. number of concurrent connections. Is there some option to just not respond if a client is sending a number of requests over a certain threshold? It might not stop them from sending the traffic but pretty soon they'd get the idea that it wasn't going to go anywhere. My concern is say there are 50 Freeswitch instances on a box (albeit 8 core, 32GB ram, 8 15K raid 10 storage) and someone starts sending thousands of rouge INVITEs to every VM on a physical box that the CPU load from just challenging the incoming INVITEs would create a DoS. We the logs regularly to try to catch people doing this sort of thing and drop them at a router upstream of the core network, but I'd like to have it happen without human intervention. Have I completely over thought this and am missing something obvious?
Thanks, Spencer
_______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
-- -- Brent Paddon Director | Over the Wire Pty Ltd brent.paddon@overthewire.com.au | www.overthewire.com.au Phone: 07 3847 9292 | Fax: 07 3847 9696 | Mobile: 0400 2400 54
participants (9)
-
Brent Paddon
-
covici＠ccs.covici.com
-
jay binks
-
Jay Binks
-
Jmesquita＠freeswitch.org
-
Ken Rice
-
mazilo
-
Spencer Thomason
-
Steven Ayre