[SOS] Help Needed with Strange Behavior
To all the crazy ones, all the gurus: TLDR: FreeSWITCH Server looses connectivity with main router (gateway), can ping any other server but cannot reach router. We have a cluster of FreeSWITCH servers running behind Kamailio. For 3 years we have been up and running smoothly without any glitches. Last week we upgraded our FreeSWITCH servers, the upgrade included: 1) Upgraded Debian Squeeze to Ubuntu LTS 14.04 2) Upgraded FreeSWITCH from version 1.2 to 1.4.18 (git: 4eed221) Ever since last week, a very strange problem happened 3 times. The FreeSWITCH servers randomly loose connectivity with our main router (gateway). FS servers can be reached and pinged from the rest of servers, they also can ping and reach all other servers but when we ping router from inside FS servers they cannot ping and reach the router. The only way we can get them working again is by: ifconfig eth1 down sleep 3 ifconfig eth1 up route add default gw 208.65.xxx.xxx eth1 The rest of our 20 servers are all running Ubuntu 14.04 and most of them are under heavy load, this only happens with FS servers. Does any one have any clue about this bizarre situation? Any one know under what circumstances a server might loose connectivity and routing to its gateway? Any pointer for debugging or suggestion is highly appreciated. Thank you, Moe
Whoever convinced you to use ubuntu must have been mad at you. Probably the problem is elementary networking mistake and nothing to do with FS. Look closer at the details is all I can suggest. If you do find FS has the power to derail you ethernet it will be a story long told. On Tuesday, May 12, 2015, Mohammad Amin Navid <manavid@gmail.com> wrote:
To all the crazy ones, all the gurus:
*TLDR: FreeSWITCH Server looses connectivity with main router (gateway), can ping any other server but cannot reach router.*
We have a cluster of FreeSWITCH servers running behind Kamailio.
For 3 years we have been up and running smoothly without any glitches.
Last week we upgraded our FreeSWITCH servers, the upgrade included:
1) Upgraded Debian Squeeze to Ubuntu LTS 14.04
2) Upgraded FreeSWITCH from version 1.2 to 1.4.18 (git: 4eed221)
Ever since last week, a very strange problem happened 3 times. The FreeSWITCH servers randomly loose connectivity with our main router (gateway).
FS servers can be reached and pinged from the rest of servers, they also can ping and reach all other servers but when we ping router from inside FS servers they cannot ping and reach the router.
The only way we can get them working again is by:
*ifconfig eth1 down* *sleep 3* *ifconfig eth1 up* *route add default gw 208.65.xxx.xxx eth1*
The rest of our 20 servers are all running Ubuntu 14.04 and most of them are under heavy load, this only happens with FS servers.
Does any one have any clue about this bizarre situation? Any one know under what circumstances a server might loose connectivity and routing to its gateway?
Any pointer for debugging or suggestion is highly appreciated.
Thank you,
Moe
-- Anthony Minessale II ♬ @anthmfs ♬ @FreeSWITCH ♬ ☞ http://freeswitch.org/ ☞ http://cluecon.com/ ☞ http://twitter.com/FreeSWITCH ☞ irc.freenode.net #freeswitch ☞ *http://freeswitch.org/g+ <http://freeswitch.org/g+>* ClueCon Weekly Development Call ☎ sip:888@conference.freeswitch.org ☎ +19193869900 https://www.youtube.com/watch?v=9XXgW34t40s https://www.youtube.com/watch?v=NLaDpGQuZDA
Hi Anthony, Thank you very much for your response. After almost 9 hours of debugging I was able to reproduce the issue. I went through all our logs and found out whenever a particular customer calls out, we loose connectivity with our router. I found out the customer had moved his IP Phone from his office to another location. That location has same private IP address as our servers. On all our servers we use 10.0.0.0/24 on eth0 and eth1 has public IP address. What I did next was isolated a FreeSWITCH server and spawn up a dedicated Kamailio to only send traffic to this FS server. I also changed my lab's LAN from 172.16.16.0/24 to 10.0.0.0/24. Here is what happens when I make a call: First call is fine and I get: switch_rtp.c:5846 Auto Changing port from 10.0.0.179:61856 to 108.178.144.243:10249 On second call, here is what happens: The value of rtp_session->flags[SWITCH_RTP_FLAG_AUTOADJ] is correctly set to 1 but the “bytes" variable is always at zero for following if statement on line 5833 in switch_rtp.c: if (bytes && rtp_session->flags[SWITCH_RTP_FLAG_AUTOADJ] && switch_sockaddr_get_port(rtp_session->from_addr)) I also did raw packet captures on eth1, I found out right after my FS receives SIP 183 with SDP from carrier, it sends out following: 1) an ICMP to my public IP address which is 108.178.144.243 2) an ARP broadcast “Who has 10.0.0.179? Tell 208.65.xxx.xxx” (208.65.xxx.xxx is the public IP of the FS server) Right after that, the router is no longer reachable and 10.0.0.179 will show up in arp table of FS with mac address of the router. I tested above scenario zillion times to make sure it’s reproducible before reporting back. What I did next was installing rtpproxy and made few changes to our Kamailio to rewrite SDP and pass the traffic through rtpproxy and when I make calls with rtpproxy in the middle, the problem does not occur. Any thoughts? Thank you, Moe
According you routing table 10.0.0.0/24 network is reachable via eth0. ARP request must be send via eth0 interface. Thinking is required to find why arp is send via "external" LAN segment to router. May be firewall rules, VLAN configuration mistake in LAN switch and etc. Also I recommend disable ARP proxy feature on router. Router must not respond to ARP request for not owned IP address. Sergey Safarov On Wed, May 13, 2015 at 12:05 PM, Moe Navid <manavid@gmail.com> wrote:
Hi Anthony,
Thank you very much for your response.
After almost 9 hours of debugging I was able to reproduce the issue.
I went through all our logs and found out whenever a particular customer calls out, we loose connectivity with our router.
I found out the customer had moved his IP Phone from his office to another location. That location has same private IP address as our servers.
On all our servers we use 10.0.0.0/24 on eth0 and eth1 has public IP address.
What I did next was isolated a FreeSWITCH server and spawn up a dedicated Kamailio to only send traffic to this FS server. I also changed my lab's LAN from 172.16.16.0/24 to 10.0.0.0/24.
Here is what happens when I make a call:
First call is fine and I get:
switch_rtp.c:5846 Auto Changing port from 10.0.0.179:61856 to 108.178.144.243:10249
On second call, here is what happens:
The value of rtp_session->flags[SWITCH_RTP_FLAG_AUTOADJ] is correctly set to 1 but the “bytes" variable is always at zero for following if statement on line 5833 in switch_rtp.c:
if (bytes && rtp_session->flags[SWITCH_RTP_FLAG_AUTOADJ] && switch_sockaddr_get_port(rtp_session->from_addr))
I also did raw packet captures on eth1, I found out right after my FS receives SIP 183 with SDP from carrier, it sends out following:
1) an ICMP to my public IP address which is 108.178.144.243
2) an ARP broadcast “Who has 10.0.0.179? Tell 208.65.xxx.xxx” (208.65.xxx.xxx is the public IP of the FS server)
Right after that, the router is no longer reachable and 10.0.0.179 will show up in arp table of FS with mac address of the router.
I tested above scenario zillion times to make sure it’s reproducible before reporting back.
What I did next was installing rtpproxy and made few changes to our Kamailio to rewrite SDP and pass the traffic through rtpproxy and when I make calls with rtpproxy in the middle, the problem does not occur.
Any thoughts?
Thank you,
Moe
_________________________________________________________________________ Professional FreeSWITCH Consulting Services: consulting@freeswitch.org http://www.freeswitchsolutions.com
Official FreeSWITCH Sites http://www.freeswitch.org http://confluence.freeswitch.org http://www.cluecon.com
FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
participants (5)
-
Anthony Minessale
-
Josh Richesin
-
Moe Navid
-
Mohammad Amin Navid
-
Sergey Safarov