[Freeswitch-users] [Solved] FS shutdown not closing TCP port causes TIME_WAIT problems

Wed Apr 4 02:37:47 MSD 2012

Happy to report (for my sake) that using the exact command below (fs_cli in Terminal) did not solve the problem (I tested 4 times), ports went to TIME_WAIT. Good idea though...  I was able to duplicate it intermittently on OpenSUSE but consistently on OSX. I will add this tidbit to the OSX wiki when I update it, hopefully soon. Thanks for the suggestion.
Mario G

On Apr 3, 2012, at 2:28 PM, Anthony Minessale wrote:

> I meant the command it uses.  One guess is that since you are using
> fs_cli the shutdown probably kicks you off the event socket mid
> shutdown while the rest of FS is still shutting down and it tries to
> restart it again too soon.
> 
> you could try fs_cli -x "fsctl shutdown now"
> this will instantly end FS
> 
> otherwise maybe its just a "mac thing" as suspected.
> 
> 
> On Tue, Apr 3, 2012 at 4:10 PM, Brian Foster <bdfoster at endigotech.com> wrote:
>> Even doing "fsctl shutdown restart" would restart freeswitch.
>> 
>> -BDF
>> 
>> On Apr 3, 2012 5:09 PM, "Brian Foster" <bdfoster at endigotech.com> wrote:
>>> 
>>> fs_cli -x "fsctl shutdown gracefully"
>>> 
>>> -BDF
>>> 
>>> On Apr 3, 2012 4:48 PM, "Mario G" <mario_fs at mgtech.com> wrote:
>>>> 
>>>> If you mean the effect, it did not stop FS although the results were the
>>>> same, it prevented the fast FS shutdown/startup from activating the internal
>>>> profile so none of the phone extensions work. In my case that meant all
>>>> phones were out from 2am to whenever I activated internal. Not good in case
>>>> of an emergency. Hope it may be useful to someone, it took me 15 days to
>>>> find and resolve.
>>>> 
>>>> If you mean how is FreeSwitch shutdown, the 2am script runs a script that
>>>> runs one command: /usr/local/freeswitch/bin/fs_cli -x "shutdown"
>>>> 
>>>> On Apr 3, 2012, at 11:24 AM, Anthony Minessale wrote:
>>>> 
>>>> How exactly does it "stop" freeswitch?
>>>> 
>>>> On Tue, Apr 3, 2012 at 11:17 AM, Mario G <mario_fs at mgtech.com> wrote:
>>>> 
>>>> I am posting this in case someone else runs into this, especially since I
>>>> 
>>>> found it may not be Mac OSX specific (I was able to duplicate on SUSE
>>>> 
>>>> although not as much).
>>>> 
>>>> 
>>>> Both of the following conditions must exist:
>>>> 
>>>> 1. You have at least one TCP local extension. In this case, due to Bria,
>>>> 
>>>> etc. on an iPad, using TCP reduced battery usage since no need to run in
>>>> 
>>>> background.
>>>> 
>>>> 2. FreeSwitch is recycled very quickly, in this case down/up within 30
>>>> 
>>>> seconds because it was automated at 2am each day. I needed it due to a
>>>> long
>>>> 
>>>> running sofia memory leak on OSX which may be resolved but I need to
>>>> test.
>>>> 
>>>> 
>>>> Problem: The extension using TCP cause a TIME_WAIT that keeps the 5060
>>>> port
>>>> 
>>>> active (but NOT open) on the TCP stack, in my case it was 30 seconds but
>>>> I
>>>> 
>>>> found the time could be different on other platforms. When FreeSwitch
>>>> comes
>>>> 
>>>> back up the "internal.xml" profile fails because port 5060 is in use. If
>>>> FS
>>>> 
>>>> is brought up AFTER the TIME_WAIT goes away there is no problem, but in
>>>> my
>>>> 
>>>> case it was very quick each night to minimize the outage.
>>>> 
>>>> 
>>>> Solution: I changed the OSX plist that auto restarts FS to run a script
>>>> that
>>>> 
>>>> starts FS, this makes no difference except a script is more flexible.
>>>> Rather
>>>> 
>>>> than use an arbitrary sleep time I check for port 5060 which is more
>>>> 
>>>> reliable and faster. I tested this many times using different scenarios
>>>> and
>>>> 
>>>> all worked. Note I tried to use the "lsof -i TCP:5060" command but it
>>>> shows
>>>> 
>>>> only OPEN ports so it won't work.
>>>> 
>>>> 
>>>> #!/bin/sh
>>>> 
>>>> while : ; do
>>>> 
>>>>    a=$(/usr/sbin/netstat -p tcp)
>>>> 
>>>>    if [[ "$a" != *sip* ]] ; then
>>>> 
>>>>      break
>>>> 
>>>>    fi
>>>> 
>>>>    echo "Port 5060 still in use - waiting for it to free up"
>>>> 
>>>>    echo "${a}"
>>>> 
>>>>    /bin/sleep 3
>>>> 
>>>> done
>>>> 
>>>> /usr/local/freeswitch/bin/freeswitch -nc -nonat
>>>> 
>>>> 
>>>> #   a=$(/usr/sbin/lsof -i TCP:5060)  <- only shows open ports
>>>> 
>>>> #   if [ "$a" = "" ] ; then
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Apr 3, 2012, at 8:53 AM, Mario G wrote:
>>>> 
>>>> 
>>>> For anyone who may run into this see "FS shutdown not closing TCP port
>>>> 
>>>> causes TIME_WAIT problems" for follow up and solution.
>>>> 
>>>> On Apr 2, 2012, at 1:33 PM, Mario G wrote:
>>>> 
>>>> 
>>>> That's what I was thinking, trying to figure out how now. Will post a
>>>> 
>>>> solution when I get one.
>>>> 
>>>> 
>>>> On Apr 2, 2012, at 1:28 PM, Anthony Minessale wrote:
>>>> 
>>>> 
>>>> From what I see, sofia uses the REUSEADDR flag so I will bet its yet
>>>> 
>>>> 
>>>> another MAC specific thing.
>>>> 
>>>> 
>>>> I am not really sure what you can do since you tend to find issues
>>>> 
>>>> 
>>>> deep inside libsofia-sip and its portability to OSX.
>>>> 
>>>> 
>>>> Maybe make your restart script sleep a bit?
>>>> 
>>>> 
>>>> 
>>>> libs/sofia-sip/libsofia-sip-ua/tport/tport_type_tcp.c:
>>>> 
>>>> 
>>>> su_setreuseaddr(socket, 1);
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Mon, Apr 2, 2012 at 3:13 PM, Mario G <mario_fs at mgtech.com> wrote:
>>>> 
>>>> 
>>>> If FS is shutdown and started too quickly, the internal profile ip/port
>>>> 
>>>> stays in use and FS will fail to start it. After searching on the wen I
>>>> 
>>>> found many references to TIME_WAIT but all fixes were to the program
>>>> source.
>>>> 
>>>> So I am wondering if when FS/sofia shuts down they are not releasing the
>>>> 
>>>> ip/port for internal. If it's true, I found some source changes to solve
>>>> it.
>>>> 
>>>> Google "TIME-WAIT" and you'll find lots of good stuff like
>>>> 
>>>> 
>>>> http://serverfault.com/questions/329845/how-to-forcibly-close-a-socket-in-time-wait.
>>>> 
>>>> Apparently there is no way to free up the port manually. Help/comments
>>>> 
>>>> anyone?
>>>> 
>>>> 
>>>> Mario G
>>>> 
>>>> 
>>>> _________________________________________________________________________
>>>> 
>>>> 
>>>> 
> 
> -- 
> Anthony Minessale II
> 
> FreeSWITCH http://www.freeswitch.org/
> ClueCon http://www.cluecon.com/
> Twitter: http://twitter.com/FreeSWITCH_wire
> 
> AIM: anthm
> MSN:anthony_minessale at hotmail.com
> GTALK/JABBER/PAYPAL:anthony.minessale at gmail.com
> IRC: irc.freenode.net #freeswitch
> 
> FreeSWITCH Developer Conference
> sip:888 at conference.freeswitch.org
> googletalk:conf+888 at conference.freeswitch.org
> pstn:+19193869900
> 
> _________________________________________________________________________
> Professional FreeSWITCH Consulting Services:
> consulting at freeswitch.org
> http://www.freeswitchsolutions.com
> 
> 
> 
> 
> Official FreeSWITCH Sites
> http://www.freeswitch.org
> http://wiki.freeswitch.org
> http://www.cluecon.com
> 
> FreeSWITCH-users mailing list
> FreeSWITCH-users at lists.freeswitch.org
> http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
> UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
> http://www.freeswitch.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.freeswitch.org/pipermail/freeswitch-users/attachments/20120403/2b0c6b25/attachment-0001.html 