Thanks Alexey for your help in diagnosing the problem.
I dug further into the TCP stack and found out we are having 2 separate restart problems.

The first problem is not that the socket is failing to be released, but apparently Sofia is trying to open the port without the SO_REUSEADDR option, which means FS must wait for all connection to close first.
So as long as there are any TCP connections to those ports in the TIME_WAIT state, FS will not start Sofia.
I tested with netcat and that program can bind to the port regardless of the state of old connections, so I suspect this is a compile error.

The second problem is that the TLS and TCP registrations in sofia_reg_internal.db cause FS to throw the assertion error “Assertion failed: orq->orq_queue, file nta.c, line 8101”.
UDP registrations are fine.
Does FS try to continue these registrations assuming that the connection is still up or reopen a connection to the same port?
The assert line is found in nta.c’s outgoing_create function. It seems that the outgoing message queue is uninitialized, so the assertion error is likely more a symptom of some other problem, instead of being the cause of the problem.
How should I go about debugging the outgoing message queue?


Thanks in advance,
Mikko

---------- Edelleenlähetetty viesti ----------
From: Alexey Sibyakin <alex@freeswitch.com>
To: FreeSWITCH Users Help <freeswitch-users@lists.freeswitch.org>
Cc: 
Bcc: 
Date: Tue, 13 Feb 2018 17:23:06 +0900
Subject: Re: [Freeswitch-users] FS fails to restart when TLS endpoints are registered (Solaris 11.3)
It looks like Sofia is not releasing ip on Solaris. If so it's a bug and it should be reported to Sofia's bug tracker or Solaris or both. If you can reproduce this problem on supported OS (Debian 8 at this moment) you can open an issue on FreeSWITCH Jira.


On Tue, Feb 13, 2018 at 2:30 AM, GBC Networks Oy <info@gbc.fi> wrote:
We're running FS on Solaris 11.3 and it is working perfectly, except that we're unable to restart FS without deleting the TLS registrations from sofia_reg_internal.db.
Our FS version is FreeSWITCH Version 1.9.0+git~20171010T231940Z~5940b04d4e~64bit (git 5940b04 2017-10-10 23:19:40Z 64bit)

Here are the symptoms:
1.    Service stops FS with fs_cli -x "fsctl shutdown now"
2.    FS stops and the service is restarted
3.    FS starts up and fs_cli is working, but ” sofia status” is empty
The console shows:
freeswitch@fs> 2018-02-12 17:06:23.875991 [ERR] sofia.c:3221 Error Creating SIP UA for profile: internal (sip:mod_sofia@123.123.123.123:5060;maddr=123.123.123.123;transport=udp,tcp) ATTEMPT 3 (RETRY IN 5 SEC)                                                                                                        
2018-02-12 17:06:23.875991 [ERR] sofia.c:3231 Error Creating SIP UA for profile: internal (sip:mod_sofia@123.123.123.123:5060;maddr=123.123.123.123;transport=udp,tcp)                                                                                                                                                   
The likely causes for this are:                                                                                                                            
1) Another application is already listening on the specified address.                                                                                      
2) The IP the profile is attempting to bind to is not local to this system.

Note: the IP is correct and nothing is listening on the port.

4.    Next we stop FS and restart
This time the console shows:
2018-02-12 13:33:51.132676 [INFO] switch_core_sqldb.c:1693 sofia:internal Starting SQL thread.
2018-02-12 13:33:51.132770 [DEBUG] sofia.c:3373 Starting thread for internal
2018-02-12 13:33:51.132777 [DEBUG] sofia.c:3020 Launching worker thread for internal
2018-02-12 13:33:51.138255 [NOTICE] sofia_reg.c:3398 Added gateway '5e4ac47f-6ce4-462d-9247-565711fe3d4e' to profile 'internal'
2018-02-12 13:33:51.138405 [NOTICE] sofia_reg.c:3398 Added gateway 'b2563fad-0051-4067-abc3-6a431e05481e' to profile 'internal'
2018-02-12 13:33:51.247731 [DEBUG] freeswitch_lua.cpp:372 DBH handle 125b300 Connected.
2018-02-12 13:33:51.254128 [DEBUG] freeswitch_lua.cpp:401 DBH handle 125b300 released.
Assertion failed: orq->orq_queue, file nta.c, line 8101

5.    sqlite3 sofia_reg_internal.db "DELETE from sip_registrations WHERE status LIKE '%TLS%'"
6.    FS starts normally


Here is a registration (non-TLS) that doesn’t break sofia:
# sqlite3 -header sofia_reg_internal.db "SELECT * from sip_registrations"
call_id|sip_user|sip_host|presence_hosts|contact|status|ping_status|ping_count|ping_time|force_ping|rpid|expires|ping_expires|user_agent|server_user|server_host|profile_name|hostname|network_ip|network_port|sip_username|sip_realm|mwi_user|mwi_host|orig_server_host|orig_hostname|sub_host
313531383139323735393532323132-m5mp9qlsdu27|USERNAME|fs.domain.com||"FS" <sip:USERNAME@REMOTE-IP:2054>|Registered(UDP)|Reachable|3|39824|1|unknown|1518455533|1518453577|snom320/8.7.5.35|USERNAME||internal|fs|REMOTE-IP|2054|USERNAME|fs.domain.com|USERNAME|fs.domain.com||fs|fs.domain.com

Here is a registration (TLS) that breaks the restart:
# sqlite3 -header sofia_reg_internal.db "SELECT * from sip_registrations"                             
call_id|sip_user|sip_host|presence_hosts|contact|status|ping_status|ping_count|ping_time|force_ping|rpid|expires|ping_expires|user_agent|server_user|server_host|profile_name|hostname|network_ip|network_port|sip_username|sip_realm|mwi_user|mwi_host|orig_server_host|orig_hostname|sub_host
313531383139323735393532323132-m5mp9qlsdu27|USERNAME|fs.domain.com||"FS" <sip:USERNAME@192.168.6.236:2170;transport=tls;fs_nat=yes;fs_path=sip%3AUSERNAME%40REMOTE-IP%3A2170%3Btransport%3Dtls>|Registered(TLS-NAT)|Reachable|0||1|unknown|1518457302|1518453664|snom320/8.7.5.35|USERNAME||internal|fs|REMOTE-IP|2170|USERNAME|fs.domain.com|USERNAME|fs.domain.com||fs|fs.domain.com


Best regards,
Mikko

_________________________________________________________________________
Professional FreeSWITCH Consulting Services:
consulting@freeswitch.org
http://www.freeswitchsolutions.com

Official FreeSWITCH Sites
http://www.freeswitch.org
http://confluence.freeswitch.org
http://www.cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org



--

Alex Sibyakin | Support Engineer
Need commercial support? Contact sales@freeswitch.com for details.

_______________________________________________
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org