While waiting for suggestions, I tried more things. In particular, I tested whether gentls_cert was present in our FS install (which is at /opt/freeswitch/ on ubuntu).
It is present, but the CA root cert step writes to {prefix}/etc/freeswitch/tls/CA/ instead of {prefix}/conf/ssl/CA/. In particular, the root CA step generates:
etc/freeswitch/tls/
agent.pem
etc/freeswitch/tls/CA/
cacert.srl
Reviewing agent.pem shows it's fine:
openssl x509 -noout -inform pem -text -in /opt/freeswitch/etc/freeswitch/tls/agent.pem
But it's owned by user root group root, so:
cd /opt/freeswitch/etc/freeswitch/tls/
chmod 640 agent.pem CA/cacert.pem
chown root.freeswitch agent.pem CA/cacert.pem
Then I edited /opt/freeswitch/etc/freeswitch/vars.xml to set internal_ssl_enable and external_ssl_enable to true.
Then I restarted FS. I checked the CLI and it shows "WS SETUP FAILED" repeatedly. *Any suggestion?*
Blazing ahead...then I exposed the public IP of the FS machine under a subdomain of the CA root cert's domain. (I used a wildcard subdomain for -org when generating both certs; maybe giving a wildcard this way is unnecessary or counterproductive.)
Now, the part that puzzles me: I'm already using a "real CA" cert in order to serve my verto client files over https from my DNS name, so browsers won't show security warnings. But I bet I need to have the same CA root cert installed in FS as I use in the webserver, right? I saw the note that commercial certs should work, but *it's not clear what steps to follow to install one*.
Cheers,
David