The switch.conf.xml setting controls a filter in the core that affects all logging. You can adjust it at runtime with the fsctl API.
The logging level to freeswitch.log is controlled through the <mappings> section, which doesn't seem to be documented on Confluence or the old Wiki yet. Look at conf/vanilla/autoload_configs/console.conf.xml in the git repo for a description of those settings. Essentially you list the levels to log in the 'all' mapping in most use cases.
fs_cli picks it's own loglevel by either through an argument or the /log command.
-Steve