"for example, there are small cut-offs every now and then"

Have you tried setting "conference-flags" to "audio-always" in conference.conf.xml?