Have you tried setting this variable for the leg in question? http://wiki.freeswitch.org/wiki/Variable_ignore_display_updates