I am facing same issue described in this old post http://lists.freeswitch.org/pipermail/freeswitch-users/2009-June/042823.html

I am using media_bypass mode.

If any more information like traces etc needed I can provide.