Never underestimate the power of desperation and monkeying around with things you don't understand :-)
In case someone else runs into this problem:
After all else failed I figured that this may have something to do with conference settings since this problem never happened outside of a conference.
I started changing conference profile params. Lo and behold, after I changed the sample rate param to 12000 (from the default 8000) the problem went away. Interestingly, at 11999 it still happened but at 12000 or above, it completely disappeared.
Frankly, I have no idea why but I'll take it :-)