I am very new to FreeSWITCH but I have been using it. Also I used Newfies and found the same problem.
I just re sampled all my recordings to WAV in 8000Hz format and it fixed the problem. I do not know much about things but could this maybe be a codec problem in that your agents recordings that "work" are connecting to system with XXX codec yet your other agent that is "not working" are connecting with a different codec?