Is Freeswitch terminating the call as well? Or media flows between IP Phone and external SIP Trunk.
Cisco Phones offer your CDR/CMR records which give you a good detail in terms of Media statistics. If using CUCM you need to enable it in Service Parameters.
You need to isolate it to:
1. Device, device firmware
2. Location
3. Time of the day when this happens.
4. Check all network path ( IP Phone -> Access switch -> Core switch -> Router -> Public Router -> Internet...) etc.
5. Validate correct NIC settings (speed) and LAN QoS.
6. Etc...