Finally solved this problem by:
1. outbound-codec-prefs to PCMU@20i,PCMA@20i
2. setting inbound-codec-negotiation = scrooge

Spent 4 hours on fixing that, maybe my solution will help someone with same problem in the future