Now I have a quick question or 2 for you. Now You said it doesn't do it on inbound calls. Now for outbound calls is it the same number you are calling or different. What I think it might be the cause the Underline carrier that is handling the outgoing call, there switch for that specific route might be having issues. I've expierenced this with Onvoy and Bandwidth.com. Now incoming might be from Level 3, wether you use viteity or Flowroute. For outgoing they have different carriers that they utilize maybe one of those carriers is having issues. Log into the asterisk console and see which media server is handling the call, and from there you can determine which carrier it is. Hope this info helps.
Once you do that install MTR on the machine and run the command mtr and the Ip of the media server u get. You will get a detailed description where the packet loss occurs