I've run the FreePBX Distro in ESXi for more than five years on a core i3 with 4gb of RAM without any issues, and even in VMWare Player (on Windows) intermittently when I was making changes to the ESXi server, and never had any issues whatsoever. I know someone else that ran FreePBX on VMWare Player 3.1 for over a year without issues. Whatever problem you're having, it is not likely to be related to ESXi.
I also doubt that it is a configuration issue, though modifying configuration parameters (such as increasing the jitter buffer) may help mask the issue.
Have you tried registering your client directly to your ITSP and seeing if the problem persists? The easiest way to resolve this problem will be to remove one potential source of the issue at a time until it goes away. Take out the PBX by registering directly to a provider, then try another ITSP, then swap your router, then swap to another internet connection (use another location if you have to), etc., until you figure it out.
I vote for a networking issue. Do you have a Wifi transmitter or any other device that generates RF (cordless phones, etc) near any of your networking equipment (switches, router, modem, etc.)?? Those device can often interfere - especially with modems and routers and switches. Move them far away from those types of devices.