What's your load? Type w at the command line or run something like htop.
N2800 is a pretty weak processor. My backup box is a Intel NUC with an i5-5250U @ 1.60GHz. It benchmarks at 3646. Your processor benchmarks at 633. Generally, voip is low on CPU demands for typical codecs like ulaw, not sure if you're hitting some limit. load will tell you.
My main box is a HP Microserver with a Xenon that benchmarks the same as that i5. That might be overkill for you, but I like it because it has real server RAM, HP server quality, compact size, and a lot of disk bays. Currently I just do 2x 500gb drives doing software raid1.
Its amazing how powerful the NUC is for the money. Not sure how I feel about a fanless design running a server but its fine as a backup and I hear some small business voip vendors use it a lot for their clients.