Maybe not. Note that according to RFC3261, the response to a SIP request that does not have an rport tag in the Via header will be sent to the address/port in Via, not the address/port that the request came from. So, an attacker with the ability to spoof the source IP address could bypass your whitelist and still have the response sent to his IP address.
This is difficult because most data centers and ISPs filter outbound packets whose source address is not assigned to them. The attacker would either need to use a data center without such filtering, or have a server in a center used by VoicePulse.
If you have packet captures of some of the SIPVicious traffic, you can determine whether this actually happened.