Well well well
They pointed me to this:
https://doc.pfsense.org/index.php/VoIP_Configuration
And what I noted is this:
A keep-alive or re-registration on the phone set for 20-30 seconds or so can also help, and is often a better solution.
So I tried to add these settings to my trunk:
qualify=yes
qualifyfreq=20
keepalive=20
But looking at the sip debug in CLI, I still get:
Expiry for XXX is 120 sec (Scheduling reregistration in 105 s)
How can I make it so everything is set up so Asterisk checks the connection every 20 seconds?