fail2ban started banning vitelity

When running iptables -L the fail2 ban chains do come up first:

 iptables -L
Chain INPUT (policy DROP)
target     prot opt source               destination
fail2ban-VSFTPD  tcp  --  anywhere             anywhere            tcp dpt:ftp
fail2ban-BadBots  tcp  --  anywhere             anywhere            multiport dports http,https
fail2ban-APACHE  tcp  --  anywhere             anywhere
fail2ban-ASTERISK  all  --  anywhere             anywhere
fail2ban-SSH  tcp  --  anywhere             anywhere            tcp dpt:ssh

The ip of the proxy was in the ignorelist, but it still banned it.

In my iptables file:

*filter
:INPUT DROP [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:fail2ban-APACHE - [0:0]
:fail2ban-ASTERISK - [0:0]
:fail2ban-BadBots - [0:0]
:fail2ban-SSH - [0:0]
:fail2ban-VSFTPD - [0:0]
-A INPUT -p tcp -m tcp --dport 21 -j fail2ban-VSFTPD
-A INPUT -p tcp -m multiport --dports 80,443 -j fail2ban-BadBots
-A INPUT -p tcp -j fail2ban-APACHE
-A INPUT -j fail2ban-ASTERISK
-A INPUT -p tcp -m tcp --dport 22 -j fail2ban-SSH
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m tcp --tcp-flags ACK ACK -j ACCEPT
-A INPUT -m state --state ESTABLISHED -j ACCEPT
-A INPUT -m state --state RELATED -j ACCEPT

Where it starts out with *filter, and I place an -A INPUT before :INPUT DROP?