I had 4 forwarding servers specified on my DNS server. I decided to try connecting to them and doing DNS lookups... 2 of the 4 timed out when using them for lookups. They were the two at the top of the forwarding list. I removed them from the list and now the internet is fast again.
I have no idea why this worked faster when the DNS server was plugged into a different switch... but after swapping the switches this problem became apparent but it looks like the problem was on the DNS server... not the switch.