Ablikim's Blog

Recently we migrated some servers to a new IP address. About a day after the migration, a client complained that their email was delayed. When we investigated, we found 10,000’s of emails jamming up the queue.

The immediate suspects were a open relay, compromised user password or web application exploit. After ruling these out, we found the problem that was obvious in hindsight.

When we migrated the server, we used IPtables to forward IPs during DNS propagation. This meant clients would see minimal impact during the migration. Emails would go through, web sites would resolve and people can pop their mail.

When looking at the email queue, we saw that all emails came from the forwarding server. This was rather odd. We had actually tried to relay mail through the forwarding server but it failed.

As it turns out, the problem was with POP before SMTP. Due to the forwarding, the server was seeing the IP for the forwarding server not the client’s IP. As a result, the forwarding server’s IP would be whitelisted to send email for 10 minutes. Given the large number of email users on the system, I suspect it was open a large portion of the time.

A spam bot found the IP and considered it an open-relay and started flooding in the messages.

The lesson — when using IP forwarding always consider any services that may cache the referring IP. We’ve seen this once before with DOS prevention type tools and bandwidth throttling systems. We had never considered the impact on POP before SMTP.

Of course, this is one reason why SMTP AUTH is preferred.