Monday, September 9, 2013

Server vs Server

Years ago I was the head tech at a small company. I was in charge of pretty much all of IT, except the things I talked the owner into hiring additional staff to take over. Legislation like CAN-SPAM and technology like ubiquitous server side spam filtering were off in the distant future, so dealing with SPAM had more of a wild west aspect than it does today.

One day I was looking into the performance issues we were having with the mail server. It was taking staff members ages to download and sort though their email over the assorted dial up connections used at the time. Naturally the bulk of the issues were spam and mailing list related. Server side mail filters that fell within the company owner's budget (free or stolen, not Open Source, that's Communist!) lacked the sophistication to address the problem effectively. I ended up configuring a complex web of routing rules to remove most the detritus. I sent unsubscribe requests to the non-work related mailing lists that seemed vaguely respectable. My goal was to stop as much of the incoming garbage as I could so I had less post-hock deletion and filtering to deal with.

One mailing list in particular was problematic. The content was overtly racist, but avoided the racial slurs that would have been caught by my manually configured rules. It also used a series of different domains to differentiate the content. Jew bashing jokes, for example, came from a different bank of URLs than the ones mocking Asians.  Since this set of related sites were responsible for most the garbage still bogging down the mail server I tracked down the originating site and found a phone number for "Joe." I quickly realized Joe was a one man operation. My request was simple and delivered politely, don't send any more mail to our domain. 

"I only send mail to a domain if someone subscribes and the form gives me permission to send a subscribe offer to anyone else on the domain."

I found this to be perplexing logic and replied, "Whoever is subscribing lacks the authority to give you that kind of permission."

"Not my problem," Joe said," Take it up with them."

"Who was the original subscriber?"

"Privacy laws. Get a warrant if you want to know."

"Then just remove my domain from-"

"Stop right there kid. It would take me WEEKS to scrub your domain from my mailing lists."

"Sounds like your mailing list management software was written by an idiot."

"I wrote it, and I don't need to scrub whole domains. Why does it bug you anyway? Are you one of THEM?"

I thought for a moment and said, "I'll just configure the mail server to send an "unsubscribe" request to anything from one of your domains."

"Good luck finding them all b****h."

I read off a list of domain names and ended with, "Did I miss any?"

There was silence for a few seconds. 

"Still there?" I asked. 

"You fu**ing hacked me."

"Nope. Just a little research. It took about an hour."

Joe hung up. 

I set up a rule to move any e-mail from the flagged domains to a dummy address I set up, then delete the message in the original mailbox. I then wrote a PERL script to check the mailbox, extract the "unsubscribe" address and send an "unsubscribe" request for the address the message was originally sent to.

Since it was now about 5:30 on a Friday I went home for the weekend and left my new system running.

When I got back on Monday the sales guys, some of who came in early to get started on correspondence, complimented me on how quickly they could get their e-mail. A few hours later the company owner, I'll call him "Dan," came in and said "I just got a call from a guy saying you hacked his server."

It took about two hours to sort out what was going on. The racist mailing list sent an email asking for confirmation in response to every unsubscribe request. My script responded to this with another unsubscribe request. This meant a single e-mail generated an endless series of back and fourth messages.
"Unsubscribe"
"You Sure?"

"Unsubscribe"
"You Sure?"

"Unsubscribe"
"You Sure?"

"Unsubscribe"
"You Sure?"

Since the racist mailing list sent us hundreds of individual e-mails a day this resulted in a LOT of e-mail. The large glut of traffic was not noticed at our end because the individual staff members were able to download their largely SPAM free e-mail quickly and easily, with the dial-up bottleneck masking any server side issues. The only connection that was bogged down was the one used by the spare computer I'd set up to send the unsubscribe requests. The mailing list server however was not faring as well. My script didn't rout e-mail through our server, but instead connected directly to the mail server processing the "unsubscribe" requests. I'd done this to reduce the load on our own mail server.

It took another hour to convey all of this to Dan in a way he could understand. He then asked what was so horrible about the mailing lists anyway.

"Well, the messages are huge. Between downloading and having to delete them they waste a lot of staff time. Then there's the content."

Dan was an Irish American. He proudly donated to the IRA to 'Defend Ireland.'" The racist mailing list included four different domains used to send jokes bashing the Irish. A lot of Polish and Scottish jokes were re-purposed for the mailing list. Then there were the images included in the mailing list, many of which depicted Irish men as flaming homosexuals with a keen interest in sheep. Dan was VERY homophobic.
"He's kinder to the Irish than he is to African Americans. This is nothing compared to how he treats the Jews though."
We got Joe on the speakerphone. Joe and Dan spent about ten minutes yelling at each other. Joe accused Dan of being a Jew shill, and Dan accused Joe of racism. Dan conveyed a muddled, but essentially accurate, explanation of what was happening at a technical level and ended with, "So go ahead and call the feds and say we're hacking you, but you and I both know you'd be lying, and the feds will figure that out."

"Stop sending me mail!" Joe screamed.
"You first," I said. "Stop ignoring our unsubscribe requests and this will stop on its own."

I heard furious typing and Joe said, "I'm blacklisting your entire f**ing domain now."

"But you said that would take weeks."

The typing was now punctuated with a string of repeated obscenities and mangled "you mamma" insults. After a few seconds he hung up.

I walked over to the PC running my PERL script. I stopped the script and emptied the dummy account's message backlog. Messages were still coming in, but after about an hour they started tapering off, finally stopping by 2:00 pm.

At 4:30, the Comptroller stormed in and demanded to know why her joke mailing lists weren't coming in anymore.

"You should discuss this with Dan," I told her, "He was part of the decision."

"Oh, I WILL," she said, her voice dripping with venom. She then stormed off in a huff. Normally when the comptroller threatened me, there was a subsequent meeting with Dan, or even an all hands meeting where I had to do things like justify the use of passwords on network logins against accusations of it being a "paranoid" practice. Strangely enough, I never heard of this particular issue again.

No comments: