Threat Level: green Handler on Duty: Brad Duncan

SANS ISC: Frustrations of ISP Abuse Handling - SANS Internet Storm Center SANS ISC InfoSec Forums


Sign Up for Free!   Forgot Password?
Log In or Sign Up for Free!
Frustrations of ISP Abuse Handling

I am the Abuse Coordinator for a small ISP in the Midwest and am very receptive and proactive when dealing with spam originating from our network.  I monitor log reports from servers and firewalls, have subscribed to all of the FBL's that I am aware of, participate in an abuse listserve, review our domain information on MS site, SenderBase and Trusted Source daily, and resolve to eliminate spam from our network as quickly as possible often times before we even receive the first official notification. We have been under a barrage of spam attacks from various ip addresses all over the world just like many others have reported and have felt the pain of email DOS first hand.  We recently implemented a Red Condor filtering system blocking over 24 million spam emails from just one of our domains in the first 3 weeks of December.  We know first hand the damage that can be done by spam.  We strive everyday to work with our customers to reduce the amount of spam coming from their home computers as well as with our business customers to ensure that they secure their mail servers to prevent abuse.  As soon as abuse is discovered it is handled.

So where am I going with this?  I am frustrated with organizations such as Trend Micro, Sorbs, etc that block IP's for NO reason whatsoever.  They simply don't like the "server name" that was chosen or the way the IP is identified in ARIN registration. One example of one of our business mail servers that was blocked because they didn't like the name....  da2.our.domain (real name masked).  They assumed that da2 stood for "dialup access" instead of "direct admin".  There had been absolutely no spam reported from the box but because they THOUGHT it was a dialup computer they blocked the IP.   We recently have been battling blocklists that are preventing email from being delivered simply because our ARIN listing does not indicate that the IP address is static.  Now these are legitimate mail servers on IP addresses that are statically assigned to our customers.  There has been absolutely no spam reports from any of the servers yet they are being blocked from sending legitimate email.  

The companies that are doing this have taken it upon themselves to act as god of the Internet.  They insist that we comply with their demands, in the manner that THEY want it done and because we won't comply they will not allow legitimate emails to be delivered.  One of the servers that they have blocked is a mail server for a small city government, for their police dept, fire dept, and EMS dept.  It was explained to Trend Micro that they were endangering the well being of this small community without justification.  I asked them if they had any examples of spam originating from the IP's and they indicated they had none. They sent an email with what needed to be done to comply with their rules.   They said that we had to comply or they WILL NOT remove the block.  

Some of you are probably thinking - why don't you just do what they want done so that it doesn't happen again?  We have considered that.  However, last week it was SORBS, this week it is Trend Micro, next week someone else, the next week someone else and we will end up spending all of our time trying to comply with every one of these groups that comes along.  We were told by Trend Micro that they want all mail servers to indicate that they are mail servers by using mail. or smtp. for the server names.  We don't control our customers mail servers. We don't tell them what they have to name the server and many times we don't even know that they put up a mail server unless they have problems delivering or receiving mail.  We don't have time to be big brother to our customers.  If the customer violates our AUP, if the customer's IP is reported for spam or copyright infringement it is handled immediately.  Otherwise, it is up to the business themselves and their IT folks what they do with the static IP's that are assigned to them.

Some people complain that the ISP's aren't doing enough to keep the Internet free from spam and malicious activity and you may be right.  It could be because the ISP's are spending all of their free time playing games with the Internet Big Brothers.  I for one am tired of hearing the criticism of ISP's, of the complaining that we aren't doing enough.  I know folks that work for other small ISP's such as ours and I know that they too are doing their best to stay ahead of the game.  I think it is time for all of the "Big Brothers" out there to get a clue, you are doing more damage to the Internet by your lack of responsibility then all of us put together.  Until we all agree to one standard, until the Internet "police" provide all of us with one set of rules that we all have to comply with we will continue to fight the battle of not only spam but also the differing opinions on how these lists should be handled.  

If these companies want to set rules, why not use SPF (Sender Policy Framework) to set these rules.  SPF has been in place for a long time and has been a recommended standard.  We are working towards SPF records for all of our mail servers and hope to have all SPF records completed within the next week.  Will this be enough to satisfy the "big brothers"?  We are also setting up RDNS records for our customers mail servers that we are aware of.  Will this be enough?  I am all for blocking computers, mail servers or home computers that are identified as sending legitimate spam.  If one of our devices is spamming, I block it on our network before it ever gets to yours.

The frustrating thing about all of this is that I know that these companies are making big bucks selling a product to their customers that will break the customers ability to receive email.  Are these companies explaining that to the customers?  Obviously not.  The folks emailing our customers expecting a response from our customers don't have a clue that it is their "filter" that is preventing the delivery of the email.  It is this Handlers opinion that we will all really need to take a step back and learn to work TOGETHER to resolve the spam problem without causing more issues for an already stressed business community.  

 

Deb Hale Long Lines, LLC

Deborah

278 Posts
ISC Handler
Well yes, Deb, but I'm fascinated as to what you think 'the industry' should do about it.
I work for one of the big tehhnology vendors; we don't currently sell 'antivirus', it's more that we buy it nowadays.

There's a rule we have, 'customers and employees only'. What business can do anything for anyone else ? Customers get warranties, employees get best-effort. We are very good, but we are too expensive for the 'domestic and small business' whose computers are typically compromised to send all the junk mail and other abuse.

Personally, I think the only solution is diversity. A mix of Windows, Linux, OSX, and various Unixes, and so on, is the only way to run a public Internet.

I have an engineering background, too. And I'm a little frustrated that my employer won't let me take the gloves off and teach the children in the local schools properly.

But that would cause such a commercial upset, that I understand why he doesn't.

Someone else has to do it. We cheer you on. We will not stop you. Keep up the good work.
Anonymous
I think that the problem is actually more extensive than you point out. My own particular "war story" ...

A while back I had a problem with my e-mail (from a large, well-known ISP) being blocked by the mail servers at a client where I was engaged at the time. After much "poking about" I found the block list entry and contacted the relevant people in the clients organization to point out that their block list was cutting me off as well as a potentially large population of customers. The response I got from the technology people in charge of mail service surprised me... I would paraphrase their response as: "We are measured on the amount of SPAM we block, blocking legitimate mail isn't considered a problem as long as we block the SPAM."

My point is that the problem goes deep into organizations where technology policies ("Block SPAM") are implemented that don't accurately reflect business policies ("Encourage customers to contact us via e-mail").
Don

3 Posts
Whilst I agree that Trend Micro and some others I could mention, go way too far (indeed, I know at least one blocklist provider that requires you pay them before they'll remove you, and they don't provide evidence to backup their block either), I must point out that many of the larger ISP's are the reason we demand so much of ISP's (just for clarity, I run or manage, several blocklists/blacklists myself, though none are SMTP related).

However, in saying this, evidence of activity should always be provided when asked for (there are of course, exceptions to this, such as when CP is involved, as was the case with an ISP I dealt with recently (UK law doesn't allow sharing such information with anyone other than LE/IWF etc)), and if non exists, the block should be removed immediately, without demanding anything.

Many of the larger ISP's, have for years, completely ignored the abuse within their networks (and I could name quite a few that actively encourage malicious activity), only seeming to do something, when they've been publicized, and some, such as Lunarpages, that carry on ignoring it even after being notified about it, and it's being publicized.
Anonymous
SPF is busted by design, don't use it.

But I feel your pain in the report frontier. I've reported many hosts "infected" with a SSH/FTP bot to the respective ISP's, and all I'd get back is a robotic thank you letter, and weeks/months later the offending host would still be pounding on my firewall. I gave up and started CIDR blocking networks.

If ISP's can't cope with their workload, they should perhaps consider hiring more people. But in places where the "Big Boys" are running the show, perhaps they should consider actually acting on the information people provide.

Roadrunner, Verizon, Comcast, Rogers... You know who you are.
Anonymous
I don't really hold the ISPs responsible. They are the 'Bus Drivers' in this scenario; common carriers, by choice and for a good commercial reason. If someone getting on their bus happens to be infected with Swine Flu, are they supposed to do anything about it ? Are they even supposed to look for signs of infection ?

The ISPs are best responsible to their own customers, for whatever they have promised their own customers they will do.

Additional regulatory burdens or public service obligations are only going to push up the cost of Internet access for Joe Q Public, and maybe also choke off a new business in its infancy.
Anonymous
For Blacklists they almost always seem to consider all blocked email by their filters to be spam, without any further verification.
To measure spam still passing their filters is easy (as that is _very_ visible). Yet it s terribly difficult to even guess -let alone accurate measure- email being blocked unjustly together wit the spam.

Those wanting ZERO spam: turn of your smtpd. yes: turn it off, fully and permanently: you'll never receive another spam email. Also you'll not get any legit email either, but that's an unfortunate side effect ...

The problem with these blacklists is that most who choose to use them do so without properly selecting them, or reviewing who they support.

For ISPs: divide your custoemrs between businesses capable of keepign their own policies and enforce them themselves, and those who won't (end-users, small non-IT businesses).
Force the latter to only send email through a filter to prevent the worst, at least rate limit their outgoing email capability, and do not allow direct port 25 to the Internet.
Those who are capable: refuse to let them offload the email onto the servers for the consumers: you'll have less trouble from them if a consumer ends up blocking your outgoing relay servers.

For the rest: I guess the old http://isc.sans.org/diary.html?storyid=3194 is still very valid.
Swa

760 Posts
(1) The requirements you described that were imposed on you for the formation of rDNS are ridiculous (but it *is* reasonable for "no rDNS" or "generic" rDNS to be a *factor* in a blacklisting--in addition to the spam being sent from an IP)

(2) Requiring payment for removal is NOT "best practices" for a DNSBL (to put it very nicely)

(3) Preemptively blocking is a "necessary evil", due to the the sneaky tactics of many spammers. However, it shouldn't be employed so arbitrarily as you've described.

...on the other hand...

Having said that, rDNS should be considered as a means of conveying "identity" and "reputation". And while the rDNS requirements you described were not reasonable, imo... it *is* reasonable to factor into blacklisting decisions things like "no rDNS", or the rDNS being "generic"--as in having greater than X numbers of hypens/dots BEFORE the ending domain name. And, ideally, the domain should be the domain of the actual company that is using the IP (not their hoster or ISP's domain)

I'm NOT saying that I agree with preemptively blacklisting based ONLY on these things not being done well. But I think there is a growing consensus within the industry that having either "no rDNS" or a "generic" rDNS is asking for trouble.

And anyone who wants respect as a legit sender should have an rDNS ending in their own Domain name. (and having that rDNS "host name" which then resolves back to the IP is even better). n fact, ALL hosters/ISPs ought to ALWAYS set this for any IP they issue to a customer which will be used by that customer for mail sending. And this is MUCH more important than SPF.

PS - SORBS is used by most for scoring instead of blocking for the reasons you mentioned. And if Trend Micro is really doing what you say (taking your word on that), then they'll lose market share to other subscription-based DNSBLs like Invaluement :) ...so the free market has a way of taking care of these things.

Rob McEwen
founder of the invaluement.com DNSBL
Swa
1 Posts
A long time ago, in the very same galaxy, I also was responsible to help a small ISP deal with all the complaints.

I made several recommendations: use submission with username/password and encryption for customers, encourage paying customers to use the ISP's antispam/antivirus relays for both incoming and outgoing mail, transparently redirect direct connections from dynamically addressed users on port tcp/25 and tcp/587 to the ISP's systems ...

At the end of the day, almost nothing was done, except putting an ACL on their routers to drop TCP/25 from dynamically addressed customers.

Another mention I made was for the ISP to send a monthly or bi-monthly information letter to all customers explaining the danger of viruses, how to handle attachments, how to secure their machine and a list of freshly updated applications with links. This was disregarded with disdain "Our customers aren't idiots and we don't want to treat them as childs, for they could get vexed and see the same service somewhere else."

No need to tell that at that time, I had to ask for BL removals at least every month. Near my mission end, my first move in the morning was to check the BL status of their mailservers.

My top 5:

1) More than 90% of the Internet population has not a freaking clue on what they do: "After all, it's only a pdf, right, it's not like it can be dangerous." -- Need more focused guides to help people ... help themselves;

2) What's the need for a machine in a dynamic range to access the whole Internet on port TCP/25? Redirect that to the ISP's relays;

3) Propose subscription where the whole range will be behind preventive ACL (no netbios in/out, no smtp out, no tcp/80 in ...);

4) Create a mailing-list for customers with last month's patches and links. People often don't know what to look for. Or where;

5) Don't hesitate to contact customers to report abnormal behavior. A customer who leaves because you told him his machine scanned a webserver for CGIs or sent hundreds of infected emails is a pain out of your neck.

From the past months, I've been thinking quality insurance from big vendors is declining and some of their actions are kind of flaky. But ISPs, both small and large, also need to understand that if they don't curb their customers, nobody will do.

J.



Swa
2 Posts
Well, when I'm at home being a 'member of the public', when I sign up with an ISP then Internet Service Provision is what I expect; like when I buy a train ticket, I expect a train journey between the approproate stations.

If you're going to provide a service that doesn't conform to the IETF RFCs (e.g. by redirecting port 25 to a different IP address), by all means do so by default for new customers; but please allow an opt-out so that hobbyists, scientists, network engineers, businesses who are interested, and anyone else who for whatever reason wants to, can have full IPv4 as it was designed.

Most people don't want 'port 25' access to all machines on the Web. Some do, though.
Swa
9 Posts
Also dealing with some of the "big" email providers is between exceedingly difficult (yahoo) and impossible (MSN/hotmail/live/whatever they're called this week).

I run a domain that is fully compliant, has forward and reverse DNS, have deployed SPF, Microsoft's variant of it, Yahoo Domain Keys, and DKIM. But for the life of me, I still couldn't get email to people on Yahoo or on the Microsoft servers.

Support calls to both were extraordinarily unhelpful, as the responses were "stop spamming and implement our technology" (DK for Yahoo, SPF2 for Microsoft).
Swa
3 Posts
if it takes you 3 weeks to figure out that 24 million emails may indicate a spammer, i am not surprised your ISP ends up on multiple blacklists.
thats roughly a million emails a day, right? how a small ISP admin would not notice that amount is beyond me.
Anonymous
Unfortunately, places like Cisco often provide blacklist services based on reputation scores.

In other words, they have no technical evidence, yet many ignorant people rely exclusively on these services because they simply do not know any better. [ignorant in the literal sense, as we are all ignorant of something]

When one manages abuse complaints for 40,000+ servers for a hosting company (as in my case), this is quite frustrating. Especially when they will not communicate with the ISP (we are the ISP).

If anyone has found other ways to deal with these sites besides banding together and having them shut down (as has happened in the past), please do share ;-)
Anonymous
I share Deb's frustration. As a malware researcher working for an antivirus firm, I have firsthand knowledge of the havoc some malware can wreak on a business or "home user" customer of a large ISP. That said, spam blacklists have been wielded as a cudgel for too long. I am personally sick of it, and wish there was some recourse for this kind of irresponsible action.

I own my own domain (for personal use), and for *years* I've struggled to get email through to friends and family. Why? Because blacklist operators' fallback policy is to block first and...no, that's it. No matter that a spammer has decided to forge my domain in his/her return addresses, and that the IP address where the spam originated isn't even on the same continent, let alone the same IP range, as my Web server. I find small personal messages to friends and family on Yahoo, Hotmail, and other free services regularly just "fails to arrive." Ever.

The entire experience is disappointing to say the least, and companies like Trend and Cisco should be tremendously embarrassed to be behind this reprehensible behavior. Reputation-based filtering, indeed. They might consider the harm to their reputations by engaging in arrogant, unilateral, and unjustifiable behavior.
Anonymous

Sign Up for Free or Log In to start participating in the conversation!