Last Updated: 2014-04-04 18:09:22 UTC
by Rob VandenBrink (Version: 3)
Thanks to Susan, one of or readers, who dropped us a line today to tell me that we (and by "we" I mean "I") missed that Windows 8.1 Update was announced on April 2.
This is an important update for all of the Windows 8.x folks out there for a couple of reasons:
- The base 8.1 update includes 2 out-of-band security updates (KB2922229 and KB2936068, both available separately)
- More importantly, it's the service baseline for 8.x, so you should have this installed before applying the in-band updates coming in May (next month).
Before installation, the 8.1 update requires a prerequisite of KB2919442 http://support.microsoft.com/kb/2919442
More on the 8.1 Update here (amongst lots of other Microsoft posts):
8.1 Update will be available as KB2919355 April 8 (this coming Patch Tuesday)
Last Updated: 2014-04-04 13:55:34 UTC
by Rob VandenBrink (Version: 1)
I had a client call me recently with a full on service outage - his servers weren't reachable, his VOIP phones were giving him more static than voice, and his Exchange server wasn't sending or receiving mail - pretty much everything was offline.
I VPN'd in (I was not onsite) and started with the firewall, because things were bad enough that's all I could initially get to from a VPN session. No surprise, I saw thousands of events per second flying by that looked like:
So right away this looks like malware, broadcasting on UDP ports 137 and 138 (netbios name services and datagrams). You''ll usually have some level of these in almost any network, but the volumes in this case where high enough to DOS just about everything, I was lucky to keep my SSH sessions (see below) going long enough to get things under control. And yes, that was me that was behind Monday's post on this if this sounds familiar
To get the network to some semblance of usability, I ssh'd to each switch in turn and put broadcast limits on each and every switch port:
storm-control broadcast level 20 (for 20 percent)
storm-control broadcast level pps 100 (for packets per second)
On HP Procurve:
broadcast-limit 5 (where x is a percentage of the total possible traffic)
On HP Comware:
int gig x/0/y
broadcast-suppression pps 200
storm-constrain broadcast pps 200 200
(you can do these in percent as well if you want)
Where I can, I try to do this in packets per second, so that the discussion with the client can be "of course we shut that port down - there's no production traffic in your environment that should generate more than 100 broadcasts per second."
With that done, I now could get to the syslog server. What we needed was a quick-and-dirty list of the infected hosts, in order of how much grief they were causing.
First, let's filter out the records of interest - everything that has a broadcast address and a matching netbios port in it - it's a Windows host, so we'll use windows commands (plus some GNU commands):
type syslogcatchall.txt | find "172.xx.yy.255/13"
But we don't really want the whole syslog record, plus this short filter still leaves us with thousands of events to go through
Let's narrow this down:
first, let's use "cut" to pull out the just source IP out of these events of interest.
cut -d " " -f 7
Unfortunately, that field also includes the source port, so let's remove that by using "/" as the field delimeter, and take only the source ip address (field one)
cut -d "/" -f 1
Use sort and uniq -c (the -c gives you a count for each source ip)
then use sort /r to do a reverse sort based on record count
Putting it all together, we get:
type syslogcatchall.txt | find "172.xx.yy.255/13" | cut -d " " -f 7 | cut -d "/" -f 1 | sort | uniq -c | sort /r > infected.txt
This gave us a 15 line file, sorted so that the worst offenders were at the top of the list, with a record count for each. My client took these 15 stations offline and started the hands-on assess and "nuke from orbit" routine on them, since their AV package didn't catch the offending malware.
What else did we learn during this incident?
- Workstations should never be on server VLANs.
- Each and every switch port needs basic security configured on it (broadcast limits today)
and, in related but not directly related lessons ...
- Their guest wireless network was being used as a base for torrent downloads
- Their guest wireless network also had an infected workstation (we popped a shun on the firewall for that one).
- Their syslog server wasn't being patched by WSUS - that poor server hadn't seen a patch since December's patch Tuesday
What did we miss?
By the time I got onsite, the infected machines had all been re-imaged, so we didn't get a chance to assess the actual malware. We don't know what it was doing besides this broadcast activity, and don't have any good way if working out for sure how it got into the environment. Though since this distilled down to one infected laptop, my guess would be this is malware that got picked up at home, but that's just a guess at the moment.
Just a side note, but an important one - cut, uniq, sed and grep are all on your syslog server if it's a *nux host, but if you run syslog on Windows, these commands are still pretty much a must-have. As you can see, with these commands we were able to distill a couple of million records down to 15 usable, actionable lines of text within a few minutes - a REALLY valuable toolset and skill to have during an incident. Microsoft provides these tools in their "SUA" - Subsystem for Unix Based Applications, which has been available for many years and for almost every version of Windows.
Or if you need to drop just a few of these commands on to a server during an incident, especially if you don't own that server, you can get what you need from gnutools (gnuwin32.sourceforge.net) - I keep this toolset on my laptop for just this sort of situation.
Once you get the hang of using these tools, you'll find that your fingers will type "grep" instead of find or findstr in no time!
Last Updated: 2014-04-04 13:09:40 UTC
by Rob VandenBrink (Version: 1)
Microsoft has posted their regular pre-announcement for Patch Tuesday here: http://technet.microsoft.com/en-us/security/bulletin/ms14-apr
We can expect:
- The final, yes final patches for XP
- The final patches for Office 2003 also - this has gotten a lot less press than XP but is just as critical ( http://office.microsoft.com/en-ca/help/support-is-ending-for-office-2003-HA103306332.aspx )
- The usual patches for other Windows and IE versions
- A couple of updates for WSUS and Windows Update. Changes to Windows Update often result in Tuesday's updates coming two parts - we'll see on Tuesday I guess
So after Tuesday, XP and Office 2003 join the ranks of the "internet of hostile things" - platforms that are no longer being patched by the vendor as new issues arise, so quickly become compromised. This includes things like your TV, your DVR or home internet router, your fridge, treadmill, IV pump, heart monitor or pacemaker - oh, and likely your phone as well - - read our stories over the last couple of months (or years really) for more on these. Unfortunately, this XP event happens all in one fell swoop - millions of hostile hosts being added to the opposing army all at once.
Fortunately, we can do something about this. Updating to Windows 7 or 8 is cheap, if your hardware is up to the task. If you've got older hardware, I'm seeing used Windows 7/8 capable desktop hardware for $100-$200 these days, and laptops seem to be in the same range. If going to a new Windows platform isn't in your future, you're probably already looking at one of the more popular Linux distributions - distros like Unbuntu, or Xubunto or Mint that try to mimic the UI that many home users are familiar with. There is a similar range of options for Office (upgrades and alternatives).
Use our comment form to let us know what you are doing or have done for your user community (or family members) that might still be on XP or Office 2003 after Tuesday.
Last Updated: 2014-04-04 12:46:04 UTC
by Stephen Hall (Version: 1)
A new version of PHP has been released. The announcement comments:
"The PHP development team announces the immediate availability of PHP 5.4.27. 6 bugs were fixed in this release, including CVE-2013-7345 in fileinfo module."
Details of CVE-2013-7345 are available at Mitre.
Steve Hall ISC Handler www.tarkie.net