ISC Diary

Intel Network Card (82574L) Packet of Death

Published: 2013-02-06,
Last Updated: 2013-02-08 21:07:10 UTC
by Johannes Ullrich (Version: 3)

[Update] Intel released a statement about this issue. and Kristian updated his blog

An interesting blog post by Kristian Kielhofer describes how a specific SPI packet can "kill" an Intel Gigabit ethernet card [1]. If a card is exposed to this traffic, the system has to be physically power cycled. A reboot will not recover the system.

The network card crashed whenever the value 0x32 or 0x33 was found at offset 0x47f. Kristian first noticed this happening for specific SIP packets, but in the end, it turned out that any packet with 0x32 at 0x47f caused the crash. Intel traced the problem to an EEPROM used in this specific card (82574L). There are some links in the comment to the blog suggesting that others have run into this problem before. For example, the commend: "ping -p 32 -s 1110 x.x.x.x" can crash an affected card remotely.

[Update] A few asked why this doesn't happen just randomly every 128th packet: Once the card receives the value "0x34" in this position, it appears to be no longer vulnerable. There are also a number of earlier bug reports about this card that sound very similar, and appear to be related to ASPM, a PCI power safe feature. Kristian claims he eliminated this issue. if you try to reproduce this issue, power up the system and then issue the "ping" command shown above quickly after reboot in order to avoid the "inoculation" wiht 0x34. We would like to hear any reports of being able to reproduce (or not) this issue.

There are also some reports about similar issues in certain 3G USB modems.

[1] http://blog.krisk.org/2013/02/packets-of-death.html

------
Johannes B. Ullrich, Ph.D.
SANS Technology Institute
Twitter

Keywords: ethernet card intel

13 comment(s)

Top of page

Comments

Actually, it's just receiving a few particular values, in a particular position in the frame, - so a web page, or even just a modified image could trigger this.

Nasty, but probably worse case is as a DoS attack tool, as a crafted request could trigger the bug.
His case was well, really obscure (a VoIP packet happened to trigger it), but only makes the story more interesting.

posted by Dom De Vitto, Wed Feb 06 2013, 22:14

Some of the newer posts in Kristians link above refer to similar problems with ZTE MMC 3G USB Mobile Broadband dongles and their drivers. I have had similar things occur (BSOD driver failures) lately that require a full reboot and they seemed to have stopped after a Java update. There still seems to be some partial issues where an individual page on a website blows up IE but just closing down the offending page and not IE fixes the problem. Weird to say the least.

posted by Laurie, Thu Feb 07 2013, 03:57

Actually the ZTE MMC is my USB not the net card driver failures referred to in the link FYI.

posted by Laurie, Thu Feb 07 2013, 03:59

If it is only on a plug-in NIC, one can always change the NIC, but is there any chance the same bug exists on motherboard mounted NICs also? That would require replacing the entire motherboard -- unless it is on-site reflashable and Intel has a patch.

posted by Moriah, Thu Feb 07 2013, 12:11

It surely can't be that easy? Or else a stream of encrypted (pseudorandom) data would trigger the bug after approx. 128 packets.

One of my server rentals has this model of NIC onboard. I sure hope they're not vulnerable, or else my provider has a whole datacentre full of these...

posted by Steven Chamberlain, Thu Feb 07 2013, 13:13

- http://www.securitytracker.com/id/1028089
"... A power cycle is required to return the system to normal operation.
The original advisory is available at:
http://blog.krisk.org/2013/02/packets-of-death.html
Impact: A remote user can cause the target controller to crash.
Solution: The vendor has issued a fix, available via customer support...
[Rots o' ruck finding that number.]
.

posted by PC.Tech, Thu Feb 07 2013, 14:59

FYI:

I used the two pcaps ( pod-http-post.pcap & pod-icmp-ping.pcap ) from http://www.kriskinc.com/intel-pod using the tcpdump-edit instructions provided. I also added "-l 30" to retry.

I have an Intel 82574L Gigabit CT adapter on an ASUS X79 board running CentOS 6.3, fully patched. The box says "Version E39199-008" "Date: 09/08/2012". I have been unable to duplicate the crash.

I "ifconfig up"ed the adapter (without an IP). "tcpdump -p -i eth3" shows the machine receiving the frames, but it doesn't stop working.

posted by Mark Jx, Thu Feb 07 2013, 16:21

Intel technical support number: 916-377-7000

posted by JeffSoh, Thu Feb 07 2013, 19:50

We uploaded the offending packet to our online packet viewer, CloudShark, with some notes on what to look for and links to Kristian's articles on the topic. Check it out here:

http://appliance.cloudshark.org/news/cloudshark-in-the-wild/intel-packet-of-death-capture/

posted by CloudShark, Thu Feb 07 2013, 19:59

I wonder if there's any chance the scope of this problem is bigger than 82574L

The symptoms sound very similar to an issue I have been having with an Intel 82567LM card over the passed 2 years. (In a Dell Optiplex 960)

It just randomly just quits talking on the network....and to fix it you can either reboot, or unplug the NIC cable for 10 seconds and then plug it back in. The problem seems to semi-randomly happen after the sleeping computer wakes up

While it is in this strange state, sometimes I can PING and sometimes I can't. If I can't ping, then if I IPCONFIG...the static IP Address no longer shows it is registered. IT shows 0.0.0.0

Very Strange!

Motherboard has been changed out 3-4 times (by Dell....who sends me the same motherboard and embedded NIC I'm assuming)
Network cable replaced
Cable run to the closet tested
Different network port used
Drivers/BIOS/chipset drivers updated

I also seem to be having sporadic problems with 82579LM cards that are in systems that were brought online last year. After the system wakes up for its nightly backup, the backup server can not contact the system. Doesn't happen all the time tho.... pretty random and sporadic....

posted by K-Dee, Fri Feb 08 2013, 00:49

Thanks for updating the article; but if there are 2 values that could trigger this and one that would prevent it, I think it would be quickly triggered on 2 out of 3 hosts. I imagine there's some other requirement for the ethernet frame, and that might limit exposure to being link-local.

Anyway I'm concerned anything like this was even possible. Why is software hidden in firmware blobs inspecting payload data? Remotely-triggerable bugs of this sort could allow DoS of whole data centres, customer premises equipment (modem/routers), or ISPs' core infrastructure.

posted by Steven Chamberlain, Fri Feb 08 2013, 12:48

Re: 'innoculation', Kris K's blog later mentions that most values other than 0x30,31,32 will do that. The chance of hitting the error becomes more like 1 in 128 and probably depends what type of traffic is sent/received during startup, or incoming broadcasts on your network.

posted by Steven Chamberlain, Fri Feb 08 2013, 12:59

I guess I'm a bit more cynical than others, looking at a weak link that was produced back in 2012 and Intel's pass the buck response. Personally I've seen bad code in their tethered and non-tethered products through HP, Dell.

Intel "worked with the author as well as the original motherboard manufacturer to investigate and determine root cause"? Intel caused the issue, not the MB manufactures with a flawed EEPROM image that could have been fixed.

We had 5 servers that ran dual NICS with this chipset.. Bravo!

A lot of us have been around this industry for years, Sadly what is becoming a trend as new products come out, there is some security flaw to them. If we did work like this, we would not be working.

posted by scsmith77, Sun Feb 10 2013, 14:58

New Comments closed for all Diaries older than two(2) weeks
Please send your comments to our Contact Form