Threat Level: green Handler on Duty: Kevin Liston

SANS ISC: Hunting for IOC's with ioc-parser - SANS Internet Storm Center SANS ISC InfoSec Forums


Sign Up for Free!   Forgot Password?
Log In or Sign Up for Free!
Hunting for IOC's with ioc-parser

Threat intelligence became a hot topic for a while. The food of threat intelligence is based on IOC's (Indicators of Compromise) which contains technical information like:

  • Files, path
  • Hashes 
  • IP addresses
  • Domains
  • Users

Mixed with other sources of information or tools, they help in detecting malicious behaviors of programs or networks. They are plenty of sources to collect IOC's. Some are publicly available while others are compiled and maintained by organizations for their customers or restricted users. DShield is of course a good source of IP addresses but Lenny (another ISC handler) is maintaining a nice list of resource on his website(1). Usually, free services offer lists of IOC's in common format that are reusable in your own environment. But sometimes, you will find interesting information published online. Many security researchers analyze pieces of malware and publish the results on their blog. Big organizations like to publish nice PDF reports containing juicy information. In both case, IOC's can be present but how to extract them automatically?

ioc-parser(2) is a nice Python script which might be very helpful in this case. It parses an input file and generates a list of IOC's in another format. It supports the following input formats: Text files, PDF files or HTML (URLs). Results can be generated in CSV, JSON, YARA or NetFlow. The idea is simple, it searches for patterns based on regular expressions. Everything is configurable and your own regexp can be added. 

Here is the list of IOC's extracted from an old PDF report about Duqu 2.0 written by Kasperky Lab:

$ ./iocp.py -p patterns.ini -i pdf -l pypdf2 The_Mystery_of_Duqu_2_0_a_sophisticated_cyberespionage_actor_returns.pdf
The_Mystery_of_Duqu_2_0_a_sophisticated_cyberespionage_actor_returns.pdf    2    Filename    msi.dll
The_Mystery_of_Duqu_2_0_a_sophisticated_cyberespionage_actor_returns.pdf    2    Filename    klif.dll
The_Mystery_of_Duqu_2_0_a_sophisticated_cyberespionage_actor_returns.pdf    2    Filename    12CTwoPENC.dll
The_Mystery_of_Duqu_2_0_a_sophisticated_cyberespionage_actor_returns.pdf    2    Filename    KMART.dll
The_Mystery_of_Duqu_2_0_a_sophisticated_cyberespionage_actor_returns.pdf    2    Filename    portserv.sys
The_Mystery_of_Duqu_2_0_a_sophisticated_cyberespionage_actor_returns.pdf    3    URL    https://en.wikipedia.org/wiki/Duqu
The_Mystery_of_Duqu_2_0_a_sophisticated_cyberespionage_actor_returns.pdf    3    URL    http://www.kaspersky.com/about/news/virus/2011/Duqu_The_Step_Brother_of_Stuxnet
The_Mystery_of_Duqu_2_0_a_sophisticated_cyberespionage_actor_returns.pdf    3    URL    http://70.auschwitz.org/index.php?lang=en
The_Mystery_of_Duqu_2_0_a_sophisticated_cyberespionage_actor_returns.pdf    3    Host    70.auschwitz.org
The_Mystery_of_Duqu_2_0_a_sophisticated_cyberespionage_actor_returns.pdf    3    CVE    CVE-2015-2360

But you can access URLs directly and extract IOC's present in the HTML code of the latest MalwareMustDie blog article:

$ ./iocp.py -p patterns.ini -i html -l requests -d http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    URL    http://www.blogger.com/go/cookiechoices
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    210.92.18.118
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.120.167.25
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.120.167.13
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.120.167.15
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.120.167.10
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.120.162.175
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.120.167.14
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.38.187.100
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.38.187.103
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.120.167.100
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.38.187.106
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.120.167.102
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.38.187.113
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.38.187.105
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.38.187.118
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    101.199.109.151
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.120.162.174
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.120.162.178
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    15.167.120.106
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.120.160.0
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.120.167.8
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    8.167.120.106
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.120.162.176
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    176.162.120.106
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    14.167.120.106
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.38.187.101
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.38.176.0
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.38.187.102
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.38.187.104
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.120.167.9
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    9.167.120.106
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    174.162.120.106
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.38.187.115
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.38.187.116
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    101.199.109.144
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    101.199.108.0
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.120.167.29
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    29.167.120.106
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    178.162.120.106
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.120.167.92
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    92.167.120.106
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.120.167.90
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    90.167.120.106
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    106.120.167.86
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    86.167.120.106
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    210.92.0.0
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    IP    222.186.34.220
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    360.cn
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    qi89.f3322.org
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    qurl.qh-lb.com
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    qup.qh-lb.com
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    sdupm.360.cn
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    sdup.360.cn
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    sdup.qh-lb.com
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    flux.sh
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    15.167.120.106.static.bjtelecom.net
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    chinatelecom.com.cn
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    167.120.106.static.bjtelecom.net
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    176.162.120.106.static.bjtelecom.net
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    14.167.120.106.static.bjtelecom.net
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    174.162.120.106.static.bjtelecom.net
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    29.167.120.106.static.bjtelecom.net
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    178.162.120.106.static.bjtelecom.net
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    92.167.120.106.static.bjtelecom.net
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    90.167.120.106.static.bjtelecom.net
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    86.167.120.106.static.bjtelecom.net
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    dshw.co.kr
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    f3322.org
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    astpbx.com
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    libworker.so
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Host    www.blogger.com
http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html    1    Email    ppyy@astpbx.com

And the same results generated in YARA format:

$ ./iocp.py -p patterns.ini -i html -l requests -d -o yara http://blog.malwaremustdie.org/2015/09/mmd-0042-2015-hunting-mr-black-ids-via.html
rule mmd_0042_2015_hunting_mr_black_ids_via
{
    strings:
        $URL1 = "http://www.blogger.com/go/cookiechoices"
        $IP1 = "210.92.18.118"
        $IP2 = "106.120.167.25"
        $IP3 = "106.120.167.13"
        $IP4 = "106.120.167.15"
        $IP5 = "106.120.167.10"
        $IP6 = "106.120.162.175"
        $IP7 = "106.120.167.14"
        $IP8 = "106.38.187.100"
        $IP9 = "106.38.187.103"
        $IP10 = "106.120.167.100"
        $IP11 = "106.38.187.106"
        $IP12 = "106.120.167.102"
        $IP13 = "106.38.187.113"
        $IP14 = "106.38.187.105"
        $IP15 = "106.38.187.118"
        $IP16 = "101.199.109.151"
        $IP17 = "106.120.162.174"
        $IP18 = "106.120.162.178"
        $IP19 = "15.167.120.106"
        $IP20 = "106.120.160.0"
        $IP21 = "106.120.167.8"
        $IP22 = "8.167.120.106"
        $IP23 = "106.120.162.176"
        $IP24 = "176.162.120.106"
        $IP25 = "14.167.120.106"
        $IP26 = "106.38.187.101"
        $IP27 = "106.38.176.0"
        $IP28 = "106.38.187.102"
        $IP29 = "106.38.187.104"
        $IP30 = "106.120.167.9"
        $IP31 = "9.167.120.106"
        $IP32 = "174.162.120.106"
        $IP33 = "106.38.187.115"
        $IP34 = "106.38.187.116"
        $IP35 = "101.199.109.144"
        $IP36 = "101.199.108.0"
        $IP37 = "106.120.167.29"
        $IP38 = "29.167.120.106"
        $IP39 = "178.162.120.106"
        $IP40 = "106.120.167.92"
        $IP41 = "92.167.120.106"
        $IP42 = "106.120.167.90"
        $IP43 = "90.167.120.106"
        $IP44 = "106.120.167.86"
        $IP45 = "86.167.120.106"
        $IP46 = "210.92.0.0"
        $IP47 = "222.186.34.220"
        $Host1 = "360.cn"
        $Host2 = "qi89.f3322.org"
        $Host3 = "qurl.qh-lb.com"
        $Host4 = "qup.qh-lb.com"
        $Host5 = "sdupm.360.cn"
        $Host6 = "sdup.360.cn"
        $Host7 = "sdup.qh-lb.com"
        $Host8 = "flux.sh"
        $Host9 = "15.167.120.106.static.bjtelecom.net"
        $Host10 = "chinatelecom.com.cn"
        $Host11 = "167.120.106.static.bjtelecom.net"
        $Host12 = "176.162.120.106.static.bjtelecom.net"
        $Host13 = "14.167.120.106.static.bjtelecom.net"
        $Host14 = "174.162.120.106.static.bjtelecom.net"
        $Host15 = "29.167.120.106.static.bjtelecom.net"
        $Host16 = "178.162.120.106.static.bjtelecom.net"
        $Host17 = "92.167.120.106.static.bjtelecom.net"
        $Host18 = "90.167.120.106.static.bjtelecom.net"
        $Host19 = "86.167.120.106.static.bjtelecom.net"
        $Host20 = "dshw.co.kr"
        $Host21 = "f3322.org"
        $Host22 = "astpbx.com"
        $Host23 = "libworker.so"
        $Host24 = "www.blogger.com"
        $Email1 = "ppyy@astpbx.com"
    condition:
        $URL1 or $IP1 or $IP2 or $IP3 or $IP4 or $IP5 or $IP6 or $IP7 or $IP8 or $IP9 or $IP10 or $IP11 or $IP12 or $IP13 or $IP14 or $IP15 or $IP16 or $IP17 or $IP18 or $IP19 or $IP20 or $IP21 or $IP22 or $IP23 or $IP24 or $IP25 or $IP26 or $IP27 or $IP28 or $IP29 or $IP30 or $IP31 or $IP32 or $IP33 or $IP34 or $IP35 or $IP36 or $IP37 or $IP38 or $IP39 or $IP40 or $IP41 or $IP42 or $IP43 or $IP44 or $IP45 or $IP46 or $IP47 or $Host1 or $Host2 or $Host3 or $Host4 or $Host5 or $Host6 or $Host7 or $Host8 or $Host9 or $Host10 or $Host11 or $Host12 or $Host13 or $Host14 or $Host15 or $Host16 or $Host17 or $Host18 or $Host19 or $Host20 or $Host21 or $Host22 or $Host23 or $Host24 or $Email1
}

This is a nice script to keep in your personal toolbox. Of course, be careful to not re-use the generated data "as is", there could be false positives or bad regular expression matches.

Happy IOC's hunting!

Xavier Mertens
ISC Handler - Freelance Security Consultant
rootshell.be
truesec.be

(1) https://zeltser.com/malicious-ip-blocklists/
(2) https://github.com/armbues/ioc_parser

Xme

396 Posts
ISC Handler
You may also want to look at the work already done here:

https://github.com/kbandla/APTnotes

and IOC exports available:

http://aptnotes.malwareconfig.com/ioc_export.csv
http://aptnotes.malwareconfig.com/ioc_export.json
http://aptnotes.malwareconfig.com/yara_export.yar

>26K IOCs ready to use :-)

Cheers,
Tom
TomU

7 Posts Posts
Thanks for sharing!
Xme

396 Posts Posts
ISC Handler
Nice! I use ioc_parser to generate ready-to-use netflow (nfdump) queries (use OUTPUT_FORMAT netflow).
cudeso

6 Posts Posts
Those using the REMnux distro (REMnux.org) will find this handy tool already installed. Just go to the /opt/remnux-ioc-parser directory.
Lenny

216 Posts Posts
ISC Handler
For IPs it looks like the feed is perfect.
For domain names & URLs, be ware that reference sites such as sans.org fireeye.com, etc... are also in the feed. So you might get lots of false positives, depending on what your users are browsing.
David Andre aka Elhoim

4 Posts Posts
Hello,

I need to get the IOC's out of FireEye (web) - any ideas? The IOC's in fireeye are called "OS changes" and it has its own format (none OpenIOC nor XML) - we can run a report and get a PDF out of it but that's it.

Please advice.
David Andre aka Elhoim
3 Posts Posts
That's a good idea! I've access to FireEye appliances, I'll investigate this way...
Xme

396 Posts Posts
ISC Handler
yes, please advise if you find anything.

The other thing I've found out is that we can also create reports in XML - the alert details report, this way we can extract the IOCs out that xml report. I figured it should be easier than a PDF report.

Chris M.
Xme
3 Posts Posts
I made some tests...
The PDF export is not doable because strings are not presents "as is" in the document.
I generated a XML report for the last 3 months. ioc-parser extract lot of stuffs but there are also a lot of false positives.. (The XML containing not only IOC's)
Xme

396 Posts Posts
ISC Handler
Hey, not finding the link to download the script.
Please share the link.

Thanks
Sudhanshu
Anonymous
Posts

Sign Up for Free or Log In to start participating in the conversation!