Infosanity's Blog

Working my way through the compilation instructions from Dionaea whilst building up my latest sensor I was reminded of some optional functionality that I’d always intended to implement, but never found the time. First on my list was p0f (that’s a zero).

From p0f’s homepage:

P0f v2 is a versatile passive OS fingerprinting tool. P0f can identify the operating system on:

- machines that connect to your box (SYN mode),
- machines you connect to (SYN+ACK mode),
- machine you cannot connect to (RST+ mode),
- machines whose communications you can observe.

P0f can also do many other tricks, and can detect or measure the following:

- firewall presence, NAT use (useful for policy enforcement),
- existence of a load balancer setup,
- the distance to the remote system and its uptime,
- other guy’s network hookup (DSL, OC3, avian carriers) and his ISP.

Setting p0f up on the sensor should have been straightforward;

• Install with: apt-get install p0f
• Run p0f as suggested in dionaea.conf: sudo p0f -i any -u root -Q /tmp/p0f.sock -q -l
• And edit dionaea.conf’s ihandler section to enable p0f

This mostly worked, watching the p0f output it was correctly (I’m assuming) providing stats about connecting systems. The problem was that p0f info wasn’t getting saved into Dionaea’s logsqlite database, dionaea-error.log was reporting the below error with each connection:

[04122010 13:48:44] connection connection.c:827-warning: Could not connect un:///tmp/p0f.sock:0 (Permission denied)

Which seemed odd, /tmp/p0f.sock was showing as globally readable. Re-reading the Dionaea compilation instructions I noticed a comment about p0f struggling with IPv6 so has problems it Dionaea is listening on ::, which mine was. Problem solved, I edited dionaea.conf so that Listen mode was set to “manual”, and provided the interface/IP details of my network connection. Only this didn’t solve my problem…

Head thoroughly hurting I swallowed my ego and asked for assistance on the mailing list, and promptly (thanks again Ryan) got a reply that provided a functional workaround.

So, why go to the effort? Main purpose behind running honeypot systems (for me) is to get a better idea understanding of what threats are actively targeting systems in the wild. At first glance the information provided by p0f can quickly help evaluate the attacking system; what OS? what connection type? is it local?

With the limited (few hours) of data I’ve already collected heres a sample of the info you can gather:

last 5 connections:

SQL Query:

select *
from p0fs
order by
connection desc
limit 5;

Breakdown by OS

SQL Query:

select count(p0f_genre) as count, p0f_genre as OS
from p0fs
group by p0f_genre
order by count(p0f_genre) desc;

Umm, so most systems spreading malware are (likely) infected Windows systems. No great surprise there…

Connectivity Types

SQL Query:

select count(p0f_link), p0f_link
from p0fs
group by p0f_link
order by count(p0f_link) desc;

Summary

Unfortunately the the information provided by p0f isn’t an exact science, and as devices and systems are constantly changing it’s only going to be as accurate as it’s latest signatures/fingerprints. But setup is fairly quick, and the information and insight provided fairly interesting. So why not give it a go?

–Andrew Waite

–Andrew Waite

(p.s. sorry, couldn’t resist…)

Markus keeps adding great features and functionality to Dionaea, when I read the post introducing a new web interface carniwwwhore I couldn’t help thinking I’d got lucky timing, start of a week’s vacation and no real plan for what to do with it. I’ve struggled previously with some of my Dionaea setups, largely because my system was running Debian, whilst Dionaea was built under Ubuntu; doesn’t cause too many problems, just a bit of google-fu, headscratching and stupidity that could have been avoided.

From this background I looked through the carniwwwhore pre-reqs with dread, plenty of version requirements that weren’t upto date with my Debian setup; so it’s time to bite the bullet and build a fresh system with Ubuntu. Unlike some of my previous setups, installation/compilation worked flawless, working on the same distro as the lead dev definitely makes life easier. If you’re looking for a fresh Dionaea installation, go with Ubuntu, you won’t regret it.

–Andrew Waite

(oh, and carniwwwhore? Vacation got the better of me so it’s added to the to-do list; watch this space…)

I’m sure this is basic for most of you, but I seem to keep making the same daft mistakes whilst updating Dionaea, so I’m hoping documenting the issues and corrections will work as a memory aid in the future.

Firstly, I can never remember Git’s equivalent to ‘svn update’, which is:

git pull

Next up is recompiling Dionaea from your updated source directory, this is no different to an initial install as per Markus’ excellent instructions.

I keep forgetting to update my configuration file to include any directives needed for the new shiny new functionality you’re upgrading to get access to. I find diff useful for identifying any new additions, for example

/opt/dionaea/etc/dionaea# diff dionaea.conf dionaea.conf.dist

Assuming there’s no major changes you should only see differences specific to your installation, for example your email address to receive analysis reports, or your VirusTotal api key. If there are any other differences you’ll need to add the new content.

From the experience I’ve had in the past week, if you encounter any unexpected problems after updating Dionaea, make sure your pre-requisites are also upto date. After updating Dionaea last week to gain access to the new integration with VirusTotal’s api my Dionaea sensor started to die randomly. Markus was a great help with troubleshooting (thanks again) and my problems were eventually corrected after it was noted that my libemu installation was outdated; after a quick ‘git pull’ and ‘make’ (again following Markus’ instructions).

As I said, this is probably basic for most of you out there, but as I keep making similar mistakes I plan to refer back to this list of daft issues before bugging anyone for support in future. You never know, it might allow someone else to retain an air of competence before proving otherwise

– Andrew

Development of new features for Dionaea has been fairly impressive of late, and I’ve been lax in keeping up to date. When Markus asked if I’d tested the graph utility that he created and wrote about here, it served as a kick to stop putting off some of the jobs I’ve got on the growing to-do list.

I won’t go into too much detail about running the script as Markus has already done a better job than I could. However I will point out that if you run your Dionaea installation on Debian stable, then your out of luck; the standard packages for sqlite are too old to take the script. Best advice is to copy your logsql.sqlite database to a Ubuntu machine and work from there (oh, and in case you didn’t guess from the script name, make sure you’ve actually installed gnuplot…).

A powerful machine is recommended, the only Ubuntu system I had to hand whilst testing was my AA1 netbook, which took 85 minutes to crunch through the script and my database.

I have immediately found the graphs produced useful as they’ve highlighted a couple of obvious spikes (see below) in activity that I would have (and did) miss if solely relying on log files and databases. This really shows the power and importance of visualising security and log information.

dionaea-overview - from gnuplotsql.py

If you’re interested the output for the InfoSanity’s installation is now online here. I’m looking to expand the statistics from the InfoSanity honeypot environment that are publicly available, this makes a nice start. As always, big thanks to Markus and carnivore.it team for the effort.

– Andrew Waite

Mercury Live DVD was initially (I believe) announced in a post to the Nepenthes Mailing list. It is a remastered Ubuntu distribution with pre-installed honeypot applications and malware analysis tools created by John Moore. From the ReadMe:

This live DVD is a remastered version of Ubuntu 10.0 Beta LTS x86_32. It was designed due to my being disappointed with another reverse engineering malware live CD that was released recently. I have decided to call my creation MERCURY, which is an acronym for Malware Enumeration, Capture, and Reverse Engineering.

The Mercury live DVD contains tools used for digital forensics, data recovery, network monitoring, and spoofing. It should primarily be used as a honeypot or network monitoring platform as well as a laboratory and teaching aid. There are three honeypots installed – honeyd, nepenthes, and dionaea. Four, if you include netcat.

The majority of the additional applications reside in /opt:

• Dionaea (0.1.0) – Dionaea is a malware collection honeypot focusing primarily on SMB emulation, covered on InfoSanity numerous times before.
• FFP – Fuzzy Fingerprinting is a util to aid SSH MitM attacks.
• jsunpack-n – Is a Javascript unpacker, perfect for analysis captured or potentially malicious URLs in more depth.
• Kippo (svn rev.169) – Kippo is an low-medium interaction SSH honeypot, Also covered
• mitm-ssh – Unsurprisingly, a utility for aiding man in the middle attacks against SSH connections.
• Origami & pdftools – Two frameworks for analysing malicious PDF files.
• Volatility – an excellent memory analysis toolkit
• Zerowine-vm – A malware behavior analysis platform. I’ve covered ZeroWine here before, and whilst I find it useful for initial analysis I found it a pain to setup and get running. The fact this works out of the box on Mercury is enough reason alone to keep the .iso handy.

Other tools are installed on the system as started, access from standard locations (/etc, /usr/bin, etc.). I won’t try to list them all, but some highlights include:

• Nepenthes – Dionaea’s predecessor
• Honeyd – Honeypot system, perfect for emulating multiple different systems from one platform. Covered in more depth here.
• John – John the Ripper, password cracker
• ircd-hybrid – irc server daemon, useful for analysis irc-based malware’s interaction with command and control systems.
• Snort – de-facto intrusion detection system.
• Wireshark – Packet capture and network analysis tools.

I could go on, but I’m sure you get the idea.

Setting up a honeypot, and analysing the results, has never been easier. And I’m sure the toolkit’s functionality will also be useful in other scenarios; incident response, general network administration or as a safe learning platform. So what are you waiting for?

–Andrew Waite

N.B. there have been several mirror’s and downloads established, the most reliable download source I’ve used is Markus’ mirror at carnivore.it

I’ve been a bit lax in writing this post; around a month ago Miguel Jacq got in contact to let me know about a couple of errors he encountered when running InfoSanity’s mimic-nepstats.py with a small data set. Basically if your log file did not include any submissions, or was for a period shorter than 24hours the script would crash out, not the biggest problem as most will be working with larger data sets but annoying non the less.

Not only did Miguel let me know about the issues, he was also gracious enough to provide a fix, the updated script can be found here. An example of the script in action is below:

cat /opt/dionaea/var/log/dionaea.log| python mimic-nepstats_v1-1.py

Statistics engine written by Andrew Waite – www.infosanity.co.uk

Number of submissions: 84
Number of unique samples: 39
Number of unique source IPs: 65

First sample seen: 2010-06-08 08:25:39.569003
Last sample seen: 2010-06-21 15:24:37.105594
System Uptime: 13 days, 6:58:57.536591
Average daily submissions: 6

Most recent submissions:
2010-06-21 15:24:37.105594, 113.37.56.28, emulate://, 56b8047f0f50238b62fa386ef109174e
2010-06-21 15:18:08.347568, 195.205.5.71, tftp://195.205.5.71/ssms.exe, fd28c5e1c38caa35bf5e1987e6167f4c
2010-06-21 15:17:08.391267, 195.117.74.62, tftp://195.117.74.62/ssms.exe, bb39f29fad85db12d9cf7195da0e1bfe
2010-06-21 06:29:03.565988, 195.160.222.101, tftp://195.160.222.101/ssms.exe, fd28c5e1c38caa35bf5e1987e6167f4c
2010-06-20 23:34:15.967299, 195.242.145.40, http://208.53.183.164/trying.exe, 094e2eae3644691711771699f4947536

– Andrew Waite

As part of a new and improved environment I’ve just finished building up a new Dionaea system. Despite the ease at which I found the install of my original system I received a lot a feedback that others had a fair amount of difficulty during system build. So this time around I decided to pay closer attention to by progress to try and assist others going through the same process.

Unfortunately I’m not sure I’m going to be able to offer as many pearls of wisdom as I originally hoped as my install went relatively smoothly. Only real problem I hit was that after following Markus’ (good documentation) my build didn’t correctly link to libemu. Bottom line, keep an eye on the output of ./configure when building Dionaea. In my case the parameters passed to the configure script didn’t match my system so needed to be modified accordingly.

On the off chance that it’s of use to others (or I forget my past failures and need a memory aid) my modified ./configure command is below:

./configure \
 --with-lcfg-include=/opt/dionaea/include/ \
 --with-lcfg-lib=/opt/dionaea/lib/ \
 --with-python=/opt/dionaea/bin/python3.1 \
 --with-cython-dir=/usr/bin \
 --with-udns-include=/opt/dionaea/include/ \
 --with-udns-lib=/opt/dionaea/lib/ \
 --with-emu-include=/opt/dionaea/include \
 --with-emu-lib=/opt/dionaea/lib/ \
 --with-gc-include=/usr/include/gc \
 --with-ev-include=/opt/dionaea/include \
 --with-ev-lib=/opt/dionaea/lib \
 --with-nl-include=/opt/dionaea/include \
 --with-nl-lib=/opt/dionaea/lib/ \
 --with-curl-config=/opt/dionaea/bin/ \
 --with-pcap-include=/opt/dionaea/include \
 --with-pcap-lib=/opt/dionaea/lib/ \
 --with-glib=/opt/dionaea

– Andrew Waite

<update 20100606> New Dionaea build encountered a problem with libemu, ./configure above has been edited to reflected additional changes I required to compile with libemu support. </update>

Earlier this week Markus released two anonymised data sets from live Dionaea installations. The full write-up and data sets can be found on the newly migrated carnivore.it news feed here. Perhaps unsurprisingly I couldn’t help but run the data through my statistics scripts to get a quick idea of  what was seen by the sensors.

This caused some immediate problems, before the data was released Markus had contacted me to point out/complain that the performance from my script is ideal. Performance wasn’t an issue I had encountered, but the database from the sensor I run is ~1MB, the smaller of the released data sets is ~300MB, with the larger being 4.1GB. I immediately tried to rectify the problem and am proud to report,…

I failed miserably. I had tried to move some of the counting and loops from the python code and migrate to more complex SQL queries, working on the theory that working with large datasets should be more efficient within databases as they are designed for working with sets of data. Theory was proved false, actually increasing run-time by about 20%, so I won’t be releasing the changes. Good job I’ve never claimed to be a developer. All this being said, the script still crunches through the raw data in 30seconds and 3minutes respectively.

Without further ado, the Berlin data-set:

Statistics engine written by Andrew Waite – www.infosanity.co.uk

Number of submissions: 2726
Number of unique samples: 133
Number of unique source IPs: 639

First sample seen: 2009-11-05 12:02:48.104760
Last sample seen: 2009-12-07 11:13:55.930130
SystemrRunning: 31 days, 23:11:07.825370
Average daily submissions: 87.935483871

Most recent submissions:
2009-12-07 11:13:55.930130, 10.48.60.253, http://zonetech.info/61.exe, ae8705a7b4bf8c13e5d8214d374e6c34
2009-12-07 11:12:59.389940, 10.13.103.23, ftp://1:1@10.101.229.251:61751/ssms.exe, 14a09a48ad23fe0ea5a180bee8cb750a
2009-12-07 11:10:27.296370, 10.13.103.23, tftp://10.13.103.23/ssms.exe, df51e3310ef609e908a6b487a28ac068
2009-12-07 10:55:24.607140, 10.183.36.128, tftp://10.183.36.128/ssms.exe, df51e3310ef609e908a6b487a28ac068
2009-12-07 10:43:48.872170, 10.183.36.128, ftp://1:1@10.20.216.112:53971/ssms.exe, 14a09a48ad23fe0ea5a180bee8cb750a

And Paris:

Statistics engine written by Andrew Waite – www.infosanity.co.uk

Number of submissions: 749518
Number of unique samples: 2064
Number of unique source IPs: 30808

First sample seen: 2009-11-30 03:10:24.591650
Last sample seen: 2009-12-07 08:46:23.657530
SystemrRunning: 7 days, 5:35:59.065880
Average daily submissions: 107074.0

Most recent submissions:
2009-12-07 08:46:23.657530, 10.46.210.146, http://10.9.0.30:3682/udqk, d45895e3980c96b077cb4ed8dc163db8
2009-12-07 08:46:20.985190, 10.98.174.44, http://10.200.78.235:2708/lzhffhai, 94e689d7d6bc7c769d09a59066727497
2009-12-07 08:46:21.000540, 10.204.219.219, http://10.38.56.49:6968/tyhxqm, 908f7f11efb709acac525c03839dc9e5
2009-12-07 08:46:18.398500, 10.174.62.175, http://10.108.210.203:3058/pghux, ed12bcac6439a640056b4795d22608da
2009-12-07 08:46:15.753080, 10.39.96.46, http://10.132.244.66:3255/dhti, 94e689d7d6bc7c769d09a59066727497

Still need to dig further into the data, they’ll be another post in the making if I uncover anything interesting…

– Andrew Waite

Following on from my work with gathering statistics from the Honeypot systems that I run I have released a limited alpha of a new script/tool that I am working on. The tool provides access to common result sets from the sqlite database, without the requirement for remembering the database architecture  and entering lengthy SQL statements by hand.

Disclaimer first: the tool doesn’t do anything outrageously new, and most of the SQL queries have been borrowed from Markus’ post on SQL logging with Dionaea when the feature was first introduced. However I have found the script makes my analysis of the honeypot logs simpler and quicker, and I’ve a positive reaction from a limited few that have had a copy of the script before this post. Hopefully it will be of use others.

Usage is relatively simple, shown below:

Dionaea database query collection
Author: Andrew Waite – www.InfoSanity.co.uk

Inspiration from carnivore.it article:

http://carnivore.it/2009/11/06/dionaea_sql_logging

Usage:
/path/to/python dionaea-sqlquery.py –query #
Where # is:
1:      Port Attack Frequency
2:      Attacks over a day
3:      Popular Malware Downloads
4:      Busy Attackers
5:      Popular Download Locations
6:      Connections in last 24 hours

The script can be found here. There is still a good level of work to be undertaken to tidy up the output, potentially allowing for output in different formats, and I also want to add additional and more complex queries as time progresses. If you have any success,  failure, comments or suggests please let me know.

– Andrew Waite