Infosanity's Blog

As proven by the logs generated by Kippo honeypot sensors have shown, attacks against SSH services are regularly seen in the wild. Even if you follow best practices for securing the service, the malicious scans will utilise resources available to your environment; CPU, bandwidth etc. In sufficient volume legitimate operation may be impacted as the server rejects failed login attempts.

This is where utilities like Breakinguard come into their own. Basically Breakinguard monitors log files for signs of malicious activity, and once a single source has triggered enough alerts blocks all connections from the source location. Other utilities (most notably fail2ban) perform the same activities, but I’m partial to Breakinguard due to it’s small size and simple configuration (and from knowing the author ).

Installation is straightforward, and for the most part automated. Once downloaded and extracted installation is handled by the configure script. On Debian based systems this will install the pre-requisite Perl modules and transfer the utilities components to the standard locations:

• breakinguard script – /usr/local/sbin/breakinguard
• config – /etc/breakinguard.conf
• init script – /etc/init.d/breakinguard

Once installed you need to edit your configuration. The breakinguard.conf file is fairly self explanatory, I normally edit:

• $alert_email -> set to the email address that you want to receive notifications of blocked attacks. On a publicly accessible system these alerts can be high volume, you may want to use a specific email account or at least setup some auto-move rules in your email client to avoid your inbox being spammed.
• $number_of_attempts -> This specifies the number of malicious log entries need to be generated by a specific IP address before the source is blocked. Due to the timing of the Breakinguard route this isn’t always an exact science, the default of 5 does a good job of avoid false positives whilst still blocking an attack in it’s infancy.
• $log_to_watch -> selects the logfile to monitor for signs of malicious activity. /var/log/auth.log is the obvious choice on a Debian based system.
• @safe_ips -> This array allows you to specify a number of trusted networks that will not be blocked, regardless of the number of times the they trigger alerts in the logs. This is useful for insuring that you don’t get locked out of your own systems in the event your keyboard ‘breaks’. On higher end systems with hardware remote management systems (iLo, DRAC, etc.) or virtual systems that provide remote access to the ‘physical’ console I leave this list to local subnet only and use the alternative options to access the server if I do lock myself out.
• $DEBUG -> set to 1 by default, this runs the utility in test mode without actually blocking malicious sources, perfect for testing configuration before going live. Once you’re good to set set $DEBUG to 0 and wait for the attacks to start. Example testing in debug mode is below:

Breakinguard - Debug Mode

Blocking and unblocking of malicious sources is handled via iptables. Once the $number_of_attempts limit is hit Breakinguard will run the $block_command (configurable in /etc/breakinguard.conf) which by default is ‘/sbin/iptables -I INPUT -s %s -j DROP’, with %s being replaced with the attacking IP. After a configurable timeout ($block_length), the $unblock_command removes the restriction.

You can see the IP addresses currently blocked as they are listed in /var/run/breakinguard/, alternatively listing the current iptables configuration will show sources currently being blocked, for example:

Breakinguard iptables

Download Breakinguard here

–Andrew Waite

I’m sure this is basic for most of you, but I seem to keep making the same daft mistakes whilst updating Dionaea, so I’m hoping documenting the issues and corrections will work as a memory aid in the future.

Firstly, I can never remember Git’s equivalent to ‘svn update’, which is:

git pull

Next up is recompiling Dionaea from your updated source directory, this is no different to an initial install as per Markus’ excellent instructions.

I keep forgetting to update my configuration file to include any directives needed for the new shiny new functionality you’re upgrading to get access to. I find diff useful for identifying any new additions, for example

/opt/dionaea/etc/dionaea# diff dionaea.conf dionaea.conf.dist

Assuming there’s no major changes you should only see differences specific to your installation, for example your email address to receive analysis reports, or your VirusTotal api key. If there are any other differences you’ll need to add the new content.

From the experience I’ve had in the past week, if you encounter any unexpected problems after updating Dionaea, make sure your pre-requisites are also upto date. After updating Dionaea last week to gain access to the new integration with VirusTotal’s api my Dionaea sensor started to die randomly. Markus was a great help with troubleshooting (thanks again) and my problems were eventually corrected after it was noted that my libemu installation was outdated; after a quick ‘git pull’ and ‘make’ (again following Markus’ instructions).

As I said, this is probably basic for most of you out there, but as I keep making similar mistakes I plan to refer back to this list of daft issues before bugging anyone for support in future. You never know, it might allow someone else to retain an air of competence before proving otherwise

– Andrew

Development of new features for Dionaea has been fairly impressive of late, and I’ve been lax in keeping up to date. When Markus asked if I’d tested the graph utility that he created and wrote about here, it served as a kick to stop putting off some of the jobs I’ve got on the growing to-do list.

I won’t go into too much detail about running the script as Markus has already done a better job than I could. However I will point out that if you run your Dionaea installation on Debian stable, then your out of luck; the standard packages for sqlite are too old to take the script. Best advice is to copy your logsql.sqlite database to a Ubuntu machine and work from there (oh, and in case you didn’t guess from the script name, make sure you’ve actually installed gnuplot…).

A powerful machine is recommended, the only Ubuntu system I had to hand whilst testing was my AA1 netbook, which took 85 minutes to crunch through the script and my database.

I have immediately found the graphs produced useful as they’ve highlighted a couple of obvious spikes (see below) in activity that I would have (and did) miss if solely relying on log files and databases. This really shows the power and importance of visualising security and log information.

dionaea-overview - from gnuplotsql.py

If you’re interested the output for the InfoSanity’s installation is now online here. I’m looking to expand the statistics from the InfoSanity honeypot environment that are publicly available, this makes a nice start. As always, big thanks to Markus and carnivore.it team for the effort.

– Andrew Waite

Mercury Live DVD was initially (I believe) announced in a post to the Nepenthes Mailing list. It is a remastered Ubuntu distribution with pre-installed honeypot applications and malware analysis tools created by John Moore. From the ReadMe:

This live DVD is a remastered version of Ubuntu 10.0 Beta LTS x86_32. It was designed due to my being disappointed with another reverse engineering malware live CD that was released recently. I have decided to call my creation MERCURY, which is an acronym for Malware Enumeration, Capture, and Reverse Engineering.

The Mercury live DVD contains tools used for digital forensics, data recovery, network monitoring, and spoofing. It should primarily be used as a honeypot or network monitoring platform as well as a laboratory and teaching aid. There are three honeypots installed – honeyd, nepenthes, and dionaea. Four, if you include netcat.

The majority of the additional applications reside in /opt:

• Dionaea (0.1.0) – Dionaea is a malware collection honeypot focusing primarily on SMB emulation, covered on InfoSanity numerous times before.
• FFP – Fuzzy Fingerprinting is a util to aid SSH MitM attacks.
• jsunpack-n – Is a Javascript unpacker, perfect for analysis captured or potentially malicious URLs in more depth.
• Kippo (svn rev.169) – Kippo is an low-medium interaction SSH honeypot, Also covered
• mitm-ssh – Unsurprisingly, a utility for aiding man in the middle attacks against SSH connections.
• Origami & pdftools – Two frameworks for analysing malicious PDF files.
• Volatility – an excellent memory analysis toolkit
• Zerowine-vm – A malware behavior analysis platform. I’ve covered ZeroWine here before, and whilst I find it useful for initial analysis I found it a pain to setup and get running. The fact this works out of the box on Mercury is enough reason alone to keep the .iso handy.

Other tools are installed on the system as started, access from standard locations (/etc, /usr/bin, etc.). I won’t try to list them all, but some highlights include:

• Nepenthes – Dionaea’s predecessor
• Honeyd – Honeypot system, perfect for emulating multiple different systems from one platform. Covered in more depth here.
• John – John the Ripper, password cracker
• ircd-hybrid – irc server daemon, useful for analysis irc-based malware’s interaction with command and control systems.
• Snort – de-facto intrusion detection system.
• Wireshark – Packet capture and network analysis tools.

I could go on, but I’m sure you get the idea.

Setting up a honeypot, and analysing the results, has never been easier. And I’m sure the toolkit’s functionality will also be useful in other scenarios; incident response, general network administration or as a safe learning platform. So what are you waiting for?

–Andrew Waite

N.B. there have been several mirror’s and downloads established, the most reliable download source I’ve used is Markus’ mirror at carnivore.it

This week I was interviewed for the Disaster Protocol Security Podcast. My theory is that everyone else was superstitious and didn’t want to risk being on number 13, so they got stuck with me…..

Basically, the interview is just me talking about honeypots and some of the results and findings that have been discussed discussed both on this blog and via Twitter. Hopefully you’ll find it an interesting listen, and hopefully you’ll be able to understand me. Seems a few people have struggled so I’ll need to work on my ‘BBC English’ next time around….

Always interested in hearing others thoughts or comments on honeypots or infosec in general; so if you liked, disliked or disagreed with any of the content let me know.

The podcast episode can be downloaded here.

–Andrew Waite

This morning I cause myself a problem. Annoyingly it was foreseeable and avoidable, this is my excuse (not great, but I’ll stick to it). But as every problem is merely an opportunity in disguise whist I’m re-building systems I might as well document the process. The original InfoSanity guide for installing Kippo was based off of the latest stable version, but I rapidly migrated to the development SVN on learning of the MySQL logging capabilities, so this guide covers that.

Packages

As I’m using a Debian system a lot of the system pre-requisites are packaged, this aren’t all needed immediately but we might as well grab them all at once.

apt-get install subversion #for svn
apt-get install python-twisted python-mysqldb # Python and required modules
apt-get install mysql-server #

Basic Kippo setup

Grab Kippo direct from svn, at time of writing I got version 160. (latest instructions):

svn checkout http://kippo.googlecode.com/svn/trunk/ /opt/kippo-svn

Now we can start the honeypot system:

./start.sh

That’s it, all that is required to get the system running. To confirm you can ssh locally with ssh -p2222 root@127.0.0.1, unless you’ve jumped ahead and edited the config, password will be 123456.

MySQL

Log into MySQL via commanline, assuming you’ve not modified the kippo.cfg database directives build the database:

create database kippo;
grant all on kippo.* to ‘kippo’@'localhost’ identified by ‘secret’;

Next edit the kippo.cfg accordingly you database/user/password and uncomment the [database] configuration directives. REMEBER to uncomment ;[database] line not just the parameters, that has now caught me out twice.

Finally, build the database structure with the script that can be found in <kippo>/doc/sql/:

doc/sql/# mysql -ukippo -psecret kippo < mysql.sql

Restart your Kippo process and you should be good; re-test access to the shell and view the database tables to confirm that logs are being written to the database.

Happy Honeypotting

–Andrew Waite

So far my Kippo honeypot installation has recieved a number of successful log ins from maliciuos users, some of which have been helpful enough to provide some tools for further analysis. A lot of the archives which have been downloaded show that the kits have been in use for a while, with some archive timestamps going back as far as 2004 (of course this could simply be an incorrect clock on the machine that created the archive). Picking on the most recent download (2010-07-18) I’ve taken a look at the archive containing gosh.tgz.

The archive was downloaded from linux<dot>hostse<dot>com<slash>gosh<tgz>, system is down at time of writing but take care if attempting to investigate yourself. Before downloading the user checked around the system with commands: w, uname -a and cat /proc/cpuinfo, and archive was downloaded and extracted in /dev/shm/.

Once extracted, the archive contains a number of files:

• Interesting files:
• Files 1 to 5, common and pass_file are password lists, totalling 235,523 potential passwords.
• mfu.txt is a list of IP addresses, mostly in the 38.99.0.0/16 address space.
• pscan2 is a fairly common and generic port scanner.
• scam is a shell script that appears to be the core brains of the toolkit. It essentially looks through scanning a different ranges of IP addresses while periodically emailing the contents of vuln.txt back to it’s master (mafia89tm@yahoo.co.uk).
• ss: appears to be another scanner used for looking for potential targets.
• ssh-scan: appears to be a Romanian tool from the message provided if run without arguments, according to Google Translate (possibly NSFW), and as you would guess from the file name is a scanner for SSH services.
• vuln.txt is blank in the archive, and will be the output of vulnerable systems located by the scanners.

All told this appears to be a kit for performing further scans for unsecured SSH sessions, and it is likely that a similar kit hosted on a different compromised machine was responsible for identifying my installation in the first place. Kits like this also quickly show the problem with tracking down the malicious user behind an compromise or attempt, it is rare for attacks to be launched from systems that can easily be traced back to the malicious user.

A quick Google search confirms that this kit (and user) has been seen in the wild attacking other systems, this posting on the Shell Person blog writes up the aftermath after a production system was compromised by the same kit.

–Andrew Waite

I’ve been running Kippo for nearly two weeks now (decided to live dangerously and go with SVN version) and have seen some interesting results.

Top 10 most common passwords attempted:

1. a (651)
2. 123456 (495)
3. password (331)
4. 12345 (302)
5. 123 (224)
6. 1234 (169)
7. 1 (139)
8. 12 (123)
9. root (105)
10. test (46)

Select count(password), password
from auth
where password <> ”
group by password
order by count(password) desc
limit 10;

Top 10 most common username attempted:

1. root (8510)
2. admin (144)
3. test (127)
4. oracle (96)
5. nagios (49)
6. mysql (47)
7. guest (43)
8. info (42)
9. user (41)
10. postgres (40)

select count(username), username
from auth
where username <>”
group by username
order by count(username)
desc limit 10;

Success ratio:

17065 attempts, 48 successful connections. (n.b. results skewed as account has purposefully poor choice of password)

select count(success),success
from auth
group by success
order by success;

Number of connections per unique IP:

1. 202.99.89.69 (5212)
2. 200.61.189.164 (1752)
3. 78.37.83.203 (1043)
4. 218.108.235.86 (848)
5. 195.14.50.8 (628)
6. 218.80.200.138 (271)
7. 58.222.200.226 (238)
8. 58.18.172.206 (158)
9. 119.188.7.174 (128)
10. 119.42.148.10 (113)

select count(ip), ip
from sessions
group by ip
order by count(ip) desc;

Number of attempts were relatively low IP address, in total 194 different source locations have attempted to access the server, with each typically only making 4 attemtps.

Packages:

Once exploited a number of attackers have proceeded to download various rootkits and utilities (thanks for these). Nothing too interesting yet, standard rootkit functionality, IRC clients and SSH scanners for further compromise. I still need to analyse some of these in more detail, so watch your RSS feeds for more to come.

One malicious user also attempted to create new user accounts on the server, if you have an account called ‘iony’ with a password of ‘ionyszaa’ then you may want to remove it…

If you’ve got a spare machine and public IP address, give Kippo a shot, setup is realitively easy; I’ve seen some interesting malicious user sessions and it turns out that some of those ’31337 haxxors’ that everyone fears really can’t type.

–Andrew Waite

As I started life as a Linux server admin I’m only too aware that many attackers see remote access functionality as a way into a system, and as SSH is the de facto standard for Linux access it is a prime target for attack. The stats collected by DShield give an indication to the extent of the problem.

As a result I’ve had the Kippo honeypot is something that I’ve had on my radar for a while. For a number of reasons I hadn’t found time to implement the system in a live environment, but a recent post on the Diatel blog suggested that installation may be quick and pain free.

Kippo is described by it’s author (Desaster) as:

Kippo is a medium interaction SSH honeypot designed to log brute force attacks and, most importantly, the entire shell interaction performed by the attacker.

Kippo is inspired, but not based on Kojoney.

Installation for me was painless, running a Debian system I downloaded the latest archive to disk, unpacked and installed the pyton-twisted package (I hadn’t read Mig5′s comment until after install so now need to go back and live on the bleeding edge…). I did hit a couple of problems when trying to start up the system, which is as simple as invoking ./start.sh

1. First, I was logged in as root when I first tried to start the system (not clever I know, was testing…). Kippo encounters an error when started by a root user. As Desaster rightly states, it’s not wise to run Kippo as a root user anyway and running as a regular user resolves the issue.
2. Second, when running as a normal user I got a ‘meaningful’ error of “Failed to load application: ‘NoneType’ object has no attribute ‘get’.” A quick piece of Google-fu lead me to this ticket, which explained Kippo was missing the file kippo.cfg, as explained copying kippo.cfg.dist to kippo.cfg correct the issue and produced a fully functional system.

There are a couple of key files that can be edited to change the feel of the system that is provided to malicious users:

• kippo.cfg contains runtime information including log location, fake hostname etc.
• kippo.tac contains an array ‘users’, which lists the username/password combination which the emulated SSH login will accept as ‘valid’.
• The honeyfs/ directory goes so far as to allow you to create a ‘real’ filesystem for the malicious user to interact with, potentially copying a live server’s filesystem to the directory to help camouflage the emulated system (after sensitive data is removed/sanitised obviously….). I haven’t tried this myself yet but is definitely on my to-do list.

From initial testing I’ve got high hopes for Kippo becoming a mainstay in my honeypot toolbox; the interaction session provided to a malicious user is reasonably convincing at first glance, and I particularly like the trick to keep users logged in after they think they’ve sent an ‘exit’ command to close the session, it could get some interesting results.

For post compromise analysis Kippo also provides some an interesting utility, utils/playlog.py. This allows you to replay a malicious terminal session in real-time, typos and all, to truely provide a feel for the malicious users interaction with the session. To help whet your apetite whilst I wait for someone to target my kippo installation, Kippo has a few demo’s of the playlog capabilities from compromise attempts. Get your demos here.

–Andrew Waite

I’ve been a bit lax in writing this post; around a month ago Miguel Jacq got in contact to let me know about a couple of errors he encountered when running InfoSanity’s mimic-nepstats.py with a small data set. Basically if your log file did not include any submissions, or was for a period shorter than 24hours the script would crash out, not the biggest problem as most will be working with larger data sets but annoying non the less.

Not only did Miguel let me know about the issues, he was also gracious enough to provide a fix, the updated script can be found here. An example of the script in action is below:

cat /opt/dionaea/var/log/dionaea.log| python mimic-nepstats_v1-1.py

Statistics engine written by Andrew Waite – www.infosanity.co.uk

Number of submissions: 84
Number of unique samples: 39
Number of unique source IPs: 65

First sample seen: 2010-06-08 08:25:39.569003
Last sample seen: 2010-06-21 15:24:37.105594
System Uptime: 13 days, 6:58:57.536591
Average daily submissions: 6

Most recent submissions:
2010-06-21 15:24:37.105594, 113.37.56.28, emulate://, 56b8047f0f50238b62fa386ef109174e
2010-06-21 15:18:08.347568, 195.205.5.71, tftp://195.205.5.71/ssms.exe, fd28c5e1c38caa35bf5e1987e6167f4c
2010-06-21 15:17:08.391267, 195.117.74.62, tftp://195.117.74.62/ssms.exe, bb39f29fad85db12d9cf7195da0e1bfe
2010-06-21 06:29:03.565988, 195.160.222.101, tftp://195.160.222.101/ssms.exe, fd28c5e1c38caa35bf5e1987e6167f4c
2010-06-20 23:34:15.967299, 195.242.145.40, http://208.53.183.164/trying.exe, 094e2eae3644691711771699f4947536

– Andrew Waite