This morning I cause myself a problem. Annoyingly it was foreseeable and avoidable, this is my excuse (not great, but I’ll stick to it). But as every problem is merely an opportunity in disguise whist I’m re-building systems I might as well document the process. The original InfoSanity guide for installing Kippo was based off of the latest stable version, but I rapidly migrated to the development SVN on learning of the MySQL logging capabilities, so this guide covers that.
As I’m using a Debian system a lot of the system pre-requisites are packaged, this aren’t all needed immediately but we might as well grab them all at once.
apt-get install subversion #for svn
apt-get install python-twisted python-mysqldb # Python and required modules
apt-get install mysql-server #
Basic Kippo setup
Grab Kippo direct from svn, at time of writing I got version 160. (latest instructions):
svn checkout http://kippo.googlecode.com/svn/trunk/ /opt/kippo-svn
Now we can start the honeypot system:
That’s it, all that is required to get the system running. To confirm you can ssh locally with ssh -p2222 firstname.lastname@example.org, unless you’ve jumped ahead and edited the config, password will be 123456.
Log into MySQL via commanline, assuming you’ve not modified the kippo.cfg database directives build the database:
create database kippo;
grant all on kippo.* to ‘kippo’@’localhost’ identified by ‘secret’;
Next edit the kippo.cfg accordingly you database/user/password and uncomment the [database] configuration directives. REMEBER to uncomment ;[database] line not just the parameters, that has now caught me out twice.
Finally, build the database structure with the script that can be found in <kippo>/doc/sql/:
doc/sql/# mysql -ukippo -psecret kippo < mysql.sql
Restart your Kippo process and you should be good; re-test access to the shell and view the database tables to confirm that logs are being written to the database.
When discussing some of my recent findings with Kippo I’ve been asked a few times for suggestions for how people can prevent their systems from being compromised via this vector. A quick Google search shows that there are already a number of good resources covering the options, including: Debian Administration Article and Securing Debian Manual. However, the high number of options can leave people unsure where to start so I’ll summarise some of those that are more common and can provide the highest return on investment for the time taken to make the change.
N.B. a lot of the suggestions below are valid for most/all remote access functionality.
Restrict access from unknown locations
If possible (it isn’t always) restrict access to only come from known and trusted sources. This can be down at multiple choke points in the network and system; perimeter firewall, host firewall (iptables etc.) or sshd config. For working with sshd the /etc/hosts.allow and /etc/hosts.deny, for example:
#Corporate HQ gateway
#Generic Deny All
It doesn’t matter how insecure your system is, if an attacker can’t connect and communicate with a vulnerable service they can’t exploit it, period.
Restrict remote root access
Preventing remote access to the root account can reduce the damage that can be caused by a compromised. With SSH this can be achieved with a single configuration line:
Only allow access to specific accounts
Does every account on you system need to be able to remotely access the system via SSH? No? Then why can it?
Remote system access can be restricted on a per user basis. This can be either as a whitelist using the AllowUsers directive or as a blacklist with the DenyUsers directive. For example, if I only wanted to allow my own account access via ssh:
These capabilities can be useful with certain honeypot systems; if you create a weak user account linked with an ftp or pop3 honeypot (for example), then the same weak accounts can be prevented from gaining access to a shell with the DenyUsers directive, limiting the weak account to only access those services that are being monitored.
Run on non-standard port
Yes, this is ‘security by obscurity’; if this is the only change you make you haven’t really improved security any, but it is still useful as part of wider security posture. Attackers are continually scanning the internet looking for new systems to exploit, currently the ISC statistics show connections to tcp22 at around 100k targets; even moving to a relatively common alternative port of 2222 drops the malicious traffic by around 90%.
This reduces the number of malcious attempts targeting the service, which will both reduce processor/network load and ‘noise’ in the log. If you now get a burst of failed log-in attempts in the logs, then this may be indicative of a specific attacker rather than just the usual background noise of bots and worms scanning for new victims.
Implementing the above can drastically improve SSH security above the defaults, with a relatively small effort required providing a great ROI. So what’s your excuse? Go harden that SSH installation
So far my Kippo honeypot installation has recieved a number of successful log ins from maliciuos users, some of which have been helpful enough to provide some tools for further analysis. A lot of the archives which have been downloaded show that the kits have been in use for a while, with some archive timestamps going back as far as 2004 (of course this could simply be an incorrect clock on the machine that created the archive). Picking on the most recent download (2010-07-18) I’ve taken a look at the archive containing gosh.tgz.
The archive was downloaded from linux<dot>hostse<dot>com<slash>gosh<tgz>, system is down at time of writing but take care if attempting to investigate yourself. Before downloading the user checked around the system with commands: w, uname -a and cat /proc/cpuinfo, and archive was downloaded and extracted in /dev/shm/.
Once extracted, the archive contains a number of files:
|1:||ISO-8859 English text, with CRLF line terminators|
|3:||ASCII C++ program text, with CRLF line terminators|
|a:||ISO-8859 text, with CRLF line terminators|
|common:||ASCII C++ program text|
|gen-pass.sh:||Bourne-Again shell script text executable|
|pscan2:||ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.2.5, not stripped|
|scam:||Bourne-Again shell script text executable|
|secure:||Bourne-Again shell script text executable|
|ss:||ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.0.0,stripped|
|ssh-scan:||ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.0.0, stripped|
- Interesting files:
- Files 1 to 5, common and pass_file are password lists, totalling 235,523 potential passwords.
- mfu.txt is a list of IP addresses, mostly in the 184.108.40.206/16 address space.
- pscan2 is a fairly common and generic port scanner.
- scam is a shell script that appears to be the core brains of the toolkit. It essentially looks through scanning a different ranges of IP addresses while periodically emailing the contents of vuln.txt back to it’s master (email@example.com).
- ss: appears to be another scanner used for looking for potential targets.
- ssh-scan: appears to be a Romanian tool from the message provided if run without arguments, according to Google Translate (possibly NSFW), and as you would guess from the file name is a scanner for SSH services.
- vuln.txt is blank in the archive, and will be the output of vulnerable systems located by the scanners.
All told this appears to be a kit for performing further scans for unsecured SSH sessions, and it is likely that a similar kit hosted on a different compromised machine was responsible for identifying my installation in the first place. Kits like this also quickly show the problem with tracking down the malicious user behind an compromise or attempt, it is rare for attacks to be launched from systems that can easily be traced back to the malicious user.
A quick Google search confirms that this kit (and user) has been seen in the wild attacking other systems, this posting on the Shell Person blog writes up the aftermath after a production system was compromised by the same kit.
I’ve been running Kippo for nearly two weeks now (decided to live dangerously and go with SVN version) and have seen some interesting results.
Top 10 most common passwords attempted:
- a (651)
- 123456 (495)
- password (331)
- 12345 (302)
- 123 (224)
- 1234 (169)
- 1 (139)
- 12 (123)
- root (105)
- test (46)
Select count(password), password
where password <> ”
group by password
order by count(password) desc
Top 10 most common username attempted:
- root (8510)
- admin (144)
- test (127)
- oracle (96)
- nagios (49)
- mysql (47)
- guest (43)
- info (42)
- user (41)
- postgres (40)
select count(username), username
where username <>”
group by username
order by count(username)
desc limit 10;
17065 attempts, 48 successful connections. (n.b. results skewed as account has purposefully poor choice of password)
group by success
order by success;
Number of connections per unique IP:
- 220.127.116.11 (5212)
- 18.104.22.168 (1752)
- 22.214.171.124 (1043)
- 126.96.36.199 (848)
- 188.8.131.52 (628)
- 184.108.40.206 (271)
- 220.127.116.11 (238)
- 18.104.22.168 (158)
- 22.214.171.124 (128)
- 126.96.36.199 (113)
select count(ip), ip
group by ip
order by count(ip) desc;
Number of attempts were relatively low IP address, in total 194 different source locations have attempted to access the server, with each typically only making 4 attemtps.
Once exploited a number of attackers have proceeded to download various rootkits and utilities (thanks for these). Nothing too interesting yet, standard rootkit functionality, IRC clients and SSH scanners for further compromise. I still need to analyse some of these in more detail, so watch your RSS feeds for more to come.
One malicious user also attempted to create new user accounts on the server, if you have an account called ‘iony’ with a password of ‘ionyszaa’ then you may want to remove it…
If you’ve got a spare machine and public IP address, give Kippo a shot, setup is realitively easy; I’ve seen some interesting malicious user sessions and it turns out that some of those ‘31337 haxxors’ that everyone fears really can’t type.
As I started life as a Linux server admin I’m only too aware that many attackers see remote access functionality as a way into a system, and as SSH is the de facto standard for Linux access it is a prime target for attack. The stats collected by DShield give an indication to the extent of the problem.
As a result I’ve had the Kippo honeypot is something that I’ve had on my radar for a while. For a number of reasons I hadn’t found time to implement the system in a live environment, but a recent post on the Diatel blog suggested that installation may be quick and pain free.
Kippo is described by it’s author (Desaster) as:
Kippo is a medium interaction SSH honeypot designed to log brute force attacks and, most importantly, the entire shell interaction performed by the attacker.
Kippo is inspired, but not based on Kojoney.
Installation for me was painless, running a Debian system I downloaded the latest archive to disk, unpacked and installed the pyton-twisted package (I hadn’t read Mig5’s comment until after install so now need to go back and live on the bleeding edge…). I did hit a couple of problems when trying to start up the system, which is as simple as invoking ./start.sh
- First, I was logged in as root when I first tried to start the system (not clever I know, was testing…). Kippo encounters an error when started by a root user. As Desaster rightly states, it’s not wise to run Kippo as a root user anyway and running as a regular user resolves the issue.
- Second, when running as a normal user I got a ‘meaningful’ error of “Failed to load application: ‘NoneType’ object has no attribute ‘get’.” A quick piece of Google-fu lead me to this ticket, which explained Kippo was missing the file kippo.cfg, as explained copying kippo.cfg.dist to kippo.cfg correct the issue and produced a fully functional system.
There are a couple of key files that can be edited to change the feel of the system that is provided to malicious users:
- kippo.cfg contains runtime information including log location, fake hostname etc.
- kippo.tac contains an array ‘users’, which lists the username/password combination which the emulated SSH login will accept as ‘valid’.
- The honeyfs/ directory goes so far as to allow you to create a ‘real’ filesystem for the malicious user to interact with, potentially copying a live server’s filesystem to the directory to help camouflage the emulated system (after sensitive data is removed/sanitised obviously….). I haven’t tried this myself yet but is definitely on my to-do list.
From initial testing I’ve got high hopes for Kippo becoming a mainstay in my honeypot toolbox; the interaction session provided to a malicious user is reasonably convincing at first glance, and I particularly like the trick to keep users logged in after they think they’ve sent an ‘exit’ command to close the session, it could get some interesting results.
For post compromise analysis Kippo also provides some an interesting utility, utils/playlog.py. This allows you to replay a malicious terminal session in real-time, typos and all, to truely provide a feel for the malicious users interaction with the session. To help whet your apetite whilst I wait for someone to target my kippo installation, Kippo has a few demo’s of the playlog capabilities from compromise attempts. Get your demos here.