Infosanity's Blog

As a security guy I find my paranoia levels are slightly high than most, a little something inside me picks up on things that general users miss that indicate that something isn’t right. This morning was no exception….

After acquiring coffee, morning inbox was opened which presented the following:

These are email alerts sent by the monitoring system Nagios, running within InfoSanity’ networks. The NRPE-check_users parameters have been modified from the Debian defaults to be more paranoid; triggering a warning alert with a single user logged into the server, and go critical if more than one. So; from this, someone is logged into the web server, and it isn’t me.

Feeling the onrush of panic, I log into server and chuck commands at a shell to see who is violating my system. Last, who and /var/log/auth all showed that no one had accessed the server at the time of my alerts. Everything good? Not if your paranoid, starting to smell a rootkit causing the system to lie to me.

There are a couple of anti-rootkit utilities that have served me well in the past, chkrootkit and rkhunter. Wondering which one to run? As we’ve already discovered I’m paranoid, answer was both. And both gave a clean bill of health to the system. Now I’m really getting paranoid.

Wanting to see if I’ve missed a trend, or if the issue is present with other servers in the environment I log into Nagios interface for a more detailed look, and find the answer:

Nagios webUI notifications

Anyone spotted it? Yep, the service alert went critical when the check’s socket timed out (network issue), and then dropped to warning showing a single user when the connectivity returned; which was correct, I’d forgotten to log out of the console on my last VMWare connection. Stepping down from high alert…..

Morale of the story?

Don’t respond to system alerts before finishing first coffee of the morning. The events weren’t a total loss though, besides getting my heart-rate up and blood flowing, it was a good(ish) refresher for incident response (can’t beat the adrenaline rush of responding to an incident, real or imagined) and rkhunter uncovered a potential weakness in the server configuration which has since been corrected (no, I’m not telling you what).

–Andrew Waite

Oh yeah, had I just opened the email, could have avoid the whole situation:

Running through my morning routine of catching up with email, twitter, etc. I came across this post showing Sequal7′s first hits on a Kippo installation. In addition to making amusing reading, it gave me a nudge to check back on the InfoSanity Kippo sensor. Initially I was looking to see if the same individual had stumbled across my sensor; they hadn’t at least not from the information I have available.

However, when checking if the newly changed password matched anything in my database I found a new ‘realm’ entry in the ‘input’ table, ‘ssh’. This got me curious, one of my ‘guests’ decided to hit another system whilst logged in to mine; ssh’ing to another IP, accepting the certificate and providing the password to said system (I’m assuming).

It should also be worth noting that by this point the user had already failed to notice that input hadn’t returned to their own system. After (attempting) to change my sensor’s root password (to ‘yahoo’, really) the user exited, but was caught out by Kippo’s trick of clearing the terminal and changing prompt to ‘localhost’, in total I viewed a ~20 minute terminal session of the user trying to compromise other systems, and failing in the same manner.

My assumption is that the user was running through a list of vulnerable systems identified by SSH scanners similar to the kit I wrote about earlier (it wasn’t the same gosh.tgz kit, but first glance shows similar functionality). From this I feel it’s safe to assume that the systems connected to in the logs available are those of other (probably 0wned) systems, rather than anything connected to my guest. Likewise it is probable that the source connection is also a compromised third party rather than belonging to by guest.

e.tgz :
For the curious, archive contents:

e/
e/exp_moosecox.c
e/funny.jpg
e/exp_powerglove.so
e/exp_ingom0wnar.c
e/pwnkernel.c
e/exp_cheddarbay.so
e/exp_powerglove.c
e/exp_ingom0wnar.so
e/exp_therebel.so
e/run_nonnull_exploits.sh
e/exp_paokara.so
e/exp_framework.h
e/exp_moosecox.so
e/exp_wunderbar.c
e/exp_cheddarbay.c
e/run_null_exploits.sh
e/exploit.c
e/exp_therebel.c
e/exp_vmware.so
e/exp_vmware.c
e/exp_wunderbar.so
e/exploit
e/exp_paokara.c

Whilst investigating the individual exploits and files; I came across this post, indicating ‘my’ archive is a known fire and forget post exploit kit. Here be Skiddies…

–Andrew Waite

I’m sure by now most people are aware of a new round of scams where victims are being called by a ‘support company’ suggesting that the victim’s computer has malware installed which they can fix. If you need it, this BBC article covers the basics. Well, I just got the call

First up the caller seemed to be auto-dialling large volumes of numbers looking for someone to pick-up as the caller (male, poor line quality meant I missed the name given) was unprepared when I answered. The caller was clearly reading from a script, I may have over-played the ‘Sorry, I’m just a dumb user that knows nothing about computers card’ but despite telling him I was clueless and willing to accept everything he told me I was still present with a long winded argument for ‘if you don’t believe us this is how I’ll prove it’ speech.

Unfortunately I wasn’t able to through the full process as, despite telling my new friend otherwise, I wasn’t able to get to a Windows machine to work through the process. Only laptop to hand was my netbook running Ubuntu, and my landline isn’t mobile so I couldn’t head up stairs. (My landline never rings, everything I do is via mobile and only have landline for ADSL connection. I’m suspicious of all landline calls before I even pick up the phone.)

After ensuring I was looking at the system wallpaper, I was instructed to press the ‘key on bottom left of keyboard with four squares that looks like the Microsoft logo’ and with another finger press the ‘r’ key. This is where I was given ‘proof’ that my system was infected, using a ‘hidden’ command that will list all infections, what is the magic command? inf (for ‘infections’), which opens Windows Explorer in C:\Windows\inf, screenshot below shows the infections on my system. I’m guessing at this point, the every user may have just entered dummy mode.

At this point I lost the caller, whether a technical fault or he’d guessed something wasn’t right (I can’t act for toffee). I’m hoping that I’ll get a second bite at the cherry at some point; my missus took a similar call a few weeks back, having spent too long listening to my security rants she immediately spotted the scam, pointed out that I was a ‘security guy’ and hung up. Information that they clearly didn’t have when ringing back (could be more that one cold calling organisation).

Unfortunately, despite my usual laughing at people who fall for these scams I can see how those with less knowledge could fall for the premise. Computers and software regularly phone home to check for updates etc, using this information to identify infected systems would/could make sense, and from an end user perspective I struggled to tell the difference between the sorts of actions I was asked to take by my ‘friend’ than those I regularly instruct friends and family members when I’m trying to provide remote support.

Be safe and spread the word to those less knowledgeable about computers that this is an active scam. Bottom line is: no legit IT company will call you to fix a problem that you weren’t aware of.

–Andrew Waite

<UPDATE>

I just received a new call following the same theme but with a different vector. This time the call came from ‘Microsoft Service Department’, and with a different convincer; this time I was baby-stepped through to opening the Security log with each entry being ‘evidence’ of the malware infection that ‘at this very moment is damaging my computer and hard drive’. To be fair, in this case they ‘could’ be right.

Other differences indicate that either this is a different group from the first caller, or they’ve improved the call systems used to implement the scam. On lifting the receiver I was placed immediately to on hold music before speaking to my ‘MS representative’ a few moments later. I believe that automated dialling is illegal within the UK, but given the nature of the call I doubt they care much either way.

REMEMBER: Microsoft will NOT call home (or business) users to inform that you’ve got a malware infection

<UPDATE>

I’ve tried messing around with SSH port forwarding in the past, but always struggled to get my head around what I was trying to connect to where, and ultimately didn’t result in anything useful. This time around I’ve put in some dedicated time to get to the bottom forwarding ports within SSH tunnels. And I’m glad I did, my with only a handful of connections the possibilities are making my head spin.

To help my head get around problems I encountered I found a number of resources helpful.

• SSH Man Pages
• Hacking Linux Exposed – SSH Tunnelling
• Symantec Port Forwarding Article

Example Scenario: A server (10.0.0.100) on an internal LAN has web services that need to be accessed, but has no remote access through the firewall. Luckily, there is remote access to an SSH server (10.0.0.200) sat on the same LAN.

From command-line:

ssh -L 8000:10.0.0.100:80 ssh-server.somedomain.tld

This connects to the machine at ssh-server.somedomain.tld and, once authenticated, forwards 10.0.0.100:80 to port 8000 on the local machine. Now accessing the remote services is as simple as pointing your browser to http://127.0.0.1:8000.

The same functionality can be achieved using configuration files. For example, edit ~/.ssh/config:

Host tunnel
HostName ssh-server.somedomain,tld
LocalForward 127.0.0.1:8000 10.0.0.100:80

To establish a connection with the configuration file in place simply run; ssh tunnel.

The port forward will remain active for as long as the SSH session is connected. If you don’t need to interact with the SSH session in addition to the forwarded port passing ssh the -fN flags will cause the session to be backgrounded once authentication can be established.

If you haven’t already, I suggest you investigate the possibilities within your own environment; and if an evil grin doesn’t spread across your face then you still don’t fully get it

–Andrew

As proven by the logs generated by Kippo honeypot sensors have shown, attacks against SSH services are regularly seen in the wild. Even if you follow best practices for securing the service, the malicious scans will utilise resources available to your environment; CPU, bandwidth etc. In sufficient volume legitimate operation may be impacted as the server rejects failed login attempts.

This is where utilities like Breakinguard come into their own. Basically Breakinguard monitors log files for signs of malicious activity, and once a single source has triggered enough alerts blocks all connections from the source location. Other utilities (most notably fail2ban) perform the same activities, but I’m partial to Breakinguard due to it’s small size and simple configuration (and from knowing the author ).

Installation is straightforward, and for the most part automated. Once downloaded and extracted installation is handled by the configure script. On Debian based systems this will install the pre-requisite Perl modules and transfer the utilities components to the standard locations:

• breakinguard script – /usr/local/sbin/breakinguard
• config – /etc/breakinguard.conf
• init script – /etc/init.d/breakinguard

Once installed you need to edit your configuration. The breakinguard.conf file is fairly self explanatory, I normally edit:

• $alert_email -> set to the email address that you want to receive notifications of blocked attacks. On a publicly accessible system these alerts can be high volume, you may want to use a specific email account or at least setup some auto-move rules in your email client to avoid your inbox being spammed.
• $number_of_attempts -> This specifies the number of malicious log entries need to be generated by a specific IP address before the source is blocked. Due to the timing of the Breakinguard route this isn’t always an exact science, the default of 5 does a good job of avoid false positives whilst still blocking an attack in it’s infancy.
• $log_to_watch -> selects the logfile to monitor for signs of malicious activity. /var/log/auth.log is the obvious choice on a Debian based system.
• @safe_ips -> This array allows you to specify a number of trusted networks that will not be blocked, regardless of the number of times the they trigger alerts in the logs. This is useful for insuring that you don’t get locked out of your own systems in the event your keyboard ‘breaks’. On higher end systems with hardware remote management systems (iLo, DRAC, etc.) or virtual systems that provide remote access to the ‘physical’ console I leave this list to local subnet only and use the alternative options to access the server if I do lock myself out.
• $DEBUG -> set to 1 by default, this runs the utility in test mode without actually blocking malicious sources, perfect for testing configuration before going live. Once you’re good to set set $DEBUG to 0 and wait for the attacks to start. Example testing in debug mode is below:

Breakinguard - Debug Mode

Blocking and unblocking of malicious sources is handled via iptables. Once the $number_of_attempts limit is hit Breakinguard will run the $block_command (configurable in /etc/breakinguard.conf) which by default is ‘/sbin/iptables -I INPUT -s %s -j DROP’, with %s being replaced with the attacking IP. After a configurable timeout ($block_length), the $unblock_command removes the restriction.

You can see the IP addresses currently blocked as they are listed in /var/run/breakinguard/, alternatively listing the current iptables configuration will show sources currently being blocked, for example:

Breakinguard iptables

Download Breakinguard here

–Andrew Waite

Mercury Live DVD was initially (I believe) announced in a post to the Nepenthes Mailing list. It is a remastered Ubuntu distribution with pre-installed honeypot applications and malware analysis tools created by John Moore. From the ReadMe:

This live DVD is a remastered version of Ubuntu 10.0 Beta LTS x86_32. It was designed due to my being disappointed with another reverse engineering malware live CD that was released recently. I have decided to call my creation MERCURY, which is an acronym for Malware Enumeration, Capture, and Reverse Engineering.

The Mercury live DVD contains tools used for digital forensics, data recovery, network monitoring, and spoofing. It should primarily be used as a honeypot or network monitoring platform as well as a laboratory and teaching aid. There are three honeypots installed – honeyd, nepenthes, and dionaea. Four, if you include netcat.

The majority of the additional applications reside in /opt:

• Dionaea (0.1.0) – Dionaea is a malware collection honeypot focusing primarily on SMB emulation, covered on InfoSanity numerous times before.
• FFP – Fuzzy Fingerprinting is a util to aid SSH MitM attacks.
• jsunpack-n – Is a Javascript unpacker, perfect for analysis captured or potentially malicious URLs in more depth.
• Kippo (svn rev.169) – Kippo is an low-medium interaction SSH honeypot, Also covered
• mitm-ssh – Unsurprisingly, a utility for aiding man in the middle attacks against SSH connections.
• Origami & pdftools – Two frameworks for analysing malicious PDF files.
• Volatility – an excellent memory analysis toolkit
• Zerowine-vm – A malware behavior analysis platform. I’ve covered ZeroWine here before, and whilst I find it useful for initial analysis I found it a pain to setup and get running. The fact this works out of the box on Mercury is enough reason alone to keep the .iso handy.

Other tools are installed on the system as started, access from standard locations (/etc, /usr/bin, etc.). I won’t try to list them all, but some highlights include:

• Nepenthes – Dionaea’s predecessor
• Honeyd – Honeypot system, perfect for emulating multiple different systems from one platform. Covered in more depth here.
• John – John the Ripper, password cracker
• ircd-hybrid – irc server daemon, useful for analysis irc-based malware’s interaction with command and control systems.
• Snort – de-facto intrusion detection system.
• Wireshark – Packet capture and network analysis tools.

I could go on, but I’m sure you get the idea.

Setting up a honeypot, and analysing the results, has never been easier. And I’m sure the toolkit’s functionality will also be useful in other scenarios; incident response, general network administration or as a safe learning platform. So what are you waiting for?

–Andrew Waite

N.B. there have been several mirror’s and downloads established, the most reliable download source I’ve used is Markus’ mirror at carnivore.it

When discussing some of my recent findings with Kippo I’ve been asked a few times for suggestions for how people can prevent their systems from being compromised via this vector. A quick Google search shows that there are already a number of good resources covering the options, including: Debian Administration Article and Securing Debian Manual. However, the high number of options can leave people unsure where to start so I’ll summarise some of those that are more common and can provide the highest return on investment for the time taken to make the change.

N.B. a lot of the suggestions below are valid for most/all remote access functionality.

Restrict access from unknown locations

If possible (it isn’t always) restrict access to only come from known and trusted sources. This can be down at multiple choke points in the network and system; perimeter firewall, host firewall (iptables etc.) or sshd config. For working with sshd the /etc/hosts.allow and /etc/hosts.deny, for example:

/etc/hosts.allow

#Corporate HQ gateway
sshd: 1.2.3.4/255.255.255.255

/etc/hosts.deny

#Generic Deny All
sshd: ALL

It doesn’t matter how insecure your system is, if an attacker can’t connect and communicate with a vulnerable service they can’t exploit it, period.

Restrict remote root access

Preventing remote access to the root account can reduce the damage that can be caused by a compromised. With SSH this can be achieved with a single configuration line:

/etc/ssh/sshd_config

PermitRootLogin no

Only allow access to specific accounts

Does every account on you system need to be able to remotely access the system via SSH? No? Then why can it?

Remote system access can be restricted on a per user basis. This can be either as a whitelist using the AllowUsers directive or as a blacklist with the DenyUsers directive. For example, if I only wanted to allow my own account access via ssh:

/etc/ssh/sshd_config

AllowUsers andrew

These capabilities can be useful with certain honeypot systems; if you create a weak user account linked with an ftp or pop3 honeypot (for example), then the same weak accounts can be prevented from gaining access to a shell with the DenyUsers directive, limiting the weak account to only access those services that are being monitored.

Run on non-standard port

Yes, this is ‘security by obscurity’; if this is the only change you make you haven’t really improved security any, but it is still useful as part of wider security posture. Attackers are continually scanning the internet looking for new systems to exploit, currently the ISC statistics show connections to tcp22 at around 100k targets; even moving to a relatively common alternative port of 2222 drops the malicious traffic by around 90%.

/etc/ssh/sshd_config

Port 2222

This reduces the number of malcious attempts targeting the service, which will both reduce processor/network load and ‘noise’ in the log. If you now get a burst of failed log-in attempts in the logs, then this may be indicative of a specific attacker rather than just the usual background noise of bots and worms scanning for new victims.

Summary

Implementing the above can drastically improve SSH security above the defaults, with a relatively small effort required providing a great ROI. So what’s your excuse? Go harden that SSH installation

–Andrew Waite

So far my Kippo honeypot installation has recieved a number of successful log ins from maliciuos users, some of which have been helpful enough to provide some tools for further analysis. A lot of the archives which have been downloaded show that the kits have been in use for a while, with some archive timestamps going back as far as 2004 (of course this could simply be an incorrect clock on the machine that created the archive). Picking on the most recent download (2010-07-18) I’ve taken a look at the archive containing gosh.tgz.

The archive was downloaded from linux<dot>hostse<dot>com<slash>gosh<tgz>, system is down at time of writing but take care if attempting to investigate yourself. Before downloading the user checked around the system with commands: w, uname -a and cat /proc/cpuinfo, and archive was downloaded and extracted in /dev/shm/.

Once extracted, the archive contains a number of files:

• Interesting files:
• Files 1 to 5, common and pass_file are password lists, totalling 235,523 potential passwords.
• mfu.txt is a list of IP addresses, mostly in the 38.99.0.0/16 address space.
• pscan2 is a fairly common and generic port scanner.
• scam is a shell script that appears to be the core brains of the toolkit. It essentially looks through scanning a different ranges of IP addresses while periodically emailing the contents of vuln.txt back to it’s master (mafia89tm@yahoo.co.uk).
• ss: appears to be another scanner used for looking for potential targets.
• ssh-scan: appears to be a Romanian tool from the message provided if run without arguments, according to Google Translate (possibly NSFW), and as you would guess from the file name is a scanner for SSH services.
• vuln.txt is blank in the archive, and will be the output of vulnerable systems located by the scanners.

All told this appears to be a kit for performing further scans for unsecured SSH sessions, and it is likely that a similar kit hosted on a different compromised machine was responsible for identifying my installation in the first place. Kits like this also quickly show the problem with tracking down the malicious user behind an compromise or attempt, it is rare for attacks to be launched from systems that can easily be traced back to the malicious user.

A quick Google search confirms that this kit (and user) has been seen in the wild attacking other systems, this posting on the Shell Person blog writes up the aftermath after a production system was compromised by the same kit.

–Andrew Waite

I’ve been running Kippo for nearly two weeks now (decided to live dangerously and go with SVN version) and have seen some interesting results.

Top 10 most common passwords attempted:

1. a (651)
2. 123456 (495)
3. password (331)
4. 12345 (302)
5. 123 (224)
6. 1234 (169)
7. 1 (139)
8. 12 (123)
9. root (105)
10. test (46)

Select count(password), password
from auth
where password <> ”
group by password
order by count(password) desc
limit 10;

Top 10 most common username attempted:

1. root (8510)
2. admin (144)
3. test (127)
4. oracle (96)
5. nagios (49)
6. mysql (47)
7. guest (43)
8. info (42)
9. user (41)
10. postgres (40)

select count(username), username
from auth
where username <>”
group by username
order by count(username)
desc limit 10;

Success ratio:

17065 attempts, 48 successful connections. (n.b. results skewed as account has purposefully poor choice of password)

select count(success),success
from auth
group by success
order by success;

Number of connections per unique IP:

1. 202.99.89.69 (5212)
2. 200.61.189.164 (1752)
3. 78.37.83.203 (1043)
4. 218.108.235.86 (848)
5. 195.14.50.8 (628)
6. 218.80.200.138 (271)
7. 58.222.200.226 (238)
8. 58.18.172.206 (158)
9. 119.188.7.174 (128)
10. 119.42.148.10 (113)

select count(ip), ip
from sessions
group by ip
order by count(ip) desc;

Number of attempts were relatively low IP address, in total 194 different source locations have attempted to access the server, with each typically only making 4 attemtps.

Packages:

Once exploited a number of attackers have proceeded to download various rootkits and utilities (thanks for these). Nothing too interesting yet, standard rootkit functionality, IRC clients and SSH scanners for further compromise. I still need to analyse some of these in more detail, so watch your RSS feeds for more to come.

One malicious user also attempted to create new user accounts on the server, if you have an account called ‘iony’ with a password of ‘ionyszaa’ then you may want to remove it…

If you’ve got a spare machine and public IP address, give Kippo a shot, setup is realitively easy; I’ve seen some interesting malicious user sessions and it turns out that some of those ’31337 haxxors’ that everyone fears really can’t type.

–Andrew Waite

As I started life as a Linux server admin I’m only too aware that many attackers see remote access functionality as a way into a system, and as SSH is the de facto standard for Linux access it is a prime target for attack. The stats collected by DShield give an indication to the extent of the problem.

As a result I’ve had the Kippo honeypot is something that I’ve had on my radar for a while. For a number of reasons I hadn’t found time to implement the system in a live environment, but a recent post on the Diatel blog suggested that installation may be quick and pain free.

Kippo is described by it’s author (Desaster) as:

Kippo is a medium interaction SSH honeypot designed to log brute force attacks and, most importantly, the entire shell interaction performed by the attacker.

Kippo is inspired, but not based on Kojoney.

Installation for me was painless, running a Debian system I downloaded the latest archive to disk, unpacked and installed the pyton-twisted package (I hadn’t read Mig5′s comment until after install so now need to go back and live on the bleeding edge…). I did hit a couple of problems when trying to start up the system, which is as simple as invoking ./start.sh

1. First, I was logged in as root when I first tried to start the system (not clever I know, was testing…). Kippo encounters an error when started by a root user. As Desaster rightly states, it’s not wise to run Kippo as a root user anyway and running as a regular user resolves the issue.
2. Second, when running as a normal user I got a ‘meaningful’ error of “Failed to load application: ‘NoneType’ object has no attribute ‘get’.” A quick piece of Google-fu lead me to this ticket, which explained Kippo was missing the file kippo.cfg, as explained copying kippo.cfg.dist to kippo.cfg correct the issue and produced a fully functional system.

There are a couple of key files that can be edited to change the feel of the system that is provided to malicious users:

• kippo.cfg contains runtime information including log location, fake hostname etc.
• kippo.tac contains an array ‘users’, which lists the username/password combination which the emulated SSH login will accept as ‘valid’.
• The honeyfs/ directory goes so far as to allow you to create a ‘real’ filesystem for the malicious user to interact with, potentially copying a live server’s filesystem to the directory to help camouflage the emulated system (after sensitive data is removed/sanitised obviously….). I haven’t tried this myself yet but is definitely on my to-do list.

From initial testing I’ve got high hopes for Kippo becoming a mainstay in my honeypot toolbox; the interaction session provided to a malicious user is reasonably convincing at first glance, and I particularly like the trick to keep users logged in after they think they’ve sent an ‘exit’ command to close the session, it could get some interesting results.

For post compromise analysis Kippo also provides some an interesting utility, utils/playlog.py. This allows you to replay a malicious terminal session in real-time, typos and all, to truely provide a feel for the malicious users interaction with the session. To help whet your apetite whilst I wait for someone to target my kippo installation, Kippo has a few demo’s of the playlog capabilities from compromise attempts. Get your demos here.

–Andrew Waite