Daily Paranoia

As a security guy I find my paranoia levels are slightly high than most, a little something inside me picks up on things that general users miss that indicate that something isn’t right. This morning was no exception….
After acquiring coffee, morning inbox was opened which presented the following:
Nagios Email Alerts
These are email alerts sent by the monitoring system Nagios, running within InfoSanity’ networks. The NRPE-check_users parameters have been modified from the Debian defaults to be more paranoid; triggering a warning alert with a single user logged into the server, and go critical if more than one. So; from this, someone is logged into the web server, and it isn’t me.
Feeling the onrush of panic, I log into server and chuck commands at a shell to see who is violating my system. Last, who and /var/log/auth all showed that no one had accessed the server at the time of my alerts. Everything good? Not if your paranoid, starting to smell a rootkit causing the system to lie to me.
There are a couple of anti-rootkit utilities that have served me well in the past, chkrootkit and rkhunter. Wondering which one to run? As we’ve already discovered I’m paranoid, answer was both. And both gave a clean bill of health to the system. Now I’m really getting paranoid.
Wanting to see if I’ve missed a trend, or if the issue is present with other servers in the environment I log into Nagios interface for a more detailed look, and find the answer:

Nagios webUI notifications
Nagios webUI notifications

Anyone spotted it? Yep, the service alert went critical when the check’s socket timed out (network issue), and then dropped to warning showing a single user when the connectivity returned; which was correct, I’d forgotten to log out of the console on my last VMWare connection. Stepping down from high alert…..
Morale of the story?
Don’t respond to system alerts before finishing first coffee of the morning. The events weren’t a total loss though, besides getting my heart-rate up and blood flowing, it was a good(ish) refresher for incident response (can’t beat the adrenaline rush of responding to an incident, real or imagined) and rkhunter uncovered a potential weakness in the server configuration which has since been corrected (no, I’m not telling you what).
–Andrew Waite
Oh yeah, had I just opened the email, could have avoid the whole situation:Nagios Email Details

Tales from the Kippo Logs: 'hackers' with poor opsec…

Running through my morning routine of catching up with email, twitter, etc. I came across this post showing Sequal7’s first hits on a Kippo installation. In addition to making amusing reading, it gave me a nudge to check back on the InfoSanity Kippo sensor. Initially I was looking to see if the same individual had stumbled across my sensor; they hadn’t at least not from the information I have available.
However, when checking if the newly changed password matched anything in my database I found a new ‘realm’ entry in the ‘input’ table, ‘ssh’. This got me curious, one of my ‘guests’ decided to hit another system whilst logged in to mine; ssh’ing to another IP, accepting the certificate and providing the password to said system (I’m assuming).
It should also be worth noting that by this point the user had already failed to notice that input hadn’t returned to their own system. After (attempting) to change my sensor’s root password (to ‘yahoo’, really) the user exited, but was caught out by Kippo’s trick of clearing the terminal and changing prompt to ‘localhost’, in total I viewed a ~20 minute terminal session of the user trying to compromise other systems, and failing in the same manner.
My assumption is that the user was running through a list of vulnerable systems identified by SSH scanners similar to the kit I wrote about earlier (it wasn’t the same gosh.tgz kit, but first glance shows similar functionality). From this I feel it’s safe to assume that the systems connected to in the logs available are those of other (probably 0wned) systems, rather than anything connected to my guest. Likewise it is probable that the source connection is also a compromised third party rather than belonging to by guest.
e.tgz :
For the curious, archive contents:

e/
e/exp_moosecox.c
e/funny.jpg
e/exp_powerglove.so
e/exp_ingom0wnar.c
e/pwnkernel.c
e/exp_cheddarbay.so
e/exp_powerglove.c
e/exp_ingom0wnar.so
e/exp_therebel.so
e/run_nonnull_exploits.sh
e/exp_paokara.so
e/exp_framework.h
e/exp_moosecox.so
e/exp_wunderbar.c
e/exp_cheddarbay.c
e/run_null_exploits.sh
e/exploit.c
e/exp_therebel.c
e/exp_vmware.so
e/exp_vmware.c
e/exp_wunderbar.so
e/exploit
e/exp_paokara.c

Whilst investigating the individual exploits and files; I came across this post, indicating ‘my’ archive is a known fire and forget post exploit kit. Here be Skiddies…
–Andrew Waite

2010: A Review

Originally I wasn’t planning on reviewing this year, didn’t think that much had happened, but during some end of year house keeping came across the InfoSanity review of 2009 and wanted to keep the trend going. In keeping with last years review. I’ll start with the non-technical (again on pain of death 😉 ); wedding plans going strong so I should be a married man early 2011.
Back to the technical: Despite my initial concerns; the site, blog and research environment are still here and still growing. To all those who’ve read, contributed and (most importantly) told me I’m wrong over the past year (you know who you are), thank you.
Lab Environment(s): To complement the home lab established in 2009, 2010 saw the introduction of a hosted virtual lab which has provided the opportunity to easily try new (and old) technologies in the real world. As part of this InfoSanity has setup (and in some cases also removed) instances of honeyd, Dionaea, Amun and Kippo. These systems have also resulted in some new utilities being developed and released as I worked through various findings.
Whilst standing on the shoulders of giants (thanks Markus), some of the findings from the InfoSanity environment are now available publically. Although I really must complete both automating the process and including findings from other systems, 2011’s to-do list is already growing.
Public Speaking: For some reason I’ve still been asked to talk in public about topics I find fascinating; so thanks to the Disaster Protocol team for having me on the show. I felt it was a great discussion of honeypot technologies and infosec in general, and from feedback I’ve had others seem to agree.
Trying new things: Whilst trying to grow and mature over the year InfoSanity tried a few different themes and topics, some worked, like basic ssh hardening guidelines (potentially more to come in new year) and some didn’t, like the ‘Infosec Triads’ series. But if you don’t stretch yourself you’ll stop learning, so expect more posts that don’t quite work in 2011.
Friends, contact and groups: As with last year, the best part of 2010 has definitely been the people I’ve either continued talking to and/or working with and those I’ve met for the first time. 2010 saw a growth spurt in local and online groups I’ve been involved in, including the start of NEBytes, ToonCon and the Kippo User Group. There are also a huge number of awesome groups which I don’t get as much time to get involved with as I’d like; EH-Net, Group51, DissectingTheHack, Exotic Liability…the list goes on.
2011?: Who knows? Every time I try to make plans or predictions the Sky Fairies and Flying Spaghetti Monsters mock me, so I won’t try to make any. But whatever the outcome, I’m not expecting a letup in the pace, and can already see some exciting new opportunities on the horizon.
Another decade down, and a new year of opportunity ahead. See you all in 2011.
–Andrew Waite

Dionaea with p0f

Working my way through the compilation instructions from Dionaea whilst building up my latest sensor I was reminded of some optional functionality that I’d always intended to implement, but never found the time. First on my list was p0f (that’s a zero).
From p0f’s homepage:

P0f v2 is a versatile passive OS fingerprinting tool. P0f can identify the operating system on:
– machines that connect to your box (SYN mode),
– machines you connect to (SYN+ACK mode),
– machine you cannot connect to (RST+ mode),
– machines whose communications you can observe.
P0f can also do many other tricks, and can detect or measure the following:
– firewall presence, NAT use (useful for policy enforcement),
– existence of a load balancer setup,
– the distance to the remote system and its uptime,
– other guy’s network hookup (DSL, OC3, avian carriers) and his ISP.

Setting p0f up on the sensor should have been straightforward;

  • Install with: apt-get install p0f
  • Run p0f as suggested in dionaea.conf: sudo p0f -i any -u root -Q /tmp/p0f.sock -q -l
  • And edit dionaea.conf’s ihandler section to enable p0f

This mostly worked, watching the p0f output it was correctly (I’m assuming) providing stats about connecting systems. The problem was that p0f info wasn’t getting saved into Dionaea’s logsqlite database, dionaea-error.log was reporting the below error with each connection:

[04122010 13:48:44] connection connection.c:827-warning: Could not connect un:///tmp/p0f.sock:0 (Permission denied)

Which seemed odd, /tmp/p0f.sock was showing as globally readable. Re-reading the Dionaea compilation instructions I noticed a comment about p0f struggling with IPv6 so has problems it Dionaea is listening on ::, which mine was. Problem solved, I edited dionaea.conf so that Listen mode was set to “manual”, and provided the interface/IP details of my network connection. Only this didn’t solve my problem…
Head thoroughly hurting I swallowed my ego and asked for assistance on the mailing list, and promptly (thanks again Ryan) got a reply that provided a functional workaround.
So, why go to the effort? Main purpose behind running honeypot systems (for me) is to get a better idea understanding of what threats are actively targeting systems in the wild. At first glance the information provided by p0f can quickly help evaluate the attacking system; what OS? what connection type? is it local?
With the limited (few hours) of data I’ve already collected heres a sample of the info you can gather:
last 5 connections:

p0f connection p0f_genre p0f_link p0f_detail p0f_uptime p0f_tos p0f_dist p0f_nat p0f_fw
328 822 Windows IPv6/IPIP 2000 SP4, XP SP1+ -1 17 0 0
327 821 Windows IPv6/IPIP 2000 SP4, XP SP1+ -1 17 0 0
326 820 Windows 2000 SP4, XP SP1+ -1 14 0 0
325 819 Windows 2000 SP4, XP SP1+ -1 14 0 0
324 818 Linux pppoe (DSL) 2.4-2.6 5 13 0 0

SQL Query:

select *
from p0fs
order by
connection desc
limit 5;

Breakdown by OS

count OS
324 Windows
24
17 Linux

SQL Query:

select count(p0f_genre) as count, p0f_genre as OS
from p0fs
group by p0f_genre
order by count(p0f_genre) desc;

Umm, so most systems spreading malware are (likely) infected Windows systems. No great surprise there…
Connectivity Types

Count Connectivity
153 IPv6/IPIP
149 ethernet/modem
40 pppoe (DSL)
21
12 (Google/AOL)
5 GPRS, T1, FreeS/WAN
3 PIX, SMC, sometimes wireless
2 sometimes DSL (2)
2 sometimes DSL (4)

SQL Query:

select count(p0f_link), p0f_link
from p0fs
group by p0f_link
order by count(p0f_link) desc;

Summary
Unfortunately the the information provided by p0f isn’t an exact science, and as devices and systems are constantly changing it’s only going to be as accurate as it’s latest signatures/fingerprints. But setup is fairly quick, and the information and insight provided fairly interesting. So why not give it a go?
–Andrew Waite

Dionaea in the key of U(buntu)

Markus keeps adding great features and functionality to Dionaea, when I read the post introducing a new web interface carniwwwhore I couldn’t help thinking I’d got lucky timing, start of a week’s vacation and no real plan for what to do with it. I’ve struggled previously with some of my Dionaea setups, largely because my system was running Debian, whilst Dionaea was built under Ubuntu; doesn’t cause too many problems, just a bit of google-fu, headscratching and stupidity that could have been avoided.
From this background I looked through the carniwwwhore pre-reqs with dread, plenty of version requirements that weren’t upto date with my Debian setup; so it’s time to bite the bullet and build a fresh system with Ubuntu. Unlike some of my previous setups, installation/compilation worked flawless, working on the same distro as the lead dev definitely makes life easier. If you’re looking for a fresh Dionaea installation, go with Ubuntu, you won’t regret it.
–Andrew Waite
(oh, and carniwwwhore? Vacation got the better of me so it’s added to the to-do list; watch this space…)
 

Cold calling IT Support

I’m sure by now most people are aware of a new round of scams where victims are being called by a ‘support company’ suggesting that the victim’s computer has malware installed which they can fix. If you need it, this BBC article covers the basics. Well, I just got the call 😉
First up the caller seemed to be auto-dialling large volumes of numbers looking for someone to pick-up as the caller (male, poor line quality meant I missed the name given) was unprepared when I answered. The caller was clearly reading from a script, I may have over-played the ‘Sorry, I’m just a dumb user that knows nothing about computers card’ but despite telling him I was clueless and willing to accept everything he told me I was still present with a long winded argument for ‘if you don’t believe us this is how I’ll prove it’ speech.
Unfortunately I wasn’t able to through the full process as, despite telling my new friend otherwise, I wasn’t able to get to a Windows machine to work through the process. Only laptop to hand was my netbook running Ubuntu, and my landline isn’t mobile so I couldn’t head up stairs. (My landline never rings, everything I do is via mobile and only have landline for ADSL connection. I’m suspicious of all landline calls before I even pick up the phone.)
After ensuring I was looking at the system wallpaper, I was instructed to press the ‘key on bottom left of keyboard with four squares that looks like the Microsoft logo’ and with another finger press the ‘r’ key. This is where I was given ‘proof’ that my system was infected, using a ‘hidden’ command that will list all infections, what is the magic command? inf (for ‘infections’), which opens Windows Explorer in C:\Windows\inf, screenshot below shows the infections on my system. I’m guessing at this point, the every user may have just entered dummy mode.

At this point I lost the caller, whether a technical fault or he’d guessed something wasn’t right (I can’t act for toffee). I’m hoping that I’ll get a second bite at the cherry at some point; my missus took a similar call a few weeks back, having spent too long listening to my security rants she immediately spotted the scam, pointed out that I was a ‘security guy’ and hung up. Information that they clearly didn’t have when ringing back (could be more that one cold calling organisation).
Unfortunately, despite my usual laughing at people who fall for these scams I can see how those with less knowledge could fall for the premise. Computers and software regularly phone home to check for updates etc, using this information to identify infected systems would/could make sense, and from an end user perspective I struggled to tell the difference between the sorts of actions I was asked to take by my ‘friend’ than those I regularly instruct friends and family members when I’m trying to provide remote support.
Be safe and spread the word to those less knowledgeable about computers that this is an active scam. Bottom line is: no legit IT company will call you to fix a problem that you weren’t aware of.
–Andrew Waite
<UPDATE>
I just received a new call following the same theme but with a different vector. This time the call came from ‘Microsoft Service Department’, and with a different convincer; this time I was baby-stepped through to opening the Security log with each entry being ‘evidence’ of the malware infection that ‘at this very moment is damaging my computer and hard drive’. To be fair, in this case they ‘could’ be right.
Other differences indicate that either this is a different group from the first caller, or they’ve improved the call systems used to implement the scam. On lifting the receiver I was placed immediately to on hold music before speaking to my ‘MS representative’ a few moments later. I believe that automated dialling is illegal within the UK, but given the nature of the call I doubt they care much either way.
REMEMBER: Microsoft will NOT call home (or business) users to inform that you’ve got a malware infection
<UPDATE>

SSH Port Forwarding 101

I’ve tried messing around with SSH port forwarding in the past, but always struggled to get my head around what I was trying to connect to where, and ultimately didn’t result in anything useful. This time around I’ve put in some dedicated time to get to the bottom forwarding ports within SSH tunnels. And I’m glad I did, my with only a handful of connections the possibilities are making my head spin.
To help my head get around problems I encountered I found a number of resources helpful.

Example Scenario: A server (10.0.0.100) on an internal LAN has web services that need to be accessed, but has no remote access through the firewall. Luckily, there is remote access to an SSH server (10.0.0.200) sat on the same LAN.
From command-line:

ssh -L 8000:10.0.0.100:80 ssh-server.somedomain.tld

This connects to the machine at ssh-server.somedomain.tld and, once authenticated, forwards 10.0.0.100:80 to port 8000 on the local machine. Now accessing the remote services is as simple as pointing your browser to http://127.0.0.1:8000.
The same functionality can be achieved using configuration files. For example, edit ~/.ssh/config:

Host tunnel
HostName ssh-server.somedomain,tld
LocalForward 127.0.0.1:8000 10.0.0.100:80

To establish a connection with the configuration file in place simply run; ssh tunnel.
The port forward will remain active for as long as the SSH session is connected. If you don’t need to interact with the SSH session in addition to the forwarded port passing ssh the -fN flags will cause the session to be backgrounded once authentication can be established.
If you haven’t already, I suggest you investigate the possibilities within your own environment; and if an evil grin doesn’t spread across your face then you still don’t fully get it 😉
–Andrew

DokuWiki and the the BeerWare License

I’m trying to do a better job of documenting the systems that make up InfoSanity. To do that it’s everyone’s favourite, a wiki. In this case I went with DokuWiki, for no real reason other than it came up first on a Google search. Installation is simple on a Debian based system, just the usual: apt-get install dokuwiki. So far so good, but I’m wondering how long I can actually keep the documentation upto date…
At the same time I came across the the BeerWare license from my Twitter feed (sorry, can’t find original message, thanks to whoever pointed me in the right direction. The license was created by Poul-Henning Kamp, and does a good job of summing up my feelings toward the tools/scripts I’ve released the last few years:

/*
 * ----------------------------------------------------------------------------
 * "THE BEER-WARE LICENSE" (Revision 42):
 * <phk@FreeBSD.ORG> wrote this file. As long as you retain this notice you
 * can do whatever you want with this stuff. If we meet some day, and you think
 * this stuff is worth it, you can buy me a beer in return Poul-Henning Kamp
 * ----------------------------------------------------------------------------
 */

So, why not combine the two together? DokuWiki has the ability to apply one of several standard licenses by default, but unfortunately BeerWare isn’t one of them. To correct this obvious mistake edit /etc/dokuwiki/license.php to include:

$license[‘beerware’] = array(
‘name’ => ‘Beerware License’,
‘url’  => ‘http://en.wikipedia.org/wiki/Beerware’,
);

n.b. I struggled to find a better license location that just deals with the license, if anyone finds better let me know.
Now it’s just a matter of editing the main dokuwiki config file, /etc/dokuwiki/dokuwiki.php:

$conf[‘license’] = ‘beerware’;

And Voila!
Dokuwiki showing the BeerWare License–Andrew Waite

SSH hardening with Breakinguard

As proven by the logs generated by Kippo honeypot sensors have shown, attacks against SSH services are regularly seen in the wild. Even if you follow best practices for securing the service, the malicious scans will utilise resources available to your environment; CPU, bandwidth etc. In sufficient volume legitimate operation may be impacted as the server rejects failed login attempts.
This is where utilities like Breakinguard come into their own. Basically Breakinguard monitors log files for signs of malicious activity, and once a single source has triggered enough alerts blocks all connections from the source location. Other utilities (most notably fail2ban) perform the same activities, but I’m partial to Breakinguard due to it’s small size and simple configuration (and from knowing the author 😉 ).
Installation is straightforward, and for the most part automated. Once downloaded and extracted installation is handled by the configure script. On Debian based systems this will install the pre-requisite Perl modules and transfer the utilities components to the standard locations:

  • breakinguard script – /usr/local/sbin/breakinguard
  • config – /etc/breakinguard.conf
  • init script – /etc/init.d/breakinguard

Once installed you need to edit your configuration. The breakinguard.conf file is fairly self explanatory, I normally edit:

  • $alert_email -> set to the email address that you want to receive notifications of blocked attacks. On a publicly accessible system these alerts can be high volume, you may want to use a specific email account or at least setup some auto-move rules in your email client to avoid your inbox being spammed.
  • $number_of_attempts -> This specifies the number of malicious log entries need to be generated by a specific IP address before the source is blocked. Due to the timing of the Breakinguard route this isn’t always an exact science, the default of 5 does a good job of avoid false positives whilst still blocking an attack in it’s infancy.
  • $log_to_watch -> selects the logfile to monitor for signs of malicious activity. /var/log/auth.log is the obvious choice on a Debian based system.
  • @safe_ips -> This array allows you to specify a number of trusted networks that will not be blocked, regardless of the number of times the they trigger alerts in the logs. This is useful for insuring that you don’t get locked out of your own systems in the event your keyboard ‘breaks’. On higher end systems with hardware remote management systems (iLo, DRAC, etc.) or virtual systems that provide remote access to the ‘physical’ console I leave this list to local subnet only and use the alternative options to access the server if I do lock myself out.
  • $DEBUG -> set to 1 by default, this runs the utility in test mode without actually blocking malicious sources, perfect for testing configuration before going live. Once you’re good to set set $DEBUG to 0 and wait for the attacks to start. Example testing in debug mode is below:

Breakinguard - Debug Mode
Breakinguard - Debug Mode

Blocking and unblocking of malicious sources is handled via iptables. Once the $number_of_attempts limit is hit Breakinguard will run the $block_command (configurable in /etc/breakinguard.conf) which by default is ‘/sbin/iptables -I INPUT -s %s -j DROP’, with %s being replaced with the attacking IP. After a configurable timeout ($block_length), the $unblock_command removes the restriction.
You can see the IP addresses currently blocked as they are listed in /var/run/breakinguard/, alternatively listing the current iptables configuration will show sources currently being blocked, for example:
Breakinguard iptables
Breakinguard iptables

Download Breakinguard here
–Andrew Waite