Network sniffing in VMware ESXi

VMWare ESXi is perfect for a self contained lab, but as I’m used to having full access to a ‘real’ network there are a few things I miss not having control over for testing and other things. The biggest of these is a spanf port (or mirror port depending on your hardware). If you’re not familiar, the basic premise is to configure one (or more ports) to reproduce any traffic flowing through any port(s). This provides packet level access for debugging network problems, passing to an I[D/P]S, etc.
ESXi doesn’t provide this functionality, but does allow you to set a vSwitch to be ‘promiscuous’. Unfortunately this isn’t as controllable as a span/mirror port as (from the quick tests I’ve run) essentially turns the vSwitch into a vHub. Not a problem in my lab environment but you probably want to give it some serious thought before enabling in a production environment; do you really want every server on the network to be able to see all traffic on the (virtual) wire?
To make the change in ESXi you need host -> Configuration -> Networking and set the properties as shown below:

vSwitch properties
vSwitch properties

Once this change is made, any guests connected to the vSwitch can all see any of the network traffic on that switch.
For testing you can build a quick lab scenario with 3 live boot BackTrack systems. Each machine has a different role; server, client and ‘sniffer’. The sniffing machine is now able to view direct communication between the other two systems. Using wireshark’s Follow TCP stream functionality shows the conversation:
sniffed TCP stream
sniffed TCP stream

–Andrew Waite

SSH Tunnelling Example

Towards the end of last year I spent a few hours trialling SSH tunnels, I knew how the process worked but hadn’t had much cause to use it in anger; so my lab got some use instead, and a post was written covering the basics; SSH port forwarding 101.
Since I now know how to quickly and successfully implement a tunnel, it turns out that I previously had plenty of cause to use tunnels in the past, I just didn’t know SSH tunnels were the right tool for the job. A couple of recent conversations has made me realise others don’t always know the flexibility of tunnels either so I wanted to try and describe a common scenario to highlight the usefulness of tunnels.

Above is a fairly common setup. You’ve got an internal resource (for example an intranet wiki for documentation), this is in turn protected by a firewall that only allows access from trusted location. Under normal circumstances all staff can access the resource without problems, and any malicious sources (human or automated) can’t access the service.
This works well, until someone needs access and they aren’t at one of the trusted locations (we’re assuming this is an unusual problem and remote access solutions aren’t in place). In a lot of environments SSH is a ‘trusted’ system management solution and is world accessible (and hopefully secured well enough to keep the barbarians from the door, but that’s for different posts).
SSH tunnels (but you guessed that). Tunnel the server’s HTTP (or whatever) service back to your local system, and then connect locally. Using the syntax I discussed previously, from a ‘nix shell you can use this command:

ssh -L 8000:

This makes an SSH connection to the server (, tunnelling the local HTTP service running on port 80 ( and binds it to your machines TCP 8000 port. Now you can connect to the service by typing into a browser, thus traversing the firewall source IP restrictions.
If you’re living in a Windows world, then the PuTTY equivalent configuration will be:

Next time you’re sat in the coffee shop on a Sunday morning, and the boss rings with an ’emergency’; are you sure that you can’t access the resources you need from where you are? If you can, that coffee (and extra slice of cake) just became expense-able 😉
–Andrew Waite

A Northern Geek's trip South

Tuesday started fine, train down the capital a chance to meet up with the London work team. So far so good, until a colleague suggested a ‘quiet’ drink after work. Ended up not being too quiet after all.
With Wednesday starting off with ‘why?….’, I found some energy and headed for Security BSides London. As I’d already reconnoitered the location on Tuesday getting to the location was a breeze, only to find the door locked. Javvad Malik to the rescue, arrived at same time and managed to call one of the organisers to let us in. After brief introductions all round I met Soraya Iggy in person for the first time, absolutely nothing like I was expecting but great in every way. After receiving goodie bag (and getting repeated grief from Iggy to change into con shirt) I enjoyed some good geek chat whilst watching the venue fill.
After the official opening of the event, I headed upstairs to track two, which started with Aaron Finnon discussing DNS tunneling techniques. I was looking forward to this talk as I’d got half of the information over a drink after Aaron gave his famous SSL talk when OWASP Leeds travelled to Newcastle. My main takeaway from the discussion was that with the use of some relatively simple tools it can be relatively simple to bypass most captive wireless portals if they aren’t sufficiently tying down egress traffic. First on my to-do list of ‘I wonder what happens if you try this in my environment?’.
Second session was David Rook and Chris Wysopal, discussing ‘Jedi Mind’ tricks for building security programs. Having watched recordings of both presents from other events I was looking forward to getting the live  experience, and neither disappointed. The presentation was great and I took a lot away for how to both discuss security issues with non-infosec people, and how to talk about the problems in business terms to get buy-in to effect real change in an organisation. I was somewhat surprised, as this started a trend of the event with my favourite presentations being non-technical in nature.
Third session was one that I’d heard a few people dismiss before the event as being a bit lame. I’d already picked it out as my preferred session for this timeslot (it was a tough call, other track was Justin Clark discuss web app attacks, but the end would have over run with the next talk I wanted to see). I’m glad I didn’t let the naysayers dissuade me, Ellen Moar and Colin McLean did a great job demonstrating just how simple it is for anyone with basic computer knowledge (script kiddie) to cut and paste their way past defensive countermeasures (AV). Content wasn’t anything groundbreaking (which is why I think some weren’t keen), but I think it’s the first time I’ve actually seen someone ‘prove’ what we all accept as gospel. Scary stuff.
Final session of the morning was Xavier Mertens discuss logging and event management. Not the most thrilling of topics I’ll admit, but it’s something that so few organisations seem to get right I was interested to find out if there were any ‘better’ ways that could improve the process. Not only are there apparently better ways, but apparently there are also free better ways, so I’m going to talk a closer look at OSSEC.
After lunch Steve Lord provided an ‘interesting’ look into different types/levels of pentester and what it means to be in the industry. The talk received a lot of laughs, but in hindsight I wish I seen a talk with more technical content. For me, bsides was for education and networking, I’ll leave comedy to the comedians.
Next talk was better, Wicked Clown, expanding on his Brucon Lightening talk showing how to break out of a restricted RDP session. This was a great presentation, and was another attack to add to my ‘what if’ to-do list. More importantly he also provided a simple fix to prevent the attack vector, considering it’s a single checkbox, and the workaround breaks how most would ‘expect’ the service to behave I’ll echo his confusion as to why Microsoft don’t have the checkbox ticked by default. Perhaps secure out of the box is too much to ask?
David Rook took to the stage again, this time alone and discussing static code analysis with Agnitio. I’d taken a look at Agnitio since David released it, but as I’m not much of a dev (see the utilities I release for proof…) haven’t been able to try it in anger. If you’re interested, the talk slides are available on the Security Ninja blog. If the tool can reach the stated end of it’s road map of being the ‘Burp Suite of static analysis’ then it should be a fantastic tool.
Next talk I saw was Manuel demo (reverse~re)engineering of DRM within Android applications. I found the talk fascinating, mostly by how quick Manuel was able to put the pieces of the puzzle together and bypass the protections put in place to do exactly what he was attempting. Whilst the presentation was good, it was one of those where you felt your comparative IQ drop as you see black magic being wielded at the keyboard before your eyes.
The event finished with ‘Security YMCA’, words cannot describe this ‘experience’ so I won’t attempt to, and leave you with this YouTube video. (WARNING: once seen, cannot be unseen). Unfortunately trying to hide at the back didn’t help in the end, so I must apologise to Ellen as I managed to say thanks for an interesting talk by subjecting her to my ‘singing’ attempts. To ensure the the guilty aren’t protected, those at the front assaulting your sensors are:

During most cons I’m usually sat in the office getting snippets from Twitter, or reading a blow by blow account as Chris John Riley posts during every session. I always wondered how he found the time to get it all done, after seeing it in person I’ve still got no clue…
Thursday provided no let-up, with Infosec Europe next on the agenda. Feeling lively I left hotel in Farringdon early just after seven, and proceed to walk to Earls Court. Yes, for those that know London this is around 5 miles as the crow flies, it’s even longer if you keep taking the wrong turn as you’re too busy admiring the sights of London (managed to cover Oxford Street, Regent Street, Piccardilly,  Buckingham Palace and Hyde Park on the way. Unsurprisingly I was slightly tired when arriving. As this was my first visit to InfoSec I was surprised by the size, but with bag for freebies in hand, I hit the stands to talk to vendors. Without a boring stand by stand account it was good to meet some people in person for the first time, and to get some hands on demos of products I hadn’t yet seen in person. Some of the marketing was in high spirits however, I got my favourite quote from a vendor who shall remain nameless (I’m nice like that) stating:
we don’t have a solution for the iPhone, as it’s a secure platform why bother?
The offering had looked promising until then; after that comment? Thanks, but I’ll pass…
I did take advantage of the Syngress stand’s discounts and filled out my to read pile. (Ninja Hacking, Seven Deadliest USB Attacks, Cybercrime and Espionage and Digital Triage Forensics). Although I didn’t take as much advantage as the gentleman in front of me in the queue, who literally bought one copy of every book on display; totalling over £450.
The trip ended with a bottle of lager sat outside the British Museum in some glorious weather (which unfortunately didn’t follow me back home). I don’t want to name names, as undoubtably I’ll forget someone, but my most common phrase this week has been ‘Good to finally be able to put a face to the twitter handle’, it really was good to meet people I’ve spoken to online for a while, and to make some new contacts as well. Looking forward to the next time we’re able to meet up.
–Andrew Waite
P.S. sorry for formatting towards the end, seems to be a strange limit with the number of paragraphs wordpress will accept per post. Will try to correct in due course.

John the Ripper 101

For those that don’t already know, John the Ripper is:

a fast password cracker, currently available for many flavors of Unix, Windows, DOS, BeOS, and OpenVMS. Its primary purpose is to detect weak Unix passwords. Besides several crypt(3) password hash types most commonly found on various Unix systems, supported out of the box are Windows LM hashes, plus many more with contributed patches.

John is commonly available for most ‘nix systems under their package management systems. However, I’d strongly recommend compiling from source to ensure John takes full advantage of more powerful hardware capabilities than the ‘generic’ build. Compiling from source is straightforward, and I’m yet to encounter any difficulties. The additional time spent with manual compilation definitely pays for itself in the long run, read the INSTALL file for instructions.
To get a feel for the available performance john includes ‘–test’ functionality. My system (iMac i7 w/8GB ram), manually compiled with system specific functions achieved:

enchmarking: Traditional DES [128/128 BS SSE2-16]… DONE
Many salts: 3370K c/s real, 3370K c/s virtual
Only one salt: 2840K c/s real, 2840K c/s virtual
Benchmarking: BSDI DES (x725) [128/128 BS SSE2-16]… DONE
Many salts: 108990 c/s real, 110080 c/s virtual
Only one salt: 107264 c/s real, 107264 c/s virtual

Using the same system and john compiled with a ‘generic’ system I achieved less than 50% performance:

Benchmarking: Traditional DES [64/64 BS]… DONE
Many salts: 1613K c/s real, 1613K c/s virtual
Only one salt: 1476K c/s real, 1487K c/s virtual
Benchmarking: BSDI DES (x725) [64/64 BS]… DONE
Many salts: 52800 c/s real, 52377 c/s virtual
Only one salt: 51961 c/s real, 51961 c/s virtual

Unfortunately, john doesn’t currently take advantage of multi-core systems and only uses a single core (see below). There are methods for sharing the processing load between multiple instances of john, or even across john instances running on multiple systems, but that’s a topic for another day.
CPU utilisation under john
Before you can crack any passwords, you first need to ‘acquire’ the hashes to be cracked. There are many options for this including the pwdump family (I’d recommend fgdump), the Cain and Abel tools and many others. For testing purposes I extracted the Windows LM hashes using Metasploit’ Meterpreter’s hashdump functionality:

meterpreter > hashdump

John comes packaged with a default password list, containing a little over 3000 entries, and several methods for generating hybrid guesses including appending numbers to entry or performing ‘l33t’ replacements. Rules for hybrid attacks can found (and edited) in the john.conf file. Even without tweaking, john is still effective  at cracking hashes as the below shows.
Cracking ‘andrew’:

Wintermute:john-testing awaite$ time /Applications/john-1.7.6-jumbo-12/run/john 192-168-1-127_Andrew.dump
Loaded 2 password hashes with no different salts (LM DES [128/128 BS SSE2-16])
ITY              (andrew:2)
INFOSAN          (andrew:1)
guesses: 2  time: 0:00:09:45 (3)  c/s: 19803K  trying: INFODTP – INFOSS9
real 9m45.359s
user 9m45.193s
sys 0m0.125s

Cracking ‘Barney’:

Wintermute:john-testing awaite$ time /Applications/john-1.7.6-jumbo-12/run/john 192-168-1-127_Barney.dump
Loaded 2 password hashes with no different salts (LM DES [128/128 BS SSE2-16])
R                (Barney:2)
DINOSAU          (Barney:1)
guesses: 2  time: 0:00:00:01 (3)  c/s: 9547K  trying: DINOSH8 – DINOVYS
real 0m1.685s
user 0m1.642s
sys 0m0.019s

Cracking ‘Charlie’:

Wintermute:john-testing awaite$ time /Applications/john-1.7.6-jumbo-12/run/john 192-168-1-127_Charlie.dump
Loaded 1 password hash (LM DES [128/128 BS SSE2-16])
CH4LK01          (Charlie)
guesses: 1  time: 0:00:28:38 (3)  c/s: 20126K  trying: CH4LB5# – CH4LCGN
real 28m38.011s
user 28m37.645s
sys 0m0.301s

Cracked hashes are stored in the john.pot file, preventing john from wasting CPU cycles cracking a hash that has already been cracked. Previously cracked passwords can be found using –show:

Wintermute:run awaite$ ./john –show /Users/awaite/Documents/john-testing/192-168-1-127.dump
5 password hashes cracked, 12 left

John has a lot of additional functionality; go install, tweak and crack (only with permission, obviously).
–Andrew Waite

Daily Paranoia

As a security guy I find my paranoia levels are slightly high than most, a little something inside me picks up on things that general users miss that indicate that something isn’t right. This morning was no exception….
After acquiring coffee, morning inbox was opened which presented the following:
Nagios Email Alerts
These are email alerts sent by the monitoring system Nagios, running within InfoSanity’ networks. The NRPE-check_users parameters have been modified from the Debian defaults to be more paranoid; triggering a warning alert with a single user logged into the server, and go critical if more than one. So; from this, someone is logged into the web server, and it isn’t me.
Feeling the onrush of panic, I log into server and chuck commands at a shell to see who is violating my system. Last, who and /var/log/auth all showed that no one had accessed the server at the time of my alerts. Everything good? Not if your paranoid, starting to smell a rootkit causing the system to lie to me.
There are a couple of anti-rootkit utilities that have served me well in the past, chkrootkit and rkhunter. Wondering which one to run? As we’ve already discovered I’m paranoid, answer was both. And both gave a clean bill of health to the system. Now I’m really getting paranoid.
Wanting to see if I’ve missed a trend, or if the issue is present with other servers in the environment I log into Nagios interface for a more detailed look, and find the answer:

Nagios webUI notifications
Nagios webUI notifications

Anyone spotted it? Yep, the service alert went critical when the check’s socket timed out (network issue), and then dropped to warning showing a single user when the connectivity returned; which was correct, I’d forgotten to log out of the console on my last VMWare connection. Stepping down from high alert…..
Morale of the story?
Don’t respond to system alerts before finishing first coffee of the morning. The events weren’t a total loss though, besides getting my heart-rate up and blood flowing, it was a good(ish) refresher for incident response (can’t beat the adrenaline rush of responding to an incident, real or imagined) and rkhunter uncovered a potential weakness in the server configuration which has since been corrected (no, I’m not telling you what).
–Andrew Waite
Oh yeah, had I just opened the email, could have avoid the whole situation:Nagios Email Details

Tales from the Kippo Logs: 'hackers' with poor opsec…

Running through my morning routine of catching up with email, twitter, etc. I came across this post showing Sequal7’s first hits on a Kippo installation. In addition to making amusing reading, it gave me a nudge to check back on the InfoSanity Kippo sensor. Initially I was looking to see if the same individual had stumbled across my sensor; they hadn’t at least not from the information I have available.
However, when checking if the newly changed password matched anything in my database I found a new ‘realm’ entry in the ‘input’ table, ‘ssh’. This got me curious, one of my ‘guests’ decided to hit another system whilst logged in to mine; ssh’ing to another IP, accepting the certificate and providing the password to said system (I’m assuming).
It should also be worth noting that by this point the user had already failed to notice that input hadn’t returned to their own system. After (attempting) to change my sensor’s root password (to ‘yahoo’, really) the user exited, but was caught out by Kippo’s trick of clearing the terminal and changing prompt to ‘localhost’, in total I viewed a ~20 minute terminal session of the user trying to compromise other systems, and failing in the same manner.
My assumption is that the user was running through a list of vulnerable systems identified by SSH scanners similar to the kit I wrote about earlier (it wasn’t the same gosh.tgz kit, but first glance shows similar functionality). From this I feel it’s safe to assume that the systems connected to in the logs available are those of other (probably 0wned) systems, rather than anything connected to my guest. Likewise it is probable that the source connection is also a compromised third party rather than belonging to by guest.
e.tgz :
For the curious, archive contents:


Whilst investigating the individual exploits and files; I came across this post, indicating ‘my’ archive is a known fire and forget post exploit kit. Here be Skiddies…
–Andrew Waite

2010: A Review

Originally I wasn’t planning on reviewing this year, didn’t think that much had happened, but during some end of year house keeping came across the InfoSanity review of 2009 and wanted to keep the trend going. In keeping with last years review. I’ll start with the non-technical (again on pain of death 😉 ); wedding plans going strong so I should be a married man early 2011.
Back to the technical: Despite my initial concerns; the site, blog and research environment are still here and still growing. To all those who’ve read, contributed and (most importantly) told me I’m wrong over the past year (you know who you are), thank you.
Lab Environment(s): To complement the home lab established in 2009, 2010 saw the introduction of a hosted virtual lab which has provided the opportunity to easily try new (and old) technologies in the real world. As part of this InfoSanity has setup (and in some cases also removed) instances of honeyd, Dionaea, Amun and Kippo. These systems have also resulted in some new utilities being developed and released as I worked through various findings.
Whilst standing on the shoulders of giants (thanks Markus), some of the findings from the InfoSanity environment are now available publically. Although I really must complete both automating the process and including findings from other systems, 2011’s to-do list is already growing.
Public Speaking: For some reason I’ve still been asked to talk in public about topics I find fascinating; so thanks to the Disaster Protocol team for having me on the show. I felt it was a great discussion of honeypot technologies and infosec in general, and from feedback I’ve had others seem to agree.
Trying new things: Whilst trying to grow and mature over the year InfoSanity tried a few different themes and topics, some worked, like basic ssh hardening guidelines (potentially more to come in new year) and some didn’t, like the ‘Infosec Triads’ series. But if you don’t stretch yourself you’ll stop learning, so expect more posts that don’t quite work in 2011.
Friends, contact and groups: As with last year, the best part of 2010 has definitely been the people I’ve either continued talking to and/or working with and those I’ve met for the first time. 2010 saw a growth spurt in local and online groups I’ve been involved in, including the start of NEBytes, ToonCon and the Kippo User Group. There are also a huge number of awesome groups which I don’t get as much time to get involved with as I’d like; EH-Net, Group51, DissectingTheHack, Exotic Liability…the list goes on.
2011?: Who knows? Every time I try to make plans or predictions the Sky Fairies and Flying Spaghetti Monsters mock me, so I won’t try to make any. But whatever the outcome, I’m not expecting a letup in the pace, and can already see some exciting new opportunities on the horizon.
Another decade down, and a new year of opportunity ahead. See you all in 2011.
–Andrew Waite

Dionaea with p0f

Working my way through the compilation instructions from Dionaea whilst building up my latest sensor I was reminded of some optional functionality that I’d always intended to implement, but never found the time. First on my list was p0f (that’s a zero).
From p0f’s homepage:

P0f v2 is a versatile passive OS fingerprinting tool. P0f can identify the operating system on:
– machines that connect to your box (SYN mode),
– machines you connect to (SYN+ACK mode),
– machine you cannot connect to (RST+ mode),
– machines whose communications you can observe.
P0f can also do many other tricks, and can detect or measure the following:
– firewall presence, NAT use (useful for policy enforcement),
– existence of a load balancer setup,
– the distance to the remote system and its uptime,
– other guy’s network hookup (DSL, OC3, avian carriers) and his ISP.

Setting p0f up on the sensor should have been straightforward;

  • Install with: apt-get install p0f
  • Run p0f as suggested in dionaea.conf: sudo p0f -i any -u root -Q /tmp/p0f.sock -q -l
  • And edit dionaea.conf’s ihandler section to enable p0f

This mostly worked, watching the p0f output it was correctly (I’m assuming) providing stats about connecting systems. The problem was that p0f info wasn’t getting saved into Dionaea’s logsqlite database, dionaea-error.log was reporting the below error with each connection:

[04122010 13:48:44] connection connection.c:827-warning: Could not connect un:///tmp/p0f.sock:0 (Permission denied)

Which seemed odd, /tmp/p0f.sock was showing as globally readable. Re-reading the Dionaea compilation instructions I noticed a comment about p0f struggling with IPv6 so has problems it Dionaea is listening on ::, which mine was. Problem solved, I edited dionaea.conf so that Listen mode was set to “manual”, and provided the interface/IP details of my network connection. Only this didn’t solve my problem…
Head thoroughly hurting I swallowed my ego and asked for assistance on the mailing list, and promptly (thanks again Ryan) got a reply that provided a functional workaround.
So, why go to the effort? Main purpose behind running honeypot systems (for me) is to get a better idea understanding of what threats are actively targeting systems in the wild. At first glance the information provided by p0f can quickly help evaluate the attacking system; what OS? what connection type? is it local?
With the limited (few hours) of data I’ve already collected heres a sample of the info you can gather:
last 5 connections:

p0f connection p0f_genre p0f_link p0f_detail p0f_uptime p0f_tos p0f_dist p0f_nat p0f_fw
328 822 Windows IPv6/IPIP 2000 SP4, XP SP1+ -1 17 0 0
327 821 Windows IPv6/IPIP 2000 SP4, XP SP1+ -1 17 0 0
326 820 Windows 2000 SP4, XP SP1+ -1 14 0 0
325 819 Windows 2000 SP4, XP SP1+ -1 14 0 0
324 818 Linux pppoe (DSL) 2.4-2.6 5 13 0 0

SQL Query:

select *
from p0fs
order by
connection desc
limit 5;

Breakdown by OS

count OS
324 Windows
17 Linux

SQL Query:

select count(p0f_genre) as count, p0f_genre as OS
from p0fs
group by p0f_genre
order by count(p0f_genre) desc;

Umm, so most systems spreading malware are (likely) infected Windows systems. No great surprise there…
Connectivity Types

Count Connectivity
153 IPv6/IPIP
149 ethernet/modem
40 pppoe (DSL)
12 (Google/AOL)
5 GPRS, T1, FreeS/WAN
3 PIX, SMC, sometimes wireless
2 sometimes DSL (2)
2 sometimes DSL (4)

SQL Query:

select count(p0f_link), p0f_link
from p0fs
group by p0f_link
order by count(p0f_link) desc;

Unfortunately the the information provided by p0f isn’t an exact science, and as devices and systems are constantly changing it’s only going to be as accurate as it’s latest signatures/fingerprints. But setup is fairly quick, and the information and insight provided fairly interesting. So why not give it a go?
–Andrew Waite

Dionaea in the key of U(buntu)

Markus keeps adding great features and functionality to Dionaea, when I read the post introducing a new web interface carniwwwhore I couldn’t help thinking I’d got lucky timing, start of a week’s vacation and no real plan for what to do with it. I’ve struggled previously with some of my Dionaea setups, largely because my system was running Debian, whilst Dionaea was built under Ubuntu; doesn’t cause too many problems, just a bit of google-fu, headscratching and stupidity that could have been avoided.
From this background I looked through the carniwwwhore pre-reqs with dread, plenty of version requirements that weren’t upto date with my Debian setup; so it’s time to bite the bullet and build a fresh system with Ubuntu. Unlike some of my previous setups, installation/compilation worked flawless, working on the same distro as the lead dev definitely makes life easier. If you’re looking for a fresh Dionaea installation, go with Ubuntu, you won’t regret it.
–Andrew Waite
(oh, and carniwwwhore? Vacation got the better of me so it’s added to the to-do list; watch this space…)