Daily Paranoia

As a security guy I find my paranoia levels are slightly high than most, a little something inside me picks up on things that general users miss that indicate that something isn’t right. This morning was no exception….
After acquiring coffee, morning inbox was opened which presented the following:

These are email alerts sent by the monitoring system Nagios, running within InfoSanity’ networks. The NRPE-check_users parameters have been modified from the Debian defaults to be more paranoid; triggering a warning alert with a single user logged into the server, and go critical if more than one. So; from this, someone is logged into the web server, and it isn’t me.
Feeling the onrush of panic, I log into server and chuck commands at a shell to see who is violating my system. Last, who and /var/log/auth all showed that no one had accessed the server at the time of my alerts. Everything good? Not if your paranoid, starting to smell a rootkit causing the system to lie to me.
There are a couple of anti-rootkit utilities that have served me well in the past, chkrootkit and rkhunter. Wondering which one to run? As we’ve already discovered I’m paranoid, answer was both. And both gave a clean bill of health to the system. Now I’m really getting paranoid.
Wanting to see if I’ve missed a trend, or if the issue is present with other servers in the environment I log into Nagios interface for a more detailed look, and find the answer:

Anyone spotted it? Yep, the service alert went critical when the check’s socket timed out (network issue), and then dropped to warning showing a single user when the connectivity returned; which was correct, I’d forgotten to log out of the console on my last VMWare connection. Stepping down from high alert…..
Morale of the story?
Don’t respond to system alerts before finishing first coffee of the morning. The events weren’t a total loss though, besides getting my heart-rate up and blood flowing, it was a good(ish) refresher for incident response (can’t beat the adrenaline rush of responding to an incident, real or imagined) and rkhunter uncovered a potential weakness in the server configuration which has since been corrected (no, I’m not telling you what).
–Andrew Waite
Oh yeah, had I just opened the email, could have avoid the whole situation: