Log analysis. It’s a thing. First reactions are probably going to be “Yuck,” “Make the new guy do it,” “Insert expletive here,” or the more common “Why? What broke now?”
Everyone in IT knows that logs are one of the single most important things in any environment. If something breaks, check the logs. Someone blaming the firewall for a site not opening? Check the logs. An end user clicked on something stupid? CHECK THE LOGS! Your logfiles are your evidence and your breadcrumbs back to the scene of the crime. (And the dunce cap goes to…..workstation D39R.)
Look at Log Analysis Differently
So to recap so far, everyone knows you should be logging. No one really likes reviewing them, even though they know that you should be. It’s dull, there’s too much noise, and you can always think of better things to do.
NOW…How do we fix this? By changing how we’re looking at log analysis. Almost all IT professionals are reactive. Either an end user has complained or there was an alarm in the SIEM or some other network device and now you’re digging for the badness. To steal a concept from John Strand, we need to be HUNTING for badness. When you’re trying to protect your environment, you need to be proactively looking for attack traffic and suspicious activity. Go out and find that C2 activity with your own eyes.
Don’t just rely on some Threat Intelligence engine to update a list somewhere and tell you about it. If you see an end user doing something stupid, tell them to knock it off! Maybe you’ll be able to stop them before they finally click on that bogus “Kim Kardashian is Pregnant Again” gossip site that actually links to a site with buffer overflow code and now you have a real problem.
SO! After many, MANY empty cases of Red Bull and a good couple of weeks of testing, this is how I made log analysis something that was actually useful:
1. Get a better logging solution
Get some sort of UTM device in your environment with some intelligent logging behind it. We like Fortinet. Good people. Good products. Fortiguard Labs rock and the stuff just works.
2. High level and prioritize
If you’re using a FortiAnalyzer, (which you should be. Just trust me on this), check FortiView for a high-level overview of what’s going on. Think of this as your 30,000-foot view of the Sarlacc pit that could potentially be your network. I usually spend 10-15 minutes on Top Threats and filter on just High and Critical to find the blatant heavy hitters. You can also look for bandwidth hogs here too. Massive traffic coming from a specific workstation may be evidence of a compromise…or Netflix.
3. UTM is your friend
High level done? Good. Let’s go a little farther down the rabbit hole. Go into Log View. Don’t click on Traffic just yet! The Fortigate’s logging capabilities auto-magically list UTM logs in their own section. Click on the Security drop down and you’ll find AntiVirus, Web Filter, App Control, and Intrusion Prevention. If there’s a UTM feature that you’re not seeing, that means that you either don’t have it on a policy (FIX THAT) or that you don’t have any traffic that’s hitting that sensor. (Congratulations! You’re a Unicorn!)
You can use these UTM specific logs to, once again, filter out the noise and focus on just that particular threat. Now you can easily tell the difference between traffic to Pandora and traffic to a Zeus botnet – call us if you find the latter! This is also a great place to find users that do dumb things. If you see the same source IP accessing sites in your web filter that can definitely be risky or at the very least unproductive, you have logs with their user connected to it and you can go slap them. Or at least tell their supervisor and he or she can slap them. Make sure HR approves actually slapping the individual in question.
4. Prepare for the hunt!
Now it’s time to get DEEP. Dive into the Traffic section. Noisy right? Let’s fix that. Filter out the “known goods.” This is going to be your search engines, Amazon, internal DNS queries, etc. Things that you know are normal and you don’t care about. Ain’t nobody got time for that noise!
Next, filter for blocks. Most of the time people don’t care when something gets blocked because the attack was stopped, but WHY is always important. Grab your network diagram and look for things that don’t make sense. Debbie in HR’s machine is trying to access the financial database over a non-standard port. Is this normal? My guess would be no. Time to check deeper on her specific traffic to see when this started and how she got popped (see above Kim Kardashian example). Check for broadcast traffic from host to host that doesn’t make sense. Same goes for massive amounts of ICMP traffic.
Speaking of non-standard ports, set your filter for known bad ports. Are you seeing 6667? Why are you using IRC? Look for 9150 or 9050 for TOR traffic. Do some research and create a hit list of known trouble ports and look for traffic that doesn’t make sense. Look for outbound DNS traffic from workstations. How many bytes are they sending out? There isn’t much in a DNS packet, so this amount should be small and that destination IP better be a known DNS server. If it’s not 8.8.8.8 or your ISP, you have a problem. Your workstations shouldn’t be calling for external DNS anyway.
Now would be a good time to check for VPN usage during non-standard business times. Grab the user name and verify that they were actually on at that time. Check what they’re accessing over the VPN. Just because they’re remote doesn’t mean the rules don’t apply to them. You are not a snowflake!
Use your resources to vet the IPs that your machines are calling out to. Obviously there are the basics such as ARIN, RIPE, etc, but you should also be using Virus Total as a regular resource to find more information on IPs that you don’t recognize. Sign up for the SANS mailing lists and check for the trouble IPs they send out to see if you’re calling out to them. Here’s a quick list of other reputation sites that I’ve used. Check them out and bookmark the ones you like:
- FortiGuard Center
- Google Transparency Report
- McAfee SiteAdvisor
- McAfee TrustedSource
- PhisTank
- BarracudaCentral
- IP Reputation Investigation
- Malware Domain List
- URL Blacklist
When checking the reputation sites, look for registrations. If the site has been registered within less than seven days or if it’s been compromised in less than 14, you may have a problem. Also look for sites registered in low security countries or known bad actors, (a streaming site registered in Belize is probably slap worthy).
5. Don’t have time for all of this? CALL INFOGRESSIVE!!!
As I’m sure you’ve been able to tell so far, I’m a special kind of OCD when it comes to log analysis. I took a lot of time to come up with these steps after MONTHS of digging through logs to look for information that mattered. It takes time, effort, and manpower to be able to effectively dig through your log sources in an intelligent and efficient manner. Depending on the size of your network, even if you spend an hour a day using these steps, you’ll still only see a small piece of the picture. The steps that I’ve given here are just the tip of the iceberg on what we train our MSSP engineers to look for. We have access to numerous resources from our various partners that let us look for IOCs before they become public knowledge, which keeps you from becoming a headline.
Shoot us an email. Make a phone call. Sometimes we’re watching for smoke signals. The point is we really want to help you protect your network. Let us know where you fit in our shield whether it’s logging analysis or another security service.
Written By: Derrick Masters, Security Engineer
