Amazon: AWS Log Reporting

I was having a conversation with a customer the other day about Amazon AWS monitoring.  He had some interesting insight on his company’s overall migration to Amazon Web Service (AWS). He started with, “Here’s the core of it, cloud based deployment isn’t going away for us. Though there are no directives, by this time next year I’m expecting all but two of our public-facing applications to be sitting outside of our buildings.”  He even went on to say, “There’s a very real possibility that within 2-3 years we decommission half of our computer rooms”. Needless to say, any application that they use for network visibility and incident response needs to support Amazon AWS monitoring. I loved his input and because of it I decided to dig deeper.amazon aws log reporting and monitoring

Consequently, the need for visibility into cloud based conversation, like AWS, isn’t a new concept. Way back in 2014 we saw rumblings of this with CompTIA’s Fifth Annual Trends in Cloud Computing Study. CompTia went out and polled 400 decision making IT professionals, business professionals and executives from U.S. IT firms to obtain their data. They found out that among companies that have progressed from the experimental stage to a non-critical use stage, 28 percent rated the transition as requiring significant effort. One of the issues they were having was being able to see the traffic inside their cloud environment. In a January 2015 Cloud Computing Trends report we see that many business are moving from using the cloud for non-critical functions like marketing or CRM and are starting to rely on this technology for storing more valuable data.

Up till now we have been able to gain visibility into encrypted traffic  (e.g. Amazon AWS and Akamai) via Fully Qualified Domain Name (FQDN) for the sites that a user visited instead of seeing only encrypted or CDN traffic URLs.  As I suggested before, the volume of encrypted traffic has tripled in less than a year and the number of connections to “cloud based” computing sites like Amazon AWS has grown at nearly the same rate. With so many connections on port 443 headed to content delivery networks, technologies like traditional NetFlow have lost much of their ability to provide insight. This made me ask the question, “So how do we get Amazon AWS monitoring into Scrutinizer?

Amazon AWS monitoring and NetFlow

The good news is that this was a priority for our development team. I spent some time reading through the “new feature” tickets and talking to the developers. After a few Red Bulls I finally had a better idea of what they have been doing. Basically, from the AWS API engine they were able to construct flow elements similar to NetFlow v5 that could be used by Scrutinizer.

Here is a basic idea of the elements we are receiving.

Field Description
version The VPC flow logs version.
account-id The AWS account ID for the flow log.
interface-id The ID of the network interface for which the log stream applies.
srcaddr The source IP address. The IP address of the network interface is always its private IP address.
dstaddr The destination IP address. The IP address of the network interface is always its private IP address.
srcport The source port of the traffic.
dstport The destination port of the traffic.
protocol The IANA protocol number of the traffic. For more information, go to Assigned Internet Protocol Numbers.
packets The number of packets transferred during the capture window.
bytes The number of bytes transferred during the capture window.
start The time, in Unix seconds, of the start of the capture window.
end The time, in Unix seconds, of the end of the capture window.
action The action associated with the traffic:
• ACCEPT: The recorded traffic was permitted by the security groups or network ACLs.
• REJECT: The recorded traffic was not permitted by the security groups or network ACLs.
log-status The logging status of the flow log:
• OK: Data is logging normally to CloudWatch Logs.
• NODATA: There was no network traffic to or from the network interface during the capture window.
• SKIPDATA: Some flow log records were skipped during the capture window. This may be because of an internal capacity constraint, or an internal error.

Amazon AWS monitoring
Since these elements resemble v5 data we should be able to determine a conversations path, interface, port and the amount of data being sent inside that cloud environment. This also means that we will be able to provide conversation pair reports, applications via Well Known Ports, alerting and more.

In addition to that, this also means that any AWS instance won’t be an isolated device since you should be able to trace the conversation from the cloud to your internal network. At that point customers can uses features like IP to Username reporting. Most importantly, with this information they can identify who or what was responsible for that conversation.  As a result, this adds another layer of visibility to your Amazon AWS monitoring.

If you are interested in knowing more about how we can help you gain better visibility into your AWS traffic make sure to contact us for an evaluation.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s