VPC Flow Logs

Introduction to VPC Flow Logs

VPC Flow Logs is an agentless log collection mechanism that collects all the logs generated in your VPC. It enables the capturing and logging of network information about the traffic flowing into and out of the network interfaces in your VPC. When a flow log is created for a VPC, every network interface within the VPC will be monitored.

The data that has been logged can be stored in Amazon CloudWatch or Amazon S3 and can be retrieved and monitored for troubleshooting network connectivity issues. These logs will be collected and stored in a different path from your IP traffic such that the performance and throughput of the network are not affected.

VPC Flow Logs are useful when troubleshooting internet connectivity or network connectivity within your VPC or when you are trying to connect to your on-premise networks and also for security and compliance in your AWS Cloud.

When we say logs, it is nothing but the IP traffic flowing through that interface or subnet, or VPC. Using VPC Flow Logs, we can analyze what kind of traffic/packets are coming in and getting accepted, what type of traffic is going out or rejected, whether there are any malicious attempts to log into your resources, etc. All this information can be derived from the flow logs, and along with this, there is much more additional information that can be obtained from the VPC Flow Logs.

Kinds of VPC Flow Logs

VPC Flow Logs can be configured at the granular level, which includes flow logging at three separate levels. Flow Logging at the VPC level, Subnet level, and Elastic Interface Network (ENI) level can be achieved.

VPC Level: VPC Flow Logs can be enabled within a particular Virtual Private Cloud (VPC), and you can monitor all the operations and activities with your VPC.

Subnet Level: Each VPC network consists of one or more IP address range called subnets. VPC Flow Logs can be configured at the subnet level for a specific subnet to monitor and analyze all the operations/activities within your subnet.

When you enable flow logs at the VPC level, all the traffic from different subnets will also be logged. To avoid the logging of traffic from all the subnets, we can enable the VPC Flow Logs at the subnet level. You can use subnet-level logging to analyze the traffic from a specific subnet.

Elastic Network Interface (ENI): An elastic network interface (ENI) is a logical networking component in a VPC that represents a virtual network card that can be attached to the EC2 instances. ENIs help facilitates network connectivity for your instances. VPC Flow Logs can be configured at the ENI level to monitor. One can log and monitor the traffic from these network interfaces to foreshadow latency and other malicious activities.

Enabling VPC Flow Logs

You can enable VPC Flow Logs using three ways. One using the AWS Management Console, another using the AWS Command Line Interface (AWS CLI), and finally, by making calls to the EC2 API. It is easy to enable flow logs from AWS Console. For advanced users, AWS Command Line Interface (AWS CLI) and EC2 API calls also serve as options to enable flow logging. Note that the VPC Flow Logs feature is not enabled by default.

To create and enable VPC Flow Logs using the console, you need to:

• Navigate to the VPC Dashboard and select the VPC for which the flow logs have to be enabled.
• Click on Create Flow Log

• The Resource field will be the VPC ID of the selected VPC (this will be auto-populated).
• Select the Filter - the type of traffic to be captured, i.e., accepted traffic, rejected traffic, or all traffic.
• Select the IAM Role for the permissions required to allow the creation and provide write access to the AWS ColudWatch groups.
• Select the Destination Log Group where you want to publish and store the logs.
• Click on 'Create Flow Log.' The log group will be created around 15 minutes after the new Flow Log is created. After the creation, it is accessible via the CloudWatch Logs dashboard.

Publishing Flow Logs

After the creation of the flow logs, they can be published in Amazon CloudWatch or Amazon S3 for easy monitoring, storage, and effective analysis.

1. VPC Flow Logs to CloudWatch Logs: A flow log captures all the network traffic and publishes the flow log records to Amazon CloudWatch Logs. A CloudWatch log group is required to publish the flow log records to Amazon CloudWatch Logs.

   The below picture is an example where the VPC Flow Log captures the network traffic for the subnets and publishes the flow log records to Amazon CloudWatch Logs. Here logging is enabled at the subnet level, so the traffic for all the ENIs in the subnets is also captured.

   

2. VPC Flow Logs to S3 Bucket: A flow log captures all the network traffic and publishes the flow log records to an S3 Bucket. An existing S3 Bucket needs to be specified to publish the flow log records to Amazon S3.

   The below picture is an example where the VPC Flow Log captures the network traffic from the ENI of an EC2 instance and publishes the flow records to an Amazon S3 Bucket.

   

How to Publish VPC Flow Logs to CloudWatch?

Flow Logs for Network Interfaces and Publish to CloudWatch

1. Go to the EC2 console
2. On the left Navigation Pane, click on Network Interfaces and select the network interface (or) network interfaces for which you want logs.
3. Click on Actions and then "Create Flow Log."
4. In the filter you can specify the type of IP traffic that you want to log. You can Select "Accepted" to log this particular traffic and "Rejected" to log this particular traffic accordingly.
5. By default, the maximum aggregation interval would be 10 minutes.
6. Give the destination as "Send to CloudWatch logs."
7. Under the destination log group, give the name of the log group where you want the logs to be published. If there are no log groups already, AWS will create a new one on its own.
8. In the IAM role, give the name of the role that has all the required permission for publishing logs to CloudWatch. If you don't have one, you can create an IAM role by following the steps below.

• Open the IAM console
• Select "Roles" in the pane (navigation ) on the left.
• Click on “Create role” > EC2 > Permissions
• In the Attach permission policy section, choose the permissions you want in the IAM role > Next: Review.
• Give the name for the role and click on Create role button.
• Next, click on the created policy in the "Roles" and choose "Add Inline policy" present in the permissions tab.
• Enter the below policy statement code,

• Click review policy
• Enter the name of the policy and click on create policy.
• In the Trust relationships tab, click on the edit trust relationship.
• Enter the below code

• Enter the name of the IAM role you created and click on "Create."

9. You can selectively choose the fields that you want to be there in the logs by going to the "Format" option.
10. Click Create

Flow Logs for VPCs/Subnets and Publish to CW

1. Go to the VPC console
2. On the left Navigation Pane, click on VPCs (or) subnets and select the VPC/subnet(or) VPCs/Subnets for which you want logs.
3. Click on Actions and then "Create Flow Log."
4. In the filter, you can specify the type of IP traffic that you want to log. You can Select "Accepted" to log this particular traffic and "Rejected" to log this particular traffic accordingly.
5. By default, the maximum aggregation interval would be 10 minutes.
6. Give the destination as "Send to CloudWatch logs."
7. Under the destination log group, give the name of the log group where you want the logs to be published. If there are no log groups already, AWS will create a new one on its own.
8. In the IAM role, give the name of the role that has all the required permission for publishing logs to CloudWatch. If you don't have one, you can create an IAM role by following the steps below.

• Open the IAM console
• Select "Roles" in the pane (navigation ) on the left.
• Click on “Create role” > EC2 > Permissions
• In the Attach permission policy section, choose the permissions you want in the IAM role > Next: Review.
• Give the name for the role and click on Create role button.
• Next,, click on the created policy in the "Roles" and choose "Add Inline policy" present in the permissions tab.
• Enter the below policy statement code,

• Click review policy
• Enter the name of the policy and click on create policy.
• In the Trust relationships tab, click on the edit trust relationship.
• Enter the below code

• Enter the name of the IAM role you created and click on "Create."

9. You can selectively choose the fields that you want to be there in the logs by going to the "Format" option.
10. Click Create

How to Publish VPC Flow Logs to S3?

Flow Logs for Network Interface and Publish to S3

1. Go to the EC2 console
2. On the left Navigation Pane, click on Network Interfaces and select the network interface (or) network interfaces for which you want logs.
3. Click on Actions and then "Create Flow Log."
4. In the filter, you can specify the type of IP traffic that you want to log. You can Select "Accepted" to log this particular traffic and "Rejected" to log this particular traffic accordingly.
5. By default, the maximum aggregation interval would be 10 minutes.
6. Give the destination as "Send to an Amazon S3 bucket".
7. In the S3 bucket ARN section, specify the ARN for the S3 bucket where you want to publish the flow logs.
8. You can selectively choose the fields that you want to be there in the logs by going to the "Format" option.
9. Click Create

Flow Logs for VPCs/Subnets and Publish to S3

1. Go to the VPC console
2. On the left Navigation Pane, click on VPCs (or) subnets and select the VPC/subnet(or) VPCs/Subnets for which you want logs.
3. Click on Actions and then "Create Flow Log."
4. In the filter you can specify the type of IP traffic that you want to log. You can Select "Accepted" to log this particular traffic and "Rejected" to log this particular traffic accordingly.
5. By default, the maximum aggregation interval would be 10 minutes.
6. Give the destination as "Send to an Amazon S3 bucket".
7. In the S3 bucket ARN section, specify the ARN for the S3 bucket where you want to publish the flow logs.
8. You can selectively choose the fields that you want to be there in the logs by going to the "Format" option.
9. Click Create

Use Cases of VPC Flow Logs

Monitoring the network: One can happily use VPC flow logs to monitor the network and further get insights into the network-related performances, latencies, and throughput.

Cutting money on expenses related to the network: One can optimize the network expenses by analyzing network usage periodically.

Network Security: You can look out for any malicious events happening by analyzing the network traffic and can also understand if any of your IP has been compromised. After the identification, one can take quick action to resolve this and avoid things like this in the future.

VPC Flow Logs Limitations

Here are some of the main limitations concerning VPC flow logs

• If you have set up VPC peering and your VPCs are in different accounts, you would not be able to enable VPC flow logs. For one to enable VPC flow logs while setting up VPC peering, both the VPCs need to be in the same AWS account.
• One cannot change the configuration of the flow log once created.
• VPC flow logs miss certain IP traffic, such as the traffic when an ec2 instance looks out for the Amazon DNS server and the traffic when an ec2 instance ( Windows) license activation happens. Other traffic like traffic involving DHCP and traffic during mirroring is also not captured in VPC flow logs

Please look into this link to find more limitations around VPC Flow Logs.