Understanding VPC Flow Logs Packet Analysis

Introduction

AWS VPC Flow Logs capture metadata about IP traffic flowing through Elastic Network Interfaces (ENIs) in your VPC. They don’t capture packet payloads (which would be both enormous and a privacy/compliance issue), but they record source/destination IPs and ports, protocol, byte counts, and whether traffic was accepted or rejected by security groups and NACLs. This data is invaluable for security analysis (detect port scans, unusual egress), network troubleshooting (verify security group rules are working), and compliance (prove that no unauthorized traffic reached sensitive resources).

flowchart TD
    A["Traffic hits an ENI in the VPC"] --> B["Flow Logs record metadata (IPs, ports, action)"]
    B --> C["Logs delivered to S3 or CloudWatch Logs"]
    C --> D["Query with Athena for security/ops analysis"]

Flow Log Record Format

version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes windowstart windowend action log-status

Example records:
2 123456789012 eni-0a1b2c3d 203.0.113.42 10.0.1.5 41820 443 6 20 4000 1704067200 1704067260 ACCEPT OK
2 123456789012 eni-0a1b2c3d 45.33.32.156 10.0.1.5 54321 22  6 5  300  1704067200 1704067260 REJECT OK
2 123456789012 eni-0a1b2c3d 10.0.1.5 8.8.8.8    52941 53  17 1  60   1704067200 1704067260 ACCEPT OK

Key fields:

• srcaddr/dstaddr: Source/destination IP addresses
• srcport/dstport: Source/destination ports
• protocol: 6=TCP, 17=UDP, 1=ICMP
• action: ACCEPT or REJECT (by security group/NACL)
• log-status: OK, NODATA (no traffic), SKIPDATA (capacity exceeded)

Setting Up Flow Logs

# Enable VPC Flow Logs to S3
aws ec2 create-flow-logs   --resource-type VPC   --resource-ids vpc-12345678   --traffic-type ALL            # ALL, ACCEPT, or REJECT
  --log-destination-type s3   --log-destination arn:aws:s3:::my-flow-logs-bucket/vpc-logs/   --log-format '${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${windowstart} ${windowend} ${action} ${log-status}'

# Terraform
resource "aws_flow_log" "main" {
  iam_role_arn    = aws_iam_role.flow_log.arn
  log_destination = aws_s3_bucket.flow_logs.arn
  traffic_type    = "ALL"
  vpc_id          = aws_vpc.main.id
  log_destination_type = "s3"
}

Querying with Athena

First, create an Athena table over the S3 path:

CREATE EXTERNAL TABLE vpc_flow_logs (
  version INT,
  account_id STRING,
  interface_id STRING,
  srcaddr STRING,
  dstaddr STRING,
  srcport INT,
  dstport INT,
  protocol BIGINT,
  packets BIGINT,
  bytes BIGINT,
  windowstart BIGINT,
  windowend BIGINT,
  action STRING,
  logstatus STRING
)
PARTITIONED BY (dt STRING)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ' '
STORED AS TEXTFILE
LOCATION 's3://my-flow-logs-bucket/vpc-logs/'
TBLPROPERTIES ("skip.header.line.count"="1");

Security Analysis Queries

-- Port scan detection: many ports from one source IP in short window
SELECT srcaddr, COUNT(DISTINCT dstport) AS ports_probed
FROM vpc_flow_logs
WHERE action = 'REJECT'
  AND dt = '2024/03/15'
  AND windowstart BETWEEN 1710461800 AND 1710462400  -- 10-minute window
GROUP BY srcaddr
HAVING COUNT(DISTINCT dstport) > 20
ORDER BY ports_probed DESC;

-- Unusual egress: high-volume traffic to unexpected external IPs
SELECT dstaddr, SUM(bytes) AS total_bytes, COUNT(*) AS flow_count
FROM vpc_flow_logs
WHERE srcaddr LIKE '10.%'              -- Internal source
  AND dstaddr NOT LIKE '10.%'         -- External destination
  AND action = 'ACCEPT'
  AND dt = '2024/03/15'
GROUP BY dstaddr
ORDER BY total_bytes DESC
LIMIT 20;

-- Verify security group rule blocked port 22 from internet
SELECT srcaddr, dstaddr, dstport, action, COUNT(*) as count
FROM vpc_flow_logs
WHERE dstport = 22
  AND dstaddr LIKE '10.0.1.%'  -- Target private subnet
  AND dt = '2024/03/15'
GROUP BY srcaddr, dstaddr, dstport, action;

-- Top talkers by bytes transferred
SELECT srcaddr, dstaddr,
       SUM(bytes) AS bytes,
       SUM(packets) AS packets
FROM vpc_flow_logs
WHERE dt = '2024/03/15'
GROUP BY srcaddr, dstaddr
ORDER BY bytes DESC
LIMIT 10;

Key Takeaways

• VPC Flow Logs capture traffic metadata (IPs, ports, bytes, action) — not payload content — at the network interface level.
• The action field (ACCEPT/REJECT) tells you whether security groups and NACLs are blocking traffic as intended — invaluable for rule debugging.
• Use Athena partitioned tables on the S3 path structure to enable efficient date-range queries without scanning all log files.
• Common security queries: port scan detection (many REJECTed ports from one source), unusual egress (large data transfers to external IPs), SSH brute force (many REJECT on port 22).
• Enable traffic type = ALL in production (not just ACCEPT) — rejected traffic is often more interesting from a security perspective.