S3 bucket notifications are a useful feature of Amazon’s Simple Storage Service (S3) that allow you to receive notifications when certain events occur in your S3 bucket. These notifications can be sent to a variety of targets, such as an AWS Lambda function, an Amazon SNS topic, or an SQS queue. In this post, we will focus on setting up S3 bucket notifications using Terraform.

Prerequisites

This post will assume you have a basic understanding of Terraform and its concepts. If you are not familiar with Terraform you can read my Introduction to Terraform. In addition, you will need an AWS account and an S3 bucket to which you want to apply the S3 bucket notifications.

Event types

Amazon S3 supports a variety of event types that can be used to trigger notifications. The available event types include:

s3:ObjectCreated - This event type is triggered when an object is created in the S3 bucket. This includes events such as when an object is uploaded to the bucket or when an object is copied to the bucket. This event can be used for things like automatically processing new files that are uploaded to the bucket. s3:ObjectRemoved - This event type is triggered when an object is removed from the S3 bucket. This can be useful for things like cleaning up associated resources or raising a compliance alert. s3:ObjectRestore - This event type is triggered when an object is restored from the S3 bucket’s versioning storage using the restore operation.

Configuring S3 Bucket Notifications to an SQS Queue

The following Terraform code will create an SQS queue and configure S3 bucket notifications to send messages to the queue when an object is created in the S3 bucket.

First, create or import an S3 bucket:

resource "aws_s3_bucket" "bucket" {
  bucket = var.bucket_name
}

Next, create an SQS queue. Note that the SQS queue must have a policy that allows the S3 bucket to send messages to the queue.

resource "aws_sqs_queue" "queue" {
  name = var.queue_name

  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "sqs:SendMessage",
      "Resource": "arn:aws:sqs:*:*:${var.queue_name}",
      "Condition": {
        "ArnEquals": { "aws:SourceArn": "${aws_s3_bucket.bucket.arn}" }
      }
    }
  ]
}
POLICY
}

Finally, create the S3 bucket notification. The filter_suffix attribute can be used to filter the notifications to only those objects that have a specific suffix. In this example, we will only receive notifications for objects that have a .log suffix.

resource "aws_s3_bucket_notification" "bucket_notification" {
  bucket = aws_s3_bucket.bucket.id

  queue {
    queue_arn     = aws_sqs_queue.queue.arn
    events        = ["s3:ObjectCreated:*"]
    filter_suffix = ".log"
  }
}

Configuring S3 Bucket Notifications to an SNS Topic

Not too different from the previous example, the following Terraform code will create an SNS topic and configure S3 bucket notifications to send messages to the topic when an object is created in the S3 bucket.

Assuming you have already created or imported an S3 bucket, create an SNS topic, with a policy that allows the S3 bucket to publish messages to the topic.

resource "aws_sns_topic" "topic" {
  name = var.topic_name

  policy = <<POLICY
{
    "Version":"2012-10-17",
    "Statement":[{
        "Effect": "Allow",
        "Principal": { "Service": "s3.amazonaws.com" },
        "Action": "SNS:Publish",
        "Resource": "arn:aws:sns:*:*:${var.topic_name}",
        "Condition":{
            "ArnLike":{"aws:SourceArn":"${aws_s3_bucket.bucket.arn}"}
        }
    }]
}
POLICY
}

Configuring S3 Bucket Notifications to an AWS Lambda Function

Finally, the Terraform code required to invoke an AWS Lambda function for S3 bucket events is a bit more involved. First, create a Lambda function and its associated IAM role. This role is assumed by the Lambda function when it is invoked.

resource "aws_lambda_function" "func" {
  function_name = var.function_name
  filename      = var.lambda_filename
  role          = aws_iam_role.iam_for_lambda.arn
  handler       = var.handler
  runtime       = var.runtime
}

resource "aws_iam_role" "iam_for_lambda" {
  name = var.lambda_role_name

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow"
    }
  ]
}
EOF
}

Next, set a Lambda permission that allows the S3 bucket to invoke the Lambda function.

resource "aws_lambda_permission" "allow_bucket" {
  statement_id  = "AllowExecutionFromS3Bucket"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.func.arn
  principal     = "s3.amazonaws.com"
  source_arn    = aws_s3_bucket.bucket.arn
}

You can now create the S3 bucket notification. Note the depends_on attribute, which creates a dependency between the S3 bucket notification and the Lambda permission. This ensures that the S3 bucket notification is created after the Lambda permission.

resource "aws_s3_bucket_notification" "bucket_notification" {
  bucket = aws_s3_bucket.bucket.id

  lambda_function {
    lambda_function_arn = aws_lambda_function.func.arn
    events              = ["s3:ObjectCreated:*"]
    filter_prefix       = "AWSLogs/"
    filter_suffix       = ".log"
  }

  depends_on = [aws_lambda_permission.allow_bucket]
}

Multiple S3 Bucket Notifications

Note: you can only create a single S3 notification configuration per bucket. Multiple aws_s3_bucket_notification resources will overwrite each other.

If you need to send notifications to multiple targets, you can create a single aws_s3_bucket_notification resource with multiple queue, topic, or lambda_function blocks.

For example, the following Terraform code will create an S3 bucket notification that sends notifications to both an SQS queue and an SNS topic for different events.

resource "aws_s3_bucket_notification" "bucket_notification" {
  bucket = aws_s3_bucket.bucket.id

  queue {
    queue_arn = aws_sqs_queue.queue.arn
    events    = ["s3:ObjectCreated:*"]
  }

  topic {
    topic_arn = aws_sns_topic.topic.arn
    events    = ["s3:ObjectRemoved:*"]
  }
}

Testing S3 Bucket Notifications

To test S3 bucket notifications, you can create or upload an object to the S3 bucket and then verify that the notifications are sent as expected.

First, create or upload an object to the S3 bucket that you have configured to send notifications. This can be done using the AWS Management Console, the AWS CLI, or an AWS SDK. For example, to upload an object using the AWS CLI, you can use the aws s3 cp command, specifying the source file and the destination bucket and path.

aws s3 cp /path/to/file s3://my-bucket/path/to/file

Next, you can verify that the notifications were sent by checking the target service where the notifications were sent. For example, if you configured the S3 bucket to send notifications to an SQS queue, you can use the aws sqs receive-message command to retrieve and view the messages in the queue.

aws sqs receive-message --queue-url <my queue url>

SNS does not provide a command-line interface for retrieving messages from a topic. However, you can subscribe to the SNS topic via email or SMS for example and check your inbox or phone for the notifications.

Finally, if you configured the S3 bucket to invoke an AWS Lambda function, you can view the function’s execution logs in CloudWatch Logs to verify that the function was invoked and executed successfully.

Conclusion

Amazon S3 bucket notifications are a powerful and versatile feature that can be used to receive notifications when certain events occur in an S3 bucket. By configuring S3 bucket notifications, you can automate processes and trigger actions in response to events such as when an object is created, deleted, or restored in the S3 bucket.

I always recommend using Terraform to configure your AWS infrastructure. Terraform is a powerful tool that allows you to define and manage your infrastructure as code, keep it versioned, and deploy it consistently and reliably.