Scheduled tasks in “the cloud” are a nightmare. If you want something to be run once every 10 minutes for example, you can’t just put that in every application server crontab. Your task would be executed every 10 minutes by all of your servers, probably at the same time.

An alternative solution is having a dedicated single cron server that performs all of your scheduled actions. While this does work, it is probably not ideal. First of all, your cron server becomes a new single point of failure. You might also have to “deploy” your app code to this dedicated server, or decide how to roll out your release if you prefer immutable infrastructure with zero cron downtime.

If you are running a Ruby on Rails app, Sidekiq does provide a very nice solution. The sidekiq-cron gem can be used to run scheduled tasks on a fleet of background job servers.

Leverage the Cloud

For those of us not lucky enough to manage a simple Rails monolith, AWS comes to the rescue with CloudWatch scheduled events.

Essentially we can configure CloudWatch to do something on a fixed schedule or even with a cron-like string.

The action taken on a CloudWatch event can be, among other things:

  • Running a CodePipeline
  • Adding a job on an SQS queue
  • Terminating an EC2 instance
  • Invoking a Lambda function

To set a CloudWatch Event to invoke a lambda function, you can use the following Terraform code:

resource "aws_lambda_function" "my_function" {
  // ... Define your lambda function here ...
}

// Allow CloudWatch to invoke our function
resource "aws_lambda_permission" "allow_cloudwatch_to_invoke" {
  function_name = aws_lambda_function.my_function.function_name
  statement_id = "CloudWatchInvoke"
  action = "lambda:InvokeFunction"

  source_arn = aws_cloudwatch_event_rule.every_day.arn
  principal = "events.amazonaws.com"
}

// Create the "cron" schedule
resource "aws_cloudwatch_event_rule" "every_day" {
  name = "daily"
  schedule_expression = "rate(1 day)"

  // or

  schedule_expression = "cron(0 0 * * *)"
}

// Set the action to perform when the event is triggered
resource "aws_cloudwatch_event_target" "invoke_lambda" {
  rule = aws_cloudwatch_event_rule.every_day.name
  arn = aws_lambda_function.my_function.arn
}