The code for this this project can be found online: https://github.com/lifechurch/terraform_static_sites
I was recently asked to help one of our teams deploy 66 different, statically built websites. Each site was to be in its own S3 bucket and be served by CDN and using SSL. As the content was being served from S3, it made sense to use AWS for the entire project. Here’s the simplified view for how this thing will look: 1. Route 53, to handle DNS 2. CloudFront, to manage caching and serve the content over SSL 3. S3, to hold the content
If we were just creating 1 or 2 sites I’d probably just do this by hand through the AWS Console, however, creating 66 sites by clicking through the portal seems a bit much.
I wanted to see if there was a tool to automate configuring of AWS services for building multiple static sites. Terraform is that tool!
They have support for most of the services we need, leaving just some SSL configuration to do by hand.
One of the first limitations I ran into was ACM (AWS Certificate Manager) has a hard limit of 100 domains per certificate, we actually need 132, as we have both *.domain.com and domain.com for each domain.
Unfortunately the wildcard *domain.com isn’t a thing. After asking Amazon to increase our ACM limit to the maximum, we’re good to start building.
I created a file called domains.tf to hold our domains. There may be a better way of doing this but essentially I created 3 variable lists, one called domain_names, and the other 2 are just the list of domains cut in half domain_group_1 and domain_group_2 , we’ll use those to build our CloudFront Distributions.
variable "domain_names" {
description = "All the domains"
type = "list"
default = ["domain-0.com", "domain-1.com", "domain-2.com", "domain-3.com"...]
}
variable "domain_group_1" {
description = "Need to split the domains up into multiple groups as AWS ACM can only provision a SAN cert for 100 aliases (we have 66 books and 2 records for each (*. and apex) so need to split in half)"
type = "list"
default = ["domain-0.com", "domain-1.com", "domain-3.com"]
}
variable "domain_group_2" {
description = "Need to split the domains up into multiple groups as AWS ACM can only provision a SAN cert for 100 aliases (we have 66 books and 2 records for each (*. and apex) so need to split in half)"
type = "list"
default = ["domain-4.com", "domain-5.com", "domain-6.com"]
}
Once we’ve defined the domains here, we need to start creating the zones in Route53.
We’ll start by creating the main.tf file and setting the AWS region we want to use:
provider "aws" {
region = "us-west-2"
}
Then we’ll add the zones, and create a delegation set which will ensure we’re using the same Nameservers for each domain, making our life easier at the registrar end. Here, we’re getting the length of each domain group and then creating a record for each item in that group, while setting the delegation_id to be the same for each domain.
resource "aws_route53_delegation_set" "main" {
reference_name = "Route53"
}
resource "aws_route53_zone" "primary_group_1" {
count = "${length(var.domain_group_1)}"
name = "${element(var.domain_group_1, count.index)}"
delegation_set_id = "${aws_route53_delegation_set.main.id}"
}
resource "aws_route53_zone" "primary_group_2" {
count = "${length(var.domain_group_2)}"
name = "${element(var.domain_group_2, count.index)}"
delegation_set_id = "${aws_route53_delegation_set.main.id}"
Now Route53 has created all our zone files we’ll add a CNAME record to send www.domain.com to domain.com. Again, we’re breaking this into 2 groups because of the limitations in Terraform and the ACM limit.
resource "aws_route53_record" "www_group_1" {
count = "${aws_route53_zone.primary_group_1.count}"
zone_id = "${element(aws_route53_zone.primary_group_1.*.zone_id, count.index)}"
name = "www"
type = "CNAME"
ttl = "300"
records = ["${element(aws_route53_zone.primary_group_1.*.name, count.index)}"]
}
resource "aws_route53_record" "www_group_2" {
count = "${aws_route53_zone.primary_group_2.count}"
zone_id = "${element(aws_route53_zone.primary_group_2.*.zone_id, count.index)}"
name = "www"
type = "CNAME"
ttl = "300"
records = ["${element(aws_route53_zone.primary_group_2.*.name, count.index)}"]
}
Once we have the base DNS records setup we can add the S3 buckets and their associated bucket policies. This is simply iterating over the list of domains and creating a bucket for each, with static hosting enabled. A basic policy gets applied to the bucket restricting direct access to anything sending the user-agent some secret. We’ll configure CloudFront to send this secret at a later stage, the purpose of this is we only want people to access the site via our CDN not directly.
resource "aws_s3_bucket" "website_bucket" {
count = "${length(var.domain_names)}"
bucket = "${element(var.domain_names, count.index)}"
website {
index_document = "index.html"
error_document = "404.html"
}
}
resource "aws_s3_bucket_policy" "website_bucket" {
count = "${length(var.domain_names)}"
bucket = "${element(var.domain_names, count.index)}"
policy =<<POLICY
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PublicReadAccess",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::${element(var.domain_names, count.index)}/*",
"Condition": {
"StringEquals": {
"aws:UserAgent": "some secret"}
}
}
]
}
POLICY
}
The next step is to create the CloudFront distributions for each domain. However, before we proceed with this step, we need to generate SSL certs for everything. I couldn’t find an easy way to do this with Terraform and all these domains, so I ended up settling on the following - 1 gotcha here is this ACM part will only currently work in US-EAST-1:
Take the list of domains from one of the groups.
Make note of both *.domain.com and domain.com
Use the aws cli tool to request the ACM certificates by using the following pattern:
aws acm request-certificate --domain-name "domain.com" --validation-method DNS --subject-alternative-names "*.domain.com", "*.domain-0.com", "domain-0.com", "*.domain-1.com", "domain-1.com"
Where –domain-name “domain.com” is the first domain in your list, and the –subject-alternative-names list contains the apex and wildcard *.domain for the remainder of the domain_group_1 variables defined in domains.tf. When executed it should return something like this:
"CertificateArn": "arn:aws:acm:us-east-1:111111111111:certificate/aabbccdd-eeff-0011-2233-445566778899"
AWS has created a SAN certificate for us with every one of those domains in the cert. We’ll use that CertificateArn when building our CloudFront Distribution template so make a note of it.
Repeat the above steps for the other domain group. You should now have 2 CertificateArn strings. You’ll need to login to the AWS console and go to Certificate Manager. You should see both of your newly created certificates listed.
I apologize in advance, this bit is manual… You’ll need to go through each domain listed under each of the 2 certificates and click on the button that says Create Record in Route53.
Once you’ve confirmed ownership of the domains, Amazon will approve the certificate for use. Now we can build the CloudFront Distributions. Again, we’re creating 2 separate Terraform resources, 1 for each domain group, they have the same policies, except for the certificate used.
We begin by getting a count of the domains in this group, and setting the origin source for the distribution - in our case the associated S3 bucket. We set our custom User-Agent header to the same secret we defined earlier in the bucket policy. This is important that they match, else you won’t be able to access your content.
We then set our aliases of both domain.com and www.domain.com. This allows us to access the site on both apex and www.
resource "aws_cloudfront_distribution" "website_cdn_1" {
lifecycle {
create_before_destroy = true
# ignore_changes = ["*"]
}
enabled = true
count = "${length(var.domain_group_1)}"
"origin" {
origin_id = "origin-bucket-${element(var.domain_group_1, count.index)}"
domain_name = "${element(var.domain_group_1, count.index)}.s3-website-us-west-2.amazonaws.com"
custom_origin_config {
origin_protocol_policy = "http-only"
http_port = "80"
https_port = "443"
origin_ssl_protocols = ["TLSv1"]
}
custom_header {
name = "User-Agent"
value = "some secret"
}
}
default_root_object = "index.html"
aliases = ["${element(var.domain_group_1, count.index)}", "www.${element(var.domain_group_1, count.index)}"]
}
The next part of the CDN config is the cache behavior (what methods we support, what methods to cache ([“GET”, “HEAD”]), any geographical restrictions on the distribution, and redirect our traffic to HTTPS.
default_cache_behavior {
allowed_methods = ["DELETE", "GET", "HEAD", "OPTIONS", "PATCH", "POST", "PUT"]
cached_methods = ["GET", "HEAD"]
target_origin_id = "origin-bucket-${element(var.domain_group_1, count.index)}"
min_ttl = "0"
default_ttl = "300"
max_ttl = "1200"
forwarded_values {
query_string = false
cookies {
forward = "none"
}
}
viewer_protocol_policy = "redirect-to-https"
compress = true
}
"restrictions" {
"geo_restriction" {
restriction_type = "none"
}
}
Finally, we add the certificate:
viewer_certificate {
cloudfront_default_certificate = true
ssl_support_method = "sni-only"
minimum_protocol_version = "TLSv1"
acm_certificate_arn = "arn:aws:acm:us-east-1:111111111111:certificate/aabbccdd-eeff-0011-2233-445566778899"
}
You’ve now built your CloudFront distribution, linked it to your S3 buckets and added the DNS zones. There’s just one more thing to link it all together. We need to tell Route53 how to route requests for domain.com to the CloudFront Distribution URL, and we can only do that once the distribution has been created.
Again, repeating the need to create 2 resource groups to do this, we make note of the Zone ID for the domain, add an A record pointing to the associated CloudFront distribution.
resource "aws_route53_record" "apex_group_1" {
count = "${aws_route53_zone.primary_group_1.count}"
zone_id = "${element(aws_route53_zone.primary_group_1.*.zone_id, count.index)}"
name = "${element(var.domain_group_1, count.index)}"
type = "A"
alias {
name = "${element(aws_cloudfront_distribution.website_cdn_1.*.domain_name, count.index)}"
zone_id = "${element(aws_cloudfront_distribution.website_cdn_1.*.hosted_zone_id, count.index)}"
evaluate_target_health = false
}
}
Repeat for the other domain group and you’re done.
In summary, this ended up being a lot more trial and error than I expected, I ran into limits with ACM. By default I believe you only get 10 Subject Alternate Names per certificate, which obviously is way too low for our use case. I had to contact AWS support to request a limit increase, to which they responded quickly and gave me the 100 domain limit.
Having to create the certificates on the command line took a little while to figure out too as there isn’t Terraform support for this yet.
There are most likely optimizations to be made here, I’m quite sure there are better ways to accomplish this without having to have split things into 2 groups and duplicate things. I think if you need less than 50 domains on this project you could just list all the domains in 1 variable list and halve the amount of code.
Again, the code is posted here at GitHub if you’d like to take a look, use it, contribute etc. : https://github.com/lifechurch/terraform_static_sites
