Today I had to make another entry in the “Things that have previously gone wrong” section of our Ops manual. We had users in the United Arab Emirates and Kuwait reporting that some of our apps were broken.

We had narrowed this down to certain audio and video files not loading over our AWS Cloudfront CDN. Initially I had assumed that there was some kind of content filter sitting between our users and our origin, as the problem had started on a Sunday evening and was affecting a significant number of people in the region, but only for some pieces of content.

When I used a VPN to test the files via the CDN in other locations, all was fine. I didn’t have a UAE or Kuwait endpoint, but it was obvious that it was an isssue there. I got some friendly users to screenshot a deeplink to a piece of content:

Screen shot showing a Cloudfront HTTP 502 error explaining that the content it was trying to fetch was refused, along with the request ID

I don’t know what I was expecting to see; I’d assumed it would be some kind of content filter screen - I wasn’t expecting a Cloudfront error.

Using the request ID, AWS confirmed that we were refusing connections to them. WTF? Then I remembered our fail2ban rules - somehow the region’s Cloudfront endpoint IP 64.252.99.116 had gotten itself into our iptables DROP list. I cleared the tables, and using AWS’s published list of its IP addresses, added this rule to our jail.local to ensure we never block any edge IP again:

aws_cloudfront_ips = 120.52.22.96/27 205.251.249.0/24 180.163.57.128/26 204.246.168.0/22 205.251.252.0/23 54.192.0.0/16 204.246.173.0/24 54.230.200.0/21 120.253.240.192/26 116.129.226.128/26 130.176.0.0/17 108.156.0.0/14 99.86.0.0/16 205.251.200.0/21 223.71.71.128/25 13.32.0.0/15 120.253.245.128/26 13.224.0.0/14 70.132.0.0/18 15.158.0.0/16 13.249.0.0/16 205.    251.208.0/20 65.9.128.0/18 130.176.128.0/18 58.254.138.0/25 54.230.208.0/20 116.129.226.0/25 52.222.128.0/17 64.252.128.0/18 205.251.254.0/24 54.230.224.0/19 71.152.0.0/17 216.137.32.0/19 204.246.172.0/24 120.52.39.128/27 118.193.97.64/26 223.71.71.96/27 54.240.128.0/18 205.251.250.0/23 180.163.57.0/25 52.46.0.0/18 223.71.11.0/27 52.82.128.0/19 54.230.0.0/17     54.230.128.0/18 54.239.128.0/18 130.176.224.0/20 36.103.232.128/26 52.84.0.0/15 143.204.0.0/16 144.220.0.0/16 120.52.153.192/26 119.147.182.0/25 120.232.236.0/25 54.182.0.0/16 58.254.138.128/26 120.253.245.192/27 54.239.192.0/19 18.64.0.0/14 120.52.12.64/26 99.84.0.0/16 130.176.192.0/19 52.124.128.0/17 204.246.164.0/22 13.35.0.0/16 204.246.174.0/23 36.103.2    32.0/25 119.147.182.128/26 118.193.97.128/25 120.232.236.128/26 204.246.176.0/20 65.8.0.0/16 65.9.0.0/17 108.138.0.0/15 120.253.241.160/27 64.252.64.0/18 13.113.196.64/26 13.113.203.0/24 52.199.127.192/26 13.124.199.0/24 3.35.130.128/25 52.78.247.128/26 13.233.177.192/26 15.207.13.128/25 15.207.213.128/25 52.66.194.128/26 13.228.69.0/24 52.220.191.0/26 13.21    0.67.128/26 13.54.63.128/26 99.79.169.0/24 18.192.142.0/23 35.158.136.0/24 52.57.254.0/24 13.48.32.0/24 18.200.212.0/23 52.212.248.0/26 3.10.17.128/25 3.11.53.0/24 52.56.127.0/25 15.188.184.0/24 52.47.139.0/24 18.229.220.192/26 54.233.255.128/26 3.231.2.0/25 3.234.232.224/27 3.236.169.192/26 3.236.48.0/23 34.195.252.0/24 34.226.14.0/24 13.59.250.0/26 18.216.    170.128/25 3.128.93.0/24 3.134.215.0/24 52.15.127.128/26 3.101.158.0/23 52.52.191.128/26 34.216.51.0/25 34.223.12.224/27 34.223.80.192/26 35.162.63.192/26 35.167.191.128/26 44.227.178.0/24 44.234.108.128/25 44.234.90.252/30

ignoreip = %(daniships)s %(ukschools)s %(aws_cloudfront_ips)s

You can generate the list of IP addresses and CIDRs really easily with jq:

curl -s https://ip-ranges.amazonaws.com/ip-ranges.json | jq '.prefixes[] | select(.service=="CLOUDFRONT") | .ip_prefix' | xargs

I’m assuming they’ll change over time, so I’ll need to keep the aws_cloudfront_ips updated every now and then.