AWS ECS Private Services

Hosting private services on ECS is pretty straight forward but I didn't find to much documentation on it when searching the webs.

What?

An AWS ECS service on a private subnet with an internally facing ALB connected to the Bastion server or micro service. This isn't for the beginners! This is an advanced article and it is expected you are already familiar with ecs-cli and have used it to successfully configure a public facing service.

Why?

• Security - In some infra setups none of the management services are exposed externally. People connecting would use an ssh tunnel through the Bastion minimizing the attack vector.
• Less dependencies - Some people don't like VPNs; it's more to keep up with. There are clients to maintain, some kind of VPN service, and keeping user lists up to date.
• Internal Services - Micro services are here to stay and not all of them are meant to be public facing. Some services support the public facing api and should be kept in private subnets away from prying eyes.

Bastion

What is a Bastion server? It sits in public facing subnet sort of like a DMZ and has access to the internal network. It is typically the entry point to sensitive systems or networks not externally exposed to the internet. The Bastion does not contain any private keys or sensitive data and only acts as a tunnel. You might be reading this post for private micro services if that's the case replace the Bastion for your internal service(s) and it works the same.

Requirements

Top level of the items you will need to configure to get this rolling.

• Application Load Balancer
• Target Group
• ECS Cluster
• 2 private subnets
• Application Load Balancer security group
• Instance security group
• DB security group
• ECS Service
• Bastion server(or micro service)
• Bastion security group(or micro service)

The big picture

For the sake of not doubling up we are using a Bastion but it can also be a micro-service. Less is more; only expose the bare minimum.

Private Subnets

Create your private subnets in your VPC and take note of the subnet-ids. These subnets should be in different AZs.

Security Groups

We need to create security groups with links between them for connectivity. Follow the diagram above as reference.

• ALB-SG - Incoming: contains the incoming traffic port 80 from Bastion-SG. Outgoing: contains the security group Instance-SG.
• Instance-SG - Incoming: contains ALB-SG. Outgoing: security group RDS-SG, ALB-SG, Bastion-SG.
• Bastion-SG - Incoming: 22 from all external or whitelist ips. Outgoing: security group ALB-SG.
• RDS-SG - Incoming: 5432 from Instance-SG. Outgoing: security group Instance-SG.

ECS-CLI Disclaimer

Launch Cluster

Next up is setting up the cluster. You will need to point the ecs-cli to different clusters whenever you want to run it.

ecs-cli configure --region us-east-1 --cluster private-demo-cluster

Now that we have a cluster name and region set we can bring it up. The big catch here is setting --no-associate-public-ip-address and setting the --subnets to the private subnet-ids. The security group should be set to the $instance-sg which should have the correct ports mapped between it and other locations.

ecs-cli up --keypair accessKey --capability-iam --size 2 --instance-type t2.medium --force --vpc vpc-xxxxxx --image-id ami-xxxxx --subnets $subnet-A,$subnet-B --security-group $instance-sg --no-associate-public-ip-address

At this point you should be able to check ECS and find a cluster running on the correct subnets without public addresses.

Internal-ALB

Create an ALB on AWS via the console.

You will see on the first page an option internet or internal. This is the money maker that tells the ALB we want an internal DNS entry. In the Availability Zones section make sure to enable the LB in the private subnets where your instances reside.

In this example, I created the Target Group which is an option on the next page.

Copy the arn of the new target-group-arn for the next step and get the DNS name from the load balancer. Notice the DNS is set to something like internal-serviceName-id.us-east-1.elb.amazonaws.com/.

Create Service

Final step, we need to create the private service.

Replace the target-group-arn and then launch.

ecs-cli compose --file docker-compose.yml service up --role serviceRole --container-name "privateService" --container-port 80 --target-group-arn "arn:aws:elasticloadbalancing:us-east-1::::"

Check the health of the instances in the Target Group and confirm they are healthy. Once the service is stable you can test out. The command below creates a tunnel from internal-dns-entry port 80 to your localhost port 8080.

ssh -A  user@$bastionServerExternalIP -N -L 8080:$internal-dns-entry:80

Open your localhost:8080 and you should now have a tunneled version of that private service. If you are running a micro service try to send a request to the new service.