Hosting private services on ECS is pretty straight forward but I didn't find to much documentation on it when searching the webs.
An AWS ECS service on a private subnet with an internally facing ALB connected to the Bastion server or micro service. This isn't for the beginners! This is an advanced article and it is expected you are already familiar with
ecs-cli and have used it to successfully configure a public facing service.
- Security - In some infra setups none of the management services are exposed externally. People connecting would use an ssh tunnel through the Bastion minimizing the attack vector.
- Less dependencies - Some people don't like VPNs; it's more to keep up with. There are clients to maintain, some kind of VPN service, and keeping user lists up to date.
- Internal Services - Micro services are here to stay and not all of them are meant to be public facing. Some services support the public facing api and should be kept in private subnets away from prying eyes.
What is a Bastion server? It sits in public facing subnet sort of like a DMZ and has access to the internal network. It is typically the entry point to sensitive systems or networks not externally exposed to the internet. The Bastion does not contain any private keys or sensitive data and only acts as a tunnel. You might be reading this post for private micro services if that's the case replace the Bastion for your internal service(s) and it works the same.
Top level of the items you will need to configure to get this rolling.
- Application Load Balancer
- Target Group
- ECS Cluster
- 2 private subnets
- Application Load Balancer security group
- Instance security group
- DB security group
- ECS Service
- Bastion server(or micro service)
- Bastion security group(or micro service)
The big picture
For the sake of not doubling up we are using a Bastion but it can also be a micro-service. Less is more; only expose the bare minimum.
Create your private subnets in your VPC and take note of the subnet-ids. These subnets should be in different AZs.
We need to create security groups with links between them for connectivity. Follow the diagram above as reference.
- ALB-SG - Incoming: contains the incoming traffic port
Bastion-SG. Outgoing: contains the security group
- Instance-SG - Incoming: contains
ALB-SG. Outgoing: security group
- Bastion-SG - Incoming:
22from all external or whitelist ips. Outgoing: security group
- RDS-SG - Incoming:
Instance-SG. Outgoing: security group
Next up is setting up the cluster. You will need to point the
ecs-cli to different clusters whenever you want to run it.
ecs-cli configure --region us-east-1 --cluster private-demo-cluster
Now that we have a cluster name and region set we can bring it up. The big catch here is setting
--no-associate-public-ip-address and setting the
--subnets to the private subnet-ids. The security group should be set to the
$instance-sg which should have the correct ports mapped between it and other locations.
ecs-cli up --keypair accessKey --capability-iam --size 2 --instance-type t2.medium --force --vpc vpc-xxxxxx --image-id ami-xxxxx --subnets $subnet-A,$subnet-B --security-group $instance-sg --no-associate-public-ip-address
At this point you should be able to check ECS and find a cluster running on the correct subnets without public addresses.
Create an ALB on AWS via the console.
You will see on the first page an option
internal. This is the money maker that tells the ALB we want an internal DNS entry. In the
Availability Zones section make sure to enable the LB in the private subnets where your instances reside.
In this example, I created the
Target Group which is an option on the next page.
Copy the arn of the new
target-group-arn for the next step and get the DNS name from the load balancer. Notice the DNS is set to something like
Final step, we need to create the private service.
target-group-arn and then launch.
ecs-cli compose --file docker-compose.yml service up --role serviceRole --container-name "privateService" --container-port 80 --target-group-arn "arn:aws:elasticloadbalancing:us-east-1::::"
Check the health of the instances in the Target Group and confirm they are healthy. Once the service is stable you can test out. The command below creates a tunnel from
80 to your localhost port
ssh -A user@$bastionServerExternalIP -N -L 8080:$internal-dns-entry:80
Open your localhost:8080 and you should now have a tunneled version of that private service. If you are running a micro service try to send a request to the new service.