Editor's note: This is a guest post written by Noah Zoschke, Co-founder & CTO at Convox. Convox is a private app deployment platform that runs in your own AWS account. Prior to starting Convox, Noah was an Engineering Manager and Platform Architect at Heroku for over 5 years and has extensive experience scaling cloud infrastructure.
Contents
- Getting Started on AWS ECS
- Getting Started on Joyent Triton
- API and CLI Experience
- Security
- Documentation
- Support
- Pricing
- Production Use
- Conclusion
Introduction
Cloud Infrastructure providers are working hard to make containers a first-class feature of their services.
Container technology is nothing new. For decades, operating systems have offered features to offer multiple isolated process spaces. The chroot mechanism, which has been around since 1982, was a key first step in securely isolating the filesystem between processes. Features like CPU, Memory, and I/O limits and network isolation were delivered over the next decades with FreeBSD jails, Linux Virtuozzo, Solaris Zones, and Linux LXC. With the correct configuration, any computer or virtual machine has been able to run containers for a long time.
However this latest wave of container services aims to make containers the first and only abstraction for running software. In a container-based world there are no more servers, VMs and arcane knowledge to configure the server and containers. We can think of containers simply as definition of a filesystem image and how much CPU and memory it requires, and leave it up to service providers to start these containers and keep them running.
On April 9th 2015, Amazon newest compute service, EC2 Container Service, became generally available[1]. According to AWS, ECS "is a highly scalable, high performance container management service that supports Docker containers and allows you to easily run applications on a managed cluster of Amazon EC2 instances."
On June 14th, 2015, Joyent’s Elastic Container Infrastructure, code-named Triton, became generally available[2]. According to Joyent, Triton lets you "securely deploy and operate containers with bare metal speed on container-native infrastructure, your cloud or ours."
Google is also working hard on their Container Engine service which is now GA, but wasn't at the time this review was written.
Getting Started on AWS ECS
Amazon offers a thorough "Setting Up with Amazon ECS" and "Getting Started with Amazon ECS" guide.
As a prerequisite I signed up for a new sandbox AWS account to experiment with EC2 and ECS. Then I followed the ECS "First Run Wizard" to create my first cluster and deploy my first containers. The wizard has steps to:
- Define tasks for httpd (Apache) and busybox
- Define an empty non-persistent volume shared between multiple containers
- Schedule 1 long-running service for the httpd and busybox tasks
- Configure an HTTP ELB on port 80
- Configure a cluster of 1 t2.micro instance running the Amazon ECS-Optimized Amazon Linux AMI
- Create an IAM ECS Instance Role
- Launch everything as a CloudFormation stack
The wizard represents a big step forward for AWS in easy onboarding onto their complex services. However there’s no hiding the complexity on AWS. Setup has you wait while Amazon provisions 15 different infrastructure components like a VPC, Security Group, AutoScaling Group, Elastic Load Balancer, IAM profile, AMI, EC2 Instance. The wizard hides this complexity behind a progress bar, but you can open up the CloudFormation Management Console to see all the resources as they are provisioned.
After waiting 5 minutes the ECS instances, Task Definitions and Service are up and running.
Getting Started on Joyent Triton
Joyent offers good "Triton Elastic Container Service", "Getting Started" and "Starting and managing Docker containers" guides.
As a prerequisite I signed up for Joyent. Then I used the Joyent Portal Dashboard to run my first containers. The guide has steps to:
- Select a Data Center
- Pull an "nginx" Docker Image
- Set the restart policy to "Always" to keep our container running
- Attach a Public IP to the container
- Configure port 80 on the Internet to forward port 80 on the container
After waiting about a minute, the container is up and running.
Getting Started Conclusion
Triton is far easier to get started on than ECS. Joyent has gone to great lengths to make containers a native part of their platform, and the experience of "just start this container" is incredible.
Amazon is doing a great job of smoothing out the setup, but there’s not much magic in ECS. It requires us to create and manage a cluster of instances to schedule containers on.
API and CLI Experience
While these hello world guides are great for understanding the basics, serious users will be configuring these container services via the API and Command Line clients. And here is where AWS and Joyent diverge greatly.
Joyent has taken an extremely bold approach to running Docker containers. They added a good portion of the Docker Remote API to their Smart Data Center APIs, allowing us to use the native docker client to manage our containers on the Joyent cloud. To play with this impressive experience, I followed the Docker Remote API implementation for Triton guide:
$ curl -O https://raw.githubusercontent.com/joyent/sdc-docker/master/tools/sdc-docker-setup.sh
$ bash sdc-docker-setup.sh https://us-east-1.api.joyent.com jill ~/.ssh/sdc-docker.id_rsa
Setting up Docker client for SDC...
$ export DOCKER_CERT_PATH=/Users/localuser/.sdc/docker/jill
$ export DOCKER_HOST=tcp://us-east-1.docker.joyent.com:2376
$ export DOCKER_TLS_VERIFY=1
$ docker run -d -p 80 nginx
latest: Pulling from nginx (dd3b19ec-0ae5-4347-933b-c7c8e4078c1b)
…
b365140281c6403394d97fd74e41d2c76a06b8c3b8c4431ab12dddd52352ccfc
$ docker exec -it b365140281c6 bash
root@b365140281c6:/#
This opens up a whole new line of thinking for running and configuring containers in the cloud. On Triton you don’t ever have to think about servers or other infrastructure services if we can compose everything out of Docker container primitives. For example, if you want to load balance between two http servers, you run an haproxy container linked to the http containers. We can define this architecture in docker-compose.yml and launch it with docker-compose up -d.
Amazon has taken a far more conservative and familiar approach to running Docker containers. They added new APIs for ECS with new concepts for "Task Definitions", "Tasks", "Services" and "Clusters". But these APIs heavily reference and depend on other AWS services like EC2 Instances, AutoScaling Groups and Elastic Load Balancers.
The APIs are big, and the CLI is clunky, but everything is thoroughly documented and works well. Task Definitions aren’t too hard to write and register to accomplish anything we can do locally with docker.
$ aws ecs register-task-definition --generate-cli-skeleton > httpd.json
$ aws ecs register-task-definition --cli-input-json file://httpd.json
{
"taskDefinition": {
"volumes": [],
"taskDefinitionArn": "arn:aws:ecs:us-east-1:901416387788:task-definition/httpd:6",
"containerDefinitions": [
{
"environment": [],
"name": "httpd",
"mountPoints": [],
"image": "httpd:2.4",
"cpu": 0,
"portMappings": [
{
"containerPort": 80,
"hostPort": 80
}
],
"entryPoint": [
""
],
"memory": 300,
"command": [
""
],
"essential": true,
"volumesFrom": []
}
],
"family": "httpd",
"revision": 6
}
}
$ aws ecs create-service --generate-cli-skeleton > service.json
$ aws ecs create-service --cli-input-json file://service.json
{
"service": {
"status": "ACTIVE",
"taskDefinition": "arn:aws:ecs:us-east-1:901416387788:task-definition/httpd:6",
"pendingCount": 0,
"loadBalancers": [
{
"containerName": "httpd",
"containerPort": 80,
"loadBalancerName": "EC2Contai-EcsElast-B6PVES22ND2V"
}
],
"roleArn": "arn:aws:iam::901416387788:role/ecsServiceRole",
"desiredCount": 1,
"serviceName": "httpd",
"clusterArn": "arn:aws:ecs:us-east-1:901416387788:cluster/default",
"serviceArn": "arn:aws:ecs:us-east-1:901416387788:service/httpd",
"deployments": [
{
"status": "PRIMARY",
"pendingCount": 0,
"createdAt": 1436798231.533,
"desiredCount": 1,
"taskDefinition": "arn:aws:ecs:us-east-1:901416387788:task-definition/httpd:6",
"updatedAt": 1436798231.533,
"id": "ecs-svc/9223370600056544274",
"runningCount": 0
}
],
"events": [],
"runningCount": 0
}
}
$ aws ecs describe-services --cluster arn:aws:ecs:us-east-1:901416387788:cluster/default --services arn:aws:ecs:us-east-1:901416387788:service/httpd
{
"services": [
{
"status": "ACTIVE",
"taskDefinition": "arn:aws:ecs:us-east-1:901416387788:task-definition/httpd:6",
"pendingCount": 0,
"loadBalancers": [
{
"containerName": "httpd",
"containerPort": 80,
"loadBalancerName": "EC2Contai-EcsElast-B6PVES22ND2V"
}
],
...
API and CLI Experience Conclusion
Joyent’s "container native" approach and Docker API compatibility again wins hands-down for quickly defining, running and inspecting containers. Particularly noteworthy is attaching to a running container with a docker exec command.
The main upside of the ECS approach is that is that the "define, create and describe" workflow extremely familiar to established AWS users. Similarly, inspection techniques are easy for anyone familiar with EC2. With security groups and SSH keys configured correctly you can SSH into the instance where you will find a standard Docker environment running your service. There are numerous devops tools like Ansible that can make this experience great.
Security
The security of containers is a much discussed subject.
AWS has not advanced the Docker container security story at all. Within a cluster, containers need to effectively trust each other, as there is nothing stopping containers on the same instance from interacting with each other. ECS relies on IAM to grant the instance and ECS agent the ability to update ELBs and nothing else, and EC2 and Security Groups to control SSH access and inbound and outbound networking.
Joyent, on the other hand, is coming out strong with their security story. "Full isolation per container in a multi-tenant environment" is extremely hard, and Joyent delivers. Based on SmartOS Zones, every container is fully isolated from every other container on the same computer.
Documentation
The Amazon EC2 Container Service Documentation is everything we expect from Amazon. They offer a Developer Guide for getting started, a full API Reference for really digging into the service and a CLI Reference that shows how to accomplish anything on ECS with the aws-cli tool. The docs are extremely accurate and thorough.
The Triton Elastic Container Service Docs leave a lot to be desired. I encountered lots of small issues following the docs, such as trouble setting up the right SSH key, getting the right DOCKER environment variables during the setup, and working with private registries. Some of the docs are laden with marketing speak, and other’s admit their own incompleteness.
Joyent has a lot more work to do here!
Support
What Joyent is lacking in documentation, they make up in support. During my experiments I opened up two support tickets and had extremely positive exchanges. On one ticket I asked about strategies to load balance between two httpd containers, a setup which I do not see addressed well in the Documentation. Within 3 hours I had a deeply technical and thorough response. Getting meaningful docker-compose.yml and HAProxy configuration is awesome service.
During the same period I opened an AWS support ticket asking for help removing an ECS Cluster that I can not delete through the API or CLI due to an error. First, it’s hard to open a technical ticket in the first place, since AWS requires a support subscription. Next it feels like normal priority tickets go into an extremely deep queue. It took 3 days for the first response, and it was another week before meaningful action occurred on the issue.
I have plenty of past experience working with AWS support on critical issues, and their support can be amazing, but it seems to be very much correlated to the amount you spend both on AWS and a paid support package.
Pricing
Joyent’s "container native" infrastructure comes with container based pricing. The smallest Docker container, 128 MB of memory, costs $0.003 per hour, or a mere $2.16 a month. The default 1 GB container is $0.026/hour or $18.72/month. The largest container listed is 64GB for $1200/month.
ECS is "free" and you pay for the underlying AWS resources, resulting in EC2 instance and cluster based pricing. A single t2.micro instance with 1 GB of memory is $0.013 per hour, or $9.36 a month. You could configure ECS to run 8 128 MB Docker containers on this no problem. Comparing RAM alone, AWS is half the price of Joyent.
Both service offer additional price competitiveness. Under the AWS Free Tier, this single t2.micro may be free for an entire year. To compete, Joyent is running a promotion where you can get $250 worth of hosting credits to try out Triton.
Unfortunately this is not apples-to-apples pricing. Joyent promises way better performance out of their containers because there is no Virtual Machine between the infrastructure and the container. The supporting data doesn’t lie and it seems quite possible to get a much better price / performance ratio on Joyent. Joyent also offers a much more granular solution. There is no way to pay for only 128 MB of memory on AWS.
More confusing is the concept of a Cluster on ECS. A best practice configuration would be to use 3 t2.smalls in 3 AZs to give your containers a chance of restarting quickly on new hardware if a single instance goes down. This will run $56/month for 6GB of capacity. The maximum container size will be limited by the single t2.small memory of 2 GB. 3 2GB containers on Triton will run $114 / month, again effectively twice the price.
Production Use
Both services are in GA. Should we begin to migrate our production service to run on Triton or ECS?
With all things "production", that depends on your service and its requirements.
For data processing, both services are absolutely ready for prime time. Instantly starting up another process or 5 to help get through a work queue is one of the places where containers shine over VMs. Both services make it easy to talk to your work queue service through the standard Docker environment techniques. Unified logging is a challenge on both services, but this is nothing new.
An HTTP service at scale raises tougher questions, the first is around load balancing. It’s here where the AWS approach of building ECS as an integration of other AWS services really shines.
ECS integrates natively with Elastic Load Balancer. Every container correctly registered the Load Balancer when it starts, de-registers when it stops, and the standard ELB health checks both stop sending traffic to unhealthy containers and instructs ECS to restart the container. Because we are extremely confident in the ELB service, an extremely well-tested, configurable and auto-scaling HTTPS Load Balancer, we can be extremely confident running a production ECS http service behind it.
How do we load balance on Triton? The "container-native" approach is to run another container. A solutions engineer at Joyent offered a recipe on how to run 2 HTTP containers linked to 1 HAProxy container, and an HAProxy configuration to round robin to the backends. This configuration is easier than ever to accomplish on Triton with the help of Docker Compose.
But personally I am not confident in this technique for production use. Adding or removing more HTTP containers safely isn’t obvious, and the availability and throughput story of a single HAProxy container is dubious.
Joyent is a seriously capable infrastructure provider, and they do offer a serious Load Balancing solution composed of Riverbed Stingray SmartMachines. I am sure that there is a good production-ready solution in here, but configuring my own load balancer instances and figuring out how to keep them in sync with where my containers are is a serious challenge.
So while I’d recommend Triton as the slickest way to get an HTTP container up and running, I can not yet recommend it for mission critical services.
Conclusion
Both Amazon and Joyent are taking containers seriously, and have put together an easy experience for running your first Docker containers in the cloud.
Joyent’s Triton wins hands-down for truly innovative container technology. The Docker Remote API and performance and security aspects of running containers via Zones will be hard to beat anywhere. On top of that, Joyent’s support is top-notch, with quick response times with meaningful solutions to technical problems. It feels like the documentation is lacking, but we can expect that to improve over time.
AWS ECS leaves a tremendous amount of work up to the user. The ECS APIs and Agent does automate defining, starting, monitoring and restarting containers around service or instance problems, no small feat. But it requires the user to configure and manage numerous additional EC2 resources.
This experience can be challenging, yet this ends up being a benefit in many ways. AWS has a tremendous eco-system of services that are immediately available to ECS. CloudFormation in particular makes it manageable to maintain and update your cluster AMIs, and Docker image versions. Elastic Load Balancers are one of the best ways to front production workloads.
While AWS might not have the best experience around managing containers, it is still the best place to manage and run production HTTP applications.
But Joyent’s offering is really compelling, and it might already be the best place to manage and run less important apps, like a chat bot or similar utility app. There is no question Joyent is capable of serving any production HTTP application, however it’s the load balancer and ecosystem of other operational services that matter, not how easy running a single container is.
[1] https://aws.amazon.com/blogs/aws/ec2-container-service-ready-for-production-use/ [2] https://www.joyent.com/about/press/joyent-ready-to-run-multi-tenant-containers-without-vms
4
1304
