Keeping AWS secrets secret with ECS

2,814
Remind
Remind is a messaging app for teachers, students, and parents to safely and easily communicate with each other. With more than 30 million users, we are one of the fastest-growing companies in edtech. We're hiring team members inspired by technology's potential to transform education, energized about solving the communication challenges that teachers face every day, and passionate about our vision of connecting every teacher, student, and parent in the world to improve education.

By Eric Holmes, Engineer at Remind.


One of the most important aspects of our security strategy in the Operations Engineering Team is to mitigate the risk of leaked AWS credentials. Even if you follow AWS best practices of putting your infrastructure within a VPC, leaking AWS credentials provides keys to the castle.

This post describes the strategy that we take to reduce the probability of AWS credentials being leaked, as well as reducing the risk in the event that they are leaked.

Background

In the beginning, there were long lived credentials.

The most straight forward way to give an application access to make requests to AWS APIs is to create an IAM user, generate an access key, and then pass the access key id and secret to the application. However, this has a number of problems:

  1. Trust. How do you pass these values to an application securely, so that they can't be accessed by unauthorized parties, and aren't stored in plaintext? In ECS, it would be bad practice to include AWS access keys as plaintext environment variables in task definitions. Any application or service that has the ecs:DescribeTaskDefinition permission will have access to all secrets.
  2. Credential Lifetime: IAM access keys don't automatically expire. The longer an access key lives, the higher the probability is that it has been accidentally, or maliciously leaked. Accidental leaks from human error are a surprisingly common source of incidents.

Instance Profiles

An alternative method is to use EC2 Instance Profiles. Instance profiles allow you to attach an IAM role to an EC2 instance, and applications running on the host can access an “instance metadata” endpoint to obtain temporary AWS credentials. This solves both trust (The EC2 instance authenticates itself with AWS) and credential expiration (credentials obtained from instance metadata only last for 1 hour, greatly minimizing the impact of a leak).

However, in the context of ECS, instance profiles have their own set of problems:

  1. Granularity: In ECS, you'll likely have many different services and applications running on the same host. To use credentials from an instance profile would require that we give the host a combination of all required permissions needed for each service, instead of giving each service it's own granular permissions. This would be a strict violation of the principle of least privilege.
  2. Unprotected Instance Metadata: It may not initially seem obvious, but instance metadata can be an incredibly dangerous feature. All it takes is an arbitrary GET request to http://169.254.169.254/ to obtain AWS credentials. What happens when software running on the host introduces a feature that downloads files from a URL? This potential exploit is best described in http://www.daemonology.net/blog/2016-10-09-EC2s-most-dangerous-feature.html.

Enter ECS Task Roles

ECS introduced Task Roles, which are similar to Instance Profiles, allowing you to attach an IAM role to an ECS task. This seems to solve all of our problems:

  1. Granularity: ECS tasks only get the exact permissions they need.
  2. Credential Lifetime: Credentials expire after 1 hour and are automatically rotated.
  3. Unprotected Instance Metadata: When obtaining credentials from ECS task roles, like instance profiles, the client within the container hits a metadata endpoint (http://169.254.169.254), which routes to the ECS agent running on the host. However, unlike instance metadata, the URL where credentials can be obtained is dynamically generated, and provided to the container via an environment variable. If an attacker gained access to make arbitrary GET requests, they would also need to know the value of the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable within the container.

At Remind, we use Stacker to manage all of our infrastructure, and then we run our services and applications with Empire. Through Stacker, we have a base “blueprint” for each Empire app which (among other things):

  1. Creates an IAM role that can be assumed by ECS.
  2. Creates a KMS key that the app can use to encrypt/decrypt SSM parameters (we may talk more about this in a future post).
  3. Passes the role to the Empire application.

Doing this ensures that we have a common starting point and convention for managing how all of our applications and services access AWS API's.

Combining the two

Not everything that we run on our ECS container instances gets run with ECS/Docker. We wanted to be able to continue using instance profiles for software running outside of Docker (generally infrastructure processes, like the Amazon SSM/ECS agents), but with the assurance that our Docker containers (user facing applications) would not be able to access userdata, or IAM credentials from the instance profile.

To do this, any requests from Docker containers going to the instance metadata endpoint get re-routed to an nginx proxy on host. This proxy denies the container access to the instance metadata endpoints for userdata and IAM credentials:

daemon off;
error_log stderr;
events {
  worker_connections 1024;
}

http {
  log_format request 'method=$request_method uri="$request_uri" host=$host ua="$http_user_agent" remote=$remote_addr status=$status';
  access_log /dev/stdout request;

  server {
    listen 127.0.0.1:7823;

    # Disallow access to credentials obtained from the instance profile.
    location ~ ^\/.*\/meta-data\/iam\/security-credentials\/.* {
      return 403;
    }

    # Disallow access to user data. In general, we shouldn't be putting secrets
    # in user data, but just in case...
    location ~ ^\/.*\/user-data.* {
      return 403;
    }

    # Proxy all other request through to instance metadata.
    location / {
      proxy_pass http://169.254.169.254:80;
    }
  }
}

# vi: ft=nginx
$ iptables -t nat -I PREROUTING -p tcp -d 169.254.169.254 --dport 80 \
  -j DNAT --to-destination 127.0.0.1:7823

This has the added benefit that any requests to instance metadata initiated from a Docker container gets logged and forwarded to our log aggregation service.

With the above in place, if an application or service that we run with ECS were to introduce an exploit that allowed an attacker to make arbitrary GET requests, we can be more confident that the attacker won't be able to obtain AWS credentials.

Remind
Remind is a messaging app for teachers, students, and parents to safely and easily communicate with each other. With more than 30 million users, we are one of the fastest-growing companies in edtech. We're hiring team members inspired by technology's potential to transform education, energized about solving the communication challenges that teachers face every day, and passionate about our vision of connecting every teacher, student, and parent in the world to improve education.
Tools mentioned in article
Senior Android Engineer
San Francisco

About Us

Remind is a communication platform that helps educators reach students and parents where they are: their phones. With 31 million active users, we’re one of the fastest-growing companies in education technology, but we have our sights set on something bigger: giving every student the opportunity to succeed.

We're hiring team members inspired by the potential to transform education, motivated to solve communication challenges in education, and passionate about our vision of connecting every teacher, student, and parent in the world.

Learn more about our Engineering Team here, and the exciting projects like Empire and MVP that we've been working on. Our Android app was featured at Google IO and this year ascended to #1 free app in the Google Play Store. We're heavily invested in Kotlin, TypeScript, and GraphQL and looking for engineers that enjoy working in highly collaborate cross-functional teams.

Our investors include First Round Capital, GSV, Kleiner Perkins Caufield Byers, and Social Capital, and we want you to join us.

Responsibilities

  • Define, write, and own product features that are used by millions of users
  • Maintain high level of stability, performance, and quality
  • Collaborate with designers and engineers to help create a fantastic experience for our users

Requirements

  • You have built and shipped an Android product to a significant user base
  • You have strong Kotlin, Java and Android programming skills
  • You are passionate about writing quality, maintainable, and performant code
  • You communicate complex ideas effortlessly, both in person and in writing

Benefits and Perks

  • Competitive salary and equity
  • 100% health coverage for you and your dependents
  • Open vacation policy
  • Parental leave
  • Catered meals and fully stocked kitchen
  • Parking and commuter benefits

Remind is an equal opportunity employer, and we're committed to diversity and inclusion in the workplace. We aim to represent the students, teachers, and parents we serve, and we welcome, support, and empower all the diverse individuals in our community.

Comments
Open jobs at Remind
Engineering Manager
San Francisco

About Us
Remind is a messaging app that helps teachers, students, and parents communicate quickly and efficiently. We believe that when communication improves, relationships get stronger. Education gets better.

We're hiring team members inspired by the potential to transform education, motivated to solve communication challenges in education, and passionate about our vision of connecting every teacher, student, and parent in the world. With more than 20 million active users, Remind is one of the fastest-growing companies in education technology.

Learn more about our Engineering Team here, and the exciting projects like Empire and Group Chat that we've been working on. We have 4 cross-functional/full stack product teams, 2 infrastructure teams, an operations engineering team and a data team.

We’ve raised $59 million in funding from Kleiner Perkins, Social Capital, and First Round Capital to bring us closer to achieving our goals, and we want you to join us.

Tech Stack
On the back-end, we are built on leading edge technologies, including Go, Ruby, node.js, Postgres, RabbitMQ, Redis, DynamoDB and more.

Our data stack consists of Interana, AWS Redshift with Mode Analytics, and we're experimenting with AWS Athena.

On the front-end, our web app is built using React, our iOS app is mostly in Swift, Realm, and MVVM, and our Android app is built on Java and MVP.

We have good automated test coverage and we push code into production multiple times per day. We welcome new ideas and experiences and like trying new things and taking measured risk.

Responsibilities:

  • Lead and inspire your team to build innovative product features
  • Demand that your team build robust, maintainable systems that seamlessly scale and perform
  • Expect operational efficiency from every member of your team
  • Collaborate with product and design to identify and specify features to delight our parents, teachers, and students
  • Enable your engineers to do more than they think that can, every day
  • Proactively manage the performance and engagement of your team
  • Expect the best from your team, your peers, and our VP of Engineering

Requirements:

  • Proven capability to set clear expectations and enable and require teams to meet and exceed those expectations
  • Proven track record of shipping code into production as part of a team
  • Proven track record of building teams that ship code
  • Proven ability to build and ship web/mobile/service applications to real users
  • Mastery of software engineering with deep experience writing, shipping, and maintaining code
  • Strong fundamentals in computer science
  • Background in distributed systems and networking
  • Proponent of writing comprehensive unit and integration tests
  • Experience with some of the following technologies: Postgres, RabbitMQ, node.js, redis, AWS, NoSQL, golang

Benefits and Perks

  • Competitive salary and equity
  • 100% health coverage for you and your dependents
  • Open vacation policy
  • Parental leave
  • Catered meals and fully stocked kitchen

Remind is an equal opportunity employer.

Android Engineer
San Francisco

About Us

Remind is a messaging app that helps teachers, students, and parents communicate quickly and efficiently. We believe that when communication improves, relationships get stronger. Education gets better. 

We're hiring team members inspired by the potential to transform education, motivated to solve communication challenges in education, and passionate about our vision of connecting every teacher, student, and parent in the world. With more than 20 million active users, Remind is one of the fastest-growing companies in education technology. 

Learn more about our Engineering Team here, and the exciting projects like Empire and Group Chat that we've been working on.

We’ve raised $59 million in funding from Kleiner Perkins, Social Capital, and First Round Capital to bring us closer to achieving our goals, and we want you to join us.

Responsibilities

  • Build new and innovative product features
  • Maintain high level performance and quality
  • Collaborate with designers and engineers alike to create an enjoyable user experience
  • We expect our engineers to play a role in deciding product strategy
  • Push code on your first day on the job with the goal of releasing to the Play Store every two weeks

Requirements

  • You have built and shipped an Android product to a large customer base
  • You have strong Java programming skills
  • You are obsessed with writing quality, maintainable code with extensive test coverage
  • You communicate complex ideas with ease, both in person and in writing
  • You enjoy debate and iteration as much as headphones-on coding

Benefits and Perks

  • Competitive salary and equity
  • 100% health coverage for you and your dependents
  • Open vacation policy
  • Parental leave
  • Catered meals and fully stocked kitchen
  • Health and education reimbursement
  • Parking and commuter benefits

Remind is an equal opportunity employer.

Backend Engineer
San Francisco

About Us
Remind is a messaging app that helps teachers, students, and parents communicate quickly and efficiently. We believe that when communication improves, relationships get stronger. Education gets better.

We're hiring team members inspired by the potential to transform education, motivated to solve communication challenges in education, and passionate about our vision of connecting every teacher, student, and parent in the world. With more than 22 million active users, Remind is one of the fastest-growing companies in education technology.

You can learn more about our Engineering Team here, and the exciting projects like Empire and Group Chat that we've been working on.

We’ve raised $59 million in funding from Kleiner Perkins, Social Capital, and First Round Capital to bring us closer to achieving our goals, and we want you to join us.

Tech Stack
On the back-end, we are built on Go, Ruby, node.js, Postgres, Kinesis, Redis, DynamoDB, Amazon Redshift, and use CD to always be pushing to production.

Responsibilities

  • Write and run code that delivers hundreds of millions of SMS, email and push messages per month and growing rapidly
  • Maintain and constantly improve scale, performance, and quality
  • Help formulate product strategy
  • Delight our users (teachers, parents and students) with new functionality and by improving site and message delivery scale, performance, and reliability
  • Ensure elasticity and scalability of our systems throughout the year
  • Push code on your first day on the job

Requirements

  • You have consistently shipped code to production as part of a team
  • You have built scalable, performant, highly available services and understand the value of a good SLA
  • You have mastered multiple programming languages, including both statically and dynamically typed languages.
  • You have built, owned, and operated distributed systems
  • You are adept at writing pragmatic, comprehensive unit and integration tests
  • You understand relational and non-relational persistence models, the CAP theorem, and enjoy reading the sundry works of Leslie Lamport
  • You have a solid grasp of the art of computer science and the science of software engineering
  • Have experience with most of the following technologies: Postgres/MySql/etc, DynamoDB, RabbitMQ, redis, AWS

Benefits and Perks

  • Competitive salary and equity
  • 100% health coverage for you and your dependents
  • Open vacation policy
  • Parental leave
  • Catered meals and fully stocked kitchen
  • Parking and commuter benefits

Remind is an equal opportunity employer.

JavaScript Engineer
San Francisco

About Us

Remind is a messaging app that helps teachers, students, and parents communicate quickly and efficiently. We believe that when communication improves, relationships get stronger. Education gets better.

We're hiring team members inspired by the potential to transform education, motivated to solve communication challenges in education, and passionate about our vision of connecting every teacher, student, and parent in the world. With more than 22 million active users, Remind is one of the fastest-growing companies in education technology.

You can learn more about our Engineering Team here, and the exciting open projects like Empire that we've been working on.

We’ve raised $59 million in funding from Kleiner Perkins, Social Capital, and First Round Capital to bring us closer to achieving our goals, and we want you to join us.

Tech Stack

  • React with Redux
  • Flow type checking
  • ES2017
  • Node service for server-side rendering
  • Migrating from REST to GraphQL with Apollo stack

Responsibilities

  • Write and operate code that supports millions of teachers, parents, and students to better collaborate
  • Work closely with our Product Managers, UX/UI designers and fellow engineers to roll out a set of highly interactive web application features for Remind
  • Maintain and constantly improve scale, performance, and quality
  • Build new and innovative product features and help formulate product strategy
  • Push code on your first day on the job

Requirements

  • You have consistently shipped web applications to a large number of users
  • You have a strong body of prior front end web work, including significant projects written in JavaScript
  • You have a good working knowledge of HTML, CSS, and one or more of AngularJS, Ember.js, or React
  • You have a deep understanding of closures, prototypal inheritance, DOM manipulation, HTTP, web security, cross browser compatibility challenges
  • You have experience writing automated tests in JavaScript

Benefits and Perks

  • Competitive salary and equity
  • 100% health coverage for you and your dependents
  • Open vacation policy
  • Parental leave
  • Catered meals and fully stocked kitchen
  • Parking and commuter benefits

Remind is an equal opportunity employer.

Verified by
You may also like
E-Commerce at Scale: Inside Shopify's Tech Stack
How SendGrid Scaled to 40 Billion Emails Per Month
How Stream Built a Modern RSS Reader With JavaScript
How Heap Built an Analytics Platform that Auto-Tracks Every User Event