How Sendwithus Sent Their First Billion Emails

2,442
D732d0fdd6c97f045433a60266ef9c92
sendwithus
Sendwithus provides a framework and tools for developers, designers, and marketers to all contribute to the same email workflow.

Editor's note: By Brad Van Vugt, ‎Cofounder at Sendwithus


What is Sendwithus?

Sendwithus is a platform for sending templated emails. We make it easy for product teams to manage and track automated emails being sent to their customers.


awesometemplate management with sendwithus


Customers use Sendwithus for

  • Welcome and Onboarding Emails
  • Notifications and Invitations
  • Product Updates
  • Invoices
  • Password Resets
  • etc

Typically automated emails are stored in source code. Making changes requires a ridiculous amount of time and effort, or in most cases, isn’t even possible. This is particularly true for larger organizations with multiple teams.

Our goal is to make automated emails collaborative and easy to manage. Instead of hardcoding email templates in HTML files, developers integrate with our API and use Sendwithus to store, send, manage, and update templates in real-time. Customers can view detailed analytics, run A/B tests, schedule drip campaigns, and manage localized content without ever updating code.


Sendwithus-cli in action


Today we power more than 10M emails each day, and process over 100M API requests between our customers and their Email Service Provider (ESP).

Engineering at Sendwithus

Our engineering team is relatively small, given the scale of our service. We have six full-time engineers and most of us work on all aspects of the platform, from backend architecture, to frontend interfaces, to open source projects.

We also have two customer success engineers that serve as our QA and support team, hunting down and resolving bugs as they occur and rigorously field-testing new features.


the sendwithus team


Initial Architecture

Sendwithus started in late 2012 as a static Django application, deployed to Amazon EC2. We ran several product and marketing experiments (affectionately named Sendwithus Zero) in an effort to learn more about who our customers were and what problems they were looking to solve.

When the time came to actually build a working product, we stuck with Django. We added Gunicorn to power our web servers, and a single PostgreSQL database storage layer to the backend.

We subscribed to the MonolithFirst pattern, with a single code base responsible for frontend, backend, api, and background workers. It worked great and let our engineering team (of two) move lightning fast.

We also switched from AWS to Heroku, taking advantage of Heroku Postgres. This was one of the best decisions we made early on. Heroku has its disadvantages (namely price and limited configuration control) but not worrying about deployment procedures and server maintenance allowed us to build and iterate very fast, focusing on what matters most. Today we still use Heroku to power some of our applications, simply because it’s so easy and fast to develop on. Our higher throughput applications have moved to AWS, primarily due to operating cost.

Early Scaling

As our first API customers came on board, we added RabbitMQ and Celery for background job processing. We initially chose a hosted RabbitMQ provider through the Heroku Add-on system. This was ok, but didn’t scale very well; more on that later.

We also added MongoDB for user-facing analytics, and Memcache to speed up our servers. We also used this Heroku Buildpack for PostgreSQL connection pooling (we still use a modified version of that buildpack today).

Our Django, PostgreSQL, Heroku stack served us well and apart from some performance tweaking here and there, got us through the early months with ease.

Queueing Millions of Events

Queueing makes up a large portion of our infrastructure needs, from queueing emails for rendering and delivery, to queueing bounce and click events for processing. Our queueing systems must also handle large, unexpected bursts of traffic.


event queue throughput


Our hosted RabbitMQ + Celery solution was first to show signs of scaling problems. We’d outgrown the capabilities of a single RabbitMQ instance and needed something that could handle more throughput.

We opted to try IronMQ, which offers a great hosted queueing service. We used IronMQ for almost a year before switching to AWS SQS. SQS has been rock solid and reasonably priced; I can’t imagine switching off anytime soon. Today we operate 50+ queues, pushing more than 100M messages/day.

Processing and Storing Millions of Events

Our second major hurdle was PostgreSQL throughput. As our customers grew so did our database needs. We had nearly 2TB of email logs and metadata data in our single PostgreSQL instance, and we were adding more than 10 GB each day.

We explored a wide variety of options, including Cassandra, MongoDB, and CitusDB, but ultimately landed on AWS DynamoDB. Our engineering team has written about our migration to DynamoDB and performance issues we ran into along the way. Today we’re very happy to have migrated when we did.

Current Architecture: Dedicated Applications and Services

To date we’ve processed nearly two billion emails for our customers. We still run the original Django monolith, but we’ve moved core pieces of functionality into dedicated applications and services. These aren’t exactly microservices - some of them are full-fledged public facing apps with complex storage layers.

The entire Sendwithus product is divided into five major applications: Landing, Application, API, Delivery, and Analytics. Each application started as a fork of the original Django project and was allowed to evolve on it’s own timeline. This meant storage layers, deployment methods, even programming languages could be selected specifically for each application.

For example, our Analytics application is currently written in Go and uses a combination of Redis, PostgreSQL, and DynamoDB to achieve high-throughput analytics and events processing. It deploys to Heroku and uses net/http and HttpRouter to serve web requests.

Our Landing application is a Jekyll project that powers our public web pages. It’s uses gulp to compile and deploys to Cloudfront using AWS S3. We also build open source projects whenever possible. Our API Docs, Victoria Startup Job Board and Template Gallery are powered by separate, open source repositories. This is great for collaborating with our customers; it’s pretty cool to show a customer a commit that fixes a bug they’ve reported.


another satisfied customer


Monitoring

As a general rule, each application uses common monitoring and logging pieces:

Developer Workflow

We use Github to store all our code repositories. Each repo maintains multiple branches, and we use Github Pull Requests for code reviews. This works great for now, but will likely change as the team grows.

We’ve also built a custom command line tool that allows anyone on our team to deploy any application by simply running the command swu deploy. The deploy process will run tests locally and then perform a rolling deploy of the application. Deployment logs are piped into Slack, providing our team with a comprehensive and searchable deploy history.

The Future

Our team is beginning to widely adopt Go, Docker, CoreOS, and EC2 for new infrastructure projects, and I imagine that will only increase as we scale. It’s likely we’ll continue to use Python, Django, and Heroku for low-throughput and UI heavy applications.

Looking to the future, we’re working to break our major applications down into smaller microservices. What these services do and how much code is shared between them is still to be determined, although we already have some good ideas.. And of course we'll continue to use our own API to power our emails :)

D732d0fdd6c97f045433a60266ef9c92
sendwithus
Sendwithus provides a framework and tools for developers, designers, and marketers to all contribute to the same email workflow.
Frontend Developer
Victoria BC
Sendwithus is looking for a Frontend Developer to build interfaces that our customers love to use. Great candidates will enjoy building modern web UIs with tools like ES6, React, Redux, Sass, and GraphQL. You will work alongside our Product Designer and Product Manager to architect and develop powerful user experiences. This position offers incredible room for professional growth and autonomy as our user base grows and our team expands. The Sendwithus platform is used by over 500 companies every month, including several Fortune 500 enterprises. Our Product Development Team is responsible for creating, building, and constantly improving every user interaction with our platform.
  • Ramp up on our tech stack and frontend architecture, working closely with senior members of the product development team
  • Complete your first project and deploy those changes to our production environment
  • Work with our Product Manager and Product Designer to prototype new UI and UX ideas
  • Identify performance and process bottlenecks, propose solutions, and lead implementation
  • Actively participate in product planning sessions
  • Gather customer feedback to learn and address common pitfalls
  • Lead development of new UI features from start to finish.
  • Developed technical best practices around UI architecture and development
  • Experimented with creative solutions to interesting and challenging UI problems
  • Collected and analyzed customer feedback regarding existing UI
  • Mentored junior members of the Product Development Team
  • React, Redux, GraphQL
  • ES6, CSS, HTML, NodeJS
  • Docker, Webpack
  • Comments
    Open jobs at sendwithus
    Full Stack Engineer
    Victoria BC
    Sendwithus is looking for a Full Stack Engineer to join our Product Team and help develop all aspects of the Sendwithus web application. You will work with the Platform Team to spec out new technical projects and gather feedback from the Customer Success team to ensure our product is top-notch. Our Product Team is cross-functional and you'll be working alongside frontend developers and UI/UX designers to execute on ambitious product roadmaps. You will design, architect, and optimize software that powers the entire user experience. We believe that user-centric design ultimately leads to the best products, so we listen very closely to our customers. We care about how code impacts their daily lives and work hard to constantly deliver exceptional experiences.
  • Get ramped up on our tech stack and architecture, working closely with senior members of the engineering team
  • Complete your first project and deploy code to production systems
  • Work with the Product Manager and Customer Success team to identify performance bottlenecks, propose solutions and lead implementations
  • Actively contribute to product planning sessions
  • Build out traffic, performance and user action monitoring systems to collect data for future feature development
  • Lead user-facing technical projects from start to finish
  • Be responsible for making high level software architecture and system design decisions for the Product Team
  • Bring focus to the details when it comes to UI performance and design
  • Constantly experiment with creative solutions and new technologies to create exceptional customer experiences
  • Ensure that customer feedback is taken into consideration during any/all feature planning
  • JavaScript, CoffeeScript, Python
  • KnockoutJS, React, Django, Node
  • PostgreSQL, Memcache
  • AWS, Heroku, Docker
  • QA Engineer
    Victoria BC
    Sendwithus is looking for a QA Engineer to join our Product Team. Great candidates will want to own the QA process for the Sendwithus web application. This position has a lot of room for growth, leadership and independence. You will be partnering with our Customer Success team to isolate and reproduce application bugs and inconsistencies, lead QA testing parties, and work with the Product Development Team to ensure an excellent customer experience across new features and software releases. As a member of our growing Product team you will become the voice for proper product testing and acceptance testing best practices. You will be working closely with our Product Manager to spec out product roadmaps, implementation planning, and feature acceptance testing. We believe that user-centric design and development leads to the best product experiences, and we really care about how our code impacts our customers. As the QA Engineer on our close-knit, cross-functional team, you’ll be an active voice in ensuring Sendwithus is continuously delivering the highest quality service to our customers.
  • Develop an expert knowledge of the Sendwithus feature set
  • Explore the customer on-boarding journey and identify areas of improvement
  • Work with the Product Manager to define and implement acceptance tests for new features
  • Work with the Customer Success team to debug and triage customer facing issues
  • Define what testing is currently being done manually and begin creating automation tools
  • Drive experimentation with creative software solutions and new technologies when it comes to testing and automation
  • Work with large, unstructured datasets to determine feature usage patterns and error rates
  • Help spec out new product features and UI requirements
  • Build out a comprehensive automation system to replace and improve manual testing processes
  • Ensure we are delivering the best possible product to our customers
  • Drive QA best practices across all teams
  • JavaScript, Python, and other scripting languages
  • SQL
  • Jenkins CI, Github
  • Unix shell scripting
  • Developer tooling
  • Senior Platform Engineer
    Victoria BC
    Sendwithus is looking for a Senior Platform Engineer to join our growing engineering team. We are looking for someone who is passionate about building distributed systems and highly available web applications. As a Senior Platform Engineer you will be architecting scalable software systems, optimizing performance, and designing creative solutions for challenging technical problems. This position offers a lot of room for professional growth and independence as our system scales and our team expands. The Sendwithus platform is responsible for over 30 million emails each day and handles thousands of requests per second. Our Platform Engineering Team ensures that our system runs smoothly, reliably, and securely. Above all else, we are customer focused; we care about how are code impacts customers.
  • Complete your first project and deploy those changes to production systems
  • Get ramped up on our tech stack and architecture while working closely with other senior members of the Platform team
  • Begin to monitor, troubleshoot, and improve system services
  • Be responsible for your first on-call rotation and technical triage
  • Actively contribute to platform roadmap and planning discussions
  • Identify performance bottlenecks, propose solutions, and lead implementation
  • Raise the bar for our engineering team by defining and iterating on engineering best practices
  • Work with engineering leads to define engineering best practices
  • Take complete ownership of a primary backend system; big-data storage layer, email delivery pipeline, API service, etc.
  • Deliver high throughput production services, ensuring high-availability
  • Lead the creation and implementation of large technical projects from start to finish
  • Make high level system architecture and software design decisions
  • Mentor new members of the engineering team
  • Go, Python
  • PostgreSQL, CassandraDB, Redshift, Redis, Memcache
  • AWS and Heroku infrastructure
  • Microservice Architecture, Docker, Kubernetes, GRPC
  • Full Stack Product Engineer
    Victoria BC
    Sendwithus is looking for a Full Stack Engineer to join our Product Team and help develop all aspects of the Sendwithus web application. You will work with the Platform Team to spec out new technical projects and gather feedback from the Customer Success team to ensure our product is top-notch. Our Product Team is cross-functional and you'll be working alongside frontend developers and UI/UX designers to execute on ambitious product roadmaps. You will design, architect, and optimize software that powers the entire user experience. We believe that user-centric design ultimately leads to the best products, so we listen very closely to our customers. We care about how code impacts their daily lives and work hard to constantly deliver exceptional experiences.
  • Get ramped up on our tech stack and architecture, working closely with senior members of the engineering team
  • Complete your first project and deploy code to production systems
  • Work with the Product Manager and Customer Success team to identify performance bottlenecks, propose solutions and lead implementations
  • Actively contribute to product planning sessions
  • Build out traffic, performance and user action monitoring systems to collect data for future feature development
  • Lead user-facing technical projects from start to finish
  • Be responsible for making high level software architecture and system design decisions for the Product Team
  • Bring focus to the details when it comes to UI performance and design
  • Constantly experiment with creative solutions and new technologies to create exceptional customer experiences
  • Ensure that customer feedback is taken into consideration during any/all feature planning
  • JavaScript, CoffeeScript, Python
  • KnockoutJS, React, Django, Node
  • PostgreSQL, Memcache
  • AWS, Heroku, Docker
  • Verified by
    You may also like
    E-Commerce at Scale: Inside Shopify's Tech Stack
    How SendGrid Scaled to 40 Billion Emails Per Month
    How Stream Built a Modern RSS Reader With JavaScript
    How Heap Built an Analytics Platform that Auto-Tracks Every User Event