How 500px serves up over 500TB of high res photos

46,703
500px
The premier community for photography.

By Paul Liu, DevOps Developer, at 500px. You can find him here on StackShare and on Twitter


Background

500px is an online community for premium photography. Millions of users all over the world share, discover, sell and buy the most beautiful images. We value design, simplicity in the code, and getting stuff done.

I'm a DevOps Developer at 500px working on the Platform Team. I work on stuff like backend systems, monitoring, configuration management, and deployment and automation. Prior to joining 500px, I spent many years as a sysadmin working in the insurance industry.

Engineering at 500px

The 500px Engineering team is split into four groups: the Web team, the Mobile team, the QA/Release Engineering team and my team, the Platform team, which is a combined team that handles building our API and backend services as well as technical operations and infrastructure management.

Our teams are highly cross functional and boundaries are fairly loose, so engineers wind up moving around a lot between teams to work on specific projects. This helps us spread knowledge and prevent siloing. There is also a very tight communications loop between engineering, product, design and the customer excellence teams, which helps to keep us honest, agile, and focused on delivering the right things.


A diverse group of people


General Architecture

The architecture of 500px can be thought of as a large Ruby on Rails monolith surrounded by a constellation of microservices. The Rails monolith serves the main 500px web application and the 500px API, which powers the web app, mobile apps and all our third party API users. The various microservices provide specific functionality of the platform to the monolith, and also serve some API endpoints directly.

The Rails monolith is a fairly typical stack: the App and API servers serve requests with Unicorn, fronted by Nginx. We have clusters of these Rails servers running behind either HAProxy or LVS load balancers, and the monolith's primary datastores are MySQL, MongoDB, Redis and Memcached. We also have a bunch of Sidekiq servers for background task processing. We currently host all of this on bare metal servers in a datacenter.

The microservices are a bit more interesting. We have about ten of them at the moment, each centered around providing an isolated and distinct business capability to the platform. Some of our microservices are:

  • Search related services, built on Elasticsearch
  • Content ingestion services, in front of S3
  • User feeds and activity streams, built on Roshi and AWS Kinesis
  • A dynamic image resizing and watermarking service
  • Specialized API frontends for our web and mobile applications

We run our microservices in Amazon EC2 and in our datacenter environment. They are mostly written in Go, though there are a couple outliers which use NodeJS or Sinatra. However, regardless of the language in use, we try to make all of our microservices good 12-factor apps, which helps to reduce the complexity of deployments and configuration management. All of our services are behind an HAProxy or an ELB.


Hard at work at 500px


The microservices pattern is great because it allows us to abstract away complex behaviour and domain-specific knowledge behind APIs and move it out of the monolith. Front-end product teams consuming these services only have to know about the service's API, and service maintainers are free to change anything about their service (as long as they maintain that API!). For example, you can query the search service without having to know a thing about Elasticsearch. This flexibility has proven to be extremely powerful for us as we evolve our platform because it lets us try out new technologies and new techniques in a safe and isolated way. If you are curious about implementing microservices yourself, former 500pxer and overall rad dude Paul Osman gave a great talk at QConSF last year about how and why we did it. My face is on one of the slides, but that is only one of the many reasons why this talk is awesome.

Image Processing

Probably the most interesting set of microservices we run at 500px are the ones having to do with serving images and image processing. Every month we ingest millions of high resolution photos from our community and we serve hundreds of terabytes of image traffic from our primary CDN, Edgecast. Last month we did about 569TB of total data transfer, with 95th percentile bandwidth of about 2308Mbps. People really like looking at cool pictures of stuff!


A representative cool picture. Also, 500px's home town, Toronto


To ingest and serve all these images we run a set of three of microservices in EC2, all built around S3, which is where we store all of our images. All three of these services are written in Go. We really like using Go for these cases because it allows us to write small, fast, and concurrent services, which means we can host them on fewer machines and keep our hosting costs under control.

The first microservice users encounter when they upload a photo is one we call the Media Service. The Media Service is fairly simple: it accepts the user's upload, does some housekeeping stuff, persists to S3, and then finally enqueues a task into RabbitMQ for further processing.

Next, consuming those tasks off of RabbitMQ is another service called (creatively) the Converter Service. The Converter Service downloads the original image from S3, does a bunch of image processing to generate various-sized thumbnails, and then saves these static conversions back to S3. We then use these conversions in lots of places around our site and for our mobile apps.

Probably so far this isn't very surprising for a photo-sharing website, and, for awhile, these two services did everything we needed -- we simply set the S3 bucket containing the resulting thumbnails as the origin for our CDN. However, as the site continued to grow, we found this solution was pretty costly and space inefficient, as well as not very flexible when new products required new sizes.

To solve this problem, we recently built what we creatively call our Resizer Service (yes, we tend to choose descriptive names for these things). This new service now acts as the CDN origin and dynamically generates any size or format of image we need using the S3 original. It can also watermark the image with a logo and apply photographer attribution, which is reassuring to our community.

The Resizer Service is fairly high throughput, with the cluster handling about 1000 requests per second during peak times. Doing all this resizing and watermarking is pretty compute-intensive, so it's a bit of a challenge to keep response times reasonable when the load is high. We've worked really hard on this problem, and at peak traffic we're able to maintain a 95th percentile response time that is below 180ms. We do this through the use of a really cool, really fast image processing library called VIPS, aggressive caching, and by optimizing like crazy. Outside of peak hours, we can usually get below 150ms.

And we're not done with this problem yet! There are almost certainly more optimizations to be found, and we hope to keep pushing those response times down further and further in the future.

Workflow

We use Github and practice continuous integration for all of our primary codebases.

For the Rails monolith, we use Semaphore and Code Climate. We use a standard rspec setup for unit testing, and a smaller set of Capybara/Selenium tests for integration testing. Fellow 500pxer and professional cool guy Devon Noel de Tilly has written at length about how we use those tools, so I won't try to out do him -- just go check it out.

For our Go microservices, we use Travis CI to run tests and to create Debian packages as build artifacts. Travis uploads these packages to S3, and then another system pulls them down, signs them, and imports them into our private Apt repository. We use FPM to create packages, and Aptly to manage our repos. Lately, though, I've trying out packagecloud.io and I really like it so far, so we may be changing how we do this in the near future.

For deployments, we use a combination of tools. At the lowest level we use Ansible and Capistrano for deploys and Chef for configuration management. At a higher level, we've really embraced chatops at 500px, so we've scripted the use of those tools into our beloved and loyal Hubot friend, BMO.


Headquarters in snowy Toronto


Anyone at 500px can easily deploy the site or a microservice with a simple chat message like bmo deploy <this thing>. BMO goes out, deploys the thing, and then posts a log back into the chat. It's a simple, easy mechanism that has done wonders to increase visibility and reduce complexity around deploys. We use Slack, which is where you interact with BMO, and it makes everything really nicely searchable. If you want to find a log or if you forget how to do something, all you have to do is search the chat. Magical.

Other Important Apps

We monitor everything with New Relic, Datadog, ELK (Elasticsearch, Logstash and Kibana), and good old Nagios. We send all our emails with Mandrill and Mailchimp and we process payments with Stripe and Paypal. To help us make decisions, we use Amazon's Elastic MapReduce and Redshift, as well as Periscope.io. We use Slack, Asana, Everhour, and Google Apps to keep everyone in sync. And when things go wrong, we've got Pagerduty and Statuspage.io to help us out and to communicate with our users.

The Future, Conan?

Right now I'm working on experimenting with running our microservice constellation in Docker containers for local dev (docker-compose up), with an eye to run them in production in the future. We've got a CI pipeline working with Travis and Docker Hub, and I'm really excited by the potential of cloud container services like Joyent Triton and Amazon ECS. As we build more and more microservices and expand the stack, we're also looking at service discovery tools like Consul and task frameworks like Mesos to make our system scale harder better and faster.


A long and winding road


More Faster Is More Better

We're expanding quickly and hiring all kinds of positions. We're looking for DevOps types, Backend and Frontend developers, Mobile developers (both Android and IOS), UX designers, and sales people. We build cool stuff, we're passionate, and we're flying by the seat of our pants at breakneck speed, building the best thing we know how to make. If you like doing awesome cool stuff and you aren't afraid to get your hands dirty, come join us.

500px
The premier community for photography.
Tools mentioned in article
Engineering Manager, Web & Mobile
Toronto

The Company

500px is an online community for sharing, discovery, buying, and selling of high-quality photography. Every month, millions of people use our website and mobile apps from around the world to find the world’s most inspiring photography.

We take pride in the products we ship and love what we do. Our engineering culture values mentorship, ownership, collaboration, and getting stuff done.

Our technology stack includes: React, Redux, ES6, Ruby on Rails, Python, Go, MySQL, MongoDB, Redis, ElasticSearch, and various Amazon AWS services. Check out our Engineering Blog for more.

The Job

We are looking for an experienced Engineering Manager who can lead the web and mobile developers, and bring the 500px platform to the next level of scale, speed, and user experience.

Role Responsibilities: 

  • Recruit, lead and mentor the web and mobile teams to design, implement, and optimize applications, to support 500px business objectives and initiatives.
  • Accountable for ensuring delivery of products and features across web and mobile applications against the product and engineering roadmaps and set timelines.
  • Have a comprehensive understanding of the existing architecture and major components of web/mobile applications.
  • Estimate project workloads under relatively vague business/product requirements, and identify potential project risks and offer solutions or walk-arounds.
  • Identify major technical debts that would hurt the product experience externally or reduce the team efficiency internally, and prioritize them according to their business impacts.
  • Collaborate with product owners and senior team members to plan the technical roadmap to fulfill product requirements and address outstanding tech debts, with the key objective of timely delivery of projects in a cross-team setup.
  • Collaborate with other functional team leads, including product, marketing, and design departments to drive consensus and ensure alignment on the product roadmap and delivery schedules. Address issues in the processes and workflows to increase team productivity.
  • Work closely with executive team to conduct budget planning, ROI analysis, and project prioritization.
  • Participate in executive meetings with senior leadership to share the progress of projects in the planned roadmap, raise concerns that could potentially affect business, and develop solutions.
  • Develop reports, dashboards and key tools for tracking KPIs and critical milestones; provide summary of status and tracking against established timelines and KPIs on a regular basis to key stakeholders across the business line and senior leadership team.
  • Communicate the main business goals and product vision to team members to ensure team alignment.
  • Build quality engineering teams through hiring, personal development events, and other activities.
  • Evaluate individual performance and conduct regular 1-1s to recognize each individual’s achievements, discuss areas of improvements, and help engineers develop career growth plans.

The ideal candidates will have:

  • Bachelor degree or above in Computer Engineering, Computer Science, or equivalent.
  • Demonstrated leadership experience
  • 5+ years combined experience on writing code
  • 3+ years management experience delivering highly successful and innovative technology products.
  • Complete comfort and familiarity with software development tools (git, Github, bug tracking), *nix environments;
  • Demonstrated experience in managing web and/or mobile product delivery from end-to-end, using lean startup and agile principles.
  • Good architectural sense and a passion for high quality code, honed by experience;
  • Grit: when the need arises you are willing to work through messy details with your team to achieve success; you persevere when others fall away.
  • A bias towards action: the ability to cut through any issue, identify the salient points, and rally a team around addressing them
  • High standards: for yourself, your work, and your colleagues
  • Exceptional communication skills
  • Track record of balancing making decisions using data vs creative vision

You'll get bonus points for:

  • Experience with Amazon Web Services, and even more bonus points for experience beyond just EC2 and S3.
  • Experience with React/Redux, Ruby on Rails, Python, or Go.
  • Experience with monitoring systems like Datadog, New Relic, or Fabric.
  • Experience with CDNs and network caching topologies.
  • A love of photography.

The Perks

  • Interesting technical challenges
  • Competitive salaries
  • Flexible hours
  • Catered lunches and snacks
  • Great health and dental benefits
  • Many professional development opportunities

Our interview process is straightforward:

  • We give you a call and chat about some basic stuff.
  • You drop by our office and show us stuff you worked on in the past and demonstrate your awesome skills.
  • You are presented with an offer.

Diversity

We believe diverse teams perform better, and we seek to increase our overall team diversity. We make active efforts to reduce the impact of unconscious bias in our hiring process.

Comments
Open jobs at 500px
Backend Developer
Toronto

Help us scale our backends!

500px is growing fast! We’re already handling hundreds of millions of requests each day, across a diverse set of backend services, datastores, and APIs, and we need your help to get to the next level.  

We are looking for smart, motivated, and passionate engineers to take charge of these backend systems and help us grow.  Our front-end teams work hard to build new and innovative products for our users, but they can’t succeed without support from the backend.

The Role

We are looking for a senior developer who can help us maintain, improve, and rebuild the 500px website. You'll have the opportunity to play a pivotal role in 500px's success as part of a small and passionate engineering team.

You will plan, implement, deploy, and maintain our backend systems, working closely with other backend developers to build out new services and improve existing ones.  You’ll play a pivotal role in designing and building the next generation of microservices for 500px.

On any given day, you might be

  • turning product plans into code, involving close collaboration with product managers, designers, and front-end web and mobile developers;
  • building highly scalable API services in Ruby and Go;
  • helping to design and maintain the 500px systems infrastructure;
  • involved in technical planning and architectural decision-making;
  • becoming intimately familiar with our large Ruby on Rails and Go applications, and forming an opinion about how to debug and improve them;
  • coordinating and collaborating with developers across the entire stack;

You’ll have ownership over your work, and you’ll support other developers in building new features and products. You will be an advocate for performance, stability, and scalability across all of 500px, and encourage others to be as well.  We are looking for a person who loves collaboration and is excited to take on new challenges.

The ideal candidate has

  • expert-level knowledge of Ruby on Rails or Go;
  • experience using JavaScript and Go, or the ability to learn them quickly based on past experience;
  • experience using, or familiarity with, MySQL, MongoDB, Redis, Memcached, and Elasticsearch;
  • solid fundamental understanding of web technologies including HTTP, RESTful services, and cloud platforms such as AWS;
  • complete comfort and familiarity with software development tools (git, Github, bug tracking), *nix environments;
  • significant experience applying those skills and knowledge to build and maintain high-traffic, consumer-focused web services;
  • good architectural sense and a passion for high quality code, honed by experience;
  • a love of photography.

This role would ideally suit someone with previous experience in a similar role. If you’re unsure, please apply – we’d love to hear from you.

Senior Backend Developer
Toronto

Help us scale our backends!

500px is growing fast! We’re already handling hundreds of millions of requests each day, across a diverse set of backend services, datastores, and APIs, and we need your help to get to the next level.  

We are looking for smart, motivated, and passionate engineers to take charge of these backend systems and help us grow.  Our front-end teams work hard to build new and innovative products for our users, but they can’t succeed without support from the backend.

The Role

We are looking for a senior developer who can help us maintain, improve, and rebuild the 500px website. You'll have the opportunity to play a pivotal role in 500px's success as part of a small and passionate engineering team.

You will plan, implement, deploy, and maintain our backend systems, working closely with other backend developers to build out new services and improve existing ones.  You’ll play a pivotal role in designing and building the next generation of microservices for 500px.

On any given day, you might be

  • turning product plans into code, involving close collaboration with product managers, designers, and front-end web and mobile developers;
  • building highly scalable API services in Ruby and Go;
  • helping to design and maintain the 500px systems infrastructure;
  • involved in technical planning and architectural decision-making;
  • becoming intimately familiar with our large Ruby on Rails and Go applications, and forming an opinion about how to debug and improve them;
  • coordinating and collaborating with developers across the entire stack;
  • providing leadership and mentorship to other team members.

You’ll have ownership over your work, and you’ll support other developers in building new features and products. You will be an advocate for performance, stability, and scalability across all of 500px, and encourage others to be as well.  We are looking for a person who loves collaboration and is excited to take on new challenges.

The ideal candidate has

  • expert-level knowledge of Ruby on Rails or Go;
  • experience using JavaScript and Go, or the ability to learn them quickly based on past experience;
  • experience using, or familiarity with, MySQL, MongoDB, Redis, Memcached, and Elasticsearch;
  • solid fundamental understanding of web technologies including HTTP, RESTful services, and cloud platforms such as AWS;
  • complete comfort and familiarity with software development tools (git, Github, bug tracking), *nix environments;
  • significant experience applying those skills and knowledge to build and maintain high-traffic, consumer-focused web services;
  • good architectural sense and a passion for high quality code, honed by experience;
  • a love of photography.

This role would ideally suit someone with previous experience in a similar role. If you’re unsure, please apply – we’d love to hear from you.

Growth Engineer
Toronto

The company

500px is an online community for sharing, discovery, buying, and selling of high-quality photography. Every month, millions of people use our website and mobile apps from around the world to find the world’s most inspiring photography.

We take pride in the products we ship and love what we do. We have an engineering-friendly culture where we value mentorship, ownership, and getting stuff done.

The job

Are you a versatile engineer? Interested in growing products through engineering all points of the funnel? Get in touch with us. You’ll have a unique opportunity to have a large impact on our roadmap. Your mission is to increase 500px’s growth rate by developing product and marketing features and implementing experiments.

You can expect to...

  • Help define the team’s goals and activities to maximize the opportunity for user and revenue growth
  • Work with a world class group of product managers, engineers, and marketers on a variety of projects to increase acquisitions, virality, referrals, retention, and revenue
  • Build, test, and learn, fast
  • Own tracking across our platform to enable insights on user behaviour and product usage
  • Develop in-house products for automating and scaling successful growth initiatives, and creating integrations with social and other platforms to drive traffic
  • Build new features and improve existing products that are used by millions of photographers
  • Be the internal subject matter expert on growth engineering, champion best practices and stay on top of industry trends
  • Drive a growth culture at 500px and work with marketing to publicize our thought leadership in the space

Requirements

  • 2+ years of Javascript, HTML, CSS
  • Experience with Ruby on Rails or a similar web framework
  • Experience with web analytics / tracking tools and methodologies
  • Knowledge of consumer marketing funnels and conversion optimization
  • Self-starter with ability to clear roadblocks and work with a small, agile team
  • Passion for learning, tackling hard problems, and standing out from the crowd
  • Great communication skills

Bonus points for…

  • A love of photography; having an active photography portfolio
  • Direct experience in growth marketing or growth hacking
  • Experience in metrics-guided product development (including A/B testing)
Mobile Developer
Toronto

Company

At its core, 500px is a community of passionate photographers. We build a platform to enable and reward visual creativity. Every month, millions of people from around the world use our website and mobile apps to find, share, and get rewarded for the world’s most inspiring photography.

We take pride in the products we ship and love what we do. Our engineering culture values mentorship, ownership, collaboration, and getting stuff done.

Our technology stack includes: Python, Ruby/Rails, Go, MongoDB, MySQL, Redis, and Elasticsearch. Check out our Engineering Blog for more.

Job

You’ll improve the mobile experience across all of 500px, without compromising on performance, usability or accessibility. You’ll take advantage of latest and greatest technologies available on iOS and Android platforms to help us build incredibly useful and innovative mobile apps that our users love. You’ll bring energy to the team, by being adaptable, productive, and by learning quickly.

What you’ll do:

  • Build and improve 500px’s mobile experiences
  • Work closely with product and design teams to ensure consistent UX and polish across our entire cross-platform experience
  • Be an advocate for mobile, and help guide our overall product roadmap

Ideally you'll have:

  • Experience publishing native apps to the major app stores
  • In-depth knowledge of either the Android or iOS platform and APIs
  • An eye for platform-specific user experience guidelines, i.e. familiarity with the iOS Human Interface Guidelines and/or the Android User Interface Guidelines
  • Flexibility, and a commitment to fix problems wherever they lie, regardless of tech stack

Perks

  • Interesting technical challenges.
  • Competitive salaries.
  • Flexible hours.
  • Your choice of computer.
  • Catered lunches and some of Toronto's best coffees and teas.
  • Great health and dental benefits.
  • Many professional development opportunities.

Interview process

  • Phone screen: <1hr conversation with a hiring manager/tech lead.
  • Coding challenge: We'll ask you to write some code, and then share it with us to review.
  • In-person interviews: 3-5 hours at our office where you'll meet multiple members of our team.

Diversity

We believe diverse teams perform better, and we seek to increase our overall team diversity. We make active efforts to reduce the impact of unconscious bias in our hiring process.

You may also like
E-Commerce at Scale: Inside Shopify's Tech Stack
How SendGrid Scaled to 40 Billion Emails Per Month
How Stream Built a Modern RSS Reader With JavaScript
How Heap Built an Analytics Platform that Auto-Tracks Every User Event