Sentry

Sentry

817 Followers
Developers use Sentry to cut time to resolution for application issues from five hours to five minutes.
@getsentry
sentry.io
San Francisco, CA
Tools by Sentry

Decisions 4

James Cunningham

Operations Engineer at Sentry

DjangoDjangoCeleryCelery
#MessageQueue

Sentry started as (and remains) an open-source project, growing out of an error logging tool built in 2008. That original build nine years ago was Django and Celery (Python’s asynchronous task codebase), with PostgreSQL as the database and Redis as the power behind Celery.

We displayed a truly shrewd notion of branding even then, giving the project a catchy name that companies the world over remain jealous of to this day: django-db-log. For the longest time, Sentry’s subtitle on GitHub was “A simple Django app, built with love.” A slightly more accurate description probably would have included Starcraft and Soylent alongside love; regardless, this captured what Sentry was all about.

#MessageQueue #InMemoryDatabases

22 209.4K

James Cunningham

Operations Engineer at Sentry

CeleryCeleryRabbitMQRabbitMQ
#MessageQueue

As Sentry runs throughout the day, there are about 50 different offline tasks that we execute—anything from “process this event, pretty please” to “send all of these cool people some emails.” There are some that we execute once a day and some that execute thousands per second.

Managing this variety requires a reliably high-throughput message-passing technology. We use Celery's RabbitMQ implementation, and we stumbled upon a great feature called Federation that allows us to partition our task queue across any number of RabbitMQ servers and gives us the confidence that, if any single server gets backlogged, others will pitch in and distribute some of the backlogged tasks to their consumers.

#MessageQueue

18 314.5K

James Cunningham

Operations Engineer at Sentry

Sentry's event processing pipeline, which is responsible for handling all of the ingested event data that makes it through to our offline task processing, is written primarily in Python.

For particularly intense code paths, like our source map processing pipeline, we have begun re-writing those bits in Rust. Rust’s lack of garbage collection makes it a particularly convenient language for embedding in Python. It allows us to easily build a Python extension where all memory is managed from the Python side (if the Python wrapper gets collected by the Python GC we clean up the Rust object as well).

18 59.8K

James Cunningham

Operations Engineer at Sentry

HAProxyHAProxy
#WebServers

When accepting events, we would be crazy to just expose the Python web process to the public Internet and say, “Alright, give me all you got!” Instead, we use two different proxying services that sit in front of our web machines:

1) NGINX, our product-aware proxy, handles many of the upper bounds that we have deemed reasonable. It is responsible for a variety of bounds, but its most popular one is protecting Sentry from exceedingly large event volumes. Ever so often, a user will run into a problem where they’ve deployed their code out into the abyss, and their event volume clocks in at a few zeroes higher than what they signed up for.

2) In front of NGINX, we use another proxying service called HAProxy, which acts as a delta of connections without any of that product awareness logic and has a lot higher throughput. All it does is accept connections and send them off to different NGINX servers, allowing us to gracefully add or remove NGINX servers as we see fit.

#WebServers #LoadBalancerReverseProxy

17 19.4K

Followers 817

Mahavir Vataliya
Alan Force
Kostas Kns
matjah4108
xq5he
Kevin Boronka
Mikołaj Kosmal
Sushilkumar Shisode
Wilfred van der Deijl
佳栋 陈
Ariel Barros
RangerDigital
Felix-Steffen Michalski
Marek  Orolín
Wannes Vermeulen
Sascha Bartl
Малик Сулайманов
Dev Jang
Xinkai Li
Vinicius Tonon
Ic3 SanDy
Dirk Preising
Alex Gemberg
Timur Israpilov
Harry Solanki
Jari Kero
Triswansyah Yuliano