Sentry

Sentry

2087 Followers
Developers use Sentry to cut time to resolution for application issues from five hours to five minutes.
@getsentry
sentry.io
San Francisco, CA
Tools by Sentry

Decisions 4

James Cunningham

Operations Engineer at Sentry

DjangoDjangoCeleryCelery
#MessageQueue

Sentry started as (and remains) an open-source project, growing out of an error logging tool built in 2008. That original build nine years ago was Django and Celery (Python’s asynchronous task codebase), with PostgreSQL as the database and Redis as the power behind Celery.

We displayed a truly shrewd notion of branding even then, giving the project a catchy name that companies the world over remain jealous of to this day: django-db-log. For the longest time, Sentry’s subtitle on GitHub was “A simple Django app, built with love.” A slightly more accurate description probably would have included Starcraft and Soylent alongside love; regardless, this captured what Sentry was all about.

#MessageQueue #InMemoryDatabases

21 272.5K

James Cunningham

Operations Engineer at Sentry

CeleryCeleryRabbitMQRabbitMQ
#MessageQueue

As Sentry runs throughout the day, there are about 50 different offline tasks that we execute—anything from “process this event, pretty please” to “send all of these cool people some emails.” There are some that we execute once a day and some that execute thousands per second.

Managing this variety requires a reliably high-throughput message-passing technology. We use Celery's RabbitMQ implementation, and we stumbled upon a great feature called Federation that allows us to partition our task queue across any number of RabbitMQ servers and gives us the confidence that, if any single server gets backlogged, others will pitch in and distribute some of the backlogged tasks to their consumers.

#MessageQueue

18 667.1K

James Cunningham

Operations Engineer at Sentry

Sentry's event processing pipeline, which is responsible for handling all of the ingested event data that makes it through to our offline task processing, is written primarily in Python.

For particularly intense code paths, like our source map processing pipeline, we have begun re-writing those bits in Rust. Rust’s lack of garbage collection makes it a particularly convenient language for embedding in Python. It allows us to easily build a Python extension where all memory is managed from the Python side (if the Python wrapper gets collected by the Python GC we clean up the Rust object as well).

18 76.2K

James Cunningham

Operations Engineer at Sentry

HAProxyHAProxy
#WebServers

When accepting events, we would be crazy to just expose the Python web process to the public Internet and say, “Alright, give me all you got!” Instead, we use two different proxying services that sit in front of our web machines:

1) NGINX, our product-aware proxy, handles many of the upper bounds that we have deemed reasonable. It is responsible for a variety of bounds, but its most popular one is protecting Sentry from exceedingly large event volumes. Ever so often, a user will run into a problem where they’ve deployed their code out into the abyss, and their event volume clocks in at a few zeroes higher than what they signed up for.

2) In front of NGINX, we use another proxying service called HAProxy, which acts as a delta of connections without any of that product awareness logic and has a lot higher throughput. All it does is accept connections and send them off to different NGINX servers, allowing us to gracefully add or remove NGINX servers as we see fit.

#WebServers #LoadBalancerReverseProxy

17 24.6K

Followers 2087

Sree Jagaths
Víctor Vila
Volodymyr Kubiv
thebpc
José Herrera Avila
Mike Cooke
Noah Davis
Alexander Cato
Naomi Maekawa
ac r
David Cortés Toledano
amin khozaei
Luca Bellenghi
Goncalo Amaral
Maxim Bulyhin
Thông Nguyễn Văn
Ishita Batra
ko0st Michalski
Andrew halan
Alfredo Neto
Earl Grant Reyes
manishapme
Andhika Desta
arihantkumar
Hursac
Thiago Ferraz
akici