By late 2018, Pinterest was running one of the largest cloud Kafka deployments in the world, handling internal use cases like metrics, reporting, and monitoring, while also powering realtime features in the app, like content recommendations and spam detection.

To deliver that breath of coverage, Pinterest runs more than 2,000 brokers on AWS, “transporting more than 800 billion messages and more than 1.2 petabytes per day, and handling more than 15 million messages per second during the peak hours.”

They run Kafka in three AWS regions, with most of the workers in us-east-1, and with MirrorMaker transporting data among the three regions. Within each region, brokers are spread among multiple clusters for topic-level isolation, limited exposure across topics if one cluster fails.