Soundcloud

Soundcloud

64 Followers
SoundCloud is the world’s leading social sound platform where anyone can create sounds and share them everywhere. Recording ... more
This is a community-generated profile. To edit this page claim this profile

Decisions 2

tim-thimmaiah

Initially, Stitch only supported real-time updates and addressed this problem with a MapReduce job named The Restorator that performed the following actions:

  • Calculated the expected totals
  • Queried Cassandra to get the values it had for each counter
  • Calculated the increments needed to apply to fix the counters
  • Applied the increments

Meanwhile, to stop the sand shifting under its feet, The Restorator needed to coordinate a locking system between itself and the real-time processors, so that the processors did not try to simultaneously apply increments to the same counter, resulting in a race-condition. It used ZooKeeper for this. Zookeeper

1 9.9K

tim-thimmaiah

Stitch is a wrapper around a Cassandra database. It has a web application that provides read-access to the counts through an HTTP API. The counts are written to Cassandra in two distinct ways, and it's possible to use either or both of them:

  • Real-time: For real-time updates, Stitch has a processor application that handles a stream of events coming from a broker and increments the appropriate counts in Cassandra.

  • Batch: The batch part is a MapReduce job running on Hadoop that reads event logs, calculates the overall totals, and bulk loads this into Cassandra. Cassandra

1 6.8K

Followers 64

Zezhe Huang
Lucky Ndovu
Shawn Rivera
Lalit Vatsal
Nicholas Olson
M Haidar Hanif
Nick Nish
Ming Wong
mrgnw
traximilian
Konstantin Mochalov
Andrew Volozhanin
Anton Cherepanov
Akiomi KAMAKURA
takahiro47
Eddie Kollar
Lucas Carvalho
disjukr
Matt Lds
Jesús Ruiz
Geoff Maddock
Shebella
tunnckoCore
elantrix
Andrey Krisanov
Kosate
rugved-inamdar