Soundcloud

Android Large Screen Optimization

Discover how SoundCloud transformed its Android app to deliver an immersive music experience on large screen devices, including tablets, foldables, and Chromebooks

An update on our API: Changes to access for inactive apps

We’re working on improving our API offering. In order to do that, we need to revoke access for some developers with **inactive apps**. We’ve reached out directly to those who will be affected via email and we’re working to mitigate impact where we can.

What I Learned in One Year as an SRE Trainee

In this post, I'll share my experiences as a new SRE, reflecting on the lessons I’ve learned and challenges I’ve faced in my first year. I hope to give you an insight into the role of an SRE, and perhaps even inspire others to consider a career in this field.

What's New with SoundCloud, November 2022

We recently made updates to our SoundCloud iOS and Android apps to address key user experience (UX) fixes according to user feedback. Read on to find out more.

Bar Raiser Interview

The Bar Raiser interview is a critical part of the final round of engineering interviews at SoundCloud, and it’s focused on increasing the quality of hiring, which ultimately has an impact on the products we develop for our users. In this post, we’ll talk about the training process we defined for the Bar Raiser interview.

What's New with SoundCloud, October 2022

We recently made updates to our SoundCloud iOS and Android apps to address key user experience (UX) fixes according to user feedback. Read on to find out more.

Android Image Loading at SoundCloud

The SoundCloud Android app recently got a significant makeover: a “new year, new me” type of thing. Our remarkable design team and outstanding engineers worked together to revamp the way we display all kinds of images in the app, and I’m about to take you behind the scenes on this “glow-up” journey.

SoundCloud Echo: Next-Level Humane Registry with Backstage

In this post, we'll describe how we took our service catalog to the next level, how we migrated our internal system to the open source Backstage developer portal technology, and how we're now providing more capabilities and empowering our engineering team, in turn speeding up software delivery and engineering effectiveness.

WINE Workshop

SoundCloud has a variety of resource groups, and one such group is called Women in Engineering, or WINE. The mission statement of the group — which is open to people of all genders — is to work “on initiatives to attract and ensure an environment for women to thrive within Engineering at SoundCloud and in the industry.”

PR Templates for Effective Pull Requests

Empty pull request descriptions slow down teams, but PR templates are a quick and easy remedy. By streamlining the PR writing process into a series of fill-in-the-blank steps, a PR becomes a valuable asset for knowledge sharing across teams that will outlive its authors.

Learning Scala at SoundCloud

This is the story of how I joined SoundCloud and learned Scala as a Golang developer with no prior knowledge of Scala.

The End of the Public API Strangler

This is the story of how we used the [Strangler pattern](https://martinfowler.com/bliki/StranglerFigApplication.html) to migrate our [public API](https://developers.soundcloud.com/docs/api/explorer/open-api) from a monolithic codebase to a fully fledged [BFF](https://developers.soundcloud.com/blog/service-architecture-1) over the course of eight years. It also discusses some of the trials and tribulations we encountered along the way.

What Is New with Periskop in 2022

We made some updates to Periskop, our internal pull-based exception monitoring service. To learn about what is new in 2022, read on.

Obvious Ownership: A Sensible Humane Registry

Imagine yourself as an engineer who just joined SoundCloud. Besides meeting your colleagues and getting your new laptop, badge, and that cool branded hoodie, the first weeks at work will be about exploration. Onboarding involves a lot of “looking around”: finding mentorship, getting to know the company’s culture, and exploring codebases — and most likely, you’ll want to contribute code too! The first question you might ask is “What do I even work on?”. With time, you’ll identify a few sibling…

Tiny Letter from Kafka

Discover the powerful design of Apache Kafka and how SoundCloud consumes events from Kafka.

Service Architecture at SoundCloud — Part 3: Domain Gateways

Discover how SoundCloud extended the concept of Value-Added Services to Domain Gateways.

Service Architecture at SoundCloud — Part 2: Value-Added Services

Learn about the evolution of architectural patterns at SoundCloud — from BFFs to Value-Added Services.

Service Architecture at SoundCloud — Part 1: Backends for Frontends

Discover how service architecture has evolved at Soundcloud over the past few years...

How We Share Knowledge as a Web Collective

There’s no single platform team that consists of only web engineers at SoundCloud, even though we consider ourselves to be part of the “Web Collective.” Instead, all web engineers are split into smaller, cross-functional teams across the organization. This structure definitely has its advantages, but it’s also left engineers in a situation where platform-specific experience and knowledge about platform-specific best practices are more difficult to access than they would be if we were all on the…

Upcoming API Security Updates — Action Required

As part of our continuous effort toward making improvements to our API with the hope that we can relaunch API access to all developers, we’re making some critical security improvements. Please note that these are important and time sensitive and may cause disruptions in your app if you don’t make the necessary security upgrades.

Did I Break You? Reverse Dependency Verification

Let's take a look at techniques we can use to confidently release updates to an internal library that's integrated into hundreds of projects across the company — even if they're API-breaking changes.

High-Tier Content in the SoundCloud API

You can now retrieve metadata for our full SoundCloud catalog through our API by providing the new `access` filter in your request.

The Journey of Corpus

How we established a team with a clear focus on data governance (and exactly what that means), allowing us to leverage the vast data available to SoundCloud — as the world’s largest open audio platform — to become a more data-driven company.

How to Successfully Hand Over Systems

In a product company, changes are inevitable so as to best support the strategy and the vision. Often during such a change, new teams are formed and other ones are restructured. While there are many challenges to be solved during a big change, there’s one in particular that’s often overlooked: system ownership. Having experienced not-so-successful handovers, I was inspired to create a guideline that will help other teams do handovers differently.

SoundCloud’s New API Track Object

As a part of our efforts to improve API use, we’re introducing a new `Track` object that’s more up to date with our current data model. The following change is intended to standardize and simplify the response data. A new object will be used for all endpoints returning track-related information.

Building a Healthy On-Call Culture

In this post, we share some of the best practices we apply when SoundCloud engineers take on out-of-hours support duties.

Dependency Inversion as a Driver to Scale Mobile Development

As our codebase scales, the need for cleaner architecture becomes more apparent. The dependency inversion principle (DIP) is one tool that helps us build cleaner relationships among our software modules. In this post, DIP is presented using atomic modules and is later applied to frameworks.

Tests Under the Magnifying Lens

We take software testing very seriously at SoundCloud. As we scale our codebases and vary the testing harness, we've also faced some challenges with maintainability. Take a look at this post to learn about some of the tooling we've built to help us manage the execution of and derive insights from the dozens of thousands of tests every day.

Leveraging a Manager Weekly Newsletter for Team Communication

Sometimes keeping your team up to date both weekly and candidly, without adding extra meetings and friction to your workday, is difficult. If only there were a newsletter they could subscribe to to keep updated…

Testing SQL for BigQuery

“To me, legacy code is simply code without tests.” — Michael Feathers If untested code is legacy code, why aren’t we testing data pipelines or ETLs (extract, transform, load)? In particular, data pipelines built in SQL are rarely tested. However, as software engineers, we know all our code should be tested. So in this post, I’ll describe how we started testing SQL data pipelines at SoundCloud.

Pagination Updates on Our API

As part of our efforts to improve our APIs, we’re introducing updates on how we paginate over tracks. This only affects developers and apps that aren’t strictly relying on the `next_href` field, but rather custom manipulating the `offset` to request a page.

DeveloperBridge: SoundCloud’s Program for Training People from Diverse Backgrounds to Become Engineers

DeveloperBridge is a year-long, full-time, paid traineeship program where participants learn from and work with engineering teams at SoundCloud. The DeveloperBridge program was launched in 2017 with the goal of further supporting diversity and inclusion at SoundCloud and giving back to the tech community.

Breaking Loose from Third-Party Lock-In with Custom Refactoring Tools

Code refactoring is an essential part of the job of software developers. As time goes on, technology evolves, product requirements change, and new features are built into a codebase. Find out how the iOS Collective at SoundCloud created a custom refactoring tool to transform more than 900 source files.

Changing the Interview Process during Remote Working

At SoundCloud, we changed our hiring strategies to adapt to hiring remotely during COVID-19.

Technical Interview Reform, Part 2: The Recruiting Perspective and Results

SoundCloud engineers and recruiters recently comprehensively rethought the backend engineering take-home challenge. This is the second in a small series of posts, and it contains the recruiting perspective and the results of the changes.

Technical Interview Reform, Part 1: Rethinking the Backend Engineering Interview Take-Home Challenge

SoundCloud engineers and recruiters recently comprehensively rethought the backend engineering take-home challenge. This is the first in a small series of posts about what we did and why.

Announcing Twinagle: Twirp and Protobuf for Finagle

This article announces the release of Twinagle, an open source implementation of the Twirp protocol for Scala/Finagle.

Stretch Opportunities for Engineers

This blog post explains how the Content Team at SoundCloud introduced a process for creating stretch opportunities for its engineers.

Open Sourcing Intervene

Open sourcing `intervene`, a tool to selectively mock API responses and proxy everything else to the real API server.

Periskop: Exception Monitoring Service

A new approach to exception monitoring, designed for high scalability...

How (Not) to Build Datasets and Consume Data at Your Company

Two years ago, if an engineering team at SoundCloud needed data produced by another team, in many cases, that team would have had to start by building a dataset from the raw data. That meant there was no standard way to structure data into a dataset, there were errors in understanding and interpreting raw data, and there was no reuse of datasets across teams. Ownership of datasets was with the teams consuming data, even when the data itself was owned and best understood by someone else. <br><br> By agreeing on an ownership strategy, a common shape for datasets, and standard tooling, SoundCloud was able to streamline these problems away. Now the data and datasets are provided by those who know best about their data, along with the help of tooling built by those who know best about building tools. Read on to learn what datasets look like at SoundCloud and how they are built.

The Magic of Generating an Xcode Project

Learn how SoundCloud improved the productivity of its iOS engineers by using Tuist to generate Xcode projects.

A Happy New Employee

For a successful onboarding, you may rely on things like finding a good mentor or writing up a detailed checklist, but there’s more you can do. In this blog post, a new employee at SoundCloud illustrates a collection of engineering practices here that allowed him to get up to speed efficiently and which made the onboarding experience a positive one.

Speeding Up Builds with Dagger Reflect

A large portion of an Android app’s build time can consist of Dagger annotation processing, and most developers agree that productivity is important, so we decided to experiment to see if we could save time when compiling the SoundCloud Android app. This blog post covers how we used Dagger Reflect to save developer time with minimal changes to our codebase.

The Power of Types for Errors

Limiting the number of primitives we use in our code in favor of using custom types instead is an easy win. In this way, we as developers will not only reduce the possibility of bugs by using the compiler, but we will also achieve more readable and self-documented types.

Implementing Dark Mode Using the Observer Pattern

Last week’s update to the SoundCloud iOS app includes support for Dark Mode. This took several months of work and collaboration between design and engineering teams across the company, so we wanted to share our approach to implementing Dark Mode and some of the obstacles we encountered along the way.

Solving Remote Build Cache Misses by Annoying Your Colleagues

This is part two in a series about solving Gradle remote build cache misses. Solving build cache misses is important to both avoid work that has already been compiled and improve build speeds. For more information, check out part one here.

Gradle Remote Build Cache Misses

Until recently, one of the top technical risks facing SoundCloud’s Android team was increasing build times. Our engineering leadership was well aware of the problem, and it was highlighted in our company’s quarterly goals and objectives as modularization. Faster build times means more productive developers. More productive developers are happier and can iterate on products more quickly.

SoundCloud Is Playing the Oboe

Media and playback are at the core of SoundCloud’s experience. For that reason, we have established and grown an engineering team that is specialized in providing the best possible streaming experience to our users across multiple platforms.

Alerting on SLOs like Pros

If there is anything like a silver bullet for creating meaningful and actionable alerts with a high signal-to-noise ratio, it is alerting based on service-level objectives (SLOs). Fulfilling a well-defined SLO is the very definition of meeting your users’ expectations. Conversely, a certain level of service errors is OK as long as you stay within the SLO — in other words, if the SLO grants you an error budget. Burning through this error budget too quickly is the ultimate signal that some rectifying action is needed. The faster the budget is burned, the more urgent it is that engineers get involved.

Playback on Web at SoundCloud

Maestro is a library we have developed to handle all playback across SoundCloud web applications. It successfully handles tens of millions of plays per day across soundcloud.com, our mobile site, our widget, Chromecast, and our Xbox application. We are considering open sourcing it, and this blog post is a technical overview of what we’ve achieved thus far with Maestro.

Taking Our Bug Bounty Program Public

We’re excited to announce the launch of our public bug bounty program with Bugcrowd — the #1 crowdsourced security platform. This public program is open to Bugcrowd’s full crowd of top, trusted whitehat hackers, and we will award up to $1,500 per vulnerability identified on our website, API, and mobile apps.

Release Quality and Mobile Trains

Once every two weeks, we prepare new versions of our mobile apps to be published to the app stores. Being confident about releasing software at that scale — with as many features and code contributions as we have and while targeting a wide range of devices like we do at SoundCloud — is no easy task. So, over the last few years, we have introduced many tools and practices in our release process to aid us.

How to Reindex One Billion Documents in One Hour at SoundCloud

In the past, the Search Team at SoundCloud had high lead times for making updates to Elasticsearch clusters, either during the implementation of a new feature or simply while fixing a bug. This was because both tasks require us to reindex our catalog from scratch, which means reindexing more than 720 million users, tracks, playlists, and albums. Altogether, this process took up to one week, though there was even one scenario where it almost took one month to roll out a bug fix.

Off-Platform Validation

Last month we launched SoundCloud Premier Distribution, which allows creators to distribute their music from SoundCloud to many other streaming platforms and stores. The result? Artists can now get paid for plays on other platforms just like they can on SoundCloud.

Radical Candor: An Experience Report

Although it can be easy to know if you’ve messed up badly as a manager, it’s not always as easy to know if you’re doing a good job. In particular, the power dynamics at play can make it hard for people on your team to feel confident letting you know what’s working well and what’s working not so well.

Garbage Collection in Redux Applications

The SoundCloud Xbox application is a regular React Redux application that runs in a native web view wrapper on Microsoft’s UWP framework. In order to allow users to play music in the background, we also ship a native C++ playback layer with the app. A thin API wrapper allows for communication between the web layer and the native layer.

Integration Testing for Memory Leaks

We write a lot of unit tests while working on the SoundCloud iOS application. Unit tests are obviously great. They’re short, they’re (hopefully) easy to read, and they give us confidence that the code we ship works as expected. But unit tests — as their name suggests — only tend to cover a single unit of code, most often a function or class. So how do we catch the bugs that exist in the interactions between classes — bugs like memory leaks?

A Pragmatic Approach to Tech Debt Reduction

Almost every company accumulates tech debt as time goes on. Tight deadlines, changing requirements, scaling issues, poor or short-sighted system designs, knowledge silos, inconsistent coding practices, turnover of key staff — these things all happen and can contribute to tech debt. So what can be done about it once it’s there?

How We Develop New Features Using Offsites and Clean Architecture

The current SoundCloud iOS App was built back in 2014, resulting in a huge codebase consisting of both Objective-C and Swift, with multiple architectural patterns and lots of legacy code. With time, the team changed and the platform evolved, so for the current iOS Team, maintenance, bug fixing, and implementing new features are all challenges.

Using Kubernetes Pod Metadata to Improve Zipkin Traces

SoundCloud is built on hundreds of microservices. This creates many challenges, among them debugging latency issues across the services engaged in the completion of a single request. As such, it’s natural that we were early adopters of Zipkin, a distributed tracing tool that emerged from Twitter, which was built in part to help engineers tackle these sorts of problems.

Hands-Off Deployment with Canary

At SoundCloud, we follow best practices around continuous delivery, i.e. deploying small incremental changes often (many times a day). In order to do this confidently and reliably, we verify these changes before releasing, using different measures including, among others, unit, integration, and contract tests.

Creating Readable Spark Jobs

Nowadays, it’s rather common to encounter Apache Spark being utilized in a lot of companies that need to process huge amounts of data, and things aren’t any different here at SoundCloud. As one can imagine, we have lots of data to process all the time. While Spark does offer a streaming API, along with other tools like Kafka Streams and Apache Flink, we mainly (but not exclusively) use batch processing with Spark to solve a variety of problems, such as:

Getting a Team Back on Track

Sometimes, an important team that’s part of an otherwise healthy company culture starts tanking and the people on the team get frustrated and even quit.

Keeping Counts In Sync

Track play counts are essential for providing a good creator experience on the SoundCloud platform. They not only help creators keep track of their most popular songs, but they also give creators a better understanding of their fanbase and global impact. This post is a continuation of an earlier post that discussed what we do at SoundCloud to ensure creators get their play stats (along with their other stats), both reliably and in real time.

Defining a Role with a Retrospective Activity

Agile retrospectives are a widely used practice within engineering teams. They provide teams with a way to reflect on how they work and become better at what they do. One of the main benefits of retrospectives is that they empower teams to define and make changes by analyzing what happened in an iteration and by determining what can be improved moving forward. Here at SoundCloud, we hold retrospectives at the end of every iteration (every two weeks), and we often do them at the end of projects as well.

Project Necromancy

In a perfect world, every project would flow smoothly — from requirement gathering, to implementation, to acceptance by stakeholders, to launch. Sadly, we don’t live in a perfect world. Sometimes projects hit bumps in the road due to changing business requirements, market conditions, or a host of other factors. This post describes our experiences managing a project that was deprioritized during implementation and lay dormant for an extended period of time before resuming in a different form.

Engineering Levels at SoundCloud

An absolutely crucial part of the experience of being an engineer at SoundCloud is learning and growing as a person. Pretty much everyone we hire mentions this aspect as one of their main motivations for joining the company. And while retaining highly talented and motivated people and helping them develop is naturally valuable for SoundCloud as a company, it's also profoundly beneficial for the employees themselves.

Managing Unplanned and Support Tasks

One challenge engineering teams often face is dealing with work that doesn’t revolve around developing new features but that still requires the team’s attention and time. The Content Engineering Team here at SoundCloud is no exception, so we iterated on a process to deal with unplanned and support tasks to end up with fewer interruptions and more time to spend on implementing planned features.

Automatic Stubbing of Network Requests to Deflakify Automation Testing

Apple introduced automated UI testing in Xcode 7. This was a great addition for developers because this native support promised, among other things, an improvement in the flakiness notoriously associated with automation tests. As many of us developers have experienced, tests can sometimes fail even when there has been no modification to the test or underlying feature code.

PageRank in Spark

SoundCloud consists of hundreds of millions of tracks, people, albums, and playlists, and navigating this vast collection of music and personalities poses a large challenge, particularly with so many covers, remixes, and original works all in one place.

Insights from SoundCloud’s DeveloperBridge Trainee Program

Back in 2016, the SoundCloud People Team collaborated with engineering management in an effort to bring more junior engineers into the company. The result was DeveloperBridge, a paid junior engineering program that ran for 12 months and was based out of our Berlin headquarters.

SoundCloud's Data Science Process

Here at SoundCloud, we’ve been working on helping our Data Scientists be more effective, happy, and productive. We revamped our organizational structure, clearly defined the role of a Data Scientist and a Data Engineer, introduced working groups to solve common problems (like this), and positioned ourselves to do incredible work! Most recently, we started thinking about the work that a Data Scientist does, and how best to describe and share the process that we use to work on a business problem. Based on the experiences of our Data Scientists, we distilled a set of steps, tips and general guidance representing the best practices that we collectively know of and agree to as a community of practitioners.

Deliver software faster by managing work in progress, not by adding overtime

Product development flow (flow) is the rate at which our products are developed, from idea to deployment. Good flow means that products should pass through the development cycle quickly and continuously.

Inside a SoundCloud Microservice

If you’re a regular visitor to this blog, you might be aware that we have been transitioning to a microservices based architecture over the past four to five years, as we have shared insights into the process and the related challenges on multiple occasions. To recap, adopting a microservices architecture has allowed us to regain team autonomy by breaking up our monolithic backend into dozens of decoupled services, each encapsulating a well defined portion of our product domain. Every service is built and deployed individually, communicating with other services over the network via light-weight data interchange formats such as JSON or Thrift. What we haven’t touched on so far is how a microservice at SoundCloud looks backstage.

Remote device sign-in

A method for signing in to a device that doesn’t have a keyboard When we were developing our SoundCloud app for Xbox One, something became very obvious during usability testing: signing in with a game controller really sucks. Entering text requires navigating a virtual keyboard to individual letters, numbers, and characters one at a time – such a nightmare! Plus, letters, numbers, and special characters are spread across three screens. The more secure your password is, the worse the experience is. Entering a password with a game controller using an onscreen keyboard

A Better Model of Data Ownership

Once upon a time, we had a single monolith of software, one mothership running everything. At SoundCloud, the proliferation of microservices came from moving functionality out of the mothership. There are plenty of benefits to splitting up features in this way. We want the same benefits for our data as well, by defining ownership of datasets and ensuring that the right teams own the right datasets.

Lessons in resilience at SoundCloud

Building and operating services distributed across a network is hard. Failures are inevitable. The way forward is having resiliency as a key part of design decisions. This post talks about two key aspects of resiliency when doing RPC at scale - the circuit breaker pattern, and its power combined with client-side load balancing.

API Sign-up Changes

Today we are launching a new API application process. Starting today, developers will need to fill out an application form to request access to our API. Here is why. As SoundCloud has grown, we’ve seen our API used in countless innovative ways that support and enhance the SoundCloud experience for our creators and listeners around the globe. At the same time, we’ve also seen a number of apps or services that act on behalf of our users without their explicit permission or attempt to use creator…

Microservices and the monolith

In a previous series of blog posts, we covered our decision to move away from a monolithic architecture, replacing it with microservices, interacting synchronously with each other over HTTP, and asynchronously using events. In this post, we review our progress toward this goal, and talk about the conditions and strategy required to decommission our monolith.

Removal of Groups endpoints

As well as adding new features and updates, we review existing features to see if they’re still beneficial to the community. As we dug into the best ways for curators to connect with artists and fans, we found that Groups aren’t working as well as reposts and curated playlists. With that in mind, we’ve decided to phase out Groups on Monday, August 22nd to make room for future updates. From this date, all Groups-related GET requests will no longer return results, and PUT requests will not persist…

Synchronous communication for microservices: current status and learnings

Since we started breaking our monolith and introduced a microservices architecture we rely a lot on synchronous request-response style communication. In this blog post we’ll go over our current status and some of the lessons we learned.

Prometheus has come of age – a reflection on the development of an open-source project

On Monday this week, the Prometheus authors have released version 1.0.0 of the central component of the Prometheus monitoring and alerting system, the Prometheus server. (Other components will follow suit over the next months.) This is a major milestone for the project. Read more about it on the Prometheus blog, and check out the announcement of the CNCF, which has recently accepted Prometheus as a hosted project.

Building radio stations at SoundCloud

Over the last 100 years we have dialed into radio stations at home, on the road, or in the office to access a curated mix of top hits delivered to us by our favorite DJ. With more and more of our daily activities taking place online, we find our source of music now comes from a mix of our mobile phones, our desktop computers and the radio - and is available to us whenever we need it. The amount of music available today is endless. The music scene has never been so vibrant. A proliferation of production tools at low cost and the ease at which an artist can share their sounds, gives everyone the opportunity to share their work at any time, with only a few clicks. Keeping this in mind, we set out to determine: How can we bring a radio like experience to SoundCloud that would enable you to tune-in and lean back?

Can a machine surprise you? We believe so.

With more than 125 million tracks from over 12 million creators heard each month on our platform, SoundCloud is uniquely positioned to offer listeners a full spectrum of music discovery. Classic hits, the latest releases, gems from underground talent and the best of what’s up-and-coming – all in one place. How can you make great content discoverable and available at ease? How can you create a unique experience for every single user?

Data pipelines with Apache Crunch and Java 8

With Java 8 now in the mainstream, Scala and Clojure are no longer the only choices to develop readable, functional code for big data technology on the JVM. In this post we see how SoundCloud is leveraging Apache Crunch and the new Crunch Lambda module to do the high-volume data processing tasks which are essential at early stages in our batch data pipeline efficiently, robustly and simply in Java 8.

Playlist search returns only compact representations as of July 15, 2016

We previously introduced new playlist representations. The compact representation returns only the playlist itself without any of the tracks. For most users, this is preferable – the compact representation significantly reduces the size of the JSON payload for large playlists and provides faster API response times. Beginning July 15, 2016, all queries using the parameter will only return this compact representation for playlists. Other playlist representations will no longer be available when…

Open-sourcing LightCycle for Android

Last week, we open-sourced LightCycle, an Android library that helps break logic out of and classes into small, self-contained components called LightCycles. Components that typically need to be aware of and lifecycle events include presenters, UI tracking code, input processors and more. We’ve been using LightCycle extensively in the SoundCloud Music & Audio and SoundCloud Pulse apps over the last year.

Congratulations, you have a lot of code!"Congratulations, you have a lot of code!" Remedying Android’s method limit - Part 2

In part one we described how running into Android’s method limit may leave you unable to build, and offered strategies you can employ to make your app fit into a single DEX file. In this part we share an alternative option: using multiple DEX files.

Introducing SoundCloud JavaScript SDK 3.0.0

We are happy to announce version 3.0.0 of our SoundCloud JavaScript SDK. The new SDK improves stream security and content uploading functionality, and modernizes the technology stack. Version 3 of the SoundCloud JavaScript SDK is a major update and is not backwards compatible. That said, the changes that you need to make your web app work with the new SDK are easy to implement. Please refer to Migrating to JavaScript SDK 3.0.0 to upgrade. ECMAScript 2015 and CommonJS The original version of the…

Congratulations, you have a lot of code!"Congratulations, you have a lot of code!" Remedying Android’s method limit - Part 1

At SoundCloud we have been building for the Android platform since 2010. Much has changed since then: the team has grown, the list of features has grown, and our audience has grown. Today, eight engineers are working full time on the official SoundCloud app, across various areas, with contributions pouring in from other parts of the organization. Due to the growing complexity and number of contributions, the app’s size has grown substantially. Currently the app consists of approximately 1200 Java source files, not counting tests, containing approximately 86000 lines of code. This doesn’t include native code, such as our playback or recording stacks. We’re not the first to run into Android’s limits in terms of build tools. An internal limitation of Dalvik’s byte code format (DEX), which I will explain in more detail, can leave you unable to build after your codebase reaches a certain size. If you fail to anticipate this, it might happen during the most inconvenient time, such as when you are preparing for a release. Part of our job in Core Engineering at SoundCloud is to make sure our developers are happy and productive; not being able to build our app anymore makes for neither happy nor productive developers. While there are a number of posts on this topic, I would like to describe in more detail what we have done to combat Android’s method limit, what things worked well and what didn’t work so well, what it actually means to use the tool’s switch and what you can do to improve application health with regards to size.

Introducing Rate Limits

At SoundCloud, we’re building an ecosystem where creativity thrives. Developers are an important part of that ecosystem. We’re continually inspired by how you use the SoundCloud API to support creators and listeners in innovative ways. But as the ecosystem has grown, we’re dealing with an increasing number of applications that abuse creator content by violating our developer Terms of Use. To help control this type of behaviour, we’re introducing a daily rate limit on API play requests. Beginning…

New playlist representations

Requests for playlists have always included the full track objects contained within. This representation may be convenient for playlists with ten or twenty tracks but can cause problems for playlists that contain hundreds or thousands of tracks. Requesting such large playlists could result in requests that take a long time to respond and that eventually timeout. Today, we introduce two new representations for the resource: and . If you add to a playlist request, the request will return only…

Apple's June 1 64-bit deadline

In October 2014, Apple announced that all submissions to the App Store must include 64-bit support by June 1, 2015. The SoundCloud API for Cocoa contains 32-bit dependencies and will not be updated, because it has been discontinued. Anyone using the SoundCloud API for Cocoa will need to will need to migrate away from it if they wish to update their app after June 1. To ease this transition we have built a sample app that demonstrates how to authorize a user via OAuth using only built-in…

Linked partitioning to replace offset-based pagination

The SoundCloud API will be dropping support for offset-based pagination on March 2, 2015, in favor of linked partitioning. To page through a JSON response, pass the linked_partitioning=1 parameter along with your request and it will return a collection, along with a next_href property if there are additional results. To fetch the next page of results, simply follow that URI. If the response does not contain a next_href property, you have reached the end of the results. You can read more about…

Prometheus: Monitoring at SoundCloud

In previous blog posts, we discussed how SoundCloud has been moving towards a microservice architecture. Soon we had hundreds of services, with many thousand instances running and changing at the same time. With our existing monitoring set-up, mostly based on StatsD and Graphite, we ran into a number of serious limitations. What we really needed was a system with the following features: A multi-dimensional data model, so that data can be sliced and diced at will, along dimensions like instance, service, endpoint, and method. Operational simplicity, so that you can spin up a monitoring server where and when you want, even on your local workstation, without setting up a distributed storage backend or reconfiguring the world. Scalable data collection and decentralized architecture, so that you can reliably monitor the many instances of your services, and independent teams can set up independent monitoring servers. Finally, a powerful query language that leverages the data model for meaningful alerting (including easy silencing) and graphing (for dashboards and for ad-hoc exploration). All of these features existed in various systems. However, we could not identify a system that combined them all until a colleague started an ambitious pet project in 2012 that aimed to do so. Shortly thereafter, we decided to develop it into SoundCloud’s monitoring system: Prometheus was born.

SoundCloud in Scalding case study by Concurrent Inc.

Recently we teamed up with Concurrent Inc., the backers of the data-processing framework Cascading, to do a case study of how we use Scalding for some of our data-driven products such as Search. Scalding enables us to iterate quickly, test easily, and it allows for loose coupling of some of our data-processing pipelines. Check back for future posts about our use of other data-processing tools, and frameworks such as Spark.

XML responses deprecated

The SoundCloud API will be dropping support for Extensible Markup Language (XML) responses. XML will be phased out on the following schedule: XML is currently the default response format for requests without an explicit format specified in the path (e.g. ) or header. This default will be changed to JSON on December 1, 2014. Explicit requests for XML — specified either in the path (e.g. ) or an header — will continue to be supported until December 15, 2014. After that point, only JSON responses…

Building the new SoundCloud iOS application — Part II: Waveform rendering

When we rebuilt our iOS app, the player was the core focus. The interactive waveform was at the center of the design. It was important both that the player be fast and look good. Initial implementation We iterated on the waveform view until it was as responsive as possible. The initial implementation focused on replicating the design, which heavily used CoreGraphics. A single custom view calculates the current bar offset based on its time property. It then draws each of the waveform samples that…

Building the new SoundCloud iOS application — Part I: The reactive paradigm

Recently, SoundCloud launched the new iOS application which was a complete rewrite of the existing iOS application. The Mobile engineering team saw this as an opportunity to build a solid foundation for the future of SoundCloud on iOS and to experiment with new technologies and processes at the same time. In the world of mobile, you deal with data, errors, threads and concurrency a lot. The common scenario starts with a user tapping on the screen. The application jumps off of the main UI thread…

Real-Time Counts with Stitch

Here at SoundCloud, in order to provide counts and a time series of counts in real time, we created something called Stitch. Stitch was initially developed to provide timelines and counts for our stats pages, which are where users can see which of their tracks are played and when. Stitch is a wrapper around a Cassandra database. It has a web application that provides read access to the counts through an HTTP API. The counts are written to Cassandra in two distinct ways, and it’s possible to use…

Building Products at SoundCloud—Part III: Microservices in Scala and Finagle

In the first two parts of this series, we talked about how SoundCloud started breaking away from a monolithic Ruby on Rails application into a microservices architecture. In this part we will talk a bit more about the platforms and languages in which we tend to write these microservices. At the same time that we started the process of building systems outside the Mothership (our Rails monolith) we started breaking our large team of engineers into smaller teams that focused on one specific area…

Building Products at SoundCloud—Part II: Breaking the Monolith

In the previous post, we talked about how we enabled our teams to build microservices in Scala, Clojure, and JRuby without coupling them with our legacy monolithic Rails system. After the architecture changes were made, our teams were free to build their new features and enhancements in a much more flexible environment. An important question remained, though: how do we extract the features from the monolithic Rails application called Mothership? Splitting a legacy application is never easy, but…

Building Products at SoundCloud —Part I: Dealing with the Monolith

Most of SoundCloud’s products are written in Scala, Clojure, or JRuby. This wasn’t always the case. Like other start-ups, SoundCloud was created as a single, monolithic Ruby on Rails application running on the MRI, Ruby’s official interpreter, and backed by memcached and MySQL. We affectionately call this system Mothership. Its architecture was a good solution for a new product used by several hundreds of thousands of artists to share their work, collaborate on tracks, and be discovered by the…

Roshi: a CRDT system for timestamped events

Let’s talk about the stream. The SoundCloud stream represents stuff that’s relevant to you primarily via your social graph, arranged in time order, newest-first. The atom of that data model, an event, is a simple enough thing. Timestamp User who did the thing Identifier of the thing that was done For example, At 2014-04-01T13:14:15.034Z, a-trak Reposted track skrillex/duck-sauce-nrg-skrillex-kill If you followed A-Trak, you’d want to see that repost event in your stream. Easy. The difficult…

Introducing JavaScript SDK version 2

SoundCloud is pleased to introduce a new major version of the SoundCloud JavaScript SDK. In version 2, we’ve rewritten much of the internal code, resulting in better performance for your JavaScript applications and support for more streaming standards, such as HTTP Live Streaming. You can test the new version by pointing your JavaScript applications to https://connect.soundcloud.com/sdk-2.0.0.js. We’ve also created a guide to help you upgrade from version 1 to version 2. JavaScript SDK version…

Irrational Fun: Find Yourself at Berlin Buzzwords

We were counting down the days until Berlin Buzzwords on May 25, when we realised that it would be great if you came too! With that in mind, we’ve created a contest. One lucky winner will receive a free ticket to Berlin Buzzwords, including travel expenses and accommodation. Here are the details about how to apply. The ratio of a circle’s circumference to its diameter, represented by the Greek letter π, is an irrational number—it never terminates or repeats. Your goal is to find the SoundCloud…

Welcome to SoundCloud's redesigned developer site

We’ve taken some time to bring all our developer resources together into a single site. In doing so, we’ve reorganized the layout to make things easier to find and also given the site a fresh new look. We hope you like it! If you have any feedback about the new design, follow @SoundCloudDev on Twitter and let us know.

Security update: Heartbleed vulnerability in OpenSSL

On Monday, April 7th, 2014, a major security vulnerability in OpenSSL was made public. The vulnerability was filed as CVE-2014-0160 and later dubbed “Heartbleed”, because the bug lies within OpenSSL’s heartbeat extension, which is used for keepalive monitoring. As a result of the bug, process memory can be read out remotely by an attacker—potentially including certificates, keys, credentials, tokens, or other sensitive data processed by the server. OpenSSL works as a cryptographic library that…

Sponsoring CocoaPods

I’m excited to announce that SoundCloud is sponsoring the development of CocoaPods through a Travis Foundation grant. CocoaPods is an open-source dependency manager for Objective-C projects. Travis Foundation is a non-profit that pairs corporate sponsors with open-source projects to make open source even better. At SoundCloud, our iOS team uses CocoaPods every day to manage the dependencies of our mobile apps. We hope that this sponsorship will lead to improvements that benefit the entire Mac…

Smooth image loading by upscaling

The site is a single-page application that displays a multitude of users’ images. At SoundCloud, we use a technique to make the loading of an image appear smooth and fast. When displaying an image on screen, we want it to display to the user as fast as possible. The images display in multiple locations from a tiny avatar on a waveform to a large profile image. For this reason, we create each image in several sizes. If you are using Gravatar, this technique also applies because you can fetch…

Hassle-free concurrency in Android using RxJava

Both our Android and iOS teams use the reactive programming paradigm to simplify asynchronous, concurrent code in our native mobile apps. For Android, we use Netflix’s RxJava. Matthias Käppler—a SoundCloud engineer and a contributor to the RxJava Android library—blogs about the HOWs and WHYs of RxJava on Android. Tomorrow in London, Matthias will be talking about RxJava at Droidcon. You can grab a drink with him and other members of our Android Team at the SoundCloud Droidcon Drinkup.

Say hello to Sketchy the spam fighter

Sketchy is a spam-fighting, open-source software framework developed by SoundCloud engineers Matt Weiden, Rany Keddo, and Michael Brückner. Sketchy reduces malicious user activity on web applications. You can use it to address several common issues: Detect when a user submits text that contains spam content. Detect when a user submits multiple texts that are nearly identical. Rate-limit malicious actions that users perform repeatedly. Check user signatures such as IP addresses against external…

Building Clojure Services at Scale

SoundCloud has a service-oriented architecture, which allows us to use different languages for different services. With concurrency and scaling in mind, we started to build some services in Clojure due to its interoperability with the JVM, the availability of good quality libraries, and we just plain like it as a language. How do you build distributed, robust, and scalable micro-services in Clojure? Read what Joseph Wilk, an engineer and Clojure enthusiast at SoundCloud, has to say.

Writing your own Karma adapter

Background When we started to work on the new version of our mobile web app, we knew we wanted to run unit tests on a wide variety of clients, mobile devices, PhantomJS, and on Chrome when running locally. Because we practice continuous integration, we knew we also wanted Git hooks and proper results formatting. We chose Karma runner, which is a project from the Angular JS team that provides developers with a “productive testing environment”. One of the advantages that Karma runner offers over…

Responsive Android applications with sane code

A common problem in Android development is that you need to jump off the main UI thread to retrieve data from an IO-based source. At SoundCloud, we use Netflix’s RxJava to simplify asynchronous flows in Android. In an interview given for the Google Developers series “root access Berlin”,  Mustafa Sezgin—who heads up our Mobile Team—explains why we chose RxJava, what it gives us, and he walks through an example-usage pattern: If you are interested in more details, be sure to catch Matthias…

Win a trip to the Barcelona Ruby Conference

The lineup for BaRuCo 2013 looks amazing, with speakers such as Aaron Patterson, Katrina Owen, Sandi Metz, and Ruby’s inventor Yukihiro Matsumoto. The conference is currently SOLD OUT, but we have one extra ticket… and it could be yours! If you win the ticket, SoundCloud will fly you from anywhere in the world to Barcelona, Spain and put you up in a nice Catalonian hotel. How do you enter to win? It’s simple. Just create a command-line interface in Ruby that uses the SoundCloud API. You can use…

Music Hack Day Coming to Toronto!

The first Music Hack Day — organized by our very own Dave Haynes — was held in July 2009 in London, UK. Since then there have been over 30 Music Hack Day events all over the world in cities like San Francisco, New York, Paris, Barcelona and Reykjavik. Music Hack Day events gather programmers, designers and artists to conceptualize, build and demo the future of music. Software, hardware, mobile, web, instruments, art — anything goes as long as it’s music related. I’m happy to announce that…

Berlin Geekettes All-Women Hackathon Roundup

Late last year, six women crowded in to a Mitte cafe booth and listened to Berlin Geekettes founder Jess Erickson share her idea: Berlin’s first all-women hackathon. With SoundCloud’s Amelie Anglade the then newly-appointed the Berlin Geekettes Tech Ambassador, we agreed that it was a great idea to produce the hackathon as a partnership between the Geekettes and the women developers of SoundCloud. Fast forward to the first weekend of March, when the vision became reality: after 24 hours of hacking, 80 women demoed 29 projects across a range of different platforms, from a belt transformed into a game controller to an app aimed to help toddlers learn to do everyday tasks.

Removing 'hotness' parameterhotness

The endpoint has traditionally accepted an parameter for ordering results by either creation date or ‘hotness’. The method for calculating a tracks ‘hotness’ has never been clearly explained, but generally speaking is based on the number of likes and listens a track receives. Recently we started to experience problems with the query that returns tracks ordered by hotness. In the past weeks, these problems started to effect and even cause outages for API users. We have decided that the best way…

SoundCloud at PennApps

Last weekend, we sponsored and attended our first PennApps, the world’s largest student run hackathon held at the University of Pennsylvania in Philadelphia. Nearly 500 students participated from a variety of universities across the US and elsewhere. Students were given 36 hours to get into teams, hack on projects, then show them off to the judges. The results were astounding. In total, over 100 hacks were submitted. Here are some of my personal favourites.

Architecture behind our new Search and Explore experience

Search is front-and-center in the new SoundCloud, key to the consumer experience. We’ve made the search box one of the first things you see, and beefed it up with suggestions that allow you to jump directly to people, sounds, groups, and sets of interest. We’ve also added a brand-new Explore section that guides you through the huge and dynamic landscape of sounds on SoundCloud. We’ve also completely overhauled our search infrastructure, which helps us provide more relevant results, scale with…

Coding for GOOD - SoundCloud API

The web has come a long way. We have APIs that allow developers to create amazing applications quickly, and browser technologies have advanced to the point where JavaScript / HTML and CSS are necessary tools for creating apps. Despite this, there’s still a skills gap and companies are having an increasingly hard time hiring for technical positions. This is the main reason the folks at GOOD teamed up with the Apollo Group to launch Coding for GOOD. A series of coding lessons and final project…

Story Hack Boston Recap

On Saturday, we hosted Story Hack Boston along with P2PU and Mashery at the MIT Media Lab. About 50 people from content and technical backgrounds joined us to create new story telling experiences. The crowd was pretty evenly split, which made for a lot of awesome collaboration.

The Next App Gallery Update

We’re making some changes to how we manage our App Gallery and wanted to take some time to explain them to you, our developer community. The App Gallery is where we highlight interesting and useful SoundCloud powered apps and services for our users. As our developer community continues to grow, it’s even more important that we keep a high bar for apps found in App Gallery. Having a high standard protects the value of being featured in App Gallery for all of our developers while giving our users…

Evolution of SoundCloud’s Architecture

This is a story of how we adapted our architecture over time to accomodate growth. Scaling is a luxury problem and surprisingly has more to do with organization than implementation. For each change we addressed the next order of magnitude of users we needed to support, starting in the thousands and now we’re designing for the hundreds of millions.  We identify our bottlenecks and addressed them as simply as possible by introducing clear integration points in our infrastructure to divide and…

How to Make a SoundCloud Powered Children's Toy

Just over a week ago we had our first internal hackathon at SoundCloud. You can read (and listen!) about it on our community blog or read some of the awesome press coverage the event received. We had over 60 people attend and over 20 projects were demoed. I joined a team organized by Josh Devins to build ToyBox, a children’s toy that plays sounds from SoundCloud in response to physical events. The team consisted of myself, Horaci Cuevas, Josh Devins, and Oliver Hookins. The first task was to…

Shoot yourself in the foot with iptables and kmod auto-loading

As some of you might know, we had an outage yesterday. We believe that in every mistake there is something to learn from, so after each outage we are writing post-mortems. Usually we do this internally because the issues we run into are very specific to our infrastructure. This time we ran into a quite nasty issue which could affect everyone running a linux system with a lot sessions on it and we thought you might be interested to know about that pitfall. What happened? At 4:40pm CEST, we got…

Introducing the CloudSeeder Devkit

Today we’re featuring a guest post from our friends at Retronyms. They’ve built some amazing community features into their app Tabletop using the SoundCloud API and have open sourced their CloudSeeder Devkit. This post was written for us by David Shu. David is a software engineer at the Retronyms and has worked on a number of iOS apps, including Tabletop and Dokobots. He currently resides in San Francisco, CA. We recently built a SoundCloud-powered community into our app Tabletop, a modular audio environment for the iPad, using the CocoaSoundCloudAPI. The project, CloudSeeder, lets users browse, stream, favorite, and comment on Tabletop tracks without ever leaving the app. As developers, we discovered tons of talented users in our Tabletop community. At the same time, our users found inspiration from each other and a new showcase for their creations. To share in the excitement of community creation with all developers, today we’re releasing the CloudSeeder Devkit as open source on Google Code.

Go at SoundCloud

SoundCloud is a polyglot company, and while we’ve always operated with Ruby on Rails at the top of our stack, we’ve got quite a wide variety of languages represented in our backend. I’d like to describe a bit about how—and why—we use Go, an open-source language that recently hit version 1. It’s in our company DNA that our engineers are generalists, rather than specialists. We hope that everyone will be at least conversant about every part of our infrastructure. Even more, we encourage engineers…

Recording and Sharing Phone Calls

A few weeks ago, I attended News Hack Day in San Francisco. News Hack Days are events that bring together journalists, developers and designers for multi day creative coding and brainstorming sessions. I really like the idea of hack days that bring together people from different backgrounds. After chatting with a few journalists, it became obvious to me that recording interviews on the phone is a real pain. I saw this as an opportunity to build a fun app that would make this easier for people.

SocialFolders Releases SoundCloud Ruby Gem

There are many approaches to building libraries that wrap HTTP APIs. For many of our officially supported SDKs we chose to build light wrappers around HTTP client libraries with a few added features to make it easier to work with the SoundCloud API. This approach has a few benefits. It guarantees a certain consistency and is relatively easy to maintain. It’s also fairly future proof. Changes in the HTTP API do not typically require updates to client libraries. Sometimes however, you might be looking for something a bit more feature-full, or with more abstraction from our HTTP API. That’s why I was really happy to see that the great folks at SocialFolders built an alternative SoundCloud Ruby gem and released it to the public. You can check out their blog post about it or go straight to the source on GitHub.

Music Hack Day Barcelona 2012

SoundCloud loves hack days. Our latest hack day adventure brought us to Music Hack Day in Barcelona and we thought we’d share a bit of the great experience we had there. Photo by Thomas Bonte

Waveforms, Let's Talk About Them

Waveforms I’ve worked at SoundCloud for over two years now, and if there’s one thing I do a lot, it’s color waveforms. Tons of them. And, I’ve done it several different ways. Today, Johannes and I are pumped to announce a new JavaScript library called Waveform.js that will assist you in your coloring efforts. But first, let’s take some time to look back and learn from past techniques.

OceaniaSmashing Pumpkins use Premiere 1.5 BETA to debut "Oceania"

Premiere 1.5 Smashing Pumpkins are using a BETA of the next version of Premiere to debut their new record “Oceania.” Here’s what’s new about the app.

Building The Next SoundCloud

This article is also available in: Serbo-Croatian: Pravljenje novog SoundCloud Armenian: SoundCloud (ձայնամպ) ծրագրավորողների համար The front-end team at SoundCloud has been building upon our experiences with the HTML5 widget to make the recently-released Next SoundCloud beta as solid as possible. Part of any learning also includes sharing your experiences, so here we outline the front-end architecture of the new site. Building a single-page application One of the core features of Next…

SoundCloud at HackTO

On Saturday, I joined over a hundred other hackers at HackTO. This has become a regular event in the Toronto tech scene, thanks to excellent organizing by Leila Boujnane and Corey Reid. SoundCloud joined several other API providers, including Atomic Reach, Context.IO, FreshBooks, Shopify, TinEye, Twilio, Trendspottr, WordPress and YellowAPI. The idea behind the event is simple: find a team (or go solo) and build an app using one or more of the APIs presented by sponsors. All apps have to be demoable by 5pm at which time you pitch to the judges who select the 1st, 2nd and 3rd place winners.

Create rich media experiences using timed comments

In the last weeks we got pretty excited about the idea of using timed comments to create and script rich media experiences. Imagine being able to trigger all kinds of visualizations & interactions for a timed comment while playing a track. Timed Comments with Media

Desktop Sharing Kits for OS X and Windows

Last week we announced a new integration with Ableton Live 8, that lets you easily share your sounds from within Ableton Live to SoundCloud. Today we’re making the technology behind that integration available to everyone through our new Desktop Sharing Kits. Mac Desktop Sharing Kit

Wave Raid powered by Timed Comments

One of the SoundCloud API’s most powerful features is the timed comment. At its core it seems simple enough: a piece of text associated with a point in time for a particular track. But where you see timed opinions, I see a light-weight game scripting engine. So when Johannes Wagener told me he was adding an event to our JS SDK, I immediately began work on Wave Raid: Quest for the Timed Comment.

Developer Contest: SoundCloud + Acapela Group Mashup

SoundCloud is teaming up with Acapela Group for our first Developer Contest. Acapela Group offers amazing text to speech solutions and have a variety of SDKs so you can write apps that create sound files using one of their voices (including hip-hop and country!). Take a listen to some sample voices. We’re calling on our developer community to mashup SoundCloud and Acapela. Show us what you can do using a text to speech service with the best audio content platform on the web. Maybe you want to have your email read (or sung!) to you layered on some instrumental or drum tracks. We think there are loads of options for making interesting and accessible apps using these two services.

HTML5 Widget API

Today we’re officially announcing our JavaScript API for the new HTML5 SoundCloud Widget. To use it, just insert the script tag on a page where you’re using our HTML5 Widget. What does it permit? Access and control the properties of HTML5 Widget Load widgets Resources take a look at our documentation for more details and technical descriptions check out our playground to get an overview of functionalities provided by the API Feedback If you’ve got a suggestion, found a bug, please let us know by…

Hacker Time – the first 3 months

Over 3 months ago we started the “Hacker Time” initiative (see here) and now it is time for a recap what’s happened so far. From the outset, people outside of the engineering department were very forthcoming in suggesting ideas for Hacker Time projects. While it was great to see so much interest, the essence of Hacker Time is that engineers create the projects that they’re personally motivated to develop. (It’s absolutely key to keep things separate from the general backlog!). Once that clicked…

Stratus 2 BETA

Stratus is a jQuery powered SoundCloud player that lives at the bottom (or top) of your website or blog. In version 2, we’ve rebuilt Stratus from the ground up to include many requested features, a nicer aesthetic, and a much easier installation. Pumped? Yeh, me too. Here’s how to participate in the BETA:

Moving API Support to Stack Overflow

The SoundCloud Developer Community has grown immensely. We have over ten thousand registered applications and over three hundred showcased in our App Gallery. Our goal as the Platform Team is to provide the best API tools possible, while also providing support and inspiration. So far, our primary channels have been this blog, our Twitter account, and our mailing list, hosted on Google Groups. In the coming months, you can expect some changes as we retool to accommodate the growth of our developer community. We are revisiting everything and doubling down on our efforts to provide the best, most accessible platform possible for building amazing applications that use sound in exciting new ways on the web.

Announcing new Python SDK

It’s no secret that SoundCloud is a Ruby shop, but that doesn’t stop us from giving some love to the Pythonistas in our community. Our old Python API wrapper has been neglected. It doesn’t support OAuth 2 or all of the resources made available by our API. It’s old and crufty and we’re sorry for letting it get that way. In order to make it up to you, we wrote a new one and made it much better.

How we do… Retrospectives

SoundCloud started its agile journey with Scrum and eventually moved to an approach based on Kanban (more on that in one of the next blog posts). Regardless of the methodology we follow, though, we strongly believe that the most important tools an agile team can use to practice continuous improvement are retrospectives. You might ask why we consider the retrospective to be the most important one? We think it is key to constantly learn and to improve based on our experiences. The best way to…

SoundCloud at Music Hack Day SF

This past weekend, hundreds of hackers showed up at the TokBox HQ with a mission to build the future of music. The event started with pitches — giving sponsors a chance to show off their APIs — followed by in-depth workshops where attendees could learn more about each platform and ask questions related to their project. Once the hacking time started, people quickly broke off into groups and got to work.

Tenacious D's Fenix rizes with SoundCloud

This is not The Greatest Website in the World, no. This is just a tribute. I was approached by Tenacious D a week ago with an incredible quest: raising a phallus shaped phoenix onto the internet (oh and the first clip of audio from their new record.) How could I raise this majestic beast with the power of SoundCloud’s legendary API? The answer was simple: Have the sound itself rize the phoenix from its ashes.

oEmbed support for the new HTML5 widget

Today we’ve added support for our new HTML5 widget to the oEmbed endpoint. Aside from a brand new user interface with lots of improvements, the HTML5 widget doesn’t require Adobe Flash anymore. That means the widget will load much faster and finally work well on most mobile platforms, including Android & iOS. And we won’t stop here. We’ll continue to improve the new widget in the coming months.

Music Hack Day London 2011

Last month we attended Music Hack Day London 2011. A special one, Music Hack Day was born in London 3 years ago and we were happy to attend a hack day again in it’s homeland. Picture For those who do not know of Music Hack Day, it’s a great weekend-long event aimed at music and sound hackers. Anyone can attend the event for free, and their goal is to conceptualize and develop innovative hacks of either software or hardware.

Stop! Hacker Time.

At SoundCloud we like to invent new ideas. But we’re not adverse to implementing really great tried and tested ideas like the 20% time concept made famous by Google. We’re calling it Hacker Time. We’re still very much in start-up mode so we’re keen to nurture the spirit of hacking. We’ve been testing out Hacker Time for a few weeks now and we’re excited about its potential, from industry-changing initiatives like “Are we playing yet” to unusual passion projects like the “Owl Octave”. Why we’re…

Front-end JavaScript bug tracking

Proper and effective error tracking is a common issue for front-end JavaScript code compared to back-end environments. We felt this pain as well and experimented with different solutions over the past months on the SoundCloud Mobile site. Analytics The first approach we had was to track errors with Google Analytics. Their library permits to fire custom events and whenever an ajax error would occur, we would log it. The biggest benefit of this tool is to monitor the stability of the site and its…

SoundCloud Strategy for OmniAuth 1.0

OmniAuth is a great little Rack library that standardizes multi-provider authentication for web applications. As OmniAuth says, “it was created to be powerful, flexible, and do as little as possible.” I use it constantly on hacks that mash up multiple API providers. You can now add SoundCloud to the growing list of strategies OmniAuth 1.0 supports.

Connect & Record with the new JavaScript SDK

Today we’re excited to announce a big update to our JavaScript SDK. Aside from improving the existing functionality like authentication, API read requests and oEmbed, we’ve added some sweet new features: API write support: Create comments, follow users, like sounds and update user profiles Audio Streaming: Build your own player powered by SoundCloud Audio recording and uploading: Let your users record and upload sounds to SoundCloud from your website. All of these features can be added to your…

Competition App - Host a contest, powered by SoundCloud

Competitions are a great way to get your fans involved around a particular theme or release. They encourage original content creation and tend to be extremely viral due to the high level of self-promotion involved with participating. The infrastructure you choose to throw these contests on should be accessible from any device while also providing a nice aesthetic backdrop to apply your branding. Users should be able to submit, vote, and comment no matter where they’re coming from: Web, iPhone, Facebook Tab, etc. Today we’re putting forth an open-source app called Competition which we think is a great solution to the problem.

Music Hack Day Boston 2011

Last weekend was another successful Music Hack Day Boston jam-packed with cool hacks, and hackers alike. We are always excited to get involved, meet new and interesting people, and of course hack!

SoundCloud launches the HTML5 Audio Improvement Initiative

We at SoundCloud want to build the best sound player for the web, and we want to do that using the Open Web standards. While working on the native audio features on our mobile site and new widgets, or even as an experiment on the main site, we have discovered that the HTML5 Audio standard is not equally well implemented across all modern browsers and some decisions can be made that would benefit the web audio users and web developers alike. Soundcloud launches the “Are We Playing Yet?” project…

Velocity Europe Birds of a Feather Meetup

With excitement building for the Velocity Conference in Berlin, we are happy to announce a pre-event meet up on Monday, November 7 at the Betahaus in Berlin. The event will be a great opportunity for visitors of the Velocity conference to beat the jet lag with a day of networking and relaxed sessions and talks. To get everyone in the Berlin-state-of-mind, the evening will be capped with a party. Berlin has quickly established itself as one of Europe’s main technology hubs. There are a lot of web…

Introducing the iOS Sharing Kit

Not long ago we released a sharing kit for Android. Today, we’re equally excited to announce a similar sharing kit for iOS. With the sharing kit, you can add SoundCloud sharing to your iPhone and iPad apps with only a few lines of code. We provide everything you need to let your users share their sounds to SoundCloud, including authentication, UI and connections to other social networks like Facebook and Twitter.

SoundCloud Signs Apache Corporate Contributor License Agreement

We just signed the corporate contributor license agreement (CCLA). SoundCloud always was big on open source – we nearly exclusively use open source software in our company and use a lot of Apache projects like Hadoop, Solr, Flume, Zookeeper and Cassandra on our large scale production site. As SoundCloud is using a lot of Apache projects and started to contribute to project we decided to sign the CCLA and enable all our developers to contribute to Apache projects even during work time if that…

Mobile: Unit Testing

When we started the Mobile project early 2011, unit testing JavaScript was one of the goals to tackle on the technical side. The history of custom JavaScript code at SoundCloud up until then rarely included unit tests, so providing references and the necessary ground research was important for both the project at hand as well as for other projects at SoundCloud. This articles aims to provide an overview of the tools we use, what worked well and what we need to improve. Tools When we started the…

Introducing the Android Sharing Kit

Are you an Android developer and want to integrate SoundCloud in your own app? With the new Android Sharing Kit this has become very easy.

Hack Your Way to Berlin

Berlin is playing host to the oh-so-sold-out JSConf.eu conference happening October 1-2. Some members of the SoundCloud engineering crew well be on-site and we thought it would be great if you could come too. We have an extra ticket for the JSConf.eu conference that we are going to raffle away in a hack competition. The rules of engagement are simple: use this podcast on the periodic table from Roman Mars and build a page that presents it in an awesome way. Be it with a very creative playback…

SoundCloud Hits the Road

A quick virtual heads-up for developers, hackers and coding enthusiasts in the Berlin-area: SoundCloud is very happy to support the upcoming Hack and Tell event at the C-base bar on Tuesday, August 30. The Hack and Tell is a monthly meet up where participants have the chance to get on stage for five minutes present their personal or professional projects. For those not presenting, it’s a great evening to mix it up with Berlin-based hackers, meet some members of the SoundCloud development team…

Doing the right thing

The recent outage of SoundCloud was the result of everybody doing the right thing. This totally jives with John Allspaw’s message that looking for a root cause will lead to you finding smart people simply doing their jobs. This is what happened. The technologies at play The number of interactions that escalated to the outage should be interesting for other Rails/MySQL shops out there. We’re using many idiomatic patterns within Rails that ended up having devastating consequences. Cache keys that…

SoundCloud mobile – Proxies

The Problem The mobile version of SoundCloud is a consumer of our own API dog food. That decision was made with the intention to deploy a self-sufficient client application that depends only on a static provider. Our early experiements showed that the attempt we made had some downsides. For example, the implementation of redirects in CORS is not behaving properly and therefore can’t be used with many of the endpoints in our API where we rely on the correct handling. Also classic XHR…

Building the SoundCloud mobile site using backbone.js

Until early this year, there was a gap. A gap between the desktop-targeted main SoundCloud site, what we call the ‘mothership’, and the native iOS (iPhone, iPod touch) and Android applications. A common and frustrating use-case was mobile Twitter: Someone would share a new favorite or upload on Twitter, you tap on it, and it tried to load the regular site on your tiny smartphone screen. Pushing the whole desktop site over a mobile connection would be a waste of precious bandwidth, if you only…

Velocity Conference 2011 – Europe

The venerable O’Reilly Velocity Conference is coming to Berlin on the 8th and 9th of November. SoundCloud is doing what we can to help organize the speaker program and want YOU to speak. Have you turned an operations challenge into a opportunity? Does web performance matter to you, and you’ve done something to prove it? Do you transform your running systems raw data into meaty information? This is an amazing opportunity to share your world class experience with European peers! Achtung! The call…

Sharing to Facebook, Twitter etc

It is now possible to share all public tracks and playlists to the connections of the authenticated user. So if a user is already connected e.g. with Facebook on SoundCloud, this connection can be used from your app to share a track.

Promoting your app in the SoundCloud App Gallery

UPDATE: We made some changes to how we manage the App Gallery. Read about them here: The Next App Gallery Update So you’ve built a fantastic app that you would like to get showcased in the SoundCloud App Gallery? Here’s what you can do to improve your chances of getting your app featured and increase the likelihood of SoundCloud users trying out your app.

New Activities

Today we announce new activities available over the API. As a reminder: Activites are the items on your Dashboard.

New Sharing Interface

Today we announce a better way to share sounds in SoundCloud via the API. The new interface resembles the way sharing happens on the website. You can retrieve, create, update and delete sharings to other users on SoundCloud based on user_id or email address for your tracks and playlists.

MySQL for Statistics – Old Faithful

MySQL turns out to be a good Swiss Army Knife for persistence, if used wisely. Understanding disk access patterns driven by your storage engine is key. Choosing a read or write optimized disk layout will get you very far. We chose a read-optimized disk layout using InnoDB and MySQL for statistics. While our wheels were spinning trying to find out why our statistics storage patterns were causing MongoDB to thrash our disks, we started looking for an emergency alternative with the technology that…

Hack Ideas

If you are looking for some inspiration for your next weekend hack project, here is a list of ideas that could be built using the SoundCloud platform. Feel free to add your own ideas in the comments!

Java API Wrapper

Alongside with the SoundCloud Android app we’ve created a new Java Wrapper for the SoundCloud API and today we can finally release it to the public! It is simple to use, supports OAuth2 and requires a minimum of external dependencies so it should be easily embeddable in both desktop and mobile applications.

Post Mortem on API Outage May 5th 2011

On Thursday May 5th 2011 we had an API outage from ~9:00 UTC to ~14:00 UTC. This was caused by a deploy that included a migration with unexpected behavior and a bug in the application code. The migration ran on tables related to API applications and made these tables unavailable for the time of the migration. When the deploy was finished, a bug causing huge memory consumption brought our app servers down. We recovered by rolling back the faulty code and migrating the database down. After the API…

Experiment 02: Destroying SoundCloud & Instagram

I was fortunate enough to be given the opportunity to help Moby premiere his new record via SoundCloud. I didn’t know what to expect from @TheLittleIdiot‘s latest piece of work. However, I soon learned he had created a phenomenal album with a perfectly crafted and inspiring theme: Destroyed ME == PUMPED I immediately started on a very simple idea: Adding Moby’s photos to a map which users could explore while experiencing the record. A good digital marketing theme cannot stand on it’s own without…

Introducing the Large Hadron Migrator

Rails style database migrations are a useful way to evolve your data schema in an agile manner. Most Rails projects start like this, and at first, making changes is fast and easy. That is until your tables grow to millions of records. At this point, the locking nature of may take your site down for an hour our more while critical tables are migrated. In order to avoid this, developers begin to design around the problem by introducing join tables or moving the data into another layer…

Web Scale Statistics – Failing with MongoDB

As SoundCloud rapidly grows our initial systems need an overhaul. Our scaling strategy has been very realistic, design for 10x our current usage. Our initial statistics system found under https://soundcloud.com/you/stats was made when we were 100k users, living long past its expiration date. Background About a year ago we started off with the goal of redesigning the statistics pages to support 500 playbacks a second. We knew that this would be a write-heavy workload and that to sustain bursts of…

Marbleo.us

This “Puzzle to Unlock” concept came to me straight from Manchester Orchestra’s wonderful label/management team, and we were able to pull it together very quickly with SoundCloud, jQuery, and SoundManager2. The band wanted to build a bit of excitement around the premiere of their new single “Simple Math.” So we developed a way to tease the song with dialogue from the artist and actual clips of audio released as “pieces” to a puzzle that will unlock both the album cover and track. Cool, right…

Stream and Download

UPDATE: Check out our guide on Playing Sounds for the most up to date information. This is an introduction to how you can power your app or website with sounds hosted on SoundCloud. Via our API, you get access to millions of different sounds, and in addition to streaming and downloading tracks, you can access meta data and social features like comments and followings.

Custom Players

UPDATE: Check out our guide on Playing Sounds for the most up to date information. We think the default audio widget from SoundCloud already is one of the sexiest on the web. But maybe you need something a bit simpler or complex and want to tailor the player to your needs. This is where the idea of “custom players” comes into play.

Sharing Sounds

UPDATE: Check out our guides on Authentication and Sharing Sounds for the most recent information. If you build an app or web service that generates any type of sound, it’s easy to connect it to SoundCloud and enable your users to share their creations across the web. Allowing users to share what they create to their existing social networks and the SoundCloud community brings great value in a variety of use cases. If you have an app for beat making, you can let your users post their creations to a dedicated group on SoundCloud. For aspiring music makers, it’s valuable and fun to get feedback from friends on Facebook. And for someone recording an interview with their phone, it’s super handy to be able to post the audio directly to a blog. In addition, a sharing feature is obviously practical for sending files to collaborators or moving audio between devices.

Hello developers!

We are happy to introduce you to our new developer portal developers.soundcloud.com. Here we are bundling up all the SoundCloud platform and developer related products and resources. It will give you the best starting point to dig into our API and all things related! Let us know what you think about it!

We ♥ OAuth 2

From today on, we advise developers to use OAuth 2 as preferred way of API authentication. Our implementation of the Draft 10 OAuth 2 specification is in production for several months months now and we made good experiences with it. Thus we move it from beta to official recommendation. OAuth 2 makes it easier for developers to implement authentication for accessing private resources. The feedback we got from external app developers and our in-house API users has been very positive. This is what…

iOS Sharing Guide

So you are familiar with the SoundCloud API Wrapper and want to use it to share the sounds you upload? And you want to use the existing connections on SoundCloud or make new ones? Awesome. Here’s how to do it.

Let’s Git it On

Being both a mediocre biz dev guy and a nerd means I get to post on the Developer blog as well as our Company blog, and today I’d like to talk to you about Git. What is Git? Git is a free & open source, version control system that when used in conjunction with social coding websites such as GitHub can greatly improve your efficiency for both small and large projects by keeping track of changes, collaborators, and more. You’ll be wondering how you ever lived without it. I’ve had to teach myself…

Music Hack Day Barcelona

Team SoundCloud hacking at Music Hack Day Barcelona Last weekend a team of SoundCloud attended Music Hack Day Barcelona. This blog post is going to talk about what a Music Hack Day is and how we experienced the one in Barcelona. At a Music Hack Day people from around the world gather to spend 24 hours hacking on music and tech. Projects can range from connecting APIs and creating new webapps to building new instruments or mashup audio. The first Music Hack Day happened in London in July 2009 and…

node.js knockout — August 28th & 29th

A few days ago a small team of SoundCloud developers (@goldjunge, @jberkel, @purzelrakete and @sohm) participated in the first node.js knockout competition. Aptly named “Team SoundCloud”, we set out to explore the current state of server side Javascript using node.js and the real-time web. (For those unfamliar with Node.js, head over to http://nodejs.org/ or the project wiki at https://github.com/ry/node/wiki for an introduction.) The node.js Knockout is a competition inspired by the popular…

Of CORS We Do

If you’re a JavaScript head, we’ve got something for you. SoundCloud now supports Cross Origin Resource Sharing, using XMLHttpRequest. Or, to put it another way: no more implausible JSON-P hacks. Some background on CORS can be found here and here.  Our implementation is super-simple:  we let you do GET requests, for our public resources. Full documentation of the feature is on our wiki, but here’s a bit of code to get you started: As we’re just setting headers, the implementation was done as an…

Let’s go backstage. Welcome to the SoundCloud development blog!

Our very cool devs Matas and Hannes. With over 15 developers working on the SoundCloud site nowadays, we’re pumping out a steady stream of new features and scalable web application code. It’s been a long journey building up such a great team, with lots of lessons learned along the way. We’re really thankful for the fact that the web startup community is so open about sharing technical knowledge–this has been super-helpful for us sofar. And we’re still regularly checking blogs of peer companies…