Dropbox

Listing the contents of all team-accessible namespaces

Learn how to list the contents of all of the namespaces across all members of a team using the Dropbox API

From AI to sustainability, why our latest data centers use 400G networking

At Dropbox, AI-powered tools and features are quickly transforming the way our customers find, organize, and understand their data. Dropbox Dash brings AI-powered universal search to all your apps, browser tabs, and cloud docs, while Dropbox AI can summarize and answer questions about the content of your files. To meet the bandwidth requirements of new and future AI workloads—and stay committed to our sustainability goals—the Dropbox networking team recently designed and launched our first data center architecture using highly efficient, cutting edge 400 gigabit per second (400G) ethernet technology.

API updates to better support team spaces

Learn about changes and updates to the Dropbox API to support the latest features of Dropbox teams

Putting everything in its right place with ML-powered file organization

Smart move uses machine learning to analyze a user’s existing subfolder structure and suggest folders where they might want to move their files. Here's how the feature was built.

Is this a date? Using ML to identify date formats in file names

File names play a vital role in facilitating effective communication and organization among teams. Files with cryptic or nonsensical names can quickly lead to chaos—whereas a well-structured naming system can streamline workflows, improve collaboration, and ensure easy retrieval of information. Consistency in naming across different files and file types enables teams to find and share content more efficiently, saving time and reducing frustration.

How we reduced the size of our JavaScript bundles by 33%

When was the last time you were about to click a button on a website, only to have the page shift—causing you to click the wrong button instead? Or the last time you rage-quit a page that took too long to load?

Beta version of major SwiftyDropbox update available

Check out an updated version of the Dropbox Swift SDK with significant new features

Dont you (forget NLP): Prompt injection with control characters in ChatGPT

Update 25/08/23: We've also published a Github repository with updated research on repeated character sequences that induce LLM instability for content-constrained queries.

How the data center site selection process works at Dropbox

For more than a decade, Dropbox has operated its own world-class, exabyte-scale storage system—a multi-metro, hybrid-cloud architecture that spans the globe. But to get to this point, we’ve had to learn a lot about what makes a good data center, and how to pick the perfect site. 

Investigating the impact of HTTP3 on network latency for search

Dropbox is well known for storing users’ files—but it’s equally important we can retrieve content quickly when our users need it most. For the Retrieval Experiences team, that means building a search experience that is as fast, simple, and powerful as possible. But when we conducted a research study in July 2022, one of the most common complaints was that search was still too slow. If search was faster, these users said, they would be more likely to use Dropbox on a regular basis.

Balancing quality and coverage with our data validation framework

At Dropbox, we store data about how people use our products and services in a Hadoop-based data lake. Various teams rely on the information in this data lake for all kinds of business purposes—for example, analytics, billing, and developing new features—and our job is to make sure that only good quality data reaches the lake. 

Accelerating our A/B experiments with machine learning

Like many companies, Dropbox runs experiments that compare two product versions—A and B—against each other to understand what works best for our users. When a company generates revenue from selling advertisements, analyzing these A/B experiments can be done promptly; did a user click on an ad or not? However, at Dropbox we sell subscriptions, which makes analysis more complex. What is the best way to analyze A/B experiments when a user’s experience over several months can affect their decision to subscribe?

Increasing Magic Pocket write throughput by removing our SSD cache disks

When Magic Pocket adopted SMR drives in 2017, one of the design decisions was to use SSDs as a write-back cache for live writes. The main motivation was that SMR disks have a reputation for being slower for random writes than their PMR counterparts. To compensate, live writes to Magic Pocket were committed to SSDs first and acknowledgements were sent to upstream services immediately. An asynchronous background process would then flush a set of these random writes to SMR disks as sequential writes. Using this approach, Magic Pocket was able to support higher disk densities while maintaining our durability and availability guarantees.

Future-proofing our metadata stack with Panda, a scalable key-value store

Metadata is crucial for serving user requests. It also takes up a lot of space—and as we’ve grown, so has the amount of metadata we’ve had to store. This isn’t a bad problem to have, but we knew it was only a matter of time before our metadata stack would need an overhaul.

How we handled a recent phishing incident that targeted Dropbox

We were recently the target of a phishing campaign that successfully accessed some of the code we store in GitHub. No one’s content, passwords, or payment information was accessed, and the issue was quickly resolved. Our core apps and infrastructure were also unaffected, as access to this code is even more limited and strictly controlled. We believe the risk to customers is minimal. Because we take our commitment to security, privacy, and transparency seriously, we have notified those affected and are sharing more here.

Using OAuth 2.0 with offline access

Learn how to use the Dropbox OAuth 2.0 app authorization flow with offline access

Everything in its write place: Cloud storage abstraction with Object Store

Dropbox originally used Amazon S3 and the Hadoop Distributed File System (HDFS) as the backbone of its data storage infrastructure. Although we migrated user file data to our internal block storage system Magic Pocket in 2015, Dropbox continued to use S3 and HDFS as a general-purpose store for other internal products and tools. Among these use cases were crash traces, build artifacts, test logs, and image caching.

Defending against SSRF attacks (with help from our bug bounty program)

Over the past few years, server-side request forgery (SSRF) has received an increasing amount of attention from security researchers. With SSRF, an attacker can retarget a request to internal services and exploit the implicit trust within the network. It often escalates into a critical vulnerability, and in 2021 it was among the top ten web application security risks identified by the Open Web Application Security Project. At Dropbox, it’s the Application Security team’s responsibility to guard against and address SSRF in a scalable manner, so that our engineers can deliver products securely and with as little friction as possible.

We’re using TTVC to measure performance on the web—and now you can too

Nobody likes waiting for software. Snappy, responsive interfaces make us happy, and research shows there’s a relationship between responsiveness and attention1. But maintaining fast-feeling websites often requires tradeoffs. This might mean diverting resources from the development of new features, paying off technical debt, or other engineering work. The key to justifying such diversions is by connecting the dots between performance and business outcomes—something we can do through measurement.

Uploading file data in a performant manner

Learn how to upload files to Dropbox using the Dropbox API in a performant manner

OpenID Connect released in preview

Learn how to Dropbox as an OpenID Connect provider

A look inside our sixth generation of server hardware

For nearly a decade, Dropbox has run its own hardware infrastructure. We have one of the few exabyte-scale storage systems in the world, and running infrastructure at this scale takes a lot of time and planning to build and maintain. Our Hardware team architects, ships, and maintains all of the physical hardware in the data centers that keep Dropbox online, and we recently hit an exciting milestone: we shipped our sixth generation hardware designs. 

Fighting the forces of clock skew when syncing password payloads

A good password manager should be able to securely store, sync, and even autofill your username and password when logging into websites and apps. A password manager like…Dropbox Passwords!

That time we unplugged a data center to test our disaster readiness

On Thursday, November 18, 2021, Dropbox did not go down. On a normal day this would be a non-event—a sign that everything was operating as usual. However, this day was anything but usual. At 5:00 pm PT, a group of Dropboxers were huddled on a Zoom call when the command was given to physically unplug our San Jose data center from the rest of the Dropbox network. 

A day in the life: Engineer onboarding at Dropbox

We’re Adam Hood and Brian Amaratunga—two senior software engineers who joined Dropbox in 2021 as Virtual First employees. This means we spend most of our time working remotely, with physical studios reserved for in-person collaboration.

Introducing Focus, a new open source Gradle plugin

Working with large projects in Gradle can be a pain. As a project grows larger over time, configuration and build times tend to grow with it. Breaking up large projects into lots of modules can help mitigate this, but unless you meticulously follow the seemingly ever-changing advice about project structure and build script configuration, it’s easy to end up with a build system that’s doing far more than it needs to. And while Gradle is doing that extra work, you have to sit and wait.

Making camera uploads for Android faster and more reliable

Camera uploads is a feature in our Android and iOS apps that automatically backs up a user’s photos and videos from their mobile device to Dropbox. The feature was first introduced in 2012, and uploads millions of photos and videos for hundreds of thousands of users every day. People who use camera uploads are some of our most dedicated and engaged users. They care deeply about their photo libraries, and expect their backups to be quick and dependable every time. It’s important that we offer a service they can trust.

How Dropbox Replay keeps everyone in sync

How do you recreate the experience of an in-person screening room with a remote, distributed team? This is one of the reasons we built Dropbox Replay, our new video collaboration tool. Dropbox customers told us the shift to virtual work had turned their once-straightforward review sessions into lengthy, inefficient video calls. It was clear their previous in-person workflows hadn’t made an elegant transition online.

Why we built a custom Rust library for Capture

Dropbox Capture is a new visual communication tool designed to make it easy for teams to asynchronously share their work using screen recordings, video messages, screenshots, or GIFs. There's no formal onboarding required, and you can start sharing your ideas in seconds. In fact, simplicity is key to the Capture experience, and it's a value that also extends down to the development of Capture’s underlying code. 

How we’re making Dropbox data centers 100% carbon neutral

As you may already know, Dropbox runs our infrastructure on a hybrid model—a mix between on-premise hardware and the public cloud. Known as Magic Pocket, it’s our own custom built multi-exabyte infrastructure. After our original migration to Magic Pocket in 2015, our primary goal was to manage large capacity growth at scale. This presented us with challenges including power efficiency and concern for our carbon footprint. We were able to check the box of powering hundreds of millions of users, but we needed to ensure we were being responsible in how we operate. 

Sharing our Engineering Career Framework with the world

At Dropbox, we strive to be a great place for all engineers to grow and be recognized for that growth. Our Engineering Career Framework helps keep us accountable there and is viewable by anyone within the company. Today, we are also making it viewable by anyone outside the company. Our goal in doing so is to be transparent externally on how we think about leveling and career progression on the Engineering team at Dropbox. It also enables others to adapt and apply it to their own organizations.

Sharing our Engineering Career Framework with the world

At Dropbox, we strive to be a great place for all engineers to grow and be recognized for that growth. Our Engineering Career Framework helps keep us accountable there and is viewable by anyone within the company. Today, we are also making it viewable by anyone outside the company. Our goal in doing so is to be transparent externally on how we think about leveling and career progression on the Engineering team at Dropbox. It also enables others to adapt and apply it to their own organizations.

How we managed our global supply chain through a year of lockdown

A little over a year ago, every Dropbox employee received the same email. We were asked to work from home for two weeks to curb the spread of COVID-19. What we didn’t realize then was that a year later, our homes would still be offices and the way we work would have changed permanently.

Optimizing payments with machine learning

It’s probably happened to you at some point: You go to use a service for which you believe you’ve got a paid subscription, only to find that it’s been canceled for non-payment. That’s not only bad for you the customer: It causes negative feelings about the brand, it disrupts what should be a steady flow of revenue to the business, and a customer who finds themselves shut off might decide not to come back.

How image search works at Dropbox

Photos are among the most common types of files in Dropbox, but searching for them by filename is even less productive than it is for text-based files.  When you're looking for that photo from a picnic a few years ago, you surely don't remember that the filename set by your camera was 2017-07-04 12.37.54.jpg.  

Boosting Dropbox upload speed and improving Windows’ TCP stack

One of the best ways to find ways to improve performance, we’ve found, is to work closely with our customers. We love shared troubleshooting sessions with their own engineering teams to find and eliminate bottlenecks. 

Detecting memory leaks in Android applications

Memory leaks occur when an application allocates memory for an object, but then fails to release the memory when the object is no longer being used. Over time, leaked memory accumulates and results in poor app performance and even crashes. Leaks can happen in any program and on any platform, but they’re especially prevalent in Android apps due to complications with activity lifecycles. Recent Android patterns such as ViewModel and LifecycleObserver can help avoid memory leaks, but if you’re following older patterns or don’t know what to look out for, it’s easy to let mistakes slip through.

Automating contract management with Dropbox and Integromat

Learn how to automate workflows like contract management with Integromat’s new Dropbox integration

Atlas: Our journey from a Python monolith to a managed platform

Dropbox, to our customers, needs to be a reliable and responsive service. As a company, we’ve had to scale constantly since our start, today serving more than 700M registered users in every time zone on the planet who generate at least 300,000 requests per second. Systems that worked great for a startup hadn’t scaled well, so we needed to devise a new model for our internal systems, and a way to get there without disrupting the use of our product.

New Paper Endpoints Released in Preview

New endpoints are available to interact with Paper in the File System using the Dropbox API.

How we sped up Dropbox Android app startup by 30%

Application startup is the first thing our users experience after they install an app and then again every single time they launch it. A simple and snappy application brings users a lot more joy than an application that has a ton of features but takes an eternity to load. Realizing this, Dropbox Android team has invested in measuring, identifying, and fixing the issues that were affecting our app startup time. We ended up improving our app start time by 30%, and this is the story of how we did it.

Cannes: How ML saves us $1.7M a year on document previews

Recently, we translated the predictive power of machine learning (ML) into $1.7 million a year in infrastructure cost savings by optimizing how Dropbox generates and caches document previews. Machine learning at Dropbox already powers commonly-used features such as search, file and folder suggestions, and OCR in document scanning. While not all our ML applications are directly visible to the user, they still drive business impact in other ways. 

Why we chose Apache Superset as our data exploration platform

Today the Apache Software Foundation announced Apache Superset as one of its official top-level projects. Apache Superset is a modern, open source data exploration and visualization platform already in use at Airbnb, American Express, Lyft, Nielsen, Rakuten Viki, Twitter, and Udemy among others.

Lessons learned in incident management

At Dropbox, we view incident management as a central element of our reliability efforts. Though we also employ proactive techniques such as Chaos engineering, how we respond to incidents has a significant bearing on our users’ experience. Every minute counts for our users during a potential site outage or product issue.

Dropbox Postman Collection for Team Admin Workflows

Use this Dropbox Postman Collection for managing Team Admin Workflows

Alki, or how we learned to stop worrying and love cold metadata

In this post, we introduce Alki, a new cost efficient petabyte-scale metadata store designed for storing cold, or infrequently accessed, metadata. We’ll discuss the motivations behind building it, its architecture, and various aspects of how we were able to rapidly prototype and then productionize the system with a small team.

PKCE: What and Why?

Come learn about the PKCE OAuth flow! How it works, why it’s valuable, and how to use it to authorize your Dropbox app.

Revamping the Android testing pipeline at Dropbox

At Dropbox, we are proud to have recently hit one billion downloads on what is still our original Android app. Historically, we have been a relatively small team of about a dozen engineers. Recently, though, our org has grown to more than 60 mobile engineers. This growth led to an explosion of features (great!) as well as an explosion of execution time for our testing pipeline (not so great). 

How we designed Dropbox ATF: an async task framework

I joined Dropbox not long after graduating with a Master’s degree in computer science. Aside from an internship, this was my first big-league engineering job. My team had already begun designing a critical internal service that most of our software would use: It would handle asynchronous computing requests behind the scenes, powering everything from dragging a file into a Dropbox folder to scheduling a marketing campaign.

HelloSign API Webinar for the UK, Germany, and France

We’re hosting webinars on the HelloSign API in the UK, Germany, and France. Register today and come learn about the HelloSign API.

Sample: Embedded Requesting Workflows Using the HelloSign and Dropbox APIs

Use this code sample to learn how combine the HelloSign and Dropbox APIs to send Dropbox files for eSignature using an embedded requesting workflow.

Our counterintuitive fix for Android path normalization

Since its inception, the Dropbox Android app has had challenges sharing files and folders with special names. These problems ranged from being unable to see folders with Cyrillic characters to crashing when attempting to load content within certain paths.

Search Files Using the Dropbox API

There’s a new version of the search endpoint available in the Dropbox API. The new version is faster, supports more file types, and has many useful features.

Building with Dropbox Webinar

Join our webinar to learn about Dropbox developer tools, meet our Developer Evangelists, and get started building with the Dropbox API.

JQuery to React: How we rewrote the HelloSign Editor

HelloSign is a Dropbox company that provides a Web-based eSignature solution: If you have a document you want someone to sign, you upload the document file, then show you an editor in which you place all the fields to build the form the recipient will fill out—signatures, dates, initials, etc. You send this prepared form to the recipient. When they’re done signing, everything is reassembled into a PDF with signatures that serves as the legally-binding document. 

Migrating App Permissions and Access Tokens

Learn to migrate your legacy Dropbox app to the new scoped app model. This document guides you through permissions, scopes, short-lived tokens, and refresh tokens.

The software engineering lifecycle: How we built the new Dropbox Plus

A few weeks ago, we released a whole bunch of new features to Dropbox Plus, our paid plan for personal users. While we started as a storage company, we‘ve grown to be a hub to manage your digital life. About 150 people worked on this launch: engineers, product managers, designers, copywriters, and many more.

New and Improved Developer Guides

New developer guides for the Dropbox API are available for OAuth, File Access, Sharing, Extensions, Team Files, Team Administration, Performance, Error Handling, and Detecting Changes.

Keeping sync fast with automated performance regression detection

Sync is a hard distributed systems problem and re-writing the heart of our sync engine on the desktop client was a monumental effort. We’ve previously discussed our efforts to heavily test durability at different layers of the system. Today, we are going to talk about how we ensured the performance of our new sync engine.

Now Available: Scoped apps and enhanced permissions

The Dropbox API has released support for scopes, PKCE, and refresh tokens. You can find the announcement and summary of new features here.

Broccoli: Syncing faster by syncing less

Dropbox syncs petabytes of data every day across millions of desktop clients. It is vital that we constantly improve the sync experience for our users, to increase our users’ productivity in their everyday lives. We also constantly strive to better leverage our infrastructure, increasing Dropbox’s operational efficiency. Because the files themselves are being sent to and from Dropbox during sync, we can leverage redundancies and recurring patterns in common file formats to send files more tersely and consequentially improve performance. Modern compression techniques identify and persist these redundancies and patterns using short bit codes, to transfer large files as smaller, losslessly compressed, files. Thus, compressing files before syncing them means less data on the wire (decreased bandwidth usage and latency for the end user!) and storing smaller pieces in the back end (increased storage cost savings!). To enable these advancements, we measured several common lossless compression algorithms on the incoming data stream at Dropbox including 7zip, zstd, zlib, and Brotli, and we chose to use a slightly modified Brotli encoder, we call Broccoli, to compress files before syncing.

How we migrated Dropbox from Nginx to Envoy

In this blogpost we’ll talk about the old Nginx-based traffic infrastructure, its pain points, and the benefits we gained by migrating to Envoy. We’ll compare Nginx to Envoy across many software engineering and operational dimensions. We’ll also briefly touch on the migration process, its current state, and some of the problems encountered on the way.

New Sample: Image Flipping Extension

Learn to build your first Dropbox Extension using sample code in node.js

How we ensure credible analytics on Dropbox mobile apps

Adding user analytics to a feature may seem like a straightforward task: add events to all the appropriate call sites, pass the event names to product analysts who will use the events for their investigations, and voilà—you have analytics for your feature! 

How Clipchamp integrated with Dropbox to make video editing easy for all

Learn how Clipchamp integrated with Dropbox to make video editing easy for anyone and empower people to share stories through video.

Now Available: Dropbox Embedder

The Dropbox Embedder is officially released and already being used in the wild. Developers can use the Embedder to embed Dropbox content on non-Dropbox surfaces.

Engineering Dropbox Transfer: Making simple even simpler

One of the challenges of application engineering within an established company like Dropbox is to break out of the cycle of incremental improvements and look at a problem fresh. Our colleagues who do user research help by regularly reminding us of the customer’s perspective, but so can our friends and family when they complain that product experiences aren’t as simple as they could be.

Build your own Dropbox Extensions

Dropbox Extensions are now open to all developers. Give users the ability to trigger a workflow in your app directly from their file in Dropbox.

Dropbox App Center in Beta

The Dropbox App Center is launched in beta. The App Center is a centralized place to find tools that enhance your Dropbox experience.

Speeding up a Git monorepo at Dropbox with <200 lines of code

We migrated from Mercurial to Git in 2014 to improve local performance and began consolidating our repositories that hosted our backend code. But as this monorepo grew, we experienced Git performance issues that grew linearly with the number of files we added. Inconveniently, this problem was the most severe on macOS—the platform most of our engineers work on. Fortunately, between upstream improvements in Git itself and a small wrapper of custom code, we’ve been able to speed Git operations up without fragmenting our unified and growing repository.

New guides on error handling and detecting changes

New developer guides! Come check out two new Dropbox API developer guides and learn about error handling and detecting changes.

The Great CoffeeScript to Typescript Migration of 2017

Editor’s preface: When I first joined Dropbox in May 2017, we were at the tail end of our CoffeeScript to TypeScript migration. If you wanted to make changes to a CoffeeScript file, it was considered common courtesy to convert it to TypeScript while you were at it. Parts of our codebase were still using react-dom-factories and we had a custom flux implementation predating Redux.

Improving your User Lifecycle Management with the Dropbox API

Take control of your Dropbox team by improving your user lifecycle management with the Dropbox API

Engineering a disruption tolerant supply chain

On January 23, the Chinese government ordered the lockdown of Wuhan and other cities in Hubei Province in an attempt to contain the COVID-19 outbreak. For many outside of China this still seemed like a regional problem, but it turned out to be the start of the global health crisis we are still living through.

Testing sync at Dropbox

Executing a full rewrite of the Dropbox sync engine was pretty daunting. (Read more about our goals and how we made the decision in our previous post here.) Doing so meant taking the engine that powers Dropbox on hundreds of millions of user’s machines and swapping it out mid-flight. To pull this off, we knew we would need a serious investment in automated testing. Our testing strategy gave us confidence that we were on the right track throughout the rewrite, and today it allows us to continue building and shipping new features on a quick release cycle.

Converting the Dropbox Activity Log into Common Event Format Events

Convert events from your Dropbox activity log into Common Event Format events by following along with this post and using the sample script.

New File and Folder Embedder Launched in Preview

Dropbox has released the File and Folder Embedder in Preview. Start here to learn how to embed Dropbox content in your app using shared links.

Building for reliability at HelloSign

Filling out forms is stressful and boring, so people avoid it. If getting people to fill out forms is critical to your job or your business, you know what a big problem this is. HelloWorks is a HelloSign product that’s solving this problem through automating the process of converting PDF forms into an intelligent, mobile-friendly format that boosts accuracy and completion rates. Data collection is critically important both to the organization requesting the information as well as the person filling out the form. The end-to-end reliability of this process is one of its most important requirements, and one we’ve spent an incredible amount of time and effort on.

File locking using the Dropbox API

Prevent unwanted edits to your files by using the new file locking feature in the Dropbox API.

Rewriting the heart of our sync engine

Over the past four years, we've been working hard on rebuilding our desktop client's sync engine from scratch. The sync engine is the magic behind the Dropbox folder on your desktop computer, and it's one of the oldest and most important pieces of code at Dropbox. We're proud to announce today that we've shipped this new sync engine (codenamed "Nucleus") to all Dropbox users. 

How Otter.ai integrated with Dropbox to streamline transcription of media

You can now automatically transcribe your media files in Dropbox by using the Otter.ai integration. Welcome to the future of work.

Dropbox bug bounty program has paid out over $1,000,000

Over the past five years, our bug bounty program has become an important part of improving our security posture, as it is now for many large tech companies. Transparency and defending the rights of legitimate researchers are cornerstones of the progress we’ve made, and the world is safer for it.

Dropbox bug bounty program has paid out over $1,000,000

Over the past five years, our bug bounty program has become an important part of improving our security posture, as it is now for many large tech companies. Transparency and defending the rights of legitimate researchers are cornerstones of the progress we’ve made, and the world is safer for it. To those outside of the security community, it may seem counterintuitive that you can make your platform safer by encouraging security researchers to attack you, but that’s exactly the value that these programs deliver. This process of discovering and remediating bugs is key to our maintaining a highly secure organization and increasingly hardened product surfaces. Our bug bounty program is only part of having a complete secure development lifecycle program.

Store grand re-opening: loading Android data with coroutines

Many moons ago, I was working at the New York Times and created a library called Store, which was “a Java library for effortless, reactive data loading.” We built Store using RxJava and patterns adopted from Guava’s Cache implementation.

Intelligent DNS based load balancing at Dropbox

The Dropbox Traffic team is charged with innovating our application networking stack to improve the experience for every one of our users—over half a billion of them.

Intelligent DNS based load balancing at Dropbox

The Dropbox Traffic team is charged with innovating our application networking stack to improve the experience for every one of our users—over half a billion of them. This article describes our work with NS1 to optimize our intelligent DNS-based global load balancing for corner cases that we uncovered while improving our point of presence (PoP) selection automation for our edge network.By co-developing the platform capabilities with NS1 to handle these outliers, we deliver positive Dropbox experiences to more users, more consistently.

Evaluating BBRv2 on the Dropbox Edge Network

Spoiler alert: BBRv2 is slower than BBRv1 but that’s a good thing. BBRv1 Congestion Control Three years have passed since “Bottleneck Bandwidth and Round-trip” (BBR) congestion control was released.

Continuous integration and deployment with Bazel

Dropbox server-side software lives in a large monorepo. One lesson we've learned scaling the monorepo is to minimize the number of global operations that operate on the repository as a whole. Years ago, it was reasonable to run our entire test corpus on every commit to the repository.

Continuous integration and deployment with Bazel

Dropbox server-side software lives in a large monorepo. One lesson we’ve learned scaling the monorepo is to minimize the number of global operations that operate on the repository as a whole. Years ago, it was reasonable to run our entire test corpus on every commit to the repository. This scheme became untenable as we added more tests. One obvious inefficiency is the pointless and wasteful execution of tests that can’t possibly be affected by a particular change.

Monitoring server applications with Vortex

At Dropbox, we use monitoring and alerting to gain insight into what our server applications are doing, how they’re performing, and to notify engineers when systems aren’t behaving as expected. Our monitoring systems operate across 1,000 machines, with 60,000 alerts evaluated continuously.

Monitoring server applications with Vortex

At Dropbox, we use monitoring and alerting to gain insight into what our server applications are doing, how they’re performing, and to notify engineers when systems aren’t behaving as expected. Our monitoring systems operate across 1,000 machines, with 60,000 alerts evaluated continuously. In 2018, we reinvented our monitoring and alerting processes, obviating the need for manual recovery and repair. We boosted query speed by up to 10000x in some cases by caching more than 200,000 queries. This work has improved both user experience and trust in the platform, allowing our engineers to monitor, debug, and ship improvements with higher velocity and efficiency.

Modernizing our Android build system: Part I, the planning

One of the biggest challenges of the mobile developer community at Dropbox in 2018 was our custom build system. Our build system was slow, hard to use, and didn’t support some use cases which were out of scope of the original design.

Modernizing our Android build system: Part II, the execution

What does an engineer do after planning and decision on an approach? More planning! This time, the planning was laser-focused on what features to build into the new Android build system.

Modernizing our Android build system: Part II, the execution

Our goal in implementing a Gradle-only solution was to introduce flexibility, but also keep some of the great features BMBF had: guaranteed layered dependency order and reduced boilerplate in build files. We had to lay down the foundation before we could remove BMBF…

How Dropbox Security builds tools for threat detection and incident response

The Dropbox Detection and Response Team (DART) detects and mitigates information security threats to our employees, infrastructure, and customer data. DART ingests security-relevant logs for building detection, threat hunting and responding to potential incidents.

Using machine learning to predict what file you need next, Part 2

At Dropbox, we are building smart features that use machine intelligence to help reduce people’s busywork. Since introducing content suggestions, which we described in our previous blog post, we have been improving the underlying infrastructure and machine learning algorithms that power content su...

Using machine learning to predict what file you need next, Part 2

At Dropbox, we are building smart features that use machine intelligence to help reduce people’s busywork. Since introducing content suggestions, which we described in our previous blog post, we have been improving the underlying infrastructure and machine learning algorithms that power content suggestions.

Enhancing Bandaid load balancing at Dropbox by leveraging real-time backend server load information

Layer-7 load balancing (LB) is a core building block to scale today’s web services. In an earlier post, we introduced Bandaid, the service proxy we built in house at Dropbox that load balances the vast majority of our user requests to backend services.

Enhancing Bandaid load balancing at Dropbox by leveraging real-time backend server load information

Layer-7 load balancing (LB) is a core building block to scale today’s web services. In an earlier post, we introduced Bandaid, the service proxy we built in house at Dropbox that load balances the vast majority of our user requests to backend services. More balanced load distribution among backend servers is desirable because it improves the performance and reliability of our services which benefits our users. Our Traffic/Runtime team recently explored leveraging real-time load information from backend servers to make better load balancing decisions in Bandaid. In this post, we will share the experiences and results, as well as discussing load balancing in general.

Our journey to type checking 4 million lines of Python

Dropbox is a big user of Python. It’s our most widely used language both for backend services and the desktop client app (we are also heavy users of Go, TypeScript, and Rust).

RunBMC: OCP hardware spec solves data center BMC pain points

Open source is not just for software. The same benefits of rapid innovation and community validation apply to hardware specifications as well. That’s why I’m happy to write that the v1.0 of the RunBMC hardware spec has been contributed to Open Compute Project (OCP).

RunBMC: OCP hardware spec solves data center BMC pain points

Open source is not just for software. The same benefits of rapid innovation and community validation apply to hardware specifications as well. That’s why I’m happy to write that the v1.0 of the RunBMC hardware spec has been contributed to Open Compute Project (OCP). Before I get into what BMCs (baseboard management controllers) are and why modern data centers are dependent on them, let’s zoom out to what companies operating at cloud scale have learned.

The (not so) hidden cost of sharing code between iOS and Android

Until very recently, Dropbox had a technical strategy on mobile of sharing code between iOS and Android via C++. The idea behind this strategy was simple—write the code once in C++ instead of twice in Java and Objective C.

The (not so) hidden cost of sharing code between iOS and Android

Until very recently, Dropbox had a technical strategy on mobile of sharing code between iOS and Android via C++. The idea behind this strategy was simple—write the code once in C++ instead of twice in Java and Objective C. We adopted this C++ strategy back in 2013, when our mobile engineering team was relatively small and needed to support a fast growing mobile roadmap. We needed to find a way to leverage this small team to quickly ship lots of code on both Android and iOS.

SMR: What we learned in our first year

A year ago, we became the first major tech company to adopt high-density SMR (Shingled Magnetic Recording) technology for our storage drives. At the time, we faced a challenge: while SMR offers major cost savings over conventional PMR (Perpendicular Magnetic Recording) drives, the technology is slow...

SMR: What we learned in our first year

When the first production machines started arriving in September, it was only natural to have a little apprehension and think, “Does this actually work? What if it breaks?!” It was raw, completely new technology—it was terrifying. And then ... nothing happened. That is, everything simply worked.

Redux with Code-Splitting and Type Checking

Before We Get Started This article assumes a working knowledge of Redux, React, React-Redux, TypeScript, and uses a little bit of Lodash for convenience. If you’re not familiar with those subjects, you might need to do some Googling. You can find the final version of all the code here.

The Programmer Mindset: Main Debug Loop

Validating ONLY with tests is basically flying the plane on instrumentation, versus being able to look out the windshield. Flying visually and by muscle-memory is both more efficient and safer, in conjunction with instrumentation. You’re much less likely to hit a mountain by mistake.

Your System is not a Sports Team

It’s the responsibility of an engineering team to do what’s right for the company, not to advocate for the system they own. Engineering teams need to be oriented around a mission not a system to avoid narrow-minded decision-making.

Your System is not a Sports Team

It’s the responsibility of an engineering team to do what’s right for the company, not to advocate for the system they own. Engineering teams need to be oriented around a mission not a system to avoid narrow-minded decision-making.

Athena: Our automated build health management system

At Dropbox, we run more than 35,000 builds and millions of automated tests every day. With so many tests, a few are bound to fail non-deterministically or “flake.” Some new code submissions are bound to break the build, which prevents developers from cutting a new release.

How we optimized Magic Pocket for cold storage

Ever since we launched Magic Pocket, our in-house multi-exabyte storage system, we’ve been continuously looking for opportunities to improve efficiency, while maintaining our high standards for reliability.

Using machine learning to predict what file you need next

As we laid out in our blog post introducing DBXi, Dropbox is building features to help users stay focused on what matters. Searching through your content can be tedious, so we built content suggestions to make it easier to find the files you need, when you need them.

Embracing papercuts

Team growth requires giving people room to make mistakes. Figuring out which mistakes are just “papercuts” and which are critical is one of the most difficult challenges in engineering leadership.

Towards better vendor security assessments

Addressing vendor security is a significant and inescapable problem for any modern company. Like many other companies, Dropbox has external third-party integrations with our products, and we also use vendors for internal services, from HR workflows to sales, marketing, and IT.

On working with designers

Editor’s note: On January, 18, 2019 the Dropbox Design blog featured a post from Product Designer, Jenny Wen, on working with engineers. This post covers the topic from an engineer’s perspective.

Don’t lead by example

This is the first in a series of posts that Dropbox Principal Engineer James Cowling has published on his personal Medium blog about technical leadership.

Incrementally migrating over one million lines of code from Python 2 to Python 3

The Dropbox desktop client is relied on by millions of users across the world to save their most important files and keep them in sync across their devices.

Finding Kafka’s throughput limit in Dropbox infrastructure

Apache Kafka is a popular solution for distributed streaming and queuing for large amounts of data. It is widely adopted in the technology industry, and Dropbox is no exception.

The scalable fabric behind our growing data center network

Dropbox needs its underlying network infrastructure to be reliable, high-performing, cost-effective, and truly scalable. In previous posts we described how the edge network was designed to improve user performance, and how the supporting multi-terabit backbone network spans continents to interconnec...

Automating Datacenter Operations at Dropbox

&nbsp; Introduction As a company that manages our own infrastructure we need to be able to rapidly install new server capacity and ensure that the equipment entering our production environment is highly reliable.

Courier: Dropbox migration to gRPC

Dropbox runs hundreds of services, written in different languages, which exchange millions of requests per second. At the core of our Service Oriented Architecture is Courier, our gRPC-based Remote Procedure Call (RPC) framework.

Cape Technical Deep Dive

We introduced Cape in a previous post. In a nutshell, Cape is a framework for enabling real-time asynchronous event processing at a large scale with strong guarantees. It has been over a year since the system was launched. Today Cape is a critical component for Dropbox infrastructure.

Offensive testing to make Dropbox (and the world) a safer place

Dropbox invests heavily in our security program. We have lots of teams dedicated to securing Dropbox, each working on exciting things. Some recent examples covered on our tech blog include: Our Product Security team rolled out support for WebAuthn to boost user adoption of two-step verification...

Cross shard transactions at 10 million requests per second

Dropbox stores petabytes of metadata to support user-facing features and to power our production infrastructure. The primary system we use to store this metadata is named Edgestore and is described in a previous blog post, (Re)Introducing Edgestore.

Crash reporting in desktop Python applications

One of the greatest challenges associated with maintaining a complex desktop application like Dropbox is that with hundreds of millions of installs, even the smallest bugs can end up affecting a very large number of users.

What we learned at our first JS Guild Summit

&nbsp; At Dropbox, we work to keep teams flowing—so earlier this month, we convened a group of our frontend engineers to do just that. At the beginning of October, we held the first JS (JavaScript) Guild Summit at our San Francisco headquarters to bring together frontend engineers from our ...

Dropbox traffic infrastructure: Edge network

In this post we will describe the Edge network part of Dropbox traffic infrastructure. This is an extended transcript of our NginxConf 2018 presentation.

Using machine learning to index text from billions of images

In our previous blog posts, we talked about how we updated the Dropbox search engine to add intelligence into our users’ workflow, and how we built our optical character recognition (OCR) pipeline.

Validating performance and reliability of the new Dropbox search engine

In our previous post, we discussed the architecture and ML capabilities that we built into our new search engine, named Nautilus. In this post, we will discuss the process that we undertook to ensure optimal performance and reliability.

Architecture of Nautilus, the new Dropbox search engine

Over the last few months, the Search Infrastructure engineering team at Dropbox has been busy releasing a new full-text search engine called Nautilus, as a replacement for our previous search engine.

How we rolled out one of the largest Python 3 migrations ever

Dropbox is one of the most popular desktop applications in the world: You can install it today on Windows, macOS, and some flavors of Linux. What you may not know is that much of the application is written using Python.

Machine intelligence at Dropbox: An update from our DBXi team

Our workdays are getting noisier. Never-ending emails, text messages, constant notifications from more apps and more platforms—it’s disruptive and distracting. And then there’s content. All kinds of documents, spreadsheets, presentations, videos, and photos.

Live-hacking Dropbox @ H1-3120

In 2018, Dropbox has focused on improving our world-class bug bounty program. From increasing bounties to protecting our researchers, we’re always looking for more creative and meaningful ways to stay ahead of the game when it comes to running this program.

Migrating from Underscore to Lodash

Modernizing the front-end stack The core Dropbox web application is 10 years old and used by millions of users per day. Hundreds of front-end engineers across multiple cities actively work on it. Unsurprisingly, our codebase is very large and somewhat irregular.

Building better compression together with DivANS

Compressing your files is a good way to save space on your hard drive. At Dropbox’s scale, it’s not just a good idea; it is essential. Even a 1% improvement in compression efficiency can make a huge difference.

Extending Magic Pocket Innovation with the first petabyte scale SMR drive deployment

Magic Pocket, the exabyte scale custom infrastructure we built to drive efficiency and performance for all Dropbox products, is an ongoing platform for innovation. We continually look for opportunities to increase storage density, reduce latency, improve reliability, and lower costs.

How we’re winning the battle against flaky tests

Testing is a crucial part of maintaining a code base, but not all tests validate what they’re testing for. Flaky tests—tests that fail sometimes but not always—are a universal problem, particularly in UI testing.

MacOS monitoring the open source way

Let’s say a machine in your corporate fleet gets infected with malware. How would you detect it? How could you find out what happened on the machine? What did the malware do? Did it steal your browser’s passwords? What network connections did the malware make? Was it looking for crypto currency?...

Protecting Security Researchers

At Dropbox, we encourage, support, and celebrate independent open security research. One way we do this is via our bug bounty program. We recently tripled our rewards to industry leading values.

Balancing open source and proprietary IP—they can co-exist

Open source software can provide significant benefits to an organization—it can decrease product development time, distribute development across a community, and attract developers to your organization. It’s because of these benefits that we at Dropbox love open source.