MongoDB

We Replaced an SSD with Storage Class Memory. Here is What We Learned.

On April 2, 2019 Intel Optane Persistent Memory became the first commercially available storage class memory (SCM). SCM is a storage device that sits on a memory bus; in contrast, traditional storage devices like SSD are attached to the PCIe bus. We were interested how SCM, being much closer to the CPU, affects performance of real applications. We compared the performance of MongoDB’s storage engine WiredTiger on two Optane storage devices: one packaged as SCM and another one packaged as SSD. Although these devices deliver vastly different performance when accessed in a raw form, their performance under the entire software stack, including OS, file system and the application, is not that different, because in-memory caches effectively mask latency.

Getting storage engines ready for fast storage devices

At MongoDB, we converted the WiredTiger storage engine to use memory-mapped files instead of system calls for I/O, and to batch the overhead of file-system housekeeping operations. These changes improved performance by up to 63% on some I/O-intensive benchmarks.

Transpiling between any programming languages (Part 1)

The Compass team has written a lightweight, extendable transpiler that translates BSON to and from any language. This tool allows developers to work in one language while being able to export the queries and aggregations they build to other languages. Most compilers are one-to-one, or less commonly, one-to-many or many-to-one. There are hardly any many-to-many transpilers. To avoid having to start from scratch, we leveraged the open source parsing tool ANTLR which provided us with a set of compiler tools along with preexisting grammars for the languages we needed. We successfully minimized the amount of additional complexity by coming up with a creative set of class hierarchies that reduced the amount of work needed from n² to n.

Repeatable performance tests: CPU options are best disabled

In an effort to improve repeatability, the MongoDB Performance team set out to reduce noise on several performance test suites run on EC2 instances. After thinking about our assumptions and the experiment setup, we began by recording data about our current setup and found no evidence of particularly good or bad EC2 instances. In the next step, we investigated IO and found that EBS instances are the stable option for us. Having found a very stable behavior as far as disks were concerned, this third and final experiment turns to tuning CPU related knobs to minimize noise from this part of the system.

Repeatable performance tests: EBS instances are the stable option

In an effort to improve repeatability, the MongoDB Performance team set out to reduce noise on several performance test suites run on EC2 instances. After thinking about our assumptions and the experiment setup, we began by recording data about our current setup and found no evidence of particularly good or bad EC2 instances. However, we found that the results of repeated tests had a high variance. Given our test data and our knowledge of the production system, we had observed that many of the noisiest tests did the most IO (being either sensitive to IO latency, or bandwidth). After performing the first baseline tests, we therefore decided to focus on IO performance through testing both AWS instance types and IO configuration on those instances.

Repeatable performance tests: EC2 instances are neither good nor bad

In an effort to improve repeatability, the MongoDB Performance team set out to reduce noise on several performance test suites run on EC2 instances. At the beginning of the project, it was unclear whether our goal of running repeatable performance tests in a public cloud was achievable. Instead of debating the issue based on assumptions and beliefs, we decided to measure noise itself and see if we could make configuration changes to minimize it. After thinking about our assumptions and the experiment setup, we began by recording data about our current setup.

Reducing variability in performance tests on EC2: Setup and Key Results

On the MongoDB Performance team, we use EC2 to run daily system performance tests. After building a continuous integration system for performance testing, we realized that there were sources of random variation in our platform and system configuration which made a lot of our results non-reproducible. The run to run variation from the platform was bigger than the changes in MongoDB performance that we wanted to capture. To reduce such variation - environmental noise - from our test configuration, we set out on a project to measure and control for the EC2 environments on which we run our tests.

Causal guarantees are anything but casual

Traditional databases, because they service reads and writes from a single node, naturally provide sequential ordering guarantees for read and write operations known as "causal consistency". A distributed system can provide these guarantees, but in order to do so, it must coordinate and order related events across all of its nodes, and limit how fast certain operations can complete. While causal consistency is easiest to understand when all data ordering guarantees are preserved – mimicking a vertically scaled database, even when the system encounters failures like node crashes or network partitions – there exist many legitimate consistency and durability tradeoffs that all systems need to make. MongoDB has been continuously running — and passing — Jepsen tests for years. Recently, we have been working with the Jepsen team to test for causal consistency. With their help, we learned how complex the failure modes become if you trade consistency guarantees for data throughput and recency.

Pruning Dynamic Rebuilds With libabigail

Complex C++ projects frequently struggle with lengthy build times. Splitting a project into multiple dynamically-linked components can give developers faster incremental rebuilds and shorter edit-compile-test cycles than relying on static linking, especially when there are a large number of test binaries. However, build systems usually do not realize all of the possible gains in dynamic incremental rebuilds due to how they handle transitive library dependencies. Red Hat's ABI introspection library libabigail offers one possible path to eliminating unnecessary transitive re-linking for some classes of source modifications.

Considering the Community Effects of Introducing an Official MongoDB Go Driver

What do you do when an open-source project you rely on no longer meets your needs?  When your choice affects not just you, but a larger community, what principles guide your decision? Submitting patches is often the first option, but you're at the mercy of the maintainer to accept them.  If the changes you need are sweeping, substantial alterations, the odds of acceptance are low.  Eventually, only a few realistic options remain: find an alternative, fork the project, or write your own replacement.  Everyone who depends on open source faces this conundrum at one time or another. After relying for years on the community-developed mgo Go driver for MongoDB, MongoDB has begun work on a brand-new, internally-developed, open-source Go driver.  We know that releasing a company-sponsored alternative to a successful, community-developed project creates tension and uncertainty for users, so we did not make this decision lightly. We carefully considered how our choice would affect current and future Go users of MongoDB.

A Founder's Reflections on 10 Years of MongoDB

As of today, on our 10 year gitaversary , MongoDB is now a publicly traded company. You can read more of my thoughts on this on our company blog .

Investing in CS4All: One Year Later

When a couple of New York City high school teachers partnered with MongoDB to teach computer science, did they succeed? Their curriculum was untested, and they were teaching in difficult districts where most students are from poor and minority families. I talked with these two teachers, Jeremy Mellema and Timothy Chen, back in September, when they had completed a summer fellowship at MongoDB and had just started teaching their curriculum; at the end of the academic year this spring, I visited Jeremy and Tim again to find out the result. Their successes were sparse and partial. They discovered that their students' poor reading skills were a barrier to learning to code, and that teaching new coders how to solve problems is, itself, an unsolved problem. With a coarse unit of iteration—a school semester—it is painfully slow to experiment and find teaching methods that work. But even partial wins make a difference for individual kids, and the support of professional engineers at companies like MongoDB can be a powerful accelerant.

Farewell, Solaris

[Update 8/29/17 3:45pm: edited to contain more specifics regarding Solaris downloads and security patches.] Solaris was the first “real operating system” I ever used. The Brown University Computer Science Department was a Sun Microsystems shop when I was an undergraduate there in the late 90s. When I took the operating systems lab class, CS-169, we implemented a toy version of Sun’s research operating system Spring OS. Several of my contemporaries in the CS Department went on to work at Sun, and developed or advanced many of the technologies that made Solaris great, like ZFS, dtrace, libumem, mdb, doors, and zones. The Solaris Linkers and Libraries Guide remains one of the best ways to develop an understanding of shared library internals. The first startup I worked for developed on Solaris x86, because the team knew Solaris well. Today, many of my co-workers on the server engineering team here at MongoDB share that formative experience with Solaris. We have a great deal of collective nostalgia and appreciation for Solaris and the amazing engineering effort that went into its development. So it is, for many of us at MongoDB, bittersweet to announce that MongoDB is terminating support for Solaris. Effective immediately, we plan to cease production of new builds of MongoDB for Solaris, across all supported versions of MongoDB. Existing release artifacts for Solaris will continue to be made available, but no new releases will be issued, barring a critical issue raised under an existing support contract covering MongoDB versions 3.0 through 3.4 running on Solaris. We will continue to fix critical flaws for the community, regardless of where found or how reported. Anyone can report a security vulnerability by using our Security project to create an account, then a ticket, describing the vulnerability. This was not an easy decision for us to make, and we feel that it is important to provide some background on why we have made what may seem at first to be a capricious decision.

A MongoDB Engineering Response to the Anti-Diversity-Effort Manifesto

The following is an email I sent to the MongoDB engineering team this past Friday. At the behest of the team, I am now making it public. —Eliot

Presenting WinKerberos

WinKerberos is a Python module providing Kerberos facilities to Python applications on Windows, where PyKerberos does not work. WinKerberos can be used as a drop-in replacement on Windows for client applications using PyKerberos. Why write a new Kerberos module for Python? The MongoDB Enterprise

Breaking the WiredTiger Logjam: The Wait-Free Solution (2/2)

Part one of this pair explored the original algorithm the WiredTiger write-ahead log used to consolidate writes in order to minimize IO. It used atomic compare-and-swap operations in two phases to accomplish this without time-consuming locking. This algorithm worked extremely well as long as there were no more than a few threads running per core. But its reliance on busy-waiting to avoid locking caused a logjam when the number of threads increased beyond that limit -- a serious problem given that many MongoDB workloads would have a large number of threads per core. This issue was blocking MongoDB’s goal of making WiredTiger the default storage engine in v3.2. This story has a happy ending thanks to my colleague, Senior Technical Service Engineer Bruce Lucas. Bruce had initially uncovered the logjam and reported it to me; together, we overcame it without compromising any other workloads. Because Bruce’s mindset was not colored by the legacy of the original approach, he was able to provide the critical insight that paved the way for the solution, allowing WiredTiger to become the default storage engine in v3.2.

Breaking the WiredTiger Logjam: The Write-Ahead Log (1/2)

Code can't be optimized; it can only be optimized for a set of conditions. When conditions change, optimizations can become bottlenecks, and when that happens, a thorough audit of assumptions might well hold the key to the solution. The WiredTiger write-ahead log exemplifies this principle. It’s a critical codepath within a high-performance storage engine, and I have optimized it heavily to avoid I/O and locking. But some of the conditions I had initially targeted became invalid when WiredTiger became a storage engine in MongoDB. When a colleague of mine investigated a case of negative scaling found during testing, he uncovered a serious bottleneck in the write-ahead log… call it a "logjam". That investigation ultimately led us to rethink our assumptions and optimize for new conditions. We validated the new approach with a quick prototype, and then covered all the intricacies and edge cases to produce a fully realized solution. In part one of this two-part series, I’ll dive deep into the innards of the WiredTiger write-ahead log. I’ll show how it orchestrates many threads writing to a single buffer without locking, and I’ll explain how two conflicts between that design and the new conditions produced the logjam. Part two will focus on how we eliminated the bottleneck. I’ll analyze its root causes, describe the key insight that enabled our solution, and detail the new algorithm and how it reflects our current conditions.

When Switching Projects, Check your Assumptions or Risk Disaster

On January 10, I released a badly broken version of the MongoDB C Driver, libmongoc 1.5.2. For most users, that version could not connect to a server at all! Luckily, in under 24 hours a developer reported the bug, I reverted the mistake and released a fix. Although it was resolved before it did any damage, this is among the most dramatic mistakes I've made since I switched from the PyMongo team to libmongoc almost two years ago. My error stemmed from three mistaken assumptions I've had ever since I changed projects. What were they? Inception Here's how the story began. In December, a libmongoc user named Alexey pointed out a longstanding limitation: it would only resolve hostnames to IPv4 addresses. Even if IPv6 address records existed for a hostname, the driver would not look them up -- when it called getaddrinfo on the hostname to do the DNS resolution, it passed AF_INET as the address family, precluding anything but IPv4. So if you passed the URI  mongodb://example.com, libmongoc resolved "example.com" to an IPv4 address like 93.184.216.34 and tried to connect to it. If the connection timed out, the driver gave up.

Testing Linearizability with Jepsen and Evergreen: “Call Me Continuously!”

What do you do with a third-party tool that proves your application lacks a feature? Add that tool to your continuous integration system (after adding the feature, of course)! In our case we have added linearizable reads to MongoDB 3.4 and use Jepsen to test it. What is Linearizability? Linearizability is a property of distributed systems first introduced by Herlihy & Wing in their July 1990 article "Linearizability: a correctness condition for concurrent objects" (ACM Transactions on Programming Languages and Systems Journal). Peter Bailis probably provides the most accessible explanation of linearizability: "writes should appear to be instantaneous. Imprecisely, once a write completes, all later reads (where “later” is defined by wall-clock start time) should return the value of that write or the value of a later write. Once a read returns a particular value, all later reads should return that value or the value of a later write."

BSD YouTubers Honor Us With Dramatic Readings

Avery Rosen | January 19, 2017 | kudos editor

D3 Round Two: How to Blend HTML5 Canvas with SVG to Speed Up Rendering

Soon after the publication of "Digging Into D3 Internals to Eliminate Jank ," I was pleased to see that it had sparked a discussion on Twitter, with D3 community members, notably Noah Veltman and Mike Bostock, sharing suggestions for improving our rendering solution. A suggestion we received both in this discussion and on lobste.rs was to use canvas to render the data points. We had originally avoided canvas because of time constraints, lack of team familiarity with canvas, and the complications it introduced with regards to mouse interactions. However, Noah proposed a combination of SVG and canvas that strikes a balance between canvas' performance and SVG's convenience, complete with a demo. It piqued my interest, and so I decided to explore it in some more detail here.