Alternatives to RethinkDB logo

Alternatives to RethinkDB

MongoDB, CouchDB, CockroachDB, Couchbase, and Firebase are the most popular alternatives and competitors to RethinkDB.
283
358
+ 1
307

What is RethinkDB and what are its top alternatives?

RethinkDB is built to store JSON documents, and scale to multiple machines with very little effort. It has a pleasant query language that supports really useful queries like table joins and group by, and is easy to setup and learn.
RethinkDB is a tool in the Databases category of a tech stack.
RethinkDB is an open source tool with 24.5K GitHub stars and 1.8K GitHub forks. Here’s a link to RethinkDB's open source repository on GitHub

Top Alternatives to RethinkDB

  • MongoDB

    MongoDB

    MongoDB stores data in JSON-like documents that can vary in structure, offering a dynamic, flexible schema. MongoDB was also designed for high availability and scalability, with built-in replication and auto-sharding. ...

  • CouchDB

    CouchDB

    Apache CouchDB is a database that uses JSON for documents, JavaScript for MapReduce indexes, and regular HTTP for its API. CouchDB is a database that completely embraces the web. Store your data with JSON documents. Access your documents and query your indexes with your web browser, via HTTP. Index, combine, and transform your documents with JavaScript. ...

  • CockroachDB

    CockroachDB

    It allows you to deploy a database on-prem, in the cloud or even across clouds, all as a single store. It is a simple and straightforward bridge to your future, cloud-based data architecture. ...

  • Couchbase

    Couchbase

    Developed as an alternative to traditionally inflexible SQL databases, the Couchbase NoSQL database is built on an open source foundation and architected to help developers solve real-world problems and meet high scalability demands. ...

  • Firebase

    Firebase

    Firebase is a cloud service designed to power real-time, collaborative applications. Simply add the Firebase library to your application to gain access to a shared data structure; any changes you make to that data are automatically synchronized with the Firebase cloud and with other clients within milliseconds. ...

  • Redis

    Redis

    Redis is an open source, BSD licensed, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets. ...

  • Cassandra

    Cassandra

    Partitioning means that Cassandra can distribute your data across multiple machines in an application-transparent matter. Cassandra will automatically repartition as machines are added and removed from the cluster. Row store means that like relational databases, Cassandra organizes data by rows and columns. The Cassandra Query Language (CQL) is a close relative of SQL. ...

  • PostgreSQL

    PostgreSQL

    PostgreSQL is an advanced object-relational database management system that supports an extended subset of the SQL standard, including transactions, foreign keys, subqueries, triggers, user-defined types and functions. ...

RethinkDB alternatives & related posts

MongoDB logo

MongoDB

51.7K
41.4K
4K
The database for giant ideas
51.7K
41.4K
+ 1
4K
PROS OF MONGODB
  • 822
    Document-oriented storage
  • 585
    No sql
  • 544
    Ease of use
  • 462
    Fast
  • 404
    High performance
  • 251
    Free
  • 212
    Open source
  • 177
    Flexible
  • 139
    Replication & high availability
  • 107
    Easy to maintain
  • 39
    Querying
  • 35
    Easy scalability
  • 34
    Auto-sharding
  • 33
    High availability
  • 29
    Map/reduce
  • 26
    Document database
  • 24
    Easy setup
  • 24
    Full index support
  • 15
    Reliable
  • 14
    Fast in-place updates
  • 13
    Agile programming, flexible, fast
  • 11
    No database migrations
  • 7
    Enterprise
  • 7
    Easy integration with Node.Js
  • 5
    Enterprise Support
  • 4
    Great NoSQL DB
  • 3
    Aggregation Framework
  • 3
    Drivers support is good
  • 3
    Support for many languages through different drivers
  • 2
    Schemaless
  • 2
    Managed service
  • 2
    Easy to Scale
  • 2
    Fast
  • 2
    Awesome
  • 1
    Consistent
CONS OF MONGODB
  • 5
    Very slowly for connected models that require joins
  • 3
    Not acid compliant
  • 1
    Proprietary query language

related MongoDB posts

Jeyabalaji Subramanian

Recently we were looking at a few robust and cost-effective ways of replicating the data that resides in our production MongoDB to a PostgreSQL database for data warehousing and business intelligence.

We set ourselves the following criteria for the optimal tool that would do this job: - The data replication must be near real-time, yet it should NOT impact the production database - The data replication must be horizontally scalable (based on the load), asynchronous & crash-resilient

Based on the above criteria, we selected the following tools to perform the end to end data replication:

We chose MongoDB Stitch for picking up the changes in the source database. It is the serverless platform from MongoDB. One of the services offered by MongoDB Stitch is Stitch Triggers. Using stitch triggers, you can execute a serverless function (in Node.js) in real time in response to changes in the database. When there are a lot of database changes, Stitch automatically "feeds forward" these changes through an asynchronous queue.

We chose Amazon SQS as the pipe / message backbone for communicating the changes from MongoDB to our own replication service. Interestingly enough, MongoDB stitch offers integration with AWS services.

In the Node.js function, we wrote minimal functionality to communicate the database changes (insert / update / delete / replace) to Amazon SQS.

Next we wrote a minimal micro-service in Python to listen to the message events on SQS, pickup the data payload & mirror the DB changes on to the target Data warehouse. We implemented source data to target data translation by modelling target table structures through SQLAlchemy . We deployed this micro-service as AWS Lambda with Zappa. With Zappa, deploying your services as event-driven & horizontally scalable Lambda service is dumb-easy.

In the end, we got to implement a highly scalable near realtime Change Data Replication service that "works" and deployed to production in a matter of few days!

See more
Robert Zuber

We use MongoDB as our primary #datastore. Mongo's approach to replica sets enables some fantastic patterns for operations like maintenance, backups, and #ETL.

As we pull #microservices from our #monolith, we are taking the opportunity to build them with their own datastores using PostgreSQL. We also use Redis to cache data we’d never store permanently, and to rate-limit our requests to partners’ APIs (like GitHub).

When we’re dealing with large blobs of immutable data (logs, artifacts, and test results), we store them in Amazon S3. We handle any side-effects of S3’s eventual consistency model within our own code. This ensures that we deal with user requests correctly while writes are in process.

See more
CouchDB logo

CouchDB

403
443
137
HTTP + JSON document database with Map Reduce views and peer-based replication
403
443
+ 1
137
PROS OF COUCHDB
  • 43
    JSON
  • 29
    Open source
  • 18
    Highly available
  • 12
    Partition tolerant
  • 11
    Eventual consistency
  • 7
    Sync
  • 4
    Attachments mechanism to docs
  • 4
    REST API
  • 4
    Multi master replication
  • 3
    Changes feed
  • 1
    REST interface
  • 1
    js- and erlang-views
CONS OF COUCHDB
    Be the first to leave a con

    related CouchDB posts

    Jonathan Pugh
    Software Engineer / Project Manager / Technical Architect · | 25 upvotes · 1.4M views

    I needed to choose a full stack of tools for cross platform mobile application design & development. After much research and trying different tools, these are what I came up with that work for me today:

    For the client coding I chose Framework7 because of its performance, easy learning curve, and very well designed, beautiful UI widgets. I think it's perfect for solo development or small teams. I didn't like React Native. It felt heavy to me and rigid. Framework7 allows the use of #CSS3, which I think is the best technology to come out of the #WWW movement. No other tech has been able to allow designers and developers to develop such flexible, high performance, customisable user interface elements that are highly responsive and hardware accelerated before. Now #CSS3 includes variables and flexboxes it is truly a powerful language and there is no longer a need for preprocessors such as #SCSS / #Sass / #less. React Native contains a very limited interpretation of #CSS3 which I found very frustrating after using #CSS3 for some years already and knowing its powerful features. The other very nice feature of Framework7 is that you can even build for the browser if you want your app to be available for desktop web browsers. The latest release also includes the ability to build for #Electron so you can have MacOS, Windows and Linux desktop apps. This is not possible with React Native yet.

    Framework7 runs on top of Apache Cordova. Cordova and webviews have been slated as being slow in the past. Having a game developer background I found the tweeks to make it run as smooth as silk. One of those tweeks is to use WKWebView. Another important one was using srcset on images.

    I use #Template7 for the for the templating system which is a no-nonsense mobile-centric #HandleBars style extensible templating system. It's easy to write custom helpers for, is fast and has a small footprint. I'm not forced into a new paradigm or learning some new syntax. It operates with standard JavaScript, HTML5 and CSS 3. It's written by the developer of Framework7 and so dovetails with it as expected.

    I configured TypeScript to work with the latest version of Framework7. I consider TypeScript to be one of the best creations to come out of Microsoft in some time. They must have an amazing team working on it. It's very powerful and flexible. It helps you catch a lot of bugs and also provides code completion in supporting IDEs. So for my IDE I use Visual Studio Code which is a blazingly fast and silky smooth editor that integrates seamlessly with TypeScript for the ultimate type checking setup (both products are produced by Microsoft).

    I use Webpack and Babel to compile the JavaScript. TypeScript can compile to JavaScript directly but Babel offers a few more options and polyfills so you can use the latest (and even prerelease) JavaScript features today and compile to be backwards compatible with virtually any browser. My favorite recent addition is "optional chaining" which greatly simplifies and increases readability of a number of sections of my code dealing with getting and setting data in nested objects.

    I use some Ruby scripts to process images with ImageMagick and pngquant to optimise for size and even auto insert responsive image code into the HTML5. Ruby is the ultimate cross platform scripting language. Even as your scripts become large, Ruby allows you to refactor your code easily and make it Object Oriented if necessary. I find it the quickest and easiest way to maintain certain aspects of my build process.

    For the user interface design and prototyping I use Figma. Figma has an almost identical user interface to #Sketch but has the added advantage of being cross platform (MacOS and Windows). Its real-time collaboration features are outstanding and I use them a often as I work mostly on remote projects. Clients can collaborate in real-time and see changes I make as I make them. The clickable prototyping features in Figma are also very well designed and mean I can send clickable prototypes to clients to try user interface updates as they are made and get immediate feedback. I'm currently also evaluating the latest version of #AdobeXD as an alternative to Figma as it has the very cool auto-animate feature. It doesn't have real-time collaboration yet, but I heard it is proposed for 2019.

    For the UI icons I use Font Awesome Pro. They have the largest selection and best looking icons you can find on the internet with several variations in styles so you can find most of the icons you want for standard projects.

    For the backend I was using the #GraphCool Framework. As I later found out, #GraphQL still has some way to go in order to provide the full power of a mature graph query language so later in my project I ripped out #GraphCool and replaced it with CouchDB and Pouchdb. Primarily so I could provide good offline app support. CouchDB with Pouchdb is very flexible and efficient combination and overcomes some of the restrictions I found in #GraphQL and hence #GraphCool also. The most impressive and important feature of CouchDB is its replication. You can configure it in various ways for backups, fault tolerance, caching or conditional merging of databases. CouchDB and Pouchdb even supports storing, retrieving and serving binary or image data or other mime types. This removes a level of complexity usually present in database implementations where binary or image data is usually referenced through an #HTML5 link. With CouchDB and Pouchdb apps can operate offline and sync later, very efficiently, when the network connection is good.

    I use PhoneGap when testing the app. It auto-reloads your app when its code is changed and you can also install it on Android phones to preview your app instantly. iOS is a bit more tricky cause of Apple's policies so it's not available on the App Store, but you can build it and install it yourself to your device.

    So that's my latest mobile stack. What tools do you use? Have you tried these ones?

    See more
    Gabriel Pa

    We implemented our first large scale EPR application from naologic.com using CouchDB .

    Very fast, replication works great, doesn't consume much RAM, queries are blazing fast but we found a problem: the queries were very hard to write, it took a long time to figure out the API, we had to go and write our own @nodejs library to make it work properly.

    It lost most of its support. Since then, we migrated to Couchbase and the learning curve was steep but all worth it. Memcached indexing out of the box, full text search works great.

    See more
    CockroachDB logo

    CockroachDB

    114
    179
    0
    A cloud-native SQL database for building global, scalable cloud services that survive disasters.
    114
    179
    + 1
    0
    PROS OF COCKROACHDB
      Be the first to leave a pro
      CONS OF COCKROACHDB
        Be the first to leave a con

        related CockroachDB posts

        Couchbase logo

        Couchbase

        338
        440
        101
        Document-Oriented NoSQL Database
        338
        440
        + 1
        101
        PROS OF COUCHBASE
        • 18
          High performance
        • 17
          Flexible data model, easy scalability, extremely fast
        • 8
          Mobile app support
        • 6
          You can query it with Ansi-92 SQL
        • 5
          All nodes can be read/write
        • 4
          Open source, community and enterprise editions
        • 4
          Local cache capability
        • 4
          Equal nodes in cluster, allowing fast, flexible changes
        • 4
          Both a key-value store and document (JSON) db
        • 3
          Automatic configuration of sharding
        • 3
          SDKs in popular programming languages
        • 3
          Elasticsearch connector
        • 3
          Easy setup
        • 3
          Web based management, query and monitoring panel
        • 3
          Linearly scalable, useful to large number of tps
        • 3
          Easy cluster administration
        • 3
          Cross data center replication
        • 2
          NoSQL
        • 2
          DBaaS available
        • 2
          Map reduce views
        • 1
          FTS + SQL together
        CONS OF COUCHBASE
        • 3
          Terrible query language

        related Couchbase posts

        Gabriel Pa

        We implemented our first large scale EPR application from naologic.com using CouchDB .

        Very fast, replication works great, doesn't consume much RAM, queries are blazing fast but we found a problem: the queries were very hard to write, it took a long time to figure out the API, we had to go and write our own @nodejs library to make it work properly.

        It lost most of its support. Since then, we migrated to Couchbase and the learning curve was steep but all worth it. Memcached indexing out of the box, full text search works great.

        See more
        Gabriel Pa

        If you want to use Pouchdb might as well use RxDB which is an observables wrapper for Pouch but much more comfortable to use. Realm is awesome but Pouchdb and RxDB give you more control. You can use Couchbase (recommended) or CouchDB to enable 2-way sync

        See more
        Firebase logo

        Firebase

        22.7K
        18.7K
        1.9K
        The Realtime App Platform
        22.7K
        18.7K
        + 1
        1.9K
        PROS OF FIREBASE
        • 357
          Realtime backend made easy
        • 261
          Fast and responsive
        • 233
          Easy setup
        • 206
          Real-time
        • 184
          JSON
        • 126
          Free
        • 120
          Backed by google
        • 80
          Angular adaptor
        • 62
          Reliable
        • 36
          Great customer support
        • 25
          Great documentation
        • 22
          Real-time synchronization
        • 19
          Mobile friendly
        • 17
          Rapid prototyping
        • 12
          Great security
        • 10
          Automatic scaling
        • 9
          Freakingly awesome
        • 8
          Super fast development
        • 8
          Chat
        • 8
          Angularfire is an amazing addition!
        • 6
          Awesome next-gen backend
        • 6
          Ios adaptor
        • 5
          Firebase hosting
        • 5
          Built in user auth/oauth
        • 4
          Very easy to use
        • 3
          Great
        • 3
          Speed of light
        • 3
          Brilliant for startups
        • 3
          It's made development super fast
        • 2
          Low battery consumption
        • 2
          The concurrent updates create a great experience
        • 2
          I can quickly create static web apps with no backend
        • 2
          Great all-round functionality
        • 1
          Easy Reactjs integration
        • 1
          Good Free Limits
        • 1
          .net
        • 1
          Faster workflow
        • 1
          Serverless
        • 1
          JS Offline and Sync suport
        • 1
          Easy to use
        • 1
          Large
        • 1
          Push notification
        CONS OF FIREBASE
        • 26
          Can become expensive
        • 14
          No open source, you depend on external company
        • 14
          Scalability is not infinite
        • 9
          Not Flexible Enough
        • 5
          Cant filter queries
        • 3
          Very unstable server
        • 2
          Too many errors
        • 2
          No Relational Data

        related Firebase posts

        Tassanai Singprom

        This is my stack in Application & Data

        JavaScript PHP HTML5 jQuery Redis Amazon EC2 Ubuntu Sass Vue.js Firebase Laravel Lumen Amazon RDS GraphQL MariaDB

        My Utilities Tools

        Google Analytics Postman Elasticsearch

        My Devops Tools

        Git GitHub GitLab npm Visual Studio Code Kibana Sentry BrowserStack

        My Business Tools

        Slack

        See more

        We are starting to work on a web-based platform aiming to connect artists (clients) and professional freelancers (service providers). In-app, timeline-based, real-time communication between users (& storing it), file transfers, and push notifications are essential core features. We are considering using Node.js, ExpressJS, React, MongoDB stack with Socket.IO & Apollo, or maybe using Real-Time Database and functionalities of Firebase.

        See more
        Redis logo

        Redis

        35.3K
        25.6K
        3.9K
        An in-memory database that persists on disk
        35.3K
        25.6K
        + 1
        3.9K
        PROS OF REDIS
        • 875
          Performance
        • 535
          Super fast
        • 510
          Ease of use
        • 442
          In-memory cache
        • 321
          Advanced key-value cache
        • 189
          Open source
        • 179
          Easy to deploy
        • 163
          Stable
        • 152
          Free
        • 120
          Fast
        • 39
          High-Performance
        • 38
          High Availability
        • 34
          Data Structures
        • 32
          Very Scalable
        • 23
          Replication
        • 20
          Great community
        • 19
          Pub/Sub
        • 17
          "NoSQL" key-value data store
        • 14
          Hashes
        • 12
          Sets
        • 10
          Sorted Sets
        • 9
          Lists
        • 8
          BSD licensed
        • 8
          NoSQL
        • 7
          Async replication
        • 7
          Integrates super easy with Sidekiq for Rails background
        • 7
          Bitmaps
        • 6
          Open Source
        • 6
          Keys with a limited time-to-live
        • 5
          Strings
        • 5
          Lua scripting
        • 4
          Awesomeness for Free!
        • 4
          Hyperloglogs
        • 3
          outstanding performance
        • 3
          Runs server side LUA
        • 3
          Networked
        • 3
          LRU eviction of keys
        • 3
          Written in ANSI C
        • 3
          Feature Rich
        • 3
          Transactions
        • 2
          Data structure server
        • 2
          Performance & ease of use
        • 1
          Existing Laravel Integration
        • 1
          Automatic failover
        • 1
          Easy to use
        • 1
          Object [key/value] size each 500 MB
        • 1
          Simple
        • 1
          Channels concept
        • 1
          Scalable
        • 1
          Temporarily kept on disk
        • 1
          Dont save data if no subscribers are found
        • 0
          Jk
        CONS OF REDIS
        • 11
          Cannot query objects directly
        • 1
          No WAL
        • 1
          No secondary indexes for non-numeric data types

        related Redis posts

        Robert Zuber

        We use MongoDB as our primary #datastore. Mongo's approach to replica sets enables some fantastic patterns for operations like maintenance, backups, and #ETL.

        As we pull #microservices from our #monolith, we are taking the opportunity to build them with their own datastores using PostgreSQL. We also use Redis to cache data we’d never store permanently, and to rate-limit our requests to partners’ APIs (like GitHub).

        When we’re dealing with large blobs of immutable data (logs, artifacts, and test results), we store them in Amazon S3. We handle any side-effects of S3’s eventual consistency model within our own code. This ensures that we deal with user requests correctly while writes are in process.

        See more

        I'm working as one of the engineering leads in RunaHR. As our platform is a Saas, we thought It'd be good to have an API (We chose Ruby and Rails for this) and a SPA (built with React and Redux ) connected. We started the SPA with Create React App since It's pretty easy to start.

        We use Jest as the testing framework and react-testing-library to test React components. In Rails we make tests using RSpec.

        Our main database is PostgreSQL, but we also use MongoDB to store some type of data. We started to use Redis  for cache and other time sensitive operations.

        We have a couple of extra projects: One is an Employee app built with React Native and the other is an internal back office dashboard built with Next.js for the client and Python in the backend side.

        Since we have different frontend apps we have found useful to have Bit to document visual components and utils in JavaScript.

        See more
        Cassandra logo

        Cassandra

        3K
        2.9K
        463
        A partitioned row store. Rows are organized into tables with a required primary key.
        3K
        2.9K
        + 1
        463
        PROS OF CASSANDRA
        • 107
          Distributed
        • 90
          High performance
        • 77
          High availability
        • 71
          Easy scalability
        • 50
          Replication
        • 25
          Reliable
        • 24
          Multi datacenter deployments
        • 6
          Schema optional
        • 6
          OLTP
        • 5
          Open source
        • 2
          Workload separation (via MDC)
        CONS OF CASSANDRA
        • 1
          Reliability of replication
        • 1
          Updates

        related Cassandra posts

        Thierry Schellenbach
        Shared insights
        on
        RedisRedisCassandraCassandraRocksDBRocksDB
        at

        1.0 of Stream leveraged Cassandra for storing the feed. Cassandra is a common choice for building feeds. Instagram, for instance started, out with Redis but eventually switched to Cassandra to handle their rapid usage growth. Cassandra can handle write heavy workloads very efficiently.

        Cassandra is a great tool that allows you to scale write capacity simply by adding more nodes, though it is also very complex. This complexity made it hard to diagnose performance fluctuations. Even though we had years of experience with running Cassandra, it still felt like a bit of a black box. When building Stream 2.0 we decided to go for a different approach and build Keevo. Keevo is our in-house key-value store built upon RocksDB, gRPC and Raft.

        RocksDB is a highly performant embeddable database library developed and maintained by Facebook’s data engineering team. RocksDB started as a fork of Google’s LevelDB that introduced several performance improvements for SSD. Nowadays RocksDB is a project on its own and is under active development. It is written in C++ and it’s fast. Have a look at how this benchmark handles 7 million QPS. In terms of technology it’s much more simple than Cassandra.

        This translates into reduced maintenance overhead, improved performance and, most importantly, more consistent performance. It’s interesting to note that LinkedIn also uses RocksDB for their feed.

        #InMemoryDatabases #DataStores #Databases

        See more
        Umair Iftikhar
        Technical Architect at Vappar · | 3 upvotes · 14.4K views

        Developing a solution that collects Telemetry Data from different devices, nearly 1000 devices minimum and maximum 12000. Each device is sending 2 packets in 1 second. This is time-series data, and this data definition and different reports are saved on PostgreSQL. Like Building information, maintenance records, etc. I want to know about the best solution. This data is required for Math and ML to run different algorithms. Also, data is raw without definitions and information stored in PostgreSQL. Initially, I went with TimescaleDB due to PostgreSQL support, but to increase in sites, I started facing many issues with timescale DB in terms of flexibility of storing data.

        My major requirement is also the replication of the database for reporting and different purposes. You may also suggest other options other than Druid and Cassandra. But an open source solution is appreciated.

        See more
        PostgreSQL logo

        PostgreSQL

        51.8K
        40K
        3.5K
        A powerful, open source object-relational database system
        51.8K
        40K
        + 1
        3.5K
        PROS OF POSTGRESQL
        • 754
          Relational database
        • 506
          High availability
        • 437
          Enterprise class database
        • 379
          Sql
        • 299
          Sql + nosql
        • 171
          Great community
        • 145
          Easy to setup
        • 129
          Heroku
        • 128
          Secure by default
        • 111
          Postgis
        • 48
          Supports Key-Value
        • 46
          Great JSON support
        • 32
          Cross platform
        • 29
          Extensible
        • 25
          Replication
        • 24
          Triggers
        • 22
          Rollback
        • 21
          Multiversion concurrency control
        • 20
          Open source
        • 17
          Heroku Add-on
        • 14
          Stable, Simple and Good Performance
        • 13
          Powerful
        • 12
          Lets be serious, what other SQL DB would you go for?
        • 9
          Good documentation
        • 7
          Scalable
        • 7
          Intelligent optimizer
        • 6
          Transactional DDL
        • 6
          Modern
        • 6
          Reliable
        • 5
          One stop solution for all things sql no matter the os
        • 5
          Free
        • 4
          Relational database with MVCC
        • 3
          Full-Text Search
        • 3
          Developer friendly
        • 3
          Faster Development
        • 2
          Excellent source code
        • 2
          Great DB for Transactional system or Application
        • 1
          Free version
        • 1
          Text
        • 1
          Open-source
        • 1
          search
        • 1
          Full-text
        CONS OF POSTGRESQL
        • 9
          Table/index bloatings

        related PostgreSQL posts

        Jeyabalaji Subramanian

        Recently we were looking at a few robust and cost-effective ways of replicating the data that resides in our production MongoDB to a PostgreSQL database for data warehousing and business intelligence.

        We set ourselves the following criteria for the optimal tool that would do this job: - The data replication must be near real-time, yet it should NOT impact the production database - The data replication must be horizontally scalable (based on the load), asynchronous & crash-resilient

        Based on the above criteria, we selected the following tools to perform the end to end data replication:

        We chose MongoDB Stitch for picking up the changes in the source database. It is the serverless platform from MongoDB. One of the services offered by MongoDB Stitch is Stitch Triggers. Using stitch triggers, you can execute a serverless function (in Node.js) in real time in response to changes in the database. When there are a lot of database changes, Stitch automatically "feeds forward" these changes through an asynchronous queue.

        We chose Amazon SQS as the pipe / message backbone for communicating the changes from MongoDB to our own replication service. Interestingly enough, MongoDB stitch offers integration with AWS services.

        In the Node.js function, we wrote minimal functionality to communicate the database changes (insert / update / delete / replace) to Amazon SQS.

        Next we wrote a minimal micro-service in Python to listen to the message events on SQS, pickup the data payload & mirror the DB changes on to the target Data warehouse. We implemented source data to target data translation by modelling target table structures through SQLAlchemy . We deployed this micro-service as AWS Lambda with Zappa. With Zappa, deploying your services as event-driven & horizontally scalable Lambda service is dumb-easy.

        In the end, we got to implement a highly scalable near realtime Change Data Replication service that "works" and deployed to production in a matter of few days!

        See more
        Tim Abbott

        We've been using PostgreSQL since the very early days of Zulip, but we actually didn't use it from the beginning. Zulip started out as a MySQL project back in 2012, because we'd heard it was a good choice for a startup with a wide community. However, we found that even though we were using the Django ORM for most of our database access, we spent a lot of time fighting with MySQL. Issues ranged from bad collation defaults, to bad query plans which required a lot of manual query tweaks.

        We ended up getting so frustrated that we tried out PostgresQL, and the results were fantastic. We didn't have to do any real customization (just some tuning settings for how big a server we had), and all of our most important queries were faster out of the box. As a result, we were able to delete a bunch of custom queries escaping the ORM that we'd written to make the MySQL query planner happy (because postgres just did the right thing automatically).

        And then after that, we've just gotten a ton of value out of postgres. We use its excellent built-in full-text search, which has helped us avoid needing to bring in a tool like Elasticsearch, and we've really enjoyed features like its partial indexes, which saved us a lot of work adding unnecessary extra tables to get good performance for things like our "unread messages" and "starred messages" indexes.

        I can't recommend it highly enough.

        See more