Alternatives to MarkLogic logo

Alternatives to MarkLogic

MongoDB, Neo4j, Oracle, Cassandra, and HBase are the most popular alternatives and competitors to MarkLogic.
42
70
+ 1
26

What is MarkLogic and what are its top alternatives?

MarkLogic is a NoSQL database that combines the benefits of a document database, a relational database, and a search engine all in one platform. Its key features include ACID transactions, horizontal scaling, flexible data model, and powerful search capabilities. However, some limitations include high cost, steep learning curve, and limited community support.

  1. MongoDB: MongoDB is a popular NoSQL database known for its scalability, flexibility, and high performance. Key features include document-based data model, horizontal scaling, and strong community support. Pros include ease of use and integration capabilities, while cons include lack of ACID transactions in earlier versions.
  2. Couchbase: Couchbase is an open-source NoSQL database with features like JSON support, distributed architecture, and strong query capabilities. Pros include high performance and scalability, while cons include limited secondary indexes and analytics capabilities.
  3. Cassandra: Apache Cassandra is a distributed NoSQL database known for its linear scalability and fault tolerance. Key features include decentralized architecture, tunable consistency levels, and support for massive amounts of data. Pros include high availability and fault tolerance, while cons include complex configuration and management.
  4. Elasticsearch: Elasticsearch is a distributed search and analytics engine known for its speed, scalability, and flexibility. Key features include full-text search, real-time data analysis, and horizontal scaling. Pros include powerful search capabilities and ease of use, while cons include lack of ACID transactions.
  5. Amazon DynamoDB: DynamoDB is a fully managed NoSQL database service offered by AWS, providing key-value and document data storage. Key features include automatic scaling, low latency, and seamless integration with other AWS services. Pros include high availability and low maintenance overhead, while cons include costs for read and write operations.
  6. Redis: Redis is an open-source, in-memory data structure store known for its speed and flexibility. Key features include data structures like strings, hashes, and lists, as well as support for high availability and clustering. Pros include low latency and high performance, while cons include limited data persistence options.
  7. Oracle NoSQL Database: Oracle NoSQL Database is a distributed, highly available NoSQL database from Oracle. Key features include automatic partitioning, replication, and failover, as well as ACID transactions. Pros include scalability and reliability, while cons include high cost and complex setup.
  8. Google Cloud Bigtable: Google Cloud Bigtable is a highly scalable, fully managed NoSQL database service offered by Google Cloud Platform. Key features include high performance, scalability to petabytes of data, and low latency. Pros include seamless integration with other GCP services and high availability, while cons include costs for storage and operations.
  9. Neo4j: Neo4j is a graph database known for its high performance and scalability in handling complex relationships. Key features include graph-based data model, ACID transactions, and powerful query language. Pros include fast query performance and strong relationship modeling capabilities, while cons include limited support for large-scale data processing.
  10. TiDB: TiDB is a distributed SQL database that combines the horizontal scalability of NoSQL with the ACID compliance of traditional relational databases. Key features include automatic sharding, strong consistency, and compatibility with MySQL protocols. Pros include scalability and ACID compliance, while cons include limited ecosystem and tooling compared to established databases.

Top Alternatives to MarkLogic

  • MongoDB
    MongoDB

    MongoDB stores data in JSON-like documents that can vary in structure, offering a dynamic, flexible schema. MongoDB was also designed for high availability and scalability, with built-in replication and auto-sharding. ...

  • Neo4j
    Neo4j

    Neo4j stores data in nodes connected by directed, typed relationships with properties on both, also known as a Property Graph. It is a high performance graph store with all the features expected of a mature and robust database, like a friendly query language and ACID transactions. ...

  • Oracle
    Oracle

    Oracle Database is an RDBMS. An RDBMS that implements object-oriented features such as user-defined types, inheritance, and polymorphism is called an object-relational database management system (ORDBMS). Oracle Database has extended the relational model to an object-relational model, making it possible to store complex business models in a relational database. ...

  • Cassandra
    Cassandra

    Partitioning means that Cassandra can distribute your data across multiple machines in an application-transparent matter. Cassandra will automatically repartition as machines are added and removed from the cluster. Row store means that like relational databases, Cassandra organizes data by rows and columns. The Cassandra Query Language (CQL) is a close relative of SQL. ...

  • HBase
    HBase

    Apache HBase is an open-source, distributed, versioned, column-oriented store modeled after Google' Bigtable: A Distributed Storage System for Structured Data by Chang et al. Just as Bigtable leverages the distributed data storage provided by the Google File System, HBase provides Bigtable-like capabilities on top of Apache Hadoop. ...

  • Hadoop
    Hadoop

    The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. ...

  • Couchbase
    Couchbase

    Developed as an alternative to traditionally inflexible SQL databases, the Couchbase NoSQL database is built on an open source foundation and architected to help developers solve real-world problems and meet high scalability demands. ...

  • Elasticsearch
    Elasticsearch

    Elasticsearch is a distributed, RESTful search and analytics engine capable of storing data and searching it in near real time. Elasticsearch, Kibana, Beats and Logstash are the Elastic Stack (sometimes called the ELK Stack). ...

MarkLogic alternatives & related posts

MongoDB logo

MongoDB

93K
80.3K
4.1K
The database for giant ideas
93K
80.3K
+ 1
4.1K
PROS OF MONGODB
  • 827
    Document-oriented storage
  • 593
    No sql
  • 553
    Ease of use
  • 464
    Fast
  • 410
    High performance
  • 255
    Free
  • 218
    Open source
  • 180
    Flexible
  • 145
    Replication & high availability
  • 112
    Easy to maintain
  • 42
    Querying
  • 39
    Easy scalability
  • 38
    Auto-sharding
  • 37
    High availability
  • 31
    Map/reduce
  • 27
    Document database
  • 25
    Easy setup
  • 25
    Full index support
  • 16
    Reliable
  • 15
    Fast in-place updates
  • 14
    Agile programming, flexible, fast
  • 12
    No database migrations
  • 8
    Easy integration with Node.Js
  • 8
    Enterprise
  • 6
    Enterprise Support
  • 5
    Great NoSQL DB
  • 4
    Support for many languages through different drivers
  • 3
    Schemaless
  • 3
    Aggregation Framework
  • 3
    Drivers support is good
  • 2
    Fast
  • 2
    Managed service
  • 2
    Easy to Scale
  • 2
    Awesome
  • 2
    Consistent
  • 1
    Good GUI
  • 1
    Acid Compliant
CONS OF MONGODB
  • 6
    Very slowly for connected models that require joins
  • 3
    Not acid compliant
  • 2
    Proprietary query language

related MongoDB posts

Shared insights
on
Node.jsNode.jsGraphQLGraphQLMongoDBMongoDB

I just finished the very first version of my new hobby project: #MovieGeeks. It is a minimalist online movie catalog for you to save the movies you want to see and for rating the movies you already saw. This is just the beginning as I am planning to add more features on the lines of sharing and discovery

For the #BackEnd I decided to use Node.js , GraphQL and MongoDB:

  1. Node.js has a huge community so it will always be a safe choice in terms of libraries and finding solutions to problems you may have

  2. GraphQL because I needed to improve my skills with it and because I was never comfortable with the usual REST approach. I believe GraphQL is a better option as it feels more natural to write apis, it improves the development velocity, by definition it fixes the over-fetching and under-fetching problem that is so common on REST apis, and on top of that, the community is getting bigger and bigger.

  3. MongoDB was my choice for the database as I already have a lot of experience working on it and because, despite of some bad reputation it has acquired in the last months, I still believe it is a powerful database for at least a very long list of use cases such as the one I needed for my website

See more
Vaibhav Taunk
Team Lead at Technovert · | 31 upvotes · 4.1M views

I am starting to become a full-stack developer, by choosing and learning .NET Core for API Development, Angular CLI / React for UI Development, MongoDB for database, as it a NoSQL DB and Flutter / React Native for Mobile App Development. Using Postman, Markdown and Visual Studio Code for development.

See more
Neo4j logo

Neo4j

1.2K
1.4K
351
The world’s leading Graph Database
1.2K
1.4K
+ 1
351
PROS OF NEO4J
  • 69
    Cypher – graph query language
  • 61
    Great graphdb
  • 33
    Open source
  • 31
    Rest api
  • 27
    High-Performance Native API
  • 23
    ACID
  • 21
    Easy setup
  • 17
    Great support
  • 11
    Clustering
  • 9
    Hot Backups
  • 8
    Great Web Admin UI
  • 7
    Powerful, flexible data model
  • 7
    Mature
  • 6
    Embeddable
  • 5
    Easy to Use and Model
  • 4
    Highly-available
  • 4
    Best Graphdb
  • 2
    It's awesome, I wanted to try it
  • 2
    Great onboarding process
  • 2
    Great query language and built in data browser
  • 2
    Used by Crunchbase
CONS OF NEO4J
  • 9
    Comparably slow
  • 4
    Can't store a vertex as JSON
  • 1
    Doesn't have a managed cloud service at low cost

related Neo4j posts

Shared insights
on
Neo4jNeo4jKafkaKafkaMySQLMySQL

Hello Stackshare. I'm currently doing some research on real-time reporting and analytics architectures. We have a use case where 1million+ records of users, 4million+ activities, and messages that we want to report against. The start was to present it directly from MySQL, which didn't go well and puts a heavy load on the database. Anybody can suggest something where we feed the data and can report in realtime? Read some articles about ElasticSearch and Kafka https://medium.com/@D11Engg/building-scalable-real-time-analytics-alerting-and-anomaly-detection-architecture-at-dream11-e20edec91d33 EDIT: also considering Neo4j

See more
Stephen Gheysens
Lead Solutions Engineer at Inscribe · | 7 upvotes · 465.4K views

Google Maps lets "property owners and their authorized representatives" upload indoor maps, but this appears to lack navigation ("wayfinding").

MappedIn is a platform and has SDKs for building indoor mapping experiences (https://www.mappedin.com/) and ESRI ArcGIS also offers some indoor mapping tools (https://www.esri.com/en-us/arcgis/indoor-gis/overview). Finally, there used to be a company called LocusLabs that is now a part of Atrius and they were often integrated into airlines' apps to provide airport maps with wayfinding (https://atrius.com/solutions/personal-experiences/personal-wayfinder/).

I previously worked at Mapbox and while I believe that it's a great platform for building map-based experiences, they don't have any simple solutions for indoor wayfinding. If I were doing this for fun as a side-project and prioritized saving money over saving time, here is what I would do:

  • Create a graph-based dataset representing the walking paths around your university, where nodes/vertexes represent the intersections of paths, and edges represent paths (literally paths outside, hallways, short path segments that represent entering rooms). You could store this in a hosted graph-based database like Neo4j, Amazon Neptune , or Azure Cosmos DB (with its Gremlin API) and use built-in "shortest path" queries, or deploy a PostgreSQL service with pgRouting.

  • Add two properties to each edge: one property for the distance between its nodes (libraries like @turf/helpers will have a distance function if you have the latitude & longitude of each node), and another property estimating the walking time (based on the distance). Once you have these values saved in a graph-based format, you should be able to easily query and find the data representation of paths between two points.

  • At this point, you'd have the routing problem solved and it would come down to building a UI. Mapbox arguably leads the industry in developer tools for custom map experiences. You could convert your nodes/edges to GeoJSON, then either upload to Mapbox and create a Tileset to visualize the paths, or add the GeoJSON to the map on the fly.

*You might be able to use open source routing tools like OSRM (https://github.com/Project-OSRM/osrm-backend/issues/6257) or Graphhopper (instead of a custom graph database implementation), but it would likely be more involved to maintain these services.

See more
Oracle logo

Oracle

2.3K
1.7K
113
An RDBMS that implements object-oriented features such as user-defined types, inheritance, and polymorphism
2.3K
1.7K
+ 1
113
PROS OF ORACLE
  • 44
    Reliable
  • 33
    Enterprise
  • 15
    High Availability
  • 5
    Hard to maintain
  • 5
    Expensive
  • 4
    Maintainable
  • 4
    Hard to use
  • 3
    High complexity
CONS OF ORACLE
  • 14
    Expensive

related Oracle posts

Saurav Pandit
Application Devloper at Bny Mellon · | 9 upvotes · 309.5K views

I have just started learning Python 3 week back. I want to create REST api using python. The api will be use to save form data in Oracle database. The front end is using AngularJS 8 with Angular Material. In python there are so many framework for developing REST ** I am looking for some suggestions which REST framework to choose? ** Here are some feature I am looking for * Easy integration and unit testing like in Angular we just run command. * Code packageing, like in Java maven project we can build and package. I am looking for something which I can push in artifactory and deploy whole code as package. *Support for swagger/ OpenAPI * Support for JSON Web Token * Support for testcase coverage report Framework can have feature included or can be available by extension.

See more
Dishi Jain
Shared insights
on
OracleOracleKubernetesKubernetes

So we are re-engineering our application database to make it cloud-native and deploy on the Kubernetes platform. Currently, our data lies on the Oracle 19c database and it is normalized extensively. We store pdfs, txt files and allow a user to edit, delete, view, create new transactions. Now I want to pick a DB, which makes the re-engineering, not a big deal but allows us to store data in a distributed manner on Kubernetes. Please assist me.

See more
Cassandra logo

Cassandra

3.6K
3.5K
507
A partitioned row store. Rows are organized into tables with a required primary key.
3.6K
3.5K
+ 1
507
PROS OF CASSANDRA
  • 119
    Distributed
  • 98
    High performance
  • 81
    High availability
  • 74
    Easy scalability
  • 53
    Replication
  • 26
    Reliable
  • 26
    Multi datacenter deployments
  • 10
    Schema optional
  • 9
    OLTP
  • 8
    Open source
  • 2
    Workload separation (via MDC)
  • 1
    Fast
CONS OF CASSANDRA
  • 3
    Reliability of replication
  • 1
    Size
  • 1
    Updates

related Cassandra posts

Thierry Schellenbach
Shared insights
on
GolangGolangPythonPythonCassandraCassandra
at

After years of optimizing our existing feed technology, we decided to make a larger leap with 2.0 of Stream. While the first iteration of Stream was powered by Python and Cassandra, for Stream 2.0 of our infrastructure we switched to Go.

The main reason why we switched from Python to Go is performance. Certain features of Stream such as aggregation, ranking and serialization were very difficult to speed up using Python.

We’ve been using Go since March 2017 and it’s been a great experience so far. Go has greatly increased the productivity of our development team. Not only has it improved the speed at which we develop, it’s also 30x faster for many components of Stream. Initially we struggled a bit with package management for Go. However, using Dep together with the VG package contributed to creating a great workflow.

Go as a language is heavily focused on performance. The built-in PPROF tool is amazing for finding performance issues. Uber’s Go-Torch library is great for visualizing data from PPROF and will be bundled in PPROF in Go 1.10.

The performance of Go greatly influenced our architecture in a positive way. With Python we often found ourselves delegating logic to the database layer purely for performance reasons. The high performance of Go gave us more flexibility in terms of architecture. This led to a huge simplification of our infrastructure and a dramatic improvement of latency. For instance, we saw a 10 to 1 reduction in web-server count thanks to the lower memory and CPU usage for the same number of requests.

#DataStores #Databases

See more
Thierry Schellenbach
Shared insights
on
RedisRedisCassandraCassandraRocksDBRocksDB
at

1.0 of Stream leveraged Cassandra for storing the feed. Cassandra is a common choice for building feeds. Instagram, for instance started, out with Redis but eventually switched to Cassandra to handle their rapid usage growth. Cassandra can handle write heavy workloads very efficiently.

Cassandra is a great tool that allows you to scale write capacity simply by adding more nodes, though it is also very complex. This complexity made it hard to diagnose performance fluctuations. Even though we had years of experience with running Cassandra, it still felt like a bit of a black box. When building Stream 2.0 we decided to go for a different approach and build Keevo. Keevo is our in-house key-value store built upon RocksDB, gRPC and Raft.

RocksDB is a highly performant embeddable database library developed and maintained by Facebook’s data engineering team. RocksDB started as a fork of Google’s LevelDB that introduced several performance improvements for SSD. Nowadays RocksDB is a project on its own and is under active development. It is written in C++ and it’s fast. Have a look at how this benchmark handles 7 million QPS. In terms of technology it’s much more simple than Cassandra.

This translates into reduced maintenance overhead, improved performance and, most importantly, more consistent performance. It’s interesting to note that LinkedIn also uses RocksDB for their feed.

#InMemoryDatabases #DataStores #Databases

See more
HBase logo

HBase

462
494
15
The Hadoop database, a distributed, scalable, big data store
462
494
+ 1
15
PROS OF HBASE
  • 9
    Performance
  • 5
    OLTP
  • 1
    Fast Point Queries
CONS OF HBASE
    Be the first to leave a con

    related HBase posts

    I am researching different querying solutions to handle ~1 trillion records of data (in the realm of a petabyte). The data is mostly textual. I have identified a few options: Milvus, HBase, RocksDB, and Elasticsearch. I was wondering if there is a good way to compare the performance of these options (or if anyone has already done something like this). I want to be able to compare the speed of ingesting and querying textual data from these tools. Does anyone have information on this or know where I can find some? Thanks in advance!

    See more

    Hi, I'm building a machine learning pipelines to store image bytes and image vectors in the backend.

    So, when users query for the random access image data (key), we return the image bytes and perform machine learning model operations on it.

    I'm currently considering going with Amazon S3 (in the future, maybe add Redis caching layer) as the backend system to store the information (s3 buckets with sharded prefixes).

    As the latency of S3 is 100-200ms (get/put) and it has a high throughput of 3500 puts/sec and 5500 gets/sec for a given bucker/prefix. In the future I need to reduce the latency, I can add Redis cache.

    Also, s3 costs are way fewer than HBase (on Amazon EC2 instances with 3x replication factor)

    I have not personally used HBase before, so can someone help me if I'm making the right choice here? I'm not aware of Hbase latencies and I have learned that the MOB feature on Hbase has to be turned on if we have store image bytes on of the column families as the avg image bytes are 240Kb.

    See more
    Hadoop logo

    Hadoop

    2.5K
    2.3K
    56
    Open-source software for reliable, scalable, distributed computing
    2.5K
    2.3K
    + 1
    56
    PROS OF HADOOP
    • 39
      Great ecosystem
    • 11
      One stack to rule them all
    • 4
      Great load balancer
    • 1
      Amazon aws
    • 1
      Java syntax
    CONS OF HADOOP
      Be the first to leave a con

      related Hadoop posts

      Shared insights
      on
      KafkaKafkaHadoopHadoop
      at

      The early data ingestion pipeline at Pinterest used Kafka as the central message transporter, with the app servers writing messages directly to Kafka, which then uploaded log files to S3.

      For databases, a custom Hadoop streamer pulled database data and wrote it to S3.

      Challenges cited for this infrastructure included high operational overhead, as well as potential data loss occurring when Kafka broker outages led to an overflow of in-memory message buffering.

      See more
      Conor Myhrvold
      Tech Brand Mgr, Office of CTO at Uber · | 7 upvotes · 3M views

      Why we built Marmaray, an open source generic data ingestion and dispersal framework and library for Apache Hadoop :

      Built and designed by our Hadoop Platform team, Marmaray is a plug-in-based framework built on top of the Hadoop ecosystem. Users can add support to ingest data from any source and disperse to any sink leveraging the use of Apache Spark . The name, Marmaray, comes from a tunnel in Turkey connecting Europe and Asia. Similarly, we envisioned Marmaray within Uber as a pipeline connecting data from any source to any sink depending on customer preference:

      https://eng.uber.com/marmaray-hadoop-ingestion-open-source/

      (Direct GitHub repo: https://github.com/uber/marmaray Kafka Kafka Manager )

      See more
      Couchbase logo

      Couchbase

      479
      603
      110
      Document-Oriented NoSQL Database
      479
      603
      + 1
      110
      PROS OF COUCHBASE
      • 18
        High performance
      • 18
        Flexible data model, easy scalability, extremely fast
      • 9
        Mobile app support
      • 7
        You can query it with Ansi-92 SQL
      • 6
        All nodes can be read/write
      • 5
        Equal nodes in cluster, allowing fast, flexible changes
      • 5
        Both a key-value store and document (JSON) db
      • 5
        Open source, community and enterprise editions
      • 4
        Automatic configuration of sharding
      • 4
        Local cache capability
      • 3
        Easy setup
      • 3
        Linearly scalable, useful to large number of tps
      • 3
        Easy cluster administration
      • 3
        Cross data center replication
      • 3
        SDKs in popular programming languages
      • 3
        Elasticsearch connector
      • 3
        Web based management, query and monitoring panel
      • 2
        Map reduce views
      • 2
        DBaaS available
      • 2
        NoSQL
      • 1
        Buckets, Scopes, Collections & Documents
      • 1
        FTS + SQL together
      CONS OF COUCHBASE
      • 3
        Terrible query language

      related Couchbase posts

      Gabriel Pa

      We implemented our first large scale EPR application from naologic.com using CouchDB .

      Very fast, replication works great, doesn't consume much RAM, queries are blazing fast but we found a problem: the queries were very hard to write, it took a long time to figure out the API, we had to go and write our own @nodejs library to make it work properly.

      It lost most of its support. Since then, we migrated to Couchbase and the learning curve was steep but all worth it. Memcached indexing out of the box, full text search works great.

      See more
      Ilias Mentzelos
      Software Engineer at Plum Fintech · | 9 upvotes · 238.3K views
      Shared insights
      on
      MongoDBMongoDBCouchbaseCouchbase

      Hey, we want to build a referral campaign mechanism that will probably contain millions of records within the next few years. We want fast read access based on IDs or some indexes, and isolation is crucial as some listeners will try to update the same document at the same time. What's your suggestion between Couchbase and MongoDB? Thanks!

      See more
      Elasticsearch logo

      Elasticsearch

      34.4K
      26.8K
      1.6K
      Open Source, Distributed, RESTful Search Engine
      34.4K
      26.8K
      + 1
      1.6K
      PROS OF ELASTICSEARCH
      • 328
        Powerful api
      • 315
        Great search engine
      • 231
        Open source
      • 214
        Restful
      • 200
        Near real-time search
      • 98
        Free
      • 85
        Search everything
      • 54
        Easy to get started
      • 45
        Analytics
      • 26
        Distributed
      • 6
        Fast search
      • 5
        More than a search engine
      • 4
        Great docs
      • 4
        Awesome, great tool
      • 3
        Highly Available
      • 3
        Easy to scale
      • 2
        Potato
      • 2
        Document Store
      • 2
        Great customer support
      • 2
        Intuitive API
      • 2
        Nosql DB
      • 2
        Great piece of software
      • 2
        Reliable
      • 2
        Fast
      • 2
        Easy setup
      • 1
        Open
      • 1
        Easy to get hot data
      • 1
        Github
      • 1
        Elaticsearch
      • 1
        Actively developing
      • 1
        Responsive maintainers on GitHub
      • 1
        Ecosystem
      • 1
        Not stable
      • 1
        Scalability
      • 0
        Community
      CONS OF ELASTICSEARCH
      • 7
        Resource hungry
      • 6
        Diffecult to get started
      • 5
        Expensive
      • 4
        Hard to keep stable at large scale

      related Elasticsearch posts

      Tim Abbott

      We've been using PostgreSQL since the very early days of Zulip, but we actually didn't use it from the beginning. Zulip started out as a MySQL project back in 2012, because we'd heard it was a good choice for a startup with a wide community. However, we found that even though we were using the Django ORM for most of our database access, we spent a lot of time fighting with MySQL. Issues ranged from bad collation defaults, to bad query plans which required a lot of manual query tweaks.

      We ended up getting so frustrated that we tried out PostgresQL, and the results were fantastic. We didn't have to do any real customization (just some tuning settings for how big a server we had), and all of our most important queries were faster out of the box. As a result, we were able to delete a bunch of custom queries escaping the ORM that we'd written to make the MySQL query planner happy (because postgres just did the right thing automatically).

      And then after that, we've just gotten a ton of value out of postgres. We use its excellent built-in full-text search, which has helped us avoid needing to bring in a tool like Elasticsearch, and we've really enjoyed features like its partial indexes, which saved us a lot of work adding unnecessary extra tables to get good performance for things like our "unread messages" and "starred messages" indexes.

      I can't recommend it highly enough.

      See more
      Tymoteusz Paul
      Devops guy at X20X Development LTD · | 23 upvotes · 9.3M views

      Often enough I have to explain my way of going about setting up a CI/CD pipeline with multiple deployment platforms. Since I am a bit tired of yapping the same every single time, I've decided to write it up and share with the world this way, and send people to read it instead ;). I will explain it on "live-example" of how the Rome got built, basing that current methodology exists only of readme.md and wishes of good luck (as it usually is ;)).

      It always starts with an app, whatever it may be and reading the readmes available while Vagrant and VirtualBox is installing and updating. Following that is the first hurdle to go over - convert all the instruction/scripts into Ansible playbook(s), and only stopping when doing a clear vagrant up or vagrant reload we will have a fully working environment. As our Vagrant environment is now functional, it's time to break it! This is the moment to look for how things can be done better (too rigid/too lose versioning? Sloppy environment setup?) and replace them with the right way to do stuff, one that won't bite us in the backside. This is the point, and the best opportunity, to upcycle the existing way of doing dev environment to produce a proper, production-grade product.

      I should probably digress here for a moment and explain why. I firmly believe that the way you deploy production is the same way you should deploy develop, shy of few debugging-friendly setting. This way you avoid the discrepancy between how production work vs how development works, which almost always causes major pains in the back of the neck, and with use of proper tools should mean no more work for the developers. That's why we start with Vagrant as developer boxes should be as easy as vagrant up, but the meat of our product lies in Ansible which will do meat of the work and can be applied to almost anything: AWS, bare metal, docker, LXC, in open net, behind vpn - you name it.

      We must also give proper consideration to monitoring and logging hoovering at this point. My generic answer here is to grab Elasticsearch, Kibana, and Logstash. While for different use cases there may be better solutions, this one is well battle-tested, performs reasonably and is very easy to scale both vertically (within some limits) and horizontally. Logstash rules are easy to write and are well supported in maintenance through Ansible, which as I've mentioned earlier, are at the very core of things, and creating triggers/reports and alerts based on Elastic and Kibana is generally a breeze, including some quite complex aggregations.

      If we are happy with the state of the Ansible it's time to move on and put all those roles and playbooks to work. Namely, we need something to manage our CI/CD pipelines. For me, the choice is obvious: TeamCity. It's modern, robust and unlike most of the light-weight alternatives, it's transparent. What I mean by that is that it doesn't tell you how to do things, doesn't limit your ways to deploy, or test, or package for that matter. Instead, it provides a developer-friendly and rich playground for your pipelines. You can do most the same with Jenkins, but it has a quite dated look and feel to it, while also missing some key functionality that must be brought in via plugins (like quality REST API which comes built-in with TeamCity). It also comes with all the common-handy plugins like Slack or Apache Maven integration.

      The exact flow between CI and CD varies too greatly from one application to another to describe, so I will outline a few rules that guide me in it: 1. Make build steps as small as possible. This way when something breaks, we know exactly where, without needing to dig and root around. 2. All security credentials besides development environment must be sources from individual Vault instances. Keys to those containers should exist only on the CI/CD box and accessible by a few people (the less the better). This is pretty self-explanatory, as anything besides dev may contain sensitive data and, at times, be public-facing. Because of that appropriate security must be present. TeamCity shines in this department with excellent secrets-management. 3. Every part of the build chain shall consume and produce artifacts. If it creates nothing, it likely shouldn't be its own build. This way if any issue shows up with any environment or version, all developer has to do it is grab appropriate artifacts to reproduce the issue locally. 4. Deployment builds should be directly tied to specific Git branches/tags. This enables much easier tracking of what caused an issue, including automated identifying and tagging the author (nothing like automated regression testing!).

      Speaking of deployments, I generally try to keep it simple but also with a close eye on the wallet. Because of that, I am more than happy with AWS or another cloud provider, but also constantly peeking at the loads and do we get the value of what we are paying for. Often enough the pattern of use is not constantly erratic, but rather has a firm baseline which could be migrated away from the cloud and into bare metal boxes. That is another part where this approach strongly triumphs over the common Docker and CircleCI setup, where you are very much tied in to use cloud providers and getting out is expensive. Here to embrace bare-metal hosting all you need is a help of some container-based self-hosting software, my personal preference is with Proxmox and LXC. Following that all you must write are ansible scripts to manage hardware of Proxmox, similar way as you do for Amazon EC2 (ansible supports both greatly) and you are good to go. One does not exclude another, quite the opposite, as they can live in great synergy and cut your costs dramatically (the heavier your base load, the bigger the savings) while providing production-grade resiliency.

      See more