Alternatives to Cilium

At my company, we are trying to move away from a monolith into microservices led architecture. We are now stuck with a problem to establish a communication mechanism between microservices. Since, we are planning to use service meshes and something like Dapr/Istio, we are not sure on how to split services between the two. Service meshes offer Traffic Routing or Splitting whereas, Dapr can offer state management and service-service invocation. At the same time both of them provide mLTS, Metrics, Resiliency and tracing. How to choose who should offer what?

As for the new support of service mesh pattern by Kong, I wonder how does it compare to Istio?

At uSwitch we wanted a way to load balance between our multiple Kubernetes clusters in AWS to give us added redundancy. We already had ingresses defined for all our applications so we wanted to build on top of that, instead of creating a new system that would require our various teams to change code/config etc.

Envoy seemed to tick a lot of boxes:

• Loadbalancing capabilities right out of the box: health checks, circuit breaking, retries etc.
• Tracing and prometheus metrics support
• Lightweight
• Good community support

This was all good but what really sold us was the api that supported dynamic configuration. This would allow us to dynamically configure envoy to route to ingresses and clusters as they were created or destroyed.

To do this we built a tool called Yggdrasil using their Go sdk. Yggdrasil effectively just creates envoy configuration from Kubernetes ingress objects, so you point Yggdrasil at your kube clusters, it generates config from the ingresses and then envoy can loadbalance between your clusters for you. This is all done dynamically so as soon as new ingress is created the envoy nodes get updated with the new config. Importantly this all worked with what we already had, no need to create new config for every application, we just put this on top of it.

We are looking to configure a load balancer with some admin UI. We are currently struggling to decide between NGINX, Traefik, HAProxy, and Envoy. We will use a load balancer in a containerized environment and the load balancer should flexible and easy to reload without changes in case containers are scaled up.

Our whole DevOps stack consists of the following tools:

• GitHub (incl. GitHub Pages/Markdown for Documentation, GettingStarted and HowTo's) for collaborative review and code management tool
• Respectively Git as revision control system
• SourceTree as Git GUI
• Visual Studio Code as IDE
• CircleCI for continuous integration (automatize development process)
• Prettier / TSLint / ESLint as code linter
• SonarQube as quality gate
• Docker as container management (incl. Docker Compose for multi-container application management)
• VirtualBox for operating system simulation tests
• Kubernetes as cluster management for docker containers
• Heroku for deploying in test environments
• nginx as web server (preferably used as facade server in production environment)
• SSLMate (using OpenSSL) for certificate management
• Amazon EC2 (incl. Amazon S3) for deploying in stage (production-like) and production environments
• PostgreSQL as preferred database system
• Redis as preferred in-memory database/store (great for caching)

The main reason we have chosen Kubernetes over Docker Swarm is related to the following artifacts:

• Key features: Easy and flexible installation, Clear dashboard, Great scaling operations, Monitoring is an integral part, Great load balancing concepts, Monitors the condition and ensures compensation in the event of failure.
• Applications: An application can be deployed using a combination of pods, deployments, and services (or micro-services).
• Functionality: Kubernetes as a complex installation and setup process, but it not as limited as Docker Swarm.
• Monitoring: It supports multiple versions of logging and monitoring when the services are deployed within the cluster (Elasticsearch/Kibana (ELK), Heapster/Grafana, Sysdig cloud integration).
• Scalability: All-in-one framework for distributed systems.
• Other Benefits: Kubernetes is backed by the Cloud Native Computing Foundation (CNCF), huge community among container orchestration tools, it is an open source and modular tool that works with any OS.