What is Common Lisp and what are its top alternatives?
Common Lisp is a powerful, expressive, and feature-rich programming language known for its interactive development environment, advanced garbage collection, macros system, and dynamic typing. It is widely used in AI, expert systems, and scientific computing. However, Common Lisp can be verbose and has a steep learning curve compared to other modern languages. It also lacks a standard threading model and has a smaller ecosystem compared to more popular languages.
- Clojure: Clojure is a modern dialect of Lisp that runs on the Java Virtual Machine (JVM). It is known for its simplicity, functional programming features, and emphasis on immutability. Clojure has a large and active community, a comprehensive standard library, and seamless interoperability with Java. However, it can be challenging for newcomers due to its syntax and reliance on JVM ecosystem tools.
- Scheme: Scheme is a minimalist dialect of Lisp known for its simplicity and elegance. It has a small core language, powerful macros system, and a focus on functional programming. Scheme is often used in educational settings and research projects. However, it lacks a large standard library and may not have as many production-ready libraries as Common Lisp.
- Racket: Racket is a general-purpose programming language based on Scheme, with a strong emphasis on creating domain-specific languages (DSLs). It provides a rich set of libraries, tools, and documentation for building complex software systems. However, Racket may have a steeper learning curve compared to Common Lisp due to its focus on language-oriented programming.
- Emacs Lisp: Emacs Lisp is a dialect of Lisp specifically designed for extending the Emacs text editor. It is used to customize and enhance Emacs functionality through the creation of macros and scripts. Emacs Lisp offers deep integration with Emacs features, but its usage is limited to Emacs extensions and may not be suitable for general-purpose programming.
- Elixir: Elixir is a functional programming language built on top of the Erlang VM (BEAM) known for its scalability and fault-tolerance. It combines the productivity of modern languages with the reliability of Erlang's concurrency and distribution features. Elixir has a growing community, vast ecosystem of libraries, and built-in support for metaprogramming through macros. However, it follows a different paradigm compared to Lisp languages.
- Haskell: Haskell is a statically-typed functional programming language known for its strong type system, purity, and laziness. It is used in industries such as finance, engineering, and academia for building high-performance and reliable systems. Haskell provides powerful abstractions, advanced type features, and a robust compiler. However, Haskell's syntax and learning curve can be challenging for beginners.
- Scala: Scala is a hybrid functional and object-oriented programming language running on the Java Virtual Machine (JVM). It combines the best features of both paradigms, providing a rich type system, concurrency primitives, and seamless interoperability with Java. Scala is popular for building scalable and performant systems, but it may have a complex syntax and slower compilation times compared to Common Lisp.
- F#: F# is a functional-first programming language targeting the .NET platform known for its succinct syntax, type inference, and expressive features. It is used in domains such as data science, web development, and finance for its productivity and interoperability with existing .NET libraries. F# encourages functional programming practices, but it may have a smaller community and ecosystem compared to Common Lisp.
- Julia: Julia is a high-level, high-performance programming language for numerical computing known for its speed, multiple dispatch, and extensive mathematical libraries. It is widely used in scientific computing, machine learning, and data analysis for its flexibility and ease of use. Julia has a growing community, strong performance, and rich ecosystem, but it may not have the same level of support for macros and metaprogramming as Lisp languages.
- Rust: Rust is a systems programming language focusing on safety, speed, and concurrency. It provides memory safety features, zero-cost abstractions, and strong type system to ensure performance and reliability. Rust is used in areas such as game development, systems programming, and web services for its memory safety guarantees and performance optimizations. However, Rust's learning curve and emphasis on low-level programming may differ from the high-level abstractions of Lisp languages.
Top Alternatives to Common Lisp
- Clojure
Clojure is designed to be a general-purpose language, combining the approachability and interactive development of a scripting language with an efficient and robust infrastructure for multithreaded programming. Clojure is a compiled language - it compiles directly to JVM bytecode, yet remains completely dynamic. Clojure is a dialect of Lisp, and shares with Lisp the code-as-data philosophy and a powerful macro system. ...
- Haskell
It is a general purpose language that can be used in any domain and use case, it is ideally suited for proprietary business logic and data analysis, fast prototyping and enhancing existing software environments with correct code, performance and scalability. ...
- Python
Python is a general purpose programming language created by Guido Van Rossum. Python is most praised for its elegant syntax and readable code, if you are just beginning your programming career python suits you best. ...
- Racket
It is a general-purpose, multi-paradigm programming language based on the Scheme dialect of Lisp. It is designed to be a platform for programming language design and implementation. It is also used for scripting, computer science education, and research. ...
- Java
Java is a programming language and computing platform first released by Sun Microsystems in 1995. There are lots of applications and websites that will not work unless you have Java installed, and more are created every day. Java is fast, secure, and reliable. From laptops to datacenters, game consoles to scientific supercomputers, cell phones to the Internet, Java is everywhere! ...
- JavaScript
JavaScript is most known as the scripting language for Web pages, but used in many non-browser environments as well such as node.js or Apache CouchDB. It is a prototype-based, multi-paradigm scripting language that is dynamic,and supports object-oriented, imperative, and functional programming styles. ...
- Git
Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. ...
Common Lisp alternatives & related posts
Clojure
- It is a lisp117
- Persistent data structures100
- Concise syntax100
- jvm-based language90
- Concurrency89
- Interactive repl81
- Code is data76
- Open source61
- Lazy data structures61
- Macros57
- Functional49
- Simplistic23
- Immutable by default22
- Excellent collections20
- Fast-growing community19
- Multiple host languages15
- Simple (not easy!)15
- Practical Lisp15
- Because it's really fun to use10
- Addictive10
- Community9
- Web friendly9
- Rapid development9
- It creates Reusable code9
- Minimalist8
- Programmable programming language6
- Java interop6
- Regained interest in programming5
- Compiles to JavaScript4
- Share a lot of code with clojurescript/use on frontend3
- EDN3
- Clojurescript1
- Cryptic stacktraces11
- Need to wrap basically every java lib5
- Toxic community4
- Good code heavily relies on local conventions3
- Tonns of abandonware3
- Slow application startup3
- Usable only with REPL1
- Hiring issues1
- It's a lisp1
- Bad documented libs1
- Macros are overused by devs1
- Tricky profiling1
- IDE with high learning curve1
- Configuration bolierplate1
- Conservative community1
- Have no good and fast fmt0
related Clojure posts
The majority of our Clojure microservices are simple web services that wrap a transactional database with CRUD operations and a little bit of business logic. We use both MySQL and PostgreSQL for transactional data persistence, having transitioned from the former to the latter for newer services to take advantage of the new features coming out of the Postgres community.
Most of our Clojure best practices can be summed up by the phrase "keep it simple." We avoid more complex web frameworks in favor of using the Ring library to build web service routes, and we prefer sending SQL directly to the JDBC library rather than using a complicated ORM or SQL DSL.
Stitch is run entirely on AWS. All of our transactional databases are run with Amazon RDS, and we rely on Amazon S3 for data persistence in various stages of our pipeline. Our product integrates with Amazon Redshift as a data destination, and we also use Redshift as an internal data warehouse (powered by Stitch, of course).
The majority of our services run on stateless Amazon EC2 instances that are managed by AWS OpsWorks. We recently introduced Kubernetes into our infrastructure to run the scheduled jobs that execute Singer code to extract data from various sources. Although we tend to be wary of shiny new toys, Kubernetes has proven to be a good fit for this problem, and its stability, strong community and helpful tooling have made it easy for us to incorporate into our operations.
While we continue to be happy with Clojure for our internal services, we felt that its relatively narrow adoption could impede Singer's growth. We chose Python both because it is well suited to the task, and it seems to have reached critical mass among data engineers. All that being said, the Singer spec is language agnostic, and integrations and libraries have been developed in JavaScript, Go, and Clojure.
- Purely-functional programming90
- Statically typed66
- Type-safe59
- Open source39
- Great community38
- Built-in concurrency31
- Built-in parallelism30
- Composable30
- Referentially transparent24
- Generics20
- Type inference15
- Intellectual satisfaction15
- If it compiles, it's correct12
- Flexible8
- Monads8
- Great type system5
- Proposition testing with QuickCheck4
- One of the most powerful languages *(see blub paradox)*4
- Purely-functional Programming4
- Highly expressive, type-safe, fast development time3
- Pattern matching and completeness checking3
- Great maintainability of the code3
- Fun3
- Reliable3
- Best in class thinking tool2
- Kind system2
- Better type-safe than sorry2
- Type classes2
- Predictable1
- Orthogonality1
- Too much distraction in language extensions9
- Error messages can be very confusing8
- Libraries have poor documentation5
- No good ABI3
- No best practices3
- Poor packaging for apps written in it for Linux distros2
- Sometimes performance is unpredictable2
- Slow compilation1
- Monads are hard to understand1
related Haskell posts
In early 2015, Uber Engineering migrated its business entities from integer identifiers to UUID identifiers as part of an initiative focused on using multiple active data centers. To do that, Uber engineers had to identify foreign key relationships between every table in the data warehouse—a nontrivial task by any accounting.
Uber’s solution was to observe and analyze incoming SQL queries to extract foreign key relationships, for which it built tool called Queryparser, which it open sourced.)
Queryparser is written in Haskell, a tool that the team wasn’t previously familiar with but has strong support for language parsing. To help each other get up to speed, engineers started a weekly reading group to discuss Haskell books and documentation.
Why I am using Haskell in my free time?
I have 3 reasons for it. I am looking for:
Fun.
Improve functional programming skill.
Improve problem-solving skill.
Laziness and mathematical abstractions behind Haskell makes it a wonderful language.
It is Pure functional, it helps me to write better Scala code.
Highly expressive language gives elegant ways to solve coding puzzle.
Python
- Great libraries1.2K
- Readable code959
- Beautiful code844
- Rapid development785
- Large community688
- Open source434
- Elegant391
- Great community280
- Object oriented272
- Dynamic typing218
- Great standard library77
- Very fast58
- Functional programming54
- Easy to learn48
- Scientific computing45
- Great documentation35
- Easy to read28
- Productivity28
- Matlab alternative28
- Simple is better than complex23
- It's the way I think20
- Imperative19
- Free18
- Very programmer and non-programmer friendly18
- Machine learning support17
- Powerfull language17
- Fast and simple16
- Scripting14
- Explicit is better than implicit12
- Ease of development11
- Clear and easy and powerfull10
- Unlimited power9
- It's lean and fun to code8
- Import antigravity8
- Python has great libraries for data processing7
- Print "life is short, use python"7
- Flat is better than nested6
- Readability counts6
- Rapid Prototyping6
- Fast coding and good for competitions6
- Now is better than never6
- There should be one-- and preferably only one --obvious6
- High Documented language6
- I love snakes6
- Although practicality beats purity6
- Great for tooling6
- Great for analytics5
- Lists, tuples, dictionaries5
- Multiple Inheritence4
- Complex is better than complicated4
- Socially engaged community4
- Easy to learn and use4
- Simple and easy to learn4
- Web scraping4
- Easy to setup and run smooth4
- Beautiful is better than ugly4
- Plotting4
- CG industry needs4
- No cruft3
- It is Very easy , simple and will you be love programmi3
- Many types of collections3
- If the implementation is easy to explain, it may be a g3
- If the implementation is hard to explain, it's a bad id3
- Special cases aren't special enough to break the rules3
- Pip install everything3
- List comprehensions3
- Generators3
- Import this3
- Flexible and easy2
- Batteries included2
- Can understand easily who are new to programming2
- Powerful language for AI2
- Should START with this but not STICK with This2
- A-to-Z2
- Because of Netflix2
- Only one way to do it2
- Better outcome2
- Good for hacking2
- Securit1
- Slow1
- Sexy af1
- Ni0
- Powerful0
- Still divided between python 2 and python 353
- Performance impact28
- Poor syntax for anonymous functions26
- GIL22
- Package management is a mess19
- Too imperative-oriented14
- Hard to understand12
- Dynamic typing12
- Very slow12
- Indentations matter a lot8
- Not everything is expression8
- Incredibly slow7
- Explicit self parameter in methods7
- Requires C functions for dynamic modules6
- Poor DSL capabilities6
- No anonymous functions6
- Fake object-oriented programming5
- Threading5
- The "lisp style" whitespaces5
- Official documentation is unclear.5
- Hard to obfuscate5
- Circular import5
- Lack of Syntax Sugar leads to "the pyramid of doom"4
- The benevolent-dictator-for-life quit4
- Not suitable for autocomplete4
- Meta classes2
- Training wheels (forced indentation)1
related Python posts
How Uber developed the open source, end-to-end distributed tracing Jaeger , now a CNCF project:
Distributed tracing is quickly becoming a must-have component in the tools that organizations use to monitor their complex, microservice-based architectures. At Uber, our open source distributed tracing system Jaeger saw large-scale internal adoption throughout 2016, integrated into hundreds of microservices and now recording thousands of traces every second.
Here is the story of how we got here, from investigating off-the-shelf solutions like Zipkin, to why we switched from pull to push architecture, and how distributed tracing will continue to evolve:
https://eng.uber.com/distributed-tracing/
(GitHub Pages : https://www.jaegertracing.io/, GitHub: https://github.com/jaegertracing/jaeger)
Bindings/Operator: Python Java Node.js Go C++ Kubernetes JavaScript OpenShift C# Apache Spark
Winds 2.0 is an open source Podcast/RSS reader developed by Stream with a core goal to enable a wide range of developers to contribute.
We chose JavaScript because nearly every developer knows or can, at the very least, read JavaScript. With ES6 and Node.js v10.x.x, it’s become a very capable language. Async/Await is powerful and easy to use (Async/Await vs Promises). Babel allows us to experiment with next-generation JavaScript (features that are not in the official JavaScript spec yet). Yarn allows us to consistently install packages quickly (and is filled with tons of new tricks)
We’re using JavaScript for everything – both front and backend. Most of our team is experienced with Go and Python, so Node was not an obvious choice for this app.
Sure... there will be haters who refuse to acknowledge that there is anything remotely positive about JavaScript (there are even rants on Hacker News about Node.js); however, without writing completely in JavaScript, we would not have seen the results we did.
#FrameworksFullStack #Languages
- Meta-programming3
- Hygienic macros3
- Pattern matching2
- Module system2
- Beginner friendly2
- Fast2
- Gradual typing2
- Nanopass compiler2
- Extensible2
- Racket Macro system2
- Cross platform GUI2
- Macro Stepper2
- Built-in concurrency2
- Built-in parallelism2
- Functional Programming2
- Open source2
- Language-oriented programming2
- FFI2
- Great libraries2
- Beautiful code2
- Rapid development2
- Racketscript1
- Great community1
- Typed Racket1
- IDE1
- Good documentation1
- LISP BASED2
- No GitHub2
related Racket posts
Java
- Great libraries599
- Widely used445
- Excellent tooling400
- Huge amount of documentation available395
- Large pool of developers available334
- Open source208
- Excellent performance202
- Great development157
- Used for android150
- Vast array of 3rd party libraries148
- Compiled Language60
- Used for Web52
- High Performance46
- Managed memory46
- Native threads44
- Statically typed43
- Easy to read35
- Great Community33
- Reliable platform29
- Sturdy garbage collection24
- JVM compatibility24
- Cross Platform Enterprise Integration22
- Universal platform20
- Good amount of APIs20
- Great Support18
- Great ecosystem14
- Backward compatible11
- Lots of boilerplate11
- Everywhere10
- Excellent SDK - JDK9
- It's Java7
- Cross-platform7
- Static typing7
- Mature language thus stable systems6
- Better than Ruby6
- Long term language6
- Portability6
- Clojure5
- Vast Collections Library5
- Used for Android development5
- Most developers favorite4
- Old tech4
- History3
- Great Structure3
- Stable platform, which many new languages depend on3
- Javadoc3
- Testable3
- Best martial for design3
- Type Safe2
- Faster than python2
- Job0
- Verbosity33
- NullpointerException27
- Nightmare to Write17
- Overcomplexity is praised in community culture16
- Boiler plate code12
- Classpath hell prior to Java 98
- No REPL6
- No property4
- Code are too long3
- Non-intuitive generic implementation2
- There is not optional parameter2
- Floating-point errors2
- Java's too statically, stronglly, and strictly typed1
- Returning Wildcard Types1
- Terrbible compared to Python/Batch Perormence1
related Java posts
How Uber developed the open source, end-to-end distributed tracing Jaeger , now a CNCF project:
Distributed tracing is quickly becoming a must-have component in the tools that organizations use to monitor their complex, microservice-based architectures. At Uber, our open source distributed tracing system Jaeger saw large-scale internal adoption throughout 2016, integrated into hundreds of microservices and now recording thousands of traces every second.
Here is the story of how we got here, from investigating off-the-shelf solutions like Zipkin, to why we switched from pull to push architecture, and how distributed tracing will continue to evolve:
https://eng.uber.com/distributed-tracing/
(GitHub Pages : https://www.jaegertracing.io/, GitHub: https://github.com/jaegertracing/jaeger)
Bindings/Operator: Python Java Node.js Go C++ Kubernetes JavaScript OpenShift C# Apache Spark
When you think about test automation, it’s crucial to make it everyone’s responsibility (not just QA Engineers'). We started with Selenium and Java, but with our platform revolving around Ruby, Elixir and JavaScript, QA Engineers were left alone to automate tests. Cypress was the answer, as we could switch to JS and simply involve more people from day one. There's a downside too, as it meant testing on Chrome only, but that was "good enough" for us + if really needed we can always cover some specific cases in a different way.
- Performance68
- Low-level49
- Portability35
- Hardware level28
- Embedded apps19
- Pure13
- Performance of assembler9
- Ubiquity8
- Great for embedded6
- Old4
- Compiles quickly3
- OpenMP2
- No garbage collection to slow it down2
- Gnu/linux interoperable1
- Low-level5
- No built in support for concurrency3
- Lack of type safety2
- No built in support for parallelism (e.g. map-reduce)2
related C lang posts
Why Uber developed H3, our open source grid system to make geospatial data visualization and exploration easier and more efficient:
We decided to create H3 to combine the benefits of a hexagonal global grid system with a hierarchical indexing system. A global grid system usually requires at least two things: a map projection and a grid laid on top of the map. For map projection, we chose to use gnomonic projections centered on icosahedron faces. This projects from Earth as a sphere to an icosahedron, a twenty-sided platonic solid. The H3 grid is constructed by laying out 122 base cells over the Earth, with ten cells per face. H3 supports sixteen resolutions: https://eng.uber.com/h3/
(GitHub Pages : https://uber.github.io/h3/#/ Written in C w/ bindings in Java & JavaScript )
One important decision for delivering a platform independent solution with low memory footprint and minimal dependencies was the choice of the programming language. We considered a few from Python (there was already a reasonably large Python code base at Thumbtack), to Go (we were taking our first steps with it), and even Rust (too immature at the time).
We ended up writing it in C. It was easy to meet all requirements with only one external dependency for implementing the web server, clearly no challenges running it on any of the Linux distributions we were maintaining, and arguably the implementation with the smallest memory footprint given the choices above.
JavaScript
- Can be used on frontend/backend1.7K
- It's everywhere1.5K
- Lots of great frameworks1.2K
- Fast896
- Light weight745
- Flexible425
- You can't get a device today that doesn't run js392
- Non-blocking i/o286
- Ubiquitousness236
- Expressive191
- Extended functionality to web pages55
- Relatively easy language49
- Executed on the client side46
- Relatively fast to the end user30
- Pure Javascript25
- Functional programming21
- Async15
- Full-stack13
- Setup is easy12
- Its everywhere12
- JavaScript is the New PHP11
- Because I love functions11
- Like it or not, JS is part of the web standard10
- Can be used in backend, frontend and DB9
- Expansive community9
- Future Language of The Web9
- Easy9
- No need to use PHP8
- For the good parts8
- Can be used both as frontend and backend as well8
- Everyone use it8
- Most Popular Language in the World8
- Easy to hire developers8
- Love-hate relationship7
- Powerful7
- Photoshop has 3 JS runtimes built in7
- Evolution of C7
- Popularized Class-Less Architecture & Lambdas7
- Agile, packages simple to use7
- Supports lambdas and closures7
- 1.6K Can be used on frontend/backend6
- It's fun6
- Hard not to use6
- Nice6
- Client side JS uses the visitors CPU to save Server Res6
- Versitile6
- It let's me use Babel & Typescript6
- Easy to make something6
- Its fun and fast6
- Can be used on frontend/backend/Mobile/create PRO Ui6
- Function expressions are useful for callbacks5
- What to add5
- Client processing5
- Everywhere5
- Scope manipulation5
- Stockholm Syndrome5
- Promise relationship5
- Clojurescript5
- Because it is so simple and lightweight4
- Only Programming language on browser4
- Hard to learn1
- Test1
- Test21
- Easy to understand1
- Not the best1
- Easy to learn1
- Subskill #41
- Hard 彤0
- A constant moving target, too much churn22
- Horribly inconsistent20
- Javascript is the New PHP15
- No ability to monitor memory utilitization9
- Shows Zero output in case of ANY error8
- Thinks strange results are better than errors7
- Can be ugly6
- No GitHub3
- Slow2
related JavaScript posts
Oof. I have truly hated JavaScript for a long time. Like, for over twenty years now. Like, since the Clinton administration. It's always been a nightmare to deal with all of the aspects of that silly language.
But wowza, things have changed. Tooling is just way, way better. I'm primarily web-oriented, and using React and Apollo together the past few years really opened my eyes to building rich apps. And I deeply apologize for using the phrase rich apps; I don't think I've ever said such Enterprisey words before.
But yeah, things are different now. I still love Rails, and still use it for a lot of apps I build. But it's that silly rich apps phrase that's the problem. Users have way more comprehensive expectations than they did even five years ago, and the JS community does a good job at building tools and tech that tackle the problems of making heavy, complicated UI and frontend work.
Obviously there's a lot of things happening here, so just saying "JavaScript isn't terrible" might encompass a huge amount of libraries and frameworks. But if you're like me, yeah, give things another shot- I'm somehow not hating on JavaScript anymore and... gulp... I kinda love it.
How Uber developed the open source, end-to-end distributed tracing Jaeger , now a CNCF project:
Distributed tracing is quickly becoming a must-have component in the tools that organizations use to monitor their complex, microservice-based architectures. At Uber, our open source distributed tracing system Jaeger saw large-scale internal adoption throughout 2016, integrated into hundreds of microservices and now recording thousands of traces every second.
Here is the story of how we got here, from investigating off-the-shelf solutions like Zipkin, to why we switched from pull to push architecture, and how distributed tracing will continue to evolve:
https://eng.uber.com/distributed-tracing/
(GitHub Pages : https://www.jaegertracing.io/, GitHub: https://github.com/jaegertracing/jaeger)
Bindings/Operator: Python Java Node.js Go C++ Kubernetes JavaScript OpenShift C# Apache Spark
- Distributed version control system1.4K
- Efficient branching and merging1.1K
- Fast959
- Open source845
- Better than svn726
- Great command-line application368
- Simple306
- Free291
- Easy to use232
- Does not require server222
- Distributed27
- Small & Fast22
- Feature based workflow18
- Staging Area15
- Most wide-spread VSC13
- Role-based codelines11
- Disposable Experimentation11
- Frictionless Context Switching7
- Data Assurance6
- Efficient5
- Just awesome4
- Github integration3
- Easy branching and merging3
- Compatible2
- Flexible2
- Possible to lose history and commits2
- Rebase supported natively; reflog; access to plumbing1
- Light1
- Team Integration1
- Fast, scalable, distributed revision control system1
- Easy1
- Flexible, easy, Safe, and fast1
- CLI is great, but the GUI tools are awesome1
- It's what you do1
- Phinx0
- Hard to learn16
- Inconsistent command line interface11
- Easy to lose uncommitted work9
- Worst documentation ever possibly made7
- Awful merge handling5
- Unexistent preventive security flows3
- Rebase hell3
- When --force is disabled, cannot rebase2
- Ironically even die-hard supporters screw up badly2
- Doesn't scale for big data1
related Git posts
Our whole DevOps stack consists of the following tools:
- GitHub (incl. GitHub Pages/Markdown for Documentation, GettingStarted and HowTo's) for collaborative review and code management tool
- Respectively Git as revision control system
- SourceTree as Git GUI
- Visual Studio Code as IDE
- CircleCI for continuous integration (automatize development process)
- Prettier / TSLint / ESLint as code linter
- SonarQube as quality gate
- Docker as container management (incl. Docker Compose for multi-container application management)
- VirtualBox for operating system simulation tests
- Kubernetes as cluster management for docker containers
- Heroku for deploying in test environments
- nginx as web server (preferably used as facade server in production environment)
- SSLMate (using OpenSSL) for certificate management
- Amazon EC2 (incl. Amazon S3) for deploying in stage (production-like) and production environments
- PostgreSQL as preferred database system
- Redis as preferred in-memory database/store (great for caching)
The main reason we have chosen Kubernetes over Docker Swarm is related to the following artifacts:
- Key features: Easy and flexible installation, Clear dashboard, Great scaling operations, Monitoring is an integral part, Great load balancing concepts, Monitors the condition and ensures compensation in the event of failure.
- Applications: An application can be deployed using a combination of pods, deployments, and services (or micro-services).
- Functionality: Kubernetes as a complex installation and setup process, but it not as limited as Docker Swarm.
- Monitoring: It supports multiple versions of logging and monitoring when the services are deployed within the cluster (Elasticsearch/Kibana (ELK), Heapster/Grafana, Sysdig cloud integration).
- Scalability: All-in-one framework for distributed systems.
- Other Benefits: Kubernetes is backed by the Cloud Native Computing Foundation (CNCF), huge community among container orchestration tools, it is an open source and modular tool that works with any OS.
Often enough I have to explain my way of going about setting up a CI/CD pipeline with multiple deployment platforms. Since I am a bit tired of yapping the same every single time, I've decided to write it up and share with the world this way, and send people to read it instead ;). I will explain it on "live-example" of how the Rome got built, basing that current methodology exists only of readme.md and wishes of good luck (as it usually is ;)).
It always starts with an app, whatever it may be and reading the readmes available while Vagrant and VirtualBox is installing and updating. Following that is the first hurdle to go over - convert all the instruction/scripts into Ansible playbook(s), and only stopping when doing a clear vagrant up
or vagrant reload
we will have a fully working environment. As our Vagrant environment is now functional, it's time to break it! This is the moment to look for how things can be done better (too rigid/too lose versioning? Sloppy environment setup?) and replace them with the right way to do stuff, one that won't bite us in the backside. This is the point, and the best opportunity, to upcycle the existing way of doing dev environment to produce a proper, production-grade product.
I should probably digress here for a moment and explain why. I firmly believe that the way you deploy production is the same way you should deploy develop, shy of few debugging-friendly setting. This way you avoid the discrepancy between how production work vs how development works, which almost always causes major pains in the back of the neck, and with use of proper tools should mean no more work for the developers. That's why we start with Vagrant as developer boxes should be as easy as vagrant up
, but the meat of our product lies in Ansible which will do meat of the work and can be applied to almost anything: AWS, bare metal, docker, LXC, in open net, behind vpn - you name it.
We must also give proper consideration to monitoring and logging hoovering at this point. My generic answer here is to grab Elasticsearch, Kibana, and Logstash. While for different use cases there may be better solutions, this one is well battle-tested, performs reasonably and is very easy to scale both vertically (within some limits) and horizontally. Logstash rules are easy to write and are well supported in maintenance through Ansible, which as I've mentioned earlier, are at the very core of things, and creating triggers/reports and alerts based on Elastic and Kibana is generally a breeze, including some quite complex aggregations.
If we are happy with the state of the Ansible it's time to move on and put all those roles and playbooks to work. Namely, we need something to manage our CI/CD pipelines. For me, the choice is obvious: TeamCity. It's modern, robust and unlike most of the light-weight alternatives, it's transparent. What I mean by that is that it doesn't tell you how to do things, doesn't limit your ways to deploy, or test, or package for that matter. Instead, it provides a developer-friendly and rich playground for your pipelines. You can do most the same with Jenkins, but it has a quite dated look and feel to it, while also missing some key functionality that must be brought in via plugins (like quality REST API which comes built-in with TeamCity). It also comes with all the common-handy plugins like Slack or Apache Maven integration.
The exact flow between CI and CD varies too greatly from one application to another to describe, so I will outline a few rules that guide me in it: 1. Make build steps as small as possible. This way when something breaks, we know exactly where, without needing to dig and root around. 2. All security credentials besides development environment must be sources from individual Vault instances. Keys to those containers should exist only on the CI/CD box and accessible by a few people (the less the better). This is pretty self-explanatory, as anything besides dev may contain sensitive data and, at times, be public-facing. Because of that appropriate security must be present. TeamCity shines in this department with excellent secrets-management. 3. Every part of the build chain shall consume and produce artifacts. If it creates nothing, it likely shouldn't be its own build. This way if any issue shows up with any environment or version, all developer has to do it is grab appropriate artifacts to reproduce the issue locally. 4. Deployment builds should be directly tied to specific Git branches/tags. This enables much easier tracking of what caused an issue, including automated identifying and tagging the author (nothing like automated regression testing!).
Speaking of deployments, I generally try to keep it simple but also with a close eye on the wallet. Because of that, I am more than happy with AWS or another cloud provider, but also constantly peeking at the loads and do we get the value of what we are paying for. Often enough the pattern of use is not constantly erratic, but rather has a firm baseline which could be migrated away from the cloud and into bare metal boxes. That is another part where this approach strongly triumphs over the common Docker and CircleCI setup, where you are very much tied in to use cloud providers and getting out is expensive. Here to embrace bare-metal hosting all you need is a help of some container-based self-hosting software, my personal preference is with Proxmox and LXC. Following that all you must write are ansible scripts to manage hardware of Proxmox, similar way as you do for Amazon EC2 (ansible supports both greatly) and you are good to go. One does not exclude another, quite the opposite, as they can live in great synergy and cut your costs dramatically (the heavier your base load, the bigger the savings) while providing production-grade resiliency.