Zulip

I have mixed feelings on GitHub as a product and our use of it for the Zulip open source project. On the one hand, I do feel that being on GitHub helps people discover Zulip, because we have enough stars (etc.) that we rank highly among projects on the platform. and there is a definite benefit for lowering barriers to contribution (which is important to us) that GitHub has such a dominant position in terms of what everyone has accounts with.

But even ignoring how one might feel about their new corporate owner (MicroSoft), in a lot of ways GitHub is a bad product for open source projects. Years after the "Dear GitHub" letter, there are still basic gaps in its issue tracker:

• You can't give someone permission to label/categorize issues without full write access to a project (including ability to merge things to master, post releases, etc.).
• You can't let anyone with a GitHub account self-assign issues to themselves.
• Many more similar issues.

It's embarrassing, because I've talked to GitHub product managers at various open source events about these things for 3 years, and they always agree the thing is important, but then nothing ever improves in the Issues product. Maybe the new management at MicroSoft will fix their product management situation, but if not, I imagine we'll eventually do the migration to GitLab.

We have a custom bot project, http://github.com/zulip/zulipbot, to deal with some of these issues where possible, and every other large project we talk to does the same thing, more or less.

We've been using PostgreSQL since the very early days of Zulip, but we actually didn't use it from the beginning. Zulip started out as a MySQL project back in 2012, because we'd heard it was a good choice for a startup with a wide community. However, we found that even though we were using the Django ORM for most of our database access, we spent a lot of time fighting with MySQL. Issues ranged from bad collation defaults, to bad query plans which required a lot of manual query tweaks.

We ended up getting so frustrated that we tried out PostgresQL, and the results were fantastic. We didn't have to do any real customization (just some tuning settings for how big a server we had), and all of our most important queries were faster out of the box. As a result, we were able to delete a bunch of custom queries escaping the ORM that we'd written to make the MySQL query planner happy (because postgres just did the right thing automatically).

And then after that, we've just gotten a ton of value out of postgres. We use its excellent built-in full-text search, which has helped us avoid needing to bring in a tool like Elasticsearch, and we've really enjoyed features like its partial indexes, which saved us a lot of work adding unnecessary extra tables to get good performance for things like our "unread messages" and "starred messages" indexes.

I can't recommend it highly enough.

We recently adopted the (previously community-developed) Docker image for Zulip, and I've gotten a bunch of experience as part of this process. We didn't have a lot of choice as to whether to use Docker -- as an open source web application, installation options are driven by what users are asking for, and in the container world, that's a Docker image.

To be honest, maintaining a Docker image is kind of a pain, even for something like Zulip where our Debian/Ubuntu install process is super clean and reliable. Docker requires you to run your application's installer in a totally crippled environment (E.g. no unicode locales are installed, so using pip to build Python packages doesn't work until you fix that) to generate a working Dockerfile. But the real pain is the fact that Docker users expect to configure everything via environment variables in e.g. a docker-compose.yml or similar Kubernetes configuration file.

And that results in just about every Docker image for a reusable web application having a giant docker-entrypoint.sh script of semi-duplicated code (ours is currently ~600LOC; those for other popular projects can easily be 2-3X that size) that does a bunch of stuff, including pass what is structurally an array through a shell environment variable and back into our Python-format settings file, which is just a mess.

Since that shell script can have bugs, this is a big maintenance (e.g. right now probably a third of user reports of issues installing a new Zulip server are Docker-specific issues, even though Docker installations are a small fraction of total installations).

Further, a lot of third-party images (e.g. for postgres or redis or RabbitMQ) end up having issues where they have an environment variable for configuring something (E.g. a postgres password), but if you change that variable after the postgres image is first booted, nothing happens, which can be confusing for users (who often assume they can just boot their image with whatever password and clean up the security stuff later).

So, while the Docker/containers ecosystem has a lot of promise, I also feel it has a long way to go before providing a Docker image isn't a significant burden on projects, even ones like us that already have a battle-tested installation process.

We've been using RabbitMQ as Zulip's queuing system since we needed a queuing system. What I like about it is that it scales really well and has good libraries for a wide range of platforms, including our own Python. So aside from getting it running, we've had to put basically 0 effort into making it scale for our needs.

However, there's several things that could be better about it: * It's error messages are absolutely terrible; if ever one of our users ends up getting an error with RabbitMQ (even for simple things like a misconfigured hostname), they always end up needing to get help from the Zulip team, because the errors logs are just inscrutable. As an open source project, we've handled this issue by really carefully scripting the installation to be a failure-proof configuration (in this case, setting the RabbitMQ hostname to 127.0.0.1, so that no user-controlled configuration can break it). But it was a real pain to get there and the process of determining we needed to do that caused a significant amount of pain to folks installing Zulip. * The pika library for Python takes a lot of time to startup a RabbitMQ connection; this means that Zulip server restarts are more disruptive than would be ideal. * It's annoying that you need to run the rabbitmqctl management commands as root.

But overall, I like that it has clean, clear semanstics and high scalability, and haven't been tempted to do the work to migrate to something like Redis (which has its own downsides).