Zulip
Avatar of tabbott
Founder at Zulip ·
Shared insights
on
GitHubGitHubGitLabGitLab
at

I have mixed feelings on GitHub as a product and our use of it for the Zulip open source project. On the one hand, I do feel that being on GitHub helps people discover Zulip, because we have enough stars (etc.) that we rank highly among projects on the platform. and there is a definite benefit for lowering barriers to contribution (which is important to us) that GitHub has such a dominant position in terms of what everyone has accounts with.

But even ignoring how one might feel about their new corporate owner (MicroSoft), in a lot of ways GitHub is a bad product for open source projects. Years after the "Dear GitHub" letter, there are still basic gaps in its issue tracker:

  • You can't give someone permission to label/categorize issues without full write access to a project (including ability to merge things to master, post releases, etc.).
  • You can't let anyone with a GitHub account self-assign issues to themselves.
  • Many more similar issues.

It's embarrassing, because I've talked to GitHub product managers at various open source events about these things for 3 years, and they always agree the thing is important, but then nothing ever improves in the Issues product. Maybe the new management at MicroSoft will fix their product management situation, but if not, I imagine we'll eventually do the migration to GitLab.

We have a custom bot project, http://github.com/zulip/zulipbot, to deal with some of these issues where possible, and every other large project we talk to does the same thing, more or less.

READ MORE
GitHub - zulip/zulip: Zulip server - powerful open source team chat (github.com)
24 upvotes·2 comments·292.5K views
Avatar of tabbott
Founder at Zulip ·

We've been using PostgreSQL since the very early days of Zulip, but we actually didn't use it from the beginning. Zulip started out as a MySQL project back in 2012, because we'd heard it was a good choice for a startup with a wide community. However, we found that even though we were using the Django ORM for most of our database access, we spent a lot of time fighting with MySQL. Issues ranged from bad collation defaults, to bad query plans which required a lot of manual query tweaks.

We ended up getting so frustrated that we tried out PostgresQL, and the results were fantastic. We didn't have to do any real customization (just some tuning settings for how big a server we had), and all of our most important queries were faster out of the box. As a result, we were able to delete a bunch of custom queries escaping the ORM that we'd written to make the MySQL query planner happy (because postgres just did the right thing automatically).

And then after that, we've just gotten a ton of value out of postgres. We use its excellent built-in full-text search, which has helped us avoid needing to bring in a tool like Elasticsearch, and we've really enjoyed features like its partial indexes, which saved us a lot of work adding unnecessary extra tables to get good performance for things like our "unread messages" and "starred messages" indexes.

I can't recommend it highly enough.

READ MORE
23 upvotes·262.2K views
Avatar of tabbott
Founder at Zulip ·
Shared insights
on
DockerDockerKubernetesKubernetes
at

We recently adopted the (previously community-developed) Docker image for Zulip, and I've gotten a bunch of experience as part of this process. We didn't have a lot of choice as to whether to use Docker -- as an open source web application, installation options are driven by what users are asking for, and in the container world, that's a Docker image.

To be honest, maintaining a Docker image is kind of a pain, even for something like Zulip where our Debian/Ubuntu install process is super clean and reliable. Docker requires you to run your application's installer in a totally crippled environment (E.g. no unicode locales are installed, so using pip to build Python packages doesn't work until you fix that) to generate a working Dockerfile. But the real pain is the fact that Docker users expect to configure everything via environment variables in e.g. a docker-compose.yml or similar Kubernetes configuration file.

And that results in just about every Docker image for a reusable web application having a giant docker-entrypoint.sh script of semi-duplicated code (ours is currently ~600LOC; those for other popular projects can easily be 2-3X that size) that does a bunch of stuff, including pass what is structurally an array through a shell environment variable and back into our Python-format settings file, which is just a mess.

Since that shell script can have bugs, this is a big maintenance (e.g. right now probably a third of user reports of issues installing a new Zulip server are Docker-specific issues, even though Docker installations are a small fraction of total installations).

Further, a lot of third-party images (e.g. for postgres or redis or RabbitMQ) end up having issues where they have an environment variable for configuring something (E.g. a postgres password), but if you change that variable after the postgres image is first booted, nothing happens, which can be confusing for users (who often assume they can just boot their image with whatever password and clean up the security stuff later).

So, while the Docker/containers ecosystem has a lot of promise, I also feel it has a long way to go before providing a Docker image isn't a significant burden on projects, even ones like us that already have a battle-tested installation process.

READ MORE
GitHub - zulip/docker-zulip: Docker Image and configuration for Zulip (github.com)
15 upvotes·14.8K views
Avatar of tabbott
Founder at Zulip ·
Shared insights
on
RabbitMQRabbitMQPythonPythonRedisRedis
at

We've been using RabbitMQ as Zulip's queuing system since we needed a queuing system. What I like about it is that it scales really well and has good libraries for a wide range of platforms, including our own Python. So aside from getting it running, we've had to put basically 0 effort into making it scale for our needs.

However, there's several things that could be better about it: * It's error messages are absolutely terrible; if ever one of our users ends up getting an error with RabbitMQ (even for simple things like a misconfigured hostname), they always end up needing to get help from the Zulip team, because the errors logs are just inscrutable. As an open source project, we've handled this issue by really carefully scripting the installation to be a failure-proof configuration (in this case, setting the RabbitMQ hostname to 127.0.0.1, so that no user-controlled configuration can break it). But it was a real pain to get there and the process of determining we needed to do that caused a significant amount of pain to folks installing Zulip. * The pika library for Python takes a lot of time to startup a RabbitMQ connection; this means that Zulip server restarts are more disruptive than would be ideal. * It's annoying that you need to run the rabbitmqctl management commands as root.

But overall, I like that it has clean, clear semanstics and high scalability, and haven't been tempted to do the work to migrate to something like Redis (which has its own downsides).

READ MORE
Queue processors — Zulip 1.9.1+git documentation (zulip.readthedocs.io)
14 upvotes·223.5K views
Avatar of tabbott
Founder at Zulip ·
Shared insights
on
Travis CITravis CICircleCICircleCI
at

We actually started out on Travis CI, but we've migrated our main builds to CircleCI, and it's been a huge improvement.

The reason it's been a huge improvement is that Travis CI has a fundamentally bad design for their images, where they start with a standard base Linux image containing tons of packages (several versions of postgres, every programming language environment, etc). This is potentially nice for the "get builds for a small project running quickly" use case, but it's a total disaster for a larger project that needs a decent number of dependencies and cares about the performance and reliability of their build.

This issue is exacerbated by their networking infrastructure being unreliable; we usually saw over 1% of builds failing due to transient networking errors in Travis CI, even after we added retries to the most frequently failing operations like apt update or pip install. And they never install Ubuntu's point release updates to their images. So doing an apt update, apt install, or especially apt upgrade would take forever. We ended up writing code to actually uninstall many of their base packages and pin the versions of hundreds of others to get a semi-fast, semi-reliable build. It was infuriating.

The CircleCI v2.0 system has the right design for a CI system: we can customize the base image to start with any expensive-to-install packages we need for our build, and we can update that image if and when we want to. The end result is that when migrating, we were able to delete all the hacky optimizations mentioned above, while still ending up with a 50% faster build latency. And we've also had 5-10x fewer issues with networking-related flakes, which means one doesn't have to constantly check whether a build failure is actually due to an issue with the code under test or "just another networking flake".

READ MORE
14 upvotes·84K views
Avatar of tabbott
Founder at Zulip ·
Shared insights
on
MarkdownMarkdown
at

We've been incredibly happy with the choice of Markdown as the composition language for Zulip:

  • Everyone can write it, since it's just like email and many other products use it.
  • It has a WYSIWYG type experience, so we don't feel the need to integrate a fancy editor like Quill.
  • There are mature markdown processors for every programming language. Having compatible, high-quality implementations for Python and JavaScript is important for doing local echo properly (see our linked Markdown documentation for details of why is necessary).
  • Rendering to HTML is fast, easy to do automated tests for, and easy to debug.

We have had to customize our Markdown implementation somewhat for the chat context, because we don't have the guarantee that content is not split across multiple messages (a good example is that you don't want the "automatically change numbered lists to start at 1" feature in default markdown; you want to restrict that to numbered lists where all the numbers are 1, for example) .

READ MORE
Markdown implementation — Zulip 1.9.1+git documentation (zulip.readthedocs.io)
14 upvotes·11.8K views
Avatar of tabbott
Founder at Zulip ·
Shared insights
on
React NativeReact Native
at

We've been using React Native for the Zulip mobile apps, and while there's definitely problems with the platform, it's also saved us a huge amount of time.

One of the things that people don't talk about enough is the product thinking cost of maintaining duplicate codebases for the same app, as one would have with a traditional fully native app for both platforms.

That said, the RN ecosystem has frequently been frustrating; RN releases often break important things, often fairly basic features of the underlying implementations (like support for automatically following redirects in various networking contexts) aren't exposed properly, which can result in a bunch of extra work. But at the same time, we're saving all of the work of maintaining two redundant mobile teams, determining and communicating details of the server/client interface with those two teams, and fixing bugs twice.

Overall, we're happy with the React Native decision, since I think it's the best platform available for what it does in 2018. But I'm also interested in whether Flutter, which in our view has a better development strategy/structure, can provide a better cross-platform mobile development experience in the coming years

READ MORE
GitHub - zulip/zulip-mobile: The Zulip app for Android and iOS. (github.com)
13 upvotes·2 comments·49K views
Avatar of tabbott
Founder at Zulip ·

Zulip has been powered by Django since the very early days of its development with Django 1.4, back in 2012. As a reasonably mature web application with significant scale, we're at the stage in many companies' development where one starts to rip out more and more of the web framework to optimize things or just make them work the way we want. (E.g. while I was at Dropbox in early 2016, we discovered we only had about 600 lines of code left from the original Pylons framework that actually ran).

One of the things that has been really fantastic about Django is that we're still happily using it for the vast majority of code in the project, and every time Django comes out with a new release, I read the changelog and get excited about several improvements that actually make my life better. While Django has made some design decisions that I don't agree with (e.g. I'm not a fan of Django REST framework, and think it makes life more difficult), Django also makes it easy to do your own thing, which we've done to great effect (see the linked article for details on our has_request_variables framework).

Overall I think we've gotten a ton of value out of Python and Django and would recommend it to anyone starting a new full-featured web application project today.

READ MORE
Writing views in Zulip — Zulip 1.9.1+git documentation (zulip.readthedocs.io)
9 upvotes·101.9K views
Avatar of tabbott
Founder at Zulip ·
Shared insights
on
DebianDebianUbuntuUbuntuFedoraFedora
at
()

We use Debian and its derivative Ubuntu because the apt ecosystem and toolchain for Debian packages is far superior to the yum-based system used by Fedora and RHEL. This is large part due to a huge amount of investment into tools like debhelper/dh over the years by the Debian community. I haven't dealt with RPM in the last couple years, but every experience I've had with RPM is that the RPM tools are slower, have less useful options, and it's more work to package software for them (and one makes more compromises in doing so).

I think everyone has seen the better experience using Ubuntu in the shift of prevalence from RHEL to Ubuntu in what most new companies are deploying on their servers, and I expect that trend to continue as long as Red Hat is using the RPM system (and I don't really see them as having a path to migrate).

The experience with Ubuntu and Debian stable releases is pretty similar: A solid release every 2 years that's supported for a few years. (While Ubuntu in theory releases every 6 months, their non-LTS releases are effectively betas: They're often unstable, only have 9 months of support, etc. I wouldn't recommend them to anyone not actively participating in Ubuntu the development community). Ubuntu has better integration of non-free drivers, which may be important if you have hardware that requires them. But it's also the case that most bugs I experience when using Ubuntu are Ubuntu-specific issues, especially on servers (in part because Ubuntu has a bunch of "cloud management" stuff pre-installed that is definitely a regression if you're not using Canonical's cloud management products).

READ MORE
9 upvotes·1 comment·51K views
Avatar of tabbott
Founder at Zulip ·
Shared insights
on
GitGitMercurialMercurial
at

I've been excited about Git ever since it got a built-in UI. It's the perfect combination of a really solid, simple data model, which allows an experienced user to predict precisely what a Git subcommand will do, often without needing to read the documentation (see the slides linked from the attached article for details). Most important to me as the lead developer of a large open source project (Zulip) is that it makes it possible to build a really clean, clear development history that I regularly use to understand details of our code history that are critical to making correct changes.

And it performs really, really well. In 2014, I managed Dropbox's migration from Mercurial to Git. And just switching tools made just about every common operation (git status, git log, git commit etc.) 2-10x faster than with Mercurial. It makes sense if you think about it, since Git was designed to perform well with Linux, one of the largest open source projects out there, but it was still a huge productivity increase that we got basically for free.

If you're learning Git, I highly recommend reading the other sections of Zulip's Git Guide; we get a lot of positive feedback from developers on it being a useful resource even for their projects unrelated to Zulip.

READ MORE
How Git is different — Zulip 1.9.1+git documentation (zulip.readthedocs.io)
9 upvotes·45.6K views