Avatar of Tim Abbott

Tim Abbott

Founder at Zulip

Decision at Zulip about GitLab, GitHub

Avatar of tabbott
Founder at Zulip ·

I have mixed feelings on GitHub as a product and our use of it for the Zulip open source project. On the one hand, I do feel that being on GitHub helps people discover Zulip, because we have enough stars (etc.) that we rank highly among projects on the platform. and there is a definite benefit for lowering barriers to contribution (which is important to us) that GitHub has such a dominant position in terms of what everyone has accounts with.

But even ignoring how one might feel about their new corporate owner (MicroSoft), in a lot of ways GitHub is a bad product for open source projects. Years after the "Dear GitHub" letter, there are still basic gaps in its issue tracker:

  • You can't give someone permission to label/categorize issues without full write access to a project (including ability to merge things to master, post releases, etc.).
  • You can't let anyone with a GitHub account self-assign issues to themselves.
  • Many more similar issues.

It's embarrassing, because I've talked to GitHub product managers at various open source events about these things for 3 years, and they always agree the thing is important, but then nothing ever improves in the Issues product. Maybe the new management at MicroSoft will fix their product management situation, but if not, I imagine we'll eventually do the migration to GitLab.

We have a custom bot project, http://github.com/zulip/zulipbot, to deal with some of these issues where possible, and every other large project we talk to does the same thing, more or less.

13 upvotes·2 comments·106.7K views

Decision at Zulip about Elasticsearch, MySQL, PostgreSQL

Avatar of tabbott
Founder at Zulip ·

We've been using PostgreSQL since the very early days of Zulip, but we actually didn't use it from the beginning. Zulip started out as a MySQL project back in 2012, because we'd heard it was a good choice for a startup with a wide community. However, we found that even though we were using the Django ORM for most of our database access, we spent a lot of time fighting with MySQL. Issues ranged from bad collation defaults, to bad query plans which required a lot of manual query tweaks.

We ended up getting so frustrated that we tried out PostgresQL, and the results were fantastic. We didn't have to do any real customization (just some tuning settings for how big a server we had), and all of our most important queries were faster out of the box. As a result, we were able to delete a bunch of custom queries escaping the ORM that we'd written to make the MySQL query planner happy (because postgres just did the right thing automatically).

And then after that, we've just gotten a ton of value out of postgres. We use its excellent built-in full-text search, which has helped us avoid needing to bring in a tool like Elasticsearch, and we've really enjoyed features like its partial indexes, which saved us a lot of work adding unnecessary extra tables to get good performance for things like our "unread messages" and "starred messages" indexes.

I can't recommend it highly enough.

12 upvotes·49.9K views

Decision at Zulip about CircleCI, Travis CI

Avatar of tabbott
Founder at Zulip ·

We actually started out on Travis CI, but we've migrated our main builds to CircleCI, and it's been a huge improvement.

The reason it's been a huge improvement is that Travis CI has a fundamentally bad design for their images, where they start with a standard base Linux image containing tons of packages (several versions of postgres, every programming language environment, etc). This is potentially nice for the "get builds for a small project running quickly" use case, but it's a total disaster for a larger project that needs a decent number of dependencies and cares about the performance and reliability of their build.

This issue is exacerbated by their networking infrastructure being unreliable; we usually saw over 1% of builds failing due to transient networking errors in Travis CI, even after we added retries to the most frequently failing operations like apt update or pip install. And they never install Ubuntu's point release updates to their images. So doing an apt update, apt install, or especially apt upgrade would take forever. We ended up writing code to actually uninstall many of their base packages and pin the versions of hundreds of others to get a semi-fast, semi-reliable build. It was infuriating.

The CircleCI v2.0 system has the right design for a CI system: we can customize the base image to start with any expensive-to-install packages we need for our build, and we can update that image if and when we want to. The end result is that when migrating, we were able to delete all the hacky optimizations mentioned above, while still ending up with a 50% faster build latency. And we've also had 5-10x fewer issues with networking-related flakes, which means one doesn't have to constantly check whether a build failure is actually due to an issue with the code under test or "just another networking flake".

11 upvotes·16.6K views

Decision at Zulip about Markdown

Avatar of tabbott
Founder at Zulip ·

We've been incredibly happy with the choice of Markdown as the composition language for Zulip:

  • Everyone can write it, since it's just like email and many other products use it.
  • It has a WYSIWYG type experience, so we don't feel the need to integrate a fancy editor like Quill.
  • There are mature markdown processors for every programming language. Having compatible, high-quality implementations for Python and JavaScript is important for doing local echo properly (see our linked Markdown documentation for details of why is necessary).
  • Rendering to HTML is fast, easy to do automated tests for, and easy to debug.

We have had to customize our Markdown implementation somewhat for the chat context, because we don't have the guarantee that content is not split across multiple messages (a good example is that you don't want the "automatically change numbered lists to start at 1" feature in default markdown; you want to restrict that to numbered lists where all the numbers are 1, for example) .

10 upvotes·4.2K views

Decision at Zulip about React Native

Avatar of tabbott
Founder at Zulip ·

We've been using React Native for the Zulip mobile apps, and while there's definitely problems with the platform, it's also saved us a huge amount of time.

One of the things that people don't talk about enough is the product thinking cost of maintaining duplicate codebases for the same app, as one would have with a traditional fully native app for both platforms.

That said, the RN ecosystem has frequently been frustrating; RN releases often break important things, often fairly basic features of the underlying implementations (like support for automatically following redirects in various networking contexts) aren't exposed properly, which can result in a bunch of extra work. But at the same time, we're saving all of the work of maintaining two redundant mobile teams, determining and communicating details of the server/client interface with those two teams, and fixing bugs twice.

Overall, we're happy with the React Native decision, since I think it's the best platform available for what it does in 2018. But I'm also interested in whether Flutter, which in our view has a better development strategy/structure, can provide a better cross-platform mobile development experience in the coming years

9 upvotes·27K views

Decision at Zulip about Fedora, Ubuntu, Debian

Avatar of tabbott
Founder at Zulip ·

We use Debian and its derivative Ubuntu because the apt ecosystem and toolchain for Debian packages is far superior to the yum-based system used by Fedora and RHEL. This is large part due to a huge amount of investment into tools like debhelper/dh over the years by the Debian community. I haven't dealt with RPM in the last couple years, but every experience I've had with RPM is that the RPM tools are slower, have less useful options, and it's more work to package software for them (and one makes more compromises in doing so).

I think everyone has seen the better experience using Ubuntu in the shift of prevalence from RHEL to Ubuntu in what most new companies are deploying on their servers, and I expect that trend to continue as long as Red Hat is using the RPM system (and I don't really see them as having a path to migrate).

The experience with Ubuntu and Debian stable releases is pretty similar: A solid release every 2 years that's supported for a few years. (While Ubuntu in theory releases every 6 months, their non-LTS releases are effectively betas: They're often unstable, only have 9 months of support, etc. I wouldn't recommend them to anyone not actively participating in Ubuntu the development community). Ubuntu has better integration of non-free drivers, which may be important if you have hardware that requires them. But it's also the case that most bugs I experience when using Ubuntu are Ubuntu-specific issues, especially on servers (in part because Ubuntu has a bunch of "cloud management" stuff pre-installed that is definitely a regression if you're not using Canonical's cloud management products).

8 upvotes·1 comment·4K views

Decision at Zulip about Redis, Python, RabbitMQ

Avatar of tabbott
Founder at Zulip ·

We've been using RabbitMQ as Zulip's queuing system since we needed a queuing system. What I like about it is that it scales really well and has good libraries for a wide range of platforms, including our own Python. So aside from getting it running, we've had to put basically 0 effort into making it scale for our needs.

However, there's several things that could be better about it: * It's error messages are absolutely terrible; if ever one of our users ends up getting an error with RabbitMQ (even for simple things like a misconfigured hostname), they always end up needing to get help from the Zulip team, because the errors logs are just inscrutable. As an open source project, we've handled this issue by really carefully scripting the installation to be a failure-proof configuration (in this case, setting the RabbitMQ hostname to, so that no user-controlled configuration can break it). But it was a real pain to get there and the process of determining we needed to do that caused a significant amount of pain to folks installing Zulip. * The pika library for Python takes a lot of time to startup a RabbitMQ connection; this means that Zulip server restarts are more disruptive than would be ideal. * It's annoying that you need to run the rabbitmqctl management commands as root.

But overall, I like that it has clean, clear semanstics and high scalability, and haven't been tempted to do the work to migrate to something like Redis (which has its own downsides).

6 upvotes·23.1K views

Decision at Zulip about Mercurial, Git

Avatar of tabbott
Founder at Zulip ·

I've been excited about Git ever since it got a built-in UI. It's the perfect combination of a really solid, simple data model, which allows an experienced user to predict precisely what a Git subcommand will do, often without needing to read the documentation (see the slides linked from the attached article for details). Most important to me as the lead developer of a large open source project (Zulip) is that it makes it possible to build a really clean, clear development history that I regularly use to understand details of our code history that are critical to making correct changes.

And it performs really, really well. In 2014, I managed Dropbox's migration from Mercurial to Git. And just switching tools made just about every common operation (git status, git log, git commit etc.) 2-10x faster than with Mercurial. It makes sense if you think about it, since Git was designed to perform well with Linux, one of the largest open source projects out there, but it was still a huge productivity increase that we got basically for free.

If you're learning Git, I highly recommend reading the other sections of Zulip's Git Guide; we get a lot of positive feedback from developers on it being a useful resource even for their projects unrelated to Zulip.

6 upvotes·12.4K views

Decision at Zulip about Go, Python

Avatar of tabbott
Founder at Zulip ·

We've been a big fan of Python ever since we adopted it for my first startup, Ksplice. But it's been an absolutely ideal tool for Zulip, which is now one of the leading alternatives to Slack. Zulip is 100% open source software, with ~10K stars on GItHub. And being written in idiomatic Python has been really helpful for our open source project, because it's such an accessible language: Any programmer can learn Python quickly. And that means we're not restricted to e.g. "folks who are excited about contributing to Zulip and ALSO know Go".

I've linked to a blog post I wrote on Python's awesome new static type system, which fixes the main complaint one might have about using Python for a large codebase, which has a lot more perspective, as well as some commentary on our Python 3 migration.

6 upvotes·9.1K views

Decision at Zulip about Webpack

Avatar of tabbott
Founder at Zulip ·

We use Webpack because it's the standard toolchain for managing frontend dependencies in 2019, and it's hard to make a nice frontend development user experience without it.

I don't like it -- their configuration system is a mess, requiring a ton of reading or expertise to do things that essentially every project wants to do by default. It has a lot of great features, which is why we use it. But as an example, it's development server hot reloading is really cool, but doesn't handle changes in the webpack configuration file itself (so adding a new file requires a restart).

My hope is that the sheer fact that everyone is using it will eventually lead to these problems being fixed or it being replaced by a similar system with a better design.

5 upvotes·9.2K views