Feed powered byStream Blue Logo Copy 5Created with Sketch.
Stream

Decision at Stream about Go, Stream, Python, Yarn, Babel, Node.js, ES6, JavaScript, Languages, FrameworksFullStack

Avatar of nparsons08
Node.js Engineer & Evangelist at Stream ·
GoGo
StreamStream
PythonPython
YarnYarn
BabelBabel
Node.jsNode.js
ES6ES6
JavaScriptJavaScript
#Languages
#FrameworksFullStack

Winds 2.0 is an open source Podcast/RSS reader developed by Stream with a core goal to enable a wide range of developers to contribute.

We chose JavaScript because nearly every developer knows or can, at the very least, read JavaScript. With ES6 and Node.js v10.x.x, it’s become a very capable language. Async/Await is powerful and easy to use (Async/Await vs Promises). Babel allows us to experiment with next-generation JavaScript (features that are not in the official JavaScript spec yet). Yarn allows us to consistently install packages quickly (and is filled with tons of new tricks)

We’re using JavaScript for everything – both front and backend. Most of our team is experienced with Go and Python, so Node was not an obvious choice for this app.

Sure... there will be haters who refuse to acknowledge that there is anything remotely positive about JavaScript (there are even rants on Hacker News about Node.js); however, without writing completely in JavaScript, we would not have seen the results we did.

#FrameworksFullStack #Languages

26 upvotes·1.4K views

Decision at Stream about Electron, CrossPlatformDesktopDevelopment

Avatar of nparsons08
Node.js Engineer & Evangelist at Stream ·
ElectronElectron
#CrossPlatformDesktopDevelopment

We wanted to experiment with building an Electron app with downloads for every Linux distro, macOS, and Windows, in addition to the web. Fundamentally, this seemed pretty easy: we write code, wrap it in an Electron shell, and release to our desired operating system… It turns out we were wrong.

Electron, though powerful, turned out to be a bigger beast than we had anticipated. Building to different distros was especially hard, even with electron-builder (granted, we had the bad luck of needing to patch electron-builder (and that bug has since been fixed), but that only accounted for some of the pain points we hit). The macOS menu bar had to be just right for the macOS store to accept our app, and performing small tasks with the Electron API, like opening a link in an external browser, turned out to be pretty difficult.

Despite the difficulties, our team moved forward with some custom tooling (all visible and open-sourced on Github) and we released not only to all of our release targets but to the web, too.

#CrossPlatformDesktopDevelopment

24 upvotes·122 views

Decision at Stream about Go, Jenkins, GitHub, Travis CI, CodeCollaborationVersionControl, ContinuousIntegration

Avatar of tschellenbach
GoGo
JenkinsJenkins
GitHubGitHub
Travis CITravis CI
#CodeCollaborationVersionControl
#ContinuousIntegration

Releasing new versions of our services is done by Travis CI. Travis first runs our test suite. Once it passes, it publishes a new release binary to GitHub.

Common tasks such as installing dependencies for the Go project, or building a binary are automated using plain old Makefiles. (We know, crazy old school, right?) Our binaries are compressed using UPX.

Travis has come a long way over the past years. I used to prefer Jenkins in some cases since it was easier to debug broken builds. With the addition of the aptly named “debug build” button, Travis is now the clear winner. It’s easy to use and free for open source, with no need to maintain anything.

#ContinuousIntegration #CodeCollaborationVersionControl

22 upvotes·1 comment·977 views

Decision at Stream about Go, Faye, Redis, Languages, InMemoryDatabases, ApplicationHosting, DataStores, RealtimeBackendApi

Avatar of tschellenbach
GoGo
FayeFaye
RedisRedis
#Languages
#InMemoryDatabases
#ApplicationHosting
#DataStores
#RealtimeBackendApi

Our real time infrastructure is based on Go , Redis and the excellent gorilla websocket library. It implements the Bayeux protocol.

In terms of architecture it’s very similar to the node based Faye library. It was interesting to read the “Ditching Go for Node.js” post on Hacker News. The author moves from Go to Node to improve performance. We actually did the exact opposite and moved from Node to Go for our real time system. The new Go-based infrastructure handles 8x the traffic per node. #InMemoryDatabases #RealtimeBackendApi #ApplicationHosting #Languages #DataStores

22 upvotes·366 views

Decision at Stream about AWS CloudFormation, BuildTestDeploy, InfrastructureBuildTools

Avatar of tschellenbach
AWS CloudFormationAWS CloudFormation
#BuildTestDeploy
#InfrastructureBuildTools

We heavily use AWS CloudFormation. Every single piece of our stack is defined in a CloudFormation template. Because our infrastructure is defined in code it has become trivial to launch new regions. If needed we are able to spawn a new dedicated shard in a few minutes. In addition, AWS Parameter Store is used to hold application settings. Our largest deployment is in US-East, but we also have regions in Tokyo, Singapore and Dublin.

#InfrastructureBuildTools #BuildTestDeploy

22 upvotes·65 views

Decision at Stream about Go, Cassandra, Python, Databases, DataStores

Avatar of tschellenbach
GoGo
CassandraCassandra
PythonPython
#Databases
#DataStores

After years of optimizing our existing feed technology, we decided to make a larger leap with 2.0 of Stream. While the first iteration of Stream was powered by Python and Cassandra, for Stream 2.0 of our infrastructure we switched to Go.

The main reason why we switched from Python to Go is performance. Certain features of Stream such as aggregation, ranking and serialization were very difficult to speed up using Python.

We’ve been using Go since March 2017 and it’s been a great experience so far. Go has greatly increased the productivity of our development team. Not only has it improved the speed at which we develop, it’s also 30x faster for many components of Stream. Initially we struggled a bit with package management for Go. However, using Dep together with the VG package contributed to creating a great workflow.

Go as a language is heavily focused on performance. The built-in PPROF tool is amazing for finding performance issues. Uber’s Go-Torch library is great for visualizing data from PPROF and will be bundled in PPROF in Go 1.10.

The performance of Go greatly influenced our architecture in a positive way. With Python we often found ourselves delegating logic to the database layer purely for performance reasons. The high performance of Go gave us more flexibility in terms of architecture. This led to a huge simplification of our infrastructure and a dramatic improvement of latency. For instance, we saw a 10 to 1 reduction in web-server count thanks to the lower memory and CPU usage for the same number of requests.

#DataStores #Databases

19 upvotes·583 views

Decision at Stream about Slack, VictorOps, Collaboration, Monitoring, MonitoringAggregation, GroupChatNotifications

Avatar of tschellenbach
SlackSlack
VictorOpsVictorOps
#Collaboration
#Monitoring
#MonitoringAggregation
#GroupChatNotifications

VictorOps is a recent addition to our support stack. The best part about VictorOps is how they use a timeline to collaborate amongst team members. VictorOps is an elegant way to keep our team in the loop about outages. It also integrates well with Slack. This setup enables us to quickly react to any problems that make it into production, work together and resolve them faster. #Collaboration #MonitoringAggregation #Monitoring #GroupChatNotifications

18 upvotes·501 views

Decision at Stream about RocksDB, Cassandra, Redis, Databases, DataStores, InMemoryDatabases

Avatar of tschellenbach
RocksDBRocksDB
CassandraCassandra
RedisRedis
#Databases
#DataStores
#InMemoryDatabases

1.0 of Stream leveraged Cassandra for storing the feed. Cassandra is a common choice for building feeds. Instagram, for instance started, out with Redis but eventually switched to Cassandra to handle their rapid usage growth. Cassandra can handle write heavy workloads very efficiently.

Cassandra is a great tool that allows you to scale write capacity simply by adding more nodes, though it is also very complex. This complexity made it hard to diagnose performance fluctuations. Even though we had years of experience with running Cassandra, it still felt like a bit of a black box. When building Stream 2.0 we decided to go for a different approach and build Keevo. Keevo is our in-house key-value store built upon RocksDB, gRPC and Raft.

RocksDB is a highly performant embeddable database library developed and maintained by Facebook’s data engineering team. RocksDB started as a fork of Google’s LevelDB that introduced several performance improvements for SSD. Nowadays RocksDB is a project on its own and is under active development. It is written in C++ and it’s fast. Have a look at how this benchmark handles 7 million QPS. In terms of technology it’s much more simple than Cassandra.

This translates into reduced maintenance overhead, improved performance and, most importantly, more consistent performance. It’s interesting to note that LinkedIn also uses RocksDB for their feed.

#InMemoryDatabases #DataStores #Databases

17 upvotes·218 views