Ruby

For Stack Decisions I needed to add Markdown in the decision composer to give our users access to some general styling when writing their decisions. We used React & GraphQL on the #Frontend and Ruby & GraphQL on the backend.

Instead of using Showdown or another tool, We decided to parse the Markdown on the backend so we had more control over what we wanted to render in Markdown because we didn't want to enable all Markdown options, we also wanted to limit any malicious code or images to be embedded into the decisions and Markdown was a fairly large to import into our component so it was going to add a lot of kilobytes that we didn't need.

We also needed to style how the markdown looked, we are currently using Glamorous so I used that but we are planning to update this to Emotion at some stage as it has a fairly easy upgrade path rather than switching over to styled-components or one of the other cssInJs alternatives.

Also we used React-Mentions for tagging tools and topics in the decisions. Typing @ will let you tag a tool, and typing # will allow you to tag a topic.

The Markdown options that we chose to support are tags: a, code, u, b, em, pre, ul, ol, li.

If there are anymore tags you'd love to see added in the composer leave me a comment below and we will look into adding them.

#StackDecisionsLaunch

#BackendsForFrontends #api #Frontend #backend

When we began to modernize our company's main software, we need to choose an architecture that could be as stable as it was for the last 20 years and at the same time, allow us to use one of the best alternatives to build an highly interactive and complex front-end.

So, to embrace the innovation while keeping an solid foundation, we decided to work with our main web framework Rails, with the React front-end, an amazing feature of one of Rails more recent versions.

The Ruby (and Rails) are our main programming language + framework for about 6 years to modernize our almost 20 years old legacy system. We already have a lot of interesting tools working together, like MongoDB, Redis, RabbitMQ, PostgreSQL etc. All of them with fantastic libraries/support for Ruby.

Besides that, our front-end team is growing fast, and wants to apply what they've been learning (in theory and practice) everyday with React .

But we don't want it to become a new monster after some months, when we passed the mainly features of our app to this new project, so we think: How we could use this new features? Which are the architecture that most adapt to our needs?

Jekyll is an open source static site generator (SSG) with a Ruby at its core which transform your plain text into static websites and blogs.

It is simple means no more databases, comment moderation, or pesky updates to install—just your content. As said earlier SSG uses Markdown, Liquid, HTML & CSS go in and come out ready for deployment. Lastly it's blog-aware permalinks, categories, pages, posts, and custom layouts are all first-class citizens here.

For stack decisions, I needed a non-sequential ID format to prevent users from guessing other IDs. My options were:

• UUIDs v4 that look like this: 496a52cd-49ba-4424-99d9-344e44803cb1
• Hashids that look like this: xpAYDx0m
• Flake IDs whose Mastodon Ruby implementation looks like this: 101151084044583231

I eventually chose flake IDs, because IMO they are better-looking and easier to type.

Although they are meant for distributed systems at scale (think Twitter), for my feature I only cared about how nice they looked. As a bonus, because the first few bits are time-based, they "feel" like good old incremental IDs. #StackDecisionsLaunch

Lumosity is home to the world's largest cognitive training database, a responsibility we take seriously. For most of the company's history, our analysis of user behavior and training data has been powered by an event stream--first a simple Node.js pub/sub app, then a heavyweight Ruby app with stronger durability. Both supported decent throughput and latency, but they lacked some major features supported by existing open-source alternatives: replaying existing messages (also lacking in most message queue-based solutions), scaling out many different readers for the same stream, the ability to leverage existing solutions for reading and writing, and possibly most importantly: the ability to hire someone externally who already had expertise.

We ultimately migrated to Kafka in early- to mid-2016, citing both industry trends in companies we'd talked to with similar durability and throughput needs, the extremely strong documentation and community. We pored over Kyle Kingsbury's Jepsen post (https://aphyr.com/posts/293-jepsen-Kafka), as well as Jay Kreps' follow-up (http://blog.empathybox.com/post/62279088548/a-few-notes-on-kafka-and-jepsen), talked at length with Confluent folks and community members, and still wound up running parallel systems for quite a long time, but ultimately, we've been very, very happy. Understanding the internals and proper levers takes some commitment, but it's taken very little maintenance once configured. Since then, the Confluent Platform community has grown and grown; we've gone from doing most development using custom Scala consumers and producers to being 60/40 Kafka Streams/Connects.

We originally looked into Storm / Heron , and we'd moved on from Redis pub/sub. Heron looks great, but we already had a programming model across services that was more akin to consuming a message consumers than required a topology of bolts, etc. Heron also had just come out while we were starting to migrate things, and the community momentum and direction of Kafka felt more substantial than the older Storm. If we were to start the process over again today, we might check out Pulsar , although the ecosystem is much younger.

To find out more, read our 2017 engineering blog post about the migration!

I chose Sqreen because it provides an out-of-the-box Security as a Service solution to protect my customer data. I get full visibility over my application security in real-time and I reduce my risk against the most common threats. My customers are happy and I don't need to spend any engineering resources or time on this. We're only alerted when our attention is required and the data that is provided helps engineering teams easily remediate vulnerabilities. The platform grows with us and will allow us to have all the right tools in place when our first security engineer joins the company. Advanced security protections against business logic threats can then be implemented.

Installation was super easy on my Node.js and Ruby apps. But Sqreen also supports Python , Java , PHP and soon Go .

It integrates well with the tools I'm using every day Slack , PagerDuty and more.

Ruby NLP C++ Grammar #BNF

At FriendlyData we had a Ruby-based pipeline for natural language processing. Our technology is centered around grammar-based natural language parsing, as well as various product features, and, as the core stack of the company historically is Ruby, the initial version of the pipeline was implemented in Ruby as well.

As we were entering the exponential growth phase, both technology- and product-wise, we looked into how could we speed up and extend the performance and flexibility of our [meta-]BNF-based parsing engine. Gradually, we built the pieces of the engine in C++.

Ultimately, the natural language parsing stack spans three universes and three software engineering paradigms: the declarative one, the functional one, and the imperative one. The imperative one was and remains implemented in Ruby, the functional one is implemented in a functional language (this part is under the NDA, while everything I am talking about here is part of the public talks we gave throughout 2017 and 2018), and the declarative part, which can loosely be thought of as being BNF-based, is now served by the C++ engine.

The C++ engine for the BNF part removed the immediate blockers, gave us 500x+ performance speedup, and enabled us to launch new product features, most notably query completions, suggestions, and spelling corrections.

I liked Sublime Text for its speed, simplicity and keyboard shortcuts which synergize well when working on scripting languages like Ruby and JavaScript. I extended the editor with custom Python scripts that improved keyboard navigability such as autofocusing the sidebar when no files are open, or changing tab closing behavior.

But customization can only get you so far, and there were little things that I still had to use the mouse for, such as scrolling, repositioning lines on the screen, selecting the line number of a failing test stack trace from a separate plugin pane, etc. After 3 years of wearily moving my arm and hand to perform the same repetitive tasks, I decided to switch to Vim for 3 reasons:

• your fingers literally don’t ever need to leave the keyboard home row (I had to remap the escape key though)
• it is a reliable tool that has been around for more than 30 years and will still be around for the next 30 years
• I wanted to "look like a hacker" by doing everything inside my terminal and by becoming a better Unix citizen

The learning curve is very steep and it took me a year to master it, but investing time to be truly comfortable with my #TextEditor was more than worth it. To me, Vim comes close to being the perfect editor and I probably won’t need to switch ever again. It feels good to ignore new editors that come out every few years, like Atom and Visual Studio Code.

delayed_job is a great Rails background job library for new projects, as it only uses what you already have: a relational database. We happily used it during the company’s first two years.

But it started to falter as our web and database transactions significantly grew. Our app interacted with users via SMS texts sent inside background jobs. Because the delayed_job daemon ran every couple seconds, this meant that users often waited several long seconds before getting text replies, which was not acceptable. Moreover, job processing was done inside AWS Elastic Beanstalk web instances, which were already under stress and not meant to handle jobs.

We needed a fast background job system that could process jobs in near real-time and integrate well with AWS. Sidekiq is a fast and popular Ruby background job library, but it does not leverage the Elastic Beanstalk worker architecture, and you have to maintain a Redis instance.

We ended up choosing active-elastic-job, which seamlessly integrates with worker instances and Amazon SQS. SQS is a fast queue and you don’t need to worry about infrastructure or scaling, as AWS handles it for you.

We noticed significant performance gains immediately after making the switch.

#BackgroundProcessing

In 2004, Shopify’s CEO and founder, Tobi Lütke, was building out an e-commerce store for snowboarding products. Unsatisfied with the existing e-commerce products on the market, Tobi decided to build his own SaaS platform using Ruby on Rails.

At that time, Rails wasn't even 1.0 yet, and the only version of the framework was exchanged as a .zip archive by email. Tobi joined Rails creator David Heinemeier Hansson (DHH) and started contributing to Ruby on Rails while building Shopify.

Shopify is now one of the world's largest and oldest Rails apps. It’s never been rewritten and still uses the original codebase, though it has matured considerably over the past decade. All of Tobi’s original commits are still in the version control history.

The bet on Rails greatly shaped how we think at Shopify and empowered us to deliver product as fast as possible. While there are parts of the framework that sometimes make it harder to scale (e.g. ActiveRecord callbacks and code organization), many of us tend to agree with Tobi that Rails is what allowed Shopify to move from a garage startup to a public company.