Alternatives to Amazon WorkSpaces

To provide employees with the critical need of interactive querying, we’ve worked with Presto, an open-source distributed SQL query engine, over the years. Operating Presto at Pinterest’s scale has involved resolving quite a few challenges like, supporting deeply nested and huge thrift schemas, slow/ bad worker detection and remediation, auto-scaling cluster, graceful cluster shutdown and impersonation support for ldap authenticator.

Our infrastructure is built on top of Amazon EC2 and we leverage Amazon S3 for storing our data. This separates compute and storage layers, and allows multiple compute clusters to share the S3 data.

We have hundreds of petabytes of data and tens of thousands of Apache Hive tables. Our Presto clusters are comprised of a fleet of 450 r4.8xl EC2 instances. Presto clusters together have over 100 TBs of memory and 14K vcpu cores. Within Pinterest, we have close to more than 1,000 monthly active users (out of total 1,600+ Pinterest employees) using Presto, who run about 400K queries on these clusters per month.

Each query submitted to Presto cluster is logged to a Kafka topic via Singer. Singer is a logging agent built at Pinterest and we talked about it in a previous post. Each query is logged when it is submitted and when it finishes. When a Presto cluster crashes, we will have query submitted events without corresponding query finished events. These events enable us to capture the effect of cluster crashes over time.

Each Presto cluster at Pinterest has workers on a mix of dedicated AWS EC2 instances and Kubernetes pods. Kubernetes platform provides us with the capability to add and remove workers from a Presto cluster very quickly. The best-case latency on bringing up a new worker on Kubernetes is less than a minute. However, when the Kubernetes cluster itself is out of resources and needs to scale up, it can take up to ten minutes. Some other advantages of deploying on Kubernetes platform is that our Presto deployment becomes agnostic of cloud vendor, instance types, OS, etc.

#BigData #AWS #DataScience #DataEngineering

Our whole DevOps stack consists of the following tools:

• GitHub (incl. GitHub Pages/Markdown for Documentation, GettingStarted and HowTo's) for collaborative review and code management tool
• Respectively Git as revision control system
• SourceTree as Git GUI
• Visual Studio Code as IDE
• CircleCI for continuous integration (automatize development process)
• Prettier / TSLint / ESLint as code linter
• SonarQube as quality gate
• Docker as container management (incl. Docker Compose for multi-container application management)
• VirtualBox for operating system simulation tests
• Kubernetes as cluster management for docker containers
• Heroku for deploying in test environments
• nginx as web server (preferably used as facade server in production environment)
• SSLMate (using OpenSSL) for certificate management
• Amazon EC2 (incl. Amazon S3) for deploying in stage (production-like) and production environments
• PostgreSQL as preferred database system
• Redis as preferred in-memory database/store (great for caching)

The main reason we have chosen Kubernetes over Docker Swarm is related to the following artifacts:

• Key features: Easy and flexible installation, Clear dashboard, Great scaling operations, Monitoring is an integral part, Great load balancing concepts, Monitors the condition and ensures compensation in the event of failure.
• Applications: An application can be deployed using a combination of pods, deployments, and services (or micro-services).
• Functionality: Kubernetes as a complex installation and setup process, but it not as limited as Docker Swarm.
• Monitoring: It supports multiple versions of logging and monitoring when the services are deployed within the cluster (Elasticsearch/Kibana (ELK), Heapster/Grafana, Sysdig cloud integration).
• Scalability: All-in-one framework for distributed systems.
• Other Benefits: Kubernetes is backed by the Cloud Native Computing Foundation (CNCF), huge community among container orchestration tools, it is an open source and modular tool that works with any OS.

Sentry has been essential to our development approach. Nobody likes errors or apps that crash. We use Sentry heavily during Node.js and React development. Our developers are able to see error reports, crashes, user's browsers, and more, all in one place. Sentry also seamlessly integrates with Asana, Slack, and GitHub.

Last time we shared there information about our decision about using YouTrack over Jira actually we found much better solution that our team have loved. Linear is a minimalistic issue tracker that integrates well with Sentry, GitHub, Slack and Figma which are our basic tools. I would like to recommend checking out Linear as a potential alternative to "heavy" issue trackers, maybe at enterprises that may not work but when we're a startup that works awesome!

So I am a huge fan of JIRA like #massive I used it for many many years, and really loved it, used it personally and at work. I would suggest every new workplace that I worked at to switch to JIRA instead of what I was using.

When I started at #StackShare we were using a Trello #Kanban board and I was so shocked at how easy the workflow was to follow, create new tasks and get tasks QA'd and deployed. What was so great about this was it didn't come with all the complexity of JIRA. Like setting up a project, user rules etc. You are able to hit the ground running with Trello and get tasks started right away without being overwhelmed with the complexity of options in JIRA

With a few TrelloPowerUps we were easily able to add GitHub integration and storyPoints to our cards and thats all we needed to get a really nice agile workflow going.

I'm not saying that JIRA is not useful, I can see larger companies being able to use the JIRA features and have the time to go through all the complex setup to get a really good workflow going. But for smaller #Startups that want to hit the ground running Trello for me is the way to go.

In saying that what I would love Trello to implement is to allow me to create custom fields. Right now we just have a Description field. So I am adding User Stories & How To Test in the Markdown of the Description if I could have these as custom fields then my #Agile workflow would be complete.

#StackDecisionsLaunch

Last time we shared there information about our decision about using YouTrack over Jira actually we found much better solution that our team have loved. Linear is a minimalistic issue tracker that integrates well with Sentry, GitHub, Slack and Figma which are our basic tools. I would like to recommend checking out Linear as a potential alternative to "heavy" issue trackers, maybe at enterprises that may not work but when we're a startup that works awesome!

So I am a huge fan of JIRA like #massive I used it for many many years, and really loved it, used it personally and at work. I would suggest every new workplace that I worked at to switch to JIRA instead of what I was using.

When I started at #StackShare we were using a Trello #Kanban board and I was so shocked at how easy the workflow was to follow, create new tasks and get tasks QA'd and deployed. What was so great about this was it didn't come with all the complexity of JIRA. Like setting up a project, user rules etc. You are able to hit the ground running with Trello and get tasks started right away without being overwhelmed with the complexity of options in JIRA

With a few TrelloPowerUps we were easily able to add GitHub integration and storyPoints to our cards and thats all we needed to get a really nice agile workflow going.

I'm not saying that JIRA is not useful, I can see larger companies being able to use the JIRA features and have the time to go through all the complex setup to get a really good workflow going. But for smaller #Startups that want to hit the ground running Trello for me is the way to go.

In saying that what I would love Trello to implement is to allow me to create custom fields. Right now we just have a Description field. So I am adding User Stories & How To Test in the Markdown of the Description if I could have these as custom fields then my #Agile workflow would be complete.

#StackDecisionsLaunch

For Etom, a side project. We wanted to test an idea for a future and bigger project.

What Etom does is searching places. Right now, it leverages the Google Maps API. For that, we found a React component that makes this integration easy because using Google Maps API is not possible via normal API requests.

You kind of need a map to work as a proxy between the software and Google Maps API.

We hate configuration(coming from Rails world) so also decided to use Create React App because setting up a React app, with all the toys, it's a hard job.

Thanks to all the people behind Create React App it's easier to start any React application.

We also chose a module called Reactstrap which is Bootstrap UI in React components.

An important thing in this side project(and in the bigger project plan) is to measure visitor through out the app. For that we researched and found that Keen was a good choice(very good free tier limits) and also it is very simple to setup and real simple to send data to

Slack and Trello are our defaults tools to comunicate ideas and discuss topics, so, no brainer using them as well for this project.

Using Screenhero via Slack was getting to be pretty horrible. Video and sound quality was often times pretty bad and worst of all the service just wasn't reliable. We all had high hopes when the acquisition went through but ultimately, the product just didn't live up to expectations. We ended up trying Zoom after I had heard about it from some friends at other companies. We noticed the video/sound quality was better, and more importantly it was super reliable. The Slack integration was awesome (just type /zoom and it starts a call)

You can schedule recurring calls which is helpful. There's a G Suite (Google Calendar) integration which lets you add a Zoom call (w/dial in info + link to web/mobile) with the click of a button.

Meeting recordings (video and audio) are really nice, you get recordings stored in the cloud on the higher tier plans. One of our engineers, Jerome, actually built a cool little Slack integration using the Slack API and Zoom API so that every time a recording is processed, a link gets posted to the "event-recordings" channel. The iOS app is great too!

#WebAndVideoConferencing #videochat

We are highly dependent on G Suite for all our collaboration and productivity needs, from Gmail and Calendar to Sheets and Docs. While it may not be as robust as Microsoft's offerings in those areas, it's totally cloud-based, we've never had any downtime issues and it integrates well with our other tools like Slack. We write and collaborate on all our specs/PRDs in Docs, share analyses via Sheets and handle our meetings via Calendar. #StackDecisionsLaunch #ProductivitySuite #Collaboration #DocumentCollaboration

We knew how we wanted to build our Design System, now it was time to choose the tools to get us there. The essence of Scrum is a small team of people. The team is highly flexible and adaptive. Perfect, so we'll work in 2 week sprints where each sprint can be a mix of new R&D stories, a presentation of decisions made, and showcasing key development milestones.

We are also able to run content stories in parallel, focusing development efforts around key areas of the site that our authors need first. Our stories would exist in a Jira backlog, documentation would be hosted in Confluence , and GitHub would host our codebase. If developers identify technical improvements during the sprint, they can be added as GitHub issues and transferred to Jira if we decide to represent them as stories for the Backlog. For Sprint Retrospectives, @groupmap proved to be a great way to include our remote members of the dev team.

This worked well for our team and allowed us to be flexible in what we wanted to build and how we wanted to build it. As we further defined our Backlog and estimated each story, we could accurately measure the team's capacity (velocity) and confidently estimate a launch date.

As a new company we could early adopt and bet on #RemoteTeam setup without cultural baggage derailing us. Our building blocks for developing remote working culture are:

• Hiring people who are self sufficient, self-disciplined and excel at video and written communication to work remotely
• Set up periodic ceremonies ( #DailyStandup, #Grooming, Release calls and chats etc) to keep the company rhythm / heartbeat going across remote cells
• Regularly train your leaders to take into account remote working aspects of organizing f2f calls, events, meetups, parties etc. when communicating and organizing workflows
• And last, but not least - select the right tools to support effective communication and collaboration:

1. All feeds and conversations come together in Slack
2. #Agile workflows in Jira
3. InProductCommunication and #CustomerSupportChat in Intercom
4. #Notes, #Documentation and #Requirements in Confluence
5. #SourceCode and ContinuousDelivery in Bitbucket
6. Persistent video streams between locations, demos, meetings run on appear.in
7. #Logging and Alerts in Papertrail

Uploadcare is mostly remote team and we're using video conferencing all the time both for internal team meetings and for external sales, support, interview, etc. calls. I think we've tried every solution there is on the market before we've decided to stop with Zoom.

Tools just plainly don't work (Skype), are painful to install for external participants (Webex and other "enterprise" solutions) can't properly handle 10+ participants calls (Google Hangouts Chat).

Zoom just works. It has all required features and even handles bad connections very graciously. One of the best tool decisions we've ever made :)

I use Slack because it offers the best experience, even on the free tier (which we're still using). As a comparison, I have had in depth experience with HipChat, Stride, Skype, Google Chat (the new service), Google Hangouts (the old service). For self hosted, Mattermost is open source and claims to support most Slack integrations, but I have not extensively investigated this claim.