Alternatives to Azure Storage

To provide employees with the critical need of interactive querying, we’ve worked with Presto, an open-source distributed SQL query engine, over the years. Operating Presto at Pinterest’s scale has involved resolving quite a few challenges like, supporting deeply nested and huge thrift schemas, slow/ bad worker detection and remediation, auto-scaling cluster, graceful cluster shutdown and impersonation support for ldap authenticator.

Our infrastructure is built on top of Amazon EC2 and we leverage Amazon S3 for storing our data. This separates compute and storage layers, and allows multiple compute clusters to share the S3 data.

We have hundreds of petabytes of data and tens of thousands of Apache Hive tables. Our Presto clusters are comprised of a fleet of 450 r4.8xl EC2 instances. Presto clusters together have over 100 TBs of memory and 14K vcpu cores. Within Pinterest, we have close to more than 1,000 monthly active users (out of total 1,600+ Pinterest employees) using Presto, who run about 400K queries on these clusters per month.

Each query submitted to Presto cluster is logged to a Kafka topic via Singer. Singer is a logging agent built at Pinterest and we talked about it in a previous post. Each query is logged when it is submitted and when it finishes. When a Presto cluster crashes, we will have query submitted events without corresponding query finished events. These events enable us to capture the effect of cluster crashes over time.

Each Presto cluster at Pinterest has workers on a mix of dedicated AWS EC2 instances and Kubernetes pods. Kubernetes platform provides us with the capability to add and remove workers from a Presto cluster very quickly. The best-case latency on bringing up a new worker on Kubernetes is less than a minute. However, when the Kubernetes cluster itself is out of resources and needs to scale up, it can take up to ten minutes. Some other advantages of deploying on Kubernetes platform is that our Presto deployment becomes agnostic of cloud vendor, instance types, OS, etc.

#BigData #AWS #DataScience #DataEngineering

Our whole DevOps stack consists of the following tools:

• GitHub (incl. GitHub Pages/Markdown for Documentation, GettingStarted and HowTo's) for collaborative review and code management tool
• Respectively Git as revision control system
• SourceTree as Git GUI
• Visual Studio Code as IDE
• CircleCI for continuous integration (automatize development process)
• Prettier / TSLint / ESLint as code linter
• SonarQube as quality gate
• Docker as container management (incl. Docker Compose for multi-container application management)
• VirtualBox for operating system simulation tests
• Kubernetes as cluster management for docker containers
• Heroku for deploying in test environments
• nginx as web server (preferably used as facade server in production environment)
• SSLMate (using OpenSSL) for certificate management
• Amazon EC2 (incl. Amazon S3) for deploying in stage (production-like) and production environments
• PostgreSQL as preferred database system
• Redis as preferred in-memory database/store (great for caching)

The main reason we have chosen Kubernetes over Docker Swarm is related to the following artifacts:

• Key features: Easy and flexible installation, Clear dashboard, Great scaling operations, Monitoring is an integral part, Great load balancing concepts, Monitors the condition and ensures compensation in the event of failure.
• Applications: An application can be deployed using a combination of pods, deployments, and services (or micro-services).
• Functionality: Kubernetes as a complex installation and setup process, but it not as limited as Docker Swarm.
• Monitoring: It supports multiple versions of logging and monitoring when the services are deployed within the cluster (Elasticsearch/Kibana (ELK), Heapster/Grafana, Sysdig cloud integration).
• Scalability: All-in-one framework for distributed systems.
• Other Benefits: Kubernetes is backed by the Cloud Native Computing Foundation (CNCF), huge community among container orchestration tools, it is an open source and modular tool that works with any OS.

Google Maps lets "property owners and their authorized representatives" upload indoor maps, but this appears to lack navigation ("wayfinding").

MappedIn is a platform and has SDKs for building indoor mapping experiences (https://www.mappedin.com/) and ESRI ArcGIS also offers some indoor mapping tools (https://www.esri.com/en-us/arcgis/indoor-gis/overview). Finally, there used to be a company called LocusLabs that is now a part of Atrius and they were often integrated into airlines' apps to provide airport maps with wayfinding (https://atrius.com/solutions/personal-experiences/personal-wayfinder/).

I previously worked at Mapbox and while I believe that it's a great platform for building map-based experiences, they don't have any simple solutions for indoor wayfinding. If I were doing this for fun as a side-project and prioritized saving money over saving time, here is what I would do:

• Create a graph-based dataset representing the walking paths around your university, where nodes/vertexes represent the intersections of paths, and edges represent paths (literally paths outside, hallways, short path segments that represent entering rooms). You could store this in a hosted graph-based database like Neo4j, Amazon Neptune , or Azure Cosmos DB (with its Gremlin API) and use built-in "shortest path" queries, or deploy a PostgreSQL service with pgRouting.

• Add two properties to each edge: one property for the distance between its nodes (libraries like @turf/helpers will have a distance function if you have the latitude & longitude of each node), and another property estimating the walking time (based on the distance). Once you have these values saved in a graph-based format, you should be able to easily query and find the data representation of paths between two points.

• At this point, you'd have the routing problem solved and it would come down to building a UI. Mapbox arguably leads the industry in developer tools for custom map experiences. You could convert your nodes/edges to GeoJSON, then either upload to Mapbox and create a Tileset to visualize the paths, or add the GeoJSON to the map on the fly.

*You might be able to use open source routing tools like OSRM (https://github.com/Project-OSRM/osrm-backend/issues/6257) or Graphhopper (instead of a custom graph database implementation), but it would likely be more involved to maintain these services.

We have an in-house build experiment management system. We produce samples as input to the next step, which then could produce 1 sample(1-1) and many samples (1 - many). There are many steps like this. So far, we are tracking genealogy (limited tracking) in the MySQL database, which is becoming hard to trace back to the original material or sample(I can give more details if required). So, we are considering a Graph database. I am requesting advice from the experts.

1. Is a graph database the right choice, or can we manage with RDBMS?
2. If RDBMS, which RDMS, which feature, or which approach could make this manageable or sustainable
3. If Graph database(Neo4j, OrientDB, Azure Cosmos DB, Amazon Neptune, ArangoDB), which one is good, and what are the best practices?

I am sorry that this might be a loaded question.

I created a simple upload/download functionality for a web application and connected it to Mongo, now I can upload, store and download files. I need advice on how to create a SPA similar to Dropbox or Google Drive in that it will be a hierarchy of folders with files within them, how would I go about creating this structure and adding this functionality to all the files within the application?

Intuitively creating a react component and adding it to a File object seems like the way to go, what are some issues to expect and how do I go about creating such an application to be as fast and UI-friendly as possible?

Anyone recommend a good connector like Kloudless for connecting a SaaS app to Dropbox/Box etc? Cheers

In #Aliadoc, we're exploring the crowdfunding option to get traction before launch. We are building a SaaS platform for website design customization.

For the Admin UI and website editor we use React and we're currently transitioning from a Create React App setup to a custom one because our needs have become more specific. We use CloudFlare as much as possible, it's a great service.

For routing dynamic resources and proxy tasks to feed websites to the editor we leverage CloudFlare Workers for improved responsiveness. We use Firebase for our hosting needs and user authentication while also using several Cloud Functions for Firebase to interact with other services along with Google App Engine and Google Cloud Storage, but also the Real Time Database is on the radar for collaborative website editing.

We generally hate configuration but honestly because of the stage of our project we lack resources for doing heavy sysops work. So we are basically just relying on Serverless technologies as much as we can to do all server side processing.

Visual Studio Code definitively makes programming a much easier and enjoyable task, we just love it. We combine it with Bitbucket for our source code control needs.

We are looking for a centralised monitoring solution for our application deployed on Amazon EKS. We would like to monitor using metrics from Kubernetes, AWS services (NeptuneDB, AWS Elastic Load Balancing (ELB), Amazon EBS, Amazon S3, etc) and application microservice's custom metrics.

We are expected to use around 80 microservices (not replicas). I think a total of 200-250 microservices will be there in the system with 10-12 slave nodes.

We tried Prometheus but it looks like maintenance is a big issue. We need to manage scaling, maintaining the storage, and dealing with multiple exporters and Grafana. I felt this itself needs few dedicated resources (at least 2-3 people) to manage. Not sure if I am thinking in the correct direction. Please confirm.

You mentioned Datadog and Sysdig charges per host. Does it charge per slave node?