Alternatives to Cloudinary

When I first built my portfolio I used GitHub for the source control and deployed directly to Netlify on a push to master. This was a perfect setup, I didn't need any knowledge about #DevOps or anything, it was all just done for me.

One of the issues I had with Netlify was I wanted to gzip my JavaScript files, I had this setup in my #Webpack file, however Netlify didn't offer an easy way to set this.

Over the weekend I decided I wanted to know more about how #DevOps worked so I decided to switch from Netlify to Amazon S3. Instead of creating any #Git Webhooks I decided to use Buddy for my pipeline and to run commands. Buddy is a fantastic tool, very easy to setup builds, copying the files to my Amazon S3 bucket, then running some #AWS console commands to set the content-encoding of the JavaScript files. - Buddy is also free if you only have a few pipelines, so I didn't need to pay anything 🤙🏻.

When I made these changes I also wanted to monitor my code, and make sure I was keeping up with the best practices so I implemented Code Climate to look over my code and tell me where there code smells, issues, and other issues I've been super happy with it so far, on the free tier so its also free.

I did plan on using Amazon CloudFront for my SSL and cacheing, however it was overly complex to setup and it costs money. So I decided to go with the free tier of CloudFlare and it is amazing, best choice I've made for caching / SSL in a long time.

I recently moved my portfolio to Amazon S3 and I needed a new way to cache and SSL my site as Amazon S3 does not come with this right out of the box. I tried Amazon CloudFront as I was already on Amazon S3 I thought this would be super easy and straight forward to setup... It was not, I was unable to get this working even though I followed all the online steps and even reached out for help to Amazon.

I'd used CloudFlare in the past, and thought let me see if I can set up CloudFlare on an Amazon S3 bucket. The setup for this was so basic and easy... I had it setup with caching and SSL within 5 minutes, and it was 100% free.

Platform Update: we’ve been using the Performance Test tool provided by KeyCDN for a long time in combination with Pingdom's similar tool and the #WebpageTest and #GoogleInsight - we decided to test out KeyCDN for static asset hosting. The results for the endpoints were superfast - almost 200% faster than CloudFlare in some tests and 370% faster than imgix . So we’ve moved Washington Brown from imgix for hosting theme images, to KeyCDN for hosting all images and static assets (Font, CSS & JS). There’s a few things that we like about “Key” apart from saving $6 a month on the monthly minimum spend ($4 vs $10 for imgix). Key allow for a custom CNAME (no more advertising imgix.com in domain requests and possible SEO improvements - and easier to swap to another host down the track). Key allows JPEG/WebP image requests based on clients ‘accept’ http headers - imgix required a ?auto=format query string on each image resource request - which can break some caches. Key allows for explicitly denying cookies to be set on a zone/domain; cookies are a big strain on limited upload bandwidth so to be able to force these off is great - Cloudflare adds a cookie to every header… for “performance reasons”… but remember “if you’re getting a product something for free…”

In mid-2018 we made a big push for speed on the site. The site, running on PHP, was taking about 7 seconds to load. The site had already been running through CloudFlare for some time but on a shared host in Sydney (which is also where most of the customers are). We found when developing the @TuffTruck site that DigitalOcean was fast - and even though it's located overseas, we still found it 2 seconds faster for Australian users. We found that some Wordpress plugins were really slowing the TTFB - with all plugins off, Wordpress would save respond 1.5-2 seconds faster. With a on/off step through of each plugin we found 2 plugins by Ontraport (a CRM type service that some forms we populating) was the main culprit. Out it went and we built our own WP plugin to do push the data to Ontraport only when required. With the TTFB acceptable, we moved on to getting the completed page load time down. Turning on CloudFlare 's HTML/CSS/JS minifications & Rocket Loader we could get our group of test pages, including the homepage, loading [in full] in just over 2 seconds. We then moved images off to imgix and put the CSS, JS and Fonts onto a mirrored subdomain (so that cookies weren't exchanged), but this only shaved about another 0.2 seconds off. We are keeping it running for the moment, but the $10 minimum a month for imgix is hardly worth it (this would be change if new images were going up all the time and needed processing). The client is overly happy with the ~70% improvement and has already seen the site move up the ranks of Google's SERP and bring down their PPC costs. AND all the new hosting providers still come in at half the price of the previous Sydney hosting service. We have a few ideas that we are testing on our staging site and will roll these out soon.

To provide employees with the critical need of interactive querying, we’ve worked with Presto, an open-source distributed SQL query engine, over the years. Operating Presto at Pinterest’s scale has involved resolving quite a few challenges like, supporting deeply nested and huge thrift schemas, slow/ bad worker detection and remediation, auto-scaling cluster, graceful cluster shutdown and impersonation support for ldap authenticator.

Our infrastructure is built on top of Amazon EC2 and we leverage Amazon S3 for storing our data. This separates compute and storage layers, and allows multiple compute clusters to share the S3 data.

We have hundreds of petabytes of data and tens of thousands of Apache Hive tables. Our Presto clusters are comprised of a fleet of 450 r4.8xl EC2 instances. Presto clusters together have over 100 TBs of memory and 14K vcpu cores. Within Pinterest, we have close to more than 1,000 monthly active users (out of total 1,600+ Pinterest employees) using Presto, who run about 400K queries on these clusters per month.

Each query submitted to Presto cluster is logged to a Kafka topic via Singer. Singer is a logging agent built at Pinterest and we talked about it in a previous post. Each query is logged when it is submitted and when it finishes. When a Presto cluster crashes, we will have query submitted events without corresponding query finished events. These events enable us to capture the effect of cluster crashes over time.

Each Presto cluster at Pinterest has workers on a mix of dedicated AWS EC2 instances and Kubernetes pods. Kubernetes platform provides us with the capability to add and remove workers from a Presto cluster very quickly. The best-case latency on bringing up a new worker on Kubernetes is less than a minute. However, when the Kubernetes cluster itself is out of resources and needs to scale up, it can take up to ten minutes. Some other advantages of deploying on Kubernetes platform is that our Presto deployment becomes agnostic of cloud vendor, instance types, OS, etc.

#BigData #AWS #DataScience #DataEngineering

Our whole DevOps stack consists of the following tools:

• GitHub (incl. GitHub Pages/Markdown for Documentation, GettingStarted and HowTo's) for collaborative review and code management tool
• Respectively Git as revision control system
• SourceTree as Git GUI
• Visual Studio Code as IDE
• CircleCI for continuous integration (automatize development process)
• Prettier / TSLint / ESLint as code linter
• SonarQube as quality gate
• Docker as container management (incl. Docker Compose for multi-container application management)
• VirtualBox for operating system simulation tests
• Kubernetes as cluster management for docker containers
• Heroku for deploying in test environments
• nginx as web server (preferably used as facade server in production environment)
• SSLMate (using OpenSSL) for certificate management
• Amazon EC2 (incl. Amazon S3) for deploying in stage (production-like) and production environments
• PostgreSQL as preferred database system
• Redis as preferred in-memory database/store (great for caching)

The main reason we have chosen Kubernetes over Docker Swarm is related to the following artifacts:

• Key features: Easy and flexible installation, Clear dashboard, Great scaling operations, Monitoring is an integral part, Great load balancing concepts, Monitors the condition and ensures compensation in the event of failure.
• Applications: An application can be deployed using a combination of pods, deployments, and services (or micro-services).
• Functionality: Kubernetes as a complex installation and setup process, but it not as limited as Docker Swarm.
• Monitoring: It supports multiple versions of logging and monitoring when the services are deployed within the cluster (Elasticsearch/Kibana (ELK), Heapster/Grafana, Sysdig cloud integration).
• Scalability: All-in-one framework for distributed systems.
• Other Benefits: Kubernetes is backed by the Cloud Native Computing Foundation (CNCF), huge community among container orchestration tools, it is an open source and modular tool that works with any OS.

Hi Otensia! I'd definitely recommend using the skills you've already got and building with JavaScript is a smart way to go these days. Most platform services have JavaScript/Node SDKs or NPM packages, many serverless platforms support Node in case you need to write any backend logic, and JavaScript is incredibly popular - meaning it will be easy to hire for, should you ever need to.

My advice would be "don't reinvent the wheel". If you already have a skill set that will work well to solve the problem at hand, and you don't need it for any other projects, don't spend the time jumping into a new language. If you're looking for an excuse to learn something new, it would be better to invest that time in learning a new platform/tool that compliments your knowledge of JavaScript. For this project, I might recommend using Netlify, Vercel, or Google Firebase to quickly and easily deploy your web app. If you need to add user authentication, there are great examples out there for Firebase Authentication, Auth0, or even Magic (a newcomer on the Auth scene, but very user friendly). All of these services work very well with a JavaScript-based application.

This is my stack in Application & Data

JavaScript PHP HTML5 jQuery Redis Amazon EC2 Ubuntu Sass Vue.js Firebase Laravel Lumen Amazon RDS GraphQL MariaDB

My Utilities Tools

Google Analytics Postman Elasticsearch

My Devops Tools

Git GitHub GitLab npm Visual Studio Code Kibana Sentry BrowserStack

My Business Tools

Slack