What is Cloudflare Workers and what are its top alternatives?
Cloudflare Workers is a serverless platform that allows developers to deploy and run code at the edge of Cloudflare's global network. Key features include the ability to execute JavaScript code, access to Cloudflare's performance and security features, and seamless integration with other Cloudflare products. However, limitations of Cloudflare Workers include limited execution time and resource limits.
- AWS Lambda: AWS Lambda is a serverless computing service provided by Amazon Web Services. Key features include support for multiple programming languages, auto-scaling, and integration with other AWS services. Pros compared to Cloudflare Workers include deeper integration with AWS ecosystem, while cons include potentially higher costs.
- Google Cloud Functions: Google Cloud Functions is Google's serverless compute solution. Key features include event-driven execution, auto-scaling, and tight integration with other Google Cloud services. Pros compared to Cloudflare Workers include Google's global infrastructure, while cons include limited support for languages.
- Microsoft Azure Functions: Azure Functions is Microsoft's serverless compute service. Key features include support for multiple programming languages, built-in triggers, and seamless integration with Azure services. Pros compared to Cloudflare Workers include tight integration with Azure ecosystem, while cons include potentially higher costs.
- Vercel: Vercel is a platform for deploying web projects with a focus on serverless functions. Key features include support for Next.js, automatic scaling, and CDN caching. Pros compared to Cloudflare Workers include easy deployment of frontend projects alongside serverless functions, while cons include limited programming language support.
- Netlify Functions: Netlify Functions allows developers to deploy serverless functions alongside their frontend projects. Key features include zero-configuration deployment, custom domains, and integration with Git workflows. Pros compared to Cloudflare Workers include seamless integration with Netlify hosting, while cons include limited execution time.
- Fastly Compute@Edge: Fastly's Compute@Edge allows developers to run serverless functions at Fastly's edge network. Key features include support for Rust, C, and JavaScript, real-time logs, and instant configuration changes. Pros compared to Cloudflare Workers include more control over edge computing, while cons include potentially steeper learning curve.
- Begin: Begin is a serverless platform that simplifies building and deploying serverless applications. Key features include support for Node.js, automatic scaling, and built-in CI/CD. Pros compared to Cloudflare Workers include built-in monitoring and debugging tools, while cons include limited language support.
- Up: Up is an open-source platform for deploying serverless functions. Key features include support for multiple cloud providers, easy configuration via a simple JSON file, and seamless deployment workflow. Pros compared to Cloudflare Workers include flexibility in cloud provider choice, while cons include potentially limited scalability.
- OpenWhisk: OpenWhisk is an open-source serverless computing platform. Key features include support for multiple programming languages, event-driven architecture, and integration with various cloud services. Pros compared to Cloudflare Workers include being open-source and customizable, while cons include potentially higher maintenance overhead.
- Render: Render is a platform for building and deploying web applications and serverless functions. Key features include built-in HTTP caching, automatic scaling, and support for multiple programming languages. Pros compared to Cloudflare Workers include simplified deployment process, while cons include limited customizability.
Top Alternatives to Cloudflare Workers
- Fastly
Fastly's real-time content delivery network gives you total control over your content, unprecedented access to performance analytics, and the ability to instantly update content in 150 milliseconds. ...
- AWS Lambda
AWS Lambda is a compute service that runs your code in response to events and automatically manages the underlying compute resources for you. You can use AWS Lambda to extend other AWS services with custom logic, or create your own back-end services that operate at AWS scale, performance, and security. ...
- Netlify
Netlify is smart enough to process your site and make sure all assets gets optimized and served with perfect caching-headers from a cookie-less domain. We make sure your HTML is served straight from our CDN edge nodes without any round-trip to our backend servers and are the only ones to give you instant cache invalidation when you push a new deploy. Netlify is also the only static hosting service with integrated continuous deployment. ...
- Azure Functions
Azure Functions is an event driven, compute-on-demand experience that extends the existing Azure application platform with capabilities to implement code triggered by events occurring in virtually any Azure or 3rd party service as well as on-premises systems. ...
- Firebase
Firebase is a cloud service designed to power real-time, collaborative applications. Simply add the Firebase library to your application to gain access to a shared data structure; any changes you make to that data are automatically synchronized with the Firebase cloud and with other clients within milliseconds. ...
- NGINX
nginx [engine x] is an HTTP and reverse proxy server, as well as a mail proxy server, written by Igor Sysoev. According to Netcraft nginx served or proxied 30.46% of the top million busiest sites in Jan 2018. ...
- Apache HTTP Server
The Apache HTTP Server is a powerful and flexible HTTP/1.1 compliant web server. Originally designed as a replacement for the NCSA HTTP Server, it has grown to be the most popular web server on the Internet. ...
- Amazon EC2
It is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easier for developers. ...
Cloudflare Workers alternatives & related posts
- Real-time updates28
- Fastest CDN26
- Powerful API22
- Great support20
- Great customer support14
- Instant Purging7
- Custom VCL7
- Good pricing6
- Tag-based Purging6
- HTTP/2 Support5
- Speed & functionality4
- Image processing on demande (Fastly IO)4
- Best CDN4
- Minimum $50/mo spend1
related Fastly posts
When my SSL cert MaxCDN was expiring on my personal site I decided it was a good time to revamp some things. Since GitHub Services is depreciated I can no longer have #CDN cache purges automated among other things. So I decided on the following: GitHub Pages, Netlify, Let's Encrypt and Jekyll. Staying the same was Bootstrap, jQuery, Grunt & #GoogleFonts.
What's awesome about GitHub Pages is that it has a #CDN (Fastly) built-in and anytime you push to master, it purges the cache instantaneously without you have to do anything special. Netlify is magic, I highly recommend it to anyone using #StaticSiteGenerators.
For the most part, everything went smoothly. The only things I had issues with were the following:
- If you want to point
www
to GitHub Pages you need to rename the repo towww
- If you edit something in the
_config.yml
you need to restartbundle exec jekyll s
or changes won't show - I had to disable the Grunt
htmlmin
module. I replaced it with Jekyll layout that compresses HTML for #webperf
Last but certainly not least, I made a donation to Let's Encrypt. If you use their service consider doing it too: https://letsencrypt.org/donate/
AWS Lambda
- No infrastructure129
- Cheap83
- Quick70
- Stateless59
- No deploy, no server, great sleep47
- AWS Lambda went down taking many sites with it12
- Event Driven Governance6
- Extensive API6
- Auto scale and cost effective6
- Easy to deploy6
- VPC Support5
- Integrated with various AWS services3
- Cant execute ruby or go7
- Compute time limited3
- Can't execute PHP w/o significant effort1
related AWS Lambda posts
Recently we were looking at a few robust and cost-effective ways of replicating the data that resides in our production MongoDB to a PostgreSQL database for data warehousing and business intelligence.
We set ourselves the following criteria for the optimal tool that would do this job: - The data replication must be near real-time, yet it should NOT impact the production database - The data replication must be horizontally scalable (based on the load), asynchronous & crash-resilient
Based on the above criteria, we selected the following tools to perform the end to end data replication:
We chose MongoDB Stitch for picking up the changes in the source database. It is the serverless platform from MongoDB. One of the services offered by MongoDB Stitch is Stitch Triggers. Using stitch triggers, you can execute a serverless function (in Node.js) in real time in response to changes in the database. When there are a lot of database changes, Stitch automatically "feeds forward" these changes through an asynchronous queue.
We chose Amazon SQS as the pipe / message backbone for communicating the changes from MongoDB to our own replication service. Interestingly enough, MongoDB stitch offers integration with AWS services.
In the Node.js function, we wrote minimal functionality to communicate the database changes (insert / update / delete / replace) to Amazon SQS.
Next we wrote a minimal micro-service in Python to listen to the message events on SQS, pickup the data payload & mirror the DB changes on to the target Data warehouse. We implemented source data to target data translation by modelling target table structures through SQLAlchemy . We deployed this micro-service as AWS Lambda with Zappa. With Zappa, deploying your services as event-driven & horizontally scalable Lambda service is dumb-easy.
In the end, we got to implement a highly scalable near realtime Change Data Replication service that "works" and deployed to production in a matter of few days!
Heroku Docker GitHub Node.js hapi Vue.js AWS Lambda Amazon S3 PostgreSQL Knex.js Checkly is a fairly young company and we're still working hard to find the correct mix of product features, price and audience.
We are focussed on tech B2B, but I always wanted to serve solo developers too. So I decided to make a $7 plan.
Why $7? Simply put, it seems to be a sweet spot for tech companies: Heroku, Docker, Github, Appoptics (Librato) all offer $7 plans. They must have done a ton of research into this, so why not piggy back that and try it out.
Enough biz talk, onto tech. The challenges were:
- Slice of a portion of the functionality so a $7 plan is still profitable. We call this the "plan limits"
- Update API and back end services to handle and enforce plan limits.
- Update the UI to kindly state plan limits are in effect on some part of the UI.
- Update the pricing page to reflect all changes.
- Keep the actual processing backend, storage and API's as untouched as possible.
In essence, we went from strictly volume based pricing to value based pricing. Here come the technical steps & decisions we made to get there.
- We updated our PostgreSQL schema so plans now have an array of "features". These are string constants that represent feature toggles.
- The Vue.js frontend reads these from the vuex store on login.
- Based on these values, the UI has simple
v-if
statements to either just show the feature or show a friendly "please upgrade" button. - The hapi API has a hook on each relevant API endpoint that checks whether a user's plan has the feature enabled, or not.
Side note: We offer 10 SMS messages per month on the developer plan. However, we were not actually counting how many people were sending. We had to update our alerting daemon (that runs on Heroku and triggers SMS messages via AWS SNS) to actually bump a counter.
What we build is basically feature-toggling based on plan features. It is very extensible for future additions. Our scheduling and storage backend that actually runs users' monitoring requests (AWS Lambda) and stores the results (S3 and Postgres) has no knowledge of all of this and remained unchanged.
Hope this helps anyone building out their SaaS and is in a similar situation.
Netlify
- Easy deploy46
- Fastest static hosting and continuous deployments43
- Free SSL support22
- Super simple deploys22
- Easy Setup and Continous deployments15
- Faster than any other option in the market10
- Free plan for personal websites10
- Deploy previews8
- Free Open Source (Pro) plan6
- Easy to use and great support4
- Analytics4
- Great loop-in material on a blog4
- Custom domains support3
- Fastest static hosting and continuous deployments3
- Great drag and drop functionality3
- Canary Releases (Split Tests)1
- Supports static site generators1
- Tech oriented support1
- Django0
- It's expensive7
- Bandwidth limitation1
related Netlify posts
I was building a personal project that I needed to store items in a real time database. I am more comfortable with my Frontend skills than my backend so I didn't want to spend time building out anything in Ruby or Go.
I stumbled on Firebase by #Google, and it was really all I needed. It had realtime data, an area for storing file uploads and best of all for the amount of data I needed it was free!
I built out my application using tools I was familiar with, React for the framework, Redux.js to manage my state across components, and styled-components for the styling.
Now as this was a project I was just working on in my free time for fun I didn't really want to pay for hosting. I did some research and I found Netlify. I had actually seen them at #ReactRally the year before and deployed a Gatsby site to Netlify already.
Netlify was very easy to setup and link to my GitHub account you select a repo and pretty much with very little configuration you have a live site that will deploy every time you push to master.
With the selection of these tools I was able to build out my application, connect it to a realtime database, and deploy to a live environment all with $0 spent.
If you're looking to build out a small app I suggest giving these tools a go as you can get your idea out into the real world for absolutely no cost.
At FundsCorner, we are on a mission to enable fast accessible credit to India’s Kirana Stores. We are an early stage startup with an ultra small Engineering team. All the tech decisions we have made until now are based on our core philosophy: "Build usable products fast".
Based on the above fundamentals, we chose Python as our base language for all our APIs and micro-services. It is ultra easy to start with, yet provides great libraries even for the most complex of use cases. Our entire backend stack runs on Python and we cannot be more happy with it! If you are looking to deploy your API as server-less, Python provides one of the least cold start times.
We build our APIs with Flask. For backend database, our natural choice was MongoDB. It frees up our time from complex database specifications - we instead use our time in doing sensible data modelling & once we finalize the data model, we integrate it into Flask using Swagger UI. Mongo supports complex queries to cull out difficult data through aggregation framework & we have even built an internal framework called "Poetry", for aggregation queries.
Our web apps are built on Vue.js , Vuetify and vuex. Initially we debated a lot around choosing Vue.js or React , but finally settled with Vue.js, mainly because of the ease of use, fast development cycles & awesome set of libraries and utilities backing Vue.
You simply cannot go wrong with Vue.js . Great documentation, the library is ultra compact & is blazing fast. Choosing Vue.js was one of the critical decisions made, which enabled us to launch our web app in under a month (which otherwise would have taken 3 months easily). For those folks who are looking for big names, Adobe, and Alibaba and Gitlab are using Vue.
By choosing Vuetify, we saved thousands of person hours in designing the CSS files. Vuetify contains all key material components for designing a smooth User experience & it just works! It's an awesome framework. All of us at FundsCorner are now lifelong fanboys of Vue.js and Vuetify.
On the infrastructure side, all our API services and backend services are deployed as server less micro-services through Zappa. Zappa makes your life super easy by packaging everything that is required to deploy your code as AWS Lambda. We are now addicted to the single - click deploys / updates through Zappa. Try it out & you will convert!
Also, if you are using Zappa, you can greatly simplify your CI / CD pipelines. Do try it! It's just awesome! and... you will be astonished by the savings you have made on AWS bills at end of the month.
Our CI / CD pipelines are built using GitLab CI. The documentation is very good & it enables you to go from from concept to production in minimal time frame.
We use Sentry for all crash reporting and resolution. Pro tip, they do have handlers for AWS Lambda , which made our integration super easy.
All our micro-services including APIs are event-driven. Our background micro-services are message oriented & we use Amazon SQS as our message pipe. We have our own in-house workflow manager to orchestrate across micro - services.
We host our static websites on Netlify. One of the cool things about Netlify is the automated CI / CD on git push. You just do a git push to deploy! Again, it is super simple to use and it just works. We were dogmatic about going server less even on static web sites & you can go server less on Netlify in a few minutes. It's just a few clicks away.
We use Google Compute Engine, especially Google Vision for our AI experiments.
For Ops automation, we use Slack. Slack provides a super-rich API (through Slack App) through which you can weave magical automation on boring ops tasks.
- Pay only when invoked14
- Great developer experience for C#11
- Multiple languages supported9
- Great debugging support7
- Can be used as lightweight https service5
- Easy scalability4
- WebHooks3
- Costo3
- Event driven2
- Azure component events for Storage, services etc2
- Poor developer experience for C#2
- No persistent (writable) file system available1
- Poor support for Linux environments1
- Sporadic server & language runtime issues1
- Not suited for long-running applications1
related Azure Functions posts





CodeFactor being a #SAAS product, our goal was to run on a cloud-native infrastructure since day one. We wanted to stay product focused, rather than having to work on the infrastructure that supports the application. We needed a cloud-hosting provider that would be reliable, economical and most efficient for our product.
CodeFactor.io aims to provide an automated and frictionless code review service for software developers. That requires agility, instant provisioning, autoscaling, security, availability and compliance management features. We looked at the top three #IAAS providers that take up the majority of market share: Amazon's Amazon EC2 , Microsoft's Microsoft Azure, and Google Compute Engine.
AWS has been available since 2006 and has developed the most extensive services ant tools variety at a massive scale. Azure and GCP are about half the AWS age, but also satisfied our technical requirements.
It is worth noting that even though all three providers support Docker containerization services, GCP has the most robust offering due to their investments in Kubernetes. Also, if you are a Microsoft shop, and develop in .NET - Visual Studio Azure shines at integration there and all your existing .NET code works seamlessly on Azure. All three providers have serverless computing offerings (AWS Lambda, Azure Functions, and Google Cloud Functions). Additionally, all three providers have machine learning tools, but GCP appears to be the most developer-friendly, intuitive and complete when it comes to #Machinelearning and #AI.
The prices between providers are competitive across the board. For our requirements, AWS would have been the most expensive, GCP the least expensive and Azure was in the middle. Plus, if you #Autoscale frequently with large deltas, note that Azure and GCP have per minute billing, where AWS bills you per hour. We also applied for the #Startup programs with all three providers, and this is where Azure shined. While AWS and GCP for startups would have covered us for about one year of infrastructure costs, Azure Sponsorship would cover about two years of CodeFactor's hosting costs. Moreover, Azure Team was terrific - I felt that they wanted to work with us where for AWS and GCP we were just another startup.
In summary, we were leaning towards GCP. GCP's advantages in containerization, automation toolset, #Devops mindset, and pricing were the driving factors there. Nevertheless, we could not say no to Azure's financial incentives and a strong sense of partnership and support throughout the process.
Bottom line is, IAAS offerings with AWS, Azure, and GCP are evolving fast. At CodeFactor, we aim to be platform agnostic where it is practical and retain the flexibility to cherry-pick the best products across providers.
In a couple of recent projects we had an opportunity to try out the new Serverless approach to building web applications. It wasn't necessarily a question if we should use any particular vendor but rather "if" we can consider serverless a viable option for building apps. Obviously our goal was also to get a feel for this technology and gain some hands-on experience.
We did consider AWS Lambda, Firebase from Google as well as Azure Functions. Eventually we went with AWS Lambdas.
PROS- No servers to manage (obviously!)
- Limited fixed costs – you pay only for used time
- Automated scaling and balancing
- Automatic failover (or, at this level of abstraction, no failover problem at all)
- Security easier to provide and audit
- Low overhead at the start (with the certain level of knowledge)
- Short time to market
- Easy handover - deployment coupled with code
- Perfect choice for lean startups with fast-paced iterations
- Augmentation for the classic cloud, server(full) approach
- Not much know-how and best practices available about structuring the code and projects on the market
- Not suitable for complex business logic due to the risk of producing highly coupled code
- Cost difficult to estimate (helpful tools: serverlesscalc.com)
- Difficulty in migration to other platforms (Vendor lock⚠️)
- Little engineers with experience in serverless on the job market
- Steep learning curve for engineers without any cloud experience
More details are on our blog: https://evojam.com/blog/2018/12/5/should-you-go-serverless-meet-the-benefits-and-flaws-of-new-wave-of-cloud-solutions I hope it helps 🙌 & I'm curious of your experiences.
- Realtime backend made easy371
- Fast and responsive270
- Easy setup242
- Real-time215
- JSON191
- Free134
- Backed by google128
- Angular adaptor83
- Reliable68
- Great customer support36
- Great documentation32
- Real-time synchronization25
- Mobile friendly21
- Rapid prototyping19
- Great security14
- Automatic scaling12
- Freakingly awesome11
- Super fast development8
- Angularfire is an amazing addition!8
- Chat8
- Firebase hosting6
- Built in user auth/oauth6
- Awesome next-gen backend6
- Ios adaptor6
- Speed of light4
- Very easy to use4
- Great3
- It's made development super fast3
- Brilliant for startups3
- Free hosting2
- Cloud functions2
- JS Offline and Sync suport2
- Low battery consumption2
- .net2
- The concurrent updates create a great experience2
- Push notification2
- I can quickly create static web apps with no backend2
- Great all-round functionality2
- Free authentication solution2
- Easy Reactjs integration1
- Google's support1
- Free SSL1
- CDN & cache out of the box1
- Easy to use1
- Large1
- Faster workflow1
- Serverless1
- Good Free Limits1
- Simple and easy1
- Can become expensive31
- No open source, you depend on external company16
- Scalability is not infinite15
- Not Flexible Enough9
- Cant filter queries7
- Very unstable server3
- No Relational Data3
- Too many errors2
- No offline sync2
related Firebase posts
Hi Otensia! I'd definitely recommend using the skills you've already got and building with JavaScript is a smart way to go these days. Most platform services have JavaScript/Node SDKs or NPM packages, many serverless platforms support Node in case you need to write any backend logic, and JavaScript is incredibly popular - meaning it will be easy to hire for, should you ever need to.
My advice would be "don't reinvent the wheel". If you already have a skill set that will work well to solve the problem at hand, and you don't need it for any other projects, don't spend the time jumping into a new language. If you're looking for an excuse to learn something new, it would be better to invest that time in learning a new platform/tool that compliments your knowledge of JavaScript. For this project, I might recommend using Netlify, Vercel, or Google Firebase to quickly and easily deploy your web app. If you need to add user authentication, there are great examples out there for Firebase Authentication, Auth0, or even Magic (a newcomer on the Auth scene, but very user friendly). All of these services work very well with a JavaScript-based application.
For inboxkitten.com, an opensource disposable email service;
We migrated our serverless workload from Cloud Functions for Firebase to CloudFlare workers, taking advantage of the lower cost and faster-performing edge computing of Cloudflare network. Made possible due to our extremely low CPU and RAM overhead of our serverless functions.
If I were to summarize the limitation of Cloudflare (as oppose to firebase/gcp functions), it would be ...
- <5ms CPU time limit
- Incompatible with express.js
- one script limitation per domain
Limitations our workload is able to conform with (YMMV)
For hosting of static files, we migrated from Firebase to CommonsHost
More details on the trade-off in between both serverless providers is in the article
NGINX
- High-performance http server1.5K
- Performance894
- Easy to configure730
- Open source607
- Load balancer530
- Free289
- Scalability288
- Web server226
- Simplicity175
- Easy setup136
- Content caching30
- Web Accelerator21
- Capability15
- Fast14
- High-latency12
- Predictability12
- Reverse Proxy8
- Supports http/27
- The best of them7
- Great Community5
- Lots of Modules5
- Enterprise version5
- High perfomance proxy server4
- Embedded Lua scripting3
- Streaming media delivery3
- Streaming media3
- Reversy Proxy3
- Blash2
- GRPC-Web2
- Lightweight2
- Fast and easy to set up2
- Slim2
- saltstack2
- Virtual hosting1
- Narrow focus. Easy to configure. Fast1
- Along with Redis Cache its the Most superior1
- Ingress controller1
- Advanced features require subscription10
related NGINX posts
Our whole DevOps stack consists of the following tools:
- GitHub (incl. GitHub Pages/Markdown for Documentation, GettingStarted and HowTo's) for collaborative review and code management tool
- Respectively Git as revision control system
- SourceTree as Git GUI
- Visual Studio Code as IDE
- CircleCI for continuous integration (automatize development process)
- Prettier / TSLint / ESLint as code linter
- SonarQube as quality gate
- Docker as container management (incl. Docker Compose for multi-container application management)
- VirtualBox for operating system simulation tests
- Kubernetes as cluster management for docker containers
- Heroku for deploying in test environments
- nginx as web server (preferably used as facade server in production environment)
- SSLMate (using OpenSSL) for certificate management
- Amazon EC2 (incl. Amazon S3) for deploying in stage (production-like) and production environments
- PostgreSQL as preferred database system
- Redis as preferred in-memory database/store (great for caching)
The main reason we have chosen Kubernetes over Docker Swarm is related to the following artifacts:
- Key features: Easy and flexible installation, Clear dashboard, Great scaling operations, Monitoring is an integral part, Great load balancing concepts, Monitors the condition and ensures compensation in the event of failure.
- Applications: An application can be deployed using a combination of pods, deployments, and services (or micro-services).
- Functionality: Kubernetes as a complex installation and setup process, but it not as limited as Docker Swarm.
- Monitoring: It supports multiple versions of logging and monitoring when the services are deployed within the cluster (Elasticsearch/Kibana (ELK), Heapster/Grafana, Sysdig cloud integration).
- Scalability: All-in-one framework for distributed systems.
- Other Benefits: Kubernetes is backed by the Cloud Native Computing Foundation (CNCF), huge community among container orchestration tools, it is an open source and modular tool that works with any OS.
We chose AWS because, at the time, it was really the only cloud provider to choose from.
We tend to use their basic building blocks (EC2, ELB, Amazon S3, Amazon RDS) rather than vendor specific components like databases and queuing. We deliberately decided to do this to ensure we could provide multi-cloud support or potentially move to another cloud provider if the offering was better for our customers.
We’ve utilized c3.large nodes for both the Node.js deployment and then for the .NET Core deployment. Both sit as backends behind an nginx instance and are managed using scaling groups in Amazon EC2 sitting behind a standard AWS Elastic Load Balancing (ELB).
While we’re satisfied with AWS, we do review our decision each year and have looked at Azure and Google Cloud offerings.
#CloudHosting #WebServers #CloudStorage #LoadBalancerReverseProxy
Apache HTTP Server
- Web server479
- Most widely-used web server305
- Virtual hosting217
- Fast148
- Ssl support138
- Since 199644
- Asynchronous28
- Robust5
- Proven over many years4
- Mature2
- Perfomance2
- Perfect Support1
- Many available modules0
- Many available modules0
- Hard to set up4
related Apache HTTP Server posts
When I joined NYT there was already broad dissatisfaction with the LAMP (Linux Apache HTTP Server MySQL PHP) Stack and the front end framework, in particular. So, I wasn't passing judgment on it. I mean, LAMP's fine, you can do good work in LAMP. It's a little dated at this point, but it's not ... I didn't want to rip it out for its own sake, but everyone else was like, "We don't like this, it's really inflexible." And I remember from being outside the company when that was called MIT FIVE when it had launched. And been observing it from the outside, and I was like, you guys took so long to do that and you did it so carefully, and yet you're not happy with your decisions. Why is that? That was more the impetus. If we're going to do this again, how are we going to do it in a way that we're gonna get a better result?
So we're moving quickly away from LAMP, I would say. So, right now, the new front end is React based and using Apollo. And we've been in a long, protracted, gradual rollout of the core experiences.
React is now talking to GraphQL as a primary API. There's a Node.js back end, to the front end, which is mainly for server-side rendering, as well.
Behind there, the main repository for the GraphQL server is a big table repository, that we call Bodega because it's a convenience store. And that reads off of a Kafka pipeline.
We've been happy with nginx as part of our stack. As an open source web application that folks install on-premise, the configuration system for the webserver is pretty important to us. I have a few complaints (e.g. the configuration syntax for conditionals is a pain), but overall we've found it pretty easy to build a configurable set of options (see link) for how to run Zulip on nginx, both directly and with a remote reverse proxy in front of it, with a minimum of code duplication.
Certainly I've been a lot happier with it than I was working with Apache HTTP Server in past projects.
- Quick and reliable cloud servers647
- Scalability515
- Easy management393
- Low cost277
- Auto-scaling271
- Market leader89
- Backed by amazon80
- Reliable79
- Free tier67
- Easy management, scalability58
- Flexible13
- Easy to Start10
- Widely used9
- Web-scale9
- Elastic9
- Node.js API7
- Industry Standard5
- Lots of configuration options4
- GPU instances2
- Simpler to understand and learn1
- Extremely simple to use1
- Amazing for individuals1
- All the Open Source CLI tools you could want.1
- Ui could use a lot of work14
- High learning curve when compared to PaaS6
- Extremely poor CPU performance3
related Amazon EC2 posts
To provide employees with the critical need of interactive querying, we’ve worked with Presto, an open-source distributed SQL query engine, over the years. Operating Presto at Pinterest’s scale has involved resolving quite a few challenges like, supporting deeply nested and huge thrift schemas, slow/ bad worker detection and remediation, auto-scaling cluster, graceful cluster shutdown and impersonation support for ldap authenticator.
Our infrastructure is built on top of Amazon EC2 and we leverage Amazon S3 for storing our data. This separates compute and storage layers, and allows multiple compute clusters to share the S3 data.
We have hundreds of petabytes of data and tens of thousands of Apache Hive tables. Our Presto clusters are comprised of a fleet of 450 r4.8xl EC2 instances. Presto clusters together have over 100 TBs of memory and 14K vcpu cores. Within Pinterest, we have close to more than 1,000 monthly active users (out of total 1,600+ Pinterest employees) using Presto, who run about 400K queries on these clusters per month.
Each query submitted to Presto cluster is logged to a Kafka topic via Singer. Singer is a logging agent built at Pinterest and we talked about it in a previous post. Each query is logged when it is submitted and when it finishes. When a Presto cluster crashes, we will have query submitted events without corresponding query finished events. These events enable us to capture the effect of cluster crashes over time.
Each Presto cluster at Pinterest has workers on a mix of dedicated AWS EC2 instances and Kubernetes pods. Kubernetes platform provides us with the capability to add and remove workers from a Presto cluster very quickly. The best-case latency on bringing up a new worker on Kubernetes is less than a minute. However, when the Kubernetes cluster itself is out of resources and needs to scale up, it can take up to ten minutes. Some other advantages of deploying on Kubernetes platform is that our Presto deployment becomes agnostic of cloud vendor, instance types, OS, etc.
#BigData #AWS #DataScience #DataEngineering
Our whole DevOps stack consists of the following tools:
- GitHub (incl. GitHub Pages/Markdown for Documentation, GettingStarted and HowTo's) for collaborative review and code management tool
- Respectively Git as revision control system
- SourceTree as Git GUI
- Visual Studio Code as IDE
- CircleCI for continuous integration (automatize development process)
- Prettier / TSLint / ESLint as code linter
- SonarQube as quality gate
- Docker as container management (incl. Docker Compose for multi-container application management)
- VirtualBox for operating system simulation tests
- Kubernetes as cluster management for docker containers
- Heroku for deploying in test environments
- nginx as web server (preferably used as facade server in production environment)
- SSLMate (using OpenSSL) for certificate management
- Amazon EC2 (incl. Amazon S3) for deploying in stage (production-like) and production environments
- PostgreSQL as preferred database system
- Redis as preferred in-memory database/store (great for caching)
The main reason we have chosen Kubernetes over Docker Swarm is related to the following artifacts:
- Key features: Easy and flexible installation, Clear dashboard, Great scaling operations, Monitoring is an integral part, Great load balancing concepts, Monitors the condition and ensures compensation in the event of failure.
- Applications: An application can be deployed using a combination of pods, deployments, and services (or micro-services).
- Functionality: Kubernetes as a complex installation and setup process, but it not as limited as Docker Swarm.
- Monitoring: It supports multiple versions of logging and monitoring when the services are deployed within the cluster (Elasticsearch/Kibana (ELK), Heapster/Grafana, Sysdig cloud integration).
- Scalability: All-in-one framework for distributed systems.
- Other Benefits: Kubernetes is backed by the Cloud Native Computing Foundation (CNCF), huge community among container orchestration tools, it is an open source and modular tool that works with any OS.