Alternatives to Spark Framework logo

Alternatives to Spark Framework

Apache Spark, Hadoop, Spring, Spring Boot, and ExpressJS are the most popular alternatives and competitors to Spark Framework.
19
22
+ 1
1

What is Spark Framework and what are its top alternatives?

It is a simple and expressive Java/Kotlin web framework DSL built for rapid development. Its intention is to provide an alternative for Kotlin/Java developers that want to develop their web applications as expressive as possible and with minimal boilerplate.
Spark Framework is a tool in the Microframeworks (Backend) category of a tech stack.

Spark Framework alternatives & related posts

related Apache Spark posts

Eric Colson
Eric Colson
Chief Algorithms Officer at Stitch Fix · | 19 upvotes · 622.2K views
atStitch FixStitch Fix
Kafka
Kafka
PostgreSQL
PostgreSQL
Amazon S3
Amazon S3
Apache Spark
Apache Spark
Presto
Presto
Python
Python
R Language
R Language
PyTorch
PyTorch
Docker
Docker
Amazon EC2 Container Service
Amazon EC2 Container Service
#AWS
#Etl
#ML
#DataScience
#DataStack
#Data

The algorithms and data infrastructure at Stitch Fix is housed in #AWS. Data acquisition is split between events flowing through Kafka, and periodic snapshots of PostgreSQL DBs. We store data in an Amazon S3 based data warehouse. Apache Spark on Yarn is our tool of choice for data movement and #ETL. Because our storage layer (s3) is decoupled from our processing layer, we are able to scale our compute environment very elastically. We have several semi-permanent, autoscaling Yarn clusters running to serve our data processing needs. While the bulk of our compute infrastructure is dedicated to algorithmic processing, we also implemented Presto for adhoc queries and dashboards.

Beyond data movement and ETL, most #ML centric jobs (e.g. model training and execution) run in a similarly elastic environment as containers running Python and R code on Amazon EC2 Container Service clusters. The execution of batch jobs on top of ECS is managed by Flotilla, a service we built in house and open sourced (see https://github.com/stitchfix/flotilla-os).

At Stitch Fix, algorithmic integrations are pervasive across the business. We have dozens of data products actively integrated systems. That requires serving layer that is robust, agile, flexible, and allows for self-service. Models produced on Flotilla are packaged for deployment in production using Khan, another framework we've developed internally. Khan provides our data scientists the ability to quickly productionize those models they've developed with open source frameworks in Python 3 (e.g. PyTorch, sklearn), by automatically packaging them as Docker containers and deploying to Amazon ECS. This provides our data scientist a one-click method of getting from their algorithms to production. We then integrate those deployments into a service mesh, which allows us to A/B test various implementations in our product.

For more info:

#DataScience #DataStack #Data

See more
Conor Myhrvold
Conor Myhrvold
Tech Brand Mgr, Office of CTO at Uber · | 7 upvotes · 283.8K views
atUber TechnologiesUber Technologies
Kafka
Kafka
Kafka Manager
Kafka Manager
Hadoop
Hadoop
Apache Spark
Apache Spark
GitHub
GitHub

Why we built Marmaray, an open source generic data ingestion and dispersal framework and library for Apache Hadoop :

Built and designed by our Hadoop Platform team, Marmaray is a plug-in-based framework built on top of the Hadoop ecosystem. Users can add support to ingest data from any source and disperse to any sink leveraging the use of Apache Spark . The name, Marmaray, comes from a tunnel in Turkey connecting Europe and Asia. Similarly, we envisioned Marmaray within Uber as a pipeline connecting data from any source to any sink depending on customer preference:

https://eng.uber.com/marmaray-hadoop-ingestion-open-source/

(Direct GitHub repo: https://github.com/uber/marmaray Kafka Kafka Manager )

See more
Hadoop logo

Hadoop

1.4K
1.2K
49
1.4K
1.2K
+ 1
49
Open-source software for reliable, scalable, distributed computing
Hadoop logo
Hadoop
VS
Spark Framework logo
Spark Framework

related Hadoop posts

StackShare Editors
StackShare Editors
| 4 upvotes · 275.4K views
atUber TechnologiesUber Technologies
Kafka
Kafka
Kibana
Kibana
Elasticsearch
Elasticsearch
Logstash
Logstash
Hadoop
Hadoop

With interactions across each other and mobile devices, logging is important as it is information for internal cases like debugging and business cases like dynamic pricing.

With multiple Kafka clusters, data is archived into Hadoop before expiration. Data is ingested in realtime and indexed into an ELK stack. The ELK stack comprises of Elasticsearch, Logstash, and Kibana for searching and visualization.

See more
StackShare Editors
StackShare Editors
Prometheus
Prometheus
Chef
Chef
Consul
Consul
Memcached
Memcached
Hack
Hack
Swift
Swift
Hadoop
Hadoop
Terraform
Terraform
Airflow
Airflow
Apache Spark
Apache Spark
Kubernetes
Kubernetes
gRPC
gRPC
HHVM (HipHop Virtual Machine)
HHVM (HipHop Virtual Machine)
Presto
Presto
Kotlin
Kotlin
Apache Thrift
Apache Thrift

Since the beginning, Cal Henderson has been the CTO of Slack. Earlier this year, he commented on a Quora question summarizing their current stack.

Apps
  • Web: a mix of JavaScript/ES6 and React.
  • Desktop: And Electron to ship it as a desktop application.
  • Android: a mix of Java and Kotlin.
  • iOS: written in a mix of Objective C and Swift.
Backend
  • The core application and the API written in PHP/Hack that runs on HHVM.
  • The data is stored in MySQL using Vitess.
  • Caching is done using Memcached and MCRouter.
  • The search service takes help from SolrCloud, with various Java services.
  • The messaging system uses WebSockets with many services in Java and Go.
  • Load balancing is done using HAproxy with Consul for configuration.
  • Most services talk to each other over gRPC,
  • Some Thrift and JSON-over-HTTP
  • Voice and video calling service was built in Elixir.
Data warehouse
  • Built using open source tools including Presto, Spark, Airflow, Hadoop and Kafka.
Etc
See more
Spring logo

Spring

2.2K
1.9K
974
2.2K
1.9K
+ 1
974
Provides a comprehensive programming and configuration model for modern Java-based enterprise applications
Spring logo
Spring
VS
Spark Framework logo
Spark Framework

related Spring posts

Spring
Spring
Spring Boot
Spring Boot
Java
Java
IntelliJ IDEA
IntelliJ IDEA
Slack
Slack

Spring Spring-Boot Java IntelliJ IDEA Slack

See more

related Spring Boot posts

Praveen Mooli
Praveen Mooli
Engineering Manager at Taylor and Francis · | 12 upvotes · 548.4K views
MongoDB Atlas
MongoDB Atlas
Java
Java
Spring Boot
Spring Boot
Node.js
Node.js
ExpressJS
ExpressJS
Python
Python
Flask
Flask
Amazon Kinesis
Amazon Kinesis
Amazon Kinesis Firehose
Amazon Kinesis Firehose
Amazon SNS
Amazon SNS
Amazon SQS
Amazon SQS
AWS Lambda
AWS Lambda
Angular 2
Angular 2
RxJS
RxJS
GitHub
GitHub
Travis CI
Travis CI
Terraform
Terraform
Docker
Docker
Serverless
Serverless
Amazon RDS
Amazon RDS
Amazon DynamoDB
Amazon DynamoDB
Amazon S3
Amazon S3
#Backend
#Microservices
#Eventsourcingframework
#Webapps
#Devops
#Data

We are in the process of building a modern content platform to deliver our content through various channels. We decided to go with Microservices architecture as we wanted scale. Microservice architecture style is an approach to developing an application as a suite of small independently deployable services built around specific business capabilities. You can gain modularity, extensive parallelism and cost-effective scaling by deploying services across many distributed servers. Microservices modularity facilitates independent updates/deployments, and helps to avoid single point of failure, which can help prevent large-scale outages. We also decided to use Event Driven Architecture pattern which is a popular distributed asynchronous architecture pattern used to produce highly scalable applications. The event-driven architecture is made up of highly decoupled, single-purpose event processing components that asynchronously receive and process events.

To build our #Backend capabilities we decided to use the following: 1. #Microservices - Java with Spring Boot , Node.js with ExpressJS and Python with Flask 2. #Eventsourcingframework - Amazon Kinesis , Amazon Kinesis Firehose , Amazon SNS , Amazon SQS, AWS Lambda 3. #Data - Amazon RDS , Amazon DynamoDB , Amazon S3 , MongoDB Atlas

To build #Webapps we decided to use Angular 2 with RxJS

#Devops - GitHub , Travis CI , Terraform , Docker , Serverless

See more
Java
Java
JavaScript
JavaScript
Node.js
Node.js
nginx
nginx
Ubuntu
Ubuntu
MongoDB
MongoDB
Amazon EC2
Amazon EC2
Redis
Redis
Amazon S3
Amazon S3
AWS Lambda
AWS Lambda
RabbitMQ
RabbitMQ
Kafka
Kafka
MySQL
MySQL
Spring Boot
Spring Boot
Dropwizard
Dropwizard
Google Analytics
Google Analytics
Elasticsearch
Elasticsearch
Amazon Route 53
Amazon Route 53
GitHub
GitHub
Docker
Docker
Webpack
Webpack
CircleCI
CircleCI
Jenkins
Jenkins
Travis CI
Travis CI
Gradle
Gradle
Apache Maven
Apache Maven
Jira
Jira
notion.so
notion.so
Trello
Trello
Vue.js
Vue.js
Flutter
Flutter
Application & Data

Java JavaScript Node.js nginx Ubuntu MongoDB Amazon EC2 Redis Amazon S3 AWS Lambda RabbitMQ Kafka MySQL Spring Boot Dropwizard Vue.js Flutter

Utilities

Google Analytics Elasticsearch Amazon Route 53

DevOps

GitHub Docker Webpack CircleCI Jenkins Travis CI Gradle Apache Maven

Cooperation Tools

Jira notion.so Trello

See more
ExpressJS logo

ExpressJS

7.4K
5.7K
1.4K
7.4K
5.7K
+ 1
1.4K
Sinatra inspired web development framework for node.js -- insanely fast, flexible, and simple
ExpressJS logo
ExpressJS
VS
Spark Framework logo
Spark Framework

related ExpressJS posts

Docker
Docker
Docker Compose
Docker Compose
Jenkins
Jenkins
Kubernetes
Kubernetes
Amazon EC2
Amazon EC2
Heroku
Heroku
FeathersJS
FeathersJS
Node.js
Node.js
ExpressJS
ExpressJS
PostgreSQL
PostgreSQL
React
React
Redux
Redux
Semantic UI React
Semantic UI React
AVA
AVA
ESLint
ESLint
nginx
nginx
GitHub
GitHub
#Containerized
#Containers
#Backend
#Stack
#Frontend

Recently I have been working on an open source stack to help people consolidate their personal health data in a single database so that AI and analytics apps can be run against it to find personalized treatments. We chose to go with a #containerized approach leveraging Docker #containers with a local development environment setup with Docker Compose and nginx for container routing. For the production environment we chose to pull code from GitHub and build/push images using Jenkins and using Kubernetes to deploy to Amazon EC2.

We also implemented a dashboard app to handle user authentication/authorization, as well as a custom SSO server that runs on Heroku which allows experts to easily visit more than one instance without having to login repeatedly. The #Backend was implemented using my favorite #Stack which consists of FeathersJS on top of Node.js and ExpressJS with PostgreSQL as the main database. The #Frontend was implemented using React, Redux.js, Semantic UI React and the FeathersJS client. Though testing was light on this project, we chose to use AVA as well as ESLint to keep the codebase clean and consistent.

See more
Paul Whittemore
Paul Whittemore
Developer and Owner at Appurist Software · | 15 upvotes · 321.4K views
Fastify
Fastify
ExpressJS
ExpressJS
Node.js
Node.js
Vuetify
Vuetify
Quasar Framework
Quasar Framework
Vue.js
Vue.js
vuex
vuex
Electron
Electron
Fastly
Fastly

I'm building most projects using: Server: either Fastify (all projects going forward) or ExpressJS on Node.js (existing, previously) on the server side, and Client app: either Vuetify (currently) or Quasar Framework (going forward) on Vue.js with vuex on Electron for the UI to deliver both web-based and desktop applications for multiple platforms.

The direct support for Android and iOS in Quasar Framework will make it my go-to client UI platform for any new client-side or web work. On the server, I'll probably use Fastly for all my server work, unless I get into Go more in the future.

Update: The mobile support in Quasar is not a sufficiently compelling reason to move me from Vuetify. I have decided to stick with Vuetify for a UI for Vue, as it is richer in components and enables a really great-looking professional result. For mobile platforms, I will just use Cordova to wrap the Vue+Vuetify app for mobile, and Electron to wrap it for desktop platforms.

See more
Flask logo

Flask

5.4K
4.4K
1.2K
5.4K
4.4K
+ 1
1.2K
a microframework for Python based on Werkzeug, Jinja 2 and good intentions.
Flask logo
Flask
VS
Spark Framework logo
Spark Framework

related Flask posts

James Man
James Man
Software Engineer at Pinterest · | 28 upvotes · 153.8K views
Flask
Flask
React
React

One of our top priorities at Pinterest is fostering a safe and trustworthy experience for all Pinners. As Pinterest’s user base and ads business grow, the review volume has been increasing exponentially, and more content types require moderation support. To solve greater engineering and operational challenges at scale, we needed a highly-reliable and performant system to detect, report, evaluate, and act on abusive content and users and so we created Pinqueue.

Pinqueue-3.0 serves as a generic platform for content moderation and human labeling. Under the hood, Pinqueue3.0 is a Flask + React app powered by Pinterest’s very own Gestalt UI framework. On the backend, Pinqueue3.0 heavily relies on PinLater, a Pinterest-built reliable asynchronous job execution system, to handle the requests for enqueueing and action-taking. Using PinLater has significantly strengthened Pinqueue3.0’s overall infra with its capability of processing a massive load of events with configurable retry policies.

Hundreds of millions of people around the world use Pinterest to discover and do what they love, and our job is to protect them from abusive and harmful content. We’re committed to providing an inspirational yet safe experience to all Pinners. Solving trust & safety problems is a joint effort requiring expertise across multiple domains. Pinqueue3.0 not only plays a critical role in responsively taking down unsafe content, it also has become an enabler for future ML/automation initiatives by providing high-quality human labels. Going forward, we will continue to improve the review experience, measure review quality and collaborate with our machine learning teams to solve content moderation beyond manual reviews at an even larger scale.

See more
Praveen Mooli
Praveen Mooli
Engineering Manager at Taylor and Francis · | 12 upvotes · 548.4K views
MongoDB Atlas
MongoDB Atlas
Java
Java
Spring Boot
Spring Boot
Node.js
Node.js
ExpressJS
ExpressJS
Python
Python
Flask
Flask
Amazon Kinesis
Amazon Kinesis
Amazon Kinesis Firehose
Amazon Kinesis Firehose
Amazon SNS
Amazon SNS
Amazon SQS
Amazon SQS
AWS Lambda
AWS Lambda
Angular 2
Angular 2
RxJS
RxJS
GitHub
GitHub
Travis CI
Travis CI
Terraform
Terraform
Docker
Docker
Serverless
Serverless
Amazon RDS
Amazon RDS
Amazon DynamoDB
Amazon DynamoDB
Amazon S3
Amazon S3
#Backend
#Microservices
#Eventsourcingframework
#Webapps
#Devops
#Data

We are in the process of building a modern content platform to deliver our content through various channels. We decided to go with Microservices architecture as we wanted scale. Microservice architecture style is an approach to developing an application as a suite of small independently deployable services built around specific business capabilities. You can gain modularity, extensive parallelism and cost-effective scaling by deploying services across many distributed servers. Microservices modularity facilitates independent updates/deployments, and helps to avoid single point of failure, which can help prevent large-scale outages. We also decided to use Event Driven Architecture pattern which is a popular distributed asynchronous architecture pattern used to produce highly scalable applications. The event-driven architecture is made up of highly decoupled, single-purpose event processing components that asynchronously receive and process events.

To build our #Backend capabilities we decided to use the following: 1. #Microservices - Java with Spring Boot , Node.js with ExpressJS and Python with Flask 2. #Eventsourcingframework - Amazon Kinesis , Amazon Kinesis Firehose , Amazon SNS , Amazon SQS, AWS Lambda 3. #Data - Amazon RDS , Amazon DynamoDB , Amazon S3 , MongoDB Atlas

To build #Webapps we decided to use Angular 2 with RxJS

#Devops - GitHub , Travis CI , Terraform , Docker , Serverless

See more

related Django REST framework posts

Tim Abbott
Tim Abbott
Founder at Zulip · | 9 upvotes · 101.5K views
atZulipZulip
Django
Django
Django REST framework
Django REST framework

Zulip has been powered by Django since the very early days of its development with Django 1.4, back in 2012. As a reasonably mature web application with significant scale, we're at the stage in many companies' development where one starts to rip out more and more of the web framework to optimize things or just make them work the way we want. (E.g. while I was at Dropbox in early 2016, we discovered we only had about 600 lines of code left from the original Pylons framework that actually ran).

One of the things that has been really fantastic about Django is that we're still happily using it for the vast majority of code in the project, and every time Django comes out with a new release, I read the changelog and get excited about several improvements that actually make my life better. While Django has made some design decisions that I don't agree with (e.g. I'm not a fan of Django REST framework, and think it makes life more difficult), Django also makes it easy to do your own thing, which we've done to great effect (see the linked article for details on our has_request_variables framework).

Overall I think we've gotten a ton of value out of Python and Django and would recommend it to anyone starting a new full-featured web application project today.

See more