Apache Spark vs Memcached

Overview

Memcached

Stacks7.9K

Followers5.7K

Votes473

GitHub Stars14.0K

Forks3.3K

Apache Spark

Stacks3.1K

Followers3.5K

Votes140

GitHub Stars42.2K

Forks28.9K

Apache Spark vs Memcached: What are the differences?

Introduction:

Apache Spark and Memcached are both widely used in the field of big data and distributed computing. However, they serve different purposes and have distinct features that set them apart from each other.

1. Scalability: Apache Spark is designed for processing large-scale data analytics workloads, enabling parallel processing on a distributed computing cluster. On the other hand, Memcached is primarily used for caching frequently accessed data in memory to improve the overall performance of web applications.

2. Programming Paradigm: Apache Spark supports multiple programming languages such as Java, Scala, Python, and R, making it more flexible for developers with diverse coding backgrounds. In contrast, Memcached is a simple key-value store with a limited set of commands and data types, designed for fast data retrieval.

3. Fault Tolerance: Apache Spark inherently provides fault tolerance by storing intermediate data in resilient distributed datasets (RDDs), enabling fault recovery in case of node failures during computation. This ensures reliable data processing across distributed systems. Conversely, Memcached does not have built-in fault tolerance mechanisms and may require additional configurations for data redundancy.

4. Data Processing: Apache Spark is equipped with powerful libraries for machine learning (MLlib), stream processing (Spark Streaming), graph processing (GraphX), and SQL queries (Spark SQL), making it a versatile tool for various data processing tasks. On the other hand, Memcached focuses on caching data in memory and does not provide extensive data processing capabilities beyond key-value storage.

5. Storage Management: Apache Spark can leverage various storage options such as in-memory processing, disk-based persistence, and external storage systems like Hadoop HDFS, enabling efficient data storage and retrieval. In contrast, Memcached relies solely on memory caching and does not support persistent storage, limiting its capacity for long-term data retention.

6. Use Cases: Apache Spark is commonly used for big data analytics, real-time processing, machine learning, and interactive querying in data-driven applications. On the other hand, Memcached is often utilized in web applications for caching database queries, session storage, and content delivery networks to improve performance and scalability.

In Summary, Apache Spark is a versatile distributed computing framework with advanced data processing capabilities, while Memcached is a high-performance caching system designed for improving web application performance through in-memory data storage.

Share your Stack

Help developers discover the tools you use. Get visibility for your team's tech choices and contribute to the community's knowledge.

View Docs

CLI (Node.js)

Manual

Advice on Memcached, Apache Spark

Nilesh

Technical Architect at Self Employed

Jul 8, 2020

Needs adviceon

Elasticsearch

Kafka

We have a Kafka topic having events of type A and type B. We need to perform an inner join on both type of events using some common field (primary-key). The joined events to be inserted in Elasticsearch.

In usual cases, type A and type B events (with same key) observed to be close upto 15 minutes. But in some cases they may be far from each other, lets say 6 hours. Sometimes event of either of the types never come.

In all cases, we should be able to find joined events instantly after they are joined and not-joined events within 15 minutes.

576k views576k

Comments

Detailed Comparison

Memcached	Apache Spark
Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.	Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.
-	Run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk;Write applications quickly in Java, Scala or Python;Combine SQL, streaming, and complex analytics;Spark runs on Hadoop, Mesos, standalone, or in the cloud. It can access diverse data sources including HDFS, Cassandra, HBase, S3
Statistics
GitHub Stars 14.0K	GitHub Stars 42.2K
GitHub Forks 3.3K	GitHub Forks 28.9K
Stacks 7.9K	Stacks 3.1K
Followers 5.7K	Followers 3.5K
Votes 473	Votes 140
Pros & Cons
Pros 139 Fast object cache 129 High-performance 91 Stable 65 Mature 33 Distributed caching system Cons 2 Only caches simple types	Pros 61 Open-source 48 Fast and Flexible 8 One platform for every big data problem 8 Great for distributed SQL like applications 6 Easy to install and to use Cons 4 Speed

What are some alternatives to Memcached, Apache Spark?

MongoDB

MongoDB stores data in JSON-like documents that can vary in structure, offering a dynamic, flexible schema. MongoDB was also designed for high availability and scalability, with built-in replication and auto-sharding.

MySQL

The MySQL software delivers a very fast, multi-threaded, multi-user, and robust SQL (Structured Query Language) database server. MySQL Server is intended for mission-critical, heavy-load production systems as well as for embedding into mass-deployed software.

PostgreSQL

PostgreSQL is an advanced object-relational database management system that supports an extended subset of the SQL standard, including transactions, foreign keys, subqueries, triggers, user-defined types and functions.

Microsoft SQL Server

Microsoft® SQL Server is a database management and analysis system for e-commerce, line-of-business, and data warehousing solutions.

SQLite

SQLite is an embedded SQL database engine. Unlike most other SQL databases, SQLite does not have a separate server process. SQLite reads and writes directly to ordinary disk files. A complete SQL database with multiple tables, indices, triggers, and views, is contained in a single disk file.

Cassandra

Partitioning means that Cassandra can distribute your data across multiple machines in an application-transparent matter. Cassandra will automatically repartition as machines are added and removed from the cluster. Row store means that like relational databases, Cassandra organizes data by rows and columns. The Cassandra Query Language (CQL) is a close relative of SQL.

MariaDB

Started by core members of the original MySQL team, MariaDB actively works with outside developers to deliver the most featureful, stable, and sanely licensed open SQL server in the industry. MariaDB is designed as a drop-in replacement of MySQL(R) with more features, new storage engines, fewer bugs, and better performance.

RethinkDB

RethinkDB is built to store JSON documents, and scale to multiple machines with very little effort. It has a pleasant query language that supports really useful queries like table joins and group by, and is easy to setup and learn.

ArangoDB

A distributed free and open-source database with a flexible data model for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.

InfluxDB

InfluxDB is a scalable datastore for metrics, events, and real-time analytics. It has a built-in HTTP API so you don't have to write any server side code to get up and running. InfluxDB is designed to be scalable, simple to install and manage, and fast to get data in and out.

Related Comparisons

Apache Spark vs Memcached: What are the differences?

Introduction:

Apache Spark vs Memcached

Overview