Maria Hatfield

Indexing in OpenSearch

The role of the OpenSearch Index can be described in two ways. Firstly, an Index is a mechanism to organize data in a user prescribed way. And secondly, the Index is a way to allocate data within an OpenSearch cluster. In this post we describe how to create, add to, delete, and reindex Indices in OpenSearch. We also include example code that you can copy and paste into your implementation.

Is Kafka a message queue?

Apache Kafka is not a traditional message queue. Kafka is a distributed messaging system that includes components of both a message queue and a publish-subscribe model. Kafka improves on the deficit of both the traditional message queue and pub-sub approaches, resulting in a fault tolerate, high throughput stream processing solution.

Learn more about the structure of Kafka at the link below.

Understanding Apache Pulsar's Two-Layer System. Apache Pulsar has two layers, a serving layer and a storage layer. The Pulsar brokers carry out the data serving. The BookKeeper bookies provide the storage.

Serving Layer: Role of Pulsar Brokers. Data is sent to Pulsar topics by producers, and read from Pulsar topics by consumers.

Multiple consumers can read from the same topic. For instance, a retail company might have a machine learning model and an alerting system both consuming messages from the same inventory database.

Each single partition topic has a single Pulsar broker that owns it. A topic with multiple partitions will be spread across many Pulsar brokers.

Continue reading about Pulsar's split serving and storage layers. https://dattell.com/data-architecture-blog/what-is-apache-pulsars-two-layer-system/

How to Query OpenSearch Using Boolean Queries

OpenSearch boolean queries find or match documents using boolean clauses. In the article linked below we describe how to construct a boolean query, or bool query. We will also work through several example OpenSearch bool queries.

Here let's discuss the four boolean clauses used for bool queries: filter, must, must_not, and should.

filter – Filter prunes the dataset; a document will either fit into a filter or be excluded by it. Filter queries can reduce datasets to a specific date range, location, or other exact match.

Filtering increases search performance because the OpenSearch cache stores filter queries. The next time a repeat filter query is run, the results get pulled from the cache. We will go into more depth about filtering below.

must – Must is like the “and” operator used when making a Google search. Must tells OpenSearch that document matches need to include all the queries that fall under the must clause. If you have more than one query, then all those queries need to match.

must-not – Mustnot is like the “not” operator used when making a Google search. It is the opposite of the must clause. Mustnot tells OpenSearch that document matches cannot include any of the queries that fall under the must_not clause.

should – It would be ideal for the matching documents to include all the queries in the should clause, but they don’t have to be included. Scoring ranks the matches.

Further down in this article we have a section on how OpenSearch uses scoring. Simply put, the more should queries that a matched document has, the higher the score for that document.

As a default setting, any match needs to contain at least one of the should queries. This minimum value is adjustable using the parameter minimumnumbershould_match.

Click on the link below to keep reading about how to query OpenSearch.

Understanding Apache Pulsar Functions

Apache Pulsar functions can process simple tasks, such as word counting. Handling computations natively within Pulsar can preserve operational simplicity by not requiring additional technology for processing.

However, Pulsar functions don't remove the need for separate technologies for more complex processes. For complex tasks you'll likely want to use Apache Heron, Apache Storm, or Apache Flink.

Check out the blog post linked below for more information on Pulsar functions.

Answers to Kafka Consumer Questions

In our latest blog post (linked below) we answer common questions about how Apache Kafka consumers work and important limitations on the number of consumers for an implementation.

Here are the questions we answer: 1.) Can a Kafka consumer read from multiple topics? 2.) How do consumer groups work? 3.) How does a consumer work? 4.) How many consumers can Kafka have? 5.) How to check if a Kafka consumer is running? 6.) How to check Kafka consumer lag?

Calculating Shard Count for OpenSearch Clusters

An important component of OpenSearch cluster optimization is assigning the proper number of shards to your cluster. OpenSearch shards enable parallelization of data processing across both a single node and multiple nodes.

In the article linked below we walk through the structure of an OpenSearch cluster and explain how to calculate the optimal number of shards for your cluster.

Keep in mind there can always be exceptions to any rule. And we talk through one of those scenarios as well.

Vector Search for OpenSearch

Here let's compare the three options available for vector search using the OpenSearch plugin.

Approximate k-NN. This method is ideal for large indices (100k vectors or more). It is a low-latency approach and is scalable. Yet, its fast performance compromises search accuracy. It also does not support pre-filtering.

Script Score k-NN. Script Score is a brute force, exact k-NN search. With brute force search the algorithm completes an exhaustive search. Brute force examines all possible solutions. This approach is more accurate than Approximate k-NN. But, it results in high latencies for large datasets. Script Score allows you to search on a subset of your vectors, also known as pre-filter search. And it can also support binary fields.

Painless Extensions. This is a custom method for complex use cases. As with Script Score, Painless Extensions is a brute force, exact k-NN search. And it allows for pre-filtering.

The Painless Extensions method also supports the use of distance functions. There are many different methods for calculating the distance between two points. The choice of distance function can improve classification accuracy.

Learn more in the linked post below.

Pulsar vs Kafka

Both Pulsar and Kafka can guarantee the delivery of messages to intended recipients. Yet, they differ in complexity and modularity, latency and throughput. Another key distinction: Kafka ensures exactly once processing at the consumer level. Pulsar only confirms exactly once in the Pulsar store.

Whether using Kafka or Pulsar, we recommend you focus on reliability. You can do this by creating an implementation that is as simple as possible.

In the linked post, we cover 8 ways to compare Kafka and Pulsar. Let us know what you think.

Elasticsearch versus OpenSearch

In the post linked below we evaluate free and open source OpenSearch, free Elasticsearch, and premium Elasticsearch. We compare the search engines for security features, support/services, documentation, ML capabilities, tools, plugins, and community.

Click the link below to learn how the two databases compare.