yurisugano

The answer to your question requires a careful study of your project's scope, demands and resources. A one-fit-all answer will be misleading at best. Here, I try to summarize a few points you should consider before making a decision. Then, I will justify my personal recommendations. There recommendations are tools that could be one of many potential solutions to your data design problem, based on your brief description.

Threshold for Distributed Processing:

• The threshold for distributed processing depends on data volume, complexity, and system performance.
  1. To determine if this threshold is reached, assess:
     • System performance under current loads.
     • Query execution times and resource utilization.
     • The scalability requirements of your system.
     • The complexity of the data and the computations being performed.

Categories of Operations:

• The answer largely depends on the kinds of operations you are performing:
  1. Data Transformation: Such as normalizing data formats, cleaning data, and transforming data structures.
  2. Aggregation and Summary Statistics: Useful for generating reports or insights from large datasets.
  3. Complex Joins: Involving multiple datasets, which can be computationally intensive.
  4. Predictive Analytics and Machine Learning: Where large volumes of data are used to train models.

Then, choose a framework that best suits your needs. Specifically for big data applications, I have experience with Apache Spark, and I have seen enormous potential with tools such as Delta Lake, so I believe they provide a versatile combination for different use cases. At the same time, I know PostgreSQL handles intensive demands extremely well, and can be seen in the stack of many top performing tech companies, likely in their business intelligence and reporting demands.

Another important hint is to plan a comprehensive stack to benefit from the advantages of different frameworks for different use cases. I can definitely envision a system where the three technologies interact to leverage the best of their abilities. As much as you can, make them your own :)