Laurent Valdes

Hi !

I would say it depends on three things:

• the volume (number of events by seconds)
• the versatility you need.
• the vendor lock on

If volume is an issue then maybe flink may be a better option. Flink is distributed and is architectured as such.

If you are ok with having everything based on Kafka then maybe Kafka streams is a better option, but if at some point you need to integrate with several systems maybe you should consider flink.

If you are okay with sticking with confluent tools it’s okay, but beware of technology. Flink is more generalist than Kafka streams.

I won’t recommend any of those tools if you plan to run thousand of analytical queries each day.

In my opinion you would better use something like Druid or Clickhouse than using bigtable or bigquery

Why ?

• first bigtable is not a database made for analytics. Or maybe you will have a way to format your data so that it can be partitioned and fitted into small rows.

• bigquery has a cost and I would say it is not the right tool for the job, despite being able to handle the load. It will work but given the datasize and the price I would say you don’t need it.

I would better launch a clickhouse install in order to prevent vendor lockdown https://clickhouse-dashboard.cube.dev/how-to-run-click-house

For less than 10 GB you can use bigquery here as this is not considered a big data load (requests can be processed on a single pc) But if you want to process more (like 1 TB) I advise to use something else as scan costs tend to be high on BQ

Both depending on the workload. Usually redis is pretty good at in-memory applications and can do key-based persistence. Postgres is more made for SQL and can do key value persistence as well. I would rather choose PostgreSQL, with a bit of redis for specific parts.