Hi Rana, good question! From my Firebase experience, 3 million records is not too big at all, as long as the cost is within reason for you. With Firebase you will be able to access the data from anywhere, including an android app, and implement fine-grained security with JSON rules. The real-time-ness works perfectly. As a fully managed database, Firebase really takes care of everything. The only thing to watch out for is if you need complex query patterns - Firestore (also in the Firebase family) can be a better fit there.

To answer question 2: the right answer will depend on what's most important to you. Algolia is like Firebase is that it is fully-managed, very easy to set up, and has great SDKs for Android. Algolia is really a full-stack search solution in this case, and it is easy to connect with your Firebase data. Bear in mind that Algolia does cost money, so you'll want to make sure the cost is okay for you, but you will save a lot of engineering time and never have to worry about scale. The search-as-you-type performance with Algolia is flawless, as that is a primary aspect of its design. Elasticsearch can store tons of data and has all the flexibility, is hosted for cheap by many cloud services, and has many users. If you haven't done a lot with search before, the learning curve is higher than Algolia for getting the results ranked properly, and there is another learning curve if you want to do the DevOps part yourself. Both are very good platforms for search, Algolia shines when buliding your app is the most important and you don't want to spend many engineering hours, Elasticsearch shines when you have a lot of data and don't mind learning how to run and optimize it.

Rana - we use Cloud Firestore at our startup. It handles many million records without any issues. It provides you the same set of features that the Firebase Realtime Database provides on top of the indexing and security trims. The only thing to watch out for is to make sure your Cloud Functions have proper exception handling and there are no infinite loop in the code. This will be too costly if not caught quickly.

For search; Algolia is a great option, but cost is a real consideration. Indexing large number of records can be cost prohibitive for most projects. Elasticsearch is a solid alternative, but requires a little additional work to configure and maintain if you want to self-host.

Hope this helps.

you can use aws glue service to convert you pipe format data to parquet format , and thus you can achieve data compression . Now you should choose Redshift to copy your data as it is very huge. To manage your data, you should partition your data in S3 bucket and also divide your data across the redshift cluster

First of all you should make your choice upon Redshift or Athena based on your use case since they are two very diferent services - Redshift is an enterprise-grade MPP Data Warehouse while Athena is a SQL layer on top of S3 with limited performance. If performance is a key factor, users are going to execute unpredictable queries and direct and managing costs are not a problem I'd definitely go for Redshift. If performance is not so critical and queries will be predictable somewhat I'd go for Athena.

Once you select the technology you'll need to optimize your data in order to get the queries executed as fast as possible. In both cases you may need to adapt the data model to fit your queries better. In the case you go for Athena you'd also proabably need to change your file format to Parquet or Avro and review your partition strategy depending on your most frequent type of query. If you choose Redshift you'll need to ingest the data from your files into it and maybe carry out some tuning tasks for performance gain.

I'll recommend Redshift for now since it can address a wider range of use cases, but we could give you better advice if you described your use case in depth.

It depend of the nature of your data (structured or not?) and of course your queries (ad-hoc or predictible?). For example you can look at partitioning and columnar format to maximize MPP capabilities for both Athena and Redshift

you can change your PSV fomat data to parquet file format with AWS GLUE and then your query performance will be improved