My Job Glasses

Hello, For security and strategic reasons, we are migrating our apps from AWS/Google to a cloud provider with more security certifications and fewer functionalities, named Outscale. So far we have been using Google BigQuery as our data warehouse with ELT workflows (using Stitch and dbt ) and we need to migrate our data ecosystem to this new cloud provider.

We are setting up a Kubernetes cluster in our new cloud provider for our apps. Regarding the data warehouse, it's not clear if there are advantages/inconvenients about setting it up on kubernetes (apart from having to create node groups and tolerations with more ram/cpu). Also, we are not sure what's the best Open source or on-premise tool to use. The main requirement is that data must remain in the secure cluster, and no external entity (especially US) can have access to it. We have a dev cluster/environment and a production cluster/environment on this cloud.

Regarding the actual DWH usage

• Today we have ~1.5TB in BigQuery in production. We're going to run our initial rests with ~50-100GB of data for our test cluster
• Most of our data comes from other databases, so in most cases, we already have replicated sources somewhere, and there are only a handful of collections whose source is directly in the DWH (such as snapshots, some external data we've fetched at some point, google analytics, etc) and needs appropriate level of replication
• We are a team of 30-ish people, we do not have critical needs regarding analytics speed, and we do not need real time. We rebuild our DBT models 2-3 times a day and this usually proves enough

Apart from postgreSQL, I haven't really found open-source or on-premise alternatives for setting up a data warehouse, and running transformations with DBT. There is also the question of data ingestion, I've selected Airbyte and @meltano and I have troubles understanding if one of the 2 is better but Airbytes seems to have a bigger community.

What do you suggest regarding the data warehouse, and the ELT workflows ?

• Kubernetes or not kubernetes ?
• Postgresql or something else ? if postgre, what are the important configs you'd have in mind ?
• Airbyte/DBT or something else.

We build a Slack app using the Bolt framework from slack https://api.slack.com/tools/bolt, a Node.js express app. It allows us to easily implement some administration features so we can easily communicate with our backend services, and we don't have to develop any frontend app since Slack block kit will do this for us. It can act as a Chatbot or handle message actions and custom slack flows for our employees.

This app is deployed as a microservice on Amazon EC2 Container Service with AWS Fargate. It uses very little memory (and money) and can communicate easily with our backend services. Slack is connected to this app through a ALB ( AWS Elastic Load Balancing (ELB) )

After splitting our monolith into a Rails API + a React Redux.js frontend app, it became a necessity to monitor frontend errors. Our frontend application is not your typical website, and features a lot of interesting SPA mechanics that need to be followed closely (many async flows, redux-saga , etc.) in addition to regular browser incompatibility issues. Rollbar kicks in so that we can monitor every bug that happens on our frontend, and aggregate this with almost 0 work. The number of occurrences and affected browsers on each occurence helps us understand the priority and severity of bugs even when our users don't tell us about them, so we can decide whether we need to fix this bug that was encountered by 1k users in less than a few days days VERSUS telling this SINGLE user to switch browsers because he's using a very outdated version that no one else uses. Now we also use Rollbar with Rails, Sidekiq and even AWS Lambda errors since the interface is quite convenient.

We decided to use AWS Lambda for several serverless tasks such as

• Managing AWS backups
• Processing emails received on @{Amazon SES}|tool:17| and stored to @{Amazon S3}|tool:25| and notified via @{Amazon SNS}|tool:396|, so as to push a message on our @{Redis}|tool:1031| so our @{Sidekiq}|tool:1078| @{Rails}|tool:990| workers can process inbound emails
• Pushing some relevant @{Amazon CloudWatch}|tool:401| metrics and alarms to @{Slack}|tool:675|

We migrated from Amazon SQS + Shoryuken to Sidekiq in order to have at-most-once delivery out of the box and more flexibility.

The UI builtin Rails makes it smoother for development and QA. Through the sidekiq rails engine we can easily see & understand which job is/was/will be executed, and even get some stats for free. Compared to SQS, we lose in scalability (need to manage the underlying Redis instance) but this is not so critical right now for our business size and the PROs clearly outweigh the CONs. Plugins allow to easily add distributed CRON scheduled jobs in there for almost free, and this is a core feature for us, so we no longer need to maintain a "scheduler" instance and we make our CRON jobs more resilient. The Sidekiq UI can easily be tweaked and for instance we have added a column that translates the CRON syntax into a human readable string, so it's easy for our Q/A to check whether the job is scheduled appropriately.

We still use Amazon SQS for some other apps, but no longer for our main Rails app.