Alternatives to Astronomer

I am working on a project that grabs a set of input data from AWS S3, pre-processes and divvies it up, spins up 10K batch containers to process the divvied data in parallel on AWS Batch, post-aggregates the data, and pushes it to S3.

I already have software patterns from other projects for Airflow + Batch but have not dealt with the scaling factors of 10k parallel tasks. Airflow is nice since I can look at which tasks failed and retry a task after debugging. But dealing with that many tasks on one Airflow EC2 instance seems like a barrier. Another option would be to have one task that kicks off the 10k containers and monitors it from there.

I have no experience with AWS Step Functions but have heard it's AWS's Airflow. There looks to be plenty of patterns online for Step Functions + Batch. Do Step Functions seem like a good path to check out for my use case? Do you get the same insights on failing jobs / ability to retry tasks as you do with Airflow?

I am looking for an open-source scheduler tool with cross-functional application dependencies. Some of the tasks I am looking to schedule are as follows:

1. Trigger Matillion ETL loads
2. Trigger Attunity Replication tasks that have downstream ETL loads
3. Trigger Golden gate Replication Tasks
4. Shell scripts, wrappers, file watchers
5. Event-driven schedules

I have used Airflow in the past, and I know we need to create DAGs for each pipeline. I am not familiar with Jenkins, but I know it works with configuration without much underlying code. I want to evaluate both and appreciate any advise

Our primary source of monitoring and alerting is Datadog. We’ve got prebuilt dashboards for every scenario and integration with PagerDuty to manage routing any alerts. We’ve definitely scaled past the point where managing dashboards is easy, but we haven’t had time to invest in using features like Anomaly Detection. We’ve started using Honeycomb for some targeted debugging of complex production issues and we are liking what we’ve seen. We capture any unhandled exceptions with Rollbar and, if we realize one will keep happening, we quickly convert the metrics to point back to Datadog, to keep Rollbar as clean as possible.

We use Segment to consolidate all of our trackers, the most important of which goes to Amplitude to analyze user patterns. However, if we need a more consolidated view, we push all of our data to our own data warehouse running PostgreSQL; this is available for analytics and dashboard creation through Looker.

Functionally, Amplitude and Mixpanel are incredibly similar. They both offer almost all the same functionality around tracking and visualizing user actions for analytics. You can track A/B test results in both. We ended up going with Amplitude at BaseDash because it has a more generous free tier for our uses (10 million actions per month, versus Mixpanel's 1000 monthly tracked users).

Segment isn't meant to compete with these tools, but instead acts as an API to send actions to them, and other analytics tools. If you're just sending event data to one of these tools, you probably don't need Segment. If you're using other analytics tools like Google Analytics and FullStory, Segment makes it easy to send events to all your tools at once.

Hi,

This is a question for best practice regarding Segment and Google Tag Manager. I would love to use Segment and GTM together when we need to implement a lot of additional tools, such as Amplitude, Appsfyler, or any other engagement tool since we can send event data without additional SDK implementation, etc.

So, my question is, if you use Segment and Google Tag Manager, how did you define what you will push through Segment and what will you push through Google Tag Manager? For example, when implementing a Facebook Pixel or any other 3rd party marketing tag?

From my point of view, implementing marketing pixels should stay in GTM because of the tag/trigger control.

If you are using Segment and GTM together, I would love to learn more about your best practice.

Thanks!