Alternatives to Beamer

Data science and engineering teams at Lyft maintain several big data pipelines that serve as the foundation for various types of analysis throughout the business.

Apache Airflow sits at the center of this big data infrastructure, allowing users to “programmatically author, schedule, and monitor data pipelines.” Airflow is an open source tool, and “Lyft is the very first Airflow adopter in production since the project was open sourced around three years ago.”

There are several key components of the architecture. A web UI allows users to view the status of their queries, along with an audit trail of any modifications the query. A metadata database stores things like job status and task instance status. A multi-process scheduler handles job requests, and triggers the executor to execute those tasks.

Airflow supports several executors, though Lyft uses CeleryExecutor to scale task execution in production. Airflow is deployed to three Amazon Auto Scaling Groups, with each associated with a celery queue.

Audit logs supplied to the web UI are powered by the existing Airflow audit logs as well as Flask signal.

Datadog, Statsd, Grafana, and PagerDuty are all used to monitor the Airflow system.

We are a young start-up with 2 developers and a team in India looking to choose our next ETL tool. We have a few processes in Azure Data Factory but are looking to switch to a better platform. We were debating Trifacta and Airflow. Or even staying with Azure Data Factory. The use case will be to feed data to front-end APIs.

Hi, We are looking to implement 2FA - so that users would be sent a Verification code over their Email and SMS to their phone.

We faced some limitations with Amazon SNS where we could either send the verification code to email OR to the phone number, while we want to send it to both.

We also are looking to make the 2FA more flexible by adding any other options later on.

What are the best alternatives to SNS for this use case and purpose? Looked at Twilio but want to explore other options before making a decision.

Would be great to know what the experience with Twilio has been, especially the limitations/issues with Twilio...

Appreciate any input from users of Twilio and others who have had similar use cases.

Searching for options for SMS that integrates with SiteLink and will allow personalization of text and tracking of both incoming/outgoing messages with reporting (Time, date, call#, etc) Have been looking at Twilio, and seems most leaning toward this. Are there any other options known that integrate into SiteLink? Also looked at Clickatell.

At StackShare we were discussing how to increase the retention of our newly signed up users. We hypothesized that if we made certain changes to the emails in our on-boarding process we could increase our retention and activation of users.

We decided to use sendwithus because it offered us the ability to A/B test our transactional emails. We also utilized the sendwithus analytics dashboard to gain real time insight into the performance of our email campaigns. Furthermore sendwithus has a Rails gem that allowed us to easily integrate the product into our application. We were also able to integrate sendwithus with our SendGrid account. #ABTestingAnalytics #TransactionalEmail

Nexmo vs Twilio ?

Back in the early days at SmartZip Analytics, that evaluation had - for whatever reason - been made by Product Management. Some developers might have been consulted, but we hadn't made the final call and some key engineering aspects of it were omitted.

When revamping the platform, I made sure to flip the decision process how it should be. Business provided an input but Engineering lead the way and has the final say on all implementation matters. My engineers and I decided on re-evaluating the criteria and vendor selection. Not only did we need SMS support, but were we not thinking about #VoiceAndSms support as the use cases evolved.

Also, on an engineering standpoint, SDK mattered. Nexmo didn't have any. Twilio did. No-one would ever want to re-build from scratch integration layers vendors should naturally come up with and provide their customers with.

Twilio won on all fronts. Including costs and implementation timelines. No-one even noticed the vendor switch.

Many years later, Twilio demonstrated its position as a leader by holding conferences in the Bay Area, announcing features like Twilio Functions. Even acquired Authy which we also used for 2FA. Twilio's growth has been amazing. Its recent acquisition of SendGrid continues to show it.

We decided to use AWS Lambda for several serverless tasks such as

• Managing AWS backups
• Processing emails received on Amazon SES and stored to Amazon S3 and notified via Amazon SNS, so as to push a message on our Redis so our Sidekiq Rails workers can process inbound emails
• Pushing some relevant Amazon CloudWatch metrics and alarms to Slack

I would like to know how I can implement a transactional email, or if it is possible to do so, like Mailchimp, using Amazon SES. I want to have the flexibility of creating emails like MailChimp, with a bulk email sending capability. Is it as simple with AWS SES as it is with MailChimp? If so, then how can I implement that for my own product? Thanks!

We've moved our transactional email away from Mandrill to Mailgun. We had continued using Mandrill after Mailchimp deprecated the service awhile back, because the amount of credits we were offered essentially made it free.

However, following a couple weeks of frequent downtime and poor service transparency from Mandrill, we decided it was time to make the switch. It appears they no longer had any engineers with the ability to identify the core problems.

Mailgun has been more reliable, yet not as reliable as we expected. We still see issues a few times per week with the API failing when we attempt to make a call. The Reporting UI is way better.

We've moved our transactional email away from Mandrill to Mailgun. We had continued using Mandrill after Mailchimp deprecated the service awhile back, because the amount of credits we were offered essentially made it free.

However, following a couple weeks of frequent downtime and poor service transparency from Mandrill, we decided it was time to make the switch. It appears they no longer had any engineers with the ability to identify the core problems.

Mailgun has been more reliable, yet not as reliable as we expected. We still see issues a few times per week with the API failing when we attempt to make a call. The Reporting UI is way better.

Hi, I've noticed my Mandrill emails are being received fine but my Mailchimp emails, about 75% are going into junk mail. I was wondering is it possible I have missed some sort of integration or can I send my Mailchimp marketing emails via mandrill?

Need help to somehow reduce the number of my emails going into junk mail, can someone help?

We decided to use AWS Lambda for several serverless tasks such as

• Managing AWS backups
• Processing emails received on Amazon SES and stored to Amazon S3 and notified via Amazon SNS, so as to push a message on our Redis so our Sidekiq Rails workers can process inbound emails
• Pushing some relevant Amazon CloudWatch metrics and alarms to Slack

Hi, We are looking to implement 2FA - so that users would be sent a Verification code over their Email and SMS to their phone.

We faced some limitations with Amazon SNS where we could either send the verification code to email OR to the phone number, while we want to send it to both.

We also are looking to make the 2FA more flexible by adding any other options later on.

What are the best alternatives to SNS for this use case and purpose? Looked at Twilio but want to explore other options before making a decision.

Would be great to know what the experience with Twilio has been, especially the limitations/issues with Twilio...

Appreciate any input from users of Twilio and others who have had similar use cases.