Alternatives to Huginn

Data science and engineering teams at Lyft maintain several big data pipelines that serve as the foundation for various types of analysis throughout the business.

Apache Airflow sits at the center of this big data infrastructure, allowing users to “programmatically author, schedule, and monitor data pipelines.” Airflow is an open source tool, and “Lyft is the very first Airflow adopter in production since the project was open sourced around three years ago.”

There are several key components of the architecture. A web UI allows users to view the status of their queries, along with an audit trail of any modifications the query. A metadata database stores things like job status and task instance status. A multi-process scheduler handles job requests, and triggers the executor to execute those tasks.

Airflow supports several executors, though Lyft uses CeleryExecutor to scale task execution in production. Airflow is deployed to three Amazon Auto Scaling Groups, with each associated with a celery queue.

Audit logs supplied to the web UI are powered by the existing Airflow audit logs as well as Flask signal.

Datadog, Statsd, Grafana, and PagerDuty are all used to monitor the Airflow system.

We are a young start-up with 2 developers and a team in India looking to choose our next ETL tool. We have a few processes in Azure Data Factory but are looking to switch to a better platform. We were debating Trifacta and Airflow. Or even staying with Azure Data Factory. The use case will be to feed data to front-end APIs.

Hi, I went through a comprehensive analysis - of headless/api content management systems - essentially to store content "bits" and publish them where needed (website, 3rd party sites, social media, etc.). I had considered many other solutions but ultimately chose Directus. I believe that was a good choice.

I had strongly considered Contentful, Strapi, Sanity, and hygraph. Hygraph came in #2 and contentful #3.

Ultimately I liked directus for:

(1) time in business

(2) open source

(3) integration with n8n and Pipedream

(4) pricing

(5) extensibility

Thoughts? Was this a good choice? We have many WordPress sites we're not (at least now) looking to replace with Directus, but instead to push to.

I'd love some feedback.

Back in 2014, I was given an opportunity to re-architect SmartZip Analytics platform, and flagship product: SmartTargeting. This is a SaaS software helping real estate professionals keeping up with their prospects and leads in a given neighborhood/territory, finding out (thanks to predictive analytics) who's the most likely to list/sell their home, and running cross-channel marketing automation against them: direct mail, online ads, email... The company also does provide Data APIs to Enterprise customers.

I had inherited years and years of technical debt and I knew things had to change radically. The first enabler to this was to make use of the cloud and go with AWS, so we would stop re-inventing the wheel, and build around managed/scalable services.

For the SaaS product, we kept on working with Rails as this was what my team had the most knowledge in. We've however broken up the monolith and decoupled the front-end application from the backend thanks to the use of Rails API so we'd get independently scalable micro-services from now on.

Our various applications could now be deployed using AWS Elastic Beanstalk so we wouldn't waste any more efforts writing time-consuming Capistrano deployment scripts for instance. Combined with Docker so our application would run within its own container, independently from the underlying host configuration.

Storage-wise, we went with Amazon S3 and ditched any pre-existing local or network storage people used to deal with in our legacy systems. On the database side: Amazon RDS / MySQL initially. Ultimately migrated to Amazon RDS for Aurora / MySQL when it got released. Once again, here you need a managed service your cloud provider handles for you.

Future improvements / technology decisions included:

Caching: Amazon ElastiCache / Memcached CDN: Amazon CloudFront Systems Integration: Segment / Zapier Data-warehousing: Amazon Redshift BI: Amazon Quicksight / Superset Search: Elasticsearch / Amazon Elasticsearch Service / Algolia Monitoring: New Relic

As our usage grows, patterns changed, and/or our business needs evolved, my role as Engineering Manager then Director of Engineering was also to ensure my team kept on learning and innovating, while delivering on business value.

One of these innovations was to get ourselves into Serverless : Adopting AWS Lambda was a big step forward. At the time, only available for Node.js (Not Ruby ) but a great way to handle cost efficiency, unpredictable traffic, sudden bursts of traffic... Ultimately you want the whole chain of services involved in a call to be serverless, and that's when we've started leveraging Amazon DynamoDB on these projects so they'd be fully scalable.

When starting a new company and building a new product w/ limited engineering we chose to optimize for expertise and rapid development, landing on Rails API, w/ AngularJS on the front.

The reality is that we're building a CRUD app, so we considered going w/ vanilla Rails MVC to optimize velocity early on (it may not be sexy, but it gets the job done). Instead, we opted to split the codebase to allow for a richer front-end experience, focus on skill specificity when hiring, and give us the flexibility to be consumed by multiple clients in the future.

We also considered .NET core or Node.js for the API layer, and React on the front-end, but our experiences dealing with mature Node APIs and the rapid-fire changes that comes with state management in React-land put us off, given our level of experience with those tools.

We're using GitHub and Trello to track issues and projects, and a plethora of other tools to help the operational team, like Zapier, MailChimp, Google Drive with some basic Vue.js & HTML5 apps for smaller internal-facing web projects.

I've heard that I have the ability to write well, at times. When it flows, it flows. I decided to start blogging in 2013 on Blogger. I started a company and joined BizPark with the Microsoft Azure allotment. I created a WordPress blog and did a migration at some point. A lot happened in the time after that migration but I stopped coding and changed cities during tumultuous times that taught me many lessons concerning mental health and productivity. I eventually graduated from BizSpark and outgrew the credit allotment. That killed the WordPress blog.

I blogged about writing again on the existing Blogger blog but it didn't feel right. I looked at a few options where I wouldn't have to worry about hosting cost indefinitely and Jekyll stood out with GitHub Pages. The Importer was fairly straightforward for the existing blog posts.

Todo * Set up redirects for all posts on blogger. The URI format is different so a complete redirect wouldn't work. Although, there may be something in Jekyll that could manage the redirects. I did notice the old URLs were stored in the front matter. I'm working on a command-line Ruby gem for the current plan. * I did find some of the lost WordPress posts on archive.org that I downloaded with the waybackmachinedownloader. I think I might write an importer for that. * I still have a few Disqus comment threads to map

hello guys, I need your help. I created a website, I've been using Elementor forever, but yesterday I bought a template after I made the purchase I knew I made a mistake, cause the template was in HTML, can anyone please show me how to put this HTML template in my WordPress so it will be the face of my website, thank you in advance.

which of the ads platform pays better? What about PurpleAds?

Google AdSense has refused to post ads on my site.

Really can not decide which one to add. Google AdSense email say that they are ready to show ads... Taboola is on review.