Alternatives to dbt logo

Alternatives to dbt

act, Airflow, Looker, Apache Spark, and Slick are the most popular alternatives and competitors to dbt.
199
208
+ 1
1

What is dbt and what are its top alternatives?

It enables analytics engineers to transform data in their warehouses by simply writing select statements. It handles turning these select statements into tables and views.
dbt is a tool in the Database Tools category of a tech stack.

Top Alternatives to dbt

  • act

    act

    Rather than having to commit/push every time you want test out the changes you are making to your .github/workflows/ files (or for any changes to embedded GitHub actions), you can use this tool to run the actions locally. The environment variables and filesystem are all configured to match what GitHub provides. ...

  • Airflow

    Airflow

    Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command lines utilities makes performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress and troubleshoot issues when needed. ...

  • Looker

    Looker

    We've built a unique data modeling language, connections to today's fastest analytical databases, and a service that you can deploy on any infrastructure, and explore on any device. Plus, we'll help you every step of the way. ...

  • Apache Spark

    Apache Spark

    Spark is a fast and general processing engine compatible with Hadoop data. It can run in Hadoop clusters through YARN or Spark's standalone mode, and it can process data in HDFS, HBase, Cassandra, Hive, and any Hadoop InputFormat. It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning. ...

  • Slick

    Slick

    It is a modern database query and access library for Scala. It allows you to work with stored data almost as if you were using Scala collections while at the same time giving you full control over when a database access happens and which data is transferred. ...

  • Spring Data

    Spring Data

    It makes it easy to use data access technologies, relational and non-relational databases, map-reduce frameworks, and cloud-based data services. This is an umbrella project which contains many subprojects that are specific to a given database. ...

  • DataGrip

    DataGrip

    A cross-platform IDE that is aimed at DBAs and developers working with SQL databases. ...

  • Microsoft SQL Server Management Studio

    Microsoft SQL Server Management Studio

    It is an integrated environment for managing any SQL infrastructure, from SQL Server to Azure SQL Database. It provides tools to configure, monitor, and administer instances of SQL Server and databases. Use it to deploy, monitor, and upgrade the data-tier components used by your applications, as well as build queries and scripts. ...

dbt alternatives & related posts

act logo

act

5
19
0
Run your GitHub Actions locally
5
19
+ 1
0
PROS OF ACT
    Be the first to leave a pro
    CONS OF ACT
      Be the first to leave a con

      related act posts

      Airflow logo

      Airflow

      1.2K
      2K
      113
      A platform to programmaticaly author, schedule and monitor data pipelines, by Airbnb
      1.2K
      2K
      + 1
      113
      PROS OF AIRFLOW
      • 44
        Features
      • 13
        Task Dependency Management
      • 12
        Beautiful UI
      • 11
        Cluster of workers
      • 10
        Extensibility
      • 5
        Open source
      • 4
        Python
      • 4
        Complex workflows
      • 3
        K
      • 2
        Custom operators
      • 2
        Dashboard
      • 2
        Good api
      • 1
        Apache project
      CONS OF AIRFLOW
      • 1
        Open source - provides minimum or no support
      • 1
        Logical separation of DAGs is not straight forward
      • 1
        Running it on kubernetes cluster relatively complex
      • 1
        Observability is not great when the DAGs exceed 250

      related Airflow posts

      Shared insights
      on
      JenkinsJenkinsAirflowAirflow

      I am looking for an open-source scheduler tool with cross-functional application dependencies. Some of the tasks I am looking to schedule are as follows:

      1. Trigger Matillion ETL loads
      2. Trigger Attunity Replication tasks that have downstream ETL loads
      3. Trigger Golden gate Replication Tasks
      4. Shell scripts, wrappers, file watchers
      5. Event-driven schedules

      I have used Airflow in the past, and I know we need to create DAGs for each pipeline. I am not familiar with Jenkins, but I know it works with configuration without much underlying code. I want to evaluate both and appreciate any advise

      See more
      Shared insights
      on
      AWS Step FunctionsAWS Step FunctionsAirflowAirflow

      I am working on a project that grabs a set of input data from AWS S3, pre-processes and divvies it up, spins up 10K batch containers to process the divvied data in parallel on AWS Batch, post-aggregates the data, and pushes it to S3.

      I already have software patterns from other projects for Airflow + Batch but have not dealt with the scaling factors of 10k parallel tasks. Airflow is nice since I can look at which tasks failed and retry a task after debugging. But dealing with that many tasks on one Airflow EC2 instance seems like a barrier. Another option would be to have one task that kicks off the 10k containers and monitors it from there.

      I have no experience with AWS Step Functions but have heard it's AWS's Airflow. There looks to be plenty of patterns online for Step Functions + Batch. Do Step Functions seem like a good path to check out for my use case? Do you get the same insights on failing jobs / ability to retry tasks as you do with Airflow?

      See more
      Looker logo

      Looker

      407
      452
      9
      Pioneering the next generation of BI, data discovery & data analytics
      407
      452
      + 1
      9
      PROS OF LOOKER
      • 4
        Real time in app customer chat support
      • 4
        GitHub integration
      • 1
        Reduces the barrier of entry to utilizing data
      CONS OF LOOKER
      • 1
        Price

      related Looker posts

      Mohan Ramanujam

      We are a consumer mobile app IOS/Android startup. The app is instrumented with branch and Firebase. We use Google BigQuery. We are looking at tools that can support engagement and cohort analysis at an early stage price which we can grow with. Data Studio is the default but it would seem Looker provides more power. We don't have much insight into Amplitude other than the fact it is a popular PM tool. Please provide some insight.

      See more
      Apache Spark logo

      Apache Spark

      2.4K
      2.8K
      132
      Fast and general engine for large-scale data processing
      2.4K
      2.8K
      + 1
      132
      PROS OF APACHE SPARK
      • 58
        Open-source
      • 48
        Fast and Flexible
      • 7
        One platform for every big data problem
      • 6
        Easy to install and to use
      • 6
        Great for distributed SQL like applications
      • 3
        Works well for most Datascience usecases
      • 2
        Machine learning libratimery, Streaming in real
      • 2
        In memory Computation
      • 0
        Interactive Query
      CONS OF APACHE SPARK
      • 3
        Speed

      related Apache Spark posts

      Eric Colson
      Chief Algorithms Officer at Stitch Fix · | 21 upvotes · 2M views

      The algorithms and data infrastructure at Stitch Fix is housed in #AWS. Data acquisition is split between events flowing through Kafka, and periodic snapshots of PostgreSQL DBs. We store data in an Amazon S3 based data warehouse. Apache Spark on Yarn is our tool of choice for data movement and #ETL. Because our storage layer (s3) is decoupled from our processing layer, we are able to scale our compute environment very elastically. We have several semi-permanent, autoscaling Yarn clusters running to serve our data processing needs. While the bulk of our compute infrastructure is dedicated to algorithmic processing, we also implemented Presto for adhoc queries and dashboards.

      Beyond data movement and ETL, most #ML centric jobs (e.g. model training and execution) run in a similarly elastic environment as containers running Python and R code on Amazon EC2 Container Service clusters. The execution of batch jobs on top of ECS is managed by Flotilla, a service we built in house and open sourced (see https://github.com/stitchfix/flotilla-os).

      At Stitch Fix, algorithmic integrations are pervasive across the business. We have dozens of data products actively integrated systems. That requires serving layer that is robust, agile, flexible, and allows for self-service. Models produced on Flotilla are packaged for deployment in production using Khan, another framework we've developed internally. Khan provides our data scientists the ability to quickly productionize those models they've developed with open source frameworks in Python 3 (e.g. PyTorch, sklearn), by automatically packaging them as Docker containers and deploying to Amazon ECS. This provides our data scientist a one-click method of getting from their algorithms to production. We then integrate those deployments into a service mesh, which allows us to A/B test various implementations in our product.

      For more info:

      #DataScience #DataStack #Data

      See more
      Conor Myhrvold
      Tech Brand Mgr, Office of CTO at Uber · | 7 upvotes · 1M views

      Why we built Marmaray, an open source generic data ingestion and dispersal framework and library for Apache Hadoop :

      Built and designed by our Hadoop Platform team, Marmaray is a plug-in-based framework built on top of the Hadoop ecosystem. Users can add support to ingest data from any source and disperse to any sink leveraging the use of Apache Spark . The name, Marmaray, comes from a tunnel in Turkey connecting Europe and Asia. Similarly, we envisioned Marmaray within Uber as a pipeline connecting data from any source to any sink depending on customer preference:

      https://eng.uber.com/marmaray-hadoop-ingestion-open-source/

      (Direct GitHub repo: https://github.com/uber/marmaray Kafka Kafka Manager )

      See more
      Slick logo

      Slick

      8.6K
      660
      0
      Database query and access library for Scala
      8.6K
      660
      + 1
      0
      PROS OF SLICK
        Be the first to leave a pro
        CONS OF SLICK
          Be the first to leave a con

          related Slick posts

          Spring Data logo

          Spring Data

          453
          311
          0
          Provides a consistent approach to data access – relational, non-relational, map-reduce, and beyond
          453
          311
          + 1
          0
          PROS OF SPRING DATA
            Be the first to leave a pro
            CONS OF SPRING DATA
              Be the first to leave a con

              related Spring Data posts

              Остап Комплікевич

              I need some advice to choose an engine for generation web pages from the Spring Boot app. Which technology is the best solution today? 1) JSP + JSTL 2) Apache FreeMarker 3) Thymeleaf Or you can suggest even other perspective tools. I am using Spring Boot, Spring Web, Spring Data, Spring Security, PostgreSQL, Apache Tomcat in my project. I have already tried to generate pages using jsp, jstl, and it went well. However, I had huge problems via carrying already created static pages, to jsp format, because of syntax. Thanks.

              See more
              DataGrip logo

              DataGrip

              365
              382
              14
              A database IDE for professional SQL developers
              365
              382
              + 1
              14
              PROS OF DATAGRIP
              • 4
                Works on Linux, Windows and MacOS
              • 2
                Wide range of DBMS support
              • 1
                Code completion
              • 1
                Generate ERD
              • 1
                Quick-fixes using keyboard shortcuts
              • 1
                Code analysis
              • 1
                Database introspection on 21 different dbms
              • 1
                Export data using a variety of formats using open api
              • 1
                Import data
              • 1
                Diff viewer
              CONS OF DATAGRIP
                Be the first to leave a con

                related DataGrip posts

                Microsoft SQL Server Management Studio logo

                Microsoft SQL Server Management Studio

                362
                279
                0
                An integrated environment for managing any SQL infrastructure
                362
                279
                + 1
                0
                PROS OF MICROSOFT SQL SERVER MANAGEMENT STUDIO
                  Be the first to leave a pro
                  CONS OF MICROSOFT SQL SERVER MANAGEMENT STUDIO
                    Be the first to leave a con

                    related Microsoft SQL Server Management Studio posts

                    Kelsey Doolittle

                    We have a 138 row, 1700 column database likely to grow at least a row and a column every week. We are mostly concerned with how user-friendly the graphical management tools are. I understand MySQL has MySQL WorkBench, and Microsoft SQL Server has Microsoft SQL Server Management Studio. We have about 6 months to migrate our Excel database to one of these DBMS, and continue (hopefully manually) importing excel files from then on. Any tips appreciated!

                    See more