Hi Teja, Jenkins is more a CI/CD tool for triggering build/test and other CD tasks, from what you're describing you may be able to get along with #Jenkins and add lot's of plugins and create pipelines. But, eventually you're going to need to know #Groovy language to orchestrate all those tasks which is going to be similar to what you do with #Airflow, So, IMHO , Airflow is more for production scheduled tasks and Jenkins is more for CI/CD non-production tasks.

For 1 - 3, Airflow is a better solution.

Cron and OSQuery may be better solutions for 4 and 5, depending on what you're actually trying to do.

In either case, Jenkins is more trouble than it's worth for these types of workloads.

If your source code is on GitHub, also take a look at Github actions. https://github.com/features/actions

Google cloud build can help you. It is hosted on cloud and also provide reasonable free quota.

I would definetly recommend Concourse to you, as it is one of the most advanced modern methods of making CI/CD while Jenkins is an old monolithic dinosaur. Concourse itself is cloudnative and containerbased which helps you to build simple, high-performance and scalable CI/CD pipelines. In my opinion, the only lack of skills you have with Concourse is your own knowledge of how to build pipelines and automate things. Technincally there is no lack, i would even say you can extend it way more easily. But as a Con it is more easy to interact with Jenkins if you are only used to UIs. Concourse needs someone which is capable of using CLIs.

For a non-streaming approach:

You could consider using more checkpoints throughout your spark jobs. Furthermore, you could consider separating your workload into multiple jobs with an intermittent data store (suggesting cassandra or you may choose based on your choice and availability) to store results , perform aggregations and store results of those.

Spark Job 1 - Fetch Data From 10 URLs and store data and metadata in a data store (cassandra) Spark Job 2..n - Check data store for unprocessed items and continue the aggregation

Alternatively for a streaming approach: Treating your data as stream might be useful also. Spark Streaming allows you to utilize a checkpoint interval - https://spark.apache.org/docs/latest/streaming-programming-guide.html#checkpointing

We use CircleCI because of the better value it provides in its plans. I'm sure we could have used Travis just as easily but we found CircleCI's pricing to be more reasonable. In the two years since we signed up, the service has improved. CircleCI is always innovating and iterating on their platform. We have been very satisfied.

As the maintainer of the Karate DSL open-source project - I found Travis CI very easy to integrate into the GitHub workflow and it has been steady sailing for more than 2 years now ! It works well for Java / Apache Maven projects and we were able to configure it to use the latest Oracle JDK as per our needs. Thanks to the Travis CI team for this service to the open-source community !

I use Google Cloud Build because it's my first foray into the CICD world(loving it so far), and I wanted to work with something GCP native to avoid giving permissions to other SaaS tools like CircleCI and Travis CI.

I really like it because it's free for the first 120 minutes, and it's one of the few CICD tools that enterprises are open to using since it's contained within GCP.

One of the unique things is that it has the Kaniko cache, which speeds up builds by creating intermediate layers within the docker image vs. pushing the full thing from the start. Helpful when you're installing just a few additional dependencies.

Feel free to checkout an example: Cloudbuild Example

I use Travis CI because of various reasons - 1. Cloud based system so no dedicated server required, and you do not need to administrate it. 2. Easy YAML configuration. 3. Supports Major Programming Languages. 4. Support of build matrix 6. Supports AWS, Azure, Docker, Heroku, Google Cloud, Github Pages, PyPi and lot more. 7. Slack Notifications.

You are probably looking at another hosted solution: Jenkins is a good tool but it way too work intensive to be used as just a backup solution.

I have good experience with Circle-CI, Codeship, Drone.io and Travis (as well as problematic experiences with all of them), but my go-to tool is Gitlab CI: simple, powerful and if you have problems with their limitations or pricing, you can always install runners somewhere and use Gitlab just for scheduling and management. Even if you don't host your git repository at Gitlab, you can have Gitlab pull changes automatically from wherever you repo lives.

If you are considering Jenkins I would recommend at least checking out Buildkite. The agents are self-hosted (like Jenkins) but the interface is hosted for you. It meshes up some of the things I like about hosted services (pipeline definitions in YAML, managed interface and authentication) with things I like about Jenkins (local customizable agent images, secrets only on own instances, custom agent level scripts, sizing instances to your needs).