Celery

Celery

Application and Data / Data Stores / Message Queue
Founder at AlterEstate·

Automations are what makes a CRM powerful. With Celery and RabbitMQ we've been able to make powerful automations that truly works for our clients. Such as for example, automatic daily reports, reminders for their activities, important notifications regarding their client activities and actions on the website and more.

We use Celery basically for everything that needs to be scheduled for the future, and using RabbitMQ as our Queue-broker is amazing since it fully integrates with Django and Celery storing on our database results of the tasks done so we can see if anything fails immediately.

READ MORE
6 upvotes·357K views
Software Engineer ·
Needs advice
on
Amazon SQSAmazon SQSKubernetesKubernetes
and
RabbitMQRabbitMQ

Hi! I am creating a scraping system in Django, which involves long running tasks between 1 minute & 1 Day. As I am new to Message Brokers and Task Queues, I need advice on which architecture to use for my system. ( Amazon SQS, RabbitMQ, or Celery). The system should be autoscalable using Kubernetes(K8) based on the number of pending tasks in the queue.

READ MORE
7 upvotes·286.8K views
Replies (1)
Founder at Odix·
Recommends
Kafka

Hello, i highly recommend Apache Kafka, to me it's the best. You can deploy it in cluster mode inside K8S, thus you can have a Highly available system (also auto scalable).

Good luck

READ MORE
1 upvote·195K views
CTO and Software Architect at Medstrat·
Needs advice
on
AppOpticsAppOptics
and
DatadogDatadog
in

We use AppOptics. I am curious what are the current leaders for APM for small companies (50 employees) that use Python, MariaDB, RabbitMQ, and Google Cloud Storage. We run both Celery and Gunicorn services. We are considering Datadog or some other deep code profiling tool that can spot I/O, DB, or other response time/request rate issues

READ MORE
3 upvotes·81.2K views
Replies (1)
Recommends
Instana

If you want to get deep insights and fast issue resolution have a look at INSTANA.

There is a public sandbox to get first insights and feeling for the tool. If you like it you can also run a free trial if you like.

READ MORE
Instana - Getting Started with APM (instana.com)
2 upvotes·2 comments·3.8K views
Gal Cohen
Gal Cohen
·
February 2nd 2021 at 7:47AM

We are running Python & Celery, our stack is based on AWS ECS. We are using NewRelic. This tool is just amazing, both for API and Offline workers. It would provide any metric I was looking for, including a profiler, SLA / SLO dashboards, infrastructure metrics. It has alerting capability that is easily integrated with Pingdom / PagerDuty / Webhooks.

·
Reply
Greg Smethells
Greg Smethells
·
September 19th 2020 at 8:29PM

Thanks, I’ll take a look.

·
Reply
Needs advice
on
CeleryCelery
and
RabbitMQRabbitMQ

I am just a beginner at these two technologies.

Problem statement: I am getting lakh of users from the sequel server for whom I need to create caches in MongoDB by making different REST API requests.

Here these users can be treated as messages. Each REST API request is a task.

I am confused about whether I should go for RabbitMQ alone or Celery.

If I have to go with RabbitMQ, I prefer to use python with Pika module. But the challenge with Pika is, it is not thread-safe. So I am not finding a way to execute a lakh of API requests in parallel using multiple threads using Pika.

If I have to go with Celery, I don't know how I can achieve better scalability in executing these API requests in parallel.

READ MORE
3 upvotes·178.6K views
Replies (1)
Recommends
rq
Redis

For large amounts of small tasks and caches I have had good luck with Redis and RQ. I have not personally used celery but I am fairly sure it would scale well, and I have not used RabbitMQ for anything besides communication between services. If you prefer python my suggestions should feel comfortable.

Sorry I do not have a more information

READ MORE
5 upvotes·161.5K views
Back End Developer at Company Hero·
Needs advice
on
GolangGolang
and
KafkaKafka

I'm analyzing companies with stacks similar to my company, as we are in a process of breaking the monolith for microservices. I noticed that your stack is very similar to ours, Python, Django, Celery, and so on. Analyzing the technology you use I could see the use of Go and Kafka which made me think that you also went through a similar process. So here's my question I would like to know what were some of the reasons why you adopted GO in your ecosystem?? scalability, performance?? We are looking into the possibility of starting to use GO but for that, I wanted to know why use GO instead of Python?? or why to use both together?? It may be something trivial but all experience and opinion are important to us?? Thanks.

READ MORE
4 upvotes·11.5K views
Replies (1)
Core Developer at OSInet·

The main reasons for using Go as a Kafka consumer at Deliveroo are (1) simplicity and reliabiliy of code (2) speed of execution.

For (1), Python works just as well and indeed, we do have some low-scale applications using Python as it tends to be more productive for smaller and/or shorter life projects. But Go remains upwards compatible since its inception while the 2->3 conversion in Python was no small matter, and even minor version conversions tend to require many dependency upgrades, which tend not to be synchronised. For (2) at high scale (think 1M transactions/5 minutes), Go is simpler to scale on less hardware. We also use Kotlin and Scala for this because, beyond pure manually written consumers, Go misses a real equivalent to Kafka Streams, as that only exists on the JVM.

READ MORE
Improving Stream Data Quality With Protobuf Schema Validation (deliveroo.engineering)
4 upvotes·1 comment·4.3K views
jazeem azeez
jazeem azeez
·
October 19th 2021 at 7:01PM

also a big thing in production systems , type-safety.

At scale python and similar systems will cry . so Go is a good candidate if you like to keep things maintainable and simple

·
Reply
Head of IT at lawpilots·

A big part of our needs fits perfectly into what Django has to offer: an ORM with support for PostgreSQL , the amazing auto-generated admin interface, consolidated tooling around the application lifecycle and a well-established community with solutions to the majority of common problems.

We use Django whenever we need the auto-generated admin and the friendly templating language to build capable web applications which are relatively easy to maintain for a comparably long time. The excellent integrations for Celery and Django REST framework make it easy to build the necessary integrations with other services.

READ MORE
5 upvotes·79.1K views