Software Engineer ·
Needs advice
on
Python
PHP
and
Go

Hi

I want to build a tool to check asset availability (video, images, etc.) from third-party vendors. These vendors have APIs. However, this process should run daily basis and update the database with the status. This is a kind of separate process. I need to know what will be the good approach and technology for this?

READ LESS
13 upvotes·108.9K views
Replies (5)
Recommends
Go

The major advantage of Go is that you can run queries in parallel. Fire off a Go thread for each vendor and each thread can check the availability of assets from a specific vendor and update the database. Go supports hundreds of threads with ease.

READ MORE
3 upvotes·1 comment·101.2K views
Noah Shanaberger
Noah Shanaberger
·
March 23rd 2021 at 7:42PM

You can do that with any language... The main difference with Go is that it implements Green Threads rather than OS threads, allowing for parallel code on a single CPU core.

·
Reply
365 Consulting Services Ltd.·
Recommends
Python

hi - I think this depends on how you want to provide the information to the user. If you want to build a Wordpress-plugin: PHP If you want to build your own website: Python+Django / PHP / JavaScript+Node.js As Desktop application?

READ MORE
7 upvotes·3 comments·107.6K views
Peter Vereshagin
Peter Vereshagin
·
March 28th 2021 at 6:14PM

Piusha, I can tell you how I implement this on my projects. I have the message queue and task scheduler the same daemon on my micrioservices which is controlled by REST API , has limitations on requesting stuff, etc. It generates the events, and another my service onboard, the API gateway, has the request fired at that same moment. The scheduler doesn't know how to authorize on vendor's API, what info is needed from there and which is not; but API gateway knows that, performs the job and has the info back to the scheduler. The next great feature of the scheduler is that it sends that same info it received on that same API gateway (or wherever I want, as it's a part of its' particular job description) but to another endpoint - to be stored then.

Among other things such a scheduler is able to repeat the job up to the limit of tries for the case if the latest request fired by that job was unsuccessful.

This comprehenses alot my other approaches, e. g., by Cron daemon built in OS.

The name of the scheduler/message-queue software is: Agenda-Rest, it can be found on github. You also may want to use Bree instead, it should be even better.

·
Reply
Piusha Kalyana
Piusha Kalyana
·
May 12th 2021 at 4:15AM

Thank you for your valuble idea. let me explore what you gave as an idea

·
Reply
Piusha Kalyana
Piusha Kalyana
·
May 12th 2021 at 4:17AM

thank you

·
Reply
View all (5)
Avatar of Doug Schwartz