Avatar of dmitry-mukhin
CTO at Uploadcare·

The 350M API requests we handle daily include many processing tasks such as image enhancements, resizing, filtering, face recognition, and GIF to video conversions.

Tornado is the one we currently use and aiohttp is the one we intend to implement in production in the near future. Both tools support handling huge amounts of requests but aiohttp is preferable as it uses asyncio which is Python-native. Since Python is in the heart of our service, we initially used PIL followed by Pillow. We kind of still do. When we figured resizing was the most taxing processing operation, Alex, our engineer, created the fork named Pillow-SIMD and implemented a good number of optimizations into it to make it 15 times faster than ImageMagick

Thanks to the optimizations, Uploadcare now needs six times fewer servers to process images. Here, by servers I also mean separate Amazon EC2 instances handling processing and the first layer of caching. The processing instances are also paired with AWS Elastic Load Balancing (ELB) which helps ingest files to the CDN.

READ LESS
How Uploadcare Built a Stack That Handles 350M File API Requests Per Day - Uploadcare Tech Stack | StackShare (stackshare.io)
20 upvotes·2 comments·94K views
Eric Plaster
Eric Plaster
·
February 26th 2020 at 10:03pm

I know this is already a year ago, but if you were to look into aiohttp, I would suggest Starlette and FastAPI (which is built on top of Starlette) instead. It's very lightweight and as fast as node.js/go. Just my 2 cents.

• https://www.starlette.io/

• https://fastapi.tiangolo.com/

·
Reply
Dmitry Mukhin
Dmitry Mukhin
·
March 9th 2020 at 1:28pm

Thanks!

Will keep in mind :)

·
Reply