A case study on solving problematic data import from excel
Back Story
The team faced an issue with the new project (Laravel 10, queues, maat)
The Problem
What needed to be solved and why
The client’s main workflow is based on excel driven produtct data upload to the public facing website. The client has many excel files with different types of products. Each of these excel files can have up to few thousands of products. The numbers have grown since the company started and one huge problem was with the old system was the slowness (it was uploading on the foreground hidering the performance for all users)
The goal
was to make a super stable and predictable excel based import what can import product data, with all their their categories and their media via url links in the background seamlessly while the whole process should not affect the ui foreground thread.
- The Approach: What technologies you chose and why (this is where technical thinking shows)
1st time I went with PHP and Laravel (v5 back then), it wasnt really paralellized, but seeing the hassle with memoory management and instababilities later with the 2.0 rewrite using the same stack but with utilizing queues I decided to go on a different approach.
I have to admit I wasn’t an ace in php, but after reasearching the topic of handling tasks in the background I found an promising clue:
a node.js native task manager which relies on Redis called BullMQ
BullMQ
BullMQ is the modern, TypeScript-rewritten successor to the popular Bull library for Node.js, designed for improved performance and reliability. While Bull is now in maintenance mode (bug fixes only), BullMQ offers advanced features, native TypeScript support, and superior job flow management, making BullMQ the recommended choice for new projects
Official website: BullMQ
- Challenges: What went wrong and how you handled it
- Results: Numbers if you have them (users, performance improvements, time saved)
- What You'd Do Differently: This shows growth mindset and self-awareness