I'm looking for an open-source/free/cheap tool to clean messy data coming from various travel APIs. We use many different APIs and save the info in our DB. However, many duplicates cannot be easily recognized as such.

We would either write an algorithm or use smart technology/tools with ML to help with product management.

While there are many things to be considered, this is one feature that it should have:

"To avoid confusion, we need to merge the suppliers & products accordingly. Products and suppliers must be able to be merged and assigned separately.

Reason: It may happen that one supplier offers different products. E.g., 1 tour operator offers 3 products via 1 API, but only 1 product with 3 (or a different amount of) variations via a different API. Also, the commission may differ for products, which we need to consider. Very often, products that are live (are bookable in real-time) on via 1 API, but are not live on the other. E.g., Supplier product 1 & 2 of API1 are live, product 3 not. For the same supplier, API2 provides live availability for products 1, 2, and 3.

Summing up, when merging the suppliers (tour operators) we need to consider:

• Are the products the same for all APIs?
• Which booking system API gives a better commission? Note: Some APIs charge us 1-5% depending on the monthly sale, which needs to be considered
• Which booking system provides live availability
• Is it the same supplier, or is the name only similar?

Most of the time, the supplier names differ even if they are the same (e.g., API1 often names them XX Pty Ltd, while API2 leaves "Pty Ltd" out). Additionally, the product title, description, etc. differ.

We need to write logic and create an algorithm to find the duplicates & to merge, assign, or (de)activate the respective supplier or product. My previous developer started a module to merge the suppliers, which does not seem to work correctly. Also, it is way too time taking considering the high amount of products that we have.

I would recommend merging, assigning etc. products and suppliers only if our algorithm says it's 90- 100% the matching supplier/product. Otherwise, admins need to be able to check & modify this. E.g. everything with a lower possibility of matching will be matched automatically, but can be undone or modified.

The next time the cron job runs, this needs to be considered to avoid recreating duplicates & creating a mess."

I am not sure in what way OpenRefine can help to achieve this and what ML tool can be connected to learn from the decisions the product management team makes. Maybe you have an idea of how other travel portals deal with messy data, duplicates, etc.?

I'm looking for the cheapest solution for a start-up, but it should do the work properly.

Power BI is really easy to start with. If you have just several Excel sheets or CSV files, or you build your first automated pipeline, it is actually quite intuitive to build your first reports.

And as we have kept growing, all the additional features and tools were just there within the Azure platform and/or Office 365.

Since we started building Mews, we have already passed several milestones in becoming start up, later also a scale up company and now getting ready to grow even further, and during all these phases Power BI was just the right tool for us.

Very easy-to-use UI. Good way to make data available inside the company for analysis.

Has some built-in visualizations and can be easily integrated with other JS visualization libraries such as D3.

Can be embedded into product to provide reporting functions.

Support team are helpful.

The only complain I have is lack of API support. Hard to track changes as codes and automate report deployment.