OpenRefine logo

OpenRefine

Desktop application for data cleanup and transformation
33
67
+ 1
0

What is OpenRefine?

It is a powerful tool for working with messy data: cleaning it; transforming it from one format into another; and extending it with web services and external data.
OpenRefine is a tool in the Big Data Tools category of a tech stack.
OpenRefine is an open source tool with 10.9K GitHub stars and 2K GitHub forks. Here’s a link to OpenRefine's open source repository on GitHub

Who uses OpenRefine?

Companies
7 companies reportedly use OpenRefine in their tech stacks, including SMARTTechStack, infoculture, and APICrafter.

Developers
25 developers on StackShare have stated that they use OpenRefine.

OpenRefine Integrations

Decisions about OpenRefine

Here are some stack decisions, common use cases and reviews by companies and developers who chose OpenRefine in their tech stack.

Shehryar Mallick
Associate Data Engineer at Virtuosoft · | 5 upvotes · 22.5K views
Needs advice
on
Apache HiveApache Hive
and
OpenRefineOpenRefine

I've been going over the documentation and couldn't find answers to different questions like:

Apache Hive is built on top of Hadoop meaning if I wanted to scale it up I could do either horizontal scaling or vertical scaling. but if I want to scale up openrefine to cater more data then how can this be achieved? the only thing I could find was to allocate more memory like 2 of 4GB but using this approach would mean that we would run out of memory to allot. so thoughts on this?

Secondly, Hadoop has MapReduce meaning a task is reduced to many mapper running in parallel to perform the task which in turn increase the processing speed, is there a similar mechanism in OpenRefine or does it only have a single processing unit (as it is running locally). thoughts?

See more

I'm looking for an open-source/free/cheap tool to clean messy data coming from various travel APIs. We use many different APIs and save the info in our DB. However, many duplicates cannot be easily recognized as such.

We would either write an algorithm or use smart technology/tools with ML to help with product management.

While there are many things to be considered, this is one feature that it should have:

"To avoid confusion, we need to merge the suppliers & products accordingly. Products and suppliers must be able to be merged and assigned separately.

Reason: It may happen that one supplier offers different products. E.g., 1 tour operator offers 3 products via 1 API, but only 1 product with 3 (or a different amount of) variations via a different API. Also, the commission may differ for products, which we need to consider. Very often, products that are live (are bookable in real-time) on via 1 API, but are not live on the other. E.g., Supplier product 1 & 2 of API1 are live, product 3 not. For the same supplier, API2 provides live availability for products 1, 2, and 3.

Summing up, when merging the suppliers (tour operators) we need to consider:

  • Are the products the same for all APIs?
  • Which booking system API gives a better commission? Note: Some APIs charge us 1-5% depending on the monthly sale, which needs to be considered
  • Which booking system provides live availability
  • Is it the same supplier, or is the name only similar?

Most of the time, the supplier names differ even if they are the same (e.g., API1 often names them XX Pty Ltd, while API2 leaves "Pty Ltd" out). Additionally, the product title, description, etc. differ.

We need to write logic and create an algorithm to find the duplicates & to merge, assign, or (de)activate the respective supplier or product. My previous developer started a module to merge the suppliers, which does not seem to work correctly. Also, it is way too time taking considering the high amount of products that we have.

I would recommend merging, assigning etc. products and suppliers only if our algorithm says it's 90- 100% the matching supplier/product. Otherwise, admins need to be able to check & modify this. E.g. everything with a lower possibility of matching will be matched automatically, but can be undone or modified.

The next time the cron job runs, this needs to be considered to avoid recreating duplicates & creating a mess."

I am not sure in what way OpenRefine can help to achieve this and what ML tool can be connected to learn from the decisions the product management team makes. Maybe you have an idea of how other travel portals deal with messy data, duplicates, etc.?

I'm looking for the cheapest solution for a start-up, but it should do the work properly.

See more

OpenRefine's Features

  • Faceting
  • Clustering
  • Editing cells
  • Reconciling
  • Extending web services

OpenRefine Alternatives & Comparisons

What are some alternatives to OpenRefine?
Trifacta
It is an Intelligent Platform that Interoperates with Your Data Investments. It sits between the data storage and processing environments and the visualization, statistical or machine learning tools used downstream
R Language
R provides a wide variety of statistical (linear and nonlinear modelling, classical statistical tests, time-series analysis, classification, clustering, ...) and graphical techniques, and is highly extensible.
Python
Python is a general purpose programming language created by Guido Van Rossum. Python is most praised for its elegant syntax and readable code, if you are just beginning your programming career python suits you best.
Pandas
Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more.
Talend
It is an open source software integration platform helps you in effortlessly turning data into business insights. It uses native code generation that lets you run your data pipelines seamlessly across all cloud providers and get optimized performance on all platforms.
See all alternatives

OpenRefine's Followers
67 developers follow OpenRefine to keep up with related blogs and decisions.