Need advice about which tool to choose?Ask the StackShare community!
Avro vs JSON: What are the differences?
Developers describe Avro as "A data serialization framework *". It is a row-oriented remote procedure call and data serialization framework developed within Apache's Hadoop project. It uses JSON for defining data types and protocols, and serializes data in a compact binary format. On the other hand, *JSON** is detailed as "A lightweight data-interchange format". JavaScript Object Notation is a lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate. It is based on a subset of the JavaScript Programming Language.
Avro can be classified as a tool in the "Serialization Frameworks" category, while JSON is grouped under "Languages".
Redsift, OTTLabs, and Mon Style are some of the popular companies that use JSON, whereas Avro is used by Liferay, LendUp, and BetterCloud. JSON has a broader approval, being mentioned in 32 company stacks & 161 developers stacks; compared to Avro, which is listed in 8 company stacks and 8 developer stacks.
Hi. Currently, I have a requirement where I have to create a new JSON file based on the input CSV file, validate the generated JSON file, and upload the JSON file into the application (which runs in AWS) using API. Kindly suggest the best language that can meet the above requirement. I feel Python will be better, but I am not sure with the justification of why python. Can you provide your views on this?
Python is very flexible and definitely up the job (although, in reality, any language will be able to cope with this task!). Python has some good libraries built in, and also some third party libraries that will help here. 1. Convert CSV -> JSON 2. Validate against a schema 3. Deploy to AWS
- The builtins include json and csv libraries, and, depending on the complexity of the csv file, it is fairly simple to convert:
import csv
import json
with open("your_input.csv", "r") as f:
csv_as_dict = list(csv.DictReader(f))[0]
with open("your_output.json", "w") as f:
json.dump(csv_as_dict, f)
The validation part is handled nicely by this library: https://pypi.org/project/jsonschema/ It allows you to create a schema and check whether what you have created works for what you want to do. It is based on the json schema standard, allowing annotation and validation of any json
It as an AWS library to automate the upload - or in fact do pretty much anything with AWS - from within your codebase: https://aws.amazon.com/sdk-for-python/ This will handle authentication to AWS and uploading / deploying the file to wherever it needs to go.
A lot depends on the last two pieces, but the converting itself is really pretty neat.
This should be pretty doable in any language. Go with whatever you're most familiar with.
That being said, there's a case to be made for using Node.js since it's trivial to convert an object to JSON and vice versa.
I would use Go. Since CSV files are flat (no hierarchy), you could use the encoding/csv package to read each row, and write out the values as JSON. See https://medium.com/@ankurraina/reading-a-simple-csv-in-go-36d7a269cecd. You just have to figure out in advance what the key is for each row.
Pros of Avro
Pros of JSON
- Simple5
- Widely supported4