Azure Blob Storage

I want to migrate files from Microsoft SharePoint Server to Azure Blob Storage using PowerShell. It seems I got a lot of files not found problems and connection/permission issues. I tried to fetch files from SharePoint Server but turns out to be not working.

Since I don't have the knowledge to evaluate this choice correctly, and since it is fundamental to the success of the project, I wanted to know the community's opinion on the matter.

Thank you :)

My process is like this: I would get data once a month, either from Google BigQuery or as parquet files from Azure Blob Storage. I have a script that does some cleaning and then stores the result as partitioned parquet files because the following process cannot handle loading all data to memory.

The next process is making a heavy computation in a parallel fashion (per partition), and storing 3 intermediate versions as parquet files: two used for statistics, and the third will be filtered and create the final files.

I make a report based on the two files in Jupyter notebook and convert it to HTML.

• Everything is done with vanilla python and Pandas.
• sometimes I may get a different format of data
• cloud service is Microsoft Azure.

What I'm considering is the following:

Get the data with Kafka or with native python, do the first processing, and store data in Druid, the second processing will be done with Apache Spark getting data from apache druid.

the intermediate states can be stored in druid too. and visualization would be with apache superset.