Google Sheets

Google Sheets

Business Tools / Back Office / Spreadsheets Online
Needs advice
on
FirebaseFirebaseMongoDBMongoDB
and
MySQLMySQL

Hey everyone, My users love Microsoft Excel, and so do I. I've been making tools for them in the form of workbooks for years, these tools usually have databases included in the spreadsheets or communicate to free APIs around the web, but now I want to distribute these tools in the form of Excel Add-ins for several reasons.

I want these Add-ins to communicate to a personal server to authorize users, read from my databases, and write to them while they're using their Excel environment. I have never built a website, so what would be a good solution for this, considering I'm new to all of these technologies? I know about the existence of Microsoft Azure, Microsoft SharePoint, and Google Sheets, but I don't know how to feel about those.

READ MORE
6 upvotes·228.9K views
Replies (2)
Recommends
Snowflake

Snowflake is a NoSQL database in the cloud, which also accepts SQL calls. Users can obtain an ODBC driver for SnowFlake, which would allow your Excel apps to write/read from the backend, locally.

READ MORE
4 upvotes·1 comment·28.3K views
woodcockjosh
woodcockjosh
·
August 26th 2021 at 1:45PM

snowflake is very slow compared to other databases IMO

·
Reply
Data Scientist ·

I recently started a new position as a data scientist at an E-commerce company. The company is founded about 4-5 years ago and is new to many data-related areas. Specifically, I'm their first data science employee. So I have to take care of both data analysis tasks as well as bringing new technologies to the company.

  1. They have used Elasticsearch (and Kibana) to have reporting dashboards on their daily purchases and users interactions on their e-commerce website.

  2. They also use the Oracle database system to keep records of their daily turnovers and lists of their current products, clients, and sellers lists.

  3. They use Data-Warehouse with cockpit 10 for generating reports on different aspects of their business including number 2 in this list.

At the moment, I grab batches of data from their system to perform predictive analytics from data science perspectives. In some cases, I use a static form of data such as monthly turnover, client values, and high-demand products, and run my predictive analysis using Python (VS code). Also, I use Google Datastudio or Google Sheets to present my findings. In other cases, I try to do time-series analysis using offline batches of data extracted from Elastic Search to do user recommendations and user personalization.

I really want to use modern data science tools such as Apache Spark, Google BigQuery, AWS, Azure, or others where they really fit. I think these tools can improve my performance as a data scientist and can provide more continuous analytics of their business interactions. But honestly, I'm not sure where each tool is needed and what part of their system should be replaced by or combined with the current state of technology to improve productivity from the above perspectives.

READ MORE
4 upvotes·214.8K views
Replies (1)
Data Scientist / Data Engineer at The Prosperity Company AG·

It's hard to make a suggestion here as your use case isn't clear enough.

Use BigQuery if you want to replicate your probably on premise Oracle and Elasticsearch databases so you can profit from the speed of BigQuery. You can do the replication via Google Cloud Functions. Your Google Sheets can be connected to BigQuery and BigQuery can easily be connected to DataStudio.

If you do data science on the data there would be BigQuery ML and Google Colab that would fit into your stack.

In case you do BigData analysis you can go with Apache Spark if you have enough resources (on-prem or Cloud). I suggest you to use a Kubernetes backbone for this as you only reserve the resources when in use and the cluster can be used for other stuff as well.

For dashboarding find your preference and the preference of your audience with DataStudio, Tableau or Apache Superset

READ MORE
8 upvotes·1 comment·3.9K views
Bob hs
Bob hs
·
March 16th 2022 at 12:09PM

Thank you for the answer.

A few days ago the head of IT told me to try AWS if I need cloud resources. I cannot migrate everything from on-premise to cloud. But, I need to choose what data I need for my Data Science tasks on the cloud. For example, I need to extract their daily sale records stored in Oracle as well as their web usage from Elasticsearch.

My main tasks would be "sale/demand forecast", "user retention prediction", "recommendation systems", and "user activity analysis". So, BigData analysis would be part of the job.

I think BigQuery and Datastudio would be out of my options. I need to use resources offered by AWS or compatible with AWS. I'm not sure if I need to grab their web data directly from their web platform's server or from Elasticsearch.

Also, what dashboarding tool is better when I use AWS for my DS pipeline?

·
Reply