I recently started a new position as a data scientist at an E-commerce company. The company is founded about 4-5 years ago and is new to many data-related areas. Specifically, I'm their first data science employee. So I have to take care of both data analysis tasks as well as bringing new technologies to the company.
They have used Elasticsearch (and Kibana) to have reporting dashboards on their daily purchases and users interactions on their e-commerce website.
They also use the Oracle database system to keep records of their daily turnovers and lists of their current products, clients, and sellers lists.
They use Data-Warehouse with cockpit 10 for generating reports on different aspects of their business including number 2 in this list.
At the moment, I grab batches of data from their system to perform predictive analytics from data science perspectives. In some cases, I use a static form of data such as monthly turnover, client values, and high-demand products, and run my predictive analysis using Python (VS code). Also, I use Google Datastudio or Google Sheets to present my findings. In other cases, I try to do time-series analysis using offline batches of data extracted from Elastic Search to do user recommendations and user personalization.
I really want to use modern data science tools such as Apache Spark, Google BigQuery, AWS, Azure, or others where they really fit. I think these tools can improve my performance as a data scientist and can provide more continuous analytics of their business interactions. But honestly, I'm not sure where each tool is needed and what part of their system should be replaced by or combined with the current state of technology to improve productivity from the above perspectives.
It's hard to make a suggestion here as your use case isn't clear enough.
Use BigQuery if you want to replicate your probably on premise Oracle and Elasticsearch databases so you can profit from the speed of BigQuery. You can do the replication via Google Cloud Functions. Your Google Sheets can be connected to BigQuery and BigQuery can easily be connected to DataStudio.
If you do data science on the data there would be BigQuery ML and Google Colab that would fit into your stack.
In case you do BigData analysis you can go with Apache Spark if you have enough resources (on-prem or Cloud). I suggest you to use a Kubernetes backbone for this as you only reserve the resources when in use and the cluster can be used for other stuff as well.
For dashboarding find your preference and the preference of your audience with DataStudio, Tableau or Apache Superset
Thank you for the answer.
A few days ago the head of IT told me to try AWS if I need cloud resources. I cannot migrate everything from on-premise to cloud. But, I need to choose what data I need for my Data Science tasks on the cloud. For example, I need to extract their daily sale records stored in Oracle as well as their web usage from Elasticsearch.
My main tasks would be "sale/demand forecast", "user retention prediction", "recommendation systems", and "user activity analysis". So, BigData analysis would be part of the job.
I think BigQuery and Datastudio would be out of my options. I need to use resources offered by AWS or compatible with AWS. I'm not sure if I need to grab their web data directly from their web platform's server or from Elasticsearch.
Also, what dashboarding tool is better when I use AWS for my DS pipeline?
BigQuery and Datastudio would be out of my options. I need to use resources offered by AWS or compatible with AWS. I'm not sure if I need to grab their web data directly from their web platform's server or from Elasticsearch.