Automate Data Pipeline To perform Incremental Load

Overview

Client database system is secure and not able to access from the outside organization, we need to extract the data from multiple tables in a different database. Then we need to upload and normalize the data in the Azure SQL staging server and export the data as Pipe delimiter format and upload it to Azure blob storage used by the Machine learning team for their analysis

Solution Overview

null

Generated BCP queries using ID and date fields to extract the latest data from the multiple tables present in a different database.

null

Built a data pipeline to extract and upload the data to the SQL server by creating a package for each table data to perform incremental load in the SQL server.

null

Uploaded data is then normalized and loaded into the staging database.

null

Configured automated SQL jobs to export the normalized data as a pipe-delimited file and uploaded it to the Azure blob storage for Machine learning team analysis.

Impact

null

Built an automated data pipeline to perform incremental uploads on weekly basis on both development and staging servers.

null

ML team will perform some competitor analysis using the exported normalized data.

Technology

Related Success Stories

Related Insights

Top 5 Machine Learning Companies in 2022

Machine learning involves computers discovering how they can perform tasks without being explicitly programmed to do so…

Read More

Alphabet Hand Gestures Recognition Using Media Pipe

Media Pipe is a cross-platform (Android, ios,web) framework used to build Machine Learning pipelines for audio, video, time-series data etc…

Read More
Free Consulting