Harsha sai gade

● Delivered end to end data engineering solutions including design, architecture and build for various organizations in the company including the revenue department and optimization department. ● Built and maintained data pipelines in Microsoft Fabric using azure data factory, spark notebooks and power apps for meta data driven pipelines giving complete control to business users to control data transformation process and automation. ● Increased the efficiency of existing pipelines by removing bottlenecks and refactoring huge sets of Pyspark notebooks. ● Built and enhanced data validation affecting data reliability for data used by the consumers for billing purposes. ● Improved the error handling capability among pipelines to make individual processes rerunnable , saving compute and time. ● Set good data engineering, database and code management standards for both business users and other developers. ● Trained and managed multiple operational managers to be able to on-board new clients, build spark notebooks. ● Built Enterprise scale data models hosting Billions of records across clients for multiple data streams. ● Collaborated with AI vendors and in-house teams to build and maintain AI agents, machine learning models. ● Ad-hoc analysis and power bi report creation and maintenance that are used for Monthly operational reports visualisation as well as Account responsible information used for invoicing summary.

Contracted for Ovation Healthcare Via Truegility INC ● Built several data pipelines in Microsoft Fabric using PySpark, MS SQL and Azure data factory to integrate and centralize data from CSV, TXT, XLSX and JSON files into Lakehouse and warehouse. ● Developed common Ingest and Export spark notebooks instead of bespoke pipelines, decreasing development time by 27%. ● Created bespoke transform notebooks depending on individual data processing requirements and join conditions to ease maintenance and updates while decreasing cross client dependency down time. ● Designed and implemented data warehouse from requirement gathering, data modelling till implementation and data loading. ● Built multiple Store procedures and APIs, integrated following CQRS design for fast reads and writes from front end. ● Designed and created an event-driven architecture using Azure Data Factory, enabled automated pipeline triggering based on events in Fabric Lakehouse for processing incoming files without manual effort. ● Built multiple ETL pipelines to integrate multiple sources of IoT data into centralized analytical data warehouses. ● Created spark notebooks to clean and transform data into OLAP data model while ensuring data integrity. ● Divided ingest, transformations into hybrid medallion architecture for additional data audit and data retention. ● Established a hierarchical staging area structure for raw data while implementing SCD Type 1 & Type 2 to ensure accurate historical data in the semantic layer for reporting and historical data preservation. ● Built multiple dashboards in power bi enabling tracking of temperature, location data from trucks while capturing missed targets and highlighting containers that fall out of optimal operating temperature.

● Developed multiple batch and real time data pipelines for transactional and analytical purposes supporting 45 dashboards. ● Created stored procedures and built multiple APIs implementing full CRUD functionality for seamless inventory management and transaction processing for the ecommerce platform capable of handling 100,000 large scale orders per day. ● Integrated data from on-premises PostgreSQL sources into AWS Redshift, using AWS Lambda and AWS Glue for orchestration to create an EDW enabling customer focused analytics and marketing.