Rudri Joshi

• Delivered ETL/ELT development using Azure Data Factory (ADF), Azure Synapse, and Azure Databricks, enabling scalable cloud-based data integration and orchestration of end-to-end pipelines. • Built streaming data pipelines in Databricks to process large-scale real-time ingestion from dispersed data sources, supporting hybrid (on-prem and cloud) data architectures. • Applied advanced SQL and Python to implement robust data preparation and transformation logic across big data environments, ensuring quality and consistency of processed datasets. • Integrated structured and unstructured data into Azure Data Lake Storage Gen2 (ADLS), applying data normalization, data conversion, and aligning with data lifecycle management standards. • Designed and deployed CI/CD pipelines with Azure DevOps and Git, automating version-controlled delivery of ADF pipelines, Databricks notebooks, and infrastructure code. • Utilized Apache Airflow for data orchestration, managing complex workflow scheduling, and implementing alerting and failure recovery mechanisms for critical data pipelines. • Collaborated with stakeholders to gather requirements for data ingestion and transformation, contributing backend logic to Power BI dashboards with real-time and batch data from Azure data services. • Tuned Spark SQL queries and optimized ADF pipeline performance, resulting in reduced latency and improved data analysis and reporting efficiency. • Supported data engineering pipelines for machine learning (ML) initiatives, assisting in scalable data provisioning for model training and deployment environments. • Promoted data governance best practices, implementing metadata management, data quality checks, and ensuring compliance with cloud data standards. • Provided production support, troubleshooting ADF, Airflow workflows, and stored procedures in highly distributed and big data scenarios

• Developed in designing, implementing, and optimizing ETL (Extract, Transform, Load) processes to efficiently extract data from various sources, transform it to meet business requirements, and load it into target systems • Mentored in utilizing Azure Data Factory to orchestrate and automate end-to-end data integration workflows, ensuring seamless data movement and transformation within Azure environments • Orchestrated in architecting and managing Azure SQL Database instances, leveraging its capabilities for scalable and reliable relational database solutions in the cloud • Implemented Azure Blob Storage and Azure Data Lake Storage for secure and cost-effective storage of structured and unstructured data, implementing data governance and access control measures as per organizational requirements • Very good at utilizing Azure Synapse Analytics for designing and implementing scalable data warehousing solutions, facilitating efficient storage, processing, and analysis of large volumes of data • Accomplished in data modeling and warehousing, proficient in creating logical and physical representations of complex data structures to support analytical and reporting needs • Skilled in designing and implementing data pipelines using Azure Stream Analytics, enabling real-time data processing and analysis for actionable insights • Designed and maintained Snowflake data warehousing solutions, optimizing data loading processes and ensuring high performance for complex queries and analytics • Streamlined in PySpark for Azure data processing, optimizing pipelines. Proficient in scalable design for efficient large-scale data processing • Utilized Power BI for interactive visualizations, connecting data sources. Experienced in transforming data for actionable insight¬