Surya Singh Kafle

• Designed and implemented scalable ETL and ELT pipelines using PySpark, Spark SQL, and Azure Databricks, processing 50M+ records daily across enterprise datasets for analytics and machine learning workloads. • Built automated data ingestion pipelines using Azure Data Factory (ADF) integrating 10+ structured and semi-structured data sources including Azure SQL Database, Blob Storage, and Azure Data Lake, reducing ingestion latency by 40%. • Implemented streaming data pipelines using Apache Kafka and Spark Streaming to support near real-time analytics and operational monitoring. • Designed and maintained Snowflake data warehouse schemas, creating optimized SQL transformations and dimensional data models for enterprise reporting and analytics. • Migrated legacy ETL processes to cloud-based data pipelines using Azure Databricks and Snowflake, improving pipeline reliability by 35% and reducing average job execution time by 50%. • Built feature engineering pipelines using PySpark and SQL to prepare large datasets used in predictive modeling and analytical experiments. • Supported development of machine learning models using TensorFlow and PyTorch by building training datasets, feature pipelines, and automated data preprocessing workflows. • Engineered multiple predictive features from behavioral and operational datasets to improve model performance and analytical insights. • Developed data pipelines to support A/B testing and experimentation frameworks, enabling teams to evaluate product and operational initiatives using data-driven approaches. • Extracted and processed large datasets using optimized SQL queries across distributed systems handling 100M+ records, achieving up to 60% improvement in query execution time. Technologies Used: Azure Data Factory, Azure Databricks, Azure Data Lake, PySpark, Spark, Snowflake, SQL, Kafka, Terraform, Azure DevOps, Power BI, Dash, TensorFlow, PyTorch, Apache Airflow, dbt, Great Expectations, OpenTelemetry

• Developed machine learning models and statistical analyses to evaluate financial and operational datasets and support business forecasting. • Built automated ETL pipelines using Python and SQL to collect, transform, and process data from multiple operational systems. • Reduced manual data processing effort by 60% by automating data preparation workflows across 5+ operational systems, improving data consistency and reliability. • Performed exploratory data analysis and feature engineering to identify key drivers of business performance. • Designed and implemented analytical data models and database schemas optimized for reporting and business intelligence. • Created visualizations and dashboards to communicate analytical insights and operational trends to stakeholders. • Wrote optimized SQL queries for large relational datasets of 10M+ rows, improving data retrieval performance by up to 50% and reducing analytical query runtime. • Collaborated with engineering and business teams to deliver data-driven insights and analytical solutions for operational decision-making. • Assisted in building data pipelines to support predictive analytics and reporting workflows. • Scheduled and monitored batch data workflows using Apache Airflow, ensuring reliable pipeline execution and automated failure alerting across data ingestion jobs. • Containerized data pipeline components using Docker and deployed on Kubernetes clusters, enabling scalable and reproducible data processing environments. Technologies Used: Python, SQL, PostgreSQL, MySQL, ETL Pipelines, Machine Learning, Data Analysis, Visualization Tools, Apache Airflow, Docker, Kubernetes, Scikit-learn, Pandas, NumPy