Gayathri Das

Architected an end-to-end real-time marketing intelligence platform on AWS, processing millions of events daily and enabling near-real-time campaign tracking. Engineered Kafka-based streaming pipelines with AWS (Kinesis, Lambda, S3) to deliver sub-minute data latency for downstream analytics. Designed a data lakehouse on S3 + Redshift + Athena using Medallion Architecture, reducing ad-hoc query costs. Developed production-grade PySpark applications for complex, fault-tolerant data transformations at scale. Implemented Change Data Capture (CDC) and incremental load patterns, significantly cutting full-load processing time for high-volume datasets. Enforced data governance and PII masking across the platform, compliant with enterprise standards. Drove CI/CD adoption for data pipelines using Jenkins and Git, reducing deployment errors and improving release velocity. Mentored junior engineers through code reviews and pair programming, elevating team coding standards.

Built scalable ETL pipelines using AWS Glue and PySpark to ingest customer data from REST APIs, relational databases, and third-party SaaS systems. Implemented a data lake on S3 with structured partitioning for JSON, Parquet, and CSV, enabling efficient downstream querying. Optimized AWS EMR Spark jobs by tuning cluster configurations, improving pipeline performance and reducing compute costs. Designed dimensional data models (star schema) in Redshift to support complex analytical workloads. Automated pipeline scheduling using Apache Airflow, achieving reliable end-to-end workflow orchestration. Built data quality validation frameworks with automated CloudWatch alerting, reducing data incidents. Partnered with business stakeholders to translate reporting requirements into scalable technical implementations.

Developed and optimized ETL pipelines using Azure Data Factory to ingest data from SQL databases, REST APIs, and flat files. Built and tuned data transformation logic in Azure Databricks with PySpark, improving processing throughput. Designed a data lake solution on ADLS Gen2 with role-based access control and lifecycle policies. Modeled schemas in Azure Synapse Analytics to support enterprise reporting and BI requirements. Prepared curated datasets to support Power BI dashboards for business analysts and leadership. Documented pipeline architecture, data lineage, and transformation logic for team onboarding and audits.

Contributed to ETL workflows (SQL + Python) to extract, transform, and load retail sales data into a centralized warehouse. Designed relational schemas and dimensional models to support sales reporting and business intelligence. Performed data cleansing, deduplication, and validation to ensure data accuracy and completeness. Assisted in optimizing SQL queries for reporting workloads, improving response times for business users.