Data Engineer with 3.5 years building and operating production data platforms on AWS. Specializes in distributed ETL with PySpark and real-time ingestion via Kinesis/Firehose, processing ~650M telemetry records/day from …
Data Engineer with 3.5 years building and operating production data platforms on AWS. Specializes in distributed ETL with PySpark and real-time ingestion via Kinesis/Firehose, processing ~650M telemetry records/day from 6.8M connected devices. Deep PostgreSQL expertise across schema-per-tenant isolation, transaction-safe migrations, and SCD Type 2 modeling. Experienced in Kafka-based data-lake delivery, orchestration (Step Functions, Airflow), lakehouse patterns (Delta Lake), and significant AWS cost optimization.
الخبرة
T
Data Engineer
Tata Elxsi Limited, Trivandrum·Adur , Kerala , India
Nov 2022 - حتى الآن · 3 سنوات 8 أشهر
Built and operate production PySpark/AWS Glue ETL processing ~650M telemetry records/day from 6.8M connected STB devices reporting every 15 minutes, transforming raw multi-tenant telemetry into analytics-ready datasets. Engineered real-time ingestion with Amazon Kinesis and Firehose; diagnosed a hot-shard bottleneck and scaled the stream 28 → 32 shards with enhanced shard-level metrics, eliminating write throttling under peak load. Delivered processed telemetry to client-side data lakes via Apache Kafka and Kafka streaming, enabling reliable downstream consumption across analytics teams. Optimized AWS Glue jobs by diagnosing data skew, tuning Adaptive Query Execution and caching (persist MEMORY_AND_DISK), and right-sizing workers (G.1X → G.2X) with autoscaling — substantially reducing job runtime and compute spend. Drove cloud cost optimization across the pipeline — Athena partition pruning and predicate pushdown, RDS storage migration gp2 → gp3 (higher IOPS at lower cost), and Glue worker right-sizing — drastically reducing monthly AWS spend. Built transaction-safe PostgreSQL patterns: atomic staging-table swaps (delete/insert), SCD Type 2 MERGE with surrogate keys and SHA-256 row hashing, and partial indexing on current records — enabling lossless retroactive reloads of historical data. Orchestrated multi-job pipelines with AWS Step Functions and EventBridge using idempotent retry/catch logic and event-driven scheduling for reliable, hands-off runs. Owned CI/CD deployment of production Glue pipelines via Jenkins with dev → prod environment promotion and CloudFormation-managed infrastructure, enabling safe, repeatable releases. Managed production schema evolution via blue/green column migration and Glue Schema Registry compatibility modes with versioned migration scripts for backward-compatible reprocessing. Built a data-quality framework — invariant-based validation (conservation, bounds, rollup consistency) and cross-database reconciliation via S3 Parquet exports and Athena full outer joins — surfacing missing/divergent records at scale.
المشاريع
I
iCX
TATA ELXSI
المدة : 15-May-2019 - 30-Jun-2026
RMS – Multi-Tenant STB Telemetry Data PlatformTata Elxsi Limited, Trivandrum | Data Engineer (~3.5 years)What it does: A production, multi-tenant AWS data platform that ingests and processes set-top box (STB) telemetry data at roughly 650 million records/day across nine tenants, powering downstream analytics and reporting.Tech stack: • Ingestion/Streaming: AWS Kinesis, Firehose • Processing: PySpark on AWS Glue • Storage: S3 (Parquet) • Query: Athena • Database: PostgreSQL (schema-per-tenant multi-tenancy • Orchestration: Step Functions, LambdaKey contributions / talking points: • Diagnosed and resolved skew-bound vs. capacity-bound Glue job performance issues • Refactored a precompute_latest job from row_number-based logic to hash aggregation for null-safe column resolution • Debugged Glue Python shell timeout issues tied to scheduled trigger configuration • Investigated and resolved Kinesis throttling on a data stream, scaling shards (28→32) • Analyzed RDS PostgreSQL ReadIOPS alarms and recommended a gp2→gp3 migration • Implemented SCD Type 2 merge logic using SHA-256 row hashing for idempotent, auditable updates • Designed multi-tenant schema-per-tenant PostgreSQL architecture • Built technical documentation for QA on metric trend comparison and weekly metrics consolidation jobsGenAI-adjacent work (differentiator):Contributed to CCAssist, an internal RAG-based support tool using AWS Strands Agents, Bedrock, Aurora PostgreSQL with pgvector, and Langfuse for observability
الشهادات
BTECH
Sree Chitra Thirunal Institute of Medical Sciences and Technology, India
· Kazhakkoottam , Kerala , India
· 2023
