J Surendranath Jampula

Constructed dependable ETL pipelines using Delta Live Tables (DLT) to facilitate automated data transformation with lineage tracking and quality checks, thereby enhancing data reliability through incremental processing and versioning using change data capture (CDC). Devised and managed efficient ETL pipelines leveraging Azure Databricks and Python for transforming, cleaning, and loading data from diverse sources into the Azure Data Lake. Engineered and implemented Unity Catalog for lineage tracking, access control, and centralised data governance across Databricks workspaces while supervising audit logs and enforcing role-based access control (RBAC) to safeguard data assets. Created streaming and batch ingestion pipelines using Databricks Autoloader, enabling automatic file detection and schema evolution for structured and semi-structured data sources, optimising incremental data processing for scalable performance with Delta Lake. Implemented real-time data ingestion pipelines using Azure Stream Analytics and Azure Event Hubs for collecting and processing data from various IoT devices, sensors, and web logs, enabling near real-time insights for informed business decisions. Automated the data integration process, minimising manual interventions by instituting scheduling and orchestration through Azure Databricks. Optimised large-scale data storage solutions in Azure Data Lake and Azure SQL Database through effective data partitioning, indexing, and compression strategies to enhance query performance. Built and optimised data warehouse solutions on Azure Synapse Analytics, facilitating faster querying and reporting. Designed a star schema model to support business intelligence (BI) and data reporting requirements, ensuring optimal performance for complex queries. Championed data processing efforts using Apache Spark in Azure Databricks, applying PySpark to perform transformations, aggregations, and data wrangling on large datasets, resulting in a 30% improvement in processing times. Utilised Azure Data Factory Monitoring and Azure Databricks performance metrics to continuously assess and optimise data pipeline performance, ensuring smooth and efficient execution. Collaborated directly with data scientists, business analysts, and data architects to identify data requirements and deliver actionable insights, promoting data-driven decision-making throughout the organisation. Implemented robust data governance policies in streaming environments, encompassing data quality validation checks, schema validation, and error handling to ensure compliance with business standards. Coordinated with Azure Data Lake Analytics to define data retention policies, ensuring consistent archiving and purging of real-time data streams based on retention rules.

Constructed dependable ETL pipelines using Delta Live Tables (DLT) to facilitate automated data transformation with lineage tracking and quality checks, thereby enhancing data reliability through incremental processing and versioning using change data capture (CDC).
Devised and managed efficient ETL pipelines leveraging Azure Databricks and Python for transforming, cleaning, and loading data from diverse sources into the Azure Data Lake.
Engineered and implemented Unity Catalog for lineage tracking, access control, and centralised data governance across Databricks workspaces while supervising audit logs and enforcing role-based access control (RBAC) to safeguard data assets.
Created streaming and batch ingestion pipelines using Databricks Autoloader, enabling automatic file detection and schema evolution for structured and semi-structured data sources, optimising incremental data processing for scalable performance with Delta Lake.
Implemented real-time data ingestion pipelines using Azure Stream Analytics and Azure Event Hubs for collecting and processing data from various IoT devices, sensors, and web logs, enabling near real-time insights for informed business decisions.
Automated the data integration process, minimising manual interventions by instituting scheduling and orchestration through Azure Databricks.
Optimised large-scale data storage solutions in Azure Data Lake and Azure SQL Database through effective data partitioning, indexing, and compression strategies to enhance query performance.
Built and optimised data warehouse solutions on Azure Synapse Analytics, facilitating faster querying and reporting.
Designed a star schema model to support business intelligence (BI) and data reporting requirements, ensuring optimal performance for complex queries.
Championed data processing efforts using Apache Spark in Azure Databricks, applying PySpark to perform transformations, aggregations, and data wrangling on large datasets, resulting in a 30% improvement in processing times.
Utilised Azure Data Factory Monitoring and Azure Databricks performance metrics to continuously assess and optimise data pipeline performance, ensuring smooth and efficient execution.
Collaborated directly with data scientists, business analysts, and data architects to identify data requirements and deliver actionable insights, promoting data-driven decision-making throughout the organisation.
Implemented robust data governance policies in streaming environments, encompassing data quality validation checks, schema validation, and error handling to ensure compliance with business standards.
Coordinated with Azure Data Lake Analytics to define data retention policies, ensuring consistent archiving and purging of real-time data streams based on retention rules.

Engaged closely with end customers to gather and comprehend requirements for designing and developing an agile data architecture in the Azure Cloud, optimally supporting the storage and management of retail data via Data Lake solutions. Collaborated with pipeline teams to automate deployment and integration pipelines, enhancing practices for data workflows in Azure Databricks. Developed and maintained Power BI dashboards and reports, transforming raw data into insightful business intelligence reports for executive and operational decision-making. Utilised Azure Monitor and Log Analytics to track pipeline health and proactively identify and resolve issues, ensuring smooth operations. Implemented Delta Lake for ACID transactions and schema evolution, safeguarding data consistency in data lakes and enhancing data governance. Constructed pipelines for managing intricate workflows of extraction, transformation, and aggregation from source systems, storing the processed data into ADLS using Azure Databricks Notebooks. Created tables and stored procedures in T-SQL for data extraction requested by business users. Assisted service developers in locating relevant content within existing reference models.

Engaged closely with end customers to gather and comprehend requirements for designing and developing an agile data architecture in the Azure Cloud, optimally supporting the storage and management of retail data via Data Lake solutions.
Collaborated with pipeline teams to automate deployment and integration pipelines, enhancing practices for data workflows in Azure Databricks.
Developed and maintained Power BI dashboards and reports, transforming raw data into insightful business intelligence reports for executive and operational decision-making.
Utilised Azure Monitor and Log Analytics to track pipeline health and proactively identify and resolve issues, ensuring smooth operations.
Implemented Delta Lake for ACID transactions and schema evolution, safeguarding data consistency in data lakes and enhancing data governance.
Constructed pipelines for managing intricate workflows of extraction, transformation, and aggregation from source systems, storing the processed data into ADLS using Azure Databricks Notebooks.
Created tables and stored procedures in T-SQL for data extraction requested by business users.
Assisted service developers in locating relevant content within existing reference models.