Vakula

Azure Data Engineer with 10 Years of IT Industry experience and 4 Years of experience in Azure Data Factory and Databricks with proven knowledge in design, development, testing, and optimizing of data pipelines in both on-premises and Azure cloud platforms. Aiming to obtain an elevated data professional career in data and analytics.

Implemented Azure Data Factory (ADF) extensively for ingesting data from different source systems like relational and unstructured data to meet business functional requirements.
Developed Batch and Near Real Time data processing using ADF and Azure Databricks.
Developed python and pyspark scripts on Azure Databricks platform for processing data from various sources and variety of formats.
Created numerous pipelines in Azure using Azure Data Factory v2 to get the data from disparate source systems by using different Azure Activities like Move &Transform, Copy, filter, for each, Databricks etc.
Experience in building Lakehouse using Azure Databricks and open-source Delta (DataLake).
Automated jobs using different triggers like Events, Schedules and Tumbling in ADF.
Used Azure Devops & Jenkins pipelines to build and deploy different resources(Code and Infrastructure) in Azure.
Experienced in developing audit, balance and control framework using SQL DB audit tables to control the ingestion, transformation and load process in Azure.
Implemented both ETL and ELT architectures in Azure using Data Factory, Databricks, SQL DB and SQL Data warehouse.
Worked with team members to resolve any technical issue, Troubleshooting, Project Risk & Issue identification, and management.
Using Azure Synapse to stage data and created external tables in azure Synapse loaded data from ADLS Location.
Created High level technical design documents and Application design document as per the requirements and delivered clear, well-communicated and complete design documents.
Scheduled, automated business processes and workflows using Azure Logic Apps.

Involved in gathering & analysis of business Requirements from End users & internal Business Analysts to Develop Strategies for ETL processes.
Experience in Performance Tuning - Identified and fixed bottlenecks and tuned the complex mappings for better Performance.
Worked on SQL Scripts, T-SQL Stored procedures, queries and packages to load data in SQL Server, and SQL Data warehouse.
Experience in developing Spark applications using Data frames, Spark SQL, and Notebooks in Databricks for data extraction, transformation, and aggregation from multiple file formats to uncover insights.
Used pandas to transform the raw data and upload to ftp sites.
Build data pipelines, data processing tools and technologies in open source and proprietary products.
Developed several Azure Data Factory re-usable pipelines, linked services and activities with parameterization techniques which can be used to cut down development time.
Used Pandas in Python for Data Cleansing and validating the source data.
Built Teradata ELT frameworks which ingests data from different sources using Teradata Legacy load utilities.
Used Python and Shell scripts to Automate Teradata and SQL Server ELT activities

Worked as a ETL SSIS, SSRS Developer.
Created SSIS packages to migrate data from different sources to data warehouse.
Successfully extracted transformed and load data into data warehouse.
Used various SSIS tasks such as conditional Split, Derived Column etc, which did Data Scrubbing, including data validation checks during Staging, before loading the data into the Data warehouse.
Created ETL packages with different data sources (SQL Server, Flat files, Excel source files, Oracle) and loaded the data into target tables by performing different kinds of transformations using SSIS.
Prepare Unit Test Plan and Integrated test Plan with interface modules.
Applied various data transformations like Aggregate, Sort, Multicasting, Conditional Split, Derived column etc.
Data quality done by using the Data cleansing by using ETL development
Designing Spotfire dashboards / reports with complex multiples data sources (SQL, Excel, Flat files) relational design and optimizing applications.
Deploying and troubleshooting the SSIS Packages.

Worked as a ETL Data test Engineer
Created Test plans and cases to test the migration of data from different sources to data warehouse.
Created a SQL Queries to Successfully extract and validate the transformed data into data warehouse.
Verified reports and data ingestion pipelines for quality
Created Test reports and finding the defects in data pipelines
Prepare Unit Test Plan and Integrated test Plan with interface modules.