Around 3 Years of Data Engineer experience in all stages of software application development life cycle of data warehousing & and Data Lake applications using various software tools. Worked in all phases of the Software …
Around 3 Years of Data Engineer experience in all stages of software application development life cycle of data warehousing & and Data Lake applications using various software tools. Worked in all phases of the Software Development Lifecycle (SDLC) - Analysis, Design and Modeling, Development, System Testing, System Implementation, and Maintenance. Actively involved in ETL Pipeline design to meet requirements for extraction, transformation, cleansing, and loading of data from source to target data structures. Proficient in Azure Data Factory (ADF) for end-to-end ETL solutions, demonstrating expertise in designing, implementing, and optimizing data pipelines for seamless Extract, Transform, and Load processes across diverse data sources and destinations. Practical understanding of the Data modeling concepts like Star-Schema Modeling, Snowflake Schema Modeling, Fact and Dimension tables. Comprehensive knowledge and experience in process improvement, normalization, de- normalization, data extraction, data cleansing, and data manipulation. Implemented Slowly Changing Dimensions-Type I&II in Dimension tables as per the requirements. Involved in various projects related to Data Modeling, System/Data Analysis, Design, and development for both OLTP and OLAP Data warehousing environments. Proficient in crafting high-performance SQL queries, optimizing database operations, and ensuring efficient data retrieval. Adept at diagnosing and resolving complex performance issues, resulting in improved database responsiveness. Over 2 years of hands-on experience in Python programming, contributing to the development of robust and scalable applications. Proficient in leveraging Python's versatility to create efficient solutions for various business needs, from data processing scripts to web applications. Utilized Pandas for efficient data extraction and transformation, enabling impactful data analytics. Employed NumPy for numerical data processing and modeling, enhancing data science capabilities Experience in setting up, configuring, and maintaining Kafka clusters for seamless real-time data streaming and event-driven architecture. Experience in Developing Spark applications using Spark - SQL in Databricks for data extraction, transformation, and aggregation from multiple file formats for analysis. Familiar with data architecture including data ingestion pipeline design, Hadoop information architecture, data modeling and data mining, machine learning, and advanced data processing. Migrated a relational database to MongoDB, denormalizing data and optimizing document structure. Query performance improved by 40x. Involved in data manipulation using Python and spark scripts, which will be useful for faster data processing. Experience in Developing Spark applications using Spark - SQL in Databricks for data extraction, transformation, and aggregation from multiple file formats for analyzing and transforming the data to uncover insights into customer usage patterns. Expertise in conceptual, logical, and physical data modeling to effectively represent business requirements. Hands-on experience designing and implementing ETL/ELT pipelines for efficient data transformation and loading. Proficient in conceptualizing and designing data models to represent business requirements effectively. Experienced in utilizing industry-standard techniques, such as entity-relationship diagrams (ERD) and normalization, to create scalable and efficient database structures. Played a key role in Agile development processes, contributing to sprint planning, daily stand-ups, and sprint reviews. Demonstrated proficiency in utilizing Agile frameworks, such as Scrum or Kanban, to foster collaboration, adaptability, and the timely delivery of high-quality software.
Designing ETL pipelines to move data from Oracle to Snowflake using ADF.
Creating STREAMS in Snowflake to capture the CDC Data.
Developed the Data Quality check code by using the Azure services and PySpark language.
Engaged in Data migration from Oracle to Snowflake Datawarehouse by using ETL Pipelines & and ADF.
Leveraged JSON, Avro, and Parquet extensively for ingesting and processing semi-structured data from various sources into Azure Data Lake.
Optimized data pipelines and ETL jobs resulting in 50% improved performance.
Performing data analysis on Snowflake & oracle tables using Python.
Experience in Creating Multiple TASKS in snowflake to load the Kafka Topics from snowflake tables, and views.
Worked on creating the JDBC connector for source and targets to communicate with KAFKA.
Developed Spark applications using Spark-SQL for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
Implementing and supporting deployments to PROD and NON-PROD.
Monitoring the jobs during the pre-production phase and jobs running on PROD.
Reduced ETL job failures by 30% through Airflow monitoring, alerts, and robust error handling.
Worked on PySpark and Spark SQL logic to eliminate the bugs.
Experience in Different kinds of Data platforms support issues (CUBE, DATAMARTS) to give a better experience to the end users while consuming the data from the above Data Platforms. Benefits.
Developed the performance tuning of Spark Applications for setting the right Batch Interval time, correct level of Parallelism, and memory tuning.
Developed Notebooks and ETL Pipeline in Azure Data Factory (ADF) that process the data.
Involved in Configuring virtual machines, storage accounts, and resource groups.
Designed the Sink connector for ADLS to copy the data from Apache Kafka Topics.
Experience utilizing Python for data analysis and automation.
Highly proficient in SQL for data manipulation and retrieval.
Experienced in leveraging Python and Pandas/NumPy for data analysis and workflow automation.
Advanced Excel skills for data visualization and reporting.
Designed and implemented data warehousing solutions, including ETL processes and schema designs, optimizing data storage and retrieval efficiency.
Implemented and managed a comprehensive data governance framework, defining policies, standards, and procedures to ensure data quality, integrity, and security across the organization.
Led initiatives to establish data stewardship roles and responsibilities.
Ensured compliance with industry regulations and data privacy standards by implementing data governance practices.
Developed and enforced policies for data classification, access controls, and audit trails, promoting a culture of responsible data management and protection.
Experience in leveraging Snowflake EDW to support analytical and business intelligence needs.
Worked on Gathering the data modeling requirements and design and implementation of the Conceptual, Logical, and Physical models.
At Wells Fargo, I worked with credit card transactions where the source data was in Oracle, and the destination was Snowflake EDW. For ETL purposes, we utilized Azure Data Factory (ADF) to copy the data to the ADLS raw layer. Subsequently, in the sanitized layer, we performed transformations, like hiding sensitive information such as SSN and credit card numbers,using Azure Data Bricks (ADB). Using ADF again, we moved the transformed data to the provision layer, and finally, leveraging ADF, we loaded the data into Snowflake EDW.
Analysis, Design and Build Modern data solutions using Azure Cloud services to support visualization of data.
Understand the current Production state of the application and determine the impact of new implementation on existing business processes.
Extract Transform and Load data from the different Sources Systems to Azure Data Storage services using a combination of Azure Data Factory, Spark SQL and Python, and Azure Data Lake Analytics.
Data Ingestion to one or more Azure Services - (Azure Data Lake, Azure Storage, Azure SQL, Azure DW) and processing the data in Azure Databricks.
Created Pipelines in ADF using Linked Services/Datasets/Pipeline/ to Extract, Transform, and load data from different sources like Azure SQL, Blob storage, Azure SQL Data warehouse, write-back tool, and backward.
Responsible for Azure Data Factory job monitoring and troubleshooting the failures and providing the resolution for the ADF jobs failures.
Experienced in designing and implementing data models in Hive, including defining tables, partitions, and optimizing schema structures for efficient querying.
Developed Spark applications using PySpark and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats for analyzing & transforming the data to uncover insights into the customer usage patterns.
Worked on Parquet file format and other kind of different file types.
Responsible for estimating the cluster size, and monitoring and troubleshooting of the Spark data bricks cluster.
Extensively used Pandas to create Shared functions to perform DATE, COUNT, and MERGE in Numerous Azure Databricks Notebooks.
Managed large datasets using Panda data frames and MySQL.
Used Pandas Modules to Analyze data stored in Databases and files and to create the reports.
Experienced in performance tuning of Spark Applications for setting the right Batch Interval time, the correct level of Parallelism, and memory tuning.
Worked on Production bugs especially those involved in Azure Databricks Notebooks bugs and provided the new PySpark and Spark SQL logics to eliminate the bugs.
Developed Notebooks and ETL Pipeline in Azure Data Factory (ADF) that process the data according to the job trigger.
Involved in creating a Data freshness Dashboard by using Power BI to generate the application health reports.
Hands-on experience in developing SQL Scripts for automation purposes.
The AI-first Recruitment Operating System that unifies and automates the entire hiring lifecycle for modern enterprises worldwide.
Dr Job FZ LLC. 2026 © All Rights Reserved
