نبذة عني
6+ years in Data Engineering and implementing Hadoop, Spark, and cloud data warehousing solutions. Expertise in providing ETL solutions for diverse business models, ensuring data integration and transformation. Proficien…
6+ years in Data Engineering and implementing Hadoop, Spark, and cloud data warehousing solutions. Expertise in providing ETL solutions for diverse business models, ensuring data integration and transformation. Proficiency in AWS services such as EC2, S3, RDS, IAM, Glue, Redshift, DynamoDB, and more, enabling cloud-based data solutions. Experience with AWS databases, NoSQL databases, and data modeling, optimizing data storage and retrieval. Collaborated with data scientists and architects, utilizing Databricks to create and maintain data models. Developed scalable systems using Hadoop technologies, including HDFS and MapReduce, for efficient data processing. Extensive experience with Informatica PowerCenter for data integration and transformation. Deployed Big Data Hadoop applications using Talend on both AWS and Azure cloud platforms. Built database models, APIs, and views using Python to develop interactive web-based solutions. Collaborated with business users to identify data gaps, developing corrective actions for data integrity. Recreated application logic in Azure Data Lake, Data Factory, SQL Database, and SQL Data Warehouse. Utilized PL/SQL procedures in ETL transformations for effective data processing. Designed and developed ETL processes using AWS Glue to migrate data from external sources into AWS Redshift. Leveraged AWS AppSync (GraphQL) for web API creation and data synchronization into Aurora Postgres or DynamoDB. Migrated ETL code from Talend to Informatica, ensuring a smooth transition of data processes. Backed up AWS Postgres to S3 on a daily basis using EMR, safeguarding data integrity. Involved in the full lifecycle development of migration projects, from development to testing and post-production support. Tuned ETL jobs in the new environment, improving process efficiency after identifying bottlenecks. Maintained the Talend admin console and provided quick assistance for production jobs, ensuring system reliability. Designed Data Models and Dimensional Modeling for OLAP and Operational Data Store (ODS) applications. Tuned performance of Informatica mappings and sessions, optimizing data processing for better performance. Utilized SQL queries in BigQuery to extract valuable insights and perform data transformations as needed. Implemented software updates and patches for system maintenance, ensuring a healthy IT ecosystem. Strong understanding of Enterprise Data Warehouse best practices, ensuring data quality and consistency Built automation regression scripts for ETL process validation between multiple databases, enhancing data quality. Worked on migration projects that involved moving web methods code to the Informatica Cloud platform. Skilled in designing and implementing cost-effective and efficient ETL architectures. Extensive experience in identifying and tuning performance bottlenecks in Ab Initio Load graphs and SQL queries. Conducted complex data analysis and provided critical reports to support various departments. Proficient in Business Intelligence tools like Business Objects and Data Visualization tools such as Tableau. Utilized Shell and Python scripting for scheduling and process automation, streamlining data operations. Developed effective working relationships with client teams, ensuring clear communication and successful project delivery. Implemented software updates and patches for Control-M to maintain system health and performance. Strong understanding of best practices in Enterprise Data Warehousing, guaranteeing data consistency and reliability. Collaborated with business users to identify root causes of data gaps and developed corrective actions to maintain data integrity. Recreated application logic in Azure Data Lake, Data Factory, SQL Database, and SQL Data Warehouse for improved performance. Utilized PL/SQL procedures in ETL transformations to optimize data processing for diverse business needs.
الخبرة
Data Engineer
Developed data pipelines for data gathering, transformation, and loading into data lakes or warehouses, a common practice in retail for data analysis.
Created efficient Hive tables with static and dynamic partitions to manage and analyze retail data.
Implemented ETL tasks to extract and load data from multiple sources, a fundamental process for retail data integration and analytics.
Developed Spark applications for data extraction, transformation, and aggregation from various file formats commonly used in retail for big data processing.
Built real-time monitoring dashboards using AWS Lambdas to track sales, inventory, and customer behavior in real time.
Worked on data visualization using Python and Tableau for presenting and interpreting retail data insights.
Proficient in data ingestion techniques, including real-time and batch data ingestion, crucial for managing the constant flow of data.
Data Engineer
Developed data pipelines for data gathering, transformation, and loading into data lakes or warehouses.
Created efficient Hive tables with static and dynamic partitions to manage and analyze retail data.
Implemented ETL tasks to extract and load data from multiple sources.
Developed custom Kafka Connect connectors in Java or Python to integrate Kafka with proprietary or legacy systems.
Developed Spark applications for data extraction, transformation, and aggregation from various file formats.
Built real-time monitoring dashboards using AWS Lambdas to track sales, inventory, and customer behavior in real time.
Worked on data visualization using Python and Tableau for presenting and interpreting retail data insights.
Proficient in data ingestion techniques, including real-time and batch data ingestion.
Implemented data caching strategies to reduce query response times.
Developed Map Reduce programs with Apache Hadoop for handling large volumes of data.
Managed Hadoop infrastructure with Cloudera Manager to ensure the stability and scalability of data processing.
Worked on job migration from Tidal to Control-M, involving scheduling retail-specific tasks and automating data processes.
Created Python wrapper scripts for data extraction using Sqoop.
Created and maintained technical documentation for launching Hadoop clusters and executing Hive queries.
Designed and implemented large-scale distributed solutions in AWS and GCP clouds.
Worked with Salesforce integration.
Conducted data extraction, transformation, comparison, and loading from DB2 into a data warehouse using Teradata Client Utilities.
Conducted testing, data validation, and data comparison between source and target applications to ensure data accuracy.
Implemented Kafka Streams API using Java or Scala to perform complex transformations on incoming data streams.
Optimized SQL queries and data processing workflows to enhance the performance and efficiency of data operations.
Worked on data modeling, including creating and maintaining data schemas, to support advanced analytics and reporting.
Developed complex streaming pipelines involving joins, aggregations, and windowed operations on Kafka topics using Data Frame and Dataset APIs in Spark.
Conducted performance tuning and optimization of data pipelines and processes.
Identify and prioritize critical business rules governing retail data, ensuring they are consistently validated in unit tests.
Collaborated with cross-functional teams to gather, analyze, and prioritize business requirements.
Conducted data profiling and analysis to identify trends and patterns in retail data.
Data Engineer
Built a reporting data warehouse from the ERP system.
Developed data models and deployed data pipelines to meet Chevron's business requirements.
Utilized Big Data and Hadoop Ecosystem components within Azure as a Data Engineer.
Worked with Neo4j, a NoSQL graph database, to create data models and relationships.
Collaborated with reporting developers to implement report and universe designs.
Assisted data scientists and architects in creating and maintaining data models using Databricks.
Developed data pipelines using tools like Flume, Sqoop, and Pig to extract and store data from weblogs.
Utilized PowerShell and UNIX scripts for file transfer, email communication, and other file-related tasks.
Participated in the deployment process from development to production.
Worked with Informatica Cloud for data integration between Salesforce, RightNow, Eloqua, and web services applications.
Monitored and optimized query performance within Big Query.
Leveraged Teradata SQL and Client Utilities for data migration to data warehouses.
Monitored and fine-tuned query performance within Azure's Big Query.
Utilized sensor data from oil wells and drilling equipment to predict and prevent equipment failures.
Integrated data from various sources, including seismic data, well logs, and drilling reports.
Worked with data scientists to develop predictive models for oil and gas production forecasts.
Demonstrated expertise in ETL processes using Spark, Spark SQL, and Spark Streaming.
Developed customized Airflow plugins within Azure to enhance data workflow management.
Collaborated on a migration project involving web methods code to Informatica Cloud.
Worked as a Data Engineer using Big Data and Hadoop Ecosystem components to build highly scalable data pipelines.
Automated data extraction from warehouses and weblogs within Azure using Oozie and NiFi.
Worked closely with business users to identify and address data gaps.
Developed custom Airflow plugins to enhance the capabilities of Airflow for managing data workflows.
Vizient, Dallas, TX
Worked as a Data Engineer using Big Data and Hadoop Ecosystem components to build highly scalable data pipelines for petroleum-related products.
Data Engineer
Designed and implemented a healthcare-specific Data Warehouse, aggregating data from various sources including claims data, EHR, and medical billing data.
Utilized data technologies and tools like Hadoop, Spark, and Hive for efficient data processing and analysis.
Developed and maintained ETL processes for accurate extraction and transformation of healthcare data.
Collaborated on the refinement and creation of BI reports and dashboards, integrating AWS QuickSight for enhanced visualization and analytics capabilities.
Administered user accounts and access in tools like PowerCenter for secure data management.
Monitored and managed Hadoop clusters to maintain data availability and system performance.
Managed user accounts and secure data access using AWS-managed services like IAM.
Employed a suite of data technologies such as Hadoop, Spark, and Hive within AWS infrastructure.
Demonstrated the impact of work through quantifiable results, such as reduced processing time and improved data accuracy.
Implemented data quality checks and validation procedures to ensure the accuracy and reliability of healthcare data.
Integrated data with healthcare systems to facilitate improved patient care.
Established an automated CI/CD pipeline using Git, Jenkins, and custom tools for efficient data processing.
Collaborated with data scientists and analysts to create predictive models and machine learning algorithms for healthcare analytics.
Developed data pipelines for the ingestion of streaming healthcare data.
Created complex SQL, PL/SQL, and Spark SQL queries for data analysis and reporting.
Worked on optimizing data storage and retrieval for large healthcare datasets.
Implemented data governance practices to maintain data integrity and enforce data security and privacy regulations.
Conducted data profiling and data lineage analysis to track the flow of healthcare data and identify potential issues.
Leveraged cloud services, including AWS, for scalable and cost-effective data solutions.
Automated data cleansing and transformation tasks to streamline the ETL process and reduce manual intervention.
Conducted performance tuning of SQL and Spark queries to optimize query execution times and resource utilization.
Assisted in disaster recovery planning and data backup strategies to ensure data availability and continuity of healthcare services.
Implemented data encryption and access control mechanisms to protect sensitive healthcare information.
Created and maintained data documentation, including data dictionaries and metadata, for data transparency and understanding.
Collaborated with third-party healthcare data providers to acquire and integrate external data sources for analysis.
Monitored data pipelines for anomalies and implemented automated alerting and notification systems.
Stayed current with healthcare industry trends and emerging data technologies to drive innovation and efficiency in data engineering processes.
Investigated and promptly resolved data discrepancies, providing swift resolution to address data-related challenges within AWS environments.
Implement robust data security measures to protect sensitive healthcare information from unauthorized access and breaches.
Implement real-time data streaming solutions for monitoring patient data, enabling immediate interventions when necessary.