Yazan Gheith

Design and implement database solutions in Azure SQL Data Warehouse and Azure SQL.
Analyze, design, and build modern data solutions using Azure PaaS service to support visualization of data.
Understand current production state of application and determine the impact of new implementation on existing business processes.
Extract, transform, and load data from source systems to Azure Data Storage services using Azure Data Factory, T-SQL, Spark SQL, and U-SQL Azure Data Lake Analytics.
Ingest data to Azure Data Lake, Azure Storage, Azure SQL, and Azure DW and process the data in Azure Databricks.
Implemented proof of concepts for SOAP and REST APIs.
Developed REST APIs to retrieve analytics data from different data feeds.
Developed Spark applications using Spark and Spark-SQL for data extraction, transformation, and aggregation from multiple file formats.
Created Databricks notebooks using Python, Scala, and Spark SQL for transforming data stored in Azure Data Lake Storage Gen2 from Raw to Stage and Curated zones.
Built technology demonstrators using Confidential Edison Arduino shield with Azure EventHub and Stream Analytics, integrated with Power BI and Azure ML.
Responsible for estimating cluster size, monitoring, and troubleshooting the Spark Databricks cluster.
Created Airflow scheduling scripts in Python.
Develop conceptual solutions and create proof-of-concepts to demonstrate viability of solutions.
Technically guide projects through to completion within target timeframes.
Collaborate with application architects and DevOps.
Identify and implement best practices, tools, and standards.
Design, set up, and maintain Azure SQL Database, Azure Analysis Service, Azure SQL Data Warehouse, and Azure Data Factory.
Develop code in Databricks and conduct unit testing before deploying to UAT and production servers.
Build complex distributed systems involving large-scale data handling, collecting metrics, building data pipelines, and analytics.
Built servers using AWS including importing volumes, launching EC2 and RDS, creating security groups, auto-scaling, and load balancers in a virtual private connection.
Managed storage in AWS using Elastic Block Storage and S3, created volumes, and configured snapshots.
Utilized AWS CLI to automate backups of ephemeral data stores to S3 buckets, EBS, and create nightly AMIs for mission critical production servers.
Set up CI/CD pipeline using Jenkins, Maven, Nexus, GitHub, Chef, Terraform, and AWS.
Worked with AWS CodePipeline and created CloudFormation JSON templates converted to Terraform for infrastructure as code.
Implemented Terraform modules for deployment of applications across multiple cloud providers.
Configured an AWS Virtual Private Cloud and Database Subnet Group for isolation of resources within the Amazon RDS Aurora DB cluster.
Used CloudFront to deliver content from AWS edge locations to users.
Used AWS Elastic Beanstalk for deploying and scaling web applications and services developed with Java, Node.js, Python, and Ruby.
Implemented and maintained monitoring and alerting of production and corporate servers and storage using CloudWatch.
Worked with DevOps practices using AWS, Elastic Beanstalk, and Docker with Kubernetes.
Managed network security using load balancer, auto scaling, security groups, and NACLs.
Used Jenkins and pipelines to drive microservices builds to the Docker registry and deployed to Kubernetes, created pods and managed using Kubernetes.
Experienced in performance Spark applications using Python utilizing DataFrames and Spark SQL API for faster processing of data.
Involved in architecture of Spark Engine for fast in-memory data processing and to save data in Hive tables.
Developed custom User Defined Functions in Spark to transform large volumes of data.
Worked in applying standardization, normalization, and transformations on data when passing through each layer.
Spark job processed data and pushed it to ADS S3 enriched data bucket using utility to write in Parquet.
Implemented usage of Amazon EMR for processing Big Data across a Hadoop cluster of virtual servers on Amazon EC2 and Amazon S3.
Improved performance and optimization of existing algorithms in Hadoop using Spark Context, Spark-SQL, and DataFrame.
Loaded structured and semi-structured data into Spark clusters using Spark SQL and DataFrames API.
Worked in applying standardization, normalization, and transformations on data when passing through each layer.
Worked on S3 buckets on AWS to store CloudFormation templates and worked on AWS to create EC2 instances.
Written AWS Lambda functions to extract incident notification data from REST APIs and wrote data to S3 by parsing JSON.
Worked on external feeds built using Amazon S3, Lambda, SNS, EMR, EFG, shell scripting, and Python.
Worked with different file formats like CSV, Avro, and Parquet for Hive querying and processing based on business logic.

• 7 years of IT experience as a Developer, Designer, & Quality Tester with cross platform integration experience using Hadoop, Spark, & Kafka. As well as Cloud Platforms such as AWS, GCP, & Azure.
• Proficiency in programming languages like Python, SQL, Scala, and data processing frameworks such as Apache Spark enabling versatile data manipulation, analysis, and API development.
• Hands on experience in installing, configuring and using Hadoop Ecosystem - HDFS, MapReduce, Pig, Hive, Oozie, Flume, HBase, Spark, Sqoop.
• Strong understanding of various Hadoop services, MapReduce and YARN architecture.
• Experience in big data technologies: Hadoop HDFS, Map-reduce, Pig, Hive, Oozie, Sqoop, Zookeeper and NoSQL.
• Hands on experience using AWS services such as Amazon EMR (Elastic MapReduce), Amazon Redshift, Amazon Glue, Amazon Kinesis, Amazon Athena, AWS Data Pipeline, AWS Lambda, Amazon S3 (for data storage), AWS Step Functions, Amazon DynamoDB (for NoSQL data), AWS Database Migration Service (DMS), AWS Data Sync, AWS Direct Connect, & Cloud Watch Events.
• Extensive expertise in AWS orchestration using AWS Step Functions and serverless data processing using AWS Lambda, ensuring scalable and cost-effective data solutions.
• Proficient in designing, developing, and maintaining data pipelines and ETL processes using AWS Glue, Apache Airflow, Spark, Sqoop, and Snowflake, with a keen understanding of optimizing data workflows for performance and reliability
• Proficient in Azure data services, including Azure Databricks, Azure Data Factory, Azure Synapse Analytics (formerly SQL Data Warehouse), and Azure Blob Storage, enabling the creation of scalable and performant data processing pipelines.
• Strong knowledge of Azure's data migration and transfer services, such as Azure Data Box, Azure Data Factory Copy Data Tool, and Azure Data Share, to facilitate data migration, replication, and synchronization.
• Proficiency in data warehousing and analytics solutions on Azure, employing services like Azure Data Warehouse, Azure Analysis Services, and Power BI for advanced data analytics and reporting.
• Strong knowledge of Google BigQuery,Google Cloud Dataproc, Google Cloud Dataflow, Google Cloud Composer, Google Cloud Pub/Sub, Google Cloud Bigtable, Google Cloud AutoML, Google Cloud Data Catalog, Google Cloud DLP, Google Cloud Storage, Google Cloud Transfer Service, and other serverless tools, enabling comprehensive data engineering solutions, real-time data processing,

Design and implement database solutions in Azure SQL Data Warehouse and Azure SQL.
Engineered a reusable Azure Data Factory based data pipeline infrastructure that transforms provisioned data to be available for consumption by Azure SQL Data Warehouse and Azure SQL DB.
Created ADF pipelines to extract data from on-premises source systems to Azure cloud data lake storage.
Worked on copy activities and implemented copy behaviors such as flatten hierarchy, preserve hierarchy, and merge hierarchy.
Implemented error handling through copy activity.
Worked on Azure Data Lake Analytics with Azure Databricks to implement SCD-1 and SCD-2 approaches.
Developed Spark notebooks to transform and partition the data and organize files in ADLS.
Worked on Azure Databricks to run Spark-Python notebooks through ADF pipelines.
Worked on migration of data from on-prem SQL Server to cloud databases including Azure Synapse Analytics and Azure SQL DB.
Created Linked Services for multiple source systems including Azure SQL Server, ADLS, BLOB, and REST API.
Implemented delta logic extractions for various sources with control tables.
Implemented data frameworks to handle deadlocks, recovery, and logging of pipeline data.
Created a Power BI data model based on analysis of the end-user workflow data provided by the client.
Imported data from SQL Server DB and Azure SQL DB to Power BI to generate reports and dashboards.
Developed analysis reports and visualization using DAX functions.
Implemented and maintained monitoring and alerting of production and corporate servers such as EC2 and storage such as S3 buckets using AWS CloudWatch.
Experience in automated deployment of EC2 instances in data centers and availability zones.
Used security groups to develop a logical firewall to ensure high security for control applications.
Used Amazon IAM to grant fine access of AWS resources to users and managed roles and permissions through IAM.
Used CloudFront to deliver content from AWS edge locations to users.
Deploying applications using Lambda, ECS, and Docker containers.
Wrote AWS Lambda functions in Python to perform various transformations and analytics on large data sets in EMR clusters.
Experience in cloud databases and data warehouses including SQL and AWS Redshift/RDS.
Designed and built multi-terabyte end-to-end data warehouse infrastructure on Amazon Redshift.
Set up Auto Scaling Groups based on memory and CPU.
Set up Elastic Load Balancers for different applications.
Used Amazon S3 to backup database instances periodically to save snapshots of data.
Used Amazon Route53 to manage DNS zones and public DNS names to load balancers.
Implemented data pipelines to move data from DynamoDB to Redshift for reporting.
Used CloudWatch logs to move application logs to S3 and create alarms.
Configured an AWS Virtual Private Cloud and database subnet group for isolation of resources within AWS RDS.
Used Amazon RDS Multi-AZ for automatic failover and high availability at the database tier for MySQL workloads.
Configured S3 versioning and lifecycle policies to backup files and archive files in Glacier.
Designed high availability applications on AWS across availability zones and regions.
Created Hive tables and worked on them using HiveQL and designed and implemented partitioning and buckets in Hive.
Involved in building applications using Maven and integrated with CI servers like Jenkins to build jobs.
Configured, deployed, and maintained multi-node Dev and Test Kafka clusters and implemented data ingestion and handling clusters in real-time processing using Kafka.
Created the cube in Talend to create different types of aggregation in the data and visualize them.
Monitored Hadoop NameNode health status, number of TaskTrackers running, number of DataNodes running and automated jobs from pulling data from different sources like MySQL to pushing results to HDFS.
Developed story-telling dashboards in Tableau Desktop and published them on Tableau Server and used GitHub version control tools to maintain project versions.

Involved in Hive/SQL queries performing Spark transformations using Spark RDDs and Python.
Created a serverless data ingestion pipeline on AWS using Lambda functions.
Configured Spark Streaming to receive real-time data from Apache Kafka and store the stream data to DynamoDB using Scala.
Developed Apache Spark applications using Scala and Python.
Implemented Apache Spark data processing module to handle data from various RDBMS and streaming sources.
Developed and scheduled various Spark Streaming and batch jobs using Python and Scala.
Developed Spark code using PySpark to apply various transformations and actions for faster data processing.
Achieved high-throughput, scalable, fault-tolerant stream processing of live data streams using Apache Spark Streaming.
Used Spark stream processing using Scala to get data into in-memory, created RDDs and DataFrames, and applied transformations and actions.
Used Python libraries with Spark to create data frames and store them to Hive.
Created Sqoop jobs and Hive queries for data ingestion from relational databases to analyze historical data.
Worked with Elastic MapReduce and setting up environments on Amazon EC2 instances.
Knowledge of handling Hive queries using Spark SQL that integrates with Spark environment.
Executed Hadoop/Spark jobs on AWS EMR using programs stored in S3 buckets.
Knowledge of creating user defined functions in Hive.
Worked with different file formats like CSV, Avro, and Parquet for Hive querying and processing based on business logic.
Worked on Sequence files, RC files, map side joins, bucketing, and partitioning for Hive performance enhancement and storage improvement.
Implemented Hive UDFs to implement business logic and performed extensive data validation using Hive.
Involved in loading structured and semi-structured data into Spark clusters using Spark SQL and DataFrames API.
Involved in developing code and generated various data frames based on the business requirement and created temporary tables in Hive.
Utilized AWS CloudWatch to monitor the performance environment instances for operational and performance metrics during load testing.
Scripting Hadoop package installation and configuration to support fully automated deployments.
Involved in Chef-infra maintenance including backup and security fix on Chef Server.
Deployed application updates using Jenkins and installed, configured, and managed Jenkins.
Triggered the SIT environment build of client remotely through Jenkins.
Deployed and configured Git repositories with branching, forks, tagging, and notifications.
Experienced and proficient in deploying and administering GitHub.
Deployed builds to production and worked with teams to identify and troubleshoot issues.
Worked on MongoDB database concepts such as locking, transactions, indexes, sharding, replication, and schema design.
Consulted with the operations team on deploying, migrating data, monitoring, analyzing, and tuning MongoDB applications.
Viewed selected issues of web interface using SonarQube.
Developed a fully functional login page for the company's user facing website with complete UI and validations.
Installed, configured, and utilized AppDynamics in the JBoss environment.
Responsible for upgradation of SonarQube using upgrade center.
Resolved tickets submitted by users, P1 issues, troubleshot errors, and documented and resolved errors.
Installed and configured Hive in Hadoop cluster and helped business users/application teams fine tune their HiveQL for optimizing performance and efficient use of resources in cluster.
Conducted performance tuning of the Hadoop cluster and MapReduce jobs.
Implemented Oozie workflow for ETL process for critical data feeds across the platform.
Configured Ethernet bonding for all nodes to double the network bandwidth.
Implemented Kerberos security authentication protocol for existing cluster.
Built high availability for major production cluster and designed automatic failover control using Zookeeper Failover Controller and Quorum Journal nodes.