Overall 3.5 years of experience in data engineering handling Big Data Ecosystem with technologies including Spark,HDFS, Sqoop, Hive, AWS, SQL Exposure on cloud technology of AWS – S3 ,EMR. Proficient in using Spark RDD b…
Overall 3.5 years of experience in data engineering handling Big Data Ecosystem with technologies including Spark,HDFS, Sqoop, Hive, AWS, SQL Exposure on cloud technology of AWS – S3 ,EMR. Proficient in using Spark RDD based data processing using Scala programming language Expertise in using Spark DataFrame transformations and actions to process large-scale structured and semi-structured data sets, including filtering, mapping, reducing, grouping, and aggregating data. Familiarity in using Spark SQL to process large-scale structured and semi-structured data sets, including querying, filtering, mapping, reducing, grouping, and aggregating data. Worked with different data serialization formats (Avro, Parquet, JSON, etc.) in Spark Enhanced performance by Implemented Spark partitioning and caching strategies. Worked on Spark functions for complex data processing. Proficient in Hive data types, schemas, partition, buckets and file formats Experienced in efficiently using Hive managed and external table with respect to the business requirement. Ability to optimize Hive queries for performance, scalability and efficiency using techniques such as indexing, partitioning, caching, etc. Knowledge of Hive table formats, including ORC, Parquet, and Avro, and their advantages and disadvantages for different use cases. Experienced in importing , exporting and writing sqoop commands for transfer of data between Hadoop and relational databases using Sqoop. Proficient in configuring and using Sqoop jobs for incremental data transfers using Sqoop's incremental import feature. Used Sqoop for Import of data with different file formats like csv, avro and parquet Worked with import and export of data from cloud platform storage services like Amazon S3 using Sqoop Have optimized Sqoop configurations for maximum performance in various data transfer scenarios. Team player with strong sense of ownership.
Processed and analysed large volumes of structured and unstructured data using Spark
Used Dataframe to query various data sources such as csv, parquet, avro, json
Performed various operations like joins, filters, maps, etc to deliver the data as per business requirements
Performed data cleansing, transformations and aggregations using Spark
Achieved 50% improvement in data processing speed and 30% reduction in resource consumption by optimizing Spark performance and tuning spark parameters
Collaborated with data-scientists and business analysts to understand the data requirements and deliver insights
Developed custom Spark functions for complex data processing using scala
Designed and optimized Spark jobs for join operations
Performed operations using AWS S3 Storage buckets, importing and exporting data to and from S3 using local terminal
Proficient in setting AWS EC2 instance creation and configuration for optimal performance
Hands-on experience with real-time scenarios with AWS EMR
Developed Spark applications for distributed data processing
Created and managed RDDs (Resilient Distributed Distributeds) for data transformations
Utilized spark sql for structured data manipulation and analysis
Designed and implemented Spark jobs using Scala
Managed Spark libraries and dependencies
Performed data cleansing and pre-processing using Spark transformations
Worked in querying Hive tables using SQL-like syntax and performing data analysis
Enhanced Hive query performance by tuning various configuration settings, such as memory allocation, parallelism, and compression
Performed hive partitions for faster query performance
Used Sqoop to load data into Hive tables for further processing and analysis
Performed imports of different types of serialized data using Sqoop
Have created and managed Hive tables, including managed, external, and partitioned tables
Experienced in designing efficient data models using Hive tables and partitions to optimize query performance
Used different Hive table formats, including ORC, Parquet, and Avro as per their advantages and disadvantages for different use cases
Performed hive schema evolution with avro file format
Involved in setting up and configuring Sqoop-based data import and export solutions for large-scale data warehousing projects
Worked on creating and managing Sqoop jobs automation for efficient import of data from RDBMS
Used Sqoop to import and export data between Hadoop clusters and data lakes such as Amazon S3
نظام التوظيف التشغيلي المعتمد على الذكاء الاصطناعي أولاً، الذي يوحّد ويؤتمت دورة التوظيف بالكامل للمؤسسات الحديثة حول العالم.
د.جوب منطقة حرة ذ.م.م. 2026 © جميع الحقوق محفوظة
