Looking for a challenging role in Data Engineering, Data ETL, Cloudera Spark and Hadoop Developer substantially experienced in designing and executing solutions for complex business problems involving large scale data wa…
Looking for a challenging role in Data Engineering, Data ETL, Cloudera Spark and Hadoop Developer substantially experienced in designing and executing solutions for complex business problems involving large scale data warehousing, real-time analytics and reporting solutions.
Creating tables and views in Impala using HUE Editor in CDP. Scheduling the workflow using Oozie with Shell scripts. Extracting the data from SAP to Google Cloud Storage for historical data. Performing the CDC changes using Dataflow and uploading the data into Bigquery. Workflow orchestration and scheduling using Cloud Composer.
Creating tables and views in Impala using HUE Editor in CDP.
Scheduling the workflow using Oozie with Shell scripts.
Extracting the data from SAP to Google Cloud Storage for historical data.
Performing the CDC changes using Dataflow and uploading the data into Bigquery.
Workflow orchestration and scheduling using Cloud Composer.
Creating tables and views in Impala using HUE Editor in CDP.
Scheduling the workflow using Oozie with Shell scripts.
Extracting the data from SAP to Google Cloud Storage for historical data.
Performing the CDC changes using Dataflow and uploading the data into Bigquery.
Workflow orchestration and scheduling using Cloud Composer.
Work on moderately to provide technical support for migration in datapipeline using Cloudera Manager and troubleshoot performance issue to escalating with Linux support team.
Implement complete End-to-end development Solutions for the customers and provide technical support on daily basis.
Primarily handle Datapipeline ETL (Hive, Sqoop, Spark) and Google Big Query projects as a dedicated engineer thereby work on new data implementations, configuration changes and job module support.
Work with Data Analyst/Scientist team and customer on low level design and implementation aspects thereby preparing job implementaion(Financial Reports)in datapipeline as pre-implementation guideline.
Plan migration support activities in datapipeline StoreSimple 5000 to StoreSimple 8100 customer requirements and inputs thereby ensuring.
On premises data report for the internal teams based and monthly reports.
Creating tables and views using Impala and scheduling jobs in Oozie.
Extracting the on-prem data into GCP Cloud Storage and Bigquery for analytics.
Capturing CDC using Dataflow and Workflow orchestration using Cloud Composer.
Managing over 60+ job modules in Datapipeline ETL – by troubleshooting and maintaining in case of
failures. Proactively identify, diagnose, analyze and troubleshoot the issues on customer data in data
ingestion as well as Google Big Query. Create and design ETL pipeline based on the client requirements in Linux platform using Shell
script/python. Import and export the data from database using Sqoop jobs. Upload the transformed data into Hive tables using partition techniques. Performance tuning of the job modules using PySpark and import data into Big Query. Implement regular upgrades to the Google Big Query based on table. Document the issues and solutions in the Confluence in Wiki for the client.
Programming Language Python
Data Ingestion Sqoop,Nifi
Data Proccessing Spark,Hive,Impala
Messaging System Kafka
Cloud Technologies GCP(Google Big Query,Google Cloud Storage)
SQL &No SQL Oracle, Mongo DB
Operating System Linux,Windows
Hadoop Ecosystem HDFS and YARN
Implement complete End-to-end development Solutions for the customers and provide technical
support on daily basis. Work closely with Data Analyst team for reports planning, design, build, test and deployment
activities for Financial and Revenue Reports using Data Studio. Responsible for interacting with internal groups and customers to coordinate the timely
implementation of job modules from different source and data services. Improved the job execution time from 18 hours to 4 hours with API processing through parallel and
batch processing.
The AI-first Recruitment Operating System that unifies and automates the entire hiring lifecycle for modern enterprises worldwide.
Dr Job FZ LLC. 2026 © All Rights Reserved
