Data Engineer

1. Design and develop real-time data processing applications using Apache Spark Streaming, leveraging its capabilities to process large volumes of streaming data efficiently.

2. Collaborate with data architects, data engineers, and business stakeholders to understand requirements and translate them into scalable and maintainable Spark Streaming applications.

3. Integrate Apache Spark Streaming with various data sources and sinks, such as Kafka, HDFS, Amazon S3, and relational databases, to enable seamless data ingestion and processing.

4. Optimize Spark Streaming applications by fine-tuning configurations, partitioning strategies, and performance tuning techniques to ensure high performance and scalability.

5. Implement complex event processing, windowing, and stateful operations on streaming data to support real-time analytics and decision-making.

6. Develop and maintain data processing pipelines that combine batch and streaming data processing using the Apache Spark ecosystem.

7. Ensure data quality, data governance, and data security best practices are followed within Spark Streaming applications.

8. Collaborate with data scientists and analysts to integrate machine learning models and algorithms into Spark Streaming applications for real-time analytics and predictions.

9. Monitor, troubleshoot, and resolve issues with Spark Streaming applications, providing root cause analysis and recommendations for improvement.

10. Create and maintain documentation, including technical designs, user guides, and troubleshooting guides for Spark Streaming applications.

Requirements

1. Bachelors degree in computer science, information systems, or a related field.

2. 3+ years of experience in big data engineering, with a focus on real-time data processing using Apache Spark Streaming.

3. Strong understanding of the Apache Spark ecosystem, including Spark Core, Spark SQL, and related technologies.

4. Proficiency in programming languages such as Scala, Java, or Python.

5. Experience with distributed messaging systems, such as Apache Kafka or Amazon Kinesis.

6. Familiarity with big data storage systems, such as HDFS, Amazon S3, and NoSQL databases.

7. Excellent problem-solving and analytical skills.

8. Knowledge in Oracle and SQL

9. Strong communication and collaboration skills, with the ability to work effectively with cross-functional teams.