Position Name: Big Data Engineer
Responsibilities:
- Build and maintain large-scale batch and real-time data pipelines with data processing frameworks and technologies like Spark, Hive, Flink, Spark-Streaming, Hadoop, etc.
- Lead and coordinate the engineering teams (or Project teams) to ensure the right architecture decisions are made, and the right development and operation processes are taken
- Responsible for technical decision of implementation (architecture and development process), and also the production system quality (operation and maintenance)
- Lead technology decisions to correctly place solutions for both functional and non-functional requirements, and to make the right decisions at the right timing for all the scaling, security, maintenance ability, etc.
- Provide estimates, risk info, and recommendations to PMs for creating the system roadmap and to support prioritization conversations.
Qualifications:
- Experience in large-scale data processing experience using Big Data tools such as Hadoop, Spark, Kafka, Hive, etc.
- Experience in making both functional and non-functional requirements, design, building and operating Data pipelines
- More than 5 years’ software development experience using the following languages (Java, Python, Go, Bash, Perl, etc.)
- More than 5 years’ software development experience (and consistent development experience in requirements definition, specification formulation, implementation, testing, release, and maintenance operation)
- Experience in database design and performance tuning.
Preferred Qualifications:
- Experience in operation for online advertising (Ad product)
- Experience in middleware selection and integration
- Experience in scrum development
Holidays/Benefits
- Five-Day Workweek
- Summer Holidays
- Winter Holidays
- Paid Holidays
- Benefits (pension, health, employment, unemployment)
- Commuting allowance
- Employee stock ownership plan
- Discounted sports club
- Childcare
- Cafeteria (free breakfast, lunch, and dinner), Library, Convenience store
- Internal clinic, pharmacy, fitness club & spa, hair salon, laundry service
- Assistance fund for club activities
- Baseball tickets
About Hiring Company
The largest eCommerce company in Japan, and third largest eCommerce marketplace company worldwide.
This organization provides a variety of consumer and business-focused services including e-commerce, e-reading, travel, banking, securities, credit card, e-money, portal and media, online marketing, and professional sports.
The company is expanding globally and currently has operations throughout Asia, Western Europe, and the Americas.
[Measures against passive smoking]
No smoking indoors allowed
Designated smoking area