What are the responsibilities and job description for the Data Engineer position at Sigmaways Inc?
Seeking Data Engineer to design and develop data pipelines to automate repetitive tasks within data science and data engineering and work across cross functional teams to solve complex problems.
Responsibilities :
- Oversee the integration and management of data from various sources and storage systems, establishing processes and pipelines to produce cohesive datasets for analysis and modeling.
- Partner with software engineering teams to deploy and validate production artifacts.
- Identify patterns and innovative solutions in existing spaces, consistently seeking opportunities to simplify, automate tasks, and build reusable components for multiple use cases and teams.
- Create data products that are well-modeled, thoroughly documented, and easy to understand and maintain.
- Comfortable leading projects in environments with undefined or loose requirements.
Qualifications :
- Bachelor’s degree in computer science, Information Systems, or another related field
- 8 years of experience in Data Engineering and implementing multiple end-to-end DW projects in a Big Data environment.
- Experience in building data pipelines through Spark with Java/Scala on Hadoop or Object storage.
- Experience working with Databases like Oracle, and Netezza and have strong SQL knowledge.
- Strong expertise and hand on experience in Databricks and deep understanding of its architecture.
- Experience of building data pipelines through Spark with Java/Scala on Hadoop or Object storage.
- Experience of working with Databases like Oracle, Netezza and have strong SQL knowledge
- Experience working in Unix environment.
- Experience in Reltio preferred.
- Experience of working in Agile teams.
- Strong analytical skills required for debugging production issues, providing root cause and implementing mitigation plan.
- Strong communication skills - both verbal and written – and strong relationship, collaboration skills and organizational skills.
- Ability to multi-task across multiple projects, interface with external / internal resources and provide technical leadership to junior team members.
- Ability to be high-energy, detail-oriented, proactive and able to function under pressure in an independent environment along with a high degree of initiative and self-motivation to drive results.
