What are the responsibilities and job description for the Data Engineer position at DRUM Associates?
Data Engineer
New York, NY
Description:
We are seeking a skilled Data Engineer or Jr. Data Engineer to join our team in Manhattan. As a Data Engineer, you will play a critical role in building and maintaining robust data pipelines, transforming raw data into actionable insights, and supporting our data-driven decision-making processes. You will work with Python, SQL, ETL, Agile methodologies, and DevOps practices to build scalable and efficient data solutions. This is an excellent opportunity for both experienced Data Engineers and those looking to grow in the field. Training will be provided on the latest industry trends and modern data strategies the company is implementing, ensuring you stay at the forefront of data engineering practices.
Responsibilities:
- Design, develop, and maintain scalable and efficient data pipelines using Python and SQL.
- Implement and optimize ETL (Extract, Transform, Load) processes for integrating data from multiple sources.
- Work closely with cross-functional teams to gather requirements, design data architectures, and ensure data accuracy.
- Monitor, troubleshoot, and optimize data pipelines for performance and reliability.
- Assist in maintaining and supporting data infrastructure, ensuring scalability and availability.
- Work in an Agile environment, participating in sprint planning, stand-ups, and retrospectives.
- Implement best practices for DevOps, ensuring seamless integration and deployment of data workflows.
- Collaborate with data scientists, analysts, and business stakeholders to ensure data solutions meet business needs.
- Continuously improve and innovate data engineering processes and technologies.
Qualifications:
- Bachelor’s degree in Information Technology, Computer Science, Data Science, or a related field.
- 2 years of hands-on experience in data engineering, with proficiency in Python, SQL, and ETL processes.
- Experience with database systems like MySQL, PostgreSQL, or similar relational databases.
- Strong understanding of ETL frameworks and data integration.
- Familiarity with cloud-based data platforms (e.g., AWS, Azure, Google Cloud) is a plus.
- Experience with Agile methodologies and DevOps tools (e.g., Jenkins, Docker, Kubernetes) is a plus.
- Familiarity with data warehousing concepts and technologies is beneficial.
- Strong problem-solving, analytical, and troubleshooting skills.
- Excellent communication and collaboration skills in a team-oriented environment.
Preferred Skills (for Data Engineer):
- Experience with big data technologies (e.g., Hadoop, Spark, Kafka).
- Knowledge of data pipeline orchestration tools (e.g., Apache Airflow, Luigi).
- Understanding of data security and governance practices.
Training and Development:
- Ongoing training will be provided to keep you updated on industry trends and emerging technologies.
- Training on modern technologies and tools the company is implementing to enhance your skills and career growth.
