What are the responsibilities and job description for the Data Engineer position at It solutions Asia pvt.LTD?
Job Title: Senior Data Engineer
Location: Initial Remote ( Bentonville, AR when requested )
Duration : Long Term
Experience Required: 6-8 Years
About the Role:
We are seeking an experienced Senior Data Engineer to join our team. The ideal candidate will have extensive experience in designing, developing, and maintaining data processing systems, with expertise in large-scale distributed systems and cloud technologies. You will play a key role in building data pipelines, managing large datasets, and enabling real-time data processing across platforms.
Key Responsibilities:
- Design and Build Data Pipelines: Create scalable, reliable, and efficient ETL/ELT pipelines using Hadoop, Spark, and streaming technologies (Kafka, Spark Streaming). - REQUIRED
- Cloud Data Engineering: Implement and optimize data infrastructure in the cloud (AWS, Azure, or GCP) to ensure high performance, reliability, and security.
- Data Warehousing: Develop and maintain robust data warehousing solutions using SQL-based and NoSQL databases. - REQUIRED
- Real-time Data Streaming: Architect and manage real-time data streaming platforms with tools like Kafka and Spark Streaming for continuous data processing and analytics.
- CI/CD Pipelines: Design and implement CI/CD processes for automated testing, integration, and deployment of data engineering solutions.
- Data Integration: Integrate various data sources (structured and unstructured) into the data platform while ensuring data quality and governance.
- Collaborate Across Teams: Work closely with data scientists, data analysts, and software engineers to understand business requirements and translate them into data engineering solutions.
- Performance Tuning: Monitor and optimize data processing workflows to achieve optimal performance and scalability.
Qualifications:
- Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
- 6-8 years of hands-on experience in data engineering, big data technologies, and cloud platforms.
- Strong problem-solving skills with the ability to design and optimize complex data workflows.
- Excellent communication skills and the ability to work in a team environment.