What are the responsibilities and job description for the Senior Data Engineer position at Infinity Solutions?
Description :
We are seeking a skilled and motivated Data Engineer to join our team and work on innovative projects with AWS. The successful candidate will be responsible for developing and managing data infrastructure, backend logic, APIs, and DevOps processes while ensuring robust logging and quality assurance.
Key Responsibilities
Infrastructure Development
Design, implement, and maintain scalable data infrastructure to support business needs.
Optimize data pipelines and workflows for performance and reliability.
Backend Logic
Develop backend systems to process and manage large-scale datasets.
Collaborate with cross-functional teams to integrate data solutions into applications.
API Development
Build and maintain secure, high-performance APIs for data access and integration.
Ensure API functionality aligns with client and project requirements.
Logging and Monitoring
Implement robust logging systems to monitor data workflows and troubleshoot issues.
Maintain logs for auditing and analytics purposes.
DevOps
Automate deployment and operational processes to improve system efficiency.
Monitor and manage cloud infrastructure on AWS to ensure high availability and security.
Quality Assurance
Conduct rigorous testing to ensure data integrity and system reliability.
Collaborate with QA teams to address issues promptly and efficiently.
Required Skills and Qualifications
Bachelor's degree in Computer Science, Engineering, or a related field.
8-10 ] years of experience in data engineering or a similar role.
Proficiency with AWS services (e.g., S3, Redshift, Lambda, EMR).
Strong programming skills in Python, Java, or Scala.
Experience with data pipelines, ETL processes, and distributed systems.
Knowledge of API design and development (e.g., RESTful, GraphQL).
Familiarity with logging frameworks and tools like ELK Stack or CloudWatch.
Expertise in DevOps tools (e.g., Jenkins, Docker, Kubernetes).
Solid understanding of QA practices, including automated testing frameworks.
Excellent problem-solving skills and the ability to work collaboratively in a fast-paced environment.
Preferred Qualifications
Experience with big data technologies such as Spark, Hadoop, or Kafka.
Familiarity with data visualization tools and BI platforms.
AWS certifications (e.g., AWS Certified Data Analytics Specialty).
