What are the responsibilities and job description for the Senior Data Engineer position at AvidMindz Inc.?
Job Details
Key Responsibilities:
Design, develop, and maintain ETL pipelines using AWS services, Python, and Spark.
Optimize data ingestion, transformation, and storage processes for high-performance data processing.
Work with structured and unstructured data, ensuring data integrity, quality, and governance.
Develop SQL queries to extract and manipulate data efficiently from relational databases.
Implement data validation and testing frameworks using Pytest to ensure data accuracy and reliability.
Collaborate with data scientists, analysts, and software engineers to build scalable data solutions.
Monitor and troubleshoot data pipelines to ensure smooth operation and minimal downtime.
Stay up-to-date with industry trends, tools, and best practices for data engineering and cloud technologies.
Required Skills & Qualifications:
Experience in Data Engineering or a related field.
Strong proficiency in AWS (S3, Glue, Lambda, EMR, Redshift, etc.) for cloud-based data processing.
Hands-on experience with Python for data processing and automation.
Expertise in Apache Spark for distributed data processing.
Solid understanding of ETL pipeline design and data warehousing concepts.
Proficiency in SQL for querying and managing relational databases.
Experience writing unit and integration tests using Pytest.
Familiarity with CI/CD pipelines and version control systems (e.g., Git).
Strong problem-solving skills and ability to work in a fast-paced environment.
Experience in Snowflake.