What are the responsibilities and job description for the Data Engineer position at Techvy Corp?
W2 Opportunity, No H1B's, Hybrid Position - 3 Days/week to Office, Phoenix/Arizona
Job Description
We are seeking a skilled Data Engineer with 6 years of experience to join our team. The ideal candidate will have experience in building and optimizing data pipelines, ensuring data quality, and implementing robust testing frameworks. This role requires a combination of technical proficiency in data engineering and a keen focus on validating data integrity and performance.
Key Responsibilities
- 6 years of experience in IT
- Design, build, and maintain scalable data pipelines and ETL processes.
- Work with structured and unstructured data from various sources.
- Optimize data workflows for performance and scalability.
- Collaborate with stakeholders to gather requirements and design data solutions.
- Develop and implement automated data validation and testing frameworks.
- Ensure data accuracy, integrity, and consistency across all pipelines.
- Perform performance testing on ETL processes and data pipelines.
- Identify and resolve data quality issues proactively.
- Partner with data analysts, scientists, and stakeholders to understand business requirements.
- Document data engineering workflows, testing protocols, and best practices.
- Provide support for data-related technical issues.
Skills & Qualifications
Technical Skills:
- Proficiency in SQL for data querying and manipulation.
- Experience with data engineering tools like Apache Spark, Kafka, or Airflow.
- Programming skills in Python, Java, or Scala for building data pipelines.
- Familiarity with cloud platforms like AWS, Azure, or Google Cloud.
- Experience with testing frameworks for data pipelines (e.g., Pytest, Great Expectations).
Testing Expertise:
- Hands-on experience with test automation for data workflows.
- Strong understanding of data validation and data quality metrics.
- Proficiency in performance and load testing for data systems.
Preferred Qualifications
- Familiarity with CI/CD pipelines for data workflows.
- Experience with BI tools like Tableau, Power BI, or Looker.
- Knowledge of big data technologies such as Hadoop or Databricks.
- Certification in cloud platforms (e.g., AWS Certified Data Analytics, Google Cloud Professional Data Engineer).
