What are the responsibilities and job description for the Principal Data Engineer in Fort Mill, SC or Austin, TX (Hybrid) position at Work For Pro Inc?
Job Details
Job Role: Principal Data Engineer
Location: Fort Mill, SC or Austin, TX (Hybrid)
Duration: Full-time role
Job Description:
We are looking for a skilled Data Engineer to join our team and help build robust, scalable, and efficient data pipelines. The ideal candidate will have strong expertise in AWS, Python, Spark, ETL Pipelines, SQL, and Pytest. This role involves designing, implementing, and optimizing data pipelines to support analytics, business intelligence, and machine learning initiatives.
Key Responsibilities:
- Design, develop, and maintain ETL pipelinesusing AWS services, Python, and Spark.
- Optimize data ingestion, transformation, and storage processes for high-performance data processing.
- Work with structured and unstructured data, ensuring data integrity, quality, and governance.
- Develop SQL queriesto extract and manipulate data efficiently from relational databases.
- Implement data validation and testingframeworks using Pytest to ensure data accuracy and reliability.
- Collaborate with data scientists, analysts, and software engineers to build scalable data solutions.
- Monitor and troubleshoot data pipelines to ensure smooth operation and minimal downtime.
- Stay up-to-date with industry trends, tools, and best practices for data engineering and cloud technologies.
Required Skills & Qualifications:
- Experiencein Data Engineering or a related field.
- Strong proficiency in AWS (S3, Glue, Lambda, EMR, Redshift, etc.)for cloud-based data processing.
- Hands-on experience with Pythonfor data processing and automation.
- Expertise in Apache Sparkfor distributed data processing.
- Solid understanding of ETL pipeline designand data warehousing concepts.
- Proficiency in SQLfor querying and managing relational databases.
- Experience writing unit and integration testsusing Pytest.
- Familiarity with CI/CD pipelinesand version control systems (e.g., Git).
- Strong problem-solving skills and ability to work in a fast-paced environment.
Experience in Snowflake.
Preferred Qualifications:
- Experience with Terraform, Docker, or Kubernetes.
- Knowledge of big data toolssuch as Apache Kafka or Airflow.
- Exposure to data governance and security best practices.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
