Senior Data Engineer

Work location: Bellevue WA, Atlanta GA, Dallas TX or Overland Park KS
Job Summary:
Should have extensive experience in building robust ETL pipelines, implementing efficient data ingestion strategies, and working with tools like Python, Spark, and Databricks.
Key Responsibilities:
- Design, develop, and maintain scalable ETL pipelines to ingest and transform data from multiple sources.
- Implement data integration strategies using Python, Spark, and Databricks to enable seamless data ingestion and transformation.
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and translate them into technical solutions.
- Develop and optimize data workflows to improve data quality, consistency, and performance.
- Ensure data security, governance, and compliance best practices are followed.
- Conduct performance tuning of Spark jobs and optimize data queries to minimize latency and improve efficiency.
- Automate data pipelines using CI/CD tools and ensure continuous data delivery.
- Troubleshoot data issues, identify root causes, and implement effective solutions.
Required Qualifications:
- Strong proficiency in Python, with experience in data manipulation libraries like Pandas and PySpark.
- Extensive experience with Apache Spark for data processing and analysis.
- Hands-on experience with Databricks for building scalable data pipelines.
- Experience with data lakes, data warehouses, and cloud platforms like AWS, Azure, or Google Cloud Platform.
- Strong SQL skills for data extraction, transformation, and querying.
- Familiarity with version control systems (e.g., Git) and CI/CD processes.
Preferred Skills:
- Experience with workflow orchestration tools like Apache Airflow, Dagster, or Prefect.
- Knowledge of data modeling techniques and best practices.
- Familiarity with data visualization tools such as Power BI, Tableau.
- Understanding of data privacy regulations and security protocols."