What are the responsibilities and job description for the Lead Data Engineer with Pyspark-Addison, TX- Only locals position at mProgen?
Job Details
Onsite Addison TX.. min 3 days/week
Job Summary
We are seeking an experienced Tech lead with 11 to 15 years of experience to join our team. The ideal candidate will have a strong background in Python, Databricks SQL, Databricks Workflows, and PySpark. This role involves designing and implementing scalable data solutions in a hybrid work model with day shifts. The candidate will play a crucial role in driving our data strategy and ensuring the successful execution of data projects.
Responsibilities
Lead the design and implementation of scalable data solutions using Python, Databricks SQL, Databricks Workflows, and PySpark.
Oversee the development and maintenance of data pipelines to ensure data integrity and availability.
Provide technical guidance and mentorship to junior team members to foster their growth and development.
Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
Ensure the performance, scalability, and reliability of data systems through continuous monitoring and optimization.
Develop and enforce best practices for data management, including data security and compliance.
Conduct code reviews to maintain high-quality code standards and ensure adherence to best practices.
Participate in the evaluation and selection of new tools and technologies to enhance the data architecture.
Create and maintain comprehensive documentation for data systems, processes, and workflows.
Drive the adoption of data-driven decision-making across the organization by providing insights and recommendations.
Facilitate effective communication between technical and non-technical stakeholders to ensure alignment on project goals.
Manage project timelines and deliverables to ensure timely completion of data initiatives.
Stay updated with the latest industry trends and advancements in data technologies to continuously improve our data capabilities.
Certifications Required
good to have - Certified Data Architect, Databricks Certified Associate Developer for Apache Spark, Python Institute PCEP
