What are the responsibilities and job description for the Lead Data Engineer (Azure) position at SPECTRAFORCE?
Job Title: Azure Data Engineer
Location: Glendale, AZ
Duration: 12 months
Job Summary:
We are seeking an experienced and highly skilled Software Engineering Lead to spearhead our data initiatives, with a primary focus on Azure Data Lake and its associated ecosystem. The ideal candidate will possess deep architectural knowledge of Azure Data Lake and Databricks, a solid understanding of security concepts within Azure, and a proven track record of leading teams in delivering scalable and secure data solutions.
US citizenship is required for this position.
Major Responsibilities:
Architectural Design and Implementation:
- Design and implement robust, scalable, and efficient data architectures leveraging Azure Data Lake and Databricks.
- Define and enforce best practices for data ingestion, storage, processing, and retrieval.
- Optimize data workflows to ensure high performance and cost efficiency.
Data Governance and Security:
- Develop and implement security measures for Azure Data Lake, ensuring compliance with organizational and regulatory standards.
- Manage role-based access control (RBAC), encryption, and other security protocols within Azure Subscriptions.
- Collaborate with security teams to perform regular audits and vulnerability assessments.
Team Leadership and Collaboration:
- Lead and mentor a team of data engineers, providing technical guidance and fostering professional development.
- Collaborate with cross-functional teams, including data scientists, business analysts, and IT teams, to deliver data-driven solutions.
- Drive agile practices and ensure timely delivery of projects.
Platform Optimization and Monitoring:
- Oversee the deployment and management of Azure Data Lake and Databricks environments.
- Implement monitoring and alerting systems to ensure system reliability and performance.
- Evaluate and incorporate new Azure services and technologies to enhance the data platform.
Strategic Planning and Roadmap Development:
- Develop and execute a roadmap for data engineering aligned with business objectives.
- Stay abreast of industry trends and advancements in data engineering and Azure technologies.
- Provide recommendations for long-term data strategy, including data lakehouse adoption and cloud optimization.
Education and Experience Requirements:
- Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
- 8 years of experience in data engineering, with at least 3 years in a leadership role.
- Extensive hands-on experience with Azure Data Lake, Databricks, and other Azure services.
- Proven expertise in architecting and implementing large-scale data solutions.
Required Knowledge and Skills:
Technical Skills:
- Proficiency in SQL.
- Deep understanding of Azure security concepts, including subscription management, RBAC, and data encryption.
- Experience with data modeling, ETL pipelines, and big data technologies.
- Familiarity with CI/CD pipelines and DevOps practices in a data engineering context.
Soft Skills:
- Strong leadership and team management abilities.
- Excellent communication and stakeholder management skills.
- Analytical mindset with a proactive approach to problem-solving.
Preferred Qualifications:
- Azure certifications, such as Azure Data Engineer Associate or Azure Solutions Architect Expert.
- Experience with implementing data lakehouse architectures.
- Familiarity with data governance frameworks like GDPR, CCPA, or HIPAA.
