Lead Data Engineer - Azure Databricks

Responsibilities

Kforce is working with a client in search of a Lead Data Engineer- Azure Databricks, to join their team in NYC. This position is full time, direct hire and on-site 1-day a week. Summary: As a Senior Data Engineer within this department, you will play a critical role in designing and developing robust ETL pipelines to integrate diverse data sources. This role requires expertise in creating efficient data pipelines, handling both streaming and batch data processing, and ensuring data integrity throughout the ETL lifecycle. The Lead Data Engineer will implement monitoring solutions, optimize ETL jobs for performance, and provide comprehensive support from data ingestion to final output. With in-depth knowledge of the Databricks platform and strong analytical skills, this position will significantly enhance the department's ability to deliver high quality, data-driven insights and solutions. Responsibilities

• Strategize, design, develop and work with a team of dynamic and passionate data engineers to deliver automated cloud infrastructure and DevOps solutions
• Design and develop ETL code to integrate various data sources
• Create and maintain efficient data pipelines on the Databricks platform
• Develop, maintain, and optimize ETL pipelines for both streaming and batch data processing
• Ensure data integrity and consistency throughout the ETL lifecycle
• Provide comprehensive support for ETL processes, from data ingestion to final output
• Write and execute unit tests and integration test cases for ETL code to ensure high-quality outcomes
• Implement monitoring solutions for ETL jobs to ensure timely and successful data processing
• Proactively identify, troubleshoot, and resolve issues in ETL workflows
• Optimize ETL jobs for maximum performance and efficiency through performance tuning and troubleshooting
• Mentor other data engineers in the team, cross-train and provides guidance
  
  

Requirements

• Bachelor's degree in Computer science, Information systems, Engineering, or Data Science
• 10 years of experience in designing, developing, and optimizing ETL processes
• 5 years of experience in developing/supporting a data platform in Azure Databricks
• Proficiency in creating and maintaining efficient data pipelines on the Databricks platform
• Well-rounded in working in a DEV Ops environment supporting processes in data platform in supporting business units and their use cases
• Strong verbal and written communication skills, with the ability to explain complex technical concepts to non-technical stakeholders
• Ability to troubleshoot and resolve issues in a timely manner
• Experience working in an Agile/Scrum environment
• Team player with the ability to work collaboratively in a cross-functional team
• Strong analytical and problem-solving skills
• Ability to work independently, handle multiple tasks simultaneously and adapt quickly to change with a variety of people and work styles
• Willingness to learn new technologies and continuously improve skills
  
  

Preferred Qualifications

• In-depth knowledge of Databricks platform and technologies, including Delta Lake, Databricks SQL, and Databricks Workflows
• Experience with Azure cloud platforms and Azure Data Lake cloud storage
• Knowledge of data warehousing, data modeling, and best practices
• Proficiency in programming languages such as Python, SQL, Scala, or R
• Experience with big data technologies such as Apache Spark, Hadoop, or Kafka
• Knowledge of working with Hadoop for migration projects from Hadoop to Databricks
• Familiarity with DevOps practices and tools such as CI/CD, Git, etc.
• Knowledge of infrastructure as Code (IaC) tools like Terraform
• Experience with implementing data governance and security measures in a cloud environment
  
  

The pay range is the lowest to highest compensation we reasonably in good faith believe we would pay at posting for this role. We may ultimately pay more or less than this range. Employee pay is based on factors like relevant education, qualifications, certifications, experience, skills, seniority, location, performance, union contract and business needs. This range may be modified in the future.

We offer comprehensive benefits including medical/dental/vision insurance, HSA, FSA, 401(k), and life, disability & ADD insurance to eligible employees. Salaried personnel receive paid time off. Hourly employees are not eligible for paid time off unless required by law. Hourly employees on a Service Contract Act project are eligible for paid sick leave.

Note: Pay is not considered compensation until it is earned, vested and determinable. The amount and availability of any compensation remains in Kforce's sole discretion unless and until paid and may be modified in its discretion consistent with the law.

This job is not eligible for bonuses, incentives or commissions.

Kforce is an Equal Opportunity/Affirmative Action Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, gender identity, national origin, age, protected veteran status, or disability status.

By clicking “Apply Today” you agree to receive calls, AI-generated calls, text messages or emails from Kforce and its affiliates, and service providers. Note that if you choose to communicate with Kforce via text messaging the frequency may vary, and message and data rates may apply. Carriers are not liable for delayed or undelivered messages. You will always have the right to cease communicating via text by using key words such as STOP.