What are the responsibilities and job description for the Sr Databricks Engineer position at Intellibee Inc?
Sr Databricks Engineer,Charlotte, NC,United States
Sr Databricks Engineer
Top 3 Skills: Data Bricks Expert, SQL, Python
Soft Skills: Comfortable with being the go-to person, Ability to collaborate with multiple teams, Ability to see big picture
What You Will Do
Sr Databricks Engineer
Top 3 Skills: Data Bricks Expert, SQL, Python
Soft Skills: Comfortable with being the go-to person, Ability to collaborate with multiple teams, Ability to see big picture
What You Will Do
- Defining and implementing scalable and flexible data analytics architecture based on current and future business needs mainly utilizing Databricks platform
- Implementing, maintaining and optimizing complex data pipelines to process large amounts of data
- Integration, consolidation, cleansing and structuring of large data sets to enable business insights, reporting and AI/ML use cases
- Delivering business insights by consolidating various data sources to understand business performance (e.g. sales data, customer satisfaction data, digital analytics data)
- Sourcing, collecting and researching new internal and open-source external data sets
- Defining and implementing data governance processes
- Plan and prepare data engineering and analysis work between technical teams
- Engage with business stakeholders to prepare and coordinate data sharing and data transfer activities
- Support deployment, monitoring and maintenance of ML/AI models
- Multiple years of experience in related roles utilizing Databricks platform and related technologies for solving data engineering challenges
- Good understanding of concepts and technologies underpinning modern data stack (storage, compute, governance, DataOps, MLOps, etc.) with focus on Databricks platform
- Experience in data modelling and designing/implementing scalable and efficient ETL/ELT pipelines on Databricks platform
- Strong experience in SQL and Python
- Understanding of AI/ML, and MLOps / DataOps experience on Databricks platform
- Experience in MLFlow or equivalent tools is highly desirable
- Understanding and experience in solving data governance challenges utilizing Unity catalog (or related technology) in the context of Databricks platform
- Experience with optimizing all aspects of Databricks workloads
- Excellent problem-solving skills and analytical mind
- Excellent verbal and written communication skills including the ability to explain technical issues, solutions and results to a non-technical audience
- Strong work ethic and willingness to share your opinion and challenge status quo
