Senior Data Engineer

Description

Basic Function

The Senior Data Engineer should be an expert familiar with all areas of data warehousing technical components (e.g., ETL, Reporting, Data Model), connected infrastructure, and their integrations. The ideal candidate will be responsible for developing the overall architecture and high-level design of the data schema environment. The candidate must have extensive experience with Star Schemas, Dimensional Models, and Data Marts. The individual is expected to build efficient, flexible, extensible, and scalable ETL design and mappings. Excellent written and verbal communication skills are required as the candidate will work very closely with diverse teams. A wide degree of creativity and latitude is expected. This position reports to the Manager of Data Services.

Typical Requirements

Requires strong technical and analytical skills, data management expertise, and business acumen to achieve results. The ideal candidate should be able to deep dive into data, perform advanced analysis, discover root causes, and design scalable long-term solutions using Databricks, Spark, and related technologies to address business questions. A strong understanding of business data needs and alignment with strategic goals will significantly enhance effectiveness. The role requires the ability to prepare high-level architectural frameworks for data services and present them to business leadership. Additionally, the candidate must work well in a collaborative environment while performing a variety of detailed tasks daily. Strong oral and written communication skills are essential, along with expertise in application design and a deep understanding of distributed computing, data lake architecture, and relational database concepts. This position requires the ability to leverage both business and technical capabilities regularly.

Essential Functions

1. Gather, structure, and process data from various sources (e.g., transactional systems, third-party applications, cloud-based  financial systems, customer feedback, etc.) using Databricks and Apache Spark to enhance business insights.

2. Develop and enforce standards, procedures, and quality control measures for data analytics in compliance with enterprise policies and best practices.

3. Partner with business stakeholders to build scalable data models and infrastructure, leveraging Databricks' Delta Lake, MLflow, and Unity Catalog.

4. Identify, analyze, and interpret complex data sets to develop insightful analytics and predictive models.

5. Utilize Databricks to design and optimize data processing pipelines for large-scale data ingestion, transformation, and storage.

6. Ensure data infrastructure completeness and compatibility to support system performance, availability, and reliability requirements.

7. Architect and implement robust data pipelines using PySpark, SQL, and Databricks Workflows for automation.

8. Provide input on technical challenges and recommend best practices for data engineering solutions within Databricks.

9. Design and optimize data models for analytical and operational use cases.

10. Develop and implement monitoring, alerting, and logging frameworks for data pipelines.

11. Lead the architecture and implementation of next-generation cloud-based data solutions.

12. Build scalable and reliable data integration pipelines using Databricks, SQL, Python, and Spark.

13. Mentor and develop junior team members, fostering a data-driven culture within the organization.

14. Develop high-quality, scalable data solutions to support business intelligence, analytics, and data science initiatives.

15. Interface with technology teams to extract, transform, and load (ETL) data from diverse data sources into Databricks.

16. Continuously improve data processes, automating and simplifying workflows for self-service analytics.

17. Work with large, complex data sets to solve non-routine analysis problems, applying advanced machine learning and data processing techniques as needed.

18. Prototype, iterate, and scale data analysis pipelines, advocating for improvements in Databricks data structures and governance.

19. Collaborate cross-functionally to present findings effectively through data visualizations and executive-level presentations.

20. Research and implement advanced analytics, forecasting, and optimization methods to drive business outcomes.

21. Stay up to date with industry trends and emerging Databricks technologies to enhance data-driven capabilities.

Decision Making

Requires judgment and decision making to a wide range of tasks to be successful in establishing a data driven model of services, while working within established rules and procedures of the company using standard practices. Requires analysis of complex principles in developing approaches and techniques for problem solving.

Supervision Received/Given

Specific direction to accomplish assigned objectives is received from the Director of Data Services for both strategic direction and when needed for special or unusual situations. Plan and prioritize own work and get clarification when needed.

Accuracy, Accountability, and Control

Probable errors may cause serious delays in project completion or may have detrimental impact to the data quality and reporting of business results. Errors would usually be detected before results became serious. They may affect the work of others within the company.

Contacts Internal/External

Internal Contacts are normally made with limited or no supervision, and appropriate judgment and discretion must be exercised to explain, interpret and influence others to avoid misunderstanding. External Contacts are made with limited or no supervision, and concern matters that require explanation, discussion, and interpretation.

Specialized Skills or Technical Knowledge

• Bachelor's degree or higher in a quantitative/technical field (e.g., Computer Science, Statistics, Engineering). A Master's degree in Computer Science, Mathematics, Statistics, or Economics is preferred.
• 5 years of experience in data engineering, business intelligence, or data analytics, with a focus on Databricks and Apache Spark.
• Extensive experience with SQL and Python for developing optimized queries and data transformations.
• Expertise in Databricks ecosystem, including Delta Lake, MLflow, and Databricks SQL.
• Experience in designing and implementing ETL/ELT pipelines on Databricks using Spark and cloud-based data platforms (Azure Data Lake, AWS S3, or Google Cloud Storage).
• Strong data modeling, data warehousing, and data governance knowledge.
• Experience in working with structured and unstructured data, including real-time and batch processing solutions.
• Familiarity with data visualization tools such as Power BI, Tableau, or Looker.
• Deep understanding of distributed computing, scalable data architecture, and cloud computing frameworks.
• Hands-on experience with CI/CD pipelines, Infrastructure as Code (IaC), and DevOps practices in data engineering.
• Proven track record of working with cross-functional teams, stakeholders, and senior management to deliver high-impact data solutions.
• Knowledge of machine learning and AI-driven analytics is a plus.
• Strong problem-solving skills and the ability to work independently and in a team-oriented environment.
• Excellent communication skills and ability to convey complex data concepts to non-technical stakeholders.
• Experience in a franchised organization is a plus.

Confidential Information

The position has access to most all proprietary business systems and critical business data, and the utmost of confidentiality and secure governance is required. Access to information is given under appropriate approval, and absolute discretion is critical for most instances.

Working Conditions

Professional office environment, operating under normal office conditions within a well-lit, climate-controlled office. Must be able to work effectively under managing multiple deadlines and commitments. May require occasional travel to franchise locations or corporate events and training sessions. Hybrid/remote work options may be available with approval.

Special Assignments

Due to the dynamic nature of our business, there are constant needs for assignments that were neither foreseen nor expected. This may be the need to evaluate or implement a new technology, gather specific relevant data, or simply to handle a challenge encountered by a franchisee.

Other Information

This position will assist in creating a new model of data services delivery to the company. This person must be a well-rounded individual and have the ability and patience to manage constant growth over the next few years while developing this team and its skillsets.

Constant education, both self-directed and classroom, is required in order to grow and keep up with changing technology, tools and business systems.

Qualifications