Lead Azure Data Engineer

Position: Lead Azure Data Engineer

Location: Atlanta, GA / Encino, CA (Hybrid)

Experience: 10 Years

Type: Full time

Must Have: Proficiency in DataVault, Data warehousing, Azure Cloud, Knowledge on Microsoft Fabric, Databricks or similar platforms

Responsibilities:

• Work in a highly collaborative team environment following the Agile methodology to assist other department personnel in the successful accomplishment of strategic and divisional objectives. Take the necessary steps to ensure our customers' needs are met to the maximum extent possible in an accurate and timely manner. Perform hands-on development and collaborate in the data management efforts including operational data to provide a high level of data integrity, security, and availability for internal and external customers.
• Establish and maintain a scalable, extensible, and maintainable architecture for the enterprise data management and analytics system. Summarize enterprise data into an intuitive analysis framework for understanding the current and future performance of the company. Provide the design, development, testing and maintenance of Data ingestion, Curation, and Dissemination processes, Data Engineering activities, Reporting and Analytics dashboards as per the Data Strategy roadmap.
• Collaborate with other business unit leaders for various projects involving enterprise data. Ensure the appropriate capture and retention of enterprise data. Work with Business Analysts to translate various business needs into EDM canonical data mapping, analytics, reporting, and dash-boarding requirements. Educate other business units and Information Technology teams about the analytics and reporting advantages available to them through Business Intelligence.
• Provide expertise to achieve system integration through database design, including data and dimensional modeling, logical and physical table design, complex queries, stored procedures and triggers, data transformation, aggregation, and enterprise application integration.
• Contribute towards the definition of data strategy, implementation of Enterprise Data management and Analytics standards and best practices. Mentor, support, and provide direction to lesser experienced teammates, including Offshore data engineers.
• Identify & deliver the analytical needs of business units by researching and analyzing data from various sources and incorporate data into existing processes and systems as needed by using expert level understanding of how data, technology, and data work together to serve our customers.
• Conduct analysis on datasets, including efficient extraction, transformation, and analysis of complex datasets. Analyze data and information in innovative ways to assist with strategic decisions and optimize process efficiencies. Ensure all necessary data elements are available for model building and strategy analysis.
• Research emerging technologies, trends, and benchmark data to make recommendations to improve customer experience, architecture, processes, and tools.

Knowledge, Skills, Abilities:

• Must have working experience with Raw and Business Data Vault Modeling and Engineering as well as building marts using Kimball based methodologies. Must have built data vault models from data requirements . Must understand how to build data quality into data pipelines for data vault and data integration pipelines
• Must have working experience with Azure based Data Pipelining , Scheduling and Monitoring and pyspark with ability to debug troublesome pipelines . Must have hands on expertise dealing with data pipelines
• Strong working experience with Big Data technologies (Spark, Data Bricks) for Data integration, & processing (ingestion, transformation, curation, etc), preferably on Azure cloud, and a clear understanding of how the resources work and integrate with cloud and on-prem.
• High level of proficiency with database and data warehouse development, including replication, staging, ETL, stored procedures, partitioning, change data capture, triggers, scheduling tools, cubes, and datamarts.
• Experience working with backend languages such as Python.
• Strong computer literacy and proficiency in data manipulation using Analytics tools/platform like Databricks, Azure Fabric using Spark engine and reporting tools like Power BI.
• Strong analytical, data mining, data modeling, and data management skills, Leadership, and judgment to analyze, evaluate, and develop solutions to complex problems.
• Exposure to Azure Fabric is a strong plus along with Automation tools for Data Vault
• Creative and critical thinking ability; excellent troubleshooting and problem-solving skills
• Self-motivated and driven to produce quality products.
• Ability to work effectively in a collaborative team environment and effectively demonstrate team building attitude and skills
• Demonstrated ability to be flexible and perform a variety of roles such as analyst, tester, and mentor

Other Desirable Knowledge:

• Experience with AI/ML models/algorithms.
• Experience providing sound technical advice to leadership and other staff in the subject field(s) related to this position.
• Excellent written, verbal and presentation skills
• Experience with a variety of industry standards regarding Microsoft database development concepts, best practices, and procedures
• Experience with financial services industry is a strong plus
• Microsoft MCDBA Certified, CBIP or cloud data warehouse certification on Azure Cloud is a strong plus.

Education & Experience:

• Bachelor’s degree in computer science or engineering, preferably with Data Science specialty, or equivalent training and experience
• 5 years of experience in Data engineering discipline implementing data lake, data warehouse, and Reporting & Analytics development (preferably on Azure) using Tools like Power BI or similar tool.
• Should have experience in architecting large-scale cloud data warehouses.
• 10 years of experience in advanced SQL scripting, the related tools and technologies.