Job Title: Principal Data Engineer (Azure)

Location: Boston, MA - Hybrid 1 - 2 days per week on-site

Duration: Full-time Direct Hire

*MUST be authorized to work independently on W2. NO sponsorship available.

Job Scope:

As a key member of the data engineering team, you will play a crucial role in designing, developing, and maintaining the data infrastructure and platform integration solutions. The Principal Data Engineer will be responsible for advancing the data engineering capabilities, ensuring data quality, and contributing to the success of data-driven initiatives.

Job Duties:

• Design end-to-end data solutions on the Azure data platform, considering Scalability, Security, Compliance and Performance.
• Contribute to technical foundation of the Data Lake platform, expanding and optimizing the data ecosystem and pipeline architecture.
• Provide expertise in selecting and implementing appropriate data platform services (e.g. Azure, Databricks, Snowflake) for various data processing and storage needs.
• Collaborate with cross-functional teams to integrate data solutions with existing data platform systems and applications, integrate new data management technologies aligned with the latest vendor products & capabilities (Microsoft Azure and Fabric, Databricks, Snowflake, Collibra, etc.) and software engineering tools into existing structures.
• Design, develop, construct, test, and maintain Data Lake architectures and large-scale data processing systems.
• Support big data, data lake, lakehouse ecosystem related tool selection and POC analysis.
• Gather and process raw data at scale including large complex data sets, meeting functional/non-functional business requirements (using ADF, Databricks, Python, Pyspark, scripts, REST API calls, SQL Queries, etc.).
• Develop data set processes for data modeling, mining, and production.
• Create and maintain optimal data pipeline architecture on cloud-based platforms (e.g., Azure) and relational data systems (SQL Server, SSIS, Snowflakes).
• Work on cross-functional teams delivering enterprise solutions for internal and external clients.
• Collaborate with Software Developers, Database Architects, Data Analysts, and Data Scientists on data initiatives.
• Support stakeholders, including the Management team, Product owners, and Architecture teams, in addressing data-related technical issues and fulfilling data infrastructure needs.
• Partner with integrated platform & data science teams to define integrated solutions to meet the evolving needs of data analysts, developers & consumers (e.g. data ingestion framework, AI/ML capabilities, scalable compute), contributing to the innovation and leadership of the organization.
• Identify, design, and implement internal process improvements, automation of manual processes, optimization of data delivery, etc.
• Build the data infrastructure required for optimal extraction, transformation, and loading of data from traditional/legacy sources.

Qualifications:

• Bachelor's or Master's degree in Computer Science, Information Technology, or related fields; or comparable work experience
• 8 years of related professional experience including 5 years in data lake development in large reporting environment(s)
• Expert in Azure cloud computing, specializing in Azure data engineering stacks like ADF (Azure Data Factory), ADLS, Event Hubs, Snowflake, Databricks, streaming, Azure PowerShell, and Log Analytics.
• Hands-on development experience with Design and Architecture of big data frameworks/tools: Azure Data Lake, Snowflake, Azure Data Bricks.
• Expert experience with Hadoop based technology (e.g. Spark) and programming (Python, SQL, PySpark)
• Knowledgeable about cloud computing costs and performance and capable of providing ongoing suggestions for cost & performance optimization.
• Solid understanding of Snowflake computing, including its integration with Azure Data Lake, utilizing ADLS as a source for data processing.
• Experience with Design and Architecture of relational SQL and NoSQL databases, including MS SQL Server, Snowflake.
• Experience with Azure DevOps, familiar with CI/CD (Continuous Integration/Continuous Deployment) processes and capable of scaling them across engineering teams, leveraging Azure native resources for data ingestion.
• Experience leading and working with cross-functional teams in a dynamic environment, demonstrated track record of team leadership, technical acumen, innovation, tactical and strategic.
• Proven verbal, communication, and presentation skills, ability to clearly and concisely communicate complex technical concepts to both technical and non-technical audiences.
• Proven ability to work independently.
• Advanced hands-on SQL, Spark, Python, PySpark knowledge and experience working with relational databases for data querying and retrieval on multiple platform.
• Proficiency in Data Modeling tools (e.g. Erwin, Visio).