What are the responsibilities and job description for the Databricks Data Engineer position at Interon IT Solutions LLC?
Job Details
Senior Databricks Data Engineer
Location: Rochester, NY (Preferred) / Remote/ Canada
What you will be doing:
Create and maintain data pipelines and foundational datasets to support analytics, modeling, experimentation, and product/business needs. Design and build database architectures with massive and complex data, balancing ergonomic benefits with computational load and cost. Collaborate closely with data science, machine learning, and engineering teams to improve the coverage, accuracy, and reliability of
instrumentation.
Develop audits for data quality at scale, implementing alerting and anomaly detection as necessary.
Create scalable dashboards and reports to support business objectives and enable data-driven decision making. Partner with data scientists, engineers, and product teams to accomplish all of the above!
What you should have:
3 years of experience building data pipelines in production environments with high volumes of consumer data.
3 years of experience in designing, developing, and maintaining robust data models from structured and unstructured sources.
3 years of experience writing accurate and effective code in SQL, Python, Pyspark
3 years of experience with a public cloud (AWS, Microsoft Azure, Google Cloud)
3 years of experience with NoSQL implementation (DynamoDB, CosmosDB)
Develop and maintain CI/CD pipelines for Databricks projects using tools like Azure DevOps
Experience implementing and monitoring audits for data quality with massive data sets (e.g., billions of rows).
Experience proactively identifying opportunities to improve ETL & dashboard performance and cost.
Experience leveraging your excellent communication and stakeholder management skills to thrive in ambiguous environments where problems are not well-defined and evolve quickly.
Experience collaborating directly with data science, machine learning, and product engineering teams.
A desire to work with amazing, passionate people who care deeply about solving challenging problems.
Last but not least - a collaborative attitude and a healthy dose of natural curiosity
Bonus Points:
Experience owning and proactively improving the data models for a functional area leveraging deep knowledge of performant scalable patterns. Experience with modern data storage and processing technologies (i.e., Airflow, and DBT or similar). Experience with data visualization and dashboarding technologies (i.e., PowerBI, Tableau, or similar). Experience with designing data architecture to power a variety of use cases, including experimentation.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
