What are the responsibilities and job description for the Sr Databricks Lead Engineer position at Pyramid Technology Solutions, Inc.?
Job Details
Job Title: Sr Databricks Lead Engineer with streaming experience
Location: ONSITE at Dallas, TX/ Houston, TX/ Baltimore, MD
Duration: Contract
Databricks lead with streaming experience. Utility domain experience
- Proficiency in Databricks tools and technologies, including Delta Lake and Databricks SQL.
- Extensive ETL development experience, particularly with large-scale data processing and transformation pipelines.
- Strong knowledge of PySpark for building and optimizing distributed data processing workflows.
- Experience working with streaming data pipelines using frameworks such as Spark Structured Streaming or similar tools.
- Familiarity with Azure platform and related data services.
- Ability to troubleshoot and optimize performance in distributed environments.
- Databricks Spark Streaming for real-time data processing
- PySpark for scalable data transformation and analysis
- Delta Live Tables for automated and reliable pipeline development
- Spark SQL for data querying and analytics.
- Autoloader for incremental data ingestion
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
