Data Engineer

Role : Snowflake DBT Data Engineer

Location: Plano, TX (day1 onsite)

Full Time

Technical/Functional Skills : Snowflake, DBT

Roles & Responsibility :

Snowflake & DBT Infrastructure Development:

• Design and implement scalable, modular, and reusable DBT models.
• Develop incremental and snapshot models to optimize data processing and reduce compute costs
• Implement Common Table Expressions (CTEs), materialized views, and table strategies to enhance performance.
• Define and implement DBT tests.
• Develop custom DBT macros.
• Optimize SQL queries for performance and cost efficiency.
• Develop and manage ETL/ELT pipelines using Snowflake’s capabilities like Snowpipe, Streams, and Tasks.

Model Deployment:

• Ensure the smooth operation of deployed models by monitoring performance and identifying potential issues.
• Manage DBT projects using Git, ensuring proper version control and collaboration.
• Set up CI/CD pipelines to automate testing, validation, and deployment of DBT models
• Set up data sharing, masking, and encryption as needed.

Data Management:

• Maintain comprehensive DBT documentation for models, sources, and transformations.
• Ensure the quality and integrity of data used in dbt models by implementing data validation and cleaning processes.
• Implement data security measures to protect sensitive data.
• Ensure transparency in data workflows by maintaining well-documented YAML files.

Monitoring and Optimization:

• Set up alerting mechanisms for failures in DBT transformations.
• Create and monitor jobs using Control M for dbt models.
• Monitor execution times, identify performance bottlenecks, and apply optimizations.
• Ensure end-to-end data pipeline reliability, from ingestion to transformation and reporting.

Collaboration and Communication:

• Work closely with cross-functional teams, including data scientists, software engineers, and DevOps engineers.
• Develop and maintain documentation for Snowflake infrastructure and dbt workflows.
• Present findings and recommendations to stakeholders through clear and effective communication.

Regards,

Shweta Sharma

shweta.sharma@msrtechnologies.com