What are the responsibilities and job description for the Data Engineer position at TechnoGen?
Data Engineer
Remote
Snowflake Expertise :
Design, develop, and optimize data pipelines within the Snowflake data platform. Implement best practices for Snowflake data modeling, warehouse architecture, security and performance optimization.
Proficient in using Snowflake Snowpipe streaming for real-time data streaming and ingestion and different batch load ingestion.
AWS Proficiency :
Utilize AWS services for data integration, storage, and processing. Hands-on experience with AWS Data Migration Service (DMS) for seamless and secure data migration.
Implement and manage streaming data solutions using AWS Kinesis. Design and develop ETL processes using AWS Glue, MSK for efficient data transformation.
Airflow Workflow Orchestration :
Implement and maintain data workflows using Apache Airflow. Ensure the reliability and scalability of data workflows for efficient data processing. Data Aggregation and Transformation :
Develop and implement data aggregation strategies for large-scale datasets. Collaborate with data scientists and analysts to understand data requirements and ensure data quality.
Architecture Design :
Collaborate with cross-functional teams to design and implement scalable and reliable data architecture. Ensure adherence to best practices for data security, privacy, and compliance.
Qualifications :
- o Minimum - 7 to 10 years of Data Engineering experience
o Strong Experience with Snowflake, AWS and ETL
o Hands-on experience with snowflake security, roles, encryption, masking, replication, snowpipe streaming.
o Hands-on experience with AWS Data Migration Service (DMS), Apache AirFlow, MSK, Kinesis Firehose for seamless and secure data migration.
o Hands-on experience with RESTapi, java script, Python, Scala, SQL and Java.
o Computer skills - a thorough knowledge of computer programming, coding, and various operating and database systems is a must
o Time management - developers should have the ability to quickly develop data warehousing systems and solve any issues to ensure the continued accuracy of business data
o Creativity - the ability to create mappings from scratch often calls for strong creative skills on the part of a developer
o Analytical thinking -Developers should be able to analyze data needs and options and understand the needs of various clients
o Troubleshooting - when data warehousing systems are down, Data engineer should be quickly assess the problem and provide a solution
o Team collaboration - Data engineer rarely work alone; they typically interact closely with database managers and other IT specialists when maintaining, storing, and retrieving da
