What are the responsibilities and job description for the Senior Data Engineer position at SSV Technologies Inc?
Job Details
Senior Data Engineer
We are looking for a highly skilled Senior Data Engineer to design, develop, and optimize streaming data pipelines and cloud-based solutions. The ideal candidate will have extensive experience with Apache Kafka, AWS Kinesis, Spark Streaming, Databricks, and cloud environments, enabling seamless real-time data processing and analytics.
Key Responsibilities:
Design, develop, and maintain streaming data pipelines using Apache Kafka, AWS Kinesis, Spark Streaming, and Databricks (Scala/Python, SQL) in a cloud environment.
Implement orchestration workflows using Apache Airflow for managing end-to-end data processing.
Deploy Apache Spark workloads on Kubernetes (K8s) or AWS ECS, optimizing for scalability and cost efficiency.
Work with Azure cloud services such as Azure Data Factory (ADF), Synapse Analytics, Fabric, Event Hubs, and Stream Analytics, as well as AWS services like Kinesis, Firehose, and ECS to manage large datasets.
Optimize data storage strategies using Azure Data Lake, Delta Lake, JSON, Parquet, SQL, and NoSQL stores for efficient data management.
Implement real-time data transformations and structured streaming for low-latency applications.
Develop dashboards and monitoring solutions using Sumologic, AWS QuickSight, Databricks Dashboards, Azure Power BI, Azure Monitor, and Log Analytics for enhanced data visualization and tracking.
Required Skills & Qualifications:
7 years of experience in data engineering or a related field.
Strong proficiency in SQL, Python, and Scala for data processing.
Expertise in streaming data technologies like Kafka, Spark Streaming, and Kinesis.
Hands-on experience with cloud platforms (Azure, AWS) and their data services.
Knowledge of containerization and orchestration (Kubernetes, AWS ECS).
Deep understanding of ETL workflows and data pipeline automation.
Experience with real-time analytics and structured streaming.
Preferred Qualifications:
Strong problem-solving and communication skills.
Ability to work collaboratively across teams to support business intelligence initiatives.
