What are the responsibilities and job description for the Senior Data Engineer position at Thinking Minds Inc.?
Job Details
Job Summary
Location: South Texas
Experience: 8 years in data engineering
Job Type: Permanent
Travel: Up to 20% (domestic)
Ideal Candidate Should Have:
Strong expertise in designing and managing large-scale data pipelines and architectures.
Hands-on experience with Apache Spark, Hadoop, and Kafka for big data processing.
Proficiency in AWS (preferred) or Azure cloud environments, including storage, compute, and data processing services.
Experience with ETL/ELT workflows and tools such as Apache Airflow or Talend.
Knowledge of Databricks, Dataiku, and AWS-native data solutions.
Proficient in SQL, Python, Java, or Scala for data engineering tasks.
Familiarity with Infrastructure-as-Code (Terraform) for cloud deployments.
Understanding of machine learning model deployment is a plus.
Strong communication skills to collaborate across technical and business teams.
Preferred certifications: AWS Certified Solution Architect, Databricks Certified Associate Developer for Apache Spark.
Job Description
Thinking Minds Inc. is seeking a Senior Data Engineer for a client in South Texas. This role is critical in designing and managing scalable data infrastructure to support advanced analytics, machine learning, and real-time data processing. The ideal candidate will bring expertise in big data technologies, cloud services, and ETL processes to enhance the client's data capabilities.
Key Responsibilities:
Architect, implement, and manage scalable and reliable data pipelines using Apache Spark, Hadoop, and Kafka.
Work within AWS (preferred) or Azure cloud environments, leveraging services such as EC2, RDS, S3, Lambda, and Azure Data Lake.
Develop and optimize data models (SQL, NoSQL, Data Lakes) to ensure data quality, accessibility, and efficiency.
Automate ETL/ELT workflows using Apache Airflow, Talend, or similar tools.
Collaborate with data scientists and analysts, providing data infrastructure to support advanced analytics and machine learning models.
Implement and maintain data governance and security policies, including encryption, masking, and access control.
Monitor and troubleshoot data pipelines and databases to optimize performance and reliability.
Stay updated with emerging data engineering technologies, recommending improvements to enhance data processing efficiency.
Qualifications:
8 years of experience in data engineering with expertise in big data architectures and large-scale data pipelines.
Strong knowledge of cloud data solutions (AWS preferred, Azure acceptable).
Hands-on experience in Databricks, Dataiku, and AWS-native data offerings.
Proficiency in SQL, Python, Java, or Scala for data manipulation and pipeline development.
Experience in Infrastructure-as-Code (Terraform) for cloud infrastructure deployment.
Familiarity with machine learning model deployment and management is a plus.
Strong problem-solving skills with the ability to work cross-functionally across technical and business teams.
This is an exciting opportunity for an experienced Senior Data Engineer to contribute to a dynamic team, working on cutting-edge data technologies in a cloud-based environment.
Location: South Texas
Experience: 8 years in data engineering
Job Type: Permanent
Travel: Up to 20% (domestic)
Ideal Candidate Should Have:
Strong expertise in designing and managing large-scale data pipelines and architectures.
Hands-on experience with Apache Spark, Hadoop, and Kafka for big data processing.
Proficiency in AWS (preferred) or Azure cloud environments, including storage, compute, and data processing services.
Experience with ETL/ELT workflows and tools such as Apache Airflow or Talend.
Knowledge of Databricks, Dataiku, and AWS-native data solutions.
Proficient in SQL, Python, Java, or Scala for data engineering tasks.
Familiarity with Infrastructure-as-Code (Terraform) for cloud deployments.
Understanding of machine learning model deployment is a plus.
Strong communication skills to collaborate across technical and business teams.
Preferred certifications: AWS Certified Solution Architect, Databricks Certified Associate Developer for Apache Spark.
Job Description
Thinking Minds Inc. is seeking a Senior Data Engineer for a client in South Texas. This role is critical in designing and managing scalable data infrastructure to support advanced analytics, machine learning, and real-time data processing. The ideal candidate will bring expertise in big data technologies, cloud services, and ETL processes to enhance the client's data capabilities.
Key Responsibilities:
Architect, implement, and manage scalable and reliable data pipelines using Apache Spark, Hadoop, and Kafka.
Work within AWS (preferred) or Azure cloud environments, leveraging services such as EC2, RDS, S3, Lambda, and Azure Data Lake.
Develop and optimize data models (SQL, NoSQL, Data Lakes) to ensure data quality, accessibility, and efficiency.
Automate ETL/ELT workflows using Apache Airflow, Talend, or similar tools.
Collaborate with data scientists and analysts, providing data infrastructure to support advanced analytics and machine learning models.
Implement and maintain data governance and security policies, including encryption, masking, and access control.
Monitor and troubleshoot data pipelines and databases to optimize performance and reliability.
Stay updated with emerging data engineering technologies, recommending improvements to enhance data processing efficiency.
Qualifications:
8 years of experience in data engineering with expertise in big data architectures and large-scale data pipelines.
Strong knowledge of cloud data solutions (AWS preferred, Azure acceptable).
Hands-on experience in Databricks, Dataiku, and AWS-native data offerings.
Proficiency in SQL, Python, Java, or Scala for data manipulation and pipeline development.
Experience in Infrastructure-as-Code (Terraform) for cloud infrastructure deployment.
Familiarity with machine learning model deployment and management is a plus.
Strong problem-solving skills with the ability to work cross-functionally across technical and business teams.
This is an exciting opportunity for an experienced Senior Data Engineer to contribute to a dynamic team, working on cutting-edge data technologies in a cloud-based environment.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
