What are the responsibilities and job description for the Big Data Engineer -W2 position at Global Bridge InfoTech Inc.?
Job Details
Big Data Engineer -W2
Note: Resource should be local to VA/MD
Location: Hybrid/Reston, VA(Primarily remote, occasional meetings occur onsite)
We are looking for a highly skilled Big Data Engineer to design, build, and optimize large-scale data processing pipelines.
The ideal candidate should have expertise in distributed computing, data ingestion, and processing frameworks such as Apache Spark, Hadoop, Kafka, and cloud-based big data solutions.
This role requires strong problem-solving skills, hands-on experience with data lakes, and a passion for working with large datasets.
Key Responsibilities:
- Design, develop, and maintain scalable and efficient big data pipelines for processing large datasets.
- Work with Apache Spark, Hadoop, Hive, and Kafka to process structured and unstructured data.
- Implement data ingestion, transformation, and storage solutions using distributed computing frameworks.
- Optimize ETL workflows, query performance, and data pipeline efficiency.
- Develop real-time and batch data processing solutions using Spark Streaming, Flink, or Kafka Streams.
- Work with SQL and NoSQL databases (PostgreSQL, MongoDB, Cassandra, HBase).
- Deploy and manage data infrastructure on cloud platforms (AWS, Azure, Google Cloud Platform) using services like Databricks, AWS EMR, Glue, BigQuery, Synapse Analytics.
- Ensure data quality, governance, and security across all pipelines.
- Work closely with Data Scientists, Analysts, and DevOps teams to support analytics and machine learning workloads.
- Implement CI/CD pipelines for automating data pipeline deployments.
Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.
