What are the responsibilities and job description for the Senior Data Engineer position at Quiet Professionals, LLC?
Position Description :
Role Overview : As a Senior Data Engineer, you will be responsible for designing, developing, and maintaining robust, fault-tolerant data pipelines that process and analyze large-scale OSINT datasets. You will work with structured, semi-structured, and unstructured data, integrating diverse data sources to enable real-time intelligence insights. You will collaborate closely with data scientists, analysts, and cloud engineers to enhance our data processing capabilities, leveraging best practices in big data, cloud computing, and machine learning .
Key Responsibilities :
- Architect, develop, and optimize high-throughput, low-latency ETL / ELT pipelines for OSINT data ingestion, transformation, and analysis.
- Implement data enrichment workflows using NLP, machine translation, Named Entity Recognition (NER), and entity resolution.
- Utilize Apache Spark, Delta Lake, and Databricks to process and analyze large-scale datasets efficiently.
- Deploy and manage data solutions in Azure, including Azure Data Factory, Azure Synapse Analytics, Azure Databricks, and Azure Kubernetes Service (AKS).
- Develop real-time streaming solutions using Apache Kafka, Azure Event Hubs, Apache Flink, and Spark Streaming.
- Implement data observability, lineage, and quality monitoring using tools such as Great Expectations, Monte Carlo, or Databricks Unity Catalog.
- Optimize query performance, indexing, and partitioning strategies for cloud-based analytics solutions.
- Work with DevOps teams to implement CI / CD for data pipelines using Terraform, GitHub Actions, or Azure DevOps.
- Secure and govern data pipelines following NIST, FedRAMP, and Zero Trust Architecture principles.
- Develop API-based data integrations with GraphQL, REST, or gRPC endpoints for internal and external data services.
Required Qualifications :
Preferred Qualifications :
Quiet Professionals is located in the Greater Tampa Bay Area. We provide innovative and sustainable solutions that enhance the operational effectiveness of our clients and partne rs.
QP is proud to be an equal opportunity, disability, and veteran employer. Our team of Quiet Professionals is highly driven, innovative, and results oriented. We empower our teams creativity, knowledge, and expertise to support client needs by fostering a diverse, inclusive, and supportive workplace. We understand that diversity fuels innovation and drives our success. We offer comprehensive benefits designed to serve the needs of our workforce.
