Splunk Engineer

Company Description

Zermount, Inc., a Certified Service-Disabled Veteran-Owned Small Business (SDVOSB), specializes in cybersecurity, information technology (IT), and management services, supporting both government and commercial clients. Founded by an industry leader with over 15 years of experience, Zermount delivers cutting-edge solutions tailored to client needs. The company is dedicated to establishing partnerships to help clients enhance efficiencies, reduce costs, and safeguard against threats. With innovative strategies and forward-thinking professionals, Zermount focuses on sustainable growth and leadership in the ever-evolving landscape of technology and cybersecurity.

Role Description

Zermount is seeking an experienced Splunk Engineer to support our client's enterprise security, operations, and monitoring environment. This role is responsible for the engineering, ongoing administration, maintenance, and enhancements of our client's Splunk environment, ensuring performance, scalability, and operational effectiveness. The ideal candidate brings proven operational experience in Splunk engineering and data ingestion, strong experience working within structured change management environments, and the ability to collaborate across infrastructure, network, and security teams.

Responsibilities

• Engineer, implement, configure, administer, maintain, upgrade, patch, and troubleshoot the Splunk Enterprise platform in accordance with client policies
• Design and continuously evaluate Splunk architecture to ensure scalability, performance, and alignment with current and future operational requirements; assess existing implementations and recommend enhancements or redesigns
• Onboard, ingest, parse, normalize, and troubleshoot new and existing data sources, including network traffic, application logs, databases, and cloud platforms
• Develop and maintain custom data parsers, field extractions, and data models to ensure accurate and efficient data integration across enterprise systems
• Install, configure, upgrade, and maintain Splunk Apps, Add-ons, and knowledge objects; extend platform functionality to meet operational needs.
• Develop and maintain custom searches, alerts, reports, and dashboards to support internal stakeholders, SOC, leadership, and external users; review and enhance detection and reporting capabilities.
• Monitor and optimize Splunk system performance, connectivity, license utilization, and overall platform health; conduct system tuning and capacity planning. Provide daily health check reports to management and stakeholders.
• Perform major version upgrades and support full platform lifecycle management, including patching, backup validation, restoration testing, and decommissioning activities
• Administer and troubleshoot Splunk infrastructure hosted on RHEL servers, including user account management, access controls, certificate maintenance, logging configuration, and configuration backups
• Troubleshoot ingestion failures, platform issues, and integration challenges; coordinate with internal teams and external vendors through issue resolution
• Develop technical documentation, architecture and data flow diagrams, and implementation strategies; participate in design reviews, testing cycles, and change management processes
• Collaborate with stakeholders and management to define requirements, translate business needs into technical deliverables, and provide accurate status updates
• Track, manage, and report on work through schedules, tickets (service, request, incident), workflows, status reports, dashboards, etc.
• Provide engineering, administrative and technical support as required to other team members or tools as a member of a cross functional security engineering team.

Qualifications

• 5 years of hands-on experience engineering and administering enterprise Splunk environments, including multi-site clustered and distributed architectures.
• Demonstrated expertise in log ingestion, data normalization, field extractions, and custom parser development across diverse data sources (network, application, database, cloud).
• Proficient with Splunk Search Processing Language (SPL), including development of complex searches, alerts, reports, and dashboards.
• Experience installing, configuring, upgrading, and performance tuning Splunk Enterprise in Linux environments (RHEL), including direct configuration of Splunk .conf files.
• Experience integrating Splunk with enterprise security and operational tools, including:
• Splunk DB Connect and custom SQL queries
• syslog-ng configuration on RHEL (SELinux environments)
• Custom integrations using Python, Bash, or PowerShell
• Experience supporting and optimizing distributed data pipelines, including administration of Cribl deployments and strategies to manage and reduce Splunk license consumption.
• Experience performing major version upgrades and lifecycle management activities within production environments.
• Experience operating within formal change management and ticket-driven workflows.
• Ability to produce technical documentation, architecture diagrams, and implementation artifacts.

EDUCATION / CERTIFICATION(S)

• Required: A minimum of Splunk Certified Administrator Certification or higher AND at least one IT Security certification reflected on the DOD 8140 IAT level II baseline.
• Preferred: The following are additional certifications that are preferred but not required: Splunk Certified Architect or Splunk Core Consultant; Linux Administration, and Cribl Certification

CLEARANCE

• Minimum Background Investigation

HOURS OF OPERATIONS

• 8:00 am ET – 4:00 pm ET
• After hours support maybe required to support emergency changes or system outages