Security Engineer - Mid - (HSEN) (Splunk Engineer)

Job Description

Job Description : \n\nDescription : Summary : Softek International, on behalf of the DHS Enterprise Security Engineering Division within the Office of the Chief Information Officer (OCIO) is responsible for the architecture, design, engineering, Tier 3 operations support, maintenance, and management of the network and security infrastructure. We are seeking a Splunk Engineer to support mission-critical systems and the ideal candidate will play a key role in enhancing our security monitoring, incident detection, and response capabilities by optimizing Splunk deployments within a secure and compliant environment. You will work closely with security, IT, and compliance teams to ensure that all logging, monitoring, and reporting align with federal regulations and security frameworks. This program portfolio includes management and oversight of the technical infrastructure and security tools, sensors, servers, and data storage devices to support the DHS Network Operations Security Center (NOSC) Cyber (SOC) and other DHS stakeholders and Components. Duties / Responsibilities :

• Design, configure, and maintain Splunk infrastructure to support security monitoring, data analytics, and operational efficiency.
• Develop and manage Splunk dashboards, alerts, reports, and visualizations that provide actionable insights for security operations and compliance auditing.
• Optimize Splunk queries (SPL) to meet agency-specific requirements for real-time monitoring, threat detection, and log analysis.
• Ensure timely onboarding of new data sources into Splunk, ensuring compliance with government data management and retention policies.
• Collaborate with internal cybersecurity, IT, and SOC teams to support threat hunting, incident response, and root-cause analysis using Splunk.
• Troubleshoot and resolve issues related to Splunk performance, indexing, and data accuracy, with a focus on maintaining compliance with government security standards (e.g., NIST, FISMA, CMMC).
• Manage Splunk upgrades, patches, and configuration changes in accordance with Change Management policies and procedures.
• Assist in maintaining Splunk environments in both on-premises and cloud-based government environments.
• Produce detailed documentation of system configurations, workflows, and processes for compliance audits and security reviews.
• Support the automation of incident detection and response processes through Splunk integrations with security orchestration tools.
• Create, manage, and support automation solutions for Splunk deployment and orchestration within a Cloud environment.
• Work closely with senior engineers, other team members and application owners to solve technical problems at the network, system and application levels.
• Conduct periodic architectural reviews of installed sensors to assess effectiveness and propose optimal installation alternatives as required.
• Conduct network security architecture reviews to determine the size, and placement of intrusion monitoring equipment during the customer onboarding process.
• Documentation and Reporting along with presentation, teamwork and DHS wide collaboration are among the expected duties and mission of the task order.
• Build, implement and administer Splunk in Windows and Linux environments.
• Conceptualize, Design, Build, and Maintain current and future NOSC supported tools and platforms. Preferred Qualifications / Requirements
• 3 to 5 years of experience in Splunk engineering, administration, and data integration within government or highly regulated environments.
• Strong knowledge of Splunk SPL (Search Processing Language) to build complex queries and optimize data extraction.
• Experience with integrating, normalizing, and managing log data from various sources, including network devices, servers, cloud services, and security tools.
• Familiarity with federal security frameworks and regulations (e.g., NIST, FISMA, CMMC, FedRAMP).
• Hands-on experience with Splunk Enterprise Security (ES) and developing security use cases for monitoring and incident detection.
• Strong understanding of network security principles, operating systems (Windows, Linux), and cybersecurity tools commonly used in government environments.
• Ability to troubleshoot and resolve issues related to system performance, log ingestion, and Splunk search efficiency.
• Experience working with cross-functional teams, including IT, security operations, and compliance.
• Proficiency managing Splunk using the Splunk command-line interface and config files
• Experience onboarding data into Splunk via forwarder, scripted inputs, TCP / UDP and modular inputs from a variety of sources.
• Proficiency onboarding data using Splunk developed add-ons for Windows, Linux, and common third-party devices and applications
• Experience collaborating with separate engineering teams to configure data sources for Splunk integration
• Experience in Linux, Windows and SQL / ODBC interfaces
• Proficiency implementing and onboarding data in Splunk DB Connect
• Experience with Splunk performing systems administration, including performing installation, configuration, monitoring system performance and availability, upgrades, and troubleshooting
• Experience developing in XML, Bash, JavaScript and Python, Perl, PowerShell scripts
• General networking and security troubleshooting (firewalls, routing, NAT, etc.)
• Splunk architecture / design, implementation, and troubleshooting experience
• Experience in managing, maintaining, and administering multi-site indexer cluster
• Scripting and development skills (BASH, python, or java) with strong knowledge of regular expressions
• Proficiency developing log ingestion and aggregation strategies per Splunk best practices
• Proficiency normalizing data to Splunk Common Information Model (CIM)
• Experience implementing and optimizing Splunk data models
• Expertise developing security-focused content for Splunk, including creation of complex threat detection log and operational dashboards
• Perform integration activities to configure, connect, and pull data with 3rd party software APIs.
• Ability to autonomously prioritize and successfully deliver across a portfolio of projects
• Undertakes day-to-day operational and user support
• Must be willing to participate to a rotating on-call support (24 / 7 / 365) for nights, weekends, holiday issues.
• Knowledge of scripting languages (e.g., Python, Bash) for automating tasks and streamlining Splunk processes.
• Experience with cloud environments (e.g., AWS GovCloud, Azure Government) and their integration with Splunk.
• Experience with automation tools (e.g., Ansible, Puppet) in government infrastructure.
• Familiarity with SIEM solutions and security orchestration tools (e.g., Swimlane, Phantom, Demisto) to enhance incident response capabilities. Experience / Years of Relevant Experience : Excellent verbal and written communication skills Ability to meet deadlines and work independently. Required Experience 5 years. Education : BA / BS in a Science, Technology, Engineering, Cybersecurity Management field Certifications (Not Required) : CISSP (Certified Information Security Systems Professional) CompTIA Security ITIL Foundations Experience with Agile-based project management (primary Kanban) Software / Hardware Experience Desired
• Splunk certifications (e.g., Splunk Core Certified Power User, Admin, Architect).
• Current Splunk Enterprise Certified Architect certification Clearance : Must be able to attain / maintain DHS EOD clearance. Must be a US Citizen and pass a background check. Maintain applicable security clearance(s) at the level required by the client and / or applicable certification(s) as Requested by Softek and / or required by Softek’s Client(s) / Customer(s). Location : Location : DHS HQ (TSA Springfield, VA), or Stennis Data Center Telework Authorized, Remote HybridRequirements : Summary : Softek International, on behalf of the DHS Enterprise Security Engineering Division within the Office of the Chief Information Officer (OCIO) is responsible for the architecture, design, engineering, Tier 3 operations support, maintenance, and management of the network and security infrastructure. We are seeking a Splunk Engineer to support mission-critical systems and the ideal candidate will play a key role in enhancing our security monitoring, incident detection, and response capabilities by optimizing Splunk deployments within a secure and compliant environment. You will work closely with security, IT, and compliance teams to ensure that all logging, monitoring, and reporting align with federal regulations and security frameworks. This program portfolio includes management and oversight of the technical infrastructure and security tools, sensors, servers, and data storage devices to support the DHS Network Operations Security Center (NOSC) Cyber (SOC) and other DHS stakeholders and Components. Duties / Responsibilities :
• Design, configure, and maintain Splunk infrastructure to support security monitoring, data analytics, and operational efficiency.
• Develop and manage Splunk dashboards, alerts, reports, and visualizations that provide actionable insights for security operations and compliance auditing.
• Optimize Splunk queries (SPL) to meet agency-specific requirements for real-time monitoring, threat detection, and log analysis.
• Ensure timely onboarding of new data sources into Splunk, ensuring compliance with government data management and retention policies.
• Collaborate with internal cybersecurity, IT, and SOC teams to support threat hunting, incident response, and root-cause analysis using Splunk.
• Troubleshoot and resolve issues related to Splunk performance, indexing, and data accuracy, with a focus on maintaining compliance with government security standards (e.g., NIST, FISMA, CMMC).
• Manage Splunk upgrades, patches, and configuration changes in accordance with Change Management policies and procedures.
• Assist in maintaining Splunk environments in both on-premises and cloud-based government environments.
• Produce detailed documentation of system configurations, workflows, and processes for compliance audits and security reviews.
• Support the automation of incident detection and response processes through Splunk integrations with security orchestration tools.
• Create, manage, and support automation solutions for Splunk deployment and orchestration within a Cloud environment.
• Work closely with senior engineers, other team members and application owners to solve technical problems at the network, system and application levels.
• Conduct periodic architectural reviews of installed sensors to assess effectiveness and propose optimal installation alternatives as required.
• Conduct network security architecture reviews to determine the size, and placement of intrusion monitoring equipment during the customer onboarding process.
• Documentation and Reporting along with presentation, teamwork and DHS wide collaboration are among the expected duties and mission of the task order.
• Build, implement and administer Splunk in Windows and Linux environments.
• Conceptualize, Design, Build, and Maintain current and future NOSC supported tools and platforms. Preferred Qualifications / Requirements
• 3 to 5 years of experience in Splunk engineering, administration, and data integration within government or highly regulated environments.
• Strong knowledge of Splunk SPL (Search Processing Language) to build complex queries and optimize data extraction.
• Experience with integrating, normalizing, and managing log data from various sources, including network devices, servers, cloud services, and security tools.
• Familiarity with federal security frameworks and regulations (e.g., NIST, FISMA, CMMC, FedRAMP).
• Hands-on experience with Splunk Enterprise Security (ES) and developing security use cases for monitoring and incident detection.
• Strong understanding of network security principles, operating systems (Windows, Linux), and cybersecurity tools commonly used in government environments.
• Ability to troubleshoot and resolve issues related to system performance, log ingestion, and Splunk search efficiency.
• Experience working with cross-functional teams, including IT, security operations, and compliance.
• Proficiency managing Splunk using the Splunk command-line interface and config files
• Experience onboarding data into Splunk via forwarder, scripted inputs, TCP / UDP and modular inputs from a variety of sources.
• Proficiency onboarding data using Splunk developed add-ons for Windows, Linux, and common third-party devices and applications
• Experience collaborating with separate engineering teams to configure data sources for Splunk integration
• Experience in Linux, Windows and SQL / ODBC interfaces
• Proficiency implementing and onboarding data in Splunk DB Connect
• Experience with Splunk performing systems administration, including performing installation, configuration, monitoring system performance and availability, upgrades, and troubleshooting
• Experience developing in XML, Bash, JavaScript and Python, Perl, PowerShell scripts
• General networking and security troubleshooting (firewalls, routing, NAT, etc.)
• Splunk architecture / design, implementation, and troubleshooting experience
• Experience in managing, maintaining, and administering multi-site indexer cluster
• Scripting and development skills (BASH, python, or java) with strong knowledge of regular expressions
• Proficiency developing log ingestion and aggregation strategies per Splunk best practices
• Proficiency normalizing data to Splunk Common Information Model (CIM)
• Experience implementing and optimizing Splunk data models
• Expertise developing security-focused content for Splunk, including creation of complex threat detection log and operational dashboards
• Perform integration activities to configure, connect, and pull data with 3rd party software APIs.
• Ability to autonomously prioritize and successfully deliver across a portfolio of projects
• Undertakes day-to-day operational and user support
• Must be willing to participate to a rotating on-call support (24 / 7 / 365) for nights, weekends, holiday issues.
• Knowledge of scripting languages (e.g., Python, Bash) for automating tasks and streamlining Splunk processes.
• Experience with cloud environments (e.g., AWS GovCloud, Azure Government) and their integration with Splunk.
• Experience with automation tools (e.g., Ansible, Puppet) in government infrastructure.
• Familiarity with SIEM solutions and security orchestration tools (e.g., Swimlane, Phantom, Demisto) to enhance incident response capabilities. Experience / Years of Relevant Experience : Excellent verbal and written communication skills Ability to meet deadlines and work independently. Required Experience 5 years. Education : BA / BS in a Science, Technology, Engineering, Cybersecurity Management field Certifications (Not Required) : CISSP (Certified Information Security Systems Professional) Com