Senior Splunk Engineer

Job Description – Principal Splunk Engineer

Position: Principal Splunk Engineer

Experience: 10 Years

Full time position

Job Summary:

We are seeking a highly skilled Principal Splunk Engineer to join our team. This role will be responsible for designing, implementing, and maintaining scalable and highly available Splunk architectures tailored to enterprise requirements. The ideal candidate will have deep expertise in Splunk administration, automation, DevOps integration, and cloud infrastructure, ensuring seamless system observability, security, and performance monitoring across all services.

Key Responsibilities:

• Splunk Architecture & Implementation:
• Design and implement scalable, distributed Splunk environments that align with business and operational needs.
• Optimize log ingestion pipelines, indexing, and storage for enhanced performance and cost efficiency.
• Maintain large clustered and distributed Splunk installations with high availability.
• Integration & Automation:
• Integrate Splunk with DevOps tools (CI/CD pipelines, GitHub, Jenkins) for real-time monitoring and security insights.
• Develop and maintain automation scripts using Python, Shell scripting, Ansible, and Terraform to streamline Splunk deployments, configurations, and maintenance.
• Ensure cloud/on-prem infrastructure meets SLAs with application and service owners.
• Security & Compliance:
• Implement Splunk security best practices to safeguard enterprise data and ensure compliance with regulatory requirements.
• Troubleshoot and resolve security, system, and application issues to maintain optimal system performance.
• Monitoring & Performance Optimization:
• Continuously monitor Splunk deployments to ensure efficient resource utilization and performance tuning.
• Identify and implement automation and scripting solutions to enhance system reliability and reduce manual efforts.
• Support incident response, root cause analysis, and performance troubleshooting for Splunk environments.
• Collaboration & Stakeholder Engagement:
• Work closely with cross-functional teams including developers, operations, security, and business stakeholders to ensure Splunk aligns with organizational goals.
• Develop and maintain custom dashboards, reports, and alerts for visibility across enterprise services.

Required Skills & Qualifications:

• 10 years of experience in infrastructure and systems engineering in an Agile environment.
• 5 years of experience in Splunk administration, automation, and backend maintenance (preferably using Ansible).
• Experience managing large-scale, distributed, and clustered Splunk deployments.
• 5 years of experience in building and maintaining Splunk apps using Python, Shell scripting, or equivalent programming languages.
• Strong knowledge of Linux & Windows environments.
• 3 years of experience working with data centers, application servers, and network infrastructure.
• 3 years of DevOps & CI/CD development experience (Git, Python, Ansible).
• Strong expertise in REST APIs, Splunk SDKs, and application integrations.
• Bachelor's degree in Computer Science, Information Systems, or a related field (or equivalent work experience).

Preferred Qualifications:

• Experience with Splunk IT Service Intelligence (ITSI) in large-scale environments.
• Knowledge of machine learning models for anomaly detection and predictive analysis within Splunk.
• Experience with Windows and Linux administration, including Docker, Kubernetes, and container orchestration.
• Familiarity with network and security monitoring using Splunk in enterprise environments.
• Understanding of cloud platforms (AWS, Azure, GCP) and hybrid cloud monitoring.