Systems Engineer/Senior Data Engineer - Splunk, ServiceNow & AppDynamics

Job DescriptionJob Description

***Security Clearance: Must possess an active TS/SCI with Full Scope Polygraph***

KDA is seeking a highly skilled and experienced Systems Engineer/Senior Data Engineer with a strong background in designing, implementing, and optimizing data pipelines and solutions for critical security and operational platforms, specifically Splunk, ServiceNow, and AppDynamics. The ideal candidate will be a proactive, problem-solving individual with a proven track record of managing complex data ingestion, normalization, and correlation processes within highly visible, operationally driven environments. This role requires an individual who thrives in a collaborative setting, possesses excellent communication skills, and is dedicated to continuous improvement and innovation.

Key Responsibilities:

• Splunk Data Engineering:
• Design, engineer, and maintain robust Splunk infrastructures, including clustered environments, for large-scale data ingestion, correlation, and reporting.
• Automate complex data ingestion methods (e.g., S3, syslog, JSON, APIs) from diverse sources across multiple enclaves.
• Develop and implement methods for data tagging and cataloging to ensure compliance with evolving security standards and facilitate efficient data discovery.
• Optimize data ingest performance and efficiency across various network environments. Familiar with Technical Add-ons.
• Parse and normalize non-standard data sets to enable comprehensive analysis and correlation within Splunk.
• Develop and refine Splunk queries, dashboards, and reports to visualize security events, infrastructure health, and operational metrics.
• Collaborate with IT operations and cyber security teams to enrich data sets, ascertain cyber threats, and bolster security posture.
• Maintain ITSI and SIEM-like tools and custom content within virtualized environments.
• Perform tuning and filtering of events and information, creating custom views and content.
• Familiar with UBA and Splunk.
• Collaborate with cross-functional teams to design and implement data integrations between various security and operational tools (including Splunk and AppDynamics) and ServiceNow.
• Develop and maintain data pipelines to ensure accurate and timely flow of security incidents, alerts, and operational metrics into ServiceNow for incident management, problem management, and reporting.
• Assist in defining and implementing data models within ServiceNow to support security operations and compliance initiatives.
• Work with third-party services for design review and optimal deployment configuration for enterprise cloud service utilization (relevant to integrations).
• Design and implement data collection strategies for AppDynamics, ensuring comprehensive monitoring of application performance and infrastructure.
• Integrate AppDynamics data with Splunk for centralized visibility and correlation with other security and operational logs.
• Troubleshoot problematic service deployments and data flows, utilizing forensic tools and audit log review (relevant to monitoring and analysis).
• Develop methods to leverage AppDynamics data for identifying potential risks and optimizing application performance.
• General Data Engineering & Systems Expertise:
• Collaborate with mission partners to develop long-term enterprise audit solutions and normalize non-standard data sets.
• Engineer and maintain secure virtualized and cloud environments for data platforms.
• Deploy and harden servers running Linux OS in accordance with CIS and other STIG guidelines.
• Develop runbooks, SOPs, and documentation for new processes and systems.
• Perform liaison duties between service providers and clients to bridge communication gaps and ensure adherence to SLAs.
• Review and evaluate data integrity and develop use cases for various data sets.
• Maintain system baselines and configuration management for data engineering tools.
• Contribute to the development of plans to safeguard data against unauthorized modification, destruction, or disclosure.
• Strong understanding of cyber security principles and experience with various security tools (e.g., Next-Gen Firewalls, IPS/IDS, Tenable Nessus, Rapid7 Nexpose, McAfee EPO, Symantec SEP).

Required Skills & Experience:

• 20+ years of progressive experience in Information Technology and Security, with a strong focus on data engineering and systems integration.
• Security Clearance: Candidate must possess an active TS/SCI with Full Scope Polygraph
• Demonstrated expertise in engineering and maintaining large-scale Splunk environments, including data ingestion, parsing, normalization, and content development.
• Experience with automating complex data ingestion methods (e.g., S3, syslog, JSON, APIs).
• Strong understanding of data tagging, cataloging, and data governance best practices.
• Proficiency with Linux OS administration and hardening.
• Familiarity with cloud security principles and deploying commercial services into protected/secured enclaves (e.g., AWS).
• Experience with SIEM solutions and their implementation, configuration, and maintenance.
• Strong scripting skills (e.g., BASH, Python, PowerShell).
• Excellent collaboration and communication skills, with the ability to work effectively in small teams and large collaborative efforts.
• Ability to troubleshoot complex technical issues and perform root cause analysis.
• Proven ability to develop and maintain documentation (runbooks, SOPs).

Desired Skills (Plus, but not required):

• Experience with ServiceNow platform administration, development, or integration.
• Experience with AppDynamics for application performance monitoring and data collection.
• Experience with configuration management tools - Git, Ansible