Senior Site Reliability Engineer

Job Purpose/Summary

The Senior Site Reliability Engineer (SRE) will serve as the intermediary between product engineering and customer environments where our tools or platform are deployed, including secure and access-limited facilities. The role is accountable for the day-to-day health, reliability, security, and operational readiness of production edge and air-gapped deployments, and for ensuring customer operators and site administrators have the software infrastructure, automation, documentation, and support needed to operate with confidence.

Customer site support is expected to account for approximately 30-50% of the role's time. Offsite, the engineer will be an active member of the infrastructure team helping to design, build, validate, and maintain the infrastructure and SecDevOps capabilities that support development, QA, release candidate (RC), and production deployments. This includes reusable infrastructure-as-code (IaC) for cloud tenancies and disconnected or air-gapped customer tenancies, kubernetes and linux platform lifecycle management, infrastructure release management, observability, incident response; and security hardening.

This is a senior, hands-on, customer-facing role with substantial autonomy. The engineer will work directly with product and engineering leads to translate customer needs and operational constraints into actionable infrastructure, reliability, and deployment requirements; coordinate site activities; lead technical communications, briefings, and debriefings, maintain clear trip plans, trip reports, action items, risks, and status updates.

As a Senior level engineer at Knowmadics, this role is expected to drive technical direction within a domain, mentors others, and improve cross-team collaboration. As such, in this role you will be expected to be the technical reference for site reliability, solving hard, cross-cutting problems and in doing so defining quality bars and ensuring they are met for our site-deployed code, infrastructure, and applications.

Success in year one includes becoming a trusted onsite technical authority, establishing repeatable site-support and release-validation practices, improving platform reliability and security, delivering standardized infrastructure patterns that scale in cloud and air-gapped deployments, and starting to educate other engineers on the challenges of deploying stable systems in these environments.

Duties and Responsibilities

• Serve as a senior day-to-day technical owner for customer deployments, including systems both inside and outside secure facilities or SCIFs.

• Spend approximately 30%-50% of time supporting customer sites, releases, maintenance, incident response, operator needs, and stakeholder activities; contribute to infrastructure team priorities during the balance of the role.

• Monitor and support the availability, performance and security of linux and kubernetes platform deployments + troubleshoot using logs, metrics, events, and system telemetry

• Lead or coordinate incident triage, recovery, root-cause analysis and reporting across customer and internal stakeholders

• Plan, coordinate, and validate upgrades, patches, configuration changes, smoke tests, acceptance checks, and rollback procedures

• Create and maintain runbooks, standard operating procedures, troubleshooting procedures, site configuration baselines, and administrator documentation

• Prepare trip plans, access and logistics requirements, agendas, pre-briefs, onsite briefings and debriefings, trip reports, status updates

• Gather operational requirements and field feedback, then work with product and engineering leads to implement new platform/infrastructure features

• Design, implement, review, test, and maintain reusable IaC for development, QA, RC, and production cloud platform deployments

• Design and maintain IaC and deployment automation for disconnected and air-gapped customer tenancies, including environment-specific configuration, offline artifacts, installation and upgrades

• Maintain infrastructure delivery and upgrade processes and release artifacts, including versioning, artifact integrity, vulnerability remediation and secrets management

• Perform actions to ensure applicable DoD/DoW platform/software infrastructure security and compliance, support platform security hardening, risk assessment, and remediation

• Participate in architecture, release, change control, risk, and post-incident reviews, establish operational standards and provide technical guidance or mentorship without requiring formal direct reports

Qualifications

Required Qualifications

• Demonstrated senior-level experience in site reliability engineering, platform or infrastructure engineering, systems administration, field engineering, or a closely related role, including independent ownership of production systems and customer-facing technical work

• Advanced linux system administration experience, familiarity with RHEL administration, hardening, patching, storage, networking, service management, and system troubleshooting is preferred

• Hands-on experience operating and troubleshooting kubernetes clusters and supporting cluster lifecycle management, upgrades, workload health, networking, storage, and access controls

• Strong experience with infrastructure-as-code and configuration management tools such as Terraform and Ansible, including reusable modules, environment-specific configuration, and state management

• Experience designing or operating cloud-native infrastructure across development, QA, RC, and production environments, including compute, networking, identity, storage, secrets, observability, and release workflows

• Experience deploying and supporting disconnected or air-gapped systems, including offline artifacts, container images or registries, environment configuration, installation, upgrades, and rollback

• Experience establishing and using operational monitoring, alerting, dashboards, logs, and incident-management practices, including root-cause analysis and remediation

• Experience with DISA Security Technical Implementation Guides and with DoD/DoW security or compliance regimes, including system hardening, vulnerability remediation, and audit or control evidence

• Proven ability to communicate with technical and non-technical stakeholders, lead briefings and debriefings, prepare clear technical reports, and represent engineering interests in access-limited environments

• Ability to work autonomously, prioritize competing site and infrastructure needs, identify and communicate risk, and coordinate work across Product, Engineering, customer, and collaborator teams

• Must hold an active U.S. Security Clearance at Secret or high level - U.S. Citizenship required

Preferred Qualifications

• Experience supporting infrastructure for data-intensive platforms, including object storage, relational database services, distributed job workers, and machine-learning or model-serving workloads

• Experience with Git-based infrastructure delivery, CI/CD or GitOps practices, Helm, container registries, secure artifacts, software supply chain controls, and release rollback

• Experience supporting systems in secure facilities or SCIFs and coordinating customer access, maintenance windows, site visits, and technical acceptance activities

• Experience defining service-level objectives, capacity plans, operational readiness criteria, and reliability improvements for multi-environment platform deployments

Working conditions

• Employees may be called upon to participate in in-person meetings, training, or company functions at Knowmadics offices or other designated locations. Travel in support of business operations may also be required, and employees are expected to comply with these obligations as part of their position

• Candidate should live within driving distance of Waldorf, MD and be able to support the customer location at Blossom Point, MD

• Approximately 20-50% of the role's time is expected to involve customer-site support or related travel, primarily local; the exact allocation may vary with releases, incidents, site schedules, and program needs

• Work may occur in secure or access-limited environments and may occasionally include scheduled maintenance, release, or incident-response activity outside normal business hours

Physical requirements

May include sitting or standing for extended periods, working with computers and technical equipment, and occasionally lifting or moving materials or tools.

Direct reports

None. The role may lead cross-functional workstreams and provide technical direction, review, and mentorship.

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or protected veteran status.