Site Reliability Engineer

Welcome to PCI Pal

PCI Pal is a leading provider of SaaS solutions that empower companies to take payments securely, adhere to strict industry governance, and remove their business from the significant risks posed by non-compliance and data loss. We are integrated and resold by some of the worlds’ leading business communications vendors, as well as major payment service providers.

We are currently looking for a Site Reliability Engineer to join our UK team.

Location: This role will require monthly travel to our London and Ipswich office

The opportunity:

You'll join PCI Pal's DevOps team as the go-to authority on observability and reliability, owning our DataDog implementation and extending monitoring, alerting and anomaly detection across infrastructure, networking and telephony. Working closely with the senior DevOps engineer and the wider team, you'll drive down manual toil, improve resiliency, and own capacity planning, availability targets and disaster recovery readiness as load grows. You're joining an existing team and will be relied on to raise the reliability and observability bar for the whole group.

You will be responsible for:

• Own the observability framework and standards across infrastructure, networking and telephony, defining what good monitoring and alerting look like, and partnering with development teams to implement against these standards, including a practical monitoring scorecard, reviewing and signing off on their coverage
• Control observability costs - right-sizing usage (hosts, custom metrics, log ingestion/retention, APM) and making deliberate cost-vs-coverage tradeoffs
• Work with teams to reduce alert noise while making sure genuine incidents are still surfaced immediately
• Act as the technical authority on monitoring maturity across the engineering function
• Mentor and upskill the wider DevOps team on reliability and observability practices, working alongside the senior DevOps engineer
• Identify and automate away manual toil, including incident response workflows, compliance evidence collection and release processes
• Drive resiliency improvements across the infrastructure estate - redundancy, failover and capacity planning
• Lead on incident response and resolution, contributing to postmortem practices that turn failures into lasting fixes
• Work with development teams to embed resiliency practices into release processes, e.g. progressive rollouts, canary deployments and fast rollback, so changes can ship quickly without risking availability

Job requirements:

• Extensive, hands-on experience with DataDog (or equivalent tools such as New Relic, Grafana/Prometheus, Dynatrace)
• Experience managing and controlling observability tooling costs at scale
• Solid experience monitoring networking infrastructure (latency, packet loss, device health, anomaly detection)
• Experience with capacity planning, demand forecasting, and redundancy/failover design
• Strong background in incident response practices and building automation around them
• Experience defining SLIs/SLOs and error-budget-based alerting
• Solid software engineering skills, writing maintainable, testable, version-controlled automation and tooling
• Experience implementing progressive delivery practices (canary deployments, staged rollouts, automated rollback)
• Confidence engaging with and influencing development teams
• Experience of PCI compliance or other similar compliance frameworks
• Experience with automated build systems (e.g. Jenkins), work management systems (e.g. Jira), and source control

Nice to haves:

• Experience with telephony/voice infrastructure monitoring
• Experience mentoring or upskilling other engineers

Talk to us:

If you have any questions or want to find out more, we’d love to hear from you.

Please contact the Recruitment Team [email protected]