Senior Software Tester

Position Summary

MS Consulting is seeking a Senior Software Tester to serve as the senior individual contributor for platform and AI testing for the clients Blanket Purchase Agreement, where MS is the Salesforce and Agentforce delivery partner under the prime contractor. This role designs and executes the test cases, evaluation transcripts, and scoring logic that validate the automated contact center solution on Salesforce, with a strong emphasis on the Agentforce / AI evaluation program under PWS 5.13.2.6. The Senior Software Tester works under the Software Testing Manager, partners closely with the Senior Software Developer / Manager and the developers, and helps calibrate the cross-clients functional testing team. This is a fully dedicated position within the MS software delivery scope.

Key Responsibilities

• Design, document, and execute functional, integration, regression, and user acceptance test cases for the Salesforce and Agentforce contact center solution.
• Build and run AI evaluations under PWS 5.13.2.6, including test cases with complete transcripts and scoring logic, regression evals, faithfulness checks, benchmarking, LLM-as-a-Judge grading, and red teaming / adversarial testing to confirm outputs stay within configured guardrails.
• Serve as a human-in-the-loop reviewer, scoring AI outputs to calibrate automated graders and validate accuracy, reliability, safety, and quality of AI-generated content.
• Log, triage, and track defects to resolution; verify fixes; and maintain traceability from test cases to PWS requirements.
• Validate Service Cloud, Service Cloud Voice, Open CTI, and integration behavior, including case creation, routing, and knowledge and content alignment.
• Produce clear test documentation, evaluation results, and release evidence, and present findings to the Software Testing Manager and stakeholders.
• Coordinate with IT, Content, Operations, Quality, and Training stakeholders to support continuous improvement across the clients AI channels.
• Support NIST SP 800-53, FISMA, and FedRAMP Moderate-aligned test and release evidence.

Required Qualifications (Mandatory)

• Minimum of three years of software testing, quality assurance, or Salesforce configuration / administration experience.
• Salesforce Certified Agentforce Specialist, current and in good standing, to support the AI evaluation and testing scope.
• Demonstrated experience testing, evaluating, or monitoring Salesforce applications or AI / large language model outputs against defined criteria.
• Certification level pre-approval ability to have any Salesforce or Agentforce certification level submitted to and approved by the client before performing any coding or configuration on the platform.
• Bachelor of Arts or Bachelor of Science degree, or equivalent relevant experience. Fields of information systems, business administration, or computer science are preferred.
• Public Trust Low Risk suitability, or the ability to begin and complete the Public Trust Low Risk suitability process at award.

Preferred Qualifications

• Hands-on experience testing and monitoring AI agents for continuous improvement, including federal Agentforce / agentic AI work.
• Experience with an AI evaluation harness or framework and with LLM-as-a-Judge, faithfulness, and regression evaluation methods.
• Service Cloud, Experience Cloud, and case management at contact center scale, including Service Cloud Voice and Open CTI.
• Salesforce administration or architecture depth, such as Administrator, Advanced Administrator, Platform App Builder, Sharing and Visibility Architect, Data Architect, or Data Cloud.
• Experience with service analytics, reporting, and dashboards for quality and performance monitoring.
• Familiarity with defect-tracking and CI/CD tooling such as Jira, Git, Salesforce DX, Copado, or Gearset.
• Federal delivery aligned to NIST SP 800-53, FISMA, and FedRAMP Moderate.

Engagement Details

• Dedication fully dedicated to the client engagement.
• Role designation software delivery support staff; not a PWS-named Key Personnel position.
• Reporting reports to the Software Testing Manager and coordinates with the Senior Software Developer / Manager and development team.
• Employer MS Consulting.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.