Senior QA Engineer (AI Systems & Automation)

Rockville, MD
Full Time
Experienced

Senior QA Engineer (AI Systems & Automation)

Essential Software Inc. (ESI)

Overview

Essential Software Inc. is a trusted partner to federal agencies, including the National Cancer Institute (NCI), delivering secure, cloud-based platforms that support large-scale cancer data and biomedical research.

As a Senior QA Engineer (AI Systems & Automation), you will lead quality strategy and test automation for critical data platforms and AI-powered experiences. You will ensure both traditional software and AI/agentic systems are reliable, explainable, and safe in a federal, mission-driven environment.

You will:

  • Own end-to-end quality for complex web, API, data, and AI/ML-powered features
  • Design AI-aware test strategies and automation that leverage GenAI and agentic frameworks
  • Mentor QA engineers and collaborate closely with cross-functional teams and government partners

Roles and Responsibilities

Core QA & Automation

  • Develop and maintain test plans, test cases, traceability, and test data for product and AI features
  • Execute manual and automated tests for web applications, RESTful APIs, data workflows, and AI/ML features
  • Own automated regression suites, release readiness criteria, and provide clear go / no-go quality signals
  • Participate in agile ceremonies, validate end-to-end functionality, and ensure user stories (including AI features) meet acceptance criteria
  • Manage the full defect lifecycle, including triage, prioritization, root cause analysis, and verification of fixes
  • Maintain QA documentation, runbooks, and quality dashboards

AI & Agentic QA Systems

  • Design and execute test strategies for AI/LLM-powered capabilities, including virtual agents, chatbots, copilots, and RAG-based systems
  • Use LLM-powered tools (e.g., ChatGPT, Claude, Copilot) to accelerate test design, data generation, exploratory testing, and script authoring
  • Build and refine QA-focused AI agents that can:
    • Scrape UI and verify DOM structures
    • Validate data against backend or ground-truth sources
    • Auto-generate and maintain test scripts
    • Run self-correcting / autonomous test flows
  • Evaluate and integrate agentic frameworks (e.g., OpenAI Assistants API, AWS Bedrock Agents, LangGraph, MCP) into QA workflows
  • Define and monitor AI-specific quality metrics (accuracy vs. ground truth, hallucination and error rates, safety / policy adherence)
  • Ensure AI and virtual agent experiences are accurate, consistent, and high quality in a federal context

Non-Functional, Data Quality & Collaboration

  • Plan and execute performance, load, and scalability testing (e.g., JMeter or equivalent)
  • Validate data integrity and transformation quality across complex biomedical data pipelines and AI-enhanced workflows
  • Partner with engineers and data scientists to ensure AI/ML models and integrations are testable, observable, and measurable post-deployment
  • Mentor QA team members in both traditional and AI-augmented QA practices
  • Collaborate with development, DevOps, product, UX, and data teams to improve testability, shift-left quality, and increase automated coverage
  • Integrate automation into CI/CD (e.g., GitHub Actions, Jenkins, Azure DevOps, GitLab CI), monitor test health and flakiness, and address coverage gaps
  • Communicate quality risks, trends, and mitigation plans to technical and non-technical stakeholders, including government partners

Basic Qualifications

  • Bachelor’s degree in computer science, Information Technology, Engineering, or related field
  • 5+ years of software QA experience (manual and automation) in production environments
  • 2+ years providing technical or process leadership (e.g., lead QA, primary product QA owner, mentor, or manager)
  • Strong experience with UI automation tools (Selenium WebDriver, Playwright, or Cypress)
  • Experience testing RESTful APIs and microservices architectures
  • Hands-on experience integrating automated tests into CI/CD pipelines (GitHub Actions, Jenkins, Azure DevOps, or GitLab CI)
  • Professional proficiency in Python or JavaScript for test automation
  • Hands-on use of GenAI tools (e.g., ChatGPT, Claude, Copilot) for QA tasks such as test-case generation, data creation, and exploratory testing
  • Understanding of AI/agentic concepts:
    • Tool-calling / function invocation
    • Multi-step / chain-of-thought workflows
    • Autonomous / self-healing test flows
    • AI-driven data comparison and validation
  • Experience with performance / load testing (e.g., JMeter or equivalent)
  • Proficiency with Jira or similar issue tracking tools
  • Strong written and verbal communication skills, including the ability to explain AI-related quality risks to stakeholders
  • Ability to prioritize, multitask, and operate effectively in complex, mission-driven environments

Preferred Qualifications

  • AWS Cloud Practitioner certification
  • Experience with modern automation stacks (Playwright or Cypress) and API testing tools (Postman, REST-assured, pytest, or similar)
  • Experience testing AI/ML-powered features (LLM applications, RAG systems, agents, recommendation engines, or chatbots)
  • Experience with one or more:
    • LangChain or LangGraph
    • AWS Bedrock Agents or OpenAI Assistants API
    • MCP (Multi-Context Protocol) or similar orchestration frameworks
  • Experience designing or testing internal QA copilots or automation bots for test authoring or execution
  • Familiarity with test management tools (e.g., TestRail, Zephyr)
  • Knowledge of accessibility standards (WCAG) and basic security testing practices
  • Prior QA experience in healthcare, life sciences, biomedical informatics, or other regulated data environments
  • ISTQB or similar certification

How We Work

  • Mission driven and customer focused, working directly with scientists, product owners, and data experts
  • Curious and learning oriented, with encouragement to experiment with new AI tools, patterns, and technologies
  • Work on cutting-edge biomedical informatics and AI projects with real-world impact
  • Full-time role; primarily remote, with approximately four days per month onsite in Rockville, MD
  • Competitive benefits, professional development opportunities, and a collaborative, supportive culture

Compensation

Salary range for this position: $107,000 - $138,000, depending on skills and experience.

Equal Opportunity Employer

Essential Software Inc. provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type, without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws. This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation, and training.

Share

Apply for this position

Required*
We've received your resume. Click here to update it.
Attach resume or Paste resume
Attach resume as .pdf, .doc, .docx, .odt, .txt, or .rtf (limit 5MB) or Paste resume

Paste your resume here or Attach resume file

To comply with government Equal Employment Opportunity and/or Affirmative Action reporting regulations, we are requesting (but NOT requiring) that you enter this personal data. This information will not be used in connection with any employment decisions, and will be used solely as permitted by state and federal law. Your voluntary cooperation would be appreciated. Learn more.
Invitation for Job Applicants to Self-Identify as a U.S. Veteran
  • A “disabled veteran” is one of the following:
    • a veteran of the U.S. military, ground, naval or air service who is entitled to compensation (or who but for the receipt of military retired pay would be entitled to compensation) under laws administered by the Secretary of Veterans Affairs; or
    • a person who was discharged or released from active duty because of a service-connected disability.
  • A “recently separated veteran” means any veteran during the three-year period beginning on the date of such veteran's discharge or release from active duty in the U.S. military, ground, naval, or air service.
  • An “active duty wartime or campaign badge veteran” means a veteran who served on active duty in the U.S. military, ground, naval or air service during a war, or in a campaign or expedition for which a campaign badge has been authorized under the laws administered by the Department of Defense.
  • An “Armed forces service medal veteran” means a veteran who, while serving on active duty in the U.S. military, ground, naval or air service, participated in a United States military operation for which an Armed Forces service medal was awarded pursuant to Executive Order 12985.
Veteran status


Voluntary Self-Identification of Disability
Voluntary Self-Identification of Disability Form CC-305
OMB Control Number 1250-0005
Expires 04/30/2026
Why are you being asked to complete this form?

We are a federal contractor or subcontractor. The law requires us to provide equal employment opportunity to qualified people with disabilities. We have a goal of having at least 7% of our workers as people with disabilities. The law says we must measure our progress towards this goal. To do this, we must ask applicants and employees if they have a disability or have ever had one. People can become disabled, so we need to ask this question at least every five years.

Completing this form is voluntary, and we hope that you will choose to do so. Your answer is confidential. No one who makes hiring decisions will see it. Your decision to complete the form and your answer will not harm you in any way. If you want to learn more about the law or this form, visit the U.S. Department of Labor’s Office of Federal Contract Compliance Programs (OFCCP) website at www.dol.gov/ofccp.

How do you know if you have a disability?

A disability is a condition that substantially limits one or more of your “major life activities.” If you have or have ever had such a condition, you are a person with a disability. Disabilities include, but are not limited to:

  • Alcohol or other substance use disorder (not currently using drugs illegally)
  • Autoimmune disorder, for example, lupus, fibromyalgia, rheumatoid arthritis, HIV/AIDS
  • Blind or low vision
  • Cancer (past or present)
  • Cardiovascular or heart disease
  • Celiac disease
  • Cerebral palsy
  • Deaf or serious difficulty hearing
  • Diabetes
  • Disfigurement, for example, disfigurement caused by burns, wounds, accidents, or congenital disorders
  • Epilepsy or other seizure disorder
  • Gastrointestinal disorders, for example, Crohn's Disease, irritable bowel syndrome
  • Intellectual or developmental disability
  • Mental health conditions, for example, depression, bipolar disorder, anxiety disorder, schizophrenia, PTSD
  • Missing limbs or partially missing limbs
  • Mobility impairment, benefiting from the use of a wheelchair, scooter, walker, leg brace(s) and/or other supports
  • Nervous system condition, for example, migraine headaches, Parkinson’s disease, multiple sclerosis (MS)
  • Neurodivergence, for example, attention-deficit/hyperactivity disorder (ADHD), autism spectrum disorder, dyslexia, dyspraxia, other learning disabilities
  • Partial or complete paralysis (any cause)
  • Pulmonary or respiratory conditions, for example, tuberculosis, asthma, emphysema
  • Short stature (dwarfism)
  • Traumatic brain injury
Please check one of the boxes below:

PUBLIC BURDEN STATEMENT: According to the Paperwork Reduction Act of 1995 no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. This survey should take about 5 minutes to complete.

You must enter your name and date
Human Check*