Add as preferred on Google
Custom Software Development Vendor Evaluation Scorecard (How to Use + Download)Read More
How to Evaluate a Custom Software Development Vendor’s QA Practices
Ask each software development vendor the same five quality assurance question groups for evaluation, then judge every answer on the evidence they can produce rather than the process they describe: quality ownership and governance, test scope and risk coverage, defect management and transparency, environments and test data, and reporting, traceability, and buyer access. Define release acceptance before development starts.
Request one anonymized artifact per group, such as a quality plan, a traceability slice, or a defect record. The failure to watch for is a certification, methodology, or tool name offered in place of an artifact.
Introduction
“High quality” is not a useful vendor commitment until it becomes observable. A buyer needs to know what will be tested, who owns each decision, what evidence will be available, and what must be true before a release is accepted.
The goal is to make quality expectations specific enough to compare vendors and turn them into QA questions to ask a software development company and record commitments before work begins.
Start With a Shared Definition of Software Quality
Software quality is more than the absence of defects. It includes whether the system meets agreed requirements, supports intended users, performs reliably under expected conditions, protects data, integrates correctly, and remains maintainable as the product changes.
Quality Assurance, or QA, is the planned and systematic work used to create confidence that quality requirements will be met. Quality Control focuses on examining work products and results. Software testing evaluates software and related artifacts to identify problems. Verification asks whether the product was built according to specified requirements and design decisions. Validation asks whether the finished system is suitable for its intended use. User Acceptance Testing, or UAT, examines the product from the business or intended-user perspective.
These practices overlap, but they are not interchangeable. UAT does not replace integration, security, performance, accessibility, or technical verification. The right assurance depth depends on system risk, user impact, integration complexity, data sensitivity, and the consequences of failure.
Ask How Quality Is Built Into Delivery
Late testing cannot repair an untestable requirement, a flawed design, missing integration assumptions, or uncontrolled code changes. Quality should begin during discovery and continue through requirements, implementation, release, and post-release remediation.
During discovery, the vendor should identify critical workflows, dependencies, constraints, and failure consequences. Those risks should become testable requirements, review criteria, and release evidence.
Ask for an anonymized quality plan, lifecycle workflow, requirements-to-test example, and recent release-readiness summary. These artifacts show whether the vendor has an operating process or only a methodology claim.
How Do You Turn Requirements Into Testable Acceptance Criteria?
A testable acceptance criterion describes an observable condition that can be evaluated. “The dashboard should be fast” is not testable. A useful criterion identifies the user action, operating conditions, expected result, data state, and an agreed performance objective when speed matters.
Ask who drafts, reviews, and approves acceptance criteria. Security, accessibility, performance, reliability, migration, and recovery expectations should be identified early enough to affect design and estimates.
Traceability should connect a business need to a requirement, acceptance criteria, tests, test results, defects, approved exceptions, and the final acceptance decision. Full traceability may be unnecessary for every minor item, but it should exist for critical workflows and high-risk requirements.
Request one anonymized vertical slice of evidence: a requirement, its acceptance criteria, linked tests, a defect produced by a failed result, the fix, the retest, and the acceptance outcome. Vague phrases such as “client satisfaction” or “production ready” should trigger follow-up questions.
Which Reviews and Tests Happen Before Code Reaches a Release Candidate?
Different controls address different failure modes. Peer review can identify design mistakes and misunderstood requirements. Static analysis examines code or related artifacts without executing the program. Unit tests evaluate small components. Integration tests examine interfaces. System tests evaluate assembled behavior. Exploratory testing uses human judgment to find risks that scripted checks may miss. Regression testing checks whether changes harmed previously working behavior.
Ask what happens when a check fails. Can required reviews block a merge? Can failed checks prevent deployment? Who can approve an exception? Request a sample workflow showing required checks, approval roles, failure states, waiver authority, and retained records.
Code coverage and test counts do not prove that requirements were correct or important failure conditions were exercised. Look for a clear connection between risk and controls.
How Do You Decide What to Automate?
Automation works well for stable, repeatable checks that provide rapid feedback or regression protection, including unit checks, API contract checks, build verification, and repeatable regression paths.
Human judgment remains important for exploratory testing, usability, business-process validation, visual behavior, and accessibility checks that tools cannot conclusively evaluate. Security testing also combines automated checks with manual analysis.
Ask who maintains automated tests, how unreliable tests are repaired, and whether test code receives review. Request a sample report showing the software version, environment, test set, failures, skipped checks, and investigation links. A smaller trusted suite can be more useful than a large suite with routinely ignored failures.
How Do You Test Integrations, Data Flows, and Failure Conditions?
Integrations often fail where responsibility crosses boundaries. Relevant areas include Application Programming Interfaces, identity providers, payment or messaging services, data migrations, queues, batch jobs, file exchanges, and third-party platforms.
Ask how the vendor tests authentication failures, malformed data, timeouts, rate limits, duplicate events, retries, partial updates, unavailable dependencies, expired credentials, and recovery after interruption.
The vendor should explain which behaviors are tested through sandboxes, contract tests, emulators, test doubles, fault injection, or limited production validation. Any limitations should appear as residual risks.
For migrations, ask how record counts, relationships, balances, transformations, exceptions, and reconciliation will be checked. Request one documented failure scenario showing preconditions, the injected fault, expected behavior, monitoring evidence, recovery steps, and unresolved limitations.
Use These Vendor Questions as a Quality Assurance Checklist
Use the same question groups with every finalist. Capture the business reason, the vendor’s answer, supporting evidence, exclusions, buyer dependencies, and unresolved questions.
Quality Ownership and Governance
Quality needs named accountability and independent challenge.
Ask:
- Who is accountable for overall engagement quality?
- Who authors and approves the test strategy?
- Who performs developer testing?
- Who provides independent validation?
- Who assigns defect severity and remediation priority?
- Who can waive a failed quality gate?
- Who approves release?
- How do unresolved risks reach the buyer?
Request a responsibility assignment matrix, anonymized quality plan, proposed team chart, and defect-triage workflow. Verify that the named roles exist in the proposed staffing model and have enough authority to act.
Be cautious when one delivery lead can lower severity, waive failed checks, and declare acceptance without buyer visibility.
Test Scope and Risk Coverage
Test depth should follow business impact rather than raw test volume.
Ask:
- Which user journeys and failure modes receive the deepest testing?
- Which functional and non-functional test types are included?
- What is excluded, and what assumptions support each exclusion?
- How are edge cases and regression risks identified?
- What additional controls apply to a higher-risk release?
- Which buyer inputs or dependencies affect coverage?
Request a risk-based test strategy, a scope matrix mapping risks to test types, and sample scenarios for one critical workflow. Higher-risk releases may require specialist review, stronger traceability, recovery exercises, or staged rollout.
Defect Management and Transparency
Defect severity describes impact. Defect priority describes how urgently the issue should be addressed. The vendor should define both and explain who can change them.
Ask what each defect record must contain, who may close it, how disputes are resolved, and how fixes are verified. A useful record includes the affected version, environment, reproducible steps, expected and actual results, owner, linked tests, fix version, and retest outcome.
Request severity definitions, an anonymized defect record, a sample dashboard, and the proposed warranty, remediation, and support terms. Confirm how the vendor distinguishes a defect from a change request after acceptance.
Environments, Test Data, and Production Readiness
Not every project needs the same number of environments, but the team must be able to isolate development, evaluate integrated behavior, and verify a release candidate without uncontrolled interference.
Ask:
- Which environments will exist, and who maintains them?
- Which production characteristics will pre-production reproduce?
- How are configuration differences tracked and approved?
- How is test data generated, masked, protected, refreshed, and removed?
- Who validates deployment, rollback, backup, restoration, and monitoring?
- Which production differences remain unresolved?
Request an environment plan, test-data approach, release checklist, and deployment and rollback runbook. For systems with demanding continuity requirements, ask whether recovery procedures have been exercised rather than merely documented.
Reporting, Traceability, and Buyer Access
Ask which quality indicators will be reported and what decisions they support. Useful indicators include escaped-defect (defect-leakage) rate, defect density, test coverage tied to risk, defect reopen rate, and mean time to resolve (MTTR), each tied to a decision, not reported for volume. Confirm whether critical requirements can be traced to acceptance criteria, tests, results, defects, and exceptions. Ask whether the buyer will have direct or timely access to issue tracking, test results, build records, release notes, and approval history.
Request an anonymized status report, traceability sample, release-readiness summary, and known-issues register. A polished dashboard is not enough when the vendor cannot explain how its status was derived.
Define Release Acceptance Before Development Begins
Acceptance should be designed with the requirements, not improvised at the end. The vendor supplies evidence that technical and contractual conditions were met. The buyer confirms that agreed business outcomes and workflows are acceptable.
The acceptance review period should not begin until the defined deliverables, environment, documentation, and test evidence are available.
What Must Be True Before a Release Is Accepted?
The acceptance plan, your explicit definition of done, should address:
- Completion and results of agreed functional and non-functional testing
- Demonstrated status of critical workflows and integrations
- Unresolved defects assessed by severity, business impact, workaround, and residual risk
- Completion of applicable security, performance, accessibility, migration, backup, recovery, and operational checks
- Availability of release notes, known limitations, operating instructions, support contacts, and handoff materials
- Confirmation that the evaluated build is the build proposed for release
- Approval authority, rejection procedure, and conditional-acceptance process
Conditional acceptance should be explicit and time-bounded. It should record the exception, affected users or functions, compensating measures, remediation owner, target resolution, verification method, and consequence if remediation is not completed.
What Happens When Quality and Schedule Conflict?
Ask who can authorize release with known defects, incomplete testing, or a failed quality gate. Schedule pressure should not silently bypass controls.
Each exception should identify deferred scope, affected risks, compensating measures, cost and timeline consequences, and the person accepting the residual risk for the buyer. Also clarify whether remediation is included in the current price, warranty, or support arrangement.
Which Quality Commitments Belong in the Contract or Statement of Work?
Operational detail can live in a quality plan. Commitments that affect scope, payment, acceptance, access, and remediation should appear in the contract or Statement of Work, or SOW.
Topics for procurement and legal review include:
- Vendor, buyer, and shared testing responsibilities
- Included test types and material exclusions
- Required environments, integrations, accounts, data, and buyer dependencies
- Deliverable-specific acceptance criteria and review windows
- Evidence required before acceptance review begins
- Defect definitions, classification governance, remediation, warranty, service-level agreements and support (SLAs) for defect response/resolution, and support.
- Required quality artifacts and buyer access rights
- Release approval, rejection, and conditional-acceptance procedures
- Treatment of third-party services and production-only limitations
- Change control when requirements, architecture, dependencies, or risk assumptions change
A named methodology, certification, or tool is not a substitute for clear outcomes and evidence. Qualified counsel should review enforceability, remedies, data obligations, warranty language, intellectual-property access, and liability allocation.
Judge the Evidence Instead of the Vocabulary
Sales teams can describe modern practices fluently. Your task is to determine whether those practices form a connected operating system.
Use a simple evidence ladder. An unsupported claim is weakest. An explained process is better. A reviewable artifact is stronger. A live demonstration or connected example is stronger still. A contractual commitment makes the expectation enforceable, but it does not ensure performance.
Certifications and standards claims should be checked for scope, issuing body, current status, and relevance to the proposed team and engagement.
Strong Answers Show a Connected Operating System
A credible vendor can explain how:
- Business risks drive quality objectives and test depth
- Requirements become observable acceptance criteria and linked tests
- Code and configuration changes trigger reviews and automated checks
- Failed checks create visible defects, corrective work, retesting, and regression protection
- Test results and operational checks feed release readiness
- Known limitations become residual risks accepted by a named authority
Ask the sales lead, delivery lead, engineer, and quality representative to walk through the same example. Their wording may differ, but ownership, workflow, evidence, and exception rules should align.
Request a connected evidence set, such as a traceability slice, change-review example, test result, defect record, and release summary.
Warning Signs That Deserve Follow-Up
Watch for:
- Quality explained only through broad claims, certifications, or tool names
- Testing concentrated at the end of development
- No named owner for the test strategy, defect disputes, exceptions, or release approval
- Universal automation or code-coverage claims without a risk rationale
- Refusal to provide any meaningful anonymized artifact or demonstration
- Unexplained exclusions for integrations, migrations, background processing, or failure conditions
- Defect severity definitions controlled solely by the delivery team
- Acceptance based on subjective satisfaction or an undefined “production-ready” label
- Essential quality activities presented as optional add-ons after promising a deployable system
One warning sign may be explainable. Several connected signs suggest that quality is being managed as a sales narrative rather than a controlled delivery discipline.
When a Deeper Technical Review Is Necessary
Escalate to a qualified technical reviewer when the software is security-sensitive, highly integrated, operationally critical, regulated, difficult to reverse, dependent on unfamiliar architecture, or capable of causing material harm through failure.
This checklist cannot establish that the architecture is sound, the code is maintainable, security controls resist attack, or performance will hold at production scale.
Turn the Answers Into a Comparable Vendor Record
Create one record per vendor with fields for the question, answer, named owner, evidence supplied, evidence date, exclusions, assumptions, buyer dependencies, risk implication, follow-up owner, and contracting action.
Use four ratings:
- Clear: The answer is coherent, specific, and supported by relevant evidence.
- Partial: The process may be credible, but ownership, scope, or evidence is incomplete.
- Unsupported: The claim lacks usable evidence or conflicts with other answers.
- Not applicable: The item does not apply, with a documented rationale.
Do not turn the ratings into false numerical precision. One unsupported answer about release authority or sensitive-data testing may matter more than several clear answers about routine reporting.
Assign an internal owner to close every material uncertainty. Transfer accepted commitments into the SOW, acceptance plan, quality plan, and governance cadence. Record any risk the buyer knowingly accepts and the authority approving it.
Strengthen Quality Before Your Next Software Release
Vendor checklists can reveal weak answers, but they cannot independently verify whether a product is secure, reliable, usable, or ready for release. Independent QA provides additional evidence before your organization accepts software or exposes it to customers and operational teams.
Explore our software testing and QA services to assess product quality, test critical workflows, evaluate release risks, and support evidence-based acceptance decisions.
Planning a new platform or application? Learn how our custom software development services integrate quality practices across discovery, engineering, testing, deployment, and ongoing improvement.
Related Blogs
