How to Evaluate Engineering Quality in a Custom Software Vendor Without Being Technical

Arbisoft Editorial TeamPosted on August 10, 2026

16-17 Min Read Time

You can't inspect a software vendor’s code. Fine. You can inspect the machine that produces the code: who reviews what, when tests run, how a release actually happens, who gets woken up when production falls over, and whether some other team could pick this thing up in eighteen months without starting from scratch.

One rule does most of the work. Prefer what you can see, count, or check over adjectives.

"Our engineering is high quality" is not information. A vendor walking you through how one real feature went from a line in a ticket to running code in front of paying users is information, and you can hold it up against the next vendor's version of the same walkthrough.

Everything below is a way of getting that second thing instead of the first.

Translate Engineering Terms Into Business Consequences

Engineering terminology matters only when it connects to a business risk you care about.

A non-technical software vendor evaluation should stay at this level.

There is little value in memorizing terms such as CI/CD, unit testing, observability, or infrastructure as code. The useful question is what those practices change in delivery and how the vendor can prove they are actually used.

Ask Questions That Grade Themselves

Broad questions usually produce broad assurances.

“Do you write automated tests?” is easy to answer with yes.

“How does a change get tested before it reaches production, and can you show me an example?” is much harder to bluff.

The same principle applies throughout software vendor due diligence.

Instead of asking:

“Do you review code?”

Ask:

“Can a developer release a change without another engineer reviewing it? What prevents that from happening?”

Instead of:

“Do you deploy frequently?”

Ask:

“How many times did a comparable client project deploy to production last month?”

There is no universal correct deployment frequency. A specific answer shows that the vendor knows its actual delivery history. “Frequently” gives you much less evidence.

Instead of:

“Do you have a quality process?”

Ask:

“Can you show me your definition of done for a real work item?”

Good questions tend to produce an example, a number, a named owner, a decision rule, or an artifact.

That gives a non-technical buyer something they can compare across vendors.

Separate Claims From Evidence

Use three levels when evaluating an engineering claim.

Claim:
“We review every important change.”

Evidence:
The vendor shows a redacted pull request, review record, or equivalent workflow showing another engineer checking the change.

Pattern:
Several examples from different dates show reviews happening consistently, with comments addressed and required checks completed.

The same test works for testing, releases, documentation, security practices, and incident handling.

One polished example proves that the vendor can produce one polished example. Patterns across ordinary work are much more useful.

Confidentiality is a reasonable limitation. A vendor can still provide sanitized screenshots, anonymized project artifacts, internal examples, open-source work, or a recreated demonstration of the same process.

Ask Them to Trace One Real Change From Request to Release

One of the strongest ways to evaluate engineering quality without reading code is to ask the vendor to reconstruct how a completed piece of work reached production.

Choose a feature, bug fix, or meaningful change.

Then follow it through the workflow:

• Request: What did the client or product team ask for?
• Definition: Where were the requirements or acceptance criteria recorded?
• Implementation: What record represents the engineering work?
• Review: Who checked the change, and where were concerns recorded?
• Testing: What automated or human checks ran?
• Release: Who decided the change was ready?
• Deployment: How can the team confirm that the reviewed version is the version that reached users?

You do not need to understand the code inside a pull request. You are checking whether the work leaves a coherent trail and whether meaningful controls exist between an idea and production.

A credible example may include changed requirements, review comments, failed tests, or follow-up work. Real software delivery is rarely spotless. What matters is whether the vendor can explain what happened and why.


If you’re comparing vendors on how they move work from requirements to production, explore our custom software development services across discovery, UX design, iterative development, integration, and security.

Pay Attention to How the Vendor Handles Pushback

Vendor behavior during the buying process can reveal as much as a methodology slide.

A strong engineering team may tell you:

• a feature should wait until a later phase;
• the proposed approach creates unnecessary risk;
• reliable pricing requires inspecting your data first;
• an assumption needs to be tested before development starts;
• there is a cheaper way to solve part of the problem.

Constant agreement deserves scrutiny.

A custom software vendor has access to experience that the buyer does not. Part of its job is to use that experience when a requested feature, deadline, or technical choice creates avoidable risk.

During vendor discussions, notice whether the team explains trade-offs clearly. A credible vendor should also be comfortable admitting uncertainty where uncertainty genuinely exists.

Find Out Who Will Actually Build Your Software

A strong company presentation does not tell you who will work on your account.

Ask for the expected team before making a final decision.

Useful questions include:

• Who will be our technical lead?
• Are these the people likely to work on the project?
• Which roles are employees and which may be subcontracted?
• How long have the key people been with the company?
• What happens if the lead engineer leaves or moves to another project?
• How is knowledge shared so that one developer does not become indispensable?

When practical, speak directly with an engineer who is likely to work on the engagement.

The purpose is to understand how engineers communicate without every answer passing through a sales or account-management layer.

Listen for plain explanations. Good engineers do not need to make complex work sound simple, but they should be able to explain why a decision matters to your business.

Ask What Happens When Something Goes Wrong

Failure handling is one of the most revealing engineering quality checks.

Ask the vendor to describe a real production problem.

Then follow the story:

How was the issue detected?
Strong teams often have monitoring and alerts rather than depending entirely on users to discover serious failures.

Who took ownership?
Responsibility should become clearer when a problem is serious.

How was the impact contained?
The team should be able to explain rollback, recovery, or another way of limiting damage.

What happened afterward?
Look for evidence that significant failures lead to corrective work, such as better testing, improved monitoring, process changes, or architecture improvements.

Be cautious if the answer is some variation of:

“We do not really have production incidents.”

Software fails. The quality signal is how a team detects problems, responds under pressure, communicates with the client, and learns afterward.

Test for Vendor Lock-In Before Signing

For many buyers, engineering quality becomes a commercial problem when changing vendors is difficult.

Ask:

“If we stopped working together four months into the project, what exactly would we have?”

The answer should cover the relevant assets for your project, which may include:

• source code and repository access;
• intellectual property rights;
• cloud and infrastructure accounts;
• credentials;
• database and deployment information;
• technical documentation;
• architecture decisions;
• instructions another team would need to operate or continue the product.

Repository access from the beginning is stronger than a promise to transfer the code at the end.

Also ask to see an anonymized handover document from a previous engagement.

The goal is practical portability. Another capable development company should have a realistic path to understand and continue the software.

Any significant hesitation around ownership deserves investigation before a contract is signed.

Use References for Due Diligence, Rather Than Testimonials

A logo on a case-study page tells you very little about what happened during the relationship.

Ask to speak with a client who has worked with the vendor for a meaningful period. Where possible, also request a reference from an engagement that has already ended.

Questions such as these produce better information than “Were you happy with them?”:

• What surprised you after the project started?
• What happened during your first serious disagreement?
• How did the final cost compare with the original expectation?
• What broke after launch?
• How did the vendor respond when something went wrong?
• How easy would it be for another development team to take over the product?

References are especially valuable for testing claims that are difficult to verify in a sales process, such as communication under pressure, cost control, long-term maintainability, and support after launch.

Read the Commercial Terms as Another Quality Signal

Engineering promises eventually meet the contract.

Review how the agreement handles:

• ownership of code and intellectual property;
• defect responsibility;
• any warranty or defect-resolution period;
• change requests;
• post-launch maintenance;
• handover obligations;
• access to repositories and infrastructure.

Ask the vendor to explain what happens when a defect is clearly caused by delivered work and who pays to correct it.

For maintenance and support, ask for the commercial model and examples from comparable projects. Avoid treating one universal percentage as a benchmark. Products with different business criticality, infrastructure needs, security requirements, and release frequency can have very different ongoing costs.

A company confident in its delivery process should be able to explain these responsibilities without hiding behind vague contract language.

Reduce the Risk With a Small Paid Engagement

Sales conversations are predictions. A short engagement gives you observed behavior.

For a significant custom software investment, consider starting with a paid discovery, technical assessment, prototype, or small pilot with:

• a clearly defined deliverable;
• named participants;
• access to the resulting work;
• a clear end point;
• the ability to continue with another provider.

A two-to-four-week discovery period can be enough for many projects to reveal how a vendor asks questions, documents decisions, handles uncertainty, communicates risk, and produces work.

This does not eliminate project risk. It replaces part of the prediction with direct evidence.

Bring in Independent Technical Judgment When the Stakes Justify It

There is also a point where non-technical evaluation reaches its limit.

For a large or business-critical investment, hiring an independent senior engineer, fractional CTO, or software architect for a limited review can be inexpensive compared with discovering serious technical problems after a large portion of the budget has been spent.

A technical reviewer can examine areas a non-technical buyer cannot reasonably assess, including:

• architecture suitability;
• source-code maintainability;
• security implementation;
• dependency risk;
• data design;
• performance and scalability;
• accumulated technical debt.

The reviewer can also sit in on finalist presentations or inspect the output from a paid discovery engagement.

For security-sensitive systems, frameworks such as the NIST Secure Software Development Framework and OWASP Application Security Verification Standard can give specialists and suppliers a common vocabulary. Buyers do not need to study those frameworks themselves.

Hiring independent technical help should be viewed as due diligence rather than an admission that the buying team has failed.

Use This Non-Technical Software Vendor Scorecard

Use substantially the same questions with each finalist.

Add Not Demonstrated as a possible result.

Missing evidence does not automatically mean poor engineering. It means uncertainty remains. On a large software investment, unresolved uncertainty has a cost of its own.

Software Development Company Red Flags to Watch For

A single warning sign should lead to another question rather than an instant rejection. Several reinforcing signals deserve more concern.

Look more closely when:

• technical jargon repeatedly replaces specific explanations;
• every requested feature receives immediate agreement;
• the vendor gives precise estimates before understanding the underlying system or data;
• the sales team cannot identify who will actually build the product;
• engineers are kept away from buyer conversations;
• testing and code review cannot be demonstrated in any form;
• production failures are described as extremely rare or nonexistent;
• documentation is treated as optional;
• repository or infrastructure access remains under vendor control;
• another development company would struggle to take over the software;
• the vendor cannot explain what happens when a reviewer, test, or security check blocks a release.

The pattern matters more than any single item.

Frequently Asked Questions

How can I tell whether a software company writes good code if I cannot read code?

Evaluate the controls and evidence around the code. Look for repeatable review, testing, release discipline, traceability, clear ownership, documentation, and credible handling of failures. For high-risk investments, have an independent technical reviewer inspect the code itself.

What evidence should I ask a software development company to show?

Useful examples include a redacted work item, code-review history, automated test or build result, release record, incident retrospective, architecture documentation, risk report, and handover document.

The evidence should connect to real delivery rather than exist only as a process template.

How much automated testing should a good vendor have?

There is no useful universal percentage. Appropriate testing depends on the product and the consequences of failure. Ask which parts of your system the vendor expects to automate, what risks those tests cover, when the tests run, and what happens when one fails.

Should a non-technical buyer hire a technical consultant before choosing a vendor?

For a large, security-sensitive, business-critical, or technically complex project, independent technical review can be sensible due diligence. A limited engagement may be enough to review finalist proposals, discovery output, architecture, or a pilot before a much larger commitment is made.

What should I own when a custom software engagement ends?

The answer depends on the contract and project structure, but buyers commonly need clear rights and practical access to source code, relevant repositories, infrastructure, credentials, documentation, and the information required for another qualified team to continue the work.

What You Are Really Trying to Prove

A non-technical buyer does not need to certify that every engineering decision is correct.

The goal is to build enough evidence that quality comes from repeatable practices rather than individual heroics or sales assurances.

Ask vendors to show ordinary work. Look for specific answers. Observe how they handle disagreement and uncertainty. Protect ownership from the start. Verify claims through references, contracts, and real delivery evidence. Use a small paid engagement or independent technical reviewer when the size of the decision warrants it.

Before selecting a finalist, ask every vendor to demonstrate two things:

One piece of real work from request through production.

One example of what happened when delivery did not go according to plan.

Those two walkthroughs can reveal more about engineering quality than a long list of technical terms.

Use the same standard on any vendor you are considering, including us.