A common way to justify an AI tool is to cite the return its vendor has already published. Such a study often has a credible analyst name on the cover, a three-year model, a net present value and an ROI percentage large enough to carry a budget slide. It goes into the business case, and by the time a board sees it, the figure can easily have lost its footnotes and become a fact about the company.
That practice is understandable. Building your own evidence is slow, and the vendor's study is free. The problem is not that these studies are dishonest. The problem is structural: the party being evaluated controls the evaluation, and no amount of care inside the study removes that.
The accepted practice: measure the tool with its vendor's evidence
Vendors publish return figures for their own AI products. Some are internal research. Some are commissioned from analyst and consulting firms. They share one property: the company whose product is being assessed is the company that paid for, scoped or reviewed the assessment.
The mechanism: who holds the instrument
Three levers, one hand
The instrument is the question being asked. A study commissioned to "examine the potential return on investment (ROI) enterprises may realize" by deploying a product is asking how much value the product creates. It is not asking whether the same money would have done more elsewhere, or whether the change would have happened anyway.
The sample is who gets asked. If the sponsor supplies the names, the sponsor decides which customers are in the room. Any sponsor asked for customer names has every reason to pick satisfied ones. No bad faith is required.
The definition of success is what counts as a benefit. Time saved, revenue influenced, cost avoided: each is a modeling choice, and each choice can be reviewed by the party with the most riding on the answer.
What the Copilot ROI study discloses
The clearest public example is one that discloses its terms in full. In March 2025 Forrester Consulting published "The Total Economic Impact of Microsoft 365 Copilot," a study commissioned by Microsoft. Its headline is a three-year, risk-adjusted 116% ROI and a $19.7 million net present value.
Read the method and the disclosures and the shape of the evidence is plain. Forrester interviewed 16 decision-makers across 12 organizations and surveyed 367 respondents with experience using Copilot. It then combined their experiences into a single composite organization, a global entity with $6.25 billion in revenue and 25,000 employees. The 116% describes that composite, which is not a real company.
The disclosures box states that Microsoft reviewed and provided feedback to Forrester, and that Microsoft provided the customer names for the interviews but did not participate in them. It also states that Forrester maintains editorial control and does not accept changes that contradict its findings, and that the survey was fielded double-blind through a third-party partner. The study adds that interviews took place "at an early stage when agents were newly implemented."
Credit where it is due: all of this is disclosed, in the study itself. Forrester's own TEI methodology page goes further. It says clients cannot purchase favorable results, and that all TEIs are marked as commissioned and "cannot be used in Forrester's syndicated content." The publisher keeps this work out of its syndicated content. That is a reasonable policy, and it is worth knowing before the number travels.
The outside authority: what sponsorship does where it has been measured
The argument so far is structural. It would be fair to ask whether the structure actually moves results. We found no study that answers that for software. Medicine has answered it, repeatedly, and at scale.
Medicine measured it
A 2017 Cochrane review by Lundh, Lexchin, Mintzes, Schroll and Bero pooled the research on industry sponsorship of drug and device studies. Industry-sponsored studies more often had favorable efficacy results, with a risk ratio of 1.27 (95% CI 1.17 to 1.37) across 25 papers, and more often had favorable conclusions, risk ratio 1.34 (95% CI 1.19 to 1.51) across 29 papers. The authors rated the first finding moderate-quality evidence and the second low-quality.
An earlier systematic review in JAMA by Bekelman, Li and Gross (2003) found the same direction. Eight articles, which together evaluated 1,140 original studies, showed a pooled odds ratio of 3.60 (95% CI 2.63 to 4.91) for pro-industry conclusions among industry-sponsored research.
Why a better-built study does not fix it
The most useful line in the Cochrane review is not the headline ratio. It is what the authors found when they checked study quality. They found no difference between industry and non-industry studies in risk of bias from sequence generation, allocation concealment, follow-up and selective outcome reporting, and on blinding industry studies more often scored better, risk ratio 1.25 across 13 papers. Conventional quality checks did not account for the favorable-results gap. The authors conclude that their analyses "suggest the existence of an industry bias that cannot be explained by standard 'Risk of bias' assessments."
They also found that in sponsored studies, conclusions agreed with results less often, risk ratio 0.83 across six papers. That is the quieter failure. The data and the summary drift apart, and the summary is the part that travels.
In medicine, sponsored studies that scored as well or better on standard quality checks still reported favorable results more often. Study quality, as conventionally measured, did not explain the gap. That is the lesson worth carrying over: a well-built study is not the same thing as an independent one.
Self-reported benefit has its own gap
Commissioned ROI models such as this one rest heavily on interviews and surveys, that is, on what customers say the product did. Independent work on self-reports urges caution. METR, an independent research nonprofit, surveyed 349 technical workers between February and April 2026 and recorded a median self-reported value gain of 1.4x to 2x from AI tools, and a median self-reported speed gain of 3x. METR's own caution is the point: its earlier randomized study, run in 2025 on a different and much smaller group of experienced developers, found people overestimated AI's effect on their task time by 40 percentage points on average.
A model built on interviews inherits whatever gap exists between what people believe the tool did and what it did.
The limits of borrowing this evidence
Three limits, stated plainly. The Cochrane and JAMA reviews cover biomedical research, not software. METR's 2026 survey is a convenience sample of technical workers with low response rates, and METR says its results plausibly suffer from significant selection bias. The medical evidence shows the mechanism is real where it has been measured. It does not tell you the size of the effect in a Copilot study.
How to read a vendor AI ROI claim
Before a vendor figure goes into a business case, answer seven questions. The third column shows how the Copilot TEI study answers them from its own text.
| Question to ask | Where to look | What the Copilot TEI study discloses |
|---|---|---|
| Who commissioned it? | Cover and disclosures | Microsoft |
| Who chose the participants? | Disclosures | Microsoft provided the customer names |
| Who reviewed the study? | Disclosures | Microsoft reviewed and gave feedback; Forrester keeps editorial control |
| What organization does the result describe? | Methodology | A composite: $6.25 billion revenue, 25,000 employees |
| How large is the evidence base? | Methodology | 16 decision-makers across 12 organizations, plus 367 survey respondents |
| When was the evidence gathered? | Methodology | At an early stage, when agents were newly implemented |
| Does the publisher use it in its syndicated content? | Publisher methodology page | No. Commissioned TEIs cannot be used in syndicated content |
None of these answers makes the study wrong. Together they make it a framework rather than a finding. Forrester says as much: the study "is not meant to be used as a competitive analysis," and Forrester "strongly advises that readers use their own estimates within the framework provided."
Take the model, replace its assumptions with your own observed data, and see what survives.
What Levos does instead
The fix is not a better vendor study. It is a measurement taken by a party that has no product in the comparison.
Levos does not sell Copilot or any of the AI tools it compares. AI Impact reads adoption depth and frequency for every AI tool from Microsoft 365, Google Workspace, GitHub, Salesforce and the AI tools themselves, so Copilot sits beside the other tools your people use rather than alone in its own console.
The comparison is your own organization, not a composite. Our measurement methodology states it plainly: we compare adopting teams to non-adopting teams, controlling for tenure, role, and tool stack. We disclose the limits of this approach openly. We do not claim full attribution. Every score carries a confidence rating, and every signal cites its source.
This is the same argument we made about seat counts as an adoption metric: the number that is easiest to obtain is rarely the one that answers the board's question. It is also the practical half of measuring AI workforce ROI without vendor self-grading. That post set out the principle. This one shows how to test a specific claim against it.
The same test applies to us
A vendor-neutral measurement company has a commercial interest in readers distrusting vendor studies. That is true, and it is the reason this post quotes the disclosures wherever it can. Every figure above can be checked against its source in a few minutes.
Hold Levos to the same seven questions. Ask who supplies our sample (you do, from your own systems), who defines success (the outcomes your business already tracks), and what we refuse to claim (full attribution). Levos has no customer results to cite yet, and we would rather say so than publish a composite.
Frequently asked questions
Is a vendor-commissioned AI ROI study reliable?
It is reliable as a description of what its interviewees, named by the sponsor, and a third-party survey sample reported, modeled onto a composite organization. It is not evidence about your organization. The Forrester TEI study of Microsoft 365 Copilot, commissioned by Microsoft and published in March 2025, discloses that Microsoft reviewed and provided feedback on the study and provided the customer names for the interviews, while Forrester states it maintains editorial control. Forrester itself advises readers to use their own estimates within the framework the study provides.
What does the Microsoft 365 Copilot TEI study actually measure?
Forrester Consulting interviewed 16 decision-makers across 12 organizations and surveyed 367 respondents with experience using Microsoft 365 Copilot, then aggregated their experiences into a single composite organization with $6.25 billion in annual revenue and 25,000 employees. The 116% ROI and $19.7 million net present value describe that composite. The study notes the interviews took place at an early stage, when agents were newly implemented.
Does sponsorship change research results?
Where it has been measured, yes. A 2017 Cochrane review by Lundh and colleagues found industry-sponsored drug and device studies more often had favorable efficacy results, risk ratio 1.27 (95% CI 1.17 to 1.37) across 25 papers, and suggested an industry bias that standard risk-of-bias assessments cannot explain. A 2003 JAMA review by Bekelman, Li and Gross pooled 8 articles evaluating 1,140 original studies and found an odds ratio of 3.60 for pro-industry conclusions. Both are medical research. We found no equivalent quantified study for software ROI research.
How should a CFO use a vendor ROI study?
As a model to populate, not as a result to cite. Check who commissioned it, who chose the participants, who reviewed the study, what organization the result describes, and when the evidence was gathered. Then measure your own rollout against comparable teams that have not adopted, from behavior in the tools where work happens, with a confidence level and stated limits attached.
Levos is accepting design partner applications from US organizations of 150 employees or more with an active AI rollout.
If a vendor study is currently carrying your AI business case, Request a Demo and we will show you what a comparison drawn from your own teams contains.
Change how work gets measured.
