Kusnetzky Group often receives messages, press releases and telephone calls from vendor PR professionals that present amazing test results and huge savings organizations would receive through the use of a specific vendor’s product or service.
Kusnetzky Group, as a general rule, doesn’t comment publicly. We thought it would be informative to present why we typically don’t comment.
A recent example
In a recent case, a supplier published a study that included, in their words “a new cost analysis of hybrid cloud hosting options.” The supplier used its technology to analyze, in their words again, “almost 1,000 real workloads to determine the cost and impact of Azure, AWS and SoftLayer.”
The message went on to point out that “The study focused on the analysis of three key areas.” Here are the areas the study purports to examine
- The impact of properly sizing cloud workloads
- The impact of different cloud models on hosting cost
- The impact of workload patterns and personalities on hosting decisions
Why we wouldn’t usually comment
The Kusnetzky Group typically doesn’t comment on research performed by other research firms or suppliers. Primarily, Kusnetzky Group isn’t in a position to verify the following things:
- How the research was conducted
- What was included and why
- What was excluded and why
- What assumptions were made during the analysis and why
This, of course, means that it wouldn’t be prudent for us to refer to that research as being fact or use it
The vendor presented the results of their cost analysis in a short, interesting video as well as in a press release. The results cited appeared quite attractive but, unfortunately, many questions about the selected configurations, the overall performance and several other factors emerged.
These questions were not answered by the video nor was the information found in the press release either. So, neither was convincing. The results appear not to have been audited either. In the end, it relies on the viewer accepting the vendor’s word as fact.
If we were advising a client, we would be forced to say that there is no way to determine how the testing was done, if the testing mentioned had any relationship whatsoever to the client’s workloads and that, in the end, the study could not be viewed as being representative of anything in particular.
Kusnetzky Group has a great deal of respect for the products and technology offered by the vendor in question. While we suspect that they would be in an excellent position to prove the results mentioned in their study for potential and actual customers, the video and press release weren’t sufficient for us to present the results as if they were indisputable fact.