When you’re considering investing in an early stage software startup, it’s difficult to predict its future performance — especially if the market is already full of more established competitors. However, one leading indicator of success is product development performance.
That’s why in this article, we’ll explain why and how investors should use software development metrics as part of their technical due diligence.
Software development performance is a leading indicator of business performance
In 2015, the Sales Engagement Platform (SEP) market was dominated by well-funded companies like Xant, ClearSlide, and Yesware. Meanwhile, a startup called Outreach was only just getting off the ground. Fast-forward to 2022, and Outreach has raised $200 million at a $4.4 billion valuation — and is seemingly the furthest ahead in the market. This development would’ve been difficult to predict back in 2015.
While there are certainly many reasons for the success of Outreach, I would argue that their ability to ship products and features faster and more reliably than their competitors has played a large part.
Arguably, the earlier stage a company is, the more value investors will place on the founding team and their perceived ability to execute. In later stages, investment decisions are typically better informed as more quantitative data on past and projected outcomes like financial performance becomes available.
Before lagging indicators like revenue growth and profitability can be evaluated, investors can absorb signals from leading indicators. One such group of metrics is product development performance, which can be used as a signal of the team’s ability to execute. This, of course, doesn’t mean that you should overlook outcomes. But instead, when there aren't many outcomes to analyze, you should look at the indicators that positively affect those outcomes.
For example, if an established competitor is shipping new features faster than the challenger, it’s unlikely that the smaller company will ever catch up. However, if the established competitor is enjoying some level of commercial success regardless of their poor engineering performance, they’ll likely start losing market share to faster-moving competitors.
This link between software development performance and business performance is well documented in a recent study by McKinsey.
Wrapping your head around software development productivity metrics
While technical due diligence is a large part of most investment evaluation processes, it rarely includes assessing product development performance, i.e. how fast and reliably the team is able to deliver new features. This is not all that surprising, since the research and best practices on how to measure software development productivity has only just started emerging.
Historically, measuring engineering productivity has been a difficult subject. The first attempts included harmful practices like measuring lines of code and stack-ranking developers based on vanity metrics. As you can imagine, these practices caused more harm than good and drove many engineering leaders, managers, and teams to give up on productivity measurement entirely.
However, ever since the DevOps Research Association (DORA) published their first annual State of DevOps report in 2014, the industry has started regaining its faith in measuring engineering productivity. In fact, many startup CTOs and VPs of Engineering are currently looking into tracking the four DORA metrics to help them gain visibility into how their engineering organization is faring against industry benchmarks.
And when the 2018 book Accelerate came out from the same authors, it finally proved that there is a strong correlation between four software delivery metrics and strong business performance. These metrics have become known as the DORA metrics, and they include change lead time (or cycle time for issues), deployment frequency, change failure rate, and mean time to recovery.
Nicole Forsgren, one of the authors of Accelerate, has recently published a whitepaper that introduces an even more holistic approach to measuring engineering productivity, the SPACE framework. The new framework suggests measuring engineering productivity across five dimensions:
- Satisfaction and well-being
- Performance
- Activity
- Communication and collaboration
- Efficiency and flow
For the first time, this framework incorporates psychological and social aspects of engineering productivity, such as developer satisfaction, well-being, communication, and collaboration.
If you’re interested in learning more about the topic, read this extensive guide about measuring software development productivity, followed by this practical article on getting started with the SPACE framework.
How to assess engineering productivity — and invest in software companies with higher confidence
With more evidence supporting the fact that software excellence fuels business performance and more research behind the metrics that matter, now is a great time to start evaluating software development productivity as part of technical due diligence. This way, you’ll be able to increase your confidence in a startup’s ability to execute and avoid passing up on the opportunity to invest in the next Outreach.
Here’s our recommendation on how to assess software development performance as part of technical due diligence:
1. Evaluate the engineering leadership’s approach to measuring software development performance
As clichéd as it may sound, it’s hard to improve what you don’t measure. Conversely, all metrics — even the wrong ones — drive behavior. That’s why it’s important to first understand whether the engineering leadership is already measuring software development performance in some way. And if they are, what metrics they’re tracking.
If, for example, the engineering leadership is using development activity metrics like lines of code to stack-rank their engineers, it would be good to understand what they’re thinking they’ll achieve with such metrics. If, instead, they’re interested in using engineering metrics to improve deployment frequency, code quality, and/or developer well-being, it’s an indication that they’re at least attempting to build a healthy engineering culture.
2. Find out how the company is doing in terms of the four DORA metrics
Even if the company you’re evaluating isn’t yet measuring research-backed engineering metrics, you can ask them to rate themselves on the four DORA metrics. In SPACE framework’s terms, the four DORA metrics give you an idea of the engineering organization's “performance”, “activity”, and “efficiency and flow”.
The DORA research group has published industry reference values, which help companies benchmark against high performers. Elite teams get code from commit to production in hours, deploy multiple times a day, restore services in a matter of hours, and have a change failure rate of 0-15%. If the four DORA metrics indicate low or medium performance, it’s a sign of possible issues in the future.
Knowing where the company stands on these metrics gives you a good baseline on how productive the engineering organization is.
3. Include a qualitative assessment of team health
Going beyond the four DORA metrics, you might also want to assess the team’s “satisfaction and well-being” and “communication and collaboration”, which are the remaining two dimensions of the SPACE framework.
According to the authors of Accelerate, a good way to measure these two dimensions is to survey the company on how they rank on the Westrum organizational culture. By asking questions like the ones below, you’ll get a pretty good idea of where the organization stands in terms of psychological safety, employee well-being, and collaboration:
- On my team, information is actively sought.
- Messengers are not punished when they deliver news of failures or other bad news.
- On my team, responsibilities are shared.
- On my team, cross-functional collaboration is encouraged and rewarded.
- On my team, failure causes inquiry.
- On my team, new ideas are welcomed.
By combining the four DORA metrics with the Westrum organizational culture survey, you’ll be able to not only reliably assess the current performance of the engineering organization but also predict its future success.
Scoring high on all the five dimensions of the SPACE framework is a signal of strong engineering performance. The scores will also allow you to identify any potential gaps the organization may want to bridge going forward.
Conclusion
Thanks to recent research, we can assert with high confidence that teams that do well in terms of DORA metrics and Westrum culture have a greater probability of successfully scaling their operations and attracting talent. That’s why we would strongly recommend that early stage investors also include product development metrics in their technical due diligence processes.
