Impact metrics are key to realizing DPI’s full potential. At the Digital Donors Exchange, our expert partners explain why.

As digital public infrastructure (DPI) gains prominence, stakeholders across the ecosystem are working to unpack the current practices, demand, and frameworks around measuring DPI’s impact on people. As part of this mission, we organized the first Digital Donor’s Exchange (DDX) on the subject in early April, bringing together key funders and other orchestrators in the field to consider collective approaches to impact metrics. We covered an array of topics, including 1) how measurement needs evolve as DPI moves from the design phase to deployment and scaling phases, 2) different forms of measurement, and 3) tradeoffs in qualitative and quantitative methods. 

To kickstart the discussion, our speakers set the stage with insightful opening remarks.  

1. Jonathan Dolan from the Digital Impact Alliance shared the motivations for measuring impact for DPI and discussed why now is a good time to focus on the subject. He also talked about the current state of impact measurements and shared DIAL’s plans for investigating this topic further. 
2. David Porteus from Integral Solutions highlighted the need to situate the ‘measuring DPI’ conversation within a spectrum without getting distracted by the need to concretely define it. He also shared his learnings and experiences from the evolution of measurements in the financial inclusion space. 
3. Adele Barzalay from the World Bank stressed the need to define the scope and boundaries of DPI before diving in to measure its impact. In addition, she explained that before defining the objectives of a DPI measurement framework, we must identify its intended audience, since governments and donors often perceive impact measurements differently. 
4. Julia Clark from the World Bank unpacked the different tiers of measurement indicators for the forum. She further explained that impact can only be estimated through a combination of qualitative and quantitative approaches. 

With that, let’s dive into some of the key themes that emerged from the discussion. 

There are many discussions around the definitions of DPI – but minimal consensus. Yet, such consensus isn’t necessarily a prerequisite to effectively measure DPI’s impact. Rather than focusing on a single, general definition, we should seek to understand DPI’s value through specific use-cases. To situate DPI within this framework, we must define it as a spectrum – while striking a balance between being too precise and too vague. Specifically, our speakers framed the DPI spectrum in three dimensions: 

1. Input dimension – indicators highlighting design choices 
2. Output dimension – indicators highlighting scale 
3. Outcome dimension – indicators highlighting impact 

While a few participants argued the need to first build consensus around the definition of DPI before attempting to measure its impact, others contended that the definitions would evolve as the DPI ecosystem matures. And rather than aligning on a specific definition right now, we can evaluate a spectrum of identifying factors to determine how a particular DPI tool or solution may be demonstrating impact.

A few participants shared learnings from their experiences in the evolution of measurement frameworks in the financial inclusion space. Initially, actors across the sector focused primarily on the inputs stage – establishing and scaling digital payment systems. But, slowly, demand for effective metrics began to grow, as the financial inclusion community sought to understand the impact of their innovations. This spurred a series of national impact measurement initiatives.  

When discussions about a global measurement standard started, the financial inclusion sector was prepared, thanks to the experiments mentioned above. These deliberations evolved into the Findex survey. It took about two years to finalize the design, and it has been evolving ever since. Findex hasn’t crowded out national initiatives. Deeper dives like RCTs over a long period help to understand the impact of financial inclusion in people’s lives.  

Our forum participants agreed on the need to draw on such important lessons and, more importantly, come together to share more about their initiatives and learnings around measuring impact. 

When thinking beyond adoption and access metrics, it is key to identify primary and secondary audiences. Who are the end users and stakeholders that we are hoping the DPI initiative will benefit? For example – if the audience is composed of residents of a specific region/country, efficiency mightn’t be relevant to them, but for an investor audience, effectiveness and efficiency for government might be important indicators. 

In addition, when creating indicators, we must determine measurement objectives and assess any differences that donors and client countries may have in their definitions of impact. While both parties might talk similarly about the key elements of DPI’s impact, such as data protection, cross border data sharing, inclusion, etc., they might be framed in different ways. 

A mix of qualitative and quantitative methods is needed to understand DPI’s impact, but use and sequencing of either method depends on the hierarchy of tiers of indicators across the DPI spectrum as defined above. 

One of our discussants broke down the different tiers of measurement indicators for DPI. 

1. At the first tier, the attempt is to understand DPI’s existence and relative maturity within different countries. This is mostly helpful when we are trying to develop a global view of DPI.  
2. The second tier is understanding the acceptance and use of DPI – both on the supply (funder/implementer) side and on the demand (country) side. This could be accomplished in the form of surveys – or adding questions or modules to existing global measurement frameworks like Findex.  
3. The third tier is impact. One of our lead discussants pointed out that impact can only be estimated over time through methods like randomized control trials. These quantitative methods should be coupled with qualitative approaches (and perception surveys) to understand the impact of DPI on people. 

When it comes to creating useful, effective DPI impact measurements, the digital development community must consider a number of different challenges. Our participants highlighted a few of the most pertinent: 

1. There was a brief discussion on the tradeoffs of using qualitative approaches for measuring DPI’s impact. While qualitative approaches can be used to estimate impact, they may not provide a global or representative understanding. As our participants explained, the challenge is picking the right sample sets and the right questions. They further agreed that there are many opportunities to improve and deliberate further on qualitative methods. 
2. Our speakers also highlighted the need to capture and understand the spillover effects of the entire DPI stack in a country – not just its components. Doing so will help foster a whole-of-DPI approach to measuring impact. 
3. Finally, they discussed the lack of a common pool of knowledge and practices, hindering countries from learning from each other and from the efforts of development agencies. 

Ultimately, this discussion emphasized the importance of collective action in conceptualizing, developing, and scaling comprehensive impact measurements for DPI. At the Digital Impact Alliance, we have been engaging with the wider DPI ecosystem through a series of activities, as we attempt to understand the current state of – and demand for – measuring such impact. We believe there is a need to collectively develop a blueprint and implementation roadmap to accurately gauge how DPI is benefiting people and communities across the world. We look forward to continuing this discussion – and others like it – with our distinguished partners in future DDX sessions.  

For more information on the importance of DPI impact metrics – and our efforts to support their development – take a look at our recent piece here.