Embracing New Approaches to Our Work When Projects ‘Fail’: What We’ve Learned About Financial Modelling for ICT4D
11 mins read
The Digital Impact Alliance (DIAL) was established to develop common digital public goods that any sector and any country could use.
The Digital Impact Alliance (DIAL) was established to develop common digital public goods that any sector and any country could use. The mission our donors set us was to move nimbly, make mistakes, and scale successes. While we have had a number of successes over the past two years (see our recent publications for examples), we have also made mistakes that we feel it is important to share with the wider digital development community as part of a shared learning agenda for the sector. This is the first of these pieces.
A great idea – a financial modelling tool for digital development
Last year, as part of its mission to identify effective and efficient digital solutions to help achieve the Sustainable Development Goals (SDGs), DIAL set out to develop a financial modelling tool explicitly designed to meet the needs of those working in ICT4D / digital development.
While many programs have developed their own tools to forecast costs, our hypothesis was that a generic tool could be developed that any government or NGO could use to appropriately quantify and model the financial requirements of their digital development programs over time, to manage multiple scenarios, to identify global data for their input costs and, in particular, to use these models to plan for sustainability at scale.
The idea emerged from needs expressed by practitioners interviewed for our Beyond Scale e-Book who identified a common challenge – appropriately forecasting the long-term financial ramifications of an early pilot going to scale.
“Development implementers face the problem of trying to forecast whether technology they’re using in a pilot will scale. Often, they face this problem without the ability to use financial models that can help them understand how scale will affect their cost structure.
Kate Wilson, CEO, DIAL
We were uncertain if it was possible to build this type of common building block in a way which was robust, easy to use and easy to adapt, but decided it was important to try. Two rounds of interviews took place with practitioners from large and small implementers from the global South and the US/Europe, and with key donors and government representatives. This led to various adaptations to the scope of the project and the user-testing of the first Excel-based prototype. Feedback from the test users led DIAL and its implementation partners (Analysys Mason and FHI 360) to unanimously agree to end further development of the tool.
The decision to end a project prematurely is not an easy one for an organization to make but DIAL learned a lot along the way, both about financial planning challenges in ICT4D, and about how to improve our product development process. Shared in a spirit of openness, we hope that these reflections will be useful for others in the digital development community too.
5 things we learned about financial modelling for ICT4D
1.) An interactive tool is not always the right solution when the problem is lack of information
Many large implementers already have well-established systems of financial planning, including templates or modelling tools to help them make financial decisions, while smaller organizations tend to have less-defined financial planning systems.
Although for some organizations, a simple, well-structured and user-friendly tool was critical, the primary barrier for most of the larger organizations was the lack of reliable information about which cost categories to include for technology projects, and ballpark estimates for these cost categories.
Identifying generic cost estimates (for handsets, airtime, messages, software development etc.) that could be reliable globally proved complex. When organizations make estimates on scaling a product, they use negotiated rates matched with international estimates from ITU/Gartner etc. Given the pricing variability in the markets in which international NGOs work, variations in pricing are such that a common set of data globally is impractical.
“The financial element isn’t complicated – it’s straightforward to list and sum costs. The real benefit of the tool would’ve been automating an ICT4D specialist’s knowledge so a health, education, or agriculture expert knows what digital solutions will and won’t work at any particular scale, as well as what costs to consider and how to quantify them. Unfortunately, that is also the part that made producing it so difficult.” – David McCann, DIAL
2.) Designing for the broadest needs can mean creating something too complicated for anyone
Attempting to develop a tool that could apply to all kinds of ICT4D work may have been too ambitious. The approach DIAL took of creating a generic ‘shell’ which could be customized for different circumstances ended up being overly complex.
In attempting to combine the use-cases into one tool, the prototype became overly complex and difficult to use – differing use-cases may be better served with specific tools or resources tailored to their needs, and some of the test users reflected that they would find worked examples of real-projects with real figures for costs and revenue more useful than a tool.
“User needs and skill levels are varied. Because of this, we developed a flexible structure which proved too complex for many users. It was also not possible to really integrate the data and know-how that DIAL initially envisaged for such a broad range of use-cases. Individually-tailored solutions are more likely to be effective but are harder to generalize. This is a recurring challenge – designing flexible tools whilst abstracting the complexity away from the users through an effective interface.”
–Richard Morgan, Analysys Mason
3.) User experience is complicated and decisions made early-on can cause problems later
We decided early on to create an Excel-based tool because many development practitioners have access to and are already comfortable with Excel, because early user feedback indicated that transparency on entered data and associated calculations was critical, and because it is comparatively cheap and rapid to develop. Through user-testing, we later discovered that an Excel-based tool was prohibitively complex for some test users. A more flexible, web-based wizard (like a ‘turbo tax for ICT4D’) may have been a better match for these users, but would have been considerably more complex and expensive to build.
4.) Some critical success factors are just too complex to automate
Beyondrelatively simple items such as SMS costs, software costs etc., there are many more complex factors that impact a products potential to scale – donor-funding cycles, changing availability of technology, buy-in from government, support of local civil society partners, end-user training etc. These softer aspects are arguably more important than simple financial inputs, and yet their variety and complexity do not lend themselves well to an automated tool.
5.) A tool on its own is not enough without business acumen and support alongside it
No matter how nuanced and well-developed a modelling tool is, if whomever is using the tool does not have the financial background or technical skills to make use of it, and to understand the different cost components that go into it, it will not be helpful.
We had hoped to build a tool that was sophisticated and flexible enough to minimize these factors, but perhaps the problem of financial modelling capability is one that is better addressed by improving the skills and capabilities of practitioners, and offering them the data and information they need, along with more targeted and tailored tools where suitable.
4 lessons DIAL learned from this journey
It is increasingly good practice in our sector to embrace iterative and adaptive approaches, to embrace learning from failure and to share results openly so that others can also learn from them. In this spirit, DIAL wanted to also share some of our internal learnings from attempting to build this financial modelling tool, and highlight some ways that we are acting on them.
1.) Ensure clear product ownership, with a strong connection to the initial vision and champion
The initial vision for this project emerged from reflections of DIAL’s CEO Kate Wilson and Sara Chamberlain from BBC Media Action during the development of Beyond Scale. Informed by Kate’s experience at PATH supporting country governments and NGOs to scale pilot projects nationwide, and Sara’s experience scaling state level systems nationally in India, they developed the concept for a tool to support development implementers in long-range financial modelling for ICT4D projects and help them forecast whether a digital development pilot would financially scale on a long-term basis.
As others took over implementation, the product ownership became dispersed and the lack of a shared vision, combined with limited direct experience of ‘the problem’ by the product team within DIAL or the vendor, inevitably led to the project goals starting to shift – from a tool for forecasting scale and sustainability of an ICT4D component of major public sector projects to a more generic financial planning and budgeting tool for ICT-based projects or initiatives.
In an ideal world, the person with a vision for a product would steward it through its development. In reality this is rarely practical.
Within agile software development, a Product Owner (PO) is defined very differently to a Project Manager and each play a different role – helping to compartmentalize the time commitment needed from the Product Owner or vision holder.
This is a significant shift from the way most aid/development organizations think, and DIAL is exploring what it can learn from ‘product owner’ discussions in the agile community, where this problem has been a subject of debate for decades and various ideas have emerged to mitigate this risk, although the challenge of responding to the lack of time of Product Owners is an ongoing challenge in every sector!
DIAL has also improved our product-visioning process and now seek to identify equivalent tools/scenarios from other sectors and include these at the ideation stage to help coalesce all stakeholders around a shared vision of what success will look like.
2.) Narrow target audiences early-on to limit later complications
“DIAL has many audiences, including donors, ICT4D practitioners, governments, social enterprises and more. Anything we produce can be helpful for any one of those audiences, but we needed to do a more exhaustive job of scoping and focusing in on one key group.”
– Melissa Johns, DIAL
Though the team worked to hone in on a specific audience and target user, it was difficult to come to a consensus that all team members were comfortable with, and consensus-building conversations continued even after the development of the tool was underway.
The tool ended up trying to serve all ‘digital development practitioners’, including NGOs, donors, governments, civil society, private sector start-ups and more. As development continued it emerged that different members of the team saw the primary audience differently leading to different expectations as to the level of technical and financial expertise these users were likely to have.
This insufficiently clear and shared understanding of the audience inevitably led to scope creep as the tool tried to serve too many of DIALs stakeholders. DIAL has since improved our product ideation practice to more clearly capture specific audiences and personas for our tools along with the thinking behind such decisions, along with ensuring that the product team and our vendors have sufficient experience with the target audience. We hope this will help ensure that in future, when scope changes inevitably occur, they do so collaboratively and intentionally.
3.) Live by the Principles for Digital Development – take Design with the User to heart
“There should have been more focus on designing with the users, ensuring that initial user research led to a deeper understanding of their needs, contexts and challenges. Instead, the project suffered from the common problem of “if we build it, they will come”. The design team thought user needs and wants were already well understood, and developed the product based on what turned out to be an incomplete and inaccurate understanding. For example, our engagement with potential users consistently identified a demand for blending in access to business-specific expertise in the tool’s functionalities, but this was not included in the design at any stage.” –Troy Etulain, FHI 360
As the steward of the Principles for Digital Development, DIAL believed it had embraced the principle of Design with the User in this project from the start. Looking back, while we did a large number of interviews about needs, we moved into developing the tool itself without continuing to involve these users meaningfully throughout the design and build stages.
Earlier prototypes or mockups of different UX options would have enabled us to get rapid feedback from users and still have time to iterate on these and pivot the direction of travel if needed. Instead, by the time we had a prototype ready to test with users, it was too late to change direction when we discovered that more than a simple course correction would be required.
DIAL was already committed to a systematic process of scoping, landscaping, and needs assessment to determine whether a planned resource/investment would meet the needs of the user. We need to get better at keeping the users involved as we shift from this landscaping phase in to actual design, development and deployment/dissemination of the resources we produce; better at ensuring their feedback is early enough that we can still adapt our plans to meet their evolving needs.
4.) Know when to stop – iterative approaches allow earlier adaptation.
“We came up with a decision matrix and had a number of meetings to discuss it. We ultimately decided that the tool was not fit for purpose and the questions around the goals, complexity, and audience were not getting resolved. We decided that we could utilize the funds in a more effective way.” – Steve Erbrick, DIAL
Ending a project is always be a difficult decision, but one that is often needed and we should feel comfortable doing so when it makes sense. Various team members reflected that it might’ve been better to stop the project earlier, as soon as the challenged discussed above started to become apparent.
As DIAL refines its processes to be more iterative and user-focused, discussions about progress and success will be possible earlier and more frequently. Each of these discussions is an opportunity to course-correct, but each is also a chance to discover when things are not going as expected and make a more significant pivot, or stop the project and re-allocate the resources to other important work.
The often mis-understood Silicon Valley mantra of ‘fail early, fail fast, fail often’ does not imply failure is inherently good, but that when we trial new things, some will work and some won’t, the earlier you discover this the better and the more things you can trial and learn from.
‘But I still need help with my ICT4D financial planning’ – what now?
The financial planning skills-gap in the digital development sector remains a concern, but based on this project, DIAL now understands that many of those working in ICT4D and digital development would benefit more from resources, training, templates and practical real-world examples to help them better understand the issues around financial planning/modelling for sustainable technology than an ICT4D-specific financial planning tool.
We would encourage donors and implementers to share any such materials of their own that might help others – particularly real budgets of real projects, demonstrating the more complex and less obvious costs
DIAL has developed what we hope is a valuable addition to these resources, a spreadsheet-guide to the various tools and resources that we analyzed during this project.
We would welcome input from organizations who would like to take this further, to add to the list of tools, to gather and share real project data, or to take on stewardship of such resources.
To discuss, please contact us at financialmodelling@dial.global
This piece is authored by Matt Haikin and Nikki Brand
