In mid-2016, we set out to create a funding mechanism in UNHCR that would encourage innovation across our 130 operations worldwide. With displacement becoming increasingly complex and causing untold suffering, we were moved by a sense of urgency to support our colleagues develop novel ways of addressing these growing challenges. Establishing an experimental budgetary space for innovation seemed like a compelling endeavour since the need to develop these solutions couldn’t come at the expense of our ongoing assistance to refugees and the communities hosting them, particularly in emergencies.
As the Innovation Fund took shape, we were faced with the difficult task of establishing metrics to support our goals and measure impact. But measuring something as nebulous as innovation and its impact perfectly is impossible, and this difficulty was compounded by our Fund’s sector-agnostic portfolio approach. Since a universal set of innovation impact metrics simply doesn’t exist, we did our best to select a handful of proxy metrics to help us gauge the impact of our funding on UNHCR and refugees.
With implementation underway, it became evident that the metrics we chose were both optimising for incremental solutions and distorting the day-to-day management of the Fund in a way we hadn’t anticipated nor wanted. This largely happened because our goals and metrics were misaligned and because our goals were ambitious but underspecified. Here are some examples to illustrate this:
While these proxy metrics revealed the number of novel solutions being tested by UNHCR, they optimised for short-term, incremental innovations that had to come to fruition in less than one year. This happened because:
1) Our metrics and goals were misaligned.
Metrics are notoriously difficult to get right, especially when they’re used to measure intangible concepts. This reality is reflected in the humanitarian sector’s overreliance on activity metrics as proxy indicators to measure complex social phenomena. Consider public attitude towards refugees as an example: in trying to cultivate empathy for refugees and improve the public’s attitude toward them, we’ve seen the metric # of public awareness campaigns conducted being used to measure the performance of advocacy work. That metric, like all activity metrics, aren’t inherently bad, but they are prone to distorting behaviour since they’re deceptively simple to measure. Because of this, they often create incentives to prioritise meeting the target (launching many public awareness campaigns) to the exclusion of the very goal that target intends to measure (creating strategic and targeted advocacy campaigns that create sustained and positive behavioural change).
In the case of the Innovation Fund, one such activity metric used was the implementation rate: a measure of expenditure against pre-defined activities in a given timeframe. This metric is applied to almost all of UNHCR’s projects and programmes as a means to ensure that resources allocated for specific activities are utilised over the course of the year. Programmes with low implementation rates at the mid-year point are often required to ‘free up’ funds that are then rechannelled to programmes and emergencies demonstrating greater need. A high implementation rate for the Fund was thought to indicate that a greater number of UNHCR operations were undertaking both incremental and transformative innovations as a result of the Innovation Fund. But what this metric actually optimised for was a de-risked Fund that favoured only incremental innovation projects that could disburse funds and implement activities in one year.
In hindsight, it’s easy to see how our goal to encourage both kinds of innovation in UNHCR was misaligned with the activity metrics we chose. We learned that alignment between our goals and specific impact measures comes with some trial and error, but that it’s important to move away from activity metrics as our sole units of measure. At best, these metrics showed we were stoking the boilers of innovation, and at worst, they created incentives that optimised for incremental solutions at the expense of transformative ones.
2) Our goals were ambitious but underspecified.
When optimally aligned with goals, the right metrics facilitate the management of complex projects and systems since they provide valuable feedback about the effect of a given intervention throughout its lifecycle. In our experience, metrics also helped us align the team’s actions toward the execution of the Fund’s goals since they gave us quantifiable indicators to converge and converse around. But we learned that for metrics to be truly effective at aligning our collective actions toward achieving our aims, we needed to have well-defined goals. As obvious as this seems, clearly articulated goals circumvent the misalignment that comes about from having underspecified goals and overpowered metrics.
Taking the Fund as an example, our goal to stimulate innovation in UNHCR field operations didn’t specify the kind of inclusive innovation we sought to encourage (both incremental and transformative). This vagueness lent itself well to misalignment between our goals and the robustly tracked activity metrics that measured things like the # of innovations adopted by UNHCR worldwide or the implementation rate of our project portfolio. This misalignment doesn’t necessarily lead to failure, but in our case, it made it more likely that whatever overpowered activity metrics we used to measure our goals would inevitably lead to mismeasurement, since they optimised for something other than the intended goals.
A way forward: what we plan on doing differently
The urgency to develop solutions that introduce new tools and processes or creatively and substantially adapt the services we provide to refugees continues to grow in importance. Inclusive innovation will be a key driver of this change, since it places emphasis on developing both incremental and transformative innovation, therefore permitting a large number of people to participate in the process. Because forced displacement is a global challenge, both bottom-up and top-down solutions need to be developed, and the metrics to measure the impact of these initiatives will play a key role in aligning our actions toward our goals.
Moving ahead, here are three considerations we have as we rearticulate our goals and reformulate our metrics:
- Maintain a portfolio approach for projects with funding streams for both incremental and radical innovation. To develop and hone this, we’d like to learn more from public funding institutions that have successfully encouraged socio-economic development and technological development by including both streams of innovation into their work. There’s a common perception that private capital has been the driving force behind investments in startups across many sectors when in fact, most early-stage, high-risk finance has been provided through public sources like the Small Business Innovation Research Program (SBIR) in the United States and SBRI in the United Kingdom. Learning how these institutions and others like them manage risk and assess the impact of their investments will help us develop better metrics.
- Develop a handful of impact metrics and avoid activity metrics to simply feed the data beast. Another common perception is that real-time data translates into real-time decision making. But having too many metrics can easily overwhelm people, who often revert to making calls based on personal judgement or intuition. Real-time data can also establish tighter feedback loops that reinforce incentives to meet the target, often at the expense of the overall goals of a given intervention. We know that measuring impact is important, but given that measurement is resource intensive, we’d like to develop a leaner model that tracks only a handful of impact metrics aligned with our goals. We hope to develop metrics that also influence and facilitate timely and thoughtful decision-making, since we’d like to see more emphasis placed on real-time programmes, instead of only real-time data.
- Set specific goals and align them with our mission. We’ve already started rearticulating our goals to counter the misalignment brought on by having underspecified goals and overpowered metrics. We’d like to make mission-oriented investments combined with patient, long-term strategic finance while maintaining a sector-agnostic portfolio. We believe this is important because our mission and goals will address challenges that cross-sectoral boundaries, since one sector alone cannot solve the political, economic and environmental challenges that result in or stem from forced displacement.
If you’re working on developing financial instruments in the public or private sector to encourage both incremental and radical innovation, we’d love to hear about your experience developing innovation impact metrics. You can reach us at any time by emailing us at: [email protected].