In our previous posts, we examined two seminal reports on the social impact of documentary films. The Fledgling Fund’s 2008 “Assessing Creative Media’s Social Impact” introduced a framework in which compelling stories create social change by going through stages of expanding awareness, deepening an audience’s engagement with an issue and thus building stronger movements. In Fledgling’s rubric, each of these stages comes its own goals and metrics for success. In “Social Issue Documentary: The Evolution of Public Engagement,” the Center for Media and Social Impact (CMSI) further developed these concepts with in-depth case studies of audience engagement campaigns tied to films that took advantage of diverse “circuits of circulation” constituting what CMSI calls “Public Media 2.0.” This is a valuable approach for thinking about how a film’s messages, rather than just a film itself, are embedded within a larger media ecosystem and are distributed through both “top-down” and “bottom-up” approaches.

Two years after CMSI’s report, UK-based BRITDOC Foundation published “Beyond the Box Office: New Documentary Valuations,” a report that picks up on this dialogue around the social impact of documentary films by undertaking an extensive evaluation of the Oscar-winning climate change documentary An Inconvenient Truth. BRITDOC has been a leading organization in the burgeoning impact field, both supporting social issue filmmakers directly and building new funding and distribution models through partnerships with Channel 4, Bertha Foundation, Puma and others. They’ve also helped launch a successful series of live pitching events called Good Pitch that connect documentary filmmakers with activists, organizations and funders working on the same issue. These events have helped formalize some of the recommendations made by both Fledgling and CMSI, creating a forum where filmmakers and activists can meet and forge new collaborations.

While events like Good Pitch have helped disseminate the model of partnership-driven audience engagement campaigns, big questions remain about how to assess the impact of such campaigns, particularly from the perspective of funders and policymakers. “Beyond the Box Office” applies a variety of new tools to the assessment of documentary film’s impact – many of them quantitative approaches drawn from the emerging fields of environmental and cultural economics, social entrepreneurship and sustainability accountancy. The report begins with a quote from Jeff Skoll, the billionaire founder of eBay and – through his company Participant Media – an Executive Producer of An Inconvenient Truth:

“One metric of success that we use is whether more good comes from the film than just putting the money directly to work in a non-profit organization involved in the same issue. Weʼve actually had cases where we looked at the risk profile of a film and said, ʻThe way this looks, chances are weʼre going to lose a million, 2 million, even 5 million dollars. But maybe weʼll get 10 million or 20 million worth of social value from itʼ. We will take risks on projects where we think we might lose money, because we hope that the good that comes from that outweighs the risk. Itʼs a different kind of philanthropy.”

Skoll’s statement points to a question at the heart of impact evaluation: how can we determine the value of social change media? The metrics for media’s impact (social or otherwise) are typically based on the logic of market economics: we measure the number of people reached, tickets sold, profits, exposure and so on. BRITDOC points out that even for a commercially-successful film like An Inconvenient Truth, there are “positive externalities” that haven’t been accounted for in the film’s financial ledgers. These externalities include individuals and companies that decided to cut carbon emissions after viewing the film, but also the rise in media coverage of climate change and the extent to which public opinion on the issue shifted as a result of the film. Altogether, these externalities make up what Skoll calls the film’s “social value,” which can be considered distinct from its market or exchange value.

The problem is that concepts like social value and positive externalities can be difficult to measure, at least compared to the “hard numbers” of a film’s box office revenues, ratings and profits. Measurement becomes further complicated when we ask whose value should be measured, since a film will have different value to different investors, audiences and communities. Filmmakers may have an instinctive notion or even anecdotal evidence of the “public good” their work is generating, but as BRITDOC points out, documentary funders and investors often “need hard data to show to colleagues, bosses and boards when it comes to media funding decisions. And they need more than just TV ratings, press cuts and awards to prove the real reach, influence and impact of their investment.”

The BRITDOC report attempts to close this gap between funders and media producers by applying recently-developed economic methods in an attempt to quantify some of the positive externalities generated by An Inconvenient Truth. Some of the indicators they draw on include:

  • the jump in Google searches for energy efficient light bulbs following the film’s release

  • increases in voluntary carbon offsets in zip codes where film was screened (compared with zip codes where film was not shown)

  • large companies that launched initiatives to cut carbon emissions inspired by the film

  • the value of the publicity generated by the film, measured by Advertising Value Equivalency (AVE) 

Measuring each of these externalities requires its own form of research and a great deal of educated guessing, but some of the numbers they produce are worth considering, despite a wide margin of error. For instance, while An Inconvenient Truth produced an estimated £1.2 million in profits in the UK, the unpaid publicity it generated were valued at £3.7 million in Advertising Value Equivalency. BRITDOC takes this analysis a step further by commissioning a survey using Contingent Value Methodologies to determine the British public’s “Willingness-to-Pay” (WTP) for the film. This method has been developed in the past few decades to estimate environmental damage in disasters like the Exxon Valdez oil spill. According to the survey results, close to 10% of the British public would be willing to pay between 20p and £500 to ensure that An Inconvenient Truth was seen, giving the film an “intrinsic valuation” of £73 million in the UK – roughly 57 times the film’s estimated £1.3 million budget. The authors admit that this number is based on a several educated guesses and a methodology that has drawn criticism in the past, but they contend that Contingent Valuation and WTP can be a valuable tool for leveraging funding, even if it only provides ballpark figures.

So what does this all mean for Video for Change practitioners? The BRITDOC report seems to reflect a growing imperative to measure impact that cuts across a variety of social change media, including commercially-successful feature documentaries, short human rights videos, websites and social media. But to what extent can we reduce social impact to a dollar value? Can methods like Willingness-to-Pay and Advertising Value Equivalency be applied effectively to participatory video or citizen journalism initiatives, or do they only work with high-profile films like An Inconvenient Truth? How do we think about “positive externalities” in projects that don’t have commercial motives? What aspects of impact do these methodologies miss?

In the report’s conclusion, the authors acknowledge the limitations of a purely quantitative approach to impact. Valuing most individual sources of externalities is “complex, time consuming and costly,” particularly if measures are not put in place prior to a project. WTP studies, they argue, “overcome the issue of translating effects into economic value” and over time could “help the documentary sector as a whole to lobby for greater resources and status.”

Such a collectivist approach to impact evaluation is an interesting one to consider from the perspective of the video4change network. But in contrast to the relative uniformity of feature documentaries, it may be more difficult to implement common standards that cut across the wide variety of Video for Change practices. Another challenge in applying the WTP method to Video for Change work would be reflecting value generated by the process of video production, rather than just the value of the video itself. For instance, if a community media project targeted a rural community with limited financial resources, a WTP study might produce disproportionately low numbers that don’t reflect the project’s true social value. Such studies might therefore carry the danger of reproducing inequalities by overvaluing social change media targeted at relatively affluent audiences.

Hard numbers may be a common currency of investors, funders and board rooms, but it is unlikely that purely economic approaches to impact assessment will ever provide a complete picture of media’s impact. Nevertheless, are there ways we can draw on these methods and combine them with qualitative assessments to build the Video for Change field as a whole? We’d love to hear your thoughts in the comments!