In this installment of the Dimensions of Impact series, we look at another report from the Center for Media and Social Impact: “Social Justice Documentary: Designing for Impact.” Written in 2011 by Jessica Clark and Barbara Abrash, this report attempts to advance models for measuring the impact of documentary film by synthesizing previous efforts and developing a framework for evaluation in a complex networked media environment.

 

CMSI’s 2009 study “Social Issue Documentary: The Evolution of Public Engagement” highlighted the expanding “circuits of circulation” through which a documentary and its messages could travel – including websites, social media channels and non-theatrical screenings. “Designing for Impact” reflects further on the rapid shifts in media during the past decade, including the multiplying number of screens (theaters, televisions, computers, mobile phones etc), platforms for distribution, and “potential publics” that can form around a film. “Documentaries travel differently in this new media ecosystem,” the report argues. “And they can also play a role in shaping its development.”

 

One of the key arguments made in “Designing for Impact” is that filmmakers need to adapt their practices to this networked media environment, both in terms of how they conceive and produce projects and in terms of how they measure impact. For instance, social issue filmmakers can borrow methods from product designers and software developers by first defining the problem they wish to address, then taking an iterative approach to their work that incorporates feedback from partners or other “users” of the documentary. The tools of design thinking and “human-centered design” (as popularized by design firm IDEO) can be used by filmmakers to integrate evaluation directly into all stages of the filmmaking process, allowing them to engage stakeholders more deeply and better understand “how a film or campaign might best represent and reach intended publics.”

 

Excerpt from “Designing for Impact”

As reimagined for social issue documentary, design thinking steps might include:

  • Define the project’s brief—design thinking aims to identify new solutions. What problem does the film solve?

  • Design with users—design thinking is user-centered. Surveys, interviews and observation before production can help to reveal how users will put a documentary project to work in policy, education and civic settings.

  • Build the production team—design thinking is multidisciplinary, and so is filmmaking. Documentary filmmakers must think collaboratively; involve users, stakeholders, researchers, developers at each stage.

  • Prototype—design thinking is iterative. Filmmakers should road test storyboards, short videos, campaigns with users to think through how their campaign and platforms will help them meet their mission.

  • Understand limits—design thinking includes a keen awareness of constraints. Doc makers should consider the desirability, feasibility, and viability of their film or campaign, and how long each phase will take.

  • Evaluate, and then iterate—design thinking relies on both qualitative and quantitative measures to determine if a design solution is working, or should be retooled.

 

Drawing from six in-depth case studies, the paper goes on map “state-of-the art methodologies for strategic design and evaluation of documentaries” including both quantitative and qualitative approaches. Qualitative methods mentioned include content analysis of media coverage, participation and dialogue, field observations, reported activities after viewing, new partnerships and visualizations of emerging community-based networks. The purpose of qualitative analysis is to “gather and synthesize anecdotes into trends and outcomes.” Quantitative metrics include numbers and diversity of viewers across platforms, sales, investment by foundations and donors, mentions of the film in the media and numbers of users engaged. The metrics also reflect the need to account for both online and offline organizing practices.

 

Some of the charts included the report offer useful strategic design questions that filmmakers can ask throughout the process to ensure that they are working towards the impact goals they’ve established. These include:

  • what problem does the film address?

  • what groups of users will the film, inform, engage and potential mobilize?

  • how do aesthetic and narrative choices relate to intended outcomes?

  • how will filmmakers engage potential users to test and respond to narratives, educational tools and calls to action?

 

 

Like the other reports we’ve covered in the , “Designing for Impact” makes the case that impact evaluation methods should vary based on the goals of the filmmaker, which is “confusing for those hoping to find a single yardstick.” But it also contends that we need a “more standardized methodological approach that combines the strengths” of qualitative and quantitative evaluation to allow more consistent assessment across projects. This seems to be one of the basic unresolved tensions underlying media impact assessment: the need for a framework that combines both qualitative and quantitative metrics, is adaptable to a project’s specific goals and universal enough to allow for consistency across projects.

 

Our challenges in measuring the impact of Video for Change work are nearly identical, although complicated by the wide variety of practices in the field. Can we develop a framework that is flexible enough to be used for participatory video, citizen journalism, video advocacy and witnessing video? And perhaps more importantly, how can we streamline impact measurement so that it is less time- and resource-intensive?

 

The tools of design thinking address some of these challenges by breaking up evaluation into smaller pieces and integrating it directly into the planning, research and production processes. Going through several cycles of evaluation and iteration ensures that a video project remains goal-driven and potentially allows project leaders to capture the impact of the process rather than focusing exclusively on the finished product(s). As Jessica Clark articulates in another CMSI report, Public Media 2.0, “People come in as participants in a media project and leave recognizing themselves as members of a public—a group of people commonly affected by an issue.” Such formations of “new publics” can be far more effective if participants’ perspectives are acknowledged throughout the production process rather than simply after a film’s release. This approach also seem particularly well-suited to the “networked media ecosystem.” By designing projects iteratively and collaboratively, both filmmakers and Video for Change practitioners can be more responsive to the changing media habits of participants and users. They can also play a larger role in reshaping that ecosystem by fostering new capacities and new “circuits of circulation.”

At a time when much attention is directed at blockbuster documentary films that help change policy or ignited widespread debate, it seems that this more subtle form of impact – shaping the development of media ecosystems – is often overlooked. But it is an area where much Video for Change work has its biggest impact, by expanding media access and bringing new voices into public awareness. This kind of work has to potential not only to raise awareness around specific issues, but also to challenge the top-down broadcast model and transform our media systems into inclusive and democratic public forums. Measuring this kind of impact may require a more collective approach that encompasses a wide range of projects and organizations, but it could also help us better articulate, as a field, the most important long-term impact of our work.