Guide to Building Data-Driven Organizations in the Public Sector

Using Big Data for Evaluation

(Team 6)Tommy Kolwicz and Dennis S Stockwell

Topic Overview

This chapter focuses on how evaluators can use Big Data as another tool to provide data points that can be used for analysis. The overall theme is that there is a significant role for Big Data to play. The first reference lays out recommendations, that if Big Data is going to be collected and used that its use should be considered early, you need to understand the role that the selected big data will play, understand the limitations of the big data that you are using (demographics of users), how you will collect data from the demographics not represented by a specific social media platform, the role language may play in the collection of data, and there may be a role of shaping Big Data thru the use of hashtags or campaigns that encourage responses that can be measured. The second reference provides insights on how big data can be used to augment and fill in a lack of knowledge about communities from a program evaluation perspective. The authors argue for the immediate requirement for evaluators to learn and use big data sources, technologies, and methodologies. The bottom line is that there is a role for Big Data as long as users understand limitations.

Chapter Summaries

Can big data be used for evaluation? A UN Women feasibility study.

This week’s reading is based on “Can Big Data be used for Evaluation” a UN Women feasibility study. The study was to “determine to what extent big data could help strengthen traditional UN women evaluations” The study focused on 2 countries Mexico and Pakistan. The study focused on Mexico due to its high internet penetration 65.3% and one of the top three twitter users representing approx. 20% of the population and Pakistan at the other side of the spectrum due to its lower internet penetration (22% of population) and over 31 million Facebook users. The Study focused on analysis of Twitter in Mexico and Facebook in Pakistan. Some of the pros and cons of Facebook were: Pros - conversations are more in depth, access to historical data, data can be collected via app (requires consent) and allows for analysis of social networks. Cons – Conversations are stovepiped, and data can only be obtained from public pages. For Twitter the Pros were analysis can be done in real time which allows for analysis of current issues and possibly identify key influencers within the network. Cons- Twitter limits conversation to so many characters, personal issues may not be discussed to the public nature of the media, and a small portion of the population are twitter users.
Ultimately the study did conclude yes, “Big Data can be used for an evaluation”, however there were several recommendations to keep in mind if you are going to use Big Data for an evaluation. Make sure that you understand the role that the selected big data will play, understand the limitations of the big data you are using (demographics of users), how you will collect data from the demographics not represented by a specific social media platform, the role language may play in the collection of data, and there may be a role of shaping Big Data thru the use of hashtags or campaigns that encourage responses that can be measured.

Integrating big data into the monitoring and evaluation of development programs

This report is meant to be a “call to action”! Big data is here. We must use it to make the world a better place. The insights from big data can be used to augment and fill in a lack of knowledge about communities from a program evaluation perspective. The authors argue for the immediate requirement for evaluators to learn and use big data sources, technologies, and methodologies. Specifically the article is focusing on the real need to gain greater insight on the “impact of development programs on the poor and vulnerable.”
This reading is an extremely in-depth report on two broad topics. Part 1 covers development evaluation in the age of big data; it’s implications, possibilities, and challenges. This part of the paper essentially lays out all of the different types of data and all the varied methods of collection, while also putting types of data into bins essentially based on quantity. Like Digital Humanitarians, it hits a lot on the potential of real-time data. Then in chapter 2, the authors discuss the challenges of big data from a standpoint of how stakeholders can begin to integrate big data into an already existing framework of program monitoring, evaluation, and learning (MEL).
Part 2 takes us deep into a program evaluation discussion. In chapter 3 the theory of how, specifically, big data could be implemented into an existing MEL framework is discussed. The discussion hits a lot on where the holes are in existing evaluation systems and how big data can fill those holes. Chapter 4 discusses building big data into program monitoring. And chapter 5 discusses how to build big data into the MEL frameworks. Chapter 6 finishes off with a discussion on managing big data inclusive evaluations, the manager’s role, and the special challenges associated with big data management. “This chapter stresses the critical role of the evaluation manager in ensuring that all evaluations address the key questions of concern to stakeholders and that the kinds of information generated can be used by a wide range of stakeholders and for different purposes.”
The UN’s Sustainable Development Goals (SDGs) are used in this paper to explain why it is important to rethink current MEL processes. The SDGs define the vision and goals for creating the “world we want” by focusing on the 5 Ps: People, Planet, Prosperity, Peace, and Partnership. There are 17 goals and 169 targets. One example is SDG–4, “Ensure inclusive and equitable quality education and promote lifelong learning opportunities for all.” Certainly we can all probably think of ways that big data could be used to facilitate the evaluation of that SDG.
There are three main ways that big data can be integrated into MEL frameworks: you can simply fit it into a conventional evaluation, you can use the data to strengthen an existing design, or you can create a new integrated design from the bottom up. Chapter 5 has some interesting case studies that I will share here.
• Using high frequency metering data for high–quality information about energy consumption and demand in rural solar micro–grids in India.
• Tablet–based financial education in Colombia. Using savings and transaction data combined with survey and telemetric tablet data.
• Assessing the effects of a government tax increase on smoking using changes in search query volume to assess the effects of a major increase in cigarette smoking in the USA. • Evaluating causal interactions between labor market shocks and internal mobility.

This paper serves as an excellent and informative primary source for how to integrate big data into program monitoring and evaluation, and what can be learned from that integration.

Discussion Questions

1) How can you use Big Data to evaluate one of your programs? 2) What are some blindspots in the evaluation that would result from relying only on Big Data as an evaluation tool? 3) Can you see a vision in your community for creation of new evaluation frameworks built from the bottom up integrating Big Data?

References