By Khulisa colleagues: Mosangoaneng Leteane & Tamar Boddé-Kekana

WhatsApp is an effective means to communicate, coordinate and teach in a world where so many people have access to smartphones.  It allows inexpensive communication across devices (Android, Apple, etc.).  Programs and projects often use WhatsApp to coordinate, communicate and provide learning resources to staff and/or beneficiaries. Thus, it has become a standard project communication tool.

Analyzing and using WhatsApp data adds a tool in Khulisa Management Services’ Evaluation Toolbox.

Khulisa was evaluating the Reading Support Project which provided coaching services to Grades 1 to 3 teachers on how to teach reading in Setswana and English.  Due to COVID-19 school closures and restrictions, the RSP  reported that they had pivoted to providing coaching through WhatsApp.  As evaluators, we had to figure out how to evaluate this new approach and assess the effectiveness of coaching.

Khulisa took those learnings and presented them as part of the gLOCAL Evaluation Week 2021 organized by the Global Evaluation Initiative (GEI). Khulisa colleagues Mosangoaneng Leteane, Jennifer Bisgard, Carole Metekoua, and Heather Dixon hosted a 2-hour webinar on WhatsApp Data Analysis.

Below is an outline of the steps and lessons learned by using WhatsApp data as an element in our recently completed evaluation of the Reading Support Project (RSP)  in South Africa. We had to pivot our evaluation and phase in WhatsApp when the RSP project moved online, due to COVID-19. We formulated evaluation questions for this data source on inspection.

Our entire gLOCAL webinar video can be found on our YouTube channel and you can have a look at the presentation slides here. To read the Summative Report of the RSP, please click here. For the Formative Report of the RSP, please click here.

We put this guide together retrospectively, due to our aforementioned pivoted approach, but our learnings can help guide the process for any future projects. In fact, we have learned important lessons that can help at different stages when taking a planned approach to WhatsApp data collection:

1. When the project is planned with all the tools in mind, incorporate WhatsApp from the onset:
2. Ensure the Evaluator for your project is a co-administrator on the WhatsApp group(s), eliminating the need to ask others to download and send you the data
3. Make sure your research objectives account for the data source
4. Formulate your research questions beforehand

• Evaluation/Program Objectives. Ask yourself:
• Will it assist our program objectives?
• Are the intended participants techno savvy enough for WhatsApp? Can you provide adequate technical support?
• What are the rules of engagement for the community?
• The granular the better!
  
  
• Legal and ethical considerations:
• Consider protection of personal information policies and laws in country of operation
• Speak to participants beforehand and get written consent
• Make sure ethical clearance accounts for WhatsApp as a source
• Think about how and when participants can revoke consent

• Data collection, planning and execution:
• What is the process map of the WhatsApp data collection method?
• How often do you intend to download the chat timeline?
• When are you conducting the analysis?
• Who are the administrators and how do they identify themselves as such on the WhatsApp group?
• Will participants be provided with data in order to engage meaningfully on WhatsApp? If so, how will that be organized and monitored? Is there a zero-rated option you can provide instead?

• Analysis:
• What kind of statistical package do you already have at your disposal?
• Do you have software for thematic, content and text analysis?
• Which software would work best for the particular objectives/questions asked?

If you are considering using WhatsApp data within your projects, whether as a pivoted or planned approach, here are some useful steps detailing how to extract, clean, and analyze such data.