When Khulisa started conceptualizing the gLOCAL 2020 Evaluation Design Competition, we planned to host it as a face-to-face interactive session. However, COVID-19 stopped this plan dead in its tracks. We had to opt for a virtual session instead. Was there any value in planning and executing an innovative learning event in the new “on-screen” normal? Join us on this reflective journey as we share the thoughts, opinions and lessons from the different competition participants, and derive some recommendations for hosting virtual competitions.
Part 1 of the gLOCAL 2020 Evaluation Design Competition blog provided an outline of the steps to follow when hosting a virtual competition. The blog also provided an easy-to-follow guide on launching a virtual competition, which included the planning, implementing, and hosting phases.
In this blog – the second in our series – we reflect on the value of hosting a virtual competition. We include feedback from competition participants, the judging panel, as well as reflections from the Khulisa team (as the competition conveners). A survey was sent to all the competition participants, and individual interviews were conducted with all the judges.
Competition applicant joins Play Africa team!
Before we dig straight into it, it’s important to celebrate a key outcome of the gLOCAL 2020 Evaluation Design Competition: One of the competitors, Jen Norins, from the winning team joined Play Africa to assist with monitoring and evaluation development within the organization after the event.
See the ‘success story’ quote below:
“I welcome the opportunity to be part of a challenging learning experience to strengthen my evaluation skills. Although I am currently pursuing my PhD in evaluation capacity building, I have not had opportunities to be exposed to, nor to work on various methodologies and apply these new skills. I welcome the opportunity to learn and work in a team”
Jen Norins
The same applicant was part of the winning team in Category 3 of the competition. In this category applicants developed aforward-looking approach that used an impact- or quasi-experimental evaluation design.
Following the gLOCAL 2020 Evaluation Design Competition, Play Africa was proud to share that Jen Norins joined the Play Africa team to support M&E Development/Capacity over the next 12 months.
The competition allowed emerging evaluators to connect with each other and with Play Africa. In turn this provided an excellent opportunity for Play Africa to link with potential individuals who can add value to the work that they do.
Unpacking the lessons learnt
Although the competition was successfully implemented under strict timelines, with individuals working across different time zones, it is important to take stock of what was learnt. Interesting dynamics exist when a group of people from diverse backgrounds come together towards a common goal and there were certainly some lessons to note down.
Lesson 1: Team dynamics can be tricky, and roles need to be clearly clarified right from the start
Managing team dynamics is unavoidable and the expectation is that all members of the teams will pull their weight in an equal capacity when planning and implementing a competition project. But that seldom happens in the real world, as competing priorities have a way of unsettling expectations from the team.
What did the teams have to say about the virtual competition?
What will we do in the future?
The convening team must make the following considerations when putting the teams together:
- Acknowledging and be more sensitive to differences in time zones
- Managing the expectations for team members who are full time employees
- Outlining expectations of each team member
The following must be considered by the competing teams:
- Team members to declare availability (e.g. full time members)
- Role clarification for each team member
- Selecting a team leader
Lesson 2: The more information, the merrier! Applicants need lots of context to plan their approach
The Play Africa Team provided valuable feedback on the value of the competition and how the different teams brought innovative solutions or approaches for the work that they do.
It is important to understand that the competition participants had limited time and information to fully execute their ideas, thus an important consideration for future competitions would be to provide access to a broader range of information. Almost half of the respondents (48%) mentioned that the information made available by Play Africa did not match their expectations.
This is what the competition participants had to say:
Lesson 3: Virtual competitions are a time-consuming business – ideally tackle such initiatives during organizational downtime
One of the critical elements of convening a virtual competition is to actively manage the available time necessary for the team to successfully execute their assignments. Time limitations provided a great learning point largely because of the following
- Team members had competing priorities
- Certain approaches needed more time to be fully executed
The judges expressed the need to have had more time to reflect and have conversations with the teams, based on their video presentations and submissions. On the actual competition day, the process of loading and playing the applicants’ videos was time consuming.
Feedback from the Judging panel:
“I think when you share all the information, it makes it easy, because the participants can actually understand how this can work in the real world. When they started working they didn’t quite understand but as we worked closely together, they were able to ask questions and make sense of it.” Futhi Mbongwe (Play Africa)
“My advice would be to do a bit of research into the real-world implications associated with the implementation of their design and think through mitigation strategies.” Katharine Tjasink (Galvmed/ former Khulisa Associate Director)
“Initially we underestimated the demands of managing the process. The project management was tense. With the challenges of connectivity etc. It would have been great to have the face to face meeting, but that wasn’t possible.” Jennifer Bisgard (Khulisa Director/Founder)
“A lot of work went into the process, and organisations and individuals need to dedicate time to ensure the success of the event.” Gretchen Wilson-Prangley (Director Play Africa)
It is important to note that initially, we underestimated the project management process, which had huge time implications for both participants, conveners and the judging team. The convening team made attempts to communicate and provide clarity to all the competing teams. This is what the competition participants had to say:
Competition highlights
Young & Emerging Evaluators gained completely novel technical skills
Overall sentiments suggest that the highlight of the gLOCAL 2020 Evaluation Design Competition was the participation of different individuals from countries across the world. This provided a diverse pool of competitors. It was commendable that teams managed to work together effectively using remote platforms, regardless of some of the challenges associated with team dynamics.
“Being on a team tasked with developing a forward-looking impact evaluation, the engagement strengthened my knowledge and understanding of the steps involved in impact studies. Furthermore, team members, shared very useful resources on different approaches. The approach we used (process tracing with Bayesian updating) was completely new to me. Also we were exposed to various online platforms that could make remote collaboration helpful.”
One of the core objectives of the competition, was to build the capacity of young and emerging evaluators. The competition succeeded in this objective in that it provided a platform for knowledge sharing and an enriching learning opportunity for participants. From the survey, 41% of respondents indicated that they gained extensive skills and knowledge in the process, while 45% indicated that they did gain ‘some’ skills and knowledge.
“Some concepts I was unfamiliar with and how to do better next time. I am still keen on learning more and getting practical work experience within the evaluation field.”
“The competition was a good exercise in designing an external evaluation from scratch, which is something I had never done before. My team members had approaches or ideas that were new to me, as well. Additionally it was also a good experience in remote team management and organization. I also used graphic design software and learned to use Piktochart.”
Competition participants gained the following skills from the gLOCAL 2020 Evaluation Design Competition:
- Data mining
- Leadership
- Time management
- Evaluation methods
- Inferential statistics
- Data visualization
The competition was important not only for developing completely new technical skills and expertise of the Young and/or Emerging Evaluators, but also for allowing learning and knowledge sharing to take place across teams.
“In our team only two of the four team members had an M&E background which resulted in the other team members learning on the job and not having a grasp of basic M&E principles. In addition I was the only team member with data mining experience out of the four team members which resulted in a lack of collaboration and ability to apply data mining approaches. I gained experience working as a co-lead on a project and having to step up. As well as exposure to developing data analysis templates.”
Play Africa gained rich insights and exposure to new talent
The competition was successful and provided a wealth of information for Play Africa. The proposed solutions addressed different challenges by translating Play Africa’s work through reporting, analysis and documentation.
This provided information ranging from implementation of programs to reporting of impact and dissemination. Competition participants were asked if they would be interested in taking part in a virtual internship with Play Africa and 67% indicated that they were interested:
And as mentioned earlier, one of the competitors, Jen Norins, is now working with the Play Africa Team!
In addition to this, even though there were challenges, most participants felt that Khulisa should host a similar competition is 2021.
Considerations for future competitions
As part of the learning process it is important for Khulisa to be aware of what to consider for the next virtual Evaluation Design Competition, and also provide advice for those seeking to implement a similar virtual competition.
From the judge’s perspectives:
- The development of the competition rubric could have been better facilitated to consider various elements
- Additional time to go through the submissions would have been valuable
Perspectives from the competing teams:
- Recommendations and direct feedback to the teams, based on the rubric would have been appreciated
- Provision of more data sources for the approaches
- An opportunity to select team members based on geographic location
Perspectives from the Competition conveners:
- Virtual feedback session with teams soon after the competition to enable direct reflection
- More time allocation from the competition activities
In summary, our recommendations are to:
- Consider time frames for all the different competition activities
- Build in flexibility around team allocations based on availability and time zones
- Provide adequate access to information/data to cover all possible scenarios
- Allow for further feedback and interaction after the competition ends, specifically direct feedback to applicants on their submissions
Work on the collaborative development of a competition rubric
This then leads us back to the original question, is there value in hosting virtual competition? Khulisa’s answer would be a resounding, ‘YES’! The competition not only provided a valuable learning point for Young and/or Emerging Evaluators, but also provided Play Africa with an opportunity to recruit new talent to their team.
