WADA Qualitative Audit Pilot

[IMPACT]

WADA and qualitative data

WADA ( West Africa Decentralized Alliance https://www.wada.org/) delivers groundbreaking projects in Africa to engage communities with Cardano. These include education and mentoring, community hubs, and engagement with students, entrepreneurs, healthcare workers, developers, and more. Many of the gains of these projects are qualitative as well as quantitative; they include what participants have learned, how included they feel, how confident they feel, how their understanding of Cardano has developed, what the facilitators noticed about how smoothly the project delivery went, and so on.

This kind of information is, naturally, extremely useful to WADA when designing future proposals, and also when developing their overall working methodology and best practice that they can share with Cardano as a whole. WADA is a very well-established organisation within the Cardano ecosystem, and has delivered a wide range of different kinds of project; so their experience is extremely useful to other organisations, particularly those who work in countries that face multiple barriers to participation.

The audit information gap

However, most of Catalyst’s existing audit requirements for funded proposals focus heavily on quantitative measures - numbers of developers, numbers of products, numbers of views. No matter how well-designed those quantitative metrics are, they inevitably miss a lot of crucial information - often, the information that represents the most important learning from the types of projects that WADA does. Although some qualitative information is used in making the final videos required to close out a project, this is essentially a marketing task, and the information that goes into making a video is rarely stored as data that can be interrogated or reused later. Instead, it stays in the heads of the project facilitators, who may not even have the time or resources to share it informally among themselves. Without a clear methodology for collecting it, qualitative and experiential data is not recorded with any consistency; and even when it is recorded informally, there is no system to document or archive it so as to make it discoverable, or to connect it or make it searchable so that patterns can be identified.

As a result, some of WADA’s most important learning about project development and community engagement on the ground is in danger of being lost. Each new WADA project is starting from scratch, other than whatever information teams are able to share informally; the voices and insights of project participants are lost; and this vital knowledge also does not reach the rest of the Cardano ecosystem, to facilitate skill-development for all of us.

WADA needs a system to:

• collect key qualitative data from projects in a way that is not onerous for project facilitators, and which focuses on the most useful and important information;
• archive this information so that it can be found, searched and used (for example, to support proposal-writing, by evidencing that a proposal is addressing the real issues as identified by previous work)
• link information from different projects, so it can be analysed, and so that different WADA projects can learn from each other, and so that an overall “best practice” can be developed, and WADA can grow in an evidence-based way;
• enable others in Catalyst to easily see and understand the insights that have been gathered; this will support WADA to share its expertise and its practice-based learning with the rest of Catalyst so that Catalyst overall can improve.

This proposal seeks to develop such a system, and pilot it with five key WADA funded projects.

We will :

• conduct insight-sharing and problem-sensing with existing WADA funded projects (talking both to facilitators and participants), to uncover what kinds of qualitative data is being missed by current audit processes; and to get a sense of what a WADA project actually *should* record in order to evaluate whether it has been successful
• use the information gathered during problem-sensing to develop a documentation / audit methodology using tools already common in the Catalyst community, such as Miro and GitBook, which will capture and collate both the qualitative AND quantitative data that facilitators and participants consider most important. The methodology will include an archiving workflow, making the data discoverable; and will enable users to search and link information from different projects
• pilot and run this methodology with 5 existing WADA funded projects, to ensure that it is usable, and that it enables facilitators to feed useful data in to project development, learning and best practice development
• share the insights from this process, and the project archive that we have developed, with the Catalyst community as a whole. We will particularly focus on the Funded Proposer and Audit communities, with a view to sharing best practice in auditing proposals that have a community engagement / qualitative focus.

The “Improve and Grow Auditability” challenge focuses on improving audit methodologies for funded proposals. By focusing on praxis and on qualitiative information, this proposal will address some key omissions in the current process, as idnetified by WADA in its projects so far. The learning from this proposal will improve how we, as a Catalyst community, audit proposals, by developing ways to collect a more complete picture of the successes of a project.

• Risk of limited engagement in the problem-sensing process, if project deliverers and participants do not have time. Mitigated by planning with them beforehand to ensure they can plan time for it; and also by paying them a contribution for their input.
• Risk of the methodology not being taken up and used by WADA projects. We will mitigate against this by ensuring it is no more time-consuming or complex than the existing process of monthly reports; and by our project team managing the participating projects’ existing IOG audit requirements for them, i.e. their monthly reports to IOG, leaving them free to focus on the new methodology.
• Risk that we devise a methodology that is unwieldy or unsuitable. Mitigate by a) ensuring that actual WADA projects are consulted at every stage and b) allowing time for iterating, and for going “back to the drawing board” if needs be.

[FEASIBILITY]

First month (May 2022)

• Collate details of what audit material WADA projects are already collecting beyond what is required for IOG reporting, particularly qualitative data; and how that material is currently stored.

Following 3 months (June -August 2022)

• Complete the problem-sensing process, canvassing opinions from 15 people (facilitators and participants), covering 6 WADA projects.
• Analyse the results of the problem-sensing, and end up with an analysis of exactly what is missing from the current evaluation process, what info WADA needs to gather, and an idea of how this could be done without placing huge evaluation burdens on project deliverers, and without the drawbacks of things like perfunctory, “by rote” evaluation forms for participants.
• Develop a suggested audit methodology, including a process/workflow for recording and archiving the information to make it discoverable and searchable.

Next 3 months (Sept-Nov 2022)

• Test out the proposed new methodology with 5 existing WADA funded proposals, and record via a mid-term evaluation in early October how well it works. Are we capturing a more complete picture of a project’s achievements? Is the methodology lightweight, easy and clear to use? Are participants’ voices amplified? Does the methodology help project deliverers with identifying and solving problems - not just the quantitative ones of “we are/are not not reaching this or that numerical target”, but “we are/are not achieving the outcomes we wanted” or “people are feeling positive/negative about such-and-such” or “the project needs to respond to such-and-such a situation”? Is the “archiving” system, the place where data is kept, easy to use and search?
• If needed, create a 2nd iteration of the methodology, and test it. Or, if this is not needed because things are working well, we will continue to run the new methodology until the end of November to check for any unexpected issues.
• Share the new methodology, and the information gathered from WADA projects, with the Catalyst community, especially funded proposers, audit community, Catalyst Circle, and IOG. This will be done via an After TownHall session at African Town Hall and another at “original” Town Hall; a written report; a video guiding people how to use the documentation process; and social media.

12 months (Dec - March 2023) - after end of project

• Ongoing sharing of the new methodology via social media and sharing of the video and report.
• Inviting the commuity to record via GitHub any learning that emerges if other funded proposals decide to use or adapt elements of the methodology for their own use in their own contexts.

• Faciliatator from QA-DAO to design and run the problem-sensing process; 6 days @$400 = $2,400
• Contribution to each WADA project faciliatator and participant for their time engaging in problem sensing (15 people x $140 per person per session) $2,100
• Facilitator and 2 members of WADA to analyse the research info and then design a process: 10 days x 3 people x $400 $12,000
• Subscriptions for archive process tools, e.g.Miro and GitBook $670
• Populate the system with all existing WADA documentation material from past projects (20 projects): 10 days @$400 = $4,000
• Train WADA project facilitators from 5 key projects to use the new system ($400 per session x 2 sessions) $800
• Compensation for time spent in report-back meetings for 5 WADA project managers (2 meetings x 5 managers x $140) $1,400
• Costs to allow project team to manage the participating projects’ existing audit responsibilities for 3 months, leaving them free to focus on the new methodology (5 projects x 3 hours x $50 per hour) $750
• Share the project learning with Catalyst (creating short video and written report, After TownHall sessions, social media) $1,400
• Treasury management (handling project wallets and distributing funds to team) $80 per month for 6 months = $480

Total : $26,000

Vanessa Cardui (@CallyFromAuron) is a community engagement professional with a background in archiving and qualitative project evaluation. She has 20+ years' experience of working with communities to record and collate their information, archive it, and make it discoverable. (see for example https://feministarchivenorth.org.uk and https://homeless.omeka.net ). She is pat of QA-DAO http://quality-assurance-dao.github.io , a Catalyst organisation for audit and quality assurance, where she is currently leading on oversight of Catalyst Circle (see https://quality-assurance-dao.gitbook.io/catalyst-circle-oversight-v3 )

Mercy (@MercyA1) is the Coordination Lead for Wada (https://www.wada.org/) . She has extensive experience in project management and process improvement. She’s the Circle v3 rep for Funded Proposers due to her passion for decentralised governance and is always ready for new challenges.

[AUDITABILITY]

We will measure:

• Number of WADA projects, facilitators/managers, and participants engaged with an exploration of qualitative audit methodologies.
• Whether project managers and participants feel heard in the problem-sensing process, and whether they feel that the proposed new audit process addresses the concerns that they have raised.
• Number of WADA project facilitators confident in using the new audit methodology, and able to train others.
• Number of WADA funded projects piloting the new methodology.
• How far projects using the new methodology feel that it is usable, lightweight and effective.
• Numbers of people in Catalyst community engaging in some way (reading report, watching video, attending After Town Halls, commenting, adapting it for their own use) with the new methodology.
• Whether or not comments and interactions from the wider Catalyst community are positive.

Success would be:

• WADA facilitators and project managers reporting an increase in audit and evaluation confidence
• WADA project deliverers and managers having space for reflective practice, and a way to capture qualitative insights
• Creation of a methodology which project leads and participants agree is usable, and also more effective in capturing and archiving the full range of data, including qualitative information;
• A methodology that can be easily scaled for big or small projects, and which can be worked into all WADA proposals
• Participants’ voices captured in WADA projects, and a workflow to enable their comments to actually influence practice and project planning
• Methodology shared with the Catalyst community, and seeing other Catalyst projects adopt and even adapt it
• Ultimately, that this project influences Catalyst’s whole evaluation/audit process, thus making it easier for qualitative, educative, community engagement and similar projects to feel at home in Catalyst, because the audit process supports them and recognises and records their achievements.

It is a new project, and also a new collaboration between QA-DAO and WADA.

Note that WADA is currently overhauling its audit process in general, as indicated by the proposal "Auditing African Proposals" https://cardano.ideascale.com/c/idea/396595; so Qualitative Audit Pilot will liaise with this overall project and feed information into it.