Catalyst QA Automation scripts

TL:DR: These scripts are a vital part of the system as it currently operates. We have built them to help take some of the manual load off IOG staff. They take time and resources to develop and maintain and we would like the community to support their continued development.

Detailed Explanation:

In the Project Catalyst process there is a phase (Quality Assurance Stage) that allows Proposers and VCAs to review the ratings and assessments done by the Community Advisors.

The assessments are evaluated using the criteria defined in the Red & Yellow card document (link here).

A Community Advisor could get a "yellow card", which will cause the exclusion of the specific assessment, a "red card", which will exclude all his assessments or positive "Constructive Feedback", reserved for assessments that comply with the guidelines and add value to the process.

Here is an overview of the process:

1. After the Community Advisors submission, the proposers have some days to report the assessments that they consider unfair or out of the guidelines, adding a rationale to their report, or to highlight assessments with "Constructive Feedback".
2. vCAs receive a master file that includes all the proposers' reports and have some day to flag assessments out of guidelines, fair, or with "Constructive Feedback".
3. The vCAs reports are aggregated and an algorithm is run to determine consensus on a decision for each specific assessment (at the moment 4 vCAs have to be involved in the decision and more than 50% of vCAs have to agree on a specific infringement criteria).
4. Yellow and Red cards are assigned to Community Advisors and invalid assessments are excluded.

This process is currently possible thanks to the implementation of the QA scripts, run by the IOG team, that create Master Files for each stage, validate inputs, calculate the consensus and produce the results used for the final ratings. It's a technical but it's very important to the current operation of the system.

The current implementation (used in Fund 5) consists in 4 scripts:

1. Create Proposer Document
   Based on the IdeaScale export is created a Google Sheet document with all the criteria defined in the R&Y card document. Proposers use this document to report assessments.
   Blank assessments are automatically marked.
2. Create VCA Master File
   Based on the output of the Proposer Document, a vCA Master File is created. It includes all the criteria defined in the R&Y card document and the new column "Proposer mark" which indicates if proposers reported specific assessments.
   The first rule of red card assignment is executed in this script: it counts the percentage of blank assessments made by every single CA and assigns a red card if the defined threshold is reached, excluding all his assessments.
   vCAs have to make a copy of this file and review the assessments.
   The output produced by this script is totally compatible with the vCA tool (included in Community Tools proposal funded in F5), offering to vCA a more user friendly way to interact with this process.
3. Create Similarity
   This script compares all the assessments to each other and assigns a similarity score for each combination. The output of this file can be used by vCA to identify bad actors and copy-pasted assessments.
4. Create VCA Aggregate
   All the single files produced by vCAs are aggregated in this script, and the consensus rule is executed for every assessment, assigning yellow and red cards to CAs.
   This script produces the final valid assessments list that are used to calculate the final ratings.

New features that will come in Fund 5:

• Use the same mechanism currently used by vCAs (create a Master File, allow vCA to create a copy and submit it to IOG) will be implemented also for Proposers, avoiding working on a single shared Google sheet. This will allow the release of the qa-proposer tool, which will offer a more user-friendly way for proposers to interact with this stage.
  This implementation will also give more control to IOG to check that proposers flag only the assessments of their own proposals.
• Adapt the scripts for the new rewarding model for CAs.
  A full update to the criteria used by vCA to classify CA reviews in Excellent / Good / Not Valid and to the consensus mechanism.

Part of this proposal is an experiment in funding already implemented value add. The funds requested for this proposal will cover the current implementation (the scripts that have been in use since Fund 4) as well as be used to maintain and adapt the scripts as changes are introduced into the Catalyst QA process until the end of 2021 (Fund 7 included).

Technical Specs

The scripts are written in Python and they use Pandas Dataframe framework to interact with the original IdeaScale export (provided by IOG as CSV). They are released as open source code on Github (https://github.com/coire1/catalyst-review-of-reviews-tools).

They also interact with the Google Docs API to publish the output as Google Sheets with some styling to improve the sheet UI.

Experience

Developer with 9+ years of experience (mainly focused on web/mobile. Languages and frameworks: PHP, Javascript, Typescript, Python, Ruby - Laravel, Ruby on Rails, Wordpress, Angular.js, Vue.js, Node.js). CA and vCA, co-creator of the Community Tools (Proposer/CA/vCA/Voter Tools).

Budget breakdown

• Initial development for F4: ~30 hours
• Data processing for F4: ~5 hours
• Testing: ~5 hours
• Updates for F5: ~15 hours
• Data processing for F5: ~2 hours
• Testing: ~2 hours
• Updates for F6: ~10 hours
• Data processing or F6: ~2 hours
• Testing: ~2 hours

Cost per hour: 50$
Total hours: 73
Total: 73 x 50$ = 3650$

KPIs

• Number of CAs (of which the assessments were processed)
  Fund 4: 213
  Fund 5: 258
• Number of vCAs (of which the reviews were processed)
  Fund 4: 20
  Fund 5: 42
• Number of CA assessments processed
  Fund 4: 7142
  Fund 5: 8494
• Number of CA assessments excluded (blanks or card assigned)
  Fund 4: 2661
  Fund 5: 3421
• Number of cards assigned:
  Fund 4: 514
  Fund 5: 768
• Number of "Constructive Feedback" collected (cumulative):
  Fund 4: 1111
  Fund 5: 2589

The same criteria could be used to evaluate the impact of the scripts for F6.