Oracle Performance Metrics

The Plan
Use the Cardano blockchain to periodically store performance meta data for deployed services such as oracles, and machine learning algorithms.
Create a tool capable of querying this meta data and aggregating metrics to produce a performance summary of automated services.
Publicly accessible via the blockchain and appropriately signed for authenticity, this meta data can be aggregated and summarised over a time series to assess reliability and utility of deployed services, providing added transparency, monitoring and validation throughout the system.
This meta data and metrics could be used by organisations who maintain oracles as well as by communities who maintain decentralised oracles. Furthermore, the communication of performance metrics amongst machine-learning algorithms enables reinforcement learning use cases and distributed AI a la SingularityNET.
Some background and rationale for the proposal can be found here: https://flyncd.github.io/
The Team
Each team member brings over a decade of experience as Data Science professionals responsible for building, assessing, deploying and monitoring credit risk, fraud, sentiment analysis, graph analytics and propensity models.
This is the team's first Catalyst proposal. Team members have sufficient experience to deliver all five proposal deliverables within a single calendar month without sourcing additional developer cost.
Team Members
Michael - https://www.linkedin.com/in/michael-hodder-417a4ba7/
Thushare - https://www.linkedin.com/in/thushare/
Steve - https://www.linkedin.com/in/steve-pirois-8b373593/
The Deliverables
Define generic assessment criteria for automated services
Performance reporting should be as flexible as possible to support different use cases (data feeds, price oracles, Machine Learning bots, risk models) while providing enough meaningful information that can fit within the 16k meta data limit.

• Deliverable #1 - Extend the schema.org SolveMathAction and MathSolver schemas to create a standards-based structure for or registration and assessment of automated services.
  Choose a specific application to create a proof of concept reporting tool
  There is a great collection of dApps currently being built on the Cardano ecosystem. The qCreditScore proposed by Liqwid Finance, is a good example of an automated service where users would desire periodic assessments of the score's predictive-power, accuracy, stability and calculation. Adding this layer of transparency and validation to the DeFi system would serve to promote take up and usage.
• Deliverable #2 - Create a simulated demo of a "Credit Risk" model which is capable of assessing the likelihood that an entity (such as an Atala Prism Decentralised Identity) will default in the next 3 months.
• Deliverable #3 - Produce a script to periodically collate and publish metrics. The summary of which is published as a blockchain transaction.
  Demonstrate reporting of metadata
  Reporting on stored metadata, particularly using metrics that can be assessed over long time periods is where the long-term utility of a proposal such as this will help the Cardano community.
• Deliverable #4 - Provide a tool which can query and aggregate metadata stored on the Cardano blockchain that targets a specific metric (eg Gini or KS for the qCreditScore model). The tool should be portable enough so that any user capable of querying the underlying Cardano blockchain metadata can produce reports.
  Report on findings
  Once the project is finalised, write up the results and publish to the project website.
• Deliverable #5 - Add a supplementary section to project website detailing lessons learnt, unsolved questions and future work.
  FAQ
• Why is the blockchain an appropriate store of this information?
  A decentralised, transparent reporting of performance metrics for oracles on a blockchain ensures authenticity of the performance data. Transparent reporting and verification of these metrics over time can produce an authentic permanent record which can be assessed using time series analysis.
• What data will be used to build the credit score simulation?
  Peer-to-peer lending data. For example, the Udacity dataset for Prosper.
• What will you use the requested funds for?
  Funds to be used as remuneration for one month of development effort, standing up development infrastructure for testnet and a mainnet demonstration application.
• What is the desired outcome of this project?
  The goal is to see the framework and tool being used amongst the Cardano community, ideally making future projects possible and successful. In the medium to long term, we hope to help the community to develop future projects and/or improve tools by releasing details about the successes, pinch points and other lessons learnt during the project development.