Risk Monitor for SPOs

[GENERAL] Name and surname of main applicant

Kenric Nelson

[GENERAL] Email address of main applicant

kenric.nelson@photrek.io

Additional applicants

Blake Anderton, James Aman, Juana Attieh, Amir Azem, Megan Hess

[GENERAL] Please specify how many months you expect your project to last (from 2-12 months)

5

[GENERAL] Please indicate if your proposal has been auto-translated into English from another language.

No

[GENERAL] Does your project have any dependencies on other organizations, technical or otherwise?

No

[GENERAL] If YES, please describe what the dependency is and why you believe it is essential for your project’s delivery. If NO, please write “No dependencies.” .

N/A

[GENERAL] Will your project’s output/s be fully open source?

Yes

[GENERAL] If NO, please describe which outputs are not going to be open source. If YES, please write “Project will be fully open source.”

N/A

[METADATA] Category of proposal

SPOs

[IMPACT] Please describe your proposed solution.

Cardano seeks to achieve the most decentralized, secure blockchain network as the foundation for an antifragile financial operating system. To achieve this goal, it is essential that the SPOs are acting independently. If a multipool or a consortium of SPOs were to share slot leader selection information, that knowledge could be used to strategically withhold block production for the purpose of creating blockchain segments controlled by these Byzantine actors. Figure 1 shows the global block production predictions and experimental measurements. Collusion is difficult to detect because it can “hide” in the inherent uncertainty of Nakamoto proof-of-stake and result in smaller pools failing to mint their block assignments to the broader network.

Figure 1: Block Production

The SPO community needs tools and educational materials to locally monitor the accuracy of the slot leader probabilities. Local alerts are needed when the accuracy of the probabilities drops below historic trends. Furthermore, analytical tools are necessary to identify which cluster of nodes are attempting a Byzantine attack.

Photrek and its partners will develop a Risk Monitor that provides Cardano SPOs with

1. A histogram of the probability forecasts of the selected slot leader;
2. An accuracy metric of the forecasts based on the geometric mean of the forecasted probabilities;
3. Robustness and Decisiveness metrics that indicate the degree of variability in the forecasts
4. Educational resources to enhance knowledge within the SPO community regarding Byzantine attacks and how the Risk Monitor can mitigate the threat.

Figure 2 shows an example of the Risk Monitor plot. This example was of an image generator algorithm. The histogram can show the distributional character of the forecasts. The Accuracy shows the central-tendency on a log scale; the Robustness shows the outlier performance; and the Decisiveness shows the ability to forecast the correct class (slot leader). 

Cardano SPOs will be able to utilize the Risk Monitor to actively track the accuracy and robustness of the slot leader forecasts. This information can provide an early warning if slot leader selection and thus block production is being manipulated via block withholding or similar network distribution based attacks. The analysis will be completed across the network nodes and for individual nodes. By developing historical records of the R-A-D metrics, alerts based on significant deviations from historical trends can be created. These alerts can for instance be used to determine whether multipool operators are withholding release of blocks in effort to increase their probability of slot leader selection.

Figure 2: Risk Monitor

How we perceive the problem we are solving:

The problem at hand involves ensuring accountability among SPOs based on the honest behavior outlined by the theoretical predictions in the Ouroboros Praos paper. This involves enabling real-time analysis of each party's data dissemination across the network. The introduction of a numerical risk model would empower each SPO to conduct an empirical analysis of network health, thereby swiftly identifying any Byzantine behavior.

• Another concern that needs addressing is the possibility of multiple pool operators leveraging their size to impact the network negatively. An example of this issue can be found in an article on https://adapulse.io/multiple-stakepool-operators-are-harming-cardano/ 
• Furthermore, an enhancement we are considering is the provision of an "active" local calculation mechanism for block creation probability. This feature would greatly aid in monitoring the performance of the network.

Our reasons for approaching it in the way that we have:

• This approach takes into account real world network behaviour instead of theoretical estimates. Nakamoto proof-of-stake allows for a larger validator set (due to being more bandwidth efficient) at the cost of short-term statistical uncertainty. This probabilistic approach coupled with a dynamically available distributed network provides a path for dishonest SPOs to accrue greater than their fair share of rewards.
• Such an effect would be small and would be more easily found by having multiple SPOs analyzing the network concurrently, in real-time

Who our project will engage:

• The SPO community will be the primary target for engagement within this project. The team plans to connect with the community through dedicated SPO chat groups on Telegram and Discord, as well as by hosting breakout rooms. The team will introduce the proposed solution and reach out to SPOs who express interest, with the aim of collaborating on testing and validation of the solution.

How we will demonstrate or prove our impact:

• The impact of this solution will be evidenced through several steps:
• A complete testnet experiment to gauge the impact on probability accuracy. This will provide an understanding of how accurate the solution works in real-world conditions.  
• A demonstration of the sensitivity of the model to the accumulation of stake. This will provide insights on the model’s ability to handle changes in stake distribution and their impact on the network.

Explanation of our unique solution, who will benefit, and why this should be important to Cardano: 

• Our solution is unique because of its localized approach to pool performance evaluation. Ouroboros is unique among proof of stake blockchains by virtue of its probabilistic nature. Local calculation of the relevant statistics for a pool are vital to understand pool performance.
• Complex model due to a combination of network topology, game theory of information propagation, and leader election process is well suited to a (multi-vectored) risk analysis.
• This comprehensive approach will benefit all participating in the network, particularly SPOs by providing a more robust and transparent understanding of network health and performance. Underscoring why this should be of significant importance to the Cardano community.

[IMPACT] How does your proposed solution address the challenge and what benefits will this bring to the Cardano ecosystem?

This project will have a significant impact on and value for the Cardano ecosystem by

• providing a tool that promotes and ensures honest behaviour of SPO operators. This results in enhancements to the overall decentralization, trustworthiness and health of the network, thus adding value to the entire Cardano ecosystem.

The project will provide significant impact through various ways:

• By enabling empirical network analysis, it can optimize the propagation of information throughout the network, leading to enhanced efficiency and reliability.
• Helping ensure fair and honest block production which is essential for the ecosystem’s growth and strength and will attract both existing and new users to engage more deeply with the Cardano ecosystem.
• Providing real-time analysis of the likelihood that a string of events has occurred. This can be instrumental in identifying irregularities or potential issues within the system.
• Ex. what are the odds that pool A has created 10 out of the last 20 blocks?

[IMPACT] How do you intend to measure the success of your project?

Describe, and where possible measure, the benefits from the project to the Cardano ecosystem and why the measures are realistic in your opinion.

• We will aim to have 3 SPOs compare local measurements of network health using the Risk Monitor. These measures will demonstrate to the Cardano community the feasibility of having a decentralized measure of network health.

How will the innovation affect Cardano’s productivity and growth, in both the short and the long term?

• This tool will promote honest SPO block creation behaviours through the community accountability afforded by introducing locally computed probability metrics. In the short term, this may lead to improved trust within the community, as SPOs can be better held accountable. This could result in higher engagement and active participation within the network. In the long term, this will enhance the level of decentralization within the ecosystem, thus strengthening the reputation and integrity of the Cardano network, attracting more users and developers.

The metrics you measure do not necessarily have to be KPIs, which are numerical performance indicators - you can also measure qualitative things, such as the opinions of your users. 

• The Risk Monitor will be tested with existing SPOs whose feedback will be incorporated into iterative refinements to both improve functionality and facilitate understanding of the Risk Monitor metrics’ meaning and utility.

[IMPACT] Please describe your plans to share the outputs and results of your project?

What are your plans to spread the project’s outputs over a reasonable timescale? How and with whom will you share the outputs/impact and opportunities that will occur as a result of your completed project?

• Risk monitoring tools and educational materials will be uploaded to a publicly available repository readily accessible to SPO operators.
• Photrek will host Catalyst After Town Hall meetings to discuss project results.

How do you expect to use the results generated from the project in further research and development activities?

• The metrics produced in this effort provide immediate insight into empirical vs. forecasted block production probabilities in stake pools. These form the basis for extensions such as identifying questionable parties present among multiple stake pools or identifying susceptibilities arising from network topology.
• Our SPO Risk Monitor is part of a larger effort to provide Risk Assessment services. Photrek is hosting the general service on the SingularityNET AI Marketplace: https://beta.singularitynet.io/servicedetails/org/Photrek/service/risk-aware-assessment

[CAPABILITY/ FEASIBILITY] What is your capability to deliver your project with high levels of trust and accountability?

Please describe your existing capabilities that demonstrate how and why you believe you’re best suited to deliver this project?

• Photrek’s expertise in analysis of probabilistic forecasts is evidenced through originating concepts in research publications (e.g., On average uncertainty, Assessing Probabilistic Inference and applying those concepts to real-world problems (including analysing election forecasts and behaviour analysis in financial markets).
• Photrek’s team includes applicable expertise including (1) experienced Cardano stake pool operators and (2) researchers in Proof-of-Stake protocols’ eligibility thresholds.
• Photrek’s team includes significant software development expertise in creating a variety of tools with contextually-appropriate User Experience/User Interface.These span (1) real-time updatable graphical analytics to (2) terminal-based minimalistic utilities such as Text User Interfaces. This will be leveraged to develop tools supporting meaningful use-cases for stake pool operators. 

Please include your steps or processes which demonstrate that you can be trusted to manage funds properly. 

• Photrek plans to submit progress reports at regular (typically monthly) intervals, ensuring that expenditures are tied to developments which incrementally achieve program objectives.
• Photrek will host Catalyst After Town Hall meetings to discuss project results.

Photrek’s successful track record in prior Catalyst funds was facilitated by business processes including web-based time reporting and weekly virtual meetings to monitor progress.

[CAPABILITY/ FEASIBILITY] What are the main goals for the project and how will you validate if your approach is feasible?

• Develop SPO Risk Monitor tool suite: Develop software performing static event data analysis for SPO to evaluate and monitor block eligibility. This includes development of appropriate visualisation and user-interface. Suite will be iteratively refined in response to validation and usability objectives’ progress.
• Validate tool operability: Initially examine historical per-epoch block eligibility for presence of byzantine behaviour. Subsequently, extend to per-slot real-time analysis. Compare empirical block production frequency against Cardano protocol’s eligibility forecasts. Consider the prospect of evaluation of byzantine behaviour detection within an appropriate SPO emulation environment (e.g., a standalone or a test net).
• Validate tool usability: Engage at least 3 SPOs for feedback on tool utility and incorporate the feedback into our incremental development.
• Present capability to Cardano community: Upload open-source Risk Monitor suite to publicly-accessible software repository along with educational materials, use-case examples, and documentation artifacts.

[CAPABILITY/ FEASIBILITY] Please provide a detailed breakdown of your project’s milestones and each of the main tasks or activities to reach the milestone plus the expected timeline for the delivery.

The following milestone durations estimate an assumed 4 month total duration.

• Milestone 1: Stand-up initial Risk Monitor (end of month 1)
• Task 1.1: Gather/analyse block eligibility data from Cardano protocol.
• Task 1.2: Incorporate eligibility data into probabilistic risk metric computation.
• Task 1.3: Present computation results in an understandable/meaningful format.
• Milestone 2: Validate operability of Risk Monitor (end of month 2)
• Task 2.1: Evaluate tool operation on historical block eligibility (static event data).
• Task 2.2: Evaluate tool operation with real-time block production data.
• Milestone 3: Validate usability of Risk Monitor (end of month 3)
• Task 3.1: Gather/incorporate SPO feedback.
• Task 3.2: Extend tool capabilities to priority use-cases identified by SPOs.
• Milestone 4: Present Risk Monitor tools and documentation (end of month 4)
• Task 4.1: Prepare educational documentation on risk metric usage/interpretation.
• Task 4.2: Upload developed software to publicly-accessible repository.

[CAPABILITY/ FEASIBILITY] Please describe the deliverables, outputs and intended outcomes of each milestone.

Milestone 1: Stand-up initial Risk Monitor 

• Deliverable: baseline repository-maintained implementation of Risk Monitor tool suite.
• Outputs: privately-accessible version-controlled baseline software implementation
• Outcome: stand-up of initial Risk Monitor capability

Milestone 2: Validate operability of Risk Monitor

• Deliverable: updated repository including operability scripts, data, and test tools.
• Outputs: initial evaluation findings from historical and emulated byzantine detection.
• Outcome: confirmation that tools operate in manner satisfying project objectives.

Milestone 3: Validate usability of Risk Monitor

• Deliverable: documented findings and capability updates from SPO feedback.
• Outputs: findings incorporated into a section of regular progress status report.
• Outcome: Risk Monitor operation confirmed as meaningful/useful to SPOs.

Milestone 4: Present Risk Monitor tools and documentation

• Deliverable: updated implementation within publicly-accessible open-source software repository.
• Outputs: Risk Monitor software, installation instructions, educational material, and testing scripts.
• Outcome: operational Risk Monitor capability and documentation in a format suitable for usage within the wider Cardano community.

[RESOURCES & VALUE FOR MONEY] Please provide a detailed budget breakdown of the proposed work and resources.

[RESOURCES & VALUE FOR MONEY] How does the cost of the project represent value for money for the Cardano ecosystem?

Photrek brings together an exceptional team of scientists, engineers, and managers. Kenric Nelson leads the team, while Blake Anderton, Juana Attieh, and Megan Hess serve on a Leadership Circle. For this project James Aman, Teaching Professor at Rice University, brings unique expertise having used the Ouroboros protocol to build the Topl Blockchain.

We price our projects at competitive rates while building teams of top talent. The rates are based on self-employment in the US & Canada. The rates that take into account the employment overheads of the resources contracted. The amounts are calculated for each milestone based on the hours to complete. For example the chart for engineering and scientific salaries in the Commonwealth of Massachusetts is provided here:  https://www.mass.gov/guides/salary-and-compensation.

In addition all the resources working on this project are taking on the currency risk of being paid in ADA. This means that a fall in the ADA price will result in being paid less or delivering less in each milestone. Any rise in the ADA price will represent a reward for investing in the Cardano ecosystem.

Consequently, given these factors, we believe this proposal offers excellent value for money in a volatile cryptocurrency environment.

[IMPORTANT NOTE] The Applicant agreed to Fund10 rules and also that data in the Submission Form and other data provided by the project team during the course of the project will be publicly available.

I Accept