Dolos - A step closer to a Rust node - Phase-1 Validations

[GENERAL] Name and surname of main applicant

[GENERAL] Email address of main applicant

Additional applicants

federico@txpipe.io

[GENERAL] Please specify how many months you expect your project to last (from 2-12 months)

[GENERAL] Please indicate if your proposal has been auto-translated into English from another language.

[GENERAL] Does your project have any dependencies on other organizations, technical or otherwise?

[GENERAL] If YES, please describe what the dependency is and why you believe it is essential for your project’s delivery. If NO, please write “No dependencies.” .

No Dependencies

[GENERAL] Will your project’s output/s be fully open source?

[GENERAL] If NO, please describe which outputs are not going to be open source. If YES, please write “Project will be fully open source.”

Project will be fully open source.

[METADATA] Category of Proposal

[IMPACT] Please describe your proposed solution.

Context

• Having node diversity (more than one implementation of the Cardano node) will make the whole ecosystem more resilient.
• A full node is a major endeavor, we estimate it to be a multi-year project with multidisciplinary engineers on the team.
• Given the magnitude, we consider that our team wouldn’t be able to achieve the goal within the context of a single Catalyst proposal.
• A Divide-and-Conquer strategy can be used to split the project into several pieces, each one achievable as stepping stones for a final implementation some time from now.
• The “Dolos” project provides a lightweight implementation of a node which is focused solely on basic data tasks (keeping track of the chain, accessing UTxO state, serving data to client applications).

Scope of the Proposal

• Slowly build on top of “Dolos” new features, each one bringing us one step closer to a full implementation.
• Maintain a flexible architecture that allows different flavors of the end application so that new features don’t compromise the lightweight characteristics of Dolos.
• As the next big feature, we plan on adding the business logic to perform “Phase-1 Validations” of the block transactions, providing as benefit a higher level of self-accountability of the chain correctness.

Technical Implementation

• Roughly speaking, Phase-1 validation is defined as all the checks required for a transaction to be added to the ledger excluding the execution of Plutus scripts. Nodes are not compensated for this work, and transactions that fail this validation are rejected without being included into a block and without paying any fee or collateral.
• All the validation rules are specified and implemented in the Cardano Ledger. The specification is splitted into several documents, one for each era, describing incrementally the modifications and additions that each era introduced. For each era there is also as part of the specification a CDDL file that precisely defines the CBOR schema used for blocks and transactions. For any aspect of the rules not specified in the documentation, the implementation of the Cardano Ledger in Haskell serves as a reference.
• To obtain a comprehensive and precise definition of the Phase-1 validation rules, it is necessary to perform a complete reverse engineering work that requires studying the specification documents for all the eras (Byron, Shelley, Mary, Alonzo and Babbage) and the Haskell source code of the ledger implementation.
• The big set of Phase-1 rules can be divided into groups according to the aspects they refer to, and to the contextual information required to check them. We arbitrarily choose to organize the rules into the following categories:

1. Basic transaction structure: CDDL compliance, integrity hashes, size limits, etc.
2. Basic ledger rules: input UTxOs existence, current slot in validity interval, etc.
3. Fees and collateral: correct amounts according to protocol parameters.
4. Output value: input/output balance, collateral balance, min lovelace per output, etc.
5. Witnesses: required signatures, scripts, redeemers and data.
6. Phase-1 scripts: native script execution.
7. Staking: credentials and withdrawals, etc.
8. Other: protocol parameter update proposals, bootstrap witnesses, etc.

• For this proposal, we plan to implement the validation rules in the first five categories (a, b, c, d and e). We understand that this set of rules accounts for an important part of the current ledger and that, given the budget constraints, it is a reasonable increment in the path to a fully functional Rust node.
• With this implementation, we will be able to synchronize all the public networks (mainnet, preprod and preview) with an important coverage of the ledger rules. This will open the road to further iterations where the remaining aspects of phase-1 validation can be completed and phase-2 validation can be fully implemented.

[IMPACT] How does your proposed solution address the challenge and what benefits will this bring to the Cardano ecosystem?

As a general concern and an indirect, long-term effect of this proposal, node diversity provides several benefits to the ecosystem:

• Resilience against vulnerability and bugs. In a single-implementation ecosystem, any zero-day vulnerability or severe bug could be “catastrophic” to the whole network.
• Resilience against organizational issues. In an ecosystem where the development of the node is handled by a single team, any organizational issues (developer rotation, shifts in priority, budget limitation, etc) can have drastic consequences on the evolution of the protocol.
• Foster different and novel approaches to existing problems. Independent teams or different tech-stacks will yield different ways of solving the same problems, potentially encountering better ways to solve already implemented concerns.

There’s also several new problems that arise from node diversity, like potential incompatibilities between implementations. We won’t discuss them as part of the proposal, but these are certainly issues that need to be addressed once a complete, full implementation of a second node is achieved.

On a more direct relationship, this proposal has short-term benefits too:

• Improve the level of self-accountability that Dolos has over the data that it receives from upstream nodes. Currently, Dolos needs to “trust” the upstream node providing the data. Phase-1 validations
• Generate detailed documentation (potentially a spec) of how Phase-1 validations works, which could be used by other teams in different node implementations.
• Abstract Phase-1 validations into a Rust library that can be embedded into other Rust-based tooling, allowing Phase-1 validations to be executed offline, without the need for a node (eg: Aiken, Lucid).
• Create a testbench for black-box testing of a ledger implementation in a way that it can be used for any current and future implementations.

[IMPACT] How do you intend to measure the success of your project?

We consider the following dimensions for measuring the success of the project:

• Activity in the open-source Github repository through metrics such as, but not limited to: # of issues, clones, external contributors, stars, visitors, etc.
• Number of downloads of binary releases.
• Web traffic on documentation site.
• Number of external repositories that include this project as a dependency.

[IMPACT] Please describe your plans to share the outputs and results of your project?

Being an open-source project, the outputs will be available to any developer in the ecosystem at every step of the development process:

• Latest version of the source-code will be available in the Github repository.
• Source code changes will be applied through a pull-request process.
• Alpha and Beta versions will be released at every milestone.

Upon reaching the end of the development process, we’ll provide:

• A LTS release version of the engine available to download in many formats (binary, Docker image, etc)
• A CLI (command line interface) binary to serve as entry point for developers
• A documentation website with instructions for usage and deployment
• An executable testing framework to run Phase-1 validation unit-test on the provided code.
• A Rust library that can be used to evaluate Phase-1 validations over arbitrary transaction data.
• A tutorial video showing a walkthrough of how to run Dolos in different environments and configurations.

[CAPABILITY/ FEASIBILITY] What is your capability to deliver your project with high levels of trust and accountability?

TxPipe is very proud of their past and current contributions to the ecosystem. Just to mention a few:

• We have developed “Pallas”, a Rust library for Cardano which is used by several high-profile project in the community (such as: cncli and Aiken)
• Through Catalyst, we have developed and delivered “Oura”, an off-chain data integration tool for Cardano used by many projects in the community (such as: Pool.io and dcSpark’s Carp, etc).
• Through Catalyst, we have developed “Dolos”, a minimalistic version of the Cardano node which is being slowly rolled out to the community as a beta version.
• We have developed “Demeter”, a cloud hosting platform for Cardano infrastructure with several high-profile clients (such as: JPG.store, SummonPlatform and TeddySwap).

The above are examples of our accountability because:

• It shows our commitment to evolving the open source community and the Cardano ecosystem.
• It shows the technical expertise required to develop and maintain Cardano infrastructure.
• It shows our commitment to the Catalyst program and its required procedures.

[CAPABILITY/ FEASIBILITY] What are the main goals for the project and how will you validate if your approach is feasible?

Our short-term direct goal is to increase the level of self-accountability of Dolos by implementing the business logic required to perform Phase-1 validations of the transactions data.

Our long-term, indirect goal is to get us closer to a Rust implementation of a full Cardano node that will improve the overall resilience of the ecosystem.

[CAPABILITY/ FEASIBILITY] Please provide a detailed breakdown of your project’s milestones and each of the main tasks or activities to reach the milestone plus the expected timeline for the delivery.

Milestone #1: Reverse Engineering (1 months)

• Evaluate available specs and documentation for ledger rules
• Evaluate existing Haskell implementation of the ledger module
• Document findings and definition of required rules

Milestone #2: Business logic implementation (2 months)

• Define module interface for ledger rule execution
• Prepare code scaffold for all required rules
• Define unit-tests for all required rules (all red)
• Implement business rules until all unit-tests pass (all green)
• Release testing framework documentation 

Milestone #3: Dolos integration (1 month)

• Connect ledger rules into the existing Dolos data pipeline
• Execute full passes over Mainnet, PreProd and Preview
• Capture performance benchmarks and compare against previous Dolos version
• Release new versions of Dolos

Milestone #4: Tooling integration (1 month)

• Abstract ledger rules as a reusable Rust library in Pallas repo
• Implement a CLI (command line interface) for evaluating arbitrary transactions
• Release binary versions of the CLI

[CAPABILITY/ FEASIBILITY] Please describe the deliverables, outputs and intended outcomes of each milestone.

Milestone #1: Reverse Engineering

• Outcome: Understand the existing ledger rules and their Haskell implementation for successful replication and improvement.
• Outputs: documentation for ledger rules, assessment of existing Haskell implementation of the ledger module, definition of rules to be implemented

Milestone #2: Business logic implementation (2 months)

• Outcome: An operational ledger rule execution module with all business rules passing their unit tests.
• Outputs: open-source code containing a ledger rule execution module, all unit-tests passing; including documentation for the testing framework.

Milestone #3: Dolos integration (1 month)

• Outcome: A fully integrated and improved Dolos data pipeline with enhanced self-accountability that has been successful executed over Mainnet, PreProd, and Preview networks.
• Outputs: open-source code with ledger rules integrated into the existing Dolos data pipeline; performance benchmarks; a new version of Dolos released.

Milestone #4: Tooling integration (1 month)

• Outcome: a Rust library that can be used by devs, either programmatically or through a CLI, to integrate Phase-1 validations on different contexts.
• Outputs: open-source code providing an implementation of the ledger rules as a reusable Rust library in the Pallas repo, and a functional CLI for evaluating arbitrary transactions.

[RESOURCES & VALUE FOR MONEY] Please provide a detailed budget breakdown of the proposed work and resources.

FTE = full-time equivalent

Values expressed in ADA (₳)

Breakdown by resource type:

• Rust developers: 1 FTE x 4 months = ₳ 128,571
• Haskell developers: 1 FTE x 1 months = ₳ 32,143
• Technical writers: 1/5 FTE x 5 months = ₳ 10,714
• Project manager: 1/5 FTE x 5 months = ₳ 10,714

Breakdown by milestone

• Milestone #1: ₳ 36,429
• Milestone #2: ₳ 72,855
• Milestone #3: ₳ 36,429
• Milestone #4: ₳ 36,429

[RESOURCES & VALUE FOR MONEY] How does the cost of the project represent value for money for the Cardano ecosystem?

All resources from the proposal will result in outputs which are either open-source code or public documentation. The developer ecosystem will gain a tangible results for their ADA investment.

The project costs have been thoughtfully assessed, taking into account factors such as complexity, expertise required, and resources invested. We've focused the proposal into a very narrow and feasible goal to ensure success.

Developers and the team involved this proposal offer a fair balance between quality and affordability within the Cardano ecosystem.

By slowly working towards the ambitious goal of having an alternative node implementation, the project aims to provide long-term value that outweighs the initial investment.

[IMPORTANT NOTE] The Applicant agreed to Fund10 rules and also that data in the Submission Form and other data provided by the project team during the course of the project will be publicly available.