Scarab - a cardano-node client implementation in functional Scala

[GENERAL] Name and Surname of Main Applicant

James Aman

[GENERAL] Email address of Main Applicant

jim.a.aman@gmail.com

Additional Applicants

N/A

[GENERAL] Please specify how many months you expect your project to last (from 2-12 months)

12

[GENERAL] Please indicate if your proposal has been auto-translated into English from another language.

No

[GENERAL] Does your project have any dependencies on other organizations, technical or otherwise?

No

[GENERAL] If YES, please describe what the dependency is and why you believe it is essential for your project’s delivery. If NO, please write “No dependencies.” in this field.

No dependencies.

[GENERAL] Will your project outputs be fully Open Source?

Yes

[GENERAL] If NO, please describe which outputs are not going to be open source. If YES, please write “Project will be fully open source.” in this field.

Project will be fully open source.

[METADATA] Category of Proposal

Node

[IMPACT] Please describe your proposed solution.

• The eventual goal of Scarab is to provide an alternative block-producing node implementation to improve upon the decentralization and security of the Cardano network. As noted in SoK: A Stratified Approach to Blockchain Decentralization, a network's resilience to implementation bugs and maintainer malfeasance is reduced when there exists only a limited number of full block-producing node implementations.
• Scarab does not seek to replace the Haskell cardano-node implementation or to fork the Cardano network, but rather to be a compatible and complementary alternative for stake-pool operators and enterprise organizations. Additionally, development of such a block-producing node would necessitate community-led specifications of functionalities (CIPs) and provide further insight regarding the implementation details within the application.
• As a first step to a block-producing node this work proposes creation of a client node, written in functional Scala, that can interact with network peers via the node-to-node protocols and users via the node-to-client protocols, maintain and extend the block history via bootstrapping and block validation, and update ledger state via transaction execution throughout each era. 
• Such a node will be able to act as a backend for user wallets and be capable of being bundled with complex off-chain applications that benefit from direct exposure to the blockchain state.

[IMPACT] How does your proposed solution address the challenge and what benefits will this bring to the Cardano ecosystem?

• Development of a Scala-based client node would directly address the “Development & Infrastructure” challenge in several ways:
• Alternative node benefits
• Increase overall decentralization of the Cardano network by developing an open-source node application independent of IOG.
• Provide a route for community-led development of a complete set of formal, language agnostic specifications to facilitate interoperability between Scarab, cardano-node, and other applications / libraries.
• Explore innovation on the design of node sub-systems such as storage.
• Functional Scala benefits
• Offers high concurrency, scalability, performance, robust static typing, and expressive syntax.
• Part of a mature JVM ecosystem with extensive libraries, IDEs, debuggers, and opens up a vast pool of resources, tools, and developers.
• Interoperability with JVM languages like Java, Kotlin, and Clojure increases its accessibility for node integrations and enterprise adoption.
• Code can run on any platform that supports JVM and additional cross-platform support is possible via GraalVM or Scala Native.
• Promotes safe and reliable code through features such as immutability and side-effect free operations.

[IMPACT] How do you intend to measure the success of your project?

• A successful proposal would result in diversification of operating client-nodes on the Cardano network via adoption of the Scarab binary for use in a decentralized application or wallet. Achieving even 1% market penetration by year-end 2024 with the client node implementation would be a considerable achievement given the current monopoly of the Haskell implementation.
• In the short-term, an alternative client will provide a means of expanding the reach of Cardano to the JVM ecosystem. While in the long-term, building on this work to create a block-producing node will contribute to trust in the resiliency and decentralization of the network.

[IMPACT] Please describe your plans to share the outputs and results of your project?

• All code will be open-sourced and available via Github for comment and review
• The project will run on a two-week sprint cadence so progress reports will be compiled, posted to Github, and shared via Twitter and Discord following each sprint.

[CAPABILITY/ FEASIBILITY] What is your capability to deliver your project with high levels of trust and accountability?

I have extensive experience developing blockchain applications in Scala having previously developed a full blockchain application in Scala based on the Akka actor model, led the research and evaluation of an Ouroboros derivative to improve block production guarantees, and spearheaded development of a greenfield functional Scala blockchain project that implemented Nakamoto proof-of-stake and an extended-utxo ledger functionality. Furthermore, several of the interfaces and implementations from my previous open-source project, such as the network mini-protocols, multiplexer, and storage design, may be ported to Scarab as they were originally inspired by Cardano specifications and design constraints. Finally, this proposal will also seek to leverage Scalus, another Fund10 open-source proposal, for implementation of the Plutus runtime within the ledger layer. Scalus is an independent effort by Alexander Nemish to provide “a Scala implementation of Cardano Plutus, PlutusTx, including UPLC evaluation machine”. From these starting points, a Scala client implementation is feasible and imminently achievable in the proposed time frame.

[CAPABILITY/ FEASIBILITY] What are the main goals for the project and how will you validate if your approach is feasible?

• The primary objective is to develop a client node implementation in functional Scala that is robust, performant, and compatible with the Haskell cardano-node. Compatibility will be evaluated along several axes including (1) network communication and data transfer, (2) chain extension and header / block validation, (3) transaction execution and utxo state management, (4) integration in multi-client testnet environments.
• A secondary goal will be to ensure Scarab is capable of processing transactions and blocks efficiently and reliably. This will be measured by compiling performance metrics such as transaction throughput, block verification time, sync speed, and resource usage (CPU, memory).
• A tertiary goal is the development of language agnostic formal specifications for aiding the development of future alternative node implementations.

[CAPABILITY/ FEASIBILITY] Please provide a detailed breakdown of your project’s milestones and each of the main tasks or activities to reach the milestone plus the expected timeline for the delivery.

• The following milestone total a 50 week or approx. 12 month project duration.
• Milestone 1: Network layer functionality
• Projected timeline: 8 weeks
• Components 
• Mini-protocols & state machines - 2 weeks
• Over-the-wire serialization - 1 week
• Node-to-client & node-to-node protocols - 3 weeks
• Multiplexer - 1 week
• Peer connection manager - 1 week
• Milestone 2: Storage (on-disk and in-memory data structures / interfaces)
• Projected timeline: 8 weeks
• Components
• On-disk serialization - 1 week
• Consensus database(s) - 3 weeks
• Ledger database - 2 weeks
• Mempool - 1 week
• Candidate chain & garbage collection - 1 week
• Milestone 3: Consensus layer functionality
• Projected timeline: 12 weeks
• Components
• Header & block validation - 2 weeks
• Chain extension & ledger interface - 2 weeks
• Chain selection & rollbacks - 2 weeks
• Epoch management - 2 weeks
• Bootstrapping - 2 weeks
• Hard fork combinator - 2 weeks
• Milestone 4: Ledger layer functionality
• Projected timeline: 10 weeks
• Components
• Chain state & UTxO management - 4 weeks
• Virtual machine integration - 2 weeks
• Transaction execution & validation - 4 weeks
• Milestone 5: Application packaging and testing
• Projected timeline: 12 weeks
• Components
• Configuration - 2 weeks
• Logging & monitoring - 2 weeks
• Integration testing - 4 weeks
• Network integration & benchmarking - 3 weeks
• Command-line interface - 1 week

[CAPABILITY/ FEASIBILITY] Please describe the deliverables, outputs and intended outcomes of each milestone.

• This project will be managed via a Kanban methodology on a two-week sprint cadence. Sprint reports will be compiled at the conclusion of each sprint and published to Github.
• Code modules will be considered complete if there are accompanying unit and integration tests that demonstrate compatibility with published specifications and test vectors.
• Milestone 1: Network layer compatibility
• Deliverable: A set of network libraries, unit tests, and integration tests will be published to Github which are compatible with the protocols as specified in the Shelley network specification.
• Milestone 2: Storage
• Deliverable: A set of storage libraries, unit tests, and integration tests will be published to Github which present a compatible interface for Consensus and Ledger operations as outlined in the Consensus and Storage report
• Milestone 3: Consensus layer functionality
• Deliverable: A set of consensus libraries, unit tests, and integration tests will be published to Github which allows for block fetching, header validation, and bootstrapping of a new node.
• Milestone 4: Ledger layer functionality
• Deliverable: A set of ledger libraries, unit tests, and integration tests will be published to Github which implement the transaction execution and ledger update mechanisms across the Byron, Shelley, Mary, Alonzo, and Babbage eras. 
• Milestone 5: Application packaging and testing
• Deliverable: A packaged binary will be published to Github that is compatible with the Haskell cardano-node implementation demonstrated by engaging in multiple rounds of testnets.

[RESOURCES & VALUE FOR MONEY] Please provide a detailed budget breakdown of the proposed work and resources.

Projected work hours (50 weeks @ 40 hours / week) - 2000 hours

Hourly rate (inclusive of taxes and other expenses) -- $75 / hour

Approximate Ada price for conversion --------------- $0.30 / Ada

Total budget ----------------------------------------- 500,000 Ada

[RESOURCES & VALUE FOR MONEY] How does the cost of the project represent value for money for the Cardano ecosystem?

This is an ambitious project for a single developer and is only feasible given the high-level of code that may be integrated or reused, the similarity of functional Scala and Haskell, and my own close familiarity with the Cardano ecosystem. Even with those caveats, this proposal is projected to be a full-time year long endeavor but one which is of great importance to achieve the targeted goal of community ownership and decentralization.

I believe this proposal offers excellent value for the investment considering the expertise involved, tax and self-employment overhead, and the current volatility of cryptocurrency markets.

[IMPORTANT NOTE] The Applicant agreed to Fund10 rules and also that data in the Submission Form and other data provided by the project team during the course of the project will be publicly available.

I Accept