Ouroboros network (NtC) in Golang

[GENERAL] Summarize your solution to the problem.

Create/develop an open source Golang library for speaking the Ouroboros protocol with a Cardano node.

[GENERAL] Summarize your relevant experience.

Cloudstruct operates the CSCS staking pool and has years of open source development experience. We already have multiple functional open source projects using Golang to interface with a Cardano node in various ways.

[IMPACT] Please describe your proposed solution.

Create a Golang library to make it easy to write applications that interact directly with a Cardano node. This library will implement support for the Ouroboros network protocol, including the various mini-protocols. This library will eventually fully support both the NtC (node-to-client) protocol (using the UNIX socket to talk to the local node) and the NtN (node-to-node) protocol (used for nodes talking to each other via TCP) as both a client and server, but this proposal is limited to client support for all NtC protocols. This will allow for things like querying the tip, submitting a transaction, or syncing the blockchain from a local node, using Golang code without shelling out to “cardano-cli” or needing to maintain another piece of infrastructure (such as cardano-submit-api, cardano-db-sync, or Ogmios).

[IMPACT] Please describe how your proposed solution will address the Challenge that you have submitted it in.

Golang is an extremely popular programming language. GitHub Language Statistics from GitHut 2.0 (code: https://github.com/madnight/githut) show that in Q4 2021, the number of pull requests on Golang repositories grew 8.161% and Haskell repositories grew 0.180% (source: https://madnight.github.io/githut/#/pull_requests/2021/4/Go,Haskell) while the number of repository stars on Golang repositories grew 11.763% and Haskell repositories grew 0.178% (source: https://madnight.github.io/githut/#/stars/2021/4/Go,Haskell). The Golang developer community is approximately 4x the size of the Haskell developer community, based on the number of Github repos with code for each. Since it’s hard to judge community size, we’re using these numbers as a proxy. We hope to bridge the gap between Golang developers and the Cardano blockchain and attract more developers to Cardano.

Creating a Golang library for interacting with Cardano nodes will allow tapping into a much larger pool of developers than the existing Haskell libraries generally allow for. This will also lower the barrier for entry into the Cardano ecosystem.

[IMPACT] What are the main risks that could prevent you from delivering the project successfully and please explain how you will mitigate each risk?

One of the biggest challenges that this project will face is the lack of documentation for particular parts of the Ouroboros network protocol outside of the official Haskell code. This usually requires reading through the Haskell code and/or combing through packet captures to determine how something is supposed to work.

Another challenge comes from Cardano using maps with bytestring keys in CBOR-encoded data (such as in block bodies, transactions, and, LocalTxSubmission refusal reasons), which cannot be natively represented in Golang. This can be mitigated with some foreknowledge of the encoded data structure and/or some custom CBOR parsing code (on top of the general “fxamacker/cbor” library).

[FEASIBILITY] Please provide a detailed plan, including timeline and key milestones for delivering your proposal.

The initial project has already been created and is released under the Apache 2.0 license.

GitHub: https://github.com/cloudstruct/go-ouroboros-network

This library will provide a Golang analog for the Haskell ouroboros-network library. It will support protocol multiplexing over a single connection using a muxer, a handshake with protocol version negotiation, and various mini-protocols with their messages and state machines as documented in the Ouroboros network spec (https://hydra.iohk.io/build/15393629/download/2/network-spec.pdf).

This proposal covers the completion of support for the client side of all documented NtC (node-to-client) protocols. This will allow someone to write a client in Golang that allows communicating with a Cardano node via the UNIX socket and performing operations such as submitting a transaction, syncing blocks from the chain, and querying the local state of the node for things like the current tip and UTXO information.

Milestone 1 was the initial implementation of this library and any work that has been done up to this point. This included:

• Lots of research into the way that the Ouroboros protocol works
• Creation of a generic multi-protocol muxer framework
• Implementation of basic CBOR message parsing for each mini-protocol
• Support for a generic per-protocol state machine
• Support for protocol version negotiation via handshake for NtC and NtN protocols

Milestone 2 is the focus of this proposal. This will include:

• Splitting each mini-protocol into separate initiator (client) and responder (server) objects with their own copies of the state machine
• Research queries and responses supported by different versions of the LocalStateQuery mini-protocol
• Implementation of helper functions for initiating queries against the local node state and parsing the responses
• Refactoring of mini-protocol callback functions to be for information gathering and response sharing rather than for specific messages supported by the mini-protocol. This will also include refactoring the mini-protocol entry points to start a process that will be driven by the mini-protocol itself rather than the library consumer
• Implementation of unit tests for NtC mini-protocol message encoding/decoding to and from CBOR
• Improve handling of TX submission refusal reasons to avoid CBOR parsing issues for maps with bytestring keys
• Expand in-code documentation of various objects and functions for display on pkg.go.dev
• Creation of standalone examples of each NtC mini-protocol

[FEASIBILITY] Please provide a detailed budget breakdown.

Milestone 1 has been completed without any funding. This took approximately 200 engineering hours to complete.

Milestone 2 completion is estimated to take 100 engineering hours to deliver.

This estimate is multiplied by a rate of $100 USD/hour. We are asking for compensation for both the already completed milestone and the in progress milestone. Prior work can be verified as already delivered, providing assurance that we can continue to deliver.

[FEASIBILITY] Please provide details of the people who will work on the project.

The project development will be completed by the CloudStruct engineering team. Our team includes Principal / Staff Engineers, with more than 60 years combined experience. This project will be primarily assigned to Andrew Gaffney (https://github.com/agaffney) for development completion, with Chris Gianelloni (https://github.com/wolf31o2) and Jason Witkowski (https://github.com/jwitko) providing code review and support.

[FEASIBILITY] If you are funded, will you return to Catalyst in a later round for further funding? Please explain why / why not.

We will be participating in later rounds for additional projects. We believe in the Cardano blockchain technology and community and want to see it continue to grow.

[FEASIBILITY] Are you or any member of your team working on any other proposals in this Fund9?

[FEASIBILITY] Are you or your team working on any other proposals from previous Funds?

[AUDITABILITY] Please describe what you will measure to track your project's progress, and how will you measure these?

Progress is tracked through GitHub Issues and a GitHub Projects Kanban board (https://github.com/orgs/cloudstruct/projects/1/views/1). Our engineering team has weekly meetings on status updates.

[AUDITABILITY] What does success for this project look like?

This project will be considered successful when the go-ouroboros-network library completely supports all of the client side operations of all NtC (node-to-client) mini-protocols.

[AUDITABILITY] Please provide information on whether this proposal is a continuation of a previously funded project in Catalyst or an entirely new one.

This is a new proposal and the first round of the Catalyst Fund project that we are applying for.