gRPC ❤️ Cardano: A streaming API for Cardano using Dolos by TxPipe

[GENERAL] Name and surname of main applicant

Santiago Carmuega

[GENERAL] Are you delivering this project as an individual or as an entity (whether formally incorporated or not)

Entity (Incorporated)

[GENERAL] Please specify how many months you expect your project to last (from 2-12 months)

5

[GENERAL] Please indicate if your proposal has been auto-translated into English from another language.

No

[GENERAL] Summarize your solution to the problem (200-character limit including spaces)

A streaming, real-time API that uses gRPC + protobuf to provide a performant and developer friendly way to interact with the Cardano blockchain.

[GENERAL] Does your project have any dependencies on other organizations, technical or otherwise?

No

[GENERAL] If YES, please describe what the dependency is and why you believe it is essential for your project’s delivery. If NO, please write “No dependencies.”

No dependencies

[GENERAL] Will your project’s output/s be fully open source?

Yes

[GENERAL] Please provide here more information on the open source status of your project outputs

We'll use Apache 2.0 License.

The following artifacts will be open-source:

• The interface spec
• The client SDK / libraries
• Any changes applied to Dolos
• The infrastructure provisioning scripts
• The infrastructure monitoring stack

[SOLUTION] Please describe your proposed solution.

Executive Summary

• We'll build a streaming, real-time API for Cardano, providing a more natural mechanism for interacting with the blockchain.
• The API will use gRPC, a very performant and robust protocol that is used extensibly across the industry.
• It is powered by Dolos, another project by TxPipe, serving as an efficient replacement of the Cardano node for data integration scenarios.
• It will be integrated into the Demeter platform to provide a zero-configuration experience. A free-tier will be available for developers.

Introduction

Connecting to the Cardano node is extremely hard due to a set of design choices. Most of the other blockchains ecosystems have a much more streamlined experience. This is an important barrier for developer adoption and a performance penalty for existing Cardano projects.

gRPC is an open-source project that originated at Google. It is used extensibly across the industry. It's slowly becoming the defacto protocol for backend APIs due to it's performance and extensive toolchain.

Dolos is a Cardano "data-node" developed by TxPipe that has a different set of design-choices that make it suitable for providing a gRPC API. In fact, it already provides gRPC API with limited functionality.

By extending Dolos' functionality to other type of gRPC interactions, we can provide an efficient and developer-friendly alternative to existing integration mechanisms.

Why gRPC instead of Ouroboros mini-protocols

Ouroboros mini-protocols is a custom binary protocol which isn't well documented (there're gaps in the spec). Connecting through this protocol requires building custom client libraries in each of programming language. This is a huge effort and prone to errors. On top of that, the restriction of having to connect through unix socket is a blocker for many use-cases.

gRPC uses a binary encoding called "protobuf" which has similar (or better under specific scenarios) performance and efficiency as CBOR, both in terms of space optimization and encoding/decoding speed.

gRPC uses a strict schema, both client and server needs to know in advance the structure of the messages that will be exchanged. What may seem like a burden, is actually a very powerful mechanism to ensure forward and backward compatibility between servers and clients.

gRPC has an extensive toolchain that provides automatic code generation for all of the main stream programming languages (Python, C++, Golang, NodeJS, Javascript, Rust, etc). By using the protocol schema, client and server libraries can be created for any of the supported languages with a strict guarantee of compatibility between them.

Why gRPC instead of traditional HTTP/JSON APIs

gRPC uses HTTP/2, a version of the HTTP protocol that support multi-plexing. This means that many client-server interactions can be bundled into the same HTTP connection, greatly improving the latency and throughput of the communication.

gRPC uses "protobuf", a much more compact binary encoding that provides between 2x - 10x reduction in payload size compared to JSON.

gRPC has streaming support. Both client-side, server-side or bidirectional streaming. Streaming provides an efficient mechanism to implement event-driven scenarios, where one party needs to wait for the other to notify about new events.

gRPC also provides an out-of-the-box bridge to websockets, allowing javascript clients to connect directly from the browser.

Why a streaming API

The blockchain is a live sequence of events (new blocks, new transactions, rollbacks). dApps need to constantly monitor these events to create a consistent view of the state and make apply corresponding changes.

Most popular APIs in Cardano rely on request / reply semantics where each request responds with a fixed view of the state of the chain (eg: give me the utxos of this address). Monitoring changes requires continuous polling of the data just in case anything has changed. This is extremely inefficient.

Streaming is a mechanism where two parties (eg: client / server) exchange a sequence messages efficiently without relying on poling. In our case, a single gRPC connection is used where the server will push a new event (message) every time a relevant change is detected.

For example, just to mention a few, a dApp can connect to our API and ask for notifications every time a change is detected on a particular Cardano address, or get a notifications when a transaction has reached the chain, or when a particular asset has moved between addresses.

About UTxO RPC

UTxO RPC is a spec built on top of gRPC, specifically designed to interact with blockchains that use the UTxO model. It provides well-defined methods for the different type of common interactions that are required (eg: chain sync, state queries, transaction submission, etc).

Having a spec which is implementation-agnostic promotes interoperability. It allows different teams and tool providers to build their own concrete, independent implementations of clients and servers while maintaining compatibility among each other.

This proposal will use UTxO RPC as the interface for the gRPC endpoints. There're other teams / projects in the Cardano ecosystem which are planning on using the same spec. This will foster interoperability.

[IMPACT] Please define the positive impact your project will have on the wider Cardano community.

Simplified development of Cardano dApps

A modern, performant and developer friendly API that provides streaming semantics will simplify development of many Cardano dApps and potentially improve the performance in specific scenarios.

Attract developers from other ecosystem

By introducing gRPC, a well-known industry standard, as a mechanism to interact with the Cardano blockchain, we're lowering the entry barrier to many developers that are already familiar with these technology.

Provide better UX for end-users in certain interactions

Using streaming semantics provides a way for dApps to implement user experiences that are more reactive and responsive that their counterparts using poling mechanisms.

Boost performance and facilitate scalability:

The introduction of gRPC provides the reliability and speed of a modern protocols, enhancing overall network responsiveness. This enables dApps to handle higher transaction volumes and user traffic with greater efficiency and lower costs.

[CAPABILITY & FEASIBILITY] What is your capability to deliver your project with high levels of trust and accountability? How do you intend to validate if your approach is feasible?

TxPipe is an active member of the Cardano ecosystem

TxPipe has been developing open-source tools for the Cardano ecosystem for over 2 years and we're not going anywhere. Evidence of our commitment can be found by evaluating the continuous activity of our public code repositories.

Experience developing in the Cardano ecosystem

TxPipe has helped developed several dApps for the Cardano ecosystem. This experience allows us to evaluate the feasibility of the project and its potential benefit from a developer's perspective.

Succesful Catalyst proposals

We have successfully completed 2 prior Catalyst proposals. This may serve as evidence that our team has the required capabilities to fulfill these type of projects.

TxPipe is one of the entities behind the UTxO RPC spec

TxPipe provided the initial design and has continued supporting the development of the UTxO RPC spec. We have a deep understanding of the protocol and we're very well suited to build a full server-side implementation.

Development process will be public and open-source

Both the output and the development process will be public and open-source. This approach provides an easy way for the Catalyst team and the Cardano community to evaluate the progress at each step of the process.

[Project Milestones] What are the key milestones you need to achieve in order to complete your project successfully?

Implement remaining UTxO RPC endpoints

Dolos already provides some of the gRPC endpoints, but not all of them. This milestone is about implementing the server side business logic for the remaining required endpoints.

Tasks:

• Implement the watch module endpoints from the UTxO RPC spec
• Implement the build module endpoints from the UTxO RPC spec
• Implement the submit module endpoints from the UTxO RPC spec

Outputs:

• Publicly available source code of the improved Dolos component including all of the required endpoints.
• A binary release of the improved Dolos component including all of the required endpoints.
• Documentation and instructions on how to interact with each of the newly available gRPC endpoints in Dolos.

Acceptance Criteria:

• A developer is able to install the binary release of Dolos and follow the provided instructions to make gRPC calls to each of the newly created endpoints (watch, build, submit modules).

Client SDKs

This milestone is about providing client libraries in many of the mainstream programming languages to facilitate access to the gRPC endpoints from consumer applications.

Tasks:

• Implement a NodeJS SDK
• Implement a Python SDK
• Implement a Rust SDK
• Implement a Golang SDK
• Implement a Web (Javascript) SDK

Outputs:

• Publicly available library packages for each of the client SDKs published on the corresponding package manager registries (eg: npm, crates.io, pypi, etc)
• Documentation and instructions on how to install and use the client SDK in each of the available languages.
• Publicly available source code for the client SDKs in each of the available languages.

Acceptance Criteria:

• A developer is able to follow the available instructions to install the client libraries in their own project and execute remote call to the gRPC endpoints of the API without having to interact with low-level protocol details.

Demeter Integration

This milestone is about the integration Dolos as part of the Demeter platform to provide a zero-configuration experience for developers that want to access the API without provisioning their own infrastructure.

Tasks:

• Automate provision and monitoring of Dolos instances on Kubernetes clusters
• Automate API access configuration in developer projects
• Rollout the infrastructure to currently active Demeter clusters

Outputs:

• Provisioning scripts to install Dolos instances on a Kuberentes cluster
• Publicly available source-code of a Kubernetes operator that controls access to the API

Acceptance Criteria:

• A developer is able to use Demeter Web Console to enable access to the gRPC API in their Demeter project in a self-service fashion.
• A developer is able to execute remote calls to the gRPC APIs on their Demeter project without having to setup any servers or backend components.

Integration with Ecosystem Libraries

This milestone is about providing a streamlined development experience to developer that already use one of the well-known frameworks for interacting with the Cardano blockchain.

Tasks:

• implement a gRPC provider for the Lucid library
• implement a gRPC provider for the MeshJS library
• Attempt to integrate changes with official upstream repositories

Outputs:

• Publicly available source code in a forked repository of the Lucid and MeshJS libraries that includes gRPC providers for the libraries' required methods.
• Documentation including simple examples of how to configure each of the libraries to use the newly added gRPC providers.

Acceptance Criteria:

• A developer is able to follow the available instructions to execute example code snippets supporting the newly created providers.
• A developer is able to switch an existing Lucid or MeshJS application to use the new gRPC provider without breaking compatibility.

Completion Report

This milestone is about wrapping up the project and providing relevant insights for the Catalyst team and larger Cardano community to evaluate the result of the proposal.

Outputs:

• a video recording summarizing the process and results of the whole project.
• publicly available source-code containing all artifacts produced throughout the project.
• written report describing the development process and any relevant findings.
• quantitative report showing any available usage metrics of the API hosted in Demeter platform.

Acceptance Criteria:

• the video recording provide enough information to asses the end result.
• a technical reviewer / developer is able to assess the validity of the source code with reasonable accuracy
• a reviewer is able to audit the development process ensuring that all required tasks have been completed.
• a reviewer is able to appreciate the usage of the API hosted in the Demeter platform.

[RESOURCES] Who is in the project team and what are their roles?

• Santiago Carmuega (Tech Lead) - Github
• Federico Weill (PM) - Linkedin
• Rodrigo Suaya (Product Manager Demeter) - Linkedin
• Alejandro Avagnina (SRE) - Github
• James Harper (Rust Developer) - Github
• Paulo Bressan (SDK Developer) - Github
• Franco Luque (Blockchain Developer) - Github
• Florencia Luna (Technical Writer) - Linkedin

[BUDGET & COSTS] Please provide a cost breakdown of the proposed work and resources.

FTE = Full-time equivalent

Total budget = ₳ 177,082

Software Development:

• Rust developer: 1 FTE x 2 months = ₳ 44,444
• SDK developer: 1 FTE x 2 months = ₳ 44,444
• Blockchain developer: 1 FTE x 1 months = ₳ 25,000
• Technical writer: 1/2 FTE x 2 months = ₳ 8,333

Cloud Infrastructure:

• Site Reliability Engineer: 1 FTE x 2 months = ₳ 44,444

Project Management:

• Project manager: 1/4 FTE x 5 months = ₳ 10,417

[VALUE FOR MONEY] How does the cost of the project represent value for money for the Cardano ecosystem?

The bulk of the budget falls under the software development categories. TxPipe has extensive experience in the field, allowing it to provide good value for money. The hourly rates are defined using fair market prices. The estimation for the level of effort takes into account all of the optimizations that our team is capable of providing after years of experience developing software solutions in the Cardano ecosystem.

[IMPORTANT NOTE] The Applicant agrees to Fund Rules and also that data in the submission form and other data provided by the project team during the course of the project will be publicly available.

I Accept