Decentralized Demeter.run - Federated Frontend Hosting - New revenue stream for SPOs

[GENERAL] Name and surname of main applicant

[GENERAL] Email address of main applicant

Additional applicants

federico@txpipe.io

[GENERAL] Please specify how many months you expect your project to last (from 2-12 months)

[GENERAL] Please indicate if your proposal has been auto-translated into English from another language.

[GENERAL] Does your project have any dependencies on other organizations, technical or otherwise?

[GENERAL] If YES, please describe what the dependency is and why you believe it is essential for your project’s delivery. If NO, please write “No dependencies.” .

TxPipe has the technical skills and infrastructure to develop the system but we’ll require help from the following independent organizations:

• A small group of SPOs that wish to participate as infrastructure providers
• A small group of dApp developers that wish to participate by hosting their frontend

[GENERAL] Will your project’s output/s be fully open source?

[GENERAL] If NO, please describe which outputs are not going to be open source. If YES, please write “Project will be fully open source.”

Project will be fully open source.

[METADATA] Category of proposal

[IMPACT] Please describe your proposed solution.

What's a frontend?

• A frontend is a web app or a mobile app that provides a friendly user interface. End-users interact with dApps through a frontend + wallet.
• Without a frontend, interacting with the blockchain would require manual crafting of really complex transactions and indexing large volumes of on-chain data locally.
• Frontends are key components to any dApp and as such, they need to be: reliable, performant, have global scale and easy to develop / maintain.
• Today, most frontends are hosted in one of a small set of big cloud providers such as AWS, Azure, Google Cloud, Vercel, etc, because they are the best available option to achieve the levels of reliability and performance required.

Problems with current approach

• Not censorship resistant: if the cloud provider decides (or is coerced) to shutdown the frontend, interacting with the dApp would be drastically affected.
• Using solutions like IPFS for content distribution would be censorship-resistant but they are extremely limited in capabilities: no server-side processing, no database, no load-balancing, no in-place updates, no detailed monitoring, etc.
• Poor competition: to get the benefits of the cloud, developers end up having to choose from a very short list of providers. These providers have little incentive for standardisation of their features, so developer often end up locked-in into particular vendors.
• Too generic: common cloud providers don't provide any Cardano-specific components (eg: Cardano Node, DBSync, Kupo, etc) and need to be manually implemented / managed by the dApp team. This adds complexity and increases the TCO.

Solution

• An open-source hosting platform (Demeter.run) that provides a developer experience similar to Vercel or Firebase. Developers build their React, NextJS or Remix frontend and then run a `publish` command.
• A set of independent stake pool operators distributed across the globe where each one runs and monitors an instance of the open-source hosting platform.
• A fair load-balancing mechanism that distributes end-user's web requests to the different available SPOs, taking into account a past-performance reputation score.
• A bookkeeping mechanism on top of a distributed ledger that takes care of charging and compensating the different actors using an on-chain smart contract.
• Although the system would not be limited to Cardano dApps, SPOs could provide Cardano-specific services (access to Cardano Node, DBSync, etc) as an extra revenue stream. This simplifies operations for dApps and reduces costs by allowing sharing of common instances.

Scope for this Proposal

The solution described above is very ambitious. The scope of this proposal is to build a PoC (proof of concept) implementation with the following characteristics:

• Include a minimum of 2 SPOs participating as infrastructure providers. This will give a total of 3 different entities providing infrastructure (since TxPipe will remain as provider).
• Include between 10 and 200 real-world frontends hosted in the platform.
• An MVP version of the bookkeeping mechanism that is suitable for a small scale group of trusted entities. Scaling the bookkeeping process to a larger group of entities will require some improvements which are out of the scope for this proposal.
• An MVP version of load-balancing mechanism that will provide all the required features 
• Personalized training for participating SPOs so that they can run their partition of the system.
• Provide infrastructure grants to pioneer dApp developers wanting to use this service. This will incentivize usage, lower dApp operating expenses and provide direct revenue to participating SPOs.

Technical Details

On our public RFC (Request For Comment) we provide some insights regarding the technical implementation of the platform.

[IMPACT] How does your proposed solution address the challenge and what benefits will this bring to the Cardano ecosystem?

Our proposal will provide:

• A real-world use-case for the Cardano blockchain: The consensus and bookkeeping mechanism that governs the hosting platform would be implemented as an Hydra state channel using Cardano as L1. More transactions on L1 means more fees for the protocol.
• A new revenue stream for Cardano SPOs: Cardano has a very rich and diverse group of SPOs which are cloud and networking experts. A federated hosting protocol would allow them to utilize their existing expertise for an extra revenue stream.

[IMPACT] How do you intend to measure the success of your project?

Our PoC requires validation of several dimensions. In particular, we need to evaluate technical and business feasibility, this will require a combination of hard data and qualitative analysis.

The following metrics will be used for data analysis:

• The number of frontends hosted in the platform
• The number of requests that frontends receive
• The throughput processed by the infrastructure
• The uptime of the frontends measured by a max timeout threshold
• The quality of service for the end-user measured by request latency
• The error rate of requests measure by HTTP status code >500

The infrastructure providers (SPOs) will be required to provide some data regarding their operation:

• Initial investment required to provision their partition of the system
• Day-2 operation costs to run their partition of the system
• Qualitative survey describing the complexity of maintaining their partition

The dApp developers using the frontend hosting service will be required to provide some feedback regarding their experience:

• Overall development experience satisfaction as compared to similar systems (Firebase, Vercel, Fly.io, etc)
• Any available end-user feedback regarding the perceived performance of the frontends hosted by the system.

[IMPACT] Please describe your plans to share the outputs and results of your project?

The proposal requires the development of the following software components:

• A Kubernetes operator that will be responsible for orchestrating the compute primitives required for running an instance of a frontend (containers, deployments, ingress, certificates, etc). This component will also be responsible for interacting with Hydra nodes for consensus between other partitions of the system.
• A CLI (command line interface) responsible for providing frontend developers with an entry point for building and deploying the required artifacts to run their frontend.
• A build system built on top of Cloud Native Buildpacks that will generate the required OCI (aka: Docker) artifacts required for runtime.
• A DNS (Domain Name System) server that is aware of the whole network topology with the goal of resolving custom subdomains pointing to the frontends hosted by the system.

Custom components developed for this system will be open-source, the outputs will be available to any developer in the ecosystem at every step of the development process:

• Latest version of the source-code will be available in the Github repository.
• Source code changes will be applied through a pull-request process.
• Alpha and Beta versions will be released at every milestone.

Upon reaching the end of the development process, we’ll provide:

• Docker images for the backend components (operator, build system, DNS)
• A CLI (command line interface) binary to serve as entry point for developers
• A documentation website with instructions for usage and deployment
• A tutorial video showing how to deploy a frontend using the CLI

Upon reaching the end of the PoC, we’ll provide:

• Raw, anonymous metrics regarding performance of the system
• A report showing our analysis of the success of the PoC. It will include both a quantitative and qualitative analysis of the metrics, SPO surveys and dApp developer surveys.
• A written roadmap describing the future plans for the system taking into account all of the information gathered throughout the PoC.

[CAPABILITY/ FEASIBILITY] What is your capability to deliver your project with high levels of trust and accountability?

TxPipe is very proud of their past and current contributions to the ecosystem. Just to mention a few:

• We have developed “Pallas”, a Rust library for Cardano which is used by several high-profile project in the community (such as: cncli and Aiken)
• Through Catalyst, we have developed and delivered “Oura”, an off-chain data integration tool for Cardano used by many projects in the community (such as: Pool.io and dcSpark’s Carp, etc).
• Through Catalyst, we have developed “Dolos”, a minimalistic version of the Cardano node which is being slowly rolled out to interested community users as a beta version.
• We have developed “Demeter”, a cloud hosting platform for Cardano infrastructure with several high-profile clients (such as: JPG.store, SummonPlatform and TeddySwap).

The above are examples of our accountability because:

• It shows our commitment to evolving the open source community and the Cardano ecosystem.
• It shows the technical expertise required to develop and maintain Cardano infrastructure.
• It shows our commitment to the Catalyst program and its required procedures.

One of the main challenges of the project is matching offer and demand. We need both SPOs offering infrastructure and dApps demanding infrastructure, which becomes a chicken-and-egg problem. To overcome this and generate trust on the project, this proposal includes resource requests that will be transferred as “infrastructure grants” for dApps wanting to participate in the PoC. We believe that this incentive is a valid way to bootstrap the system.

[CAPABILITY/ FEASIBILITY] What are the main goals for the project and how will you validate if your approach is feasible?

The main goal of our PoC is to validate the hypothesis that there’s a valid business model for SPOs and infrastructure providers in general around hosting dApp Frontends in a more decentralized fashion.

We consider the project to be a success if we manage to gather enough information to answer the following questions:

• Is the approach cost-effective for dApp developers?
• Is the approach a valid revenue stream for infrastructure providers (SPOs)?
• Is the quality of service for end-users comparable to large cloud-providers?
• Is the reliability of the distributed system comparable to large cloud-providers?

As a secondary success criteria, we’ll consider this proposal a complete success if the hypothesis turns out to be true, and this PoC turns into an MVP that can be expanded into a larger and self-sustainable grid of small infrastructure providers serving dApp frontends.

[CAPABILITY/ FEASIBILITY] Please provide a detailed breakdown of your project’s milestones and each of the main tasks or activities to reach the milestone plus the expected timeline for the delivery.

Milestone #1: Distributed Kubernetes Operator (1 month)

• Develop open-source Kuberentes operator to handle Frontend workloads
• Integrate operator with distributed consensus mechanism
• Provision a 2-party staging environment with fake workloads

Milestone #2: Distributed Load Balancer (1 month)

• Integrate DNS component for awareness of global topology
• Integrate reverse-proxy component for awareness of global topology
• Update 2-party staging environment for distributed load balancing
• Perform long-running operation tests with fake requests

Milestone #3: Distributed Bookkeeping (1 month)

• Develop credit bookkeeping component
• Develop usage-tracking bookkeeping component
• Update 2-party staging environment for distributed bookkeeping
• Perform long-running operation tests with fake requests

Milestone #4: SPO Onboarding (2 weeks)

• Reach-out to interested SPOs to participate in the PoC
• Train SPOs on technical operation of the infrastructure
• Support SPOs throughout the process of provisioning their infrastructure
• Setup a distributed staging environment among all participating SPOs
• Perform long-running operation tests with fake requests

Milestone #5: Frontend CI Pipelines (1 Month)

• Integrate an automated CI pipeline for Frontend builds
• Update the distributed staging environment to support build jobs
• Perform integration testing focusing on build pipelines

Milestone #6: Frontend Developer Tooling (1 Month)

• Develop an API to execute RPC to request builds & deployments
• Develop a CLI application to interact with the API
• Release binaries and end-user documentation for the CLI

Milestone #7: dApp Onboarding (2 weeks)

• Reach-out to interested dApps to participate in the PoC
• Train dApp developers on how to use the system
• Support the dApps developers throughout the process of running their frontends
• Deploy staging versions of the frontends
• Perform long-running operation tests with fake requests

Milestone #8: Stress & Penetration Testing (2 weeks)

• Perform integration tests on bookkeeping subsystem
• Perform integration tests on CI pipelines
• Perform stress test of frontend request throughput
• Perform stress test of concurrent frontends
• Perform stress test on API endpoints
• Perform penetration test on API endpoints

Milestone #9: Production Release (1 month)

• Setup distributed production environment
• Instruct dApps to deploy production frontends and partially redirect traffic
• Measure HTTP request metrics
• Measure compute resource metrics

Milestone #10: Real-world data gathering and analysis (2 weeks)

• Perform SPO survey
• Perform dApp developer survey
• Analyze raw metric data
• Prepare final report and share with community

[CAPABILITY/ FEASIBILITY] Please describe the deliverables, outputs and intended outcomes of each milestone.

Milestone #1: Distributed Kubernetes Operator

• Output: An open-source Kubernetes operator integrated with a distributed consensus mechanism (publicly available repository), including provisioning script and tutorial.
• Outcome: An operator capable of handling frontend workloads, tested and validated in a two-party staging environment.

Milestone #2: Distributed Load Balancer

• Output: Source code for a new feature of the operator that integrates a DNS server and reverse-proxy components for global topology awareness, including provisioning script and tutorial.
• Outcome: A distributed load balancer mechanism, verified for long-running operations in a two-party staging environment.

Milestone #3: Distributed Bookkeeping

• Output: Source code for a new feature of the operator that integrates credit and usage-tracking bookkeeping components.
• Outcome: A working distributed bookkeeping mechanism, verified through long-running tests in a two-party staging environment.

Milestone #4: SPO Onboarding 

• Output: At least 2 SPOs on-boarded and participating in the PoC, capable of managing their partition of the infrastructure.
• Outcome: A working distributed staging environment managed by multiple SPOs, resulting in a federated network of infrastructure providers.

Milestone #5: Frontend CI Pipelines

• Output: Source code for a new feature of the operator that integrates an automated CI pipeline for frontend builds.
• Outcome: An working CI pipeline process for building frontend artifacts that it’s validated through integration tests.

Milestone #6: Frontend Developer Tooling

• Output: Source code for an API for RPC execution and a CLI application. Binary releases for a CLI application. End-user documentation site for deploying frontends.
• Outcome: A working, developer-friendly toolchain for deploying frontends that has been validated through integration tests.

Milestone #7: dApp Onboarding

• Output: At least 5 trained dApp developers capable of using the system to deploy their frontend.
• Outcome: Staging versions of frontends deployed and long-term operation tests executed, resulting in a network that is ready for more extensive use.

Milestone #8: Stress & Penetration Testing

• Output: Comprehensive reports from integration tests, stress tests, and penetration tests.
• Outcome: A system that has been thoroughly tested for security, scalability, and robustness, reducing potential risks and vulnerabilities.

Milestone #9: Production Release

• Output: A distributed production environment.
• Outcome: dApps deploy production frontends and partially redirect traffic, resulting in real-world usage and measurement of HTTP request metrics and compute resource metrics.

Milestone #10: Real-world data gathering and analysis

• Output: Survey results from SPOs and dApp developers, analysis of raw metric data, and a final report.
• Outcome: Valuable insights into real-world operation of the system, supporting further development and community engagement.

[RESOURCES & VALUE FOR MONEY] Please provide a detailed budget breakdown of the proposed work and resources.

FTE = full-time equivalent

Values expressed in ADA (₳)

Breakdown by resource type

• Rust developers: 1 FTE x 5 months = ₳ 160,714
• Frontend / React developers: 1 FTE x 2 month = ₳ 28,571
• Technical writers: 1 FTE x 2 months = ₳ 21,429
• Project manager: 1/4 FTE x 8 months = ₳ 21,429
• Site-reliability engineers: 1 FTE x 3 months = ₳ 85,714
• Hosting grants for dApps: 200 frontends x 6 months = ₳ 85,714
• Staging environment: 1 environment x 6 months = ₳ 4,286

Breakdown by milestone

• Milestone #1: ₳ 49,108
• Milestone #2: ₳ 49,108
• Milestone #3: ₳ 42,678
• Milestone #4: ₳ 27,589
• Milestone #5: ₳ 33,750
• Milestone #6: ₳ 56,964
• Milestone #7: ₳ 21,339
• Milestone #8: ₳ 11,517
• Milestone #9: ₳ 108,750
• Milestone #10: ₳ 7,054

[RESOURCES & VALUE FOR MONEY] How does the cost of the project represent value for money for the Cardano ecosystem?

The resources associated with the development process will result in open-source code that can be leveraged by any team or member of the Cardano ecosystem.

The resources used for grants will serve as direct revenue for SPOs participating as infrastructure providers and also as value for dApps since they get hosting for free for 6 months.

The resources used to generate the reports will provide the Cardano ecosystem, in particular TxPipe, Demeter.run and SPOs, valuable insight for a potential larger scale project that can bring real-world use and traffic in the chain.

[IMPORTANT NOTE] The Applicant agreed to Fund10 rules and also that data in the Submission Form and other data provided by the project team during the course of the project will be publicly available.