GridRepublic LLM: Intelligence-as-a-Service for the Cardano Ecosystem

[GENERAL] Name and surname of main applicant

Matthew Blumberg

[GENERAL] Are you delivering this project as an individual or as an entity (whether formally incorporated or not)

Entity (Incorporated)

[GENERAL] Co-proposers and additional applicants

No additional applicants.

[GENERAL] Please specify how many months you expect your project to last (from 2-12 months)

7

[GENERAL] Please indicate if your proposal has been auto-translated into English from another language

No

[GENERAL] Summarize your solution to the problem (200-character limit including spaces)

A community-powered LLM inference service. The system will support a range of models (and languages), powered by an open network of distributed resource providers, and is available to users worldwide.

[GENERAL] Does your project have any dependencies on other organizations, technical or otherwise?

No

[GENERAL] If YES, please describe what the dependency is and why you believe it is essential for your project’s delivery. If NO, please write “No dependencies.”

No dependencies.

[GENERAL] Will your project’s output/s be fully open source?

No

[GENERAL] Please provide here more information on the open source status of your project outputs

The LLMs used will be open-source. The distributed computing tools and infrastructure, however, are provided by GridRepublic; some portions of this are open-source and some are not. 

[METADATA] SDG rating

SDG 09 - "Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation"

RELATED, The U.N.'s 1948 "Universal Declaration of Human Rights" recognizes people’s right to “share in scientific advancement and its benefits”. This vision is not being realized in the domain of AI, however – The enormous infrastructure requirements of these services result in centralization of control, and usage costs which are prohibitive for people in many regions.

[SOLUTION] Please describe your proposed solution

Video Demo: GridRepublic Prototype & Proposal Summary (click image)

PROTOTYPE: Distributed inference service WebUI

SUMMARY

As our problem statement suggests, the construction and utilization of LLM inference infrastructure is costly and centralized, placing control in the hands of a few major corporations and making it inaccessible to global populations. With this project we seek to develop a decentralized, community-powered, "intelligence ecosystem" – one that operates at lower costs, and supports a wide range of languages.

The project builds upon a prototype developed under a Fund11 "Concept" grant: Wolfram: AI - LLM Distributed Inference Services. This prototype includes the following components:

• FOR RESOURCE PROVIDERS: An easy-to-deploy LLM-Server application which can be run on computing devices with appropriate specs, and then automatically plugs into and integrates with the global inference service. "The Network is the computer", as they used to say at Sun Microsystems. (Documentation: Installing and running the GridRepublic LLM client)

• FOR INFERENCE USERS: A range of interfaces, by which users (or 3rd party applications) can make inference requests which are fulfilled by the network of distributed resources running the LLM-Server application. Current interfaces provided by the prototype system include an API, a WebUI, and a Notebook interface. This modular system provides a large library of models, to offer users a range of price-performance options. (Demo: Distributed inference service WebUI)

The current proposal aims to further develop this prototype into a product, most notably by:

{1} Adding billing and payment systems, including ADA-based payments – to enable {a} users to pay for LLM services, and {b} resource-providers to get paid for providing the compute resources which power these services. And also,

{2} Developing the prototype into a platform for Intelligence-as-a-Service within Cardano. That is: by providing a Haskell interface and Plutus integration, we will facilitate deep integration of inference services into the Cardano ecosystem.

[IMPACT] Please define the positive impact your project will have on the wider Cardano community

General Impact. The objective of this work is to create a distributed infrastructure for providing LLM-based services – to lower costs, increase availability across languages, and enhance democratic control of these critical systems.

Cardano-specific Impact. Together, ADA-based payments, Haskell APIs, and Plutus integration comprise an "intelligence layer" for the Cardano ecosystem: i.e. an open and easy means to integrate low-cost and multilingual LLM services into a wide range Cardano applications and services.

[CAPABILITY & FEASIBILITY] What is your capability to deliver your project with high levels of trust and accountability? How do you intend to validate if your approach is feasible?

GridRepublic has over a decade of experience building distributed computational systems, most notably Charity Engine. (N.b. Wolfram Research has been a Charity Engine customer and user for many years; see also Wolfram Language Batch Compute)

The GridRepublic team has operated distributed applications running on as many as a million simultaneous CPU cores, in domains ranging from molecular simulation, advanced mathematics, and genomics. For example:

• For Math Fans: A Hitchhiker’s Guide to the Number 42 (In partnership with MIT + University of Bristol)

• Finding Candida auris in public metagenomic repositories (*in partnership with the US Center for Disease Control and Prevention (CDC); live reporting portal here.)

Furthermore, a prototype of our distributed LLM service has already been built, with support from a Cardano Fund11 "Concept" grant, Wolfram: AI - LLM Distributed Inference Services. 

This working prototype, together with our demonstrated ability to run distributed applications at scale in a wide range of contexts (see examples above), supports confidence in the feasibility of this project.

[PROJECT MILESTONES] What are the key milestones you need to achieve in order to complete your project successfully?

ACCOUNTING AND BILLING SYSTEMS

Outputs

An accounting and billing system to serve both {1} resource providers, and {2} users of the service. I.e. through this system, resource providers should be able to track runtime provided, and see payments due for that runtime; and inference users should be able to see data on their usage, and charges due for that usage. This data should be available via a clean and simple WebUI (the "Accounting Portal").

Acceptance Criteria

1. Resource providers should be able to {a} download and install the LLM-server application, {b} "attach" this to the network, and {c} view the device in the accounting portal
2. Resource Providers should then see data tracking usage of their attached devices, and payments due for this activity, in the accounting portal
3. Documentation sufficient to complete #1 and #2
4. Inference-users should be able to see data tracking their usage of the service, and payments due for this activity
5. Documentation sufficient for #4

Evidence

• A short video walkthrough to demonstrate {a} downloading and installation of the LLM-server application, {b} "attachment" of the installed client to the network, and {c} viewing the device in the accounting portal
• The above video should also show, within the accounting portal, usage and billing data for the attached resource
• Documentation for LLM-server installation and Accounting Portal access
• A short video walkthrough to demonstrate {a} use of the inference service, and {b} login to the inference portal, to view activity and billing data associated with that usage.
• User documentation, for basic usage and accounting portal access.

PAYMENTS

Outputs

{1} Payment Functionality: Inference users should be able to pay for their services via the Accounting Portal; and resource providers should be able to receive compensation for runtime provided. Payments should be possible via at least one fiat currency (USD) and Cardano (ADA).

{2} Accounting and Payment API: Accounting data should be available, and payments supported, via API.

Acceptance Criteria

• Inference users should be able to pay for services via US and ADA
• Resource providers should be able to receive compensation via USD and ADA
• Activity and accounting data should be available via API
• Payments should be possible via API

Evidence

• A short video walkthrough to demonstrate an inference user paying for use of services, via USD and ADA
• A short video walkthrough to demonstrate a resource provider receiving compensation for runtime, via USD and ADA
• A short video walkthrough to demonstrate availability of activity and accounting data via API
• A short video walkthrough to demonstrate payments via API, in USA and ADA: {1} for a resource provider and {2} for an inference user
• Documentation: Payments for Inference Users (including payment by WebUI and API)
• Documentation: Payments for Resource Providers (including payment by WebUI and API)

ENHANCED CARDANO ECOSYSTEM INTEGRATION

Outputs

{1} An API module for Haskell, to simplify interaction with our inference services from Cardano smart contracts and other Cardano services.

{2} A Plutus plugin, to facilitate integration of LLM capabilities into Cardano Smart Contracts.

Together, these two features should facilitate deep integration of LLM Inference capabilities into a wide range of Cardano applications and services.

Acceptance Criteria

• Use of Haskell API {1} to use the inference service; {2} to pay for inference service; {3} to get paid for compute resources supporting inference activity
• Use of Plutus plugin {1} to use the inference service; {2} to pay for inference service; {3} to get paid for compute resources supporting inference activity

Evidence

• A short video walkthrough to demonstrate use of inference services via the Haskell interface
• A short video walkthrough to demonstrate use of the Plutus plugin
• Documentation of the Haskell interface and Plutus Plugin

CLOSE OUT REPORTING

Outputs

PDF report documenting the completion of the three milestones: 1) Accounting and Billing Systems, 2) Payments, 3) Enhanced Cardano Ecosystem Integration.

Acceptance Criteria

PDF report containing

• links to the Accounting Portal and accounting APIs, and evidence of functionality
• evidence of functional billing and payments
• links to a functional Haskell interface and Plutus integration, and evidence of functionality for these features

Evidence

• PDF report documenting the completion of the three milestones: 1) Accounting and Billing Systems, 2) Payments, 3) Enhanced Cardano Ecosystem Integration.

[RESOURCES] Who is in the project team and what are their roles?

Matthew Blumberg, Co-founder and Executive Director, GridRepublic

Matthew has been working in the fields of network computing and large scale collaboration for 15 years. He is Executive Director of GridRepublic and Co-Founder of Charity Engine, two large-scale distributed computing services. Past projects include work as Fellow at Harvard's MetaLAB; Visiting Fellow at the Laboratory for Innovation Science at Harvard (LISH); Section Editor of "The Handbook of Human Computation"; Consultant to DARPA’s “Social Computing Seedling”; and Partner in TGT Energy, an industrial-scale energy storage venture.

Tristan Olive & Rytis Slatkevičius, Co-CTOs, GridRepublic

As the technical team behind Charity Engine, GridRepublic, Progress Thru Processors, PrimeGrid, and other global distributed computing projects, Rytis and Tristan bring decades each of experience in massively high-throughput computing, as well as development and management of platforms and infrastructure on a global scale.

Jon Woodard, WBL CEO

Jon Woodard is the CEO at Wolfram Blockchain Labs, where Jon coordinates the decentralized projects that connect the Wolfram Technology ecosystem to different DLT ecosystems. Previously at Wolfram Research Jon worked on projects at the direction of Wolfram Research CEO Stephen Wolfram and prior to that was a member of the team who worked on the monetization strategies and execution for Wolfram|Alpha. Jon has a background in economics and computational neuroscience. He enjoys cycling in his spare time.

Steph Macurdy, WBL Head of Research and Education

Steph Macurdy has a background in economics, with a focus on complex systems. He attended the Real World Risk Institute in 2019, lead by Nassim Taleb, and has been investing in the crypto asset space since 2015. He previously worked for Tesla as an energy advisor and Cambridge Associates as an investment analyst. Steph is a youth soccer coach in the Philadelphia area and is interested in permaculture.

Gabriela Guerra Galan, WBL Business Operations Specialist

Gabriela Guerra Galan: Gabriela has 15+ years of experience leading projects. She is a certified PMP and Product Owner with bachelor's degree in Mechatronical Engineering, complemented by a master's degree in Automotive Engineering. As the co-founder of Bloinx, a startup that secured funding from the UNICEF Innovation Fund, she has demonstrated a passion for driving innovation and social impact.

[BUDGET & COSTS] Please provide a cost breakdown of the proposed work and resources

Milestone #1

Accounting and Billing Systems

8 weeks. $80,000 ADA

• Installer for the LLM-Server application run by resource providers, plus associated documentation
• Accounting Portal: an accounting and billing system to serve both {1} resource providers, and {2} users of the service.

Milestone #2

Payments

8 weeks. $80,000 ADA

• Payment services, in both USD and ADA, for {1} resource providers, and {2} users of the service.
• Accounting and Payment API

Milestone #3

Enhanced Cardano Ecosystem Integration

12 weeks. $120,000 ADA

• API module for Haskell
• Plutus plugin

[VALUE FOR MONEY] How does the cost of the project represent value for money for the Cardano ecosystem?

{1} We believe there is general social utility to a community-powered AI service.

{2} This project aims to deliver a range of price-performance options for LLM services, at price points lower than current-market rates. The origin of this project demonstrates the utility of the proposed service to the Cardano community. As described in our Fund11 proposal, submitted with our partners Wolfram Blockchain Labs:

Wolfram Blockchain Labs (WBL) is engaging actively in Cardano Catalyst-sponsored research to develop assistance chat bot services for Cardano Catalyst in Fund10 (note – we've now changed the name of that project to the Cardano “Catalyst Navigator”). 

In the course of this work, it’s become clear that in order to appropriately deploy LLM-based applications for Catalyst, blockchain and general communities worldwide, lower-cost inference resources are needed

Thus, in order to build appropriate distributed infrastructure to power such low-cost inferences, WBL is collaborating with GridRepublic (an organization with nearly two decades of experience in distributed computing applications, at global scale)....

{3} We believe the proposed "intelligence layer" for Cardano – LLM services available via Haskell API and Plutus, payable via ADA – will greatly simplify development of an exciting wide range of intelligence-enhanced Smart Contracts, Dapps, and other services within the Cardano ecosystem.

{*} In sum, we believe delivering a service with the breadth and depth of functionality described here, for the proposed budget, comprises an excellent value for the Cardano community and will provide a very solid platform for further growth of Cardano-based Intelligence Services.