Wolfram: The AI Revolution and Implications to Project Catalyst

[GENERAL] Name and surname of main applicant

[GENERAL] Email address of main applicant

Additional applicants

Steph Macurdy

Gabriela Guerra

[GENERAL] Please specify how many months you expect your project to last (from 2-12 months)

[GENERAL] Please indicate if your proposal has been auto-translated into English from another language.

[GENERAL] Does your project have any dependencies on other organizations, technical or otherwise?

[GENERAL] If YES, please describe what the dependency is and why you believe it is essential for your project’s delivery. If NO, please write “No dependencies.”

No dependencies.

[GENERAL] Will your project’s output/s be fully open source?

[GENERAL] If NO, please describe which outputs are not going to be open source. If YES, please write “Project will be fully open source.”

Project will be fully open source.

[METADATA] Category of proposal

[IMPACT] Please describe your proposed solution.

Introduction

Due to the advancements in large language models (LLMs), the entire world has undergone significant changes. Any governance process based on proposals is now subject to the open creation possibilities by LLMs as well as the useful review and evaluations from them.

Wolfram: AI Revolution & Project Catalyst Video Presentation

The Problem and Our Solution

At present, understanding the full Catalyst process is a significant undertaking. From problem sensing to challenge interpretation, from proposal writing to feedback sharing and ultimately submission; proposers have extensive guidelines and community standards to follow for accurate completions. Writing proposals is a significant undertaking, and thus a strong deterrent against less serious proposals. However, a global community of participants potentially struggle to comprehend, articulate and convey how their proposal will deliver impact in a highly feasible and cost effective way. And while these strict guidelines are necessary, they may simultaneously increase the difficulty in articulating solutions to real problems and ultimately decrease participation. It would be encouraging to this community if supportive and generative "railings" existed to help facilitate the process. A grassroots system of innovation may benefit from a supportive boost in the area of proposal writing.

Conversely, the field of community reviewers face a similar (yet opposite) undertaking when evaluating proposals in each new funding round. With financial incentives and tools like LLMs, what support are we providing these reviewers to combat and enforce the auditability standards? With over 1,000 submissions in each of the last three rounds, what standardized procedures can reviewers unequivocally rely upon when facing the full spectrum of community submitted proposals? Even the most veteran reviewers might bias depending on native language or recognition of "friendly" proposals. We know there's a wide variety of reasons why community reviewers need support in the functions they provide the Catalyst community. So they may support the idea of an assistant that boosts their ability to efficiently and effectively draw conclusions about impact, feasibility and cost effectiveness of each proposal in a standardized way.

So, our solution is to research what is needed to create an LLM-based assistant that acts as a guiding companion in the Catalyst process. The objective of this research is twofold: to explore the feasibility of providing users with support throughout the stages of proposal submission, evaluation, and voting, and to understand the preferred implementation approach according to the community's perspective. It's important to note that this proposal does not encompass the actual implementation of the LLM assistant.

With the support of Wolfram Research's extensive work on building LLM tools over the past six months, with the successful Wolfram plug-in to ChatGPT and more, we are well-equipped to research the requirements of a tool that addresses the proposal process's pain points. The potential of LLMs is vast, opening up numerous possibilities for assistance on all stages of the proposal process – from researching, to writing, to reviewing and, finally, to voting. Our team acknowledges both the extreme benefits and drawbacks of LLMs inside a community process based primarily on written communication. However, it would be prudent to acknowledge that LLMs are now a part of society and we can likely find a balance so that LLMs can be used effectively. In fact, this is how open systems naturally evolve and grow.

Researching LLM with Guided Participation

Our research will center around what is needed to provide step-by-step guidance throughout the Catalyst process through a chatbot type interface. Critically, this is not the creation of an AI chatbot or AI Assistant for Catalyst. That is an enormous undertaking at this point. Rather, this project is significant research and exploration to see if the useful creation of an assistant is possible—by investigating novel approaches and proofs of concept—and if so, how to keep the operational costs low enough to provide a significant return on investment for the Catalyst community. Because this project is fully open source, we believe leading this research effort would establish a strong precedent for community actors to get up to speed and build from there.

Researching the Catalyst Assistant

To build a robust and cost-effective LLM-based Catalyst Assistant, extensive research is required.

Literature Review & Community Surveys

The initial phase of the work will involve conducting thorough literature reviews and community surveys. Wolfram has already dedicated significant time to studying and engaging in discussions about the utilization of LLMs for developing expansive virtual assistants. However, it is essential to delve into detailed discussions and provide comprehensive summaries of various approaches. In addition to researching existing literature, conducting surveys among Cardano Catalyst participants is necessary to gain a deeper insight into the execution of different roles within Catalyst and understand the expectations users would have for a potential assistant. 

Content Collection & Curation 

There have been 10 Cardano Catalyst funds, and Wolfram Blockchain Labs (WBL) has already constructed curated collections of past proposals, including analyses for ranking proposals based on community voting and reviews in our existing dashboard solution. However, for this specific use case, we will need to further curate these proposals. Additionally, we must gather all instructional content related to participating in different roles within Catalyst. Furthermore, it is likely that we will need to develop a significant amount of new material on how to engage in Catalyst. Initial assessments of the existing material reveal that many instructional resources assume a significant amount of prior knowledge—unfortunately we can’t assume that instructional content would be available in different LLMs, nor would it be validated or trustworthy. These tasks would provide curated content and a deep understanding of the unique aspects of Catalyst proposals that would be the key inputs into LLM-based assistance in detail aspects of proposal creation, refining, and evaluation.

LLM Evaluation

Drawing from our extensive experience collaborating with various LLM providers, such as OpenAI, we are aware of the wide range of performance capabilities and inference costs associated with these models. To develop a highly efficient system, striking a delicate balance between LLM performance and inference expenses is crucial. This entails establishing a test suite and evaluating different LLMs specifically for the use case of an LLM enhanced Catalyst Assistant. Furthermore, during the LLM exploration process, we will have the opportunity to delve into other aspects of the training pipeline to create and experiment with use cases, and design and refine proofs of concept and rank them for the possible benefits they may provide to the proposal pipeline.

Final Output

The culmination of our research and development efforts will result in a detailed research report. This report will document the exploration of LLMs that meet the necessary licensing requirements and ROI, presenting the findings to the Cardano community. Insights gathered from surveys conducted within the Cardano community will also be included in the research report, Computational essays will provide a step-by-step account of the potential development process, highlighting the critical aspects of building a potential assistant. Specifications and architecture will be detailed to provide transparency and clarity on potential functionality and implementation.

Project Objectives Summary:

• Conduct a comprehensive literature review
• Implement community survey
• Collect all instructional content related to Catalyst
• Evaluate LLMs for suitability for explored assistant use cases
• Research specification architectures for LLM-based catalysis of proposal pipeline

Overall Benefits: 

• New instructional content for potential training material
• Research report of potential LLM assistant architectures
• Specification architectures of an assistant and beneficial model
• Community discussion on application of new technology

Conclusion

The introduction of an LLM-based assistant for the Cardano Catalyst proposal process holds immense potential for transformation. Harnessing the capabilities of large language models, we can equip participants with seamless navigation through Cardano Catalyst, offering real-time assistance and guidance. Together, we can revolutionize the proposal process, foster innovation, and propel the Cardano Catalyst community to unprecedented achievements.

[IMPACT] How does your proposed solution address the challenge and what benefits will this bring to the Cardano ecosystem?

The "Catalyst System improvements" challenge asks "what research and development is required to advance the state of the art of Catalyst and allow Catalyst to serve the community's needs better?'

Our proposed solution addresses this question by researching LLMs and trying to understand through a community supported research process how it can be applied to Project Catalyst.

We believe this proposal directly addresses the Catalyst Systems Improvements because this research has a high likelihood of advancing the state of the art for the innovation platform for Cardano. LLMs are at their infancy of development and we're just beginning to see the wide-ranging effects. A decentralized governance and proposal-based innovation fund would greatly benefit from understanding the upsides and downsides of LLM assistance. This is a direction that our community will inevitably face more in the future, so adoption now may prevent catastrophic failure in the future.

The challenge goes on to detail the category of Academic research. "Clearly defines a known Catalyst-specific problem-space where the intention is to identify facts and/or clearly stated opinions that will likely assist in solving Catalyst-specific problems, or a detailed study of a Catalyst-specific subject, especially in order to discover (new) information or reach a (new) understanding."

LLMs are state of the art technology and Wolfram Research has been exploring many ways to use it.

Link here: https://writings.stephenwolfram.com/2023/03/will-ais-take-all-our-jobs-and-end-human-history-or-not-well-its-complicated/

One of the critical issues is how many text-based processes will change with the introduction of such revolutionary technologies. We believe it's important to explore how to integrate LLM services in ways that help all members of the Catalyst community participate in all roles of the Catalyst process. Participants should not be hampered, nor should the intrinsic value of their proposals be lessened by, for instance, writing in a second language or having limited experience in proposal writing. In a way, establishing the use case for individuals ensures that power is shared more democratically in a process that is based on decentralization as core tenet.

[IMPACT] How do you intend to measure the success of your project?

A general community agreement that this line of research is interesting and worth exploring further. Clear success would be if a this research enables other participants success using LLMs and/or implementations into Catalyst.

[IMPACT] Please describe your plans to share the outputs and results of your project?

We have deliverables for public repository of curated Catalyst training data, a research report, breakout rooms to share findings and receive community feedback, and twitter spaces to engage in discussion.

[CAPABILITY/ FEASIBILITY] What is your capability to deliver your project with high levels of trust and accountability?

Our capability to deliver the project with high levels of trust and accountability is grounded in several key factors:

• Track Record: Wolfram Blockchain Labs (WBL) has a good track record in successfully completing Catalyst projects, including four already completed projects and three nearly complete since Fund 8. This experience has led to developing our ability to meet project objectives, timelines, and budgetary constraints.

• Experienced Team: Our team comprises professionals from diverse disciplines, including blockchain, finance, Wolfram technology, LLMs and natural language processing. This multidisciplinary approach allows us to bring a comprehensive and holistic perspective to the project, ensuring that all aspects are addressed with a high level of expertise.

• On-Time Delivery: Our successful completions demonstrates our ability to manage project resources efficiently and effectively. The recognition and recommendation we received from the Cardano catalyst community further reinforce our capability to deliver projects with accountability and quality.

Our track record, experienced team, firsthand expertise, and adherence to strong financial management practices is why we're well-suited to deliver the project with high levels of trust and accountability.

[CAPABILITY/ FEASIBILITY] What are the main goals for the project and how will you validate if your approach is feasible?

Milestone 1: Literature Review & Community Surveys

• The purpose of the literature review and community surveys is to gather a strong understanding of how LLMs can be used in a proposal process like Project Catalyst's and to learn what the community would benefit most from.

Milestone 2: Content Collection and Curation

• The collection of all existing Cardano Catalyst instructional materials and relevant proposals/reviews in preparation for the design and implementation of the LLM assistant.

Milestone 3: LLM Evaluation 

• Evaluate different LLMs to see which ones might be the best to use for this specific purpose
• Chose the optimal LLM for the Catalyst use case
• Reduce training costs based on LLM capabilities

Milestone 4: Experimentation & Alterations

• Explore an infrastructure pipeline to handle Project Catalyst LLM assistant data processing, storage, and retrieval functionality.
• Examine alterations to the proposal structure

Milestone 5: Report Writing and Presentation

• Deliver research report
• Share results with community

[CAPABILITY/ FEASIBILITY] Please provide a detailed breakdown of your project’s milestones and each of the main tasks or activities to reach the milestone plus the expected timeline for the delivery.

Milestone 1: Literature Review & Community Surveys (2 months)

• Conduct a literature review
• Conduct community surveys

Milestone 2: Content Collection and Curation (2 months)

• Collection of all existing Cardano Catalyst instructional materials
• Collect proposals, community reviews from Catalyst API
• Potentially gather transcripts of existing video and audio recordings for LLM decoding
• Discussion of results in after town hall breakout room with community to ensure community review feedback

Milestone 3: LLM Evaluation (3 months)

• Review different LLMs
• Narrow the choices for the optimal LLM for the Catalyst use case
• Research how to reduce training costs based on LLM capabilities
• Discussion of results in after town hall breakout room with community to ensure community review feedback

Milestone 4: Experimentation & Alterations (1 month)

• Explore the infrastructure pipeline to handle LLM data processing, storage, and retrieval functionality
• Examine alterations to the proposal structure
• Discussion of results in after town hall breakout room with community to ensure community review feedback

Milestone 5: Report Writing and Presentation (1 month)

• Deliver research report
• Prepare visualizations, tables, or charts to present the findings.
• Share results with community

[CAPABILITY/ FEASIBILITY] Please describe the deliverables, outputs and intended outcomes of each milestone.

Milestone 1: Literature Review & Community Surveys

• Deliverable: Literature review and community survey
• Output: Report to community
• Outcome: Shared understanding of LLM literature and community feedback

Milestone 2: Content Collection and Curation

• Deliverable: Collected and curated data in open source repository
• Output: Open source repository of data
• Outcome: Collected data serving as input to the future training of LLM

Milestone 3: LLM Evaluation 

• Deliverable: Report reviewing different LLMs
• Output: Report to community
• Outcome: Optimal LLM to be used in future implementation

Milestone 4: Experimentation & Alterations

• Deliverable: Technical architecture description
• Output: Description of the LLM data infrastructure (processing, storage, retrieval) and potential alterations
• Outcome: Reduced friction of design, experimentation and cost for future implementations

Milestone 5: Report Writing and Presentation

• Deliverable: Research report
• Output: Report with visualizations, tables, and/or charts to present the findings
• Outcome: Wolfram and Catalyst community will have shared knowledge and approach of AI that can support the evolution of the Catalyst system processes.

[RESOURCES & VALUE FOR MONEY] Please provide a detailed budget breakdown of the proposed work and resources.

Milestone 1: Literature Review & Community Surveys - ₳94,600

• Conduct a literature review
• Conduct community surveys

Milestone 2: Content Collection and Curation - ₳189,600

• Collection of all existing Cardano Catalyst instructional materials
• Collect proposals, community reviews from Catalyst API
• Potentially gather transcripts of existing video and audio recordings for LLM decoding
• Discussion of results in after town hall breakout room with community to ensure community review feedback

Milestone 3: LLM Evaluation - ₳283,900

• Review different LLMs
• Narrow the choices for the optimal LLM for the Catalyst use case
• Research how to reduce training costs based on LLM capabilities
• Discussion of results in after town hall breakout room with community to ensure community review feedback

Milestone 4: Experimentation & Alterations - ₳94,600

• Explore the infrastructure pipeline to handle LLM data processing, storage, and retrieval functionality
• Examine alterations to the proposal structure
• Discussion of results in after town hall breakout room with community to ensure community review feedback

Milestone 5: Report Writing and Presentation - ₳47,300

• Deliver research report
• Prepare visualizations, tables, or charts to present the findings.
• Share results with community

[RESOURCES & VALUE FOR MONEY] How does the cost of the project represent value for money for the Cardano ecosystem?

The research of an LLM resource for the Cardano Catalyst process could be a game-changer. By leveraging the power of large language models, we can empower participants to navigate Cardano Catalyst with ease, providing real-time answers, guidance and reviewer support. Our comprehensive approach to research, content creation, and planning ensures the research will be produced effectively. Together, we can revolutionize the Catalyst system, foster innovation, and propel the Cardano community to new heights.

[IMPORTANT NOTE] The Applicant agreed to Fund10 rules and also that data in the Submission Form and other data provided by the project team during the course of the project will be publicly available.