Legacy Databases to Blockchain

Many thousands of applications are written to use traditional databases including MongoDB, CouchDB, Postgres, Mysql and others. Anyone* can write a so-called "oracle contract" that polls these databases for content, but these contents would not necessarily have demonstrable cryptographic integrity. These integrations would also be "one-off".

We seek to prototype and develop at least one meaningful solution for integrating traditional database applications, allowing their data to be used for reliably making progress in on-chain contracts, and re-integrating the results back to those applications and/or their databases.

Creating a positive experience for developers: We will provide recipes and guidance for developers to create meaningful connections between existing off-chain applications (or even new off-chain apps!) using well-known database technology, with integrations to the Cardano mainchain for key in-app activities not limited to acceptance of payments.

Other challenge-brief details:

• We expect that our work will directly impact the Number of developers actively building on top of Cardano because they'll be able to integrate existing applications and known operational platforms
• ...and attract developers from outside of our current community, for the same reasons
• "have database integration solutions" is one direct answer to What do developers want, to get onboard in Cardano
• ... and the same is true for enterprise dev managers - reducing their risk profile during blockchain integration projects

Gimbalabs will conduct this effort using our project-based learning program. We will also actively involve YOU, the community, in recognizing and building out the scope. Want to have your name on a budget line item for this proposal? Join us at a Gimbalabs Playground and be together with us.

The first slice: Database-to-sidechain

The first part of our current scope involves streaming changes from at least two databases, selected in consultation with the community, into a merkle-tree capturing the state of those databases as they change.

Once connected to the database's evolving state, capturing the progress of the database (similar to tracking the master branch of a git repository) involves:

1. tracking the changing merkle root (commit hash @master, over time)
2. committing the merkle root to the Cardano blockchain on an application-specified schedule using a smart contract
3. retaining the underlying data for future reference

In this last step, all of this database content can be held privately, all the data blinded on-chain behind cryptographic hashes. Its on-chain usefulness arrives in the third slice of our scope.

We plan to implement at least two connectors for traditional database servers and a "server agent" that implements the steps described above.

For the side-chain storage tier, we will evaluate at least three technical storage options that produce blockchain-compatible merkle trees, and choose at least two to use for prototyping, proving key success criteria for each storage technique, and reporting on other factors found in our study and research.

The second slice: Additional data extraction

A useful detail at step 1.1) above depends heavily on the application: some applications will get value from posting certain levels of additional detail about their database's merkle tree. These details can range from simply showing additional levels of their tree (still cryptographically blinded), to reveal selected public information onchain.

Because every application's needs are expected to be distinct if not unique, it's difficult to provide much additional detail about the capabilities at this step, beyond this: While these policies must be crafted to the specific needs of the application, their capabilities need not be limited by other than the applied ethics of the policy authors, constraints from the consensus rule of jurisdictional law, and consent of the other stakeholders engaged in relationship with the database oracle's owner/sponsor.

In this second slice, we will research and implement one or more mechanisms useful for application authors to indicate additional merkle-proofs and other data details to be anchored on-chain as part of the connector's normal operation.

We will also explore considerations and techniques targeted at balancing performance, transaction fees, and other runtime considerations, based on computational characteristics of the involved data structures, as well as through experience gained while applying such integrations. Exploring this technical landscape is a key set of research activities needed to identify, validate and mitigate risks, constraints, and opportunities... combining technique with vision and practical testing.

We will create a whitepaper reporting on the mechanisms we explored and results we found, with general recommendations and information about tradeoffs found during the research.

We will also provide training material demonstrating some practical use cases for this data-extraction capability in action, including integrating the results back to applications for further dataflow and human workflow.

The third slice: side-chain to contract: validating transactions with off-chain data

In the third slice of our scope, we will create prototype on-chain contracts to perform critical business-logic, taking into account key pieces of the underlying data. Remember, in slice one, hashes found in the merkle tree were the primary data anchored in on-chain transitions. At this stage, we address a need for on-chain smart contracts to USE some of the underlying data.

In this slice, we will develop protocols satisfying data-privacy goals and technical constraints, and allowing smart contracts to make progress while ensuring that the off-chain data from (1.1) above is and remains satisfying so-called "business-logic" constraints of the application.

The specific outcomes of this phase will include both the learnings found during the prototyping and research, which we will present as a whitepaper detailing practical considerations in achieving the goals, as well as reference smart contracts demonstrating key techniques for satisfying critical so-called "business logic" requirements for contracts collaborating with off-chain database applications.

This step involves some research and exploration of the range of possible implementation paths. Gimbalabs project-based-learning provides a perfect environment for exploring the problem space, analyzing effectiveness, and feeding back the results of the explorations back to the community.

Future: the fourth slice: re-integrating on-chain transactions to the traditional database

As transactions are executed on-chain (either in the transactions described above, or other transactions occurring on those same contracts), it will be useful to provide hooks for monitoring and re-integrating the results of those transactions to the application and/or to its database.

In the current proposal scope, we don't plan to deliver a result on for this slice, but we will incentivize GPLers to stretch and achieve it, providing at least two reference implementations of hooks that reflect meaningful updates back into the application tier, completing the cycle of integrating blockchain with traditional database applications.

Execution Plan: Community Engagement and Project-Based Learning

At Gimbalabs, learning through experience is one of our core values. Developing community and shared ownership is another.

Existing followers of the Plutus Pioneers program will be invited to be part of our project-based learning program, where the Catalyst funds will be used to grow the experience of Plutus learners and provide them with rewards for learning.

We will sponsor 3 community-development leaders, who will be rewarded directly for working together with our learners and for integrating the best results of the learning teams, forming the implementation. We will also sponsor a technical product manager and architect to provide leadership and guidance, assist with development of incremental, actionable milestones to the learning team. 

We are also allocating a focused reward for part-time contribution from two technical team members as needed to ensure we have every skill needed.  Are you at the top of your class?  Come and be with us on this amazing journey.

Details TBD: all of our plutus learners will earn a participation reward. We will issue bonus rewards for the first effective solution, the first fully unit-tested solution, and the best quality solution (passing the unit tests) at each milestone. Milestone-level rewards will be distributed from the total reward pool according to the number and difficulty of the separate milestones.

Randall Harmon: https://www.linkedin.com/in/randall-harmon-aa52765

This proposal is cosponsored by Gimbalabs, where we are mobilizing everyone to develop tools and applications through a unique experience of co-creation that facilitates adoption of the Cardano protocol, reveals new possibilities, and ignites the public imagination worldwide. Gimbalabs serves a large group of developers through its Playground network, which provides networking and professional development to the Cardano community.

Gimbalabs has been doing the hard work during the last 4 funding rounds, creating products such as the Dandelion APIs and the Dandelion Incentivized Alpha Network. We have been champions of project-based learning, and we're pleased to refer you to our latest video update (8 Sept 2021), in which we're demoing our 1:1 Scheduling application, highlighting our DiD functionality, and a revenue-enabling platform for people to trade skills, assistance, and ADA.

https://www.youtube.com/watch?v=DQxDmU6GqPY

ROADMAP

Our proposed roadmap, to be developed further in teamwork with the community and Gimbalabs PBL teams:

Phase 1: @1 Month:

• Enjoying Charles Hoskinson's highlighting these kinds of sidechain initiatives as part of the Cardano story
• Essential connector to read change-streams from one trad database for Slice 1
• Plutus prototyping of on-chain validation using off-chain data with merkle proofs for Slice 3 (using mock side-chain); explore and test multiple approaches to enable high-assurance progress for such contracts.
• Reports on review of side-chain solutions
• Plutus contract for posting side-chain results as an oracle contract
• (stretch): Prototype implementation of one blockchain-compatible data-capture technology for Slice 1.

Phase 2: @3 Months:

• Whitepaper: reports on learnings from phase 1
• Connector for a second trad database for Slice 1
• Prototype for second blockchain-compatible data-capture technology for Slice 1
• Experiment with Slice 2 data extraction and on-chain db-state anchoring, focusing on performance and transaction size/complexity/fees.
• Prototype for Slice 3: experiment with techniques guarding security and privacy, in tandem with Slice 2 experiments.
• Enhance Plutus prototypes based on community and peer review
• SUCCESS: delivery of above, and one or more videos as needed to present and explore the problem/solution space covered

Phase 3: @6 Months:

• Whitepaper: report on learnings from Slice 3
• Reference smart contracts demonstrating techniques for business-logic assurance
• SUCCESS: delivery of above, and one or more videos to broaden accessibility of the information and learnings
• DELIVERY by Jun 1, 2022

...at 12 months:

 SUCCESS: Gimbalabs and other organizations using the developed techniques to integrate assurance of data in off-chain applications into the blockchain record. 

Example projects integrating traditional database applications with blockchain

...Compared to other proposals:

This proposal is distinct from:

https://cardano.ideascale.com/a/dtd/Graph-DB-Sidechain-with-Fluree/352531-48088

The Fluree sidechain DB project from Catalyst Fund 5 is one data-storage layer that this project could be connected with, but that project does not seek to address the consideration of integrating with existing off-chain applications. We will consider Fluree as a side-chain storage layer and support any of our learners who choose Fluree for prototyping

Proposed Budget:

• Architectural planning and review: $7,000
• Project Management: $10,100
• Learning Rewards: $21,000
• Community development leaders: $26,400
• Additional developers (x2, part-time): $6,024
• Develop and present training materials: $6,500

Total: $77,024