Cardano on BigQuery: scalably querying Cardano’s authenticated blockchain data on BigQuery

[GENERAL] Name and surname of main applicant

Alexander Diemand

[GENERAL] Email address of main applicant

adiemand@proton.me

Additional applicants

Thomas Kaliakos (@thomaska)

Bitseat Tadesse (@bitseatt)

[GENERAL] Please specify how many months you expect your project to last (from 2-12 months)

12

[GENERAL] Please indicate if your proposal has been auto-translated into English from another language.

No

[GENERAL] Does your project have any dependencies on other organizations, technical or otherwise?

No

[GENERAL] If YES, please describe what the dependency is and why you believe it is essential for your project’s delivery. If NO, please write “No dependencies.” .

No dependencies.

[GENERAL] Will your project’s output/s be fully open source?

Yes

[GENERAL] If NO, please describe which outputs are not going to be open source. If YES, please write “Project will be fully open source.”

Project will be fully open source.

[METADATA] Category of Proposal

Data tools

[IMPACT] Please describe your proposed solution.

This project addresses the challenge of providing free and open data access to Cardano’s blockchain which constantly grows and requires increasing technology to be thrown at the task. Because the network has agreed on the finalisation of the on-chain data, these data can be shared with everybody in its most practical and cheapest form, trustless. Our back validation procedure builds a proof of data authenticity of the exported data in BigQuery.

This project might have a great impact on the Cardano community as it provides an accessible and affordable data basis to launch DApps, web3, NFT, data analytics, and other projects with ease.

[IMPACT] How does your proposed solution address the challenge and what benefits will this bring to the Cardano ecosystem?

Earlier stages of this project have been originating in 2022 within IOG’s Data Analytics team and we know of a number of projects in the Cardano community which already build on Cardano_on_BigQuery.

To secure long-term stability of the project we ask for funding the transition to a community owned project which is completely open source.

The more users are involved the greater the potential for future development. The more shoulders that carry the burden of maintaining Cardano_on_BigQuery the lower the costs.

We believe that this project enables small and mid sized projects on the Cardano blockchain to implement their data layers: scalability at minimal, usage related costs.

[IMPACT] How do you intend to measure the success of your project?

As success for this project we consider the raising number of great projects developing on Cardano. This project enables DApps, web3, transaction explorers, on-chain analytics, and other types of adoption related works.

We are looking for testimonials of users of the BigQuery dataset. Maybe through our Twitter account: @CardanoBigQuery

There are no metrics available to determine the number of users connecting to the public BigQuery dataset. We can however get some access statistics from the back validation dashboard. And, we hope to get magnificent contributions in the Github repository.

[IMPACT] Please describe your plans to share the outputs and results of your project?

We are going open source from day one as all development and documentation is happening in our public Github repository.

All data in BigQuery is publicly available.

All meta data and the results of the back validation is publicly accessible as well as the dashboard summarising the results.

[CAPABILITY/ FEASIBILITY] What is your capability to deliver your project with high levels of trust and accountability?

We have more than two years experience with our own developed Db-Sync Enterprise protocol which minimises downtime of a Db-Sync pipeline.

The export of the Cardano blockchain data to BigQuery has been running for ten continuous months without interruption, until June 2023.

The project team covers all aspects required to achieve the stated goals.

[CAPABILITY/ FEASIBILITY] What are the main goals for the project and how will you validate if your approach is feasible?

G1: Continuous update of the BigQuery dataset with new block data from Cardano’s chain

Feasibility: back validation as a proof of data authenticity

G2: Open source the code and document the processes

Feasibility: Github repo with wiki

G3: Communication with users of the data and the Cardano community

Feasibility: feedback from the community, collaboration with users

[CAPABILITY/ FEASIBILITY] Please provide a detailed breakdown of your project’s milestones and each of the main tasks or activities to reach the milestone plus the expected timeline for the delivery.

M1: create infrastructure for the project (1 month, in-progress)

M2: setup of the export and its continuous processing (1 month)

M3: deep comparison to create authoritative data equivalence with Cardano’s blockchain (1 month)

M4: fully document the update process and the data quality monitoring (1 month)

[CAPABILITY/ FEASIBILITY] Please describe the deliverables, outputs and intended outcomes of each milestone.

D1: BigQuery dataset of Cardano blockchain data publicly available

D2: Open source code repository containing all code of the update and monitoring processes

D3: Metadata and dashboard of back validation publicly available

D4: complete documentation

D5: continuous communication with the community on the update of the BigQuery dataset

[RESOURCES & VALUE FOR MONEY] Please provide a detailed budget breakdown of the proposed work and resources.

B1: 50 PD for architecture, design, project management, documentation, communication

B1.1: 10 PD architecture & design

B1.2: 10 PD project management

B1.3: 10 PD documentation

B1.4: 20 PD communication

B2: 20 PD for systems engineering (devops)

B2.1: 10 PD infrastructure setup (redundant hardware, high-availability)

B2.2: 10 PD process monitoring, alerting, mitigation procedures

B3: 55 PD for data engineering

B3.1: 5 PD PostgreSQL optimisations

B3.2: 10 PD BigQuery maintenance

B3.3: 20 PD Update process

B3.4: 20 PD Deep comparison process (back validation)

B4: Infrastructure costs

B4.1: $680 per month for redundant server hardware

(PD = person day; 8 hrs/day; 1 hr = $90)

Sum PD = 125 person days

At rate $90/h, 8 hrs/day: Sum budget PD = $90,000

12 months running costs: Hardware $680 x 12 = $8160

Total Budget: $98,160

[RESOURCES & VALUE FOR MONEY] How does the cost of the project represent value for money for the Cardano ecosystem?

Blockchain data is by its definition equal for all participants of the network. So it makes sense to share these data in their most practical form such that each participant can independently work with it. We believe that SQL is the most accessible way of querying data and everybody will find a way in their own setup to connect and query from the BigQuery dataset which is always on.

Trustless data querying is enabled by our back validation which proves that the data in BigQuery really represents the on-chain Cardano blockchain.

Running the complete Db-Sync pipeline amounts to costs of several hundred dollars per month. On the other hand, BigQuery offers a free monthly quota of 1 TB queried data and usually incurs no costs if used sparingly.

[IMPORTANT NOTE] The Applicant agreed to Fund10 rules and also that data in the Submission Form and other data provided by the project team during the course of the project will be publicly available.

I Accept