Wolfram Cardano Exchange Data Infrastructure

[GENERAL] Name and Surname of Main Applicant

Jon Woodard

[GENERAL] Email address of Main Applicant

jwoodard@wolfram.com

Additional Applicants

Steph Macurdy

Gabriela Guerra

[GENERAL] Please specify how many months you expect your project to last (from 2-12 months)

9

[GENERAL] Please indicate if your proposal has been auto-translated into English from another language.

No

[GENERAL] Does your project have any dependencies on other organizations, technical or otherwise?

Yes

[GENERAL] If YES, please describe what the dependency is and why you believe it is essential for your project’s delivery. If NO, please write “No dependencies.” in this field.

We use the CCXT library which provides quick access to market data for storage. CCXT is an open-source data and trading library that supports over 100 exchanges.

[GENERAL] Will your project outputs be fully Open Source?

Yes

[GENERAL] If NO, please describe which outputs are not going to be open source. If YES, please write “Project will be fully open source.” in this field.

Project will be fully open source.

[METADATA] Category of Proposal

Dev Tools, API or Library

[IMPACT] Please describe your proposed solution.

Wolfram Blockchain Labs’ (WBL) mission is to advance the science of blockchains, making blockchain data computational and accessible. The task of collecting high-resolution financial data from a wide range of centralized crypto exchanges (CEXs) and decentralized crypto exchanges (DEXs) AND making it useful to developers is a complex undertaking.

Wolfram Exchange Data Infrastructure: Video Presentation

Real-time data aggregation services are critical components of financial applications. Providing an open-source component should advance the quantity of financial applications in the Cardano ecosystem by making it easier to start development.

Wolfram Blockchain Labs (WBL) is taking two simultaneous tactics to provide this data—collecting vertically through on-chain decentralized exchanges (DEXs) and horizontally across chains and centralized exchanges (CEXs).

Our open-source software will allow developers to deploy their own data collectors to store pair pricing data from the exchange sources with customizable intervals so they can build applications on top. Potential applications might range from visual dashboards to on-chain price feeds.

Horizontal data collection from CEXs

Through the CCXT library, we get quick access to market data for storage. CCXT is an open-source data and trading library that supports over 100 exchanges. 

We have set up one collector per exchange that pulls the following trading pair data: 

• Order book 
• Trade 
• Ticker 
• Candlestick 

Once the response is received, this data is stored in a database, making it easily retrievable by app developers. In our PoC (available in the GitHub repo attached to this application), we have integrated five exchanges and five trading pairs.

We will expand this coverage to 10+ exchanges and 100+ trading pairs. Also, we will connect the tokens and coins to ITSA.global ITINs for linking additional data.

Vertical data collection from DEXs 

We will build a solution that incorporates the top native token pairs listed in Cardano’s decentralized exchanges (DEXs) to empower developers to generate new applications on top. We will also connect the relevant Cardano tokens (up to a maximum of 50) to ITSA ITINs for linking additional data.

Additionally, we plan to incorporate pricing information obtained from the leading three Ethereum DEXs, including Uniswap.

Connecting to ITSA's "International Token Identification Number" establishes an industry standard for token identification which could be helpful for developers across blockchains, applications and interoperability protocols.

Development 

To ensure the smooth and consistent deployment of our services, we have implemented a CI/CD pipeline that constructs Docker images of the updated services, allowing them to be retrieved by other developers on the target machine or cloud service.

Service

As a benefit to the Cardano community, we will provide an API with access to the data we collect. This API will operate continuously for a duration of two years, ensuring sustained availability of the service, provided the volatility of ADA is low. To maintain a reasonable and secure usage environment, appropriate rate limits will be implemented to prevent misuse that could potentially impact the overall performance and reliability of the system. By offering this open API, we aim to empower developers and stakeholders within the Cardano ecosystem with convenient and reliable access to our collected data, fostering innovation and enabling the development of diverse applications and services.

General Improvements 

We will also add several general improvements to ensure that the project is more usable as a component for financial applications, such as queuing to improve data collection, historical data compression and system status monitoring. Additionally, WBL recognizes the significance of the broader cryptocurrency exchange ecosystem for pricing data. For example, significant volume and economic activity occurs on decentralized exchanges like Uniswap. With this in mind, the Wolfram Cardano Exchange Data Infrastructure project aims to incorporate DEXs across the spectrum. These DEX services represent significant data sources for obtaining a more comprehensive understanding of the broader cryptocurrency ecosystem.

Summary of Project Objectives: 

• Make significant architectural improvements to enhance the stability of data collection and enable simultaneous searching across multiple exchanges, among other features 
• Generate an archive with historical pricing data for relevant pairs 
• Enable data streaming of aggregate pricing data from Centralized Exchanges through an API 
• Incorporate Cardano DEX pairs into our current data collectors 
• Incorporate top three Ethereum DEXs into our current DEX data collectors for additional pricing data
• Provide an open-source package that enables other developers to generate a pricing database according to their financial application needs

Overall Benefits: 

• Open-source pricing database 
• API (REST and WebSocket) of open, high, low, close, pricing, and coin/token volume data 
• Ability to adapt/fork the database
• Expansion to the top 100 trading pairs with connections to ITSA.global ITINs for additional information 
• Expansion to incorporate Cardano DEXs and price data for the top 100 Cardano tokens 
• Expansion to incorporate top 3 Ethereum DEXs
• Reduced friction for third-party developers to build dashboards, trading strategies, smart contracts, etc. 
• Database compression algorithm to save space and reduce cost overtime.
• Tool to enable system monitoring of the servers collecting the data.

Our team has experience integrating various blockchain ecosystems into our technology stack and is excited to continue incorporating Cardano exchange data into the foundation of our services.

[IMPACT] How does your proposed solution address the challenge and what benefits will this bring to the Cardano ecosystem?

The Development and Infrastructure challenge asks, "what research, tools or software can improve the developer ecosystem or infrastructure to make it easier to build and scale on the Cardano blockchain?"

Our proposed solution addresses this challenge through the following sub-category:

• Data oracles, indexers & explorers
• Tools and software that integrate real world data into Cardano or that make existing Cardano data more easy to use and accessible.

Our proposed solution makes it easier for developers to access financial exchange data (both CEXs and DEXs) with open software tools. We hope to enable enhanced data accessibility to this information and empower developers to build applications. We strive to catalyze their creativity by building on top of the software tools we make available.

We believe that achieving this goal directly answers the challenge proposed in the Development and Infrastructure category. This tool will enable ourselves (Wolfram Blockchain Labs) and other developers to contribute to the overall growth of the Cardano ecosystem.

[IMPACT] How do you intend to measure the success of your project?

The success of our project will be reflected in the availability of our software to the Cardano developer community with instructions to easily spin up in their own hardware and/or cloud service. We will measure the success of our project in the following ways:

• Coverage and Data Quality:
• Assess the extent of coverage in terms of the number of exchanges, trading pairs, and Cardano DEXs incorporated into the data collectors.
• Additionally, evaluate the quality and reliability of the collected data from exchanges by analyzing factors like accuracy, consistency, and timeliness.
• We will also provide access to monitor the status of the server architecture with the intention to understand the overall effectiveness of system in action.
• Scalability:
• Server infrastructure, data storage, and resource utilization have been designed with scalability in mind.
• Additionally, this architecture can accommodate an increasing number of exchanges, trading pairs, and users.
• Usage
• Any incremental usage from open source developers will be considered a success.

[IMPACT] Please describe your plans to share the outputs and results of your project?

The outputs and results of our project will be shared through a dedicated repository. The repository will serve as a central hub where we will make the code, documentation, and other relevant project artifacts available to the public.

[CAPABILITY/ FEASIBILITY] What is your capability to deliver your project with high levels of trust and accountability?

Wolfram Blockchain Labs (WBL) is a subsidiary of the 36-year-old Wolfram Research that is exclusively licensed with high-performance blockchain technologies and specifically designed to extend ecosystem tools for application development for distributed ledger technologies (DLTs). WBL often uses additional expertise in specialized areas from both Wolfram Research and Wolfram Technical Consulting teams. The latter is an organization that has completed over three hundred consulting projects in the past decade.

Our capability to deliver the project with high levels of trust and accountability is grounded in several key factors:

• Track Record: Wolfram Blockchain Labs (WBL) has a proven track record in successfully completing substantial projects (Catalyst included), for which we have completed four projects already and have three more near completion. We've gone through the Milestone pilot in Fund 9 and the subsequent revision process, through two sets of proof-of-achievement processes and one of our projects was used as an example for good documentation (DADF - Fund 9). We understand the governance process and support the mission to improve and grow auditability of the Catalyst governance system.
• Experienced Team: Our team comprises professionals from diverse disciplines, including blockchain technology, finance and project management. Our multidisciplinary approach brings a comprehensive and holistic perspective to the project, ensuring that all aspects are addressed with a high level of expertise.
• All of four completed projects across Fund 8 and Fund 9 have occurred within budget.

This experience relays our ability to meet project objectives, timelines, and budgetary constraints and deliver results. We believe we have been good stewards of the Catalyst process and are well-suited to deliver the project with high levels of trust and accountability.

[CAPABILITY/ FEASIBILITY] What are the main goals for the project and how will you validate if your approach is feasible?

We know this approach is feasible and have a proof-of-concept completed to show for it. The main goals of the project are to expand data coverage and to create a tool for others to use as well. Below we explain the feasibility of our approach:

• Data Coverage: Expand data coverage in terms of the number of exchanges (10), trading pairs (100), Cardano DEXs (5) and several Ethereum DEXs (3) incorporated into the DEX data collectors. We will monitor the progress by tracking the percentage increase in coverage over time.

• Open Tool: Freely accessible tool for developers to employ their own software and hardware infrastructure to collect data from both centralized and decentralized exchanges. This will be made available through our open repository.

[CAPABILITY/ FEASIBILITY] Please provide a detailed breakdown of your project’s milestones and each of the main tasks or activities to reach the milestone plus the expected timeline for the delivery.

1. Historical data and source expansion (1 month)
2. Expand number of pairs and exchanges
3. Identify publicly available sources for historical pricing data of the specified pairs
4. Implement data fetching from these sources
5. Testing
6. DEX Collectors (2 months)
7. Implement driver for Cardano node
8. Implement DEX contract drivers
9. Implement collectors for DEX contracts
10. DEX Collector REST API and Testing
11. Update REST API
12. Testing
13. Data compression (2 months)
14. Data compression algorithm
15. Data compression Testing
16. System Status Monitoring (2 months)
17. Data visualization
18. Stress testing
19. Equity Data Infrastructure (2 months)
20. Equity data infrastructure design
21. Equity data infrastructure implementation
22. Equity data infrastructure testing

[CAPABILITY/ FEASIBILITY] Please describe the deliverables, outputs and intended outcomes of each milestone.

1. Historical data and source expansion
2. Deliverable: Function(s) that allow historical data to be loaded in the deployed database
3. Output: Functional component of the docker image which provides access to populate historical (CEX) financial data easily
4. Outcome: Devs will have the ability of populating historical data for their desired use case
5. DEX Collectors
6. Deliverable: Software module that connects to the Cardano Blockchain to get DEX pricing data. Software module that connects to Ethereum DEXs to get pricing data.
7. Output: Functional component of the docker image which provides access to start collecting DEX financial data.
8. Outcome: Devs will have the ability to store native Cardano pair pricing from various exchanges as a basis for their applications.
9. Data compression
10. Deliverable: Platform functions that after a determined amount of time take a section of the data and replace it with a single aggregate line.
11. Output: A function to compress data across time intervals to enable greater storage capacity.
12. Outcome: Data compression and long term resource optimization.
13. System Status Monitoring
14. Deliverable: A simple tool that allows monitoring of the deployed servers.
15. Output: Metrics for understanding system status.
16. Outcome: The system status monitoring tool will give developers valuable information to ensure the reliability of their deployed servers.
17. Equity Data Infrastructure
18. Deliverable: A data collector infrastructure for traditional equity data.
19. Output: Functional component of the docker image which provides access to start collecting traditional equity financial data.
20. Outcome: This collector will allow developers to store traditional equity data (not including the required data licenses).

[RESOURCES & VALUE FOR MONEY] Please provide a detailed budget breakdown of the proposed work and resources.

1. Historical data and source expansion (1 month) - ₳78,400
2. Expand number of pairs and exchanges
3. Identify publicly available sources for historical pricing data of the specified pairs.
4. Implement data fetching from these sources.
5. Testing
6. DEX Collectors (2 months) - ₳187,500
7. Implement driver for Cardano node
8. Implement DEX contract driver
9. Implement collector for DEX contracts
10. DEX Collector REST API and Testing
11. Update REST API
12. Testing
13. Data compression (2 months) - ₳187,500
14. Data compression algorithm
15. Data compression Testing
16. System Status Monitoring (2 months) - ₳119,300
17. Data visualization
18. Stress testing
19. Equity Data Infrastructure (2 months) - ₳255,700
20. Equity Data Infrastructure Design
21. Equity Data Infrastructure implementation
22. Equity Data Infrastructure Testing

[RESOURCES & VALUE FOR MONEY] How does the cost of the project represent value for money for the Cardano ecosystem?

WBL wants to supply Cardano and it's users with real-time crypto financial data and price feeds. In order to bring this application to market, we needed to build the infrastructure so that we could access the complex spectrum of crypto financial data. Crypto financial data is difficult and expensive to work with. The reason it is difficult is because the true value of crypto financial data is that it needs to be as real-time as possible. If crypto financial data is not reflecting the real price in real time, then the application or entity providing the data can be arbitraged. Arbitrage is the process by which a difference in price is exploited in order to remove the discrepancy. When dealing in crypto financial markets, this can be a catastrophic error and cost applications or entities potentially billions of dollars.

For this reason, crypto financial data requires high performing hardware, efficient software and data management, and human expertise to implement a solution.

Furthermore, apart from cryptocurrency financial data, the solution has the capability to accommodate equities financial data as well. However, it's important to note that this particular price does not cover subscriptions to that data (which can run into multiple hundreds of thousands a year).

Because WBL operates within the Cardano ecosystem, our solution to this problem is to open source the tool we've built to eventually use for ourselves in the application for delivering financial data and price feeds back to the community. We've consulted many organizations around this problem and have found expertise in many areas and feel confident that we can deliver this solution. This has been a time consuming and expensive process, it will require further expertise to bring the full solution to market, and has several moving parts and teams to manage. This is why we are requesting the amount of $200,000 USD to complete and deliver the project.

[IMPORTANT NOTE] The Applicant agreed to Fund10 rules and also that data in the Submission Form and other data provided by the project team during the course of the project will be publicly available.

I Accept