Koios - Elastic Cardano Query Layer

Problem 1:

The current minimum requirements for building projects that need to query the blockchain in the Cardano ecosystem usually include running cardano-node and cardano-db-sync components that take significant resources/understanding/time to set up and maintain, as well as incur high financial costs to run this base layer infrastructure if adding failover support. Moreover, as Cardano grows, these requirements are getting more and more expensive (https://github.com/input-output-hk/cardano-node/issues/2978). We anticipate that future requirements will be too high for individuals to run such large setups.

Problem 2:

At the moment, there is no standard on what certain blockchain information sets should represent and what should be included. This means that every builder is taking their own approach to querying what should be the same dataset from the blockchain. This leads to re-inventing the wheel with almost every project jumping through similar hoops for working out their queries and leads to differences in object representation between builders. For example, different explorers show different values for the "live stake" dataset.

Koios API

The Koios API (Figure 1) will be an optimized query layer that qualifies as a flexible launchpad for builders removing the overhead costs of setting up, running and maintaining Cardano components (Problem 1), as well as offering API endpoints for both simple and complex data through open-source standardised queries (Problem 2). Figure 2 provides a more technical overview of the Koios API by illustrating most of its features listed below.

Koios features

1. Elastic infrastructure that is RESTful and easily scalable horizontally by adding more Koios instances. The instances are independent and capable of forming an independent branch as long as the health checks pass.
2. Flexibility for consumers, giving them the option to either query the KoiosAPI remotely with no software (light mode) or run a Koios stack locally if needed. A hybrid approach is also possible, with a preference for local connection but with failover to proxy (for example, during local upgrades).
3. Security - the components in use will allow easy encryption/authentication/DDoS control layers and not fall into smart queries causing slowness (one of the limitations if going down the GraphQL path).
4. No single point of failure from three different perspectives - entity-wise, component-wise and infrastructure-wise.
5. Standardized queries for Cardano datasets provide an easy way to add new ones, improve or use the existing queries to build on by having all queries open-source. These also include provisioning for cached layers of queries that can be resource-intensive and are not readily available (e.g. the live stake dataset for all pools). This has two major benefits:
   a) It drastically reduces the setting-up time cost for builders as they do not have to reinvent the wheel to get specific data from the blockchain (especially so for complex queries).
   b) It will reduce differences in object representations between different builders and encourage a standardized approach peer-reviewed by the community.
6. Open-source and always open to contributions from the community. This includes not just queries for different data, but also infrastructure setup scripts, documentation, etc. Everything in Koios remains open-source!
7. Architecture is carefully thought-over with optimal performance, failover, decentralization and prevention of duplicated work in mind.

How Koios differs from other projects with similar aims

This is best shown in points 1), 4), 5), 6) and 7), but we will try to shortly summarize it here. The Koios elastic infrastructure combined with being open-source allows easy scaling and flexibility for both lightweight and heavyweight clients - it is simple for projects to configure even their own private Koios clusters. Moreover, Koios is not controlled by any centralized entity, nor does it have a single point of failure in any aspect. Finally, open-source queries and object standards will encourage knowledge sharing and unified efforts to solve problems common to all builders.

Who we are

The Guild is a collection of Cardano enthusiasts consisting of multiple OGs/current/ex-ambassadors that have continually been making immense contributions to the ecosystem without any additional marketing or financial cost requests. We are creators of some of the many popular tools and resources in Cardano, including CNTools, gLiveView and topologyUpdater along with comprehensive Cardano documentation (https://cardano-community.github.io/guild-operators/). Through our contributions, we have developed a deep understanding of the Cardano blockchain and its components. Apart from our work on Cardano, Guild members have extensive experience in the technology field, with multiple (8+) years of experience across system design, DevOps, system administration, networking, and software development.

Priyank - https://github.com/rdlrt

Ola - https://github.com/Scitz0

Markus - https://github.com/gufmar

Michael - https://github.com/redoracle

Homer - https://github.com/hodlonaut

Damjan - https://github.com/DamjanOstrelic

Deliverables

• Setup scripts for creating complete onboarding of Koios instances with additional custom endpoints and making queries available on GitHub.
• Approach and work with interested explorer/wallet providers to collaborate and unify object representations, while making these available to the community.
• The first batch of data queries including all the base queries (e.g. list of registered stake pools), and the most requested complex ones (e.g. live stake distribution across pools).
• A commitment for stable setup of initial core Koios instances run by the core members until we see adoption & beyond. These will ensure live data availability to serve API, and the setup will be extended horizontally by other members who wish to join the project.
• Creating a set of rules for each endpoint to ensure any instance lacking in data automatically fails over to a working instance.
• Providing the same query layer availability over Testnet, Mainnet and Guild (a separate 30-min epoch network for running quick tests) Networks.
• Continuous maintenance and improvements to infrastructure and query layers.

Community Engagement

As the Cardano community represents the users of Koios, we want to hear and cater to its needs. Hence, 60-minute community meetings will be held on a bi-weekly basis open to anyone interested in the project. These will also provide a space for the core development team to report on current progress and issues, as well as open the floor to the community to give feedback, make requests, and get involved in discussions about Koios.

Budget

Costs for Koios can be separated into several categories:

1. Core instances server costs (12 months) - 4000 USD
2. Compensation for initial development phase (3 months) + maintenance (12 months) - 24000 USD
3. Community incentives (3 months) - 2000 USD

The community incentives will be used to incentivize top community collaborators on the project during the initial development phase.

Metrics

Contributions, activity and the continued development can be tracked through the public GitHub repository.

Code:

https://github.com/cardano-community/guild-operators

Documentation:

http://koios.rest/docs

Website:

https://koios.rest

Success timeline

0 - 3 months:

• Setup scripts for creating complete onboarding of Koios instances with additional custom endpoints and making queries available on GitHub.
• Approach and work with interested explorer/wallet providers to collaborate and unify object representations, while making these available to the community.
• The first batch of data queries including all the base queries (e.g. list of registered stake pools), and the most requested complex ones (e.g. live stake distribution across pools).
• Create a set of rules for each endpoint to ensure any instance lacking in data automatically fails over to a working instance.
• Provide same query layer availability over Testnet, Mainnet and Guild (a separate 30-min epoch network for running quick tests) Networks.

3 - 12 months:

• A commitment for stable setup of initial core Koios instances run by the core members until we see adoption & beyond. These will ensure live data availability to serve API, and the setup will be extended horizontally by other members who wish to join the project.
• Continuous maintenance and improvements to infrastructure and query layers.