NuNet: Enabling Elasticsearch Clusters on decentralized hardware // Elasticsearch is a fast and scalable search and analytics engine, providing powerful analytics and visualization tools. NuNet will enable running it on its decentralized network.

[GENERAL] Name and Surname of Main Applicant

Ilija Radeljic

[GENERAL] Email address of Main Applicant

ilija.radeljic@nunet.io

Additional Applicants

Yitbarek Yohannes, MSc CS

Lead researcher and developer

Kabir Veitas, PhD

CEO and lead architect

[GENERAL] Please specify how many months you expect your project to last (from 2-12 months)

11

[GENERAL] Please indicate if your proposal has been auto-translated into English from another language.

No

[GENERAL] Does your project have any dependencies on other organizations, technical or otherwise?

Yes

[GENERAL] If YES, please describe what the dependency is and why you believe it is essential for your project’s delivery. If NO, please write “No dependencies.” in this field.

For the development we shall use only open source frameworks as follows: 

• Containerisation (e.g. Docker) for hardware standardization
• Selected components of Elastic Stack
• Native peer-to-peer libraries Go libp2p for decentralized network communication (e.g. Go libp2p)

These technical dependencies do not require permissions to use outside Open Source licensing and are taken into account in our feasibility study. Therefore, integrating these dependencies will not cause any delays.

[GENERAL] Will your project outputs be fully Open Source?

Yes

[GENERAL] If NO, please describe which outputs are not going to be open source. If YES, please write “Project will be fully open source.” in this field.

Project will be fully open source in line with our licensing policy.

[METADATA] Category of Proposal

Data, Explorers, Trackers

[METADATA] SDG rating

SDG Goals

• 9 - Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation
• 10 - Reduce inequality within and among countries

SDG Subgoals

• 9.1 - Develop quality, reliable, sustainable and resilient infrastructure, including regional and transborder infrastructure, to support economic development and human well-being, with a focus on affordable and equitable access for all
• 9.5 - Enhance scientific research, upgrade the technological capabilities of industrial sectors in all countries, in particular developing countries, including, by 2030, encouraging innovation and substantially increasing the number of research and development workers per 1 million people and public and private research and development spending

[IMPACT] Please describe your proposed solution.

Problem:

Modern applications, especially the ones involving ML and AI need access to massive amounts of big data. Hosting databases on a cloud is expensive and makes data storage centralized to big data providers, hindering access to data sources directly from decentralized networks.

Unique solution:

The proposed solution is to enable consumers to deploy an Elasticsearch cluster with custom configuration on decentralized hardware available via NuNet.

NuNet will enable anyone with eligible devices to host Elasticsearch nodes, configure them into clusters and be rewarded via Cardano Smart Contracts. For these purposes a number of new features will be added to the platform.

More info on the many application areas of Elasticsearch can be found here:

https://www.elastic.co/

All these application areas could be enabled to run on NuNet as an alternative to big cloud service providers.

Detailed approach:

This solution will add new features on the existing NuNet platform. The solution can be seen from three perspectives: 1) consumer, 2) provider and the 3) NuNet platform. 

Consumers:

A consumer will be able to request the deployment of Elasticsearch cluster with custom configuration via the NuNet network. When this solution is done, a consumer will either select nodes to create a cluster or they will be automatically assigned machines based on cluster configuration and available machines. The consumer will be able to access, monitor and manage the cluster.

Providers:

Currently device providers on NuNet can onboard their device and provide compute resources for running machine learning workflows. This use-case solution and new platform features will enable providers to host Elasticsearch nodes as they will.

NuNet platform:

New features will be added on the NuNet platform so that Elasticsearch clusters can be formed dynamically as per consumer requests and hosted for the predefined time or indefinitely. Under the hood, NuNet platform will treat Elasticsearch nodes as a single continuously running job. So, the existing design and components of NuNet are suitable to manage the Elasticsearch cluster. 

Benefits for the Cardano ecosystem:

This solution is important to Cardano in two ways:

First, projects which need data hosting and indexing solutions, like NuNet’s partner IAMX, which retrieve data from the Cardano nodes and store it on the cloud providers, will have a better alternative to store their data on decentralized hardware. Elasticsearch provides many other application areas where it could be deployed.

Second, the payment for these services will be in NTX (a Cardano native token), where each transaction will be executed as a Smart Contract on the Cardano blockchain which will directly increase the volume of tx, volume of CNT as well as provide unique use cases to be built on top of it for the Cardano ecosystem. This will provide flexible data hosting pricing and use-case monetization options for Web3 application developers.

[IMPACT] How does your proposed solution address the challenge and what benefits will this bring to the Cardano ecosystem?

The proposal addresses the following directions of the challenge:

• Data oracles, indexers & explorers

This project helps solutions that store and consume blockchain data, along with any other data that needs to be efficiently indexed and queried. Cardano blockchain explorers can deploy Elasticsearch clusters on NuNet and configure them in order to perform fast custom queries for internal use or providing services for the community. This encourages more blockchain exploration and data storage / indexing solutions to be implemented and monetized via the Cardano ecosystem. 

Also, other startups building novel solutions in the ecosystem that need a database for their application logic will get an advantage and a head start.

IAMX partnership

One of the direct project benefits is the deployment of Elasticsearch clusters on NuNet, specifically configured for project partner’s IAMX use-case, which will serve for the first real-world usage and reference implementation. IAMX currently has a workflow where they have the need for an Elasticsearch cluster, which is in turn deployed on their cloud/bare metal systems.

IAMX is developing their own KYC/KYC solution on the Cardano blockchain and uses a Cardano native token IAMX for their processes.

This partnership between two Cardano native projects will be a direct benefit for the Cardano ecosystem. In addition, when live, this feature will enable Elasticsearch use cases to be used by the wider Cardano community as well as anybody willing to deploy it on decentralized systems - NuNet.

More about IAMX: https://iamx.id/

[IMPACT] How do you intend to measure the success of your project?

This proposal is an implementation and can be measured with the total number of consumers, providers and Elasticsearch cluster deployments.

NuNet will be monitoring the adoption of the solution and reporting direct interest and users during the project milestones. We anticipate at least 1 partner to be onboarded.

Some of the direct benefits to the Cardano ecosystem are:

• Computing resources used in the processes will be compensated in NTX, which is a Cardano Native Token
• Each exchange of value will be done as a Smart Contract on Cardano
• Currently over 2000+ people are in NuNet Discord testing the various builds of the NuNet platform

The indirect benefits to the Cardano ecosystem is:

• Other solutions in the Cardano ecosystem will be able to deploy Elasticsearch cluster on NuNet and provide a decentralized hardware provision for data storage and exploration needs;
• Moreover, implementation of platform features, needed to deploy Elasticsearch clusters via NuNet will enable further development of data hosting solutions via NuNet using mainstream databases

[IMPACT] Please describe your plans to share the outputs and results of your project?

Spreading Outputs Over a Timescale

Our project plan includes clear milestones and deliverables, which will be shared publicly as they are completed. This incremental release of outputs will ensure a continuous stream of updates for the community.

This approach lets us provide updates on a regular basis, and offers users the chance to provide feedback that we can use to guide subsequent development.

Sharing Outputs, Impacts, and Opportunities

We intend to leverage various communication channels to share our project's outputs, impacts, and opportunities:

1. GitLab: The primary hub for our technical work, hosting our codebase, documentation, and issue tracking. This will be the main point of reference for the details of our project.
2. Social Platforms: We plan to regularly post updates on our progress on platforms like Twitter, LinkedIn, and Reddit. This will include major milestones, bug fixes, and insights from our development work.
3. Technical Discussions: We will continue to hold weekly technical discussion where we discuss the technical aspects of our work. This provides a forum for live Q&A and discussion with our community.
4. Blogs: A regular blogs to summarize the progress we have made, highlighting key achievements and outlining the next steps in our project.

Testing and further research

As an open-source project, our outputs will be freely accessible for further research and development. We encourage the community's involvement in testing our solutions to enhance their real-world performance.

Community Testing: We'll invite our users to participate in alpha and beta testing phases, where they can help identify bugs and suggest improvements. We'll use GitLab's issue tracking for managing feedback and provide guidelines for issue reporting and feature suggestions.

Internally, we'll use project insights and community feedback to guide our future work, optimize performance, and prioritize new features. Our aim is to foster a collaborative development ecosystem that is robust, relevant, and of high quality.

[CAPABILITY/ FEASIBILITY] What is your capability to deliver your project with high levels of trust and accountability?

Illustration of Capacity:

Our organization comes with a history of successfully bringing intricate technology projects to fruition. The pillars of our success lie in our deep-rooted technical understanding, stringent project management practices, and an unwavering focus on transparency and responsibility.

Our team is populated with seasoned software engineers with excellent skills to leverage containerization (Docker), Elastic Stack , Blockchain and peer-to-peer networking (Go libp2p). NuNet past work includes the implementation of projects in some sense similar to the one proposed here, showcasing our readiness to tackle the unique challenges this project poses.

NuNet is committed to Open Source Software development from the inception. Therefore, all our development and progress is available for public scrutiny at all times as well as open collaboration with the community. We actively invite and work with the community in regards to contribution, usage, work and testing of the platform codebase.

Link: https://gitlab.com/nunet

NuNet licencing policy:

https://docs.nunet.io/nunet-licensing-policy/

Openness and Responsibility:

We have established a robust framework to ensure openness and responsibility in the execution of the project and the management of finances:

1. Elaborate Budgeting: We present an exhaustive budget layout at the start of the project that details the fund allocation across various tasks. This leaves no room for ambiguity regarding the utilization of funds.

2. Periodic Reporting: Regular updates regarding the project and financial statements will be shared, offering complete transparency into the progression of the project and the use of funds.

3. External Auditing: We are open to audits conducted by independent third parties at regular intervals. This ensures responsibility and openness in our financial management.

4. Escrow Mechanisms: To further reassure proper use of funds, we can utilize an escrow service. This arrangement ensures that the project funds are held by a third party and released according to pre-set milestones. This provides an extra layer of assurance for the funds.

5. Payment Based on Milestones: Our payment structure is built around specific, agreed-upon milestones. This ensures that funds are released as we achieve these milestones. The completion of each milestone can be verified, ensuring you pay only for verifiable progress.

These measures reflect our commitment to openness, responsibility, and proper management of funds. We believe that these factors, along with our technical capabilities, make us an ideal choice to successfully execute this project.

We understand that not all steps we have implemented are valid for the Catalyst proposal but it demonstrates the internal working procedures we have in place.

Catalyst Experience

NuNet also has received the funding for proposals in Fund7 and Fund8. One proposal is successfully closed and the other is close to completion with one technical obstacle left to be solved. Overall, the funds were spent as intended on the development which can be monitored on Gitlab with daily commits since the award.

https://gitlab.com/groups/nunet/-/milestones/19#tab-issues

https://gitlab.com/groups/nunet/-/milestones/20#tab-issues

Financial Stability

As a 28+ strong team, we have independent funding to develop the core platform with a cash runway for at least 1-1.5 years. Cardano Catalyst proposals are used to extend the functionality and add features to the platform in order to enrich the possible use cases.

The financial report is publicly available and can be reviewed here:

https://medium.com/nunet/nunet-financial-report-2022-and-outlook-for-2023-405d38397629

[CAPABILITY/ FEASIBILITY] What are the main goals for the project and how will you validate if your approach is feasible?

Main goals:

1. Enable consumers to deploy Elasticsearch clusters on NuNet

This project enables a consumer to deploy an Elasticsearch cluster with a fair price. On the other hand, a computer device provider can host an Elasticsearch node and can get paid for that.

2. Decentralize Elasticsearch cluster 

The existing approach of deploying databases on cloud providers leads data to be centrally hosted by few big tech companies. By allowing databases to be deployed and monetized on decentralized hardware, this project opens up an unlimited number of use cases and further development of decentralized data storage and granular monetization leveraging mainstream database solutions.

3. An alternative data hosting for NuNet partners including IAMX 

Use case partners or third party developers leveraging the NuNet network (like IAMX) will be able to deploy Elasticsearch services on NuNet. These companies will have a decentralized, more dynamic and potentially cheaper alternative to current solutions. 

4. NuNet will use it for the system observability purposes

Currently NuNet is running its observability infrastructure that involves Elasticsearch cluster deployed on a cloud server. This solution will allow NuNet to run its own infrastructure on its own network and enable this for other software development companies.

[CAPABILITY/ FEASIBILITY] Please provide a detailed breakdown of your project’s milestones and each of the main tasks or activities to reach the milestone plus the expected timeline for the delivery.

Milestone 1: Project Commencement and detailed architecture design

• Main Tasks / Activities: 
• Forming the squad, building communication platforms, conducting weekly brainstorming sessions for comprehensive understanding of the Cardano universe and its latest computational offerings, identifying the prerequisites for a distributed database hosting, drafting a comprehensive project blueprint and architecture design, and conducting a practicability study based on the specified prerequisites and available resources.
• Expected Delivery Time: 5-8 Weeks

Milestone 2: Implementation of detailed architecture

• Main Tasks / Activities: 
• Implement ability for compute providers to broadcast their availability to run Elasticsearch nodes and network topology to the network;
• Implement functionality to create Elasticsearch cluster configuration and propagate it to the network;
• Implement functionality to correctly receive deployment request on a selected compute provider;
• Implement functionality to deploy Elasticsearch node container and connect to the cluster;
• Expected Delivery Time: 4-12 Weeks

Milestone 3: Internal, community testing and resolving additional issues

• Main Tasks / Activities: 
• Internal testing on closed testing network;
• Testing in cooperation with partner IAMX by deploying the service and their specific processes;
• Large scale community testing on testnet/preprod;
• Resolving bug reports and implement identified additional features;
• Expected Delivery Time: 4-12 Weeks

Milestone 4: Production release

• Main Tasks / Activities: 
• Release of the tested solution and features on the Cardano mainnet;
• Expected Delivery Time: 3-8 Weeks

Milestone 5: Technical documentation and result dissemination;

• Main Tasks / Activities: 
• Writing documentation 
• Dissemination of results in the community
• Expected Delivery Time: 2.5-4 Weeks

[CAPABILITY/ FEASIBILITY] Please describe the deliverables, outputs and intended outcomes of each milestone.

Milestone 1: Project Commencement and detailed architecture design

• Deliverables: Project blueprint, general architecture and applied research, practicability analysis, project roadmap, defining risk management strategy and resource allocation plan.
• Outcomes: Detailed documents of all strategies, including a timeline, resource distribution, and dependencies.

Milestone 2: Implementation of detailed architecture

• Deliverables: Updated platform components (DMS) with modified data format, advertising willingness to host Elasticsearch nodes, functionality of required hardware and network capacity properties, matching and searching for required nodes for the specified Elasticsearch cluster functionality, connecting nodes into the cluster inside NuNet’s peer-to-peer network; 
• Outcomes: New device and job metadata format and related functionality, option for data hosting availability advertising on the platform, increased number of applications using NuNet network with requirement of data access. 

Milestone 3: Internal, community testing and resolving additional issues

• Deliverables: Extensive testing of the system under different conditions and gathering feedback to make improvements to the system specified in a documented format; fixes based on the community testing
• Outcomes: A reliable and robust system that meets the defined requirements and delivers the desired functionality.

Milestone 4: Production release

• Deliverables: Deployment of the finalized and tested system on the Cardano mainnet.
• Outcomes: A live, functioning system on the Cardano mainnet enabling customers to request Elasticsearch clusters and compute providers to deploy their nodes and get rewarded for actual compute resources used for that.

Milestone 5: Technical documentation and result dissemination;

• Deliverables: Prepared comprehensive technical and usage documentation for users and community developers;
• Outcomes: Users are trained and informed on how to effectively use the system and deploy Elasticsearch clusters, improved ability to develop full applications involving data access and storage on mainstream databases on decentralized hardware, increased number of NTX settlement transactions on Cardano blockchain and in general.

[RESOURCES & VALUE FOR MONEY] Please provide a detailed budget breakdown of the proposed work and resources.

Each project is examined in great detail which can be seen in the proposed budgeting sheet. This results in pre-feasibility and feasibility studies which minimize the risk of budget overruns.

Project management in NuNet is on a high level with employed techniques such as Agile, Scrum, CCPM and others resulting in a good daily overview of the project progress.

The project is complex and involves research and development uncertainties however, NuNet is a well funded deep tech startup and in case of budget overruns will continue to develop until delivered due to this being a critical part of the overall NuNet development plan. This is evidenced by the funding received in Cardano Catalyst Fund 7 and 8 where NuNet has continued the work despite the substantial unexpected technical roadblocks and time impact.

[RESOURCES & VALUE FOR MONEY] How does the cost of the project represent value for money for the Cardano ecosystem?

The costs of the project are based on the average salary levels of engineers currently employed by NuNet. Since the team is fully distributed and remote, it is challenging to have a suitable median cost that covers the range of countries (India, Pakistan, Ethiopia, Brasil, Egypt, UAE, UK, Italy and others).

We believe that the costs are reasonable and reflect the seniority and knowledge of various positions involved in the delivering of the proposal.

In line of full openness, in the budget table can be seen the very granular distribution of costs, all the way to the hours of each position for each milestone.

In addition, fully remote workers can compete for jobs in Western countries driving the individual compensation levels much higher than in their native countries.

[IMPORTANT NOTE] The Applicant agreed to Fund10 rules and also that data in the Submission Form and other data provided by the project team during the course of the project will be publicly available.

I Accept