StreamCardano: serverless infra

[IMPACT]

We build build software-as-a-service that will allow queries of Cardano blockchain status and receive notifications on updates.

Our service will be connected by JavaScript/TypeScript/PureScript library that subscribes to live updates of Cardano blockchain. There will be also user interface for trying the queries during development (IDE), and setup automated runs of Cardano Wallet scripts whenever query triggers.

Any Cardano developer may create a free account to get a developer access token.

This per-project access token will enable developer for custom queries of entire Cardano blockchain database, extracting details of the transactions, and smart contract redeemer data.

Minimal online IDE will be sufficient for test queries, and status page will allow confirmation that service is working flawlessly.

The platform will also allow real-time notification of the change on the blockchain by means of server-sent events, wallet script execution, or webhook calls.

When app will be deployed to production, then we will ask for a subscription fee to cover infrastructure costs. This service will allow building entirely serverless dApps.

Among our main deliverables will also be tutorials and example apps that developers can adapt to use our serverless service.

We will also provide developer support for Catalyst projects that want to use the service. This way we grow ecosystem, and share the cost and burden of infrastructure necessary for Cardano dApp projects.

Using software-as-a-service will allow shared cache and instant-on dApp building, beside using less electricity and money.

Solution will make dApp creation more accessible, support dApp authors and make Cardano much easier platform to develop on.

We want to make the self-sustaining product that fuels ecosystem growth.

To achieve this we focus on developer ergonomics. Developer support is primary driver of the project.

Milestones M2 and M3 will include free beta plans, and milestone M5 will fund the Customer Success Engineering. From M2 forwards, we will help our partners develop dApps, instead of just giving away a free but clunky service. To assure developer accessibility, milestones M1 and M2 fund observability of the platform development and availability.

It would be ideal to sustain free trial dev accounts for indefinite future, and make project self-sustaining using production apps only. At the same time, thinking about the choice of investing in ergonomics versus making the cheapest service, we know that wise investment will fuel the growth of the developer ecosystem.

That is why we want to prioritise support of the projects funded by Catalyst.

Hosting production apps on our serverless platform may be cheaper and greener than self-hosting, but still needs to pay for the infrastructure and engineering costs.

As for the cost of code review, platform audit, and pentest: In order to maximize value to Cardano, we selected with great care those review, audit and pentest providers that promise to deliver this service within our tight budget constraints.

How to make the project more cost-effective and impactful, while keeping it sustainable in the future?

Main challenges are threefold:

• to improve developer ergonomics sufficiently to open a floodgate for dApp development
• to make the service secure, which we plan to address by SQL parsing, and sandboxing, as well as user account separation
• make it easily accessible to developers, which we plan to address by dogfooding the service, and making sure that developer documentation is our first priority when developing the service.

[FEASIBILITY]

Milestones:

M1. Technology teaser: we will deliver technology teaser, where frontend apps are running:

• seeing the latest testnet block status
• website with live updates of ADA transaction volume
• website with live updates of a MILK token price

M2. Closed beta: We will request users to register for closed beta of our serverless query service, and help them to develop their apps using this service.

• benchmarks for number of queries supported by service in parallel
• free user support to help users build apps, with direct connection to developers
• service status page
• functionality test tracking pages
• online IDE
• within 2 month from the project start.

M3. Public beta:

1. We will accept user registrations for the full SaaS.

• at this moment we will allow user registrations
• we will also provide a smart contract calling service that would be automatically triggered by the updates on the blockchain
• customer support site for filing tickets
• webhooks triggered by updates on the Cardano blockchain
• Within 4 months from the project start

M4. Official release:

• official SLA for public users
• optional AWS marketplace image for corporate users
• live monitor of query isolation
• billings
• account quota and billing warnings
• Cardano Summit presentation
• Within 6 months from the project start.

M5. Community Growth stage:

• trainings at blockchain conferences
• marketing within Cardano local communities
• service support and monitoring
• features and bug fixing upon user request
• monitoring service security
• biweekly Q&A, and monthly use case
• Security audit
• Within 12 months from project start.

Costs per milestone

==============

In case of running costs, they are described upon first milestone when they start to be counted.

M1:

• project website, public issue tracker, public status tracker - 2k$ for configuration and necessary updates over the life of the project
• streaming query operation (development includes event streaming, connection pooling, caching, and app token validation) - 12k$
• serverless service demo showing live updates of token price (frontend app using streaming query operation above) - 5k$
• online service status page (publicly available over entire course of the project) - configuration and subscription cost counted together with test page below
• online test page for developed functionality - CI configuration and subscription costs over entire lifecycle 3k$
• Online documentation with CI updates - CI configuration and subscription costs over entire lifecycle 3k$

M2:

• JS, TS binding library with a tutorial - 5k$
• PureScript REST binding library with tutorial - 5k$
• online IDE allowing logged users to explore Cardano with arbitrary queries - 10k$
• Haskell backend API with CI deployment tutorial - 10k$
• closed beta test for developers of read-only query apps supported!

M3:

• WebUI with user subscription and per-project access tokens (this is main user interface for the app developer and billing subscriber, it will provide all functionality essential for managing different apps, and monitoring them) - 26k$
• Running wallet script and submitting transaction upon change on the Cardano blockchain (for example, selling token when it reaches the price). - 10k$
• security compartment for scripted wallets (separate REST process that is responsible for separation of actions on wallets from all other components of the application, and preventing wallets from either being lost, or stolen) - 10k$
• public beta-tests for developers!

M4:

• demo trading on threshold using transaction submission upon change of blockchain state - 10k$
• professional services (ToS, SLA, privacy policy etc.- the necessary legal documents need to be available before commercial subscriptions are possible, this is costly because it needs precise coordination between legal counsel, application architect, and cloud providers) - 15k$
• webhook calling upon change on Cardano blockchain (for example the price of the token)
• first commercial subscriptions possible!

M5:

• Developer success engineering for 6 months after public release:
• weekly Q&A with "answers in code" and "user debugging" sessions - 11k$
• monthly demo use case - 24k$
• bugfixing outside Q&A and use cases - 11k$
• service security monitoring - 11k$ (essential for multitenant system)
• virtual machine for multicloud marketplace for customers needing extra security: 5k$
• first single-tenant top security enterprise subscriptions possible!

Running costs over all milestones:

• Cloud hosting service costs for a full year - 18k$
• biweekly update posts - 6k$
• management support: 20k$
• code reviews
• cloud architecture review
• pentest
• security audit

Detailed cloud hosting service costs for a full year:

====================================

Most of the AWS costs are for the Kubernetes (K8S) clusters that would host test, staging, and production environments.

I assume we need to host three environment:

• test for running CI jobs
• staging for beta-testing
• production

We have tested an early alpha on configuration recommended by CardanoDBSync developer (32GB RAM, 4 cores, 80GB disk space), so we use this as a minimal node configuration.

Going below recommended configuration is not recommended on K8S, since they will split each node into different containers anyway.

Both staging and production environments should have at least one spare node to allow migration in case of hardware problem. Production environment should additionally have a replication of each service, so that failover is quick. That means that staging should have at least 2 nodes, and production should have a minimum of 3 nodes. We hope that the 4th additional production node and extra memory (64GB instead of 32GB) will be sufficient to accomodate user demand and production apps within the first year due to efficient multitenancy.

Here is the AWS cost breakdown:

A. Dev Kubernetes test cluster (single node) at ~200$/mo for 12 months: 2.4k$

B. Beta-test (staging) Kubernetes cluster with minimal node redundancy (2 nodes) for hosting beta-testers and trial users at 320$/mo for 12 mo: 3.84k$

C. Kubernetes cluster allowing service scaling (4 nodes, at least 64GB memory each) at 0.9k/mo for 8 months: 10.8k$

D. Hosting container images at ~100$/mo in ECR: 1.2k$

E. Storage buckets for website, documentation for 12 months (assuming 100GB/mo outgoing traffic to CDN) at 10$/mo: 120$

Overall: 18.36k$

Notes:

1. We do not plan for geo-redundancy or multi-region failover in the first year.
2. Test and staging nodes are to have recommended CardanoDBSync configuration from https://github.com/input-output-hk/cardano-db-sync are: 32GB RAM, 4 CPU, 60k+ IOPS, 80GB+. We have successfully tested such a single-node configuration for development purposes.
3. May be verified using AWS cost estimation service: https://calculator.aws
4. We reserve right to use competing cloud services where reliability and price permit.
5. I am looking for any suggestions on how to improve this estimate.

Notes on methodology of software development estimation

===========================================

COSMIC function points methodology was used for estimation of development effort.

This methodology assigns one point of effort for each data piece entering, leaving,

or getting updated within a user story describing the application.

More details and software architecture diagram are provided in the PDF attachment.

• Michał (https://gitlab.com/mjgajda, https://www.linkedin.com/in/mjgajda/) is a professional software developer with a background in functional programming, logic, data analytics, and agile software development. Within the past five years, he presented at events associated with ICFP, SkillsExchange, LambdaDays, Haskell.Love, XP Agile Programming Conference, Data Science Europe, EACSL, CCC, LAP. He is excited by software technology, constructive logic, data analytics, and epistemology.

• Julian Ospald(https://migamake.com/team/julian.html, https://www.linkedin.com/in/julian-ospald-2ab78ba6/) is an experienced Cardano developer with many contributions to open source in Haskell ecosystem.

• Gabriel Araujo(https://migamake.com/team/gabriel.html,https://github.com/Gab0) is experienced Haskeller, MSc in bioinformatics, specialized in parsing and data analytics

• Migamake.com (https://www.migamake.com) will provide resources for the work, including high quality code reviews.

Our advisors:

• James Haver(https://www.codementor.io/@mchaver, https://youtu.be/1MWn9v7VrUU) is the original and primary developer of the Dexter decentralized exchange.

• Szymon Zioło(https://migamake.com/team/szymon.html, https://www.linkedin.com/in/szymon-ziolo-a5b591) is experienced business analysts, who worked for numerous public contracts on software design, specification, and planning.

[AUDITABILITY]

We will publish the progress of the project by the following means:

• Online documentation of the REST interface on the web page:

• number of endpoints

• textual descriptions and explanations

• working examples in a public repository

• Online functionality tests:

• public status page of our service

• live test status page enumerating app functionality working in the latest released version

• release milestones that indicate completion of the functionality listed on the status pages

• Blogging updates on statistics on user uptake on the project web page:
• project milestones reached

• number of registered users

• number of registered app projects

• number of active project tokens per day

• number of commercial subscriptions

Many Cardano dApp&Integration developers use our service to build their next Cardano dApp or Integration. They may deploy dApps using our serverless platform without the cost and maintenance of additional servers. Cardano developers would consider our platform as a first stop to develop and prototype Cardano applications.

This is a new project.

Our past proposal is here: https://cardano.ideascale.com/c/idea/383011

SDG Rating

SaaS will:

• decrease resource consumption per Cardano dApp
• decrease inequality between developers by decreasing threshold on entry
• promote innovation and growth in the number and quality of dApps within Cardano community

SDG goals:

Goal 8. Promote sustained, inclusive and sustainable economic growth, full and productive employment and decent work for all (-> by making dApp deployment cheaper and more accessible and thus more sustainable)

Goal 9. Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation

SDG subgoals:

10.3 Ensure equal opportunity and reduce inequalities of outcome, including by eliminating discriminatory laws, policies and practices and promoting appropriate legislation, policies and action in this regard (-> by making Cardano development equally accessible to all frontend developers)

7.3 By 2030, double the global rate of improvement in energy efficiency (-> by sharing multitenant SaaS infrastructure)

8.2 Achieve higher levels of economic productivity through diversification, technological upgrading and innovation, including through a focus on high-value added and labour-intensive sectors (-> increase value added by high level development platform)

8.4 Improve progressively, through 2030, global resource efficiency in consumption and production and endeavour to decouple economic growth from environmental degradation, in accordance with the 10‑Year Framework of Programmes on Sustainable Consumption and Production, with developed countries taking the lead

(-> Increase efficiency by using multitenant platforms for apps)

Key Performance Indicator (KPI):

7.3.1 Energy intensity measured in terms of primary energy used by hosted dApps as compared with independent dApps

8.2.1 Annual growth rate of real GDP per employed person by estimating increase in efficiency of dApp developers using SaaS

8.4.1 Decreased footprint, by estimating footprint used by SaaS-hosted dApps as compared with independent dApps