CardanoPy: 5 min extensible node

• Github: https://github.com/floydcraft/cardano-py
• Discord: https://discord.gg/FyDz4Xrt4x

The YouTube video walks through the workflow within the first few mins! Check it out!
https://www.youtube.com/watch?v=fBW1_-kHjx0

Technical Walkthrough + Deep dive (Part 1: walkthrough, Part 2: substitutions, Part 3: kubernetes)

https://www.youtube.com/watch?v=8Eumu4j80sk&t=32s
*NOTE: this is the natural evolution of my FUND3 Proposal CardanoETL https://cardano.ideascale.com/a/dtd/Cardano-ETL-Public-BigQuery-Data/334530-48088*

# Problem
Cardano node setup and ops are overtly difficult. Onboarding and development frictions need to be significantly reduced

• IOHK docker image is NOT extensible
• Building and dockerzing cardano nodes is a complex task
• Developers want to focus on their business use cases. not solving infrastructure and development productivity issues.
• Cardano has a high friction to learning atm. This approach will make it much easier and learnable.

# Solution / Quickstart
I already built a Proof of Concept (POC), at the Github link above, that you can install and use today (v0.1.6). Keep in mind the goal of ease of use / learnable approach to Cardano. Please check it out!

Within a few mins you can have a Cardano node running on your local computer using the following four commands from a new developer perspective.

1. pip3 install cardanopy
2. cardanopy create -t basic -n testnet app
3. cardanopy docker run app
4. cardanopy cli query tip

The solution is to extend that workflow to include "templates" for the following use cases.

NOTE: Initial focus will be local and GCP and later expand to AWS.

• basic : run a simple node just because you can
• bp-k8s: blocker producer node for kubernetes
• relay-k8s: relay node for kubernetes
• bp-dc: block producer node for docker compose
• relay-dc: relay node for docker compose
• db-sync: db-sync postgres sidecar
• wallet: wallet sidecar

Using this workflow we can "inject" configuration (substitutions) while retaining the same base images for most of these workflows. Developers can easily extend the base docker image (or even make their own new base image that follows the spec) for their dApps and ETL workflows.

Additional scope: this solution will also provide some additional benefits to developers outside of the simple bootstrap, customization, and extensible features mentioned above.

• Helper CLI for cardano-cli that makes it easier / more useful (e.g., no more --testnet 1097911063, …)
• MVP cardanopy etl command - BigQuery / json exports mentioned in my fund 3 proposal CardanoETL. Python is the go to for data engineering and data science and cardano etl WILL be the first place most of them look to =)
• Build scripts for Google Cloud Build and Github Actions for anyone who wants to setup their own CI/CD using my examples.
• Documentation for python module and all of the source
• Weekly YouTube videos during both development and the initial production release (v1.0.0). Currently v0.1.5

See Roadmap for more details https://github.com/floydcraft/cardano-py#roadmap

# Experience

You should checkout my Linkedin here: http://bit.ly/linkedin-bfloyd
development + devops engineer with 8 years of mobile games engineering experience + 7 years of military simulations and training engineering experience. I've also built ETL pipelines supporting massive real time event pipelines in GCP and AWS in addition to my normal game backend / client and infrastructure work.

I've also setup and operated a jormungandr stakepool (CHBFI) for a few months during the initial testnet.
In the last 3 weeks I've built CardanoPy from nothing to v0.1.5. Excluding merges, I've have pushed 330 commits to master. On master, 93 files have changed and there have been 1,059 additions and 1,112 deletions. It currently capable of deploying and operating single nodes, pools (relays + block producers), and db-sync exporting to postgres).

# Target Impact

• 40+ hours per developer decrease in time spent for developers to bootstrap Cardano and start using it's complex features in a useful and productive developer workflow that uses CardanoPy. Allowing on developers to focus on their business problems instead of making their own development workflows for Cardano.
• 10+ contributions to a first class citizen docker cardano node from the community instead of a custom solution for everyone.
• 3+ contributors to a first class citizen docker cardano node from the community instead of a custom solution for everyone.
• +5 dApps in development using CardanoPy
• +5 stakepools using CardanoPy (even if passive testnet)
• Easy and Free: open source and no paid services required with clear bootstrap quckstarts
• Easy access to Cardano blockchain data through CardanoPy etl command / tools

# Auditability

• Open Source [cardano-py](https://github.com/floydcraft/cardano-py) Apache License Version 2.0
• Metrics for github repos (activity, commits, community, …) and cardanopy anonymous (opt out) usage data (downloads, installs, …)
• Youtube video metrics (views, session time, subscribers) to CardanoPy channel (new)
• Metrics for cardano-py-slim (base) image anonymous (opt out) usage data (if the community is in favor of this)
• Chat / Social metrics for CardanoPy

# Feasibility: Approach
I've already proven the feasibility technically (see YouTube video or github). Additionally, the python community is robust and active in the data engineering and data science disciplines. I expect there will be significant interest in CardanoPy as adoption of Cardano continues to accelerate. I'll focus on finding 1 or 2 primary contributors outside of myself during the project as it becomes more robust and useful to developers. Though for the scope of this project it isn't necessary, the long term health of the project is the end goal.

# Estimates
See Roadmap for more details https://github.com/floydcraft/cardano-py#roadmap

• (DONE) March - MVP key capabilities / developer workflow

Development ($10K)

• March - Kubernetes capabilities
• April - docker capabilities and Cardano CLI capabilities (native assets)
• May - Wallet capabilities and Cardano CLI capabilities (nft)
• June - db-sync capabilities (postgres) and Cardano CLI capabilities (smart contracts)
• July / August - data exports to json delimited, CSV, GCP PubSub, and BigQuery.
• September - Official Release

Operations ($10K)

• ~6 months of server operations (development + etl operations / testing)

Support ($0)

• ~50+ hours of community support (bug fixes, …)

Community / Marketing ($0)

• ~100+ hours of content creation (YouTube, …)

# Future Funding (for both operations and developments)

• Will continue to contribute to the catalyst proposals as needed for supporting the evolution of this project
• Will operate/market a 10% margin stakepool TBD to fund CardanoPy for the long term
• Donation / Treasury wallet (only for CardanoPy development)
• Considering a Native token approach (governance of treasury / github priorities?)
• Other ideas welcome!

# Releases
NOTE: you can use these features already! `pip3 install --upgrade cardanopy`
## 0.1.x
3/14/2021 - MVP key capabilities / developer workflow

• `cardanopy run` - executes cardano-node command using cardanopy.yaml config to generate *.template.json configs.
• `cardanopy create` - creates a cardanopy app from a template ('basic, producer-k8s, relay-k8s'). e.g., cardanopy.yaml, config.template.json, …
• `cardanopy generate` - generates configs from cardanopy app using cardanopy.yaml
• `cardanopy docker run` - runs or attaches to docker container by name using cardanopy.yaml docker.* configs.
• `cardanopy docker stop` - stops docker container by name using cardanopy.yaml docker.name config.
• `cardanopy cli address` - same as cardano-cli except automaticaly provides cardano network param (testnet/mainnet)
• `cardanopy cli stake-address` - same as cardano-cli except automaticaly provides cardano network param (testnet/mainnet)
• `cardanopy cli query` - same as cardano-cli except automaticaly provides cardano network param (testnet/mainnet)
• `-d, --dry-run` - flag to print without mutation what would be executed. useful for most commands like docker and kubernetes.
• substitutions - core capability of cardanopy. cardanopy.yaml can both in memory and write to disk substitutions to configs before running a cardano-node. e.g., _BP_ENABLED=True or _BP_ENABLED=False to enable / disable a blocker producer.
• `-s KEY=VALUE, --sub KEY=VALUE` - optional override to substitutions defaults in cardanpy.yaml config.
• [Cardano Node: Basic Example](https://github.com/floydcraft/cardano-py-examples/tree/master/basic-example)
• YouTube: CardanoPy - Overview + Examples

# Roadmap
Prioritized, but open to feedback!
## 0.2.0
March - Kubernetes capabilities
NOTE: WIP now. Already have testnet working. See [CardanoPy](https://github.com/floydcraft/cardano-py) GitHub

• `cardanopy k8s apply/*` - full node pool capabilities for both local kubernetes and GCP / GKE kubernetes.
• [Cardano Node: Kubernetes Example](https://github.com/floydcraft/cardano-py-examples)
• YouTube Overview + Examples

## 0.3.0
April - Cardano CLI capabilities

• `cardanopy cli *` - more core cli capabilities
• `cardanopy cli native-tokens
• [Cardano Node: Native Token Example](https://github.com/floydcraft/cardano-py-examples)
• YouTube: Docker compose - Overview + Examples

## 0.4.0
April - docker capabilities

• `cardanopy docker run/stop` - add additional support for python docker (instead of requiring external dependency of docker)
• `cardanopy docker-compose up/*`
• [Cardano Node: Docker compose Example](https://github.com/floydcraft/cardano-py-examples)
• YouTube: Docker compose - Overview + Examples

## 0.5.0
May - Wallet capabilities

• `cardanopy wallet *`
• [Cardano Node: Wallet Example](https://github.com/floydcraft/cardano-py-examples)
• YouTube: Wallet - Overview + Examples

## 0.6.0
June - db-sync capabilities (postgres)

• `cardanopy db-sync *`
• [Cardano Node: Postgres Example](https://github.com/floydcraft/cardano-py-examples)
• YouTube: Postgres - Overview + Examples

## 0.7.0
July through August - data exports to json delimited, csv, GCP PubSub, and BigQuery.

• cardanopy data *` - export capabilities like db-sync. except to useful formats for python data engineering, data science, and data exploration/insights. Includes JSON delimited, CSV, PubSub, and BigQuery.
• [Cardano Node: Json / BigQuery Example](https://github.com/floydcraft/cardano-py-examples)
• YouTube: Json / BigQuery - Overview + Examples

## 1.0.0
September - Official Release

• bug fixes and improvements to all features 0.1.x - 0.7.x and documentation
• YouTube Official Release

20000