Helios unit testing framework with code-coverage statistics

[GENERAL] Name and surname of main applicant

Christian Schmitz

[GENERAL] Are you delivering this project as an individual or as an entity (whether formally incorporated or not)

Individual

[GENERAL] Requested funds in ada

32000

[GENERAL] Please specify how many months you expect your project to last (from 2-12 months)

5

[GENERAL] Please indicate if your proposal has been auto-translated into English from another language.

No

[GENERAL] Summarize your solution to the problem (200-character limit including spaces)

I propose an OOP-like approach, by creating a new built-in called TestingContext`, which has conventional test-runner methods. The UPLC CEK machine will be extended to calculate code-coverage stats.

[GENERAL] Does your project have any dependencies on other organizations, technical or otherwise?

No

[GENERAL] If YES, please describe what the dependency is and why you believe it is essential for your project’s delivery. If NO, please write “No dependencies.”

No dependencies

[GENERAL] Will your project’s output/s be fully open source?

Yes

[GENERAL] Please provide here more information on the open source status of your project outputs

These features will be part of the following repository: https://github.com/HyperionBT/helios

This repository is already open-source. The license is BSD-3-Clause

[SOLUTION] Please describe your proposed solution.

Implement the ability to test Helios functions from within Helios scripts themselves. The CLI can then automatically detect any testing scripts in the workspace/repository, run them, and generate a test report.

An OOP unit-testing framework, which fits nicely into the current Helios language paradigm, will allow us to avoid introducing new syntax that is only intended for unit testing. The test-runner methods will support automatic fuzzy arguments of the functions being tested.

Finally, code-coverage stats will be collected from test runs by using the code mapping that is included in UPLC scripts generated by the Helios compiler.

[IMPACT] Please define the positive impact your project will have on the wider Cardano community.

The lack of in-language unit testing is deterring some teams from using Helios. The current approach of wrapping Helios test scripts in JS test-runners is too cumbersome.

This proposal will alleviate these issues and allow developers to write Helios unit tests in a way that is hopefully on par with mainstream languages.

[CAPABILITY & FEASIBILITY] What is your capability to deliver your project with high levels of trust and accountability? How do you intend to validate if your approach is feasible?

The Helios language already has many built-in types and macros. A TestingContext object would introduce a few additional macros, for which the core architecture is already there.

The current Helios JS fuzzy tester can be used in the background to generate fuzzy arguments for test runs.

Collecting code-coverage statistics requires collecting code-map sites from UPLC run. This code mapping has already been implemented.

[Project Milestones] What are the key milestones you need to achieve in order to complete your project successfully?

Investigate how unit-testing frameworks work in a variety of mainstream languages. Make a synthesis of this investigation in order to determine the best possible design of TestingContext . Consult members of the Helios discord for their opinions/recommendations/experiences.

Acceptance criteria:

Share a report containing the details and conclusion of this investigation

Collect branch/function scope info upon compiling a Helios program. Collect evaluated code-map sites from a UPLC evaluation. Combines these two pieces of information to generate code-coverage reports.

Acceptance criteria:

Add a tutorial to the Helios book demonstrating how to generate code-coverage reports during a test run or emulator run.

Implement the TestingContext type along with fuzzy testing and other macros. (Macros are Helios-specific UPLC functions that connect to a service/library outside of the UPLC environment).

Make sure the CLI can autodetect all testing scripts in a repository/workspace, run them, and then generate a report.

Acceptance criteria:

Add a tutorial to the Helios book on how to write unit tests and run them.

[RESOURCES] Who is in the project team and what are their roles?

Christian Schmitz: creator of Helios

https://www.linkedin.com/in/christian-schmitz-ba0a23b9/

I will try to dedicate 0.2 FTE to this. If I can't find the time I will reach out to other Cardano developers via the Helios discord.

Required skills for code-coverage stats implementation and CLI changes: significant experience with JS/TS, experience with the Helios codebase (bonus).

Required skills for the investigative phase: significant experience with unit-testing frameworks.

[BUDGET & COSTS] Please provide a cost breakdown of the proposed work and resources.

Milestone 1: 5 FTE days

Milestone 2: 5 FTE days

Milestone 3: 10 FTE days

Total: 20 FTE days => 160 hours @ 8hours/day => 12000 USD @ 75 USD/hour => 32000 ADA @ 0.38USD/ADA

Timeline: though milestones 1 and 2 can run in parallel, there aren't enough FTEs available to do that. 20 FTE days => 100 days @ 0.2 FTE => 5 months

[VALUE FOR MONEY] How does the cost of the project represent value for money for the Cardano ecosystem?

An easy-to-use unit-testing framework and code-coverage stats could bring more teams to use Helios.

Let's assume that Helios will be able to attract 10% more teams annually coming from outside the Cardano ecosystem. At the current rate that is about 0.5 teams per year (not including very small teams and hobbyists).

Assuming that only 1 in 10 teams can deliver a successful dApp on Cardano, and that single successful dApp brings a TVL of 1M USD.

According to this estimate, this feature will roughly be able to add 50k USD TVL to Cardano per year.

[IMPORTANT NOTE] The Applicant agrees to Fund Rules and also that data in the submission form and other data provided by the project team during the course of the project will be publicly available.

I Accept