Oura v2

[GENERAL] Summarize your solution to the problem.

We'll assign a fixed development bandwidth to develop a new version of Oura that incorporates new features, improves performance and evolves the event model.

[GENERAL] Summarize your relevant experience.

We're TxPipe, a small team of seasoned developers passionate about Open Source and Cardano. It is the team behind existing projects such as Oura & Pallas. We specialize in backend development, data pipelines and cloud architecture. Rust is our language of choice.

[IMPACT] Please describe your proposed solution.

Oura v1 has received good level of adoption judging by the activity on the Github repository. High-profile projects are using Oura either as a base library or to fulfill specific data integration roles within their infrastructure.

Project maintainers have managed to complete most of the development roadmap initially defined and have kept the codebase updated with required bug-fixes and upstream changes.

Regardless of this effort, we have identified, with the help of the community, several improvements that would require a high-level of dedication:

• Improve chain-sync performance by incorporating a new concurrency model at the multiplexer level. We expect at least 5x speed improvement in node-to-client connections.
• An overhaul of the event data model that focuses on less type of events and more nested structures, allowing consumer to process the full Tx payloads.
• A new type of filter / sink that allows user to run custom code by providing WebAssembly file. This will allow integrators to integrate their downstream services without the need of a broker.
• New sinks that integrate directly with stream-processing services from Azure cloud provider.
• Finalize the ongoing effort to include new sinks that integrate directly with stream-processing services from Google Cloud Platform, such as PubSub and Cloud Functions.
• A new Kafka data source that would allow integrators to consume event from topics already populated with chain data. This opens the possibility to execute several Oura instances consuming from a single Kafka topic, without the need for a direct connection to the Cardano node.
• A new AWS S3 data source that would allow Oura to read from a pre-populated bucket of raw block data. This is useful for workloads that require to traverse the whole history without impacting on the performance of their Cardano node.

[IMPACT] Please describe how your proposed solution will address the Challenge that you have submitted it in.

Adding the described features to Oura will provide new data integration options. Having new tech-stacks and services integrated with Cardano data will widen the spectrum of tools available, improving the chances for developer adoption.

Improved performance and data model will open up the possibilities for new use-cases.

[IMPACT] What are the main risks that could prevent you from delivering the project successfully and please explain how you will mitigate each risk?

Our experience developing Oura v1 allows us to estimate the LoE for this proposal with a high level of accuracy. Regardless of this advantage, there are some known-unknowns that could present a risk:

• Our development methodology is driven by community feedback. Committing to a strict and specific set of features in advance might limit the capacity to adapt to feedback during the development phase.
• Proposed changes require breaking compatibility at the config, data and library levels. Communicating and supporting the existing Oura users during the migration to the new version will require effort. Estimating this effort is very difficult on an open-source project without strict knowledge of the existing implementations.

[FEASIBILITY] Please provide a detailed plan, including timeline and key milestones for delivering your proposal.

Milestone #1: Data Model / Performance

• 1 month development
• 1 full-time Rust developer
• 1 part-time technical writer
• Deliverables
• Oura v2.0
• Performance improvements
• Data-model overhaul
• Updated docs

Milestone #2: New Sinks

• 1 month development
• 1 full-time Rust developer
• 1 part-time technical writer
• Deliverables
• Oura v2.1
• WebAssembly Sink
• Azure Sink
• GCP Sink
• Updated docs

Milestone #3: New sources

• 1 month development
• 1 full-time Rust developer
• 1 part-time technical writer
• Deliverables
• Oura v2.2
• S3 Source
• Kafka Source
• Updated docs

[FEASIBILITY] Please provide a detailed budget breakdown.

Hourly rates:

• Rust Developer: 75 usd / hs
• Technical Writer: 36 usd / hs

Required Hours:

• Rust Developer: 480 hs
• Technical Writer: 240 hs

Total Budget:

• Rust Developer: 36,000 USD
• Technical Writer: 8,640 USD

[FEASIBILITY] Please provide details of the people who will work on the project.

Santiago Carmuega will lead the software development effort. He is a senior developer with over 20 years of experience in software development and very active in Cardano Open Source ecosystem leading TxPipe. 

Github: https://github.com/scarmuega

Twitter: https://twitter.com/santicarmuega

Lucas Rosa will contribute with the software development. He is a senior engineer with a diverse technical background and a passion for compilers. In addition to building toy languages he spent over a year helping build Roc a new functional programming language targeting backend development inspired by Elm.

https://github.com/rvcas

https://twitter.com/rvcas

Florencia Luna will be in charge of the technical writing. She is a junior developer with experience in technical writing.

[FEASIBILITY] If you are funded, will you return to Catalyst in a later round for further funding? Please explain why / why not.

If the project continues to gain adoption we plan on returning to get funding to implement a 3rd iteration of new features and improvements in response to community requests.

[FEASIBILITY] Are you or any member of your team working on any other proposals in this Fund9?

[FEASIBILITY] Are you or your team working on any other proposals from previous Funds?

[AUDITABILITY] Please describe what you will measure to track your project's progress, and how will you measure these?

Progress of the project will be measured by released versions matching the scope of the predetermined milestones.

• A detailed roadmap will be presented as part of the open source repository.
• Each milestone will be presented as a partial but working "release" within the repository
• Milestone and tasks will be tracked via Github using their project management tool.
• Direct communication with the development team will be able through TxPipe's Discord server.
• Weekly update summaries will be posted via TxPipe's Twitter account.

[AUDITABILITY] What does success for this project look like?

Projects within the Cardano ecosystem continue to use Oura to integrate their data pipelines, community around the tools grows and the new features receive good feedback.

[AUDITABILITY] Please provide information on whether this proposal is a continuation of a previously funded project in Catalyst or an entirely new one.

This is an entirely new proposal.