Automated Website Scraper for Dead Link Detection in Cardano official Documentation. Open Source - Github client

[GENERAL] Name and surname of main applicant

Aditya Singh

[GENERAL] Are you delivering this project as an individual or as an entity (whether formally incorporated or not)

Individual

[GENERAL] Co-proposers and additional applicants

Nil

[GENERAL] Please specify how many months you expect your project to last (from 2-12 months)

6

[GENERAL] Please indicate if your proposal has been auto-translated into English from another language

No

[GENERAL] Summarize your solution to the problem (200-character limit including spaces)

Develop an automated website scraper that periodically checks all links within Cardano documentation for functionality. The scraper will identify dead or non-functioning links and generate alerts.

[GENERAL] Does your project have any dependencies on other organizations, technical or otherwise?

No

[GENERAL] If YES, please describe what the dependency is and why you believe it is essential for your project’s delivery. If NO, please write “No dependencies.”

No dependencies.

[GENERAL] Will your project’s output/s be fully open source?

Yes

[GENERAL] Please provide here more information on the open source status of your project outputs

We intend to use the MIT License for our open-source project. This license allows for broad use, modification, and distribution of our code with minimal restrictions. The core library functions, utility scripts, and documentation will be open source.

[METADATA] Horizons

Governance

[SOLUTION] Please describe your proposed solution

Objective: Enhance the reliability and user experience of Cardano’s documentation by ensuring all links are functional and up-to-date.

Solution Overview: Develop an automated website scraper that periodically checks all links within Cardano documentation for functionality. The scraper will identify dead or non-functioning links and generate reports for the documentation team to update or fix these links promptly.

Key Features:

• Automated Link Checking: The scraper will run at scheduled intervals (e.g., daily, weekly) to check all links within the documentation. It will verify the status of each link, ensuring they lead to valid and accessible pages.
• Error Reporting: The scraper will generate detailed reports highlighting dead or non-functioning links. Reports will include the URL of the broken link, the page where it was found, and the type of error encountered (e.g., 404 Not Found, 500 Internal Server Error).
• Notification System: Automated notifications will be sent to the documentation team via email or a project management tool (e.g., Jira, Trello) whenever broken links are detected. This ensures prompt attention and resolution of issues.
• Dashboard for Monitoring: A web-based dashboard will provide an overview of the link-checking status, historical data, and trends. The dashboard will allow the documentation team to track the health of the documentation links over time.
• Integration with Version Control: The scraper can be integrated with the version control system (e.g., GitHub) to automatically check links in new documentation updates before they are published. This helps prevent broken links from being introduced in the first place.

Benefits:

• Improved User Experience: Ensures that users accessing Cardano documentation have a seamless experience without encountering broken links.
• Increased Efficiency: Automates the tedious task of manually checking links, allowing the documentation team to focus on content quality.
• Proactive Maintenance: Identifies and addresses issues before they become widespread, maintaining the integrity of the documentation.
• Data-Driven Insights: Provides valuable insights into the frequency and types of link issues, helping to improve the overall documentation process.

Implementation Plan:

• Requirement Analysis: Define the scope and requirements of the scraper, including the frequency of checks and the reporting format.
• Development: Develop the scraper using a suitable programming language (e.g., Python) and libraries (e.g., BeautifulSoup, Requests). Implement the reporting and notification system.
• Testing: Test the scraper on a subset of the documentation to ensure accuracy and reliability. Conduct performance testing to handle large volumes of links.
• Deployment: Deploy the scraper to run at scheduled intervals. Set up the dashboard and integrate with the version control system.
• Monitoring: Continuously monitor the scraper’s performance and update the scraper to handle changes in the documentation structure or new types of links.

[IMPACT] Please define the positive impact your project will have on the wider Cardano community

Improved User Experience: Ensures that users and developers have access to up-to-date and functional documentation.

Efficiency: Automates the tedious task of link checking, saving time and resources.

Reliability: Enhances the credibility and reliability of Cardano’s documentation.

Security: Makes sure no false links are being displayed in your docs. Stops stealing of domains and Phishing Attacks.

[CAPABILITY & FEASIBILITY] What is your capability to deliver your project with high levels of trust and accountability? How do you intend to validate if your approach is feasible?

Technical Expertise:

• Deep knowledge of web technologies, including HTML, CSS, JavaScript, and Python, which are essential for developing the scraper. We have experience with web scraping libraries such as BeautifulSoup and Scrapy, and automation tools.

Budget Planning:

• Detailed budget plans are created, outlining all expected expenses, including development costs, infrastructure, and contingency funds.
• Regular reviews and adjustments will be made to stay within budget.

[PROJECT MILESTONES] What are the key milestones you need to achieve in order to complete your project successfully?

Research and Planning:

Identify the scope of documentation to be covered.

Define the technical requirements and tools needed for the scraper.

Development:

• Build the website scraper using a suitable programming language (e.g., Python).
• Implement functionality to check links and generate reports.
• Test the scraper on a subset of documentation.

Deployment and Monitoring:

Deploy the scraper to run periodically (e.g., weekly).

Set up monitoring and alert systems for dead links.

Maintenance and Updates:

Regularly update the scraper to handle new types of documentation and links.

Address any issues or bugs that arise.

Ensure all the functionalities are met and provide a demonstration and proof to ensure the problem statement has been solved. All the requirements are fulfilled and the project is working as proposed.

[RESOURCES] Who is in the project team and what are their roles?

• Aditya Singh (Aditya Singh Solanki | LinkedIn) Project Lead.
• Devlopers x2 TBA.

[BUDGET & COSTS] Please provide a cost breakdown of the proposed work and resources

Development Costs:

• $33.04 per hour (Developer + documentation), that is about 100 ADA per Hour
• 2 developers x 400 hours x 100 ADA/hour = 80,000
• 7000 ADA project management

Total Budget 87,000 ADA

[VALUE FOR MONEY] How does the cost of the project represent value for money for the Cardano ecosystem?

Development Costs: 87000 ADA

• Justification: This includes the salaries for developers who will design, build, and test the scraper. The cost is based on average freelance rates for software developers, which typically range from $30 to $100 per hour. Assuming a mid-range rate of $33.04 per hour, the budget allows for approximately 800 hours of development work. This is a reasonable estimate for building a robust and reliable scraper with a useful notification system to it. It also needs to be robust enough to handle updates.
• Value: Investing in skilled developers ensures the scraper is built efficiently and effectively, reducing the likelihood of bugs and ensuring long-term reliability.