A tool used to parse hundreds of thousands of pages per minute to build a database of backlinks. The system efficiently collects information about valuable links on the internet by scanning 2 billion pages each month.

This process works in multiple stages:

  • Stage 1: Scanning and parsing web pages.
  • Stage 2: Analyzing link quality and relevancy.
  • Stage 3: Compiling data into a structured format.

Outcome: Ultimately, a database is created which enables:

  • Extraction of data from individual pages.
  • Compilation of comprehensive statistics.
  • Evaluation of links in terms of their value and content.

The system leverages a MongoDB database to generate insightful reports. These reports are crucial for:

  • SEO optimization strategies.
  • Analysis of competitive landscapes.
  • In-depth understanding of specific internet niches.

Tech stack

  • Front-End: Vue.js, HTML5, CSS3
  • Back-End: Go
  • Databases: MongoDB
  • DevOps: Docker, Linux

Project Links

Check out the project on GitHub: