dex-scraper

A Kotlin-based web scraping application built with Quarkus for extracting structured data from Yu-Gi-Oh! card databases.

Overview

This project is a high-performance web scraper designed to extract structured data about Yu-Gi-Oh! card sets, regional variations, and card prints from various online sources. It leverages Quarkus for fast startup times and low memory usage, with a focus on reliability through CommonCrawl web archives.

Features

• Multi-source scraping: Configurable scraping targets for different card databases
• Historical data retrieval: Uses CommonCrawl web archives for reliable historical data
• Structured extraction: Extracts card sets, regional sets, and card prints with validation
• Flexible configuration: YAML-based configuration for easy customization
• REST API: Provides HTTP endpoints for data access

Architecture

The scraper follows a modular architecture:

• Controller Layer: REST endpoints for API access
• Service Layer:
  • ScrapeService: Orchestrates the scraping process
  • SourceService: Manages provider configurations
  • CommonCrawlService: Handles web archive queries
  • Extraction services for different data types
• Configuration: YAML-based configuration in application.yml

API Endpoints

Scrape Card Set

GET /api/{provider}/set?setName={name}

Returns structured card set information.

Scrape Regional Set

GET /api/{provider}/regionalSet?setName={name}

Returns regional variation information for a card set.

Scrape Card (Not Implemented)

GET /api/{provider}/card?cardName={name}

Currently not implemented.

Configuration

The scraper is configured through src/main/resources/application.yml. Currently supports:

• Yu-Gi-Oh Fandom Wiki: The primary target for card data extraction
• Configurable CSS and XPath selectors for element targeting
• Validation and transformation rules for data cleaning

Getting Started

Prerequisites

• JDK 21+
• Gradle

Running in Development Mode

./gradlew quarkusDev

Building the Application

./gradlew build

The application can be run with:

java -jar build/quarkus-app/quarkus-run.jar

Creating a Native Executable

./gradlew build -Dquarkus.native.enabled=true

Development

This project uses:

• Quarkus: Supersonic Subatomic Java Framework
• Kotlin: Primary programming language
• Jsoup: HTML parsing library
• CommonCrawl: Web archive data retrieval

Contributing

1. Fork the repository
2. Create a feature branch
3. Commit your changes
4. Push to the branch
5. Open a pull request

Easily start your REST Web Services

Related guide section...