# dex-scraper

A Kotlin-based web scraping application built with Quarkus for extracting structured data from Yu-Gi-Oh! card databases.

## Overview

This project is a high-performance web scraper designed to extract structured data about Yu-Gi-Oh! card sets, regional variations, and card prints from various online sources. It leverages Quarkus for fast startup times and low memory usage, with a focus on reliability through CommonCrawl web archives.

## Features

- **Multi-source scraping**: Configurable scraping targets for different card databases
- **Historical data retrieval**: Uses CommonCrawl web archives for reliable historical data
- **Structured extraction**: Extracts card sets, regional sets, and card prints with validation
- **Flexible configuration**: YAML-based configuration for easy customization
- **REST API**: Provides HTTP endpoints for data access

## Architecture

The scraper follows a modular architecture:
- **Controller Layer**: REST endpoints for API access
- **Service Layer**: 
  - `ScrapeService`: Orchestrates the scraping process
  - `SourceService`: Manages provider configurations
  - `CommonCrawlService`: Handles web archive queries
  - Extraction services for different data types
- **Configuration**: YAML-based configuration in `application.yml`

## API Endpoints

### Scrape Card Set
```
GET /api/{provider}/set?setName={name}
```
Returns structured card set information.

### Scrape Regional Set
```
GET /api/{provider}/regionalSet?setName={name}
```
Returns regional variation information for a card set.

### Scrape Card (Not Implemented)
```
GET /api/{provider}/card?cardName={name}
```
Currently not implemented.

## Configuration

The scraper is configured through `src/main/resources/application.yml`. Currently supports:

- **Yu-Gi-Oh Fandom Wiki**: The primary target for card data extraction
- Configurable CSS and XPath selectors for element targeting
- Validation and transformation rules for data cleaning

## Getting Started

### Prerequisites
- JDK 21+
- Gradle

### Running in Development Mode
```bash
./gradlew quarkusDev
```

### Building the Application
```bash
./gradlew build
```

The application can be run with:
```bash
java -jar build/quarkus-app/quarkus-run.jar
```

### Creating a Native Executable
```bash
./gradlew build -Dquarkus.native.enabled=true
```

## Development

This project uses:
- **Quarkus**: Supersonic Subatomic Java Framework
- **Kotlin**: Primary programming language
- **Jsoup**: HTML parsing library
- **CommonCrawl**: Web archive data retrieval

## Contributing

1. Fork the repository
2. Create a feature branch
3. Commit your changes
4. Push to the branch
5. Open a pull request

Easily start your REST Web Services

[Related guide section...](https://quarkus.io/guides/getting-started-reactive#reactive-jax-rs-resources)