101 lines
2.8 KiB
Markdown
101 lines
2.8 KiB
Markdown
# dex-scraper
|
|
|
|
A Kotlin-based web scraping application built with Quarkus for extracting structured data from Yu-Gi-Oh! card databases.
|
|
|
|
## Overview
|
|
|
|
This project is a high-performance web scraper designed to extract structured data about Yu-Gi-Oh! card sets, regional variations, and card prints from various online sources. It leverages Quarkus for fast startup times and low memory usage, with a focus on reliability through CommonCrawl web archives.
|
|
|
|
## Features
|
|
|
|
- **Multi-source scraping**: Configurable scraping targets for different card databases
|
|
- **Historical data retrieval**: Uses CommonCrawl web archives for reliable historical data
|
|
- **Structured extraction**: Extracts card sets, regional sets, and card prints with validation
|
|
- **Flexible configuration**: YAML-based configuration for easy customization
|
|
- **REST API**: Provides HTTP endpoints for data access
|
|
|
|
## Architecture
|
|
|
|
The scraper follows a modular architecture:
|
|
- **Controller Layer**: REST endpoints for API access
|
|
- **Service Layer**:
|
|
- `ScrapeService`: Orchestrates the scraping process
|
|
- `SourceService`: Manages provider configurations
|
|
- `CommonCrawlService`: Handles web archive queries
|
|
- Extraction services for different data types
|
|
- **Configuration**: YAML-based configuration in `application.yml`
|
|
|
|
## API Endpoints
|
|
|
|
### Scrape Card Set
|
|
```
|
|
GET /api/{provider}/set?setName={name}
|
|
```
|
|
Returns structured card set information.
|
|
|
|
### Scrape Regional Set
|
|
```
|
|
GET /api/{provider}/regionalSet?setName={name}
|
|
```
|
|
Returns regional variation information for a card set.
|
|
|
|
### Scrape Card (Not Implemented)
|
|
```
|
|
GET /api/{provider}/card?cardName={name}
|
|
```
|
|
Currently not implemented.
|
|
|
|
## Configuration
|
|
|
|
The scraper is configured through `src/main/resources/application.yml`. Currently supports:
|
|
|
|
- **Yu-Gi-Oh Fandom Wiki**: The primary target for card data extraction
|
|
- Configurable CSS and XPath selectors for element targeting
|
|
- Validation and transformation rules for data cleaning
|
|
|
|
## Getting Started
|
|
|
|
### Prerequisites
|
|
- JDK 21+
|
|
- Gradle
|
|
|
|
### Running in Development Mode
|
|
```bash
|
|
./gradlew quarkusDev
|
|
```
|
|
|
|
### Building the Application
|
|
```bash
|
|
./gradlew build
|
|
```
|
|
|
|
The application can be run with:
|
|
```bash
|
|
java -jar build/quarkus-app/quarkus-run.jar
|
|
```
|
|
|
|
### Creating a Native Executable
|
|
```bash
|
|
./gradlew build -Dquarkus.native.enabled=true
|
|
```
|
|
|
|
## Development
|
|
|
|
This project uses:
|
|
- **Quarkus**: Supersonic Subatomic Java Framework
|
|
- **Kotlin**: Primary programming language
|
|
- **Jsoup**: HTML parsing library
|
|
- **CommonCrawl**: Web archive data retrieval
|
|
|
|
## Contributing
|
|
|
|
1. Fork the repository
|
|
2. Create a feature branch
|
|
3. Commit your changes
|
|
4. Push to the branch
|
|
5. Open a pull request
|
|
|
|
Easily start your REST Web Services
|
|
|
|
[Related guide section...](https://quarkus.io/guides/getting-started-reactive#reactive-jax-rs-resources)
|