Update README

2026-09-25 02:23:43 +02:00
parent 406c104158
commit 9be35c1523
1 changed files with 77 additions and 55 deletions
@@ -1,77 +1,99 @@
-# jsoup-scraper
+# dex-scraper

-This project uses Quarkus, the Supersonic Subatomic Java Framework.
+A Kotlin-based web scraping application built with Quarkus for extracting structured data from Yu-Gi-Oh! card databases.

-If you want to learn more about Quarkus, please visit its website: <https://quarkus.io/>.
+## Overview

-## Running the application in dev mode
+This project is a high-performance web scraper designed to extract structured data about Yu-Gi-Oh! card sets, regional variations, and card prints from various online sources. It leverages Quarkus for fast startup times and low memory usage, with a focus on reliability through CommonCrawl web archives.

-You can run your application in dev mode that enables live coding using:
+## Features

-```shell script
+- **Multi-source scraping**: Configurable scraping targets for different card databases
+- **Historical data retrieval**: Uses CommonCrawl web archives for reliable historical data
+- **Structured extraction**: Extracts card sets, regional sets, and card prints with validation
+- **Flexible configuration**: YAML-based configuration for easy customization
+- **REST API**: Provides HTTP endpoints for data access
+
+## Architecture
+
+The scraper follows a modular architecture:
+- **Controller Layer**: REST endpoints for API access
+- **Service Layer**: 
+  - `ScrapeService`: Orchestrates the scraping process
+  - `SourceService`: Manages provider configurations
+  - `CommonCrawlService`: Handles web archive queries
+  - Extraction services for different data types
+- **Configuration**: YAML-based configuration in `application.yml`
+
+## API Endpoints
+
+### Scrape Card Set
+```
+GET /api/{provider}/set?setName={name}
+```
+Returns structured card set information.
+
+### Scrape Regional Set
+```
+GET /api/{provider}/regionalSet?setName={name}
+```
+Returns regional variation information for a card set.
+
+### Scrape Card (Not Implemented)
+```
+GET /api/{provider}/card?cardName={name}
+```
+Currently not implemented.
+
+## Configuration
+
+The scraper is configured through `src/main/resources/application.yml`. Currently supports:
+
+- **Yu-Gi-Oh Fandom Wiki**: The primary target for card data extraction
+- Configurable CSS and XPath selectors for element targeting
+- Validation and transformation rules for data cleaning
+
+## Getting Started
+
+### Prerequisites
+- JDK 21+
+- Gradle
+
+### Running in Development Mode
+```bash
 ./gradlew quarkusDev
 ```

-> **_NOTE:_**  Quarkus now ships with a Dev UI, which is available in dev mode only at <http://localhost:8080/q/dev/>.
-
-## Packaging and running the application
-
-The application can be packaged using:
-
-```shell script
+### Building the Application
+```bash
 ./gradlew build
 ```

-It produces the `quarkus-run.jar` file in the `build/quarkus-app/` directory.
-Be aware that it’s not an _über-jar_ as the dependencies are copied into the `build/quarkus-app/lib/` directory.
-
-The application is now runnable using `java -jar build/quarkus-app/quarkus-run.jar`.
-
-If you want to build an _über-jar_, execute the following command:
-
-```shell script
-./gradlew build -Dquarkus.package.jar.type=uber-jar
+The application can be run with:
+```bash
+java -jar build/quarkus-app/quarkus-run.jar
 ```

-The application, packaged as an _über-jar_, is now runnable using `java -jar build/*-runner.jar`.
-
-## Creating a native executable
-
-You can create a native executable using:
-
-```shell script
+### Creating a Native Executable
+```bash
 ./gradlew build -Dquarkus.native.enabled=true
 ```

-Or, if you don't have GraalVM installed, you can run the native executable build in a container using:
+## Development

-```shell script
-./gradlew build -Dquarkus.native.enabled=true -Dquarkus.native.container-build=true
-```
+This project uses:
+- **Quarkus**: Supersonic Subatomic Java Framework
+- **Kotlin**: Primary programming language
+- **Jsoup**: HTML parsing library
+- **CommonCrawl**: Web archive data retrieval

-You can then execute your native executable with: `./build/jsoup-scraper-1.0-SNAPSHOT-runner`
+## Contributing

-If you want to learn more about building native executables, please consult <https://quarkus.io/guides/gradle-tooling>.
-
-## Related Guides
-
- REST ([guide](https://quarkus.io/guides/rest)): A Jakarta REST implementation utilizing build time processing and
-  Vert.x. This extension is not compatible with the quarkus-resteasy extension, or any of the extensions that depend on
-  it.
- REST Client ([guide](https://quarkus.io/guides/rest-client)): Call REST services
- REST Jackson ([guide](https://quarkus.io/guides/rest#json-serialisation)): Jackson serialization support for Quarkus
-  REST. This extension is not compatible with the quarkus-resteasy extension, or any of the extensions that depend on it
- Kotlin ([guide](https://quarkus.io/guides/kotlin)): Write your services in Kotlin
-
-## Provided Code
-
-### REST Client
-
-Invoke different services through REST with JSON
-
-[Related guide section...](https://quarkus.io/guides/rest-client)
-
-### REST
+1. Fork the repository
+2. Create a feature branch
+3. Commit your changes
+4. Push to the branch
+5. Open a pull request

 Easily start your REST Web Services