From 9be35c15239a7935907060a7293db68a270a8641 Mon Sep 17 00:00:00 2001 From: rakyah Date: Fri, 25 Sep 2026 02:23:43 +0200 Subject: [PATCH] Update README --- README.md | 132 +++++++++++++++++++++++++++++++----------------------- 1 file changed, 77 insertions(+), 55 deletions(-) diff --git a/README.md b/README.md index 22a1860..1f9731d 100644 --- a/README.md +++ b/README.md @@ -1,77 +1,99 @@ -# jsoup-scraper +# dex-scraper -This project uses Quarkus, the Supersonic Subatomic Java Framework. +A Kotlin-based web scraping application built with Quarkus for extracting structured data from Yu-Gi-Oh! card databases. -If you want to learn more about Quarkus, please visit its website: . +## Overview -## Running the application in dev mode +This project is a high-performance web scraper designed to extract structured data about Yu-Gi-Oh! card sets, regional variations, and card prints from various online sources. It leverages Quarkus for fast startup times and low memory usage, with a focus on reliability through CommonCrawl web archives. -You can run your application in dev mode that enables live coding using: +## Features -```shell script +- **Multi-source scraping**: Configurable scraping targets for different card databases +- **Historical data retrieval**: Uses CommonCrawl web archives for reliable historical data +- **Structured extraction**: Extracts card sets, regional sets, and card prints with validation +- **Flexible configuration**: YAML-based configuration for easy customization +- **REST API**: Provides HTTP endpoints for data access + +## Architecture + +The scraper follows a modular architecture: +- **Controller Layer**: REST endpoints for API access +- **Service Layer**: + - `ScrapeService`: Orchestrates the scraping process + - `SourceService`: Manages provider configurations + - `CommonCrawlService`: Handles web archive queries + - Extraction services for different data types +- **Configuration**: YAML-based configuration in `application.yml` + +## API Endpoints + +### Scrape Card Set +``` +GET /api/{provider}/set?setName={name} +``` +Returns structured card set information. + +### Scrape Regional Set +``` +GET /api/{provider}/regionalSet?setName={name} +``` +Returns regional variation information for a card set. + +### Scrape Card (Not Implemented) +``` +GET /api/{provider}/card?cardName={name} +``` +Currently not implemented. + +## Configuration + +The scraper is configured through `src/main/resources/application.yml`. Currently supports: + +- **Yu-Gi-Oh Fandom Wiki**: The primary target for card data extraction +- Configurable CSS and XPath selectors for element targeting +- Validation and transformation rules for data cleaning + +## Getting Started + +### Prerequisites +- JDK 21+ +- Gradle + +### Running in Development Mode +```bash ./gradlew quarkusDev ``` -> **_NOTE:_** Quarkus now ships with a Dev UI, which is available in dev mode only at . - -## Packaging and running the application - -The application can be packaged using: - -```shell script +### Building the Application +```bash ./gradlew build ``` -It produces the `quarkus-run.jar` file in the `build/quarkus-app/` directory. -Be aware that it’s not an _über-jar_ as the dependencies are copied into the `build/quarkus-app/lib/` directory. - -The application is now runnable using `java -jar build/quarkus-app/quarkus-run.jar`. - -If you want to build an _über-jar_, execute the following command: - -```shell script -./gradlew build -Dquarkus.package.jar.type=uber-jar +The application can be run with: +```bash +java -jar build/quarkus-app/quarkus-run.jar ``` -The application, packaged as an _über-jar_, is now runnable using `java -jar build/*-runner.jar`. - -## Creating a native executable - -You can create a native executable using: - -```shell script +### Creating a Native Executable +```bash ./gradlew build -Dquarkus.native.enabled=true ``` -Or, if you don't have GraalVM installed, you can run the native executable build in a container using: +## Development -```shell script -./gradlew build -Dquarkus.native.enabled=true -Dquarkus.native.container-build=true -``` +This project uses: +- **Quarkus**: Supersonic Subatomic Java Framework +- **Kotlin**: Primary programming language +- **Jsoup**: HTML parsing library +- **CommonCrawl**: Web archive data retrieval -You can then execute your native executable with: `./build/jsoup-scraper-1.0-SNAPSHOT-runner` +## Contributing -If you want to learn more about building native executables, please consult . - -## Related Guides - -- REST ([guide](https://quarkus.io/guides/rest)): A Jakarta REST implementation utilizing build time processing and - Vert.x. This extension is not compatible with the quarkus-resteasy extension, or any of the extensions that depend on - it. -- REST Client ([guide](https://quarkus.io/guides/rest-client)): Call REST services -- REST Jackson ([guide](https://quarkus.io/guides/rest#json-serialisation)): Jackson serialization support for Quarkus - REST. This extension is not compatible with the quarkus-resteasy extension, or any of the extensions that depend on it -- Kotlin ([guide](https://quarkus.io/guides/kotlin)): Write your services in Kotlin - -## Provided Code - -### REST Client - -Invoke different services through REST with JSON - -[Related guide section...](https://quarkus.io/guides/rest-client) - -### REST +1. Fork the repository +2. Create a feature branch +3. Commit your changes +4. Push to the branch +5. Open a pull request Easily start your REST Web Services