Update README
This commit is contained in:
@@ -1,77 +1,99 @@
|
|||||||
# jsoup-scraper
|
# dex-scraper
|
||||||
|
|
||||||
This project uses Quarkus, the Supersonic Subatomic Java Framework.
|
A Kotlin-based web scraping application built with Quarkus for extracting structured data from Yu-Gi-Oh! card databases.
|
||||||
|
|
||||||
If you want to learn more about Quarkus, please visit its website: <https://quarkus.io/>.
|
## Overview
|
||||||
|
|
||||||
## Running the application in dev mode
|
This project is a high-performance web scraper designed to extract structured data about Yu-Gi-Oh! card sets, regional variations, and card prints from various online sources. It leverages Quarkus for fast startup times and low memory usage, with a focus on reliability through CommonCrawl web archives.
|
||||||
|
|
||||||
You can run your application in dev mode that enables live coding using:
|
## Features
|
||||||
|
|
||||||
```shell script
|
- **Multi-source scraping**: Configurable scraping targets for different card databases
|
||||||
|
- **Historical data retrieval**: Uses CommonCrawl web archives for reliable historical data
|
||||||
|
- **Structured extraction**: Extracts card sets, regional sets, and card prints with validation
|
||||||
|
- **Flexible configuration**: YAML-based configuration for easy customization
|
||||||
|
- **REST API**: Provides HTTP endpoints for data access
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
The scraper follows a modular architecture:
|
||||||
|
- **Controller Layer**: REST endpoints for API access
|
||||||
|
- **Service Layer**:
|
||||||
|
- `ScrapeService`: Orchestrates the scraping process
|
||||||
|
- `SourceService`: Manages provider configurations
|
||||||
|
- `CommonCrawlService`: Handles web archive queries
|
||||||
|
- Extraction services for different data types
|
||||||
|
- **Configuration**: YAML-based configuration in `application.yml`
|
||||||
|
|
||||||
|
## API Endpoints
|
||||||
|
|
||||||
|
### Scrape Card Set
|
||||||
|
```
|
||||||
|
GET /api/{provider}/set?setName={name}
|
||||||
|
```
|
||||||
|
Returns structured card set information.
|
||||||
|
|
||||||
|
### Scrape Regional Set
|
||||||
|
```
|
||||||
|
GET /api/{provider}/regionalSet?setName={name}
|
||||||
|
```
|
||||||
|
Returns regional variation information for a card set.
|
||||||
|
|
||||||
|
### Scrape Card (Not Implemented)
|
||||||
|
```
|
||||||
|
GET /api/{provider}/card?cardName={name}
|
||||||
|
```
|
||||||
|
Currently not implemented.
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
The scraper is configured through `src/main/resources/application.yml`. Currently supports:
|
||||||
|
|
||||||
|
- **Yu-Gi-Oh Fandom Wiki**: The primary target for card data extraction
|
||||||
|
- Configurable CSS and XPath selectors for element targeting
|
||||||
|
- Validation and transformation rules for data cleaning
|
||||||
|
|
||||||
|
## Getting Started
|
||||||
|
|
||||||
|
### Prerequisites
|
||||||
|
- JDK 21+
|
||||||
|
- Gradle
|
||||||
|
|
||||||
|
### Running in Development Mode
|
||||||
|
```bash
|
||||||
./gradlew quarkusDev
|
./gradlew quarkusDev
|
||||||
```
|
```
|
||||||
|
|
||||||
> **_NOTE:_** Quarkus now ships with a Dev UI, which is available in dev mode only at <http://localhost:8080/q/dev/>.
|
### Building the Application
|
||||||
|
```bash
|
||||||
## Packaging and running the application
|
|
||||||
|
|
||||||
The application can be packaged using:
|
|
||||||
|
|
||||||
```shell script
|
|
||||||
./gradlew build
|
./gradlew build
|
||||||
```
|
```
|
||||||
|
|
||||||
It produces the `quarkus-run.jar` file in the `build/quarkus-app/` directory.
|
The application can be run with:
|
||||||
Be aware that it’s not an _über-jar_ as the dependencies are copied into the `build/quarkus-app/lib/` directory.
|
```bash
|
||||||
|
java -jar build/quarkus-app/quarkus-run.jar
|
||||||
The application is now runnable using `java -jar build/quarkus-app/quarkus-run.jar`.
|
|
||||||
|
|
||||||
If you want to build an _über-jar_, execute the following command:
|
|
||||||
|
|
||||||
```shell script
|
|
||||||
./gradlew build -Dquarkus.package.jar.type=uber-jar
|
|
||||||
```
|
```
|
||||||
|
|
||||||
The application, packaged as an _über-jar_, is now runnable using `java -jar build/*-runner.jar`.
|
### Creating a Native Executable
|
||||||
|
```bash
|
||||||
## Creating a native executable
|
|
||||||
|
|
||||||
You can create a native executable using:
|
|
||||||
|
|
||||||
```shell script
|
|
||||||
./gradlew build -Dquarkus.native.enabled=true
|
./gradlew build -Dquarkus.native.enabled=true
|
||||||
```
|
```
|
||||||
|
|
||||||
Or, if you don't have GraalVM installed, you can run the native executable build in a container using:
|
## Development
|
||||||
|
|
||||||
```shell script
|
This project uses:
|
||||||
./gradlew build -Dquarkus.native.enabled=true -Dquarkus.native.container-build=true
|
- **Quarkus**: Supersonic Subatomic Java Framework
|
||||||
```
|
- **Kotlin**: Primary programming language
|
||||||
|
- **Jsoup**: HTML parsing library
|
||||||
|
- **CommonCrawl**: Web archive data retrieval
|
||||||
|
|
||||||
You can then execute your native executable with: `./build/jsoup-scraper-1.0-SNAPSHOT-runner`
|
## Contributing
|
||||||
|
|
||||||
If you want to learn more about building native executables, please consult <https://quarkus.io/guides/gradle-tooling>.
|
1. Fork the repository
|
||||||
|
2. Create a feature branch
|
||||||
## Related Guides
|
3. Commit your changes
|
||||||
|
4. Push to the branch
|
||||||
- REST ([guide](https://quarkus.io/guides/rest)): A Jakarta REST implementation utilizing build time processing and
|
5. Open a pull request
|
||||||
Vert.x. This extension is not compatible with the quarkus-resteasy extension, or any of the extensions that depend on
|
|
||||||
it.
|
|
||||||
- REST Client ([guide](https://quarkus.io/guides/rest-client)): Call REST services
|
|
||||||
- REST Jackson ([guide](https://quarkus.io/guides/rest#json-serialisation)): Jackson serialization support for Quarkus
|
|
||||||
REST. This extension is not compatible with the quarkus-resteasy extension, or any of the extensions that depend on it
|
|
||||||
- Kotlin ([guide](https://quarkus.io/guides/kotlin)): Write your services in Kotlin
|
|
||||||
|
|
||||||
## Provided Code
|
|
||||||
|
|
||||||
### REST Client
|
|
||||||
|
|
||||||
Invoke different services through REST with JSON
|
|
||||||
|
|
||||||
[Related guide section...](https://quarkus.io/guides/rest-client)
|
|
||||||
|
|
||||||
### REST
|
|
||||||
|
|
||||||
Easily start your REST Web Services
|
Easily start your REST Web Services
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user