Skip to content

Commit

Permalink
v2.3.10: 搭建readthedocs文档网站并迁移所有文档,新增插件—下载图片后缀过滤器,优化代码。(#152)
Browse files Browse the repository at this point in the history
  • Loading branch information
hect0x7 authored Oct 23, 2023
1 parent 6ab3bc4 commit 4eeee1e
Show file tree
Hide file tree
Showing 53 changed files with 741 additions and 739 deletions.
File renamed without changes.
2 changes: 1 addition & 1 deletion .github/workflows/download.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ on:
paths:
- '.github/workflows/download.yml' # 工作流定义
- 'usage/workflow_download.py' # 下载脚本
- 'assets/config/option_workflow_download.yml' # 配置文件
- 'assets/option/option_workflow_download.yml' # 配置文件

jobs:
crawler:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release_auto.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ jobs:
id: tb
run: |
commit_message=$(git log --format=%B -n 1 ${{ github.sha }})
python release.py "$commit_message"
python .github/release.py "$commit_message"
- name: Create Release
uses: softprops/action-gh-release@v1
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/test_api.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,19 +10,19 @@ on:
- 'src/**/*.py'
- 'tests/**/*.py'
- '.github/workflows/test_api.yml'
- 'assets/config/option_test_api.yml'
- 'assets/option/option_test_api.yml'

jobs:
test: # This code is based on https://github.com/gaogaotiantian/viztracer/blob/master/.github/workflows/python-package.yml
strategy:
matrix:
python-version: [ 3.7, 3.8, 3.9, "3.10", "3.11" ]
python-version: ['3.7', '3.8', '3.11', '3.12']
os: [ ubuntu-latest ]
runs-on: ${{ matrix.os }}
timeout-minutes: 5
env:
# 配置文件路径
JM_OPTION_PATH_TEST: ./assets/config/option_test_api.yml
JM_OPTION_PATH_TEST: ./assets/option/option_test_api.yml

steps:
- name: Checkout repository
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/test_html.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,19 +10,19 @@ on:
- 'src/**/*.py'
- 'tests/**/*.py'
- '.github/workflows/test_html.yml'
- 'assets/config/option_test_html.yml'
- 'assets/option/option_test_html.yml'

jobs:
test: # This code is based on https://github.com/gaogaotiantian/viztracer/blob/master/.github/workflows/python-package.yml
strategy:
matrix:
python-version: [ 3.7, 3.8, 3.9, "3.10", "3.11" ]
python-version: ['3.7', '3.8', '3.11', '3.12']
os: [ ubuntu-latest ]
runs-on: ${{ matrix.os }}
timeout-minutes: 5
env:
# 配置文件路径
JM_OPTION_PATH_TEST: ./assets/config/option_test_html.yml
JM_OPTION_PATH_TEST: ./assets/option/option_test_html.yml

steps:
- name: Checkout repository
Expand Down
15 changes: 15 additions & 0 deletions .readthedocs.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Required
version: 2

# Set the OS, Python version and other tools you might need
build:
os: ubuntu-22.04
tools:
python: "3.12"
mkdocs:
configuration: assets/docs/mkdocs.yml

# Optionally declare the Python requirements required to build your docs
python:
install:
- requirements: assets/docs/requirements.txt
59 changes: 24 additions & 35 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@
```shell
pip install jmcomic -i https://pypi.org/project --upgrade
```
* 本地安装
* 通过源代码安装

```shell
pip install -e ./
pip install git+https://github.com/hect0x7/JMComic-Crawler-Python
```

## 快速上手
Expand All @@ -26,63 +26,52 @@
```python
import jmcomic # 导入此模块,需要先安装.
jmcomic.download_album('422866') # 传入要下载的album的id,即可下载整个album到本地.
# 上面的这行代码,还有一个可选参数option: JmOption,表示配置项,
# 配置项的作用是告诉程序下载时候的一些选择,
# 比如,要下载到哪个文件夹,使用怎样的路径组织规则(比如[/作者/本子id/图片] 或者 [/作者/本子名称/图片]).
# 如果没有配置,则会使用 JmOption.default(),下载的路径是[当前工作文件夹/本子名称/图片].
# 如果你想要配置,请参考文件 assets/config/常用配置介绍.yml
```

* v2.2.9: 新增命令行调用方式,上述的代码可以转为一行命令

```bash
# 下载album_id为422866的本子
$ jmcomic 422866
# 更多用法请参考文件 usage/usage_cl.py (命令行使用介绍)
```

## 进阶使用

进阶使用可以参考本repo下usage文件夹内的示例代码文件,下面是各个文件的作用,你可以挑感兴趣的阅读:

- API上手介绍: `getting_started.py`
- 命令行使用介绍: `usage_cl.py`
- 配置客户端的实现(网页端、移动端): `usage_configure_client_impl.py`
- 使用API实现简单功能: `usage_simple.py`
- 演示jmcomic模块的可自定义功能点: `usage_custom.py`
- 使用API的Filter过滤功能: `usage_feature_filter.py`
- 演示jmcomic模块的Plugin插件体系: `usage_feature_plugin.py`
- 演示一个综合使用实例: `usage_advanced.py`
- 包括6个功能需求的介绍、实现方案和完整运行日志
- 实现方案非常简洁,充分jmcomic的便利性,以及强大的插件扩展机制

以及一些趣味用法:

- 测试你的ip可以访问哪些禁漫域名: `pick_domain.py`
- 基于GitHub Actions下载本子: `workflow_download.py`

## 项目特点

- **绕过Cloudflare的反爬虫**
- 支持使用**命令行**下载本子,无需写Python代码,简单易用
- 支持使用**GitHub Actions**下载本子,网页上直接输入本子id就能下载([教程:使用GitHub Actions下载禁漫本子](./assets/docs/教程:使用GitHub%20Actions下载禁漫本子.md)
- 用法多样:

- GitHub Actions:网页上直接输入本子id就能下载([教程:使用GitHub Actions下载禁漫本子](./assets/docs/sources/tutorial/1_github_actions.md)
- 命令行:无需写Python代码,简单易用([教程:使用命令行下载禁漫本子](./assets/docs/sources/tutorial/2_command_line.md)
- Python代码:最直接的使用方式,需要你有一定的python编程基础
- 支持**网页端****移动端**两种客户端实现,可通过配置切换(**移动端不限ip兼容性好,网页端限制ip地区但效率高**
- 支持**自动重试和域名切换**机制
- **多线程下载**(可细化到一图一线程,效率极高)
- 跟进了JM最新的图片分割算法(2023-02-08)
- **可配置性强**

- 不配置也能使用,十分方便
- 配置可以从**配置文件**生成,支持多种文件格式
- 配置可以从配置文件生成,支持多种文件格式
- 配置点有:`请求域名` `客户端实现` `是否使用磁盘缓存` `同时下载的章节/图片数量` `图片格式转换` `下载路径规则` `请求元信息(headers,cookies,proxies)`
- **可扩展性强**

- **支持Plugin插件,可以方便地扩展功能,以及使用别人的插件**
- 目前内置支持的插件有:`登录插件` `硬件占用监控插件` `只下载新章插件` `压缩文件插件`
- 目前内置支持的插件有:`登录插件` `硬件占用监控插件` `只下载新章插件` `压缩文件插件` `下载特定后缀图片插件`
- 支持自定义本子/章节/图片下载前后的回调函数
- 支持自定义debug日志
- 支持自定义类:`Downloader(负责调度)` `Option(负责配置)` `Client(负责请求)` `实体类`

## 进阶使用

进阶使用请查阅文档:[文档](https://jmcomic.readthedocs.io/en/latest)

下面列出一些常用的文档链接:

* [option配置文件语法](./assets/docs/sources/option_file_syntax.md)
* [常用类和方法演示(下载本子、获取实体类、搜索本子)](assets/docs/sources/tutorial/3_demo.md)
* [命令行使用教程](assets/docs/sources/tutorial/2_command_line.md)
* [GitHub Actions使用教程](./assets/docs/sources/tutorial/1_github_actions.md)
* [插件机制](assets/docs/sources/tutorial/6_plugin.md)
* [下载过滤器机制](assets/docs/sources/tutorial/5_filter.md)

## 使用小说明

* Python >= 3.7
Expand All @@ -97,8 +86,8 @@ $ jmcomic 422866
* src:存放源代码

* jmcomic:`jmcomic`模块
* tests:测试目录,存放测试代码,使用unittest
* usage:用法目录,存放示例/使用代码
* tests:测试目录,存放测试代码,使用unittest
* usage:用法目录,存放示例/使用代码

## 感谢以下项目

Expand Down
25 changes: 0 additions & 25 deletions assets/config/option_plugin.yml

This file was deleted.

22 changes: 22 additions & 0 deletions assets/docs/mkdocs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
site_name: jmcomic
theme:
name: readthedocs
highlightjs: true
plugins:
- search
- mkdocstrings:
custom_templates: templates
handlers:
# See: https://mkdocstrings.github.io/python/usage/
python:
paths: [ '../../src/' ]
options:
docstring_style: sphinx


markdown_extensions:
- markdown_include.include:
base_path: .
- admonition

docs_dir: sources
4 changes: 4 additions & 0 deletions assets/docs/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
mkdocs
mkdocstrings[python]
markdown-include
mkdocs-material
2 changes: 1 addition & 1 deletion TODO.md → assets/docs/sources/TODO.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# 大版本更新内容计划
# Plan For Update Content

| 版本范围 | 更新内容 |
|:--------:|:--------------------------------------:|
Expand Down
4 changes: 4 additions & 0 deletions assets/docs/sources/api/client.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# client

::: jmcomic.jm_client_impl

3 changes: 3 additions & 0 deletions assets/docs/sources/api/config.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# config

::: jmcomic.jm_config.JmModuleConfig
4 changes: 4 additions & 0 deletions assets/docs/sources/api/download.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# download

::: jmcomic.api

3 changes: 3 additions & 0 deletions assets/docs/sources/api/entity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# entity

::: jmcomic.jm_entity
3 changes: 3 additions & 0 deletions assets/docs/sources/api/option.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# option

::: jmcomic.jm_option.JmOption
3 changes: 3 additions & 0 deletions assets/docs/sources/api/plugin.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# plugin

::: jmcomic.jm_plugin
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
File renamed without changes
102 changes: 102 additions & 0 deletions assets/docs/sources/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
# jmcomic

Python API for JMComic(禁漫天堂)

## Features

- Bypasses Cloudflare anti-bot measures.
- Multiple usage ways:

- GitHub Actions: Requires only a GitHub account. (See
tutorial → [Tutorial - Download Album via GitHub Actions](./tutorial/1_github_actions.md))

- Command line: No need to write Python code, simple and easy to use. (See tutorial → [Tutorial - Download Album via Command Line](./tutorial/2_command_line.md))
- Python code: The most flexible and powerful way, requires some basic knowledge of Python programming.

- Supports two client implementations: web-based and mobile-based. Switchable through configuration (mobile-based has
better IP compatibility, web-based has higher efficiency).
- Supports automatic request retry and domain switching mechanism.
- Multi-threaded downloading (can be fine-tuned to one thread per image, highly efficient).
- Highly configurable:

- Can be used without configuration, very convenient.
- Configuration can be generated from a configuration file, supports multiple file formats.
- Configuration options
include: `request domain`, `client implementation`, `number of chapters/images downloaded simultaneously`, `image format conversion`, `download path rules`, `request metadata (headers, cookies, proxies)`,
and more.

- Highly extensible:

- Supports Plugin plugins for easy functionality extension and use of other plugins.
- Currently built-in
plugins: `login plugin`, `hardware usage monitoring plugin`, `only download new chapters plugin`, `zip compression plugin`.
- Supports custom callback functions before and after downloading album/chapter/images.
- Supports custom debug logging.
- Supports custom core
classes: `Downloader (responsible for scheduling)`, `Option (responsible for configuration)`, `Client (responsible for requests)`, `entity classes`,
and more.

## Install

- Install via official pip source (recommended, and also used for updates):

```
pip install jmcomic -i https://pypi.org/project --upgrade
```

- Install via GitHub code:

```
pip install git+https://github.com/hect0x7/JMComic-Crawler-Python
```

## Getting Started

### 1. Download album:

- Python code

```python
import jmcomic
# Pass the ID of the album you want to download, and it will download all chapters of the album to your local machine.
jmcomic.download_album('422866')
```

- Command line

```
jmcomic 422866
```

### 2. Customize download behavior using an option:

For example, if you want to convert all downloaded images to the .jpg format, you can create a YAML file with the
following content (refer to [option file syntax](./option_file_syntax.md)):

```yml
download:
image:
suffix: .jpg # Don't forget the '.'
```
Then, use one of the following ways:
* Python code
```python
from jmcomic import download_album, create_option
option = create_option('/path/to/your/optionfile')
download_album('422866', option)
```

* Command line

```
jmcomic 422866 --option="/path/to/your/optionfile"
```

## Acknowledgement

### Image Segmentation Algorithm Code + JMComic Mobile API

[![Readme Card](https://github-readme-stats.vercel.app/api/pin/?username=tonquer&repo=JMComic-qt)](https://github.com/tonquer/JMComic-qt)
Loading

0 comments on commit 4eeee1e

Please sign in to comment.