28 Commits

Author SHA1 Message Date
nianzhibai 00aaeed736 feat: add recent watch sorting 2026-06-20 15:08:47 +08:00
nianzhibai 5efbceb205 fix: add safe-area home screen icons 2026-06-20 14:36:24 +08:00
nianzhibai 0faeaf408f fix(crawlers): stabilize manual upload workflow
Add a manual crawler upload action in the admin UI and backend so users can retry uploads when automatic migration leaves local crawler videos behind.

Keep the button always clickable and return clear refusal messages when there are no local videos, no upload target, unfinished fingerprints/previews, failed generated assets, or active crawler work.

Simplify crawler cards by removing pipeline/status capsules, dropping the ready pill, and aligning the preview toggle with the existing action button style.

Avoid the small-/tmp upload bug by reusing seekable local files for PikPak GCID calculation/uploads and by routing fallback upload temp files for PikPak, 115, 123Pan, and WoPan into the application data upload-tmp directory.

Add regression coverage for crawler manual upload handling, frontend form expectations, configured upload temp dirs, and PikPak seekable-reader uploads.

Verification: npm run lint; npm test; npm run build; go test ./... -count=1.
2026-06-20 00:14:37 +08:00
nianzhibai 1b5eda92b0 Merge pull request #64 from iBenzene/feat/google-drive-openlist-api-url 2026-06-19 10:57:15 +08:00
nianzhibai 840a858dbd chore: remove bundled 91VideoSpider script 2026-06-19 10:42:35 +08:00
nianzhibai 1ee5ee35be Merge pull request #63 from iBenzene/fix/rescan-sync-renamed-videos
fix(scanner): sync renamed video metadata on rescan
2026-06-19 10:32:20 +08:00
iBenzene 12b737b6fe feat: add google drive openlist api url field 2026-06-19 10:23:08 +08:00
iBenzene bd33d26a1f fix(scanner): sync renamed video metadata on rescan 2026-06-19 08:40:38 +08:00
nianzhibai 36fe32cb84 feat: optimize toast feedback and bundle splitting
- lazy-load route pages and use the HLS light build to remove large chunk warnings

- copy toast messages to the clipboard with success/failure feedback

- make scriptcrawler dry-run stderr capture deterministic
2026-06-18 23:29:16 +08:00
nianzhibai 194d98895a fix: allow guangyapan crawler uploads and improve admin toasts 2026-06-17 17:33:58 +08:00
nianzhibai 2437fbd779 发布v0.1.9 版本
1. 修复Apple设备短视频页面无法播放问题
2. 优化短视频播放页面交互
2026-06-15 20:23:08 +08:00
nianzhibai 4dd66b8120 fix: stabilize shorts playback and iPhone fullscreen controls 2026-06-15 20:19:28 +08:00
nianzhibai 30b736cf36 Merge pull request #54 2026-06-15 17:38:32 +08:00
Long Chen 57391e0e98 fix: render short videos on iOS by dropping fixed positioning
Short-video mode showed a black screen on iOS Safari/WebKit while working
on desktop. iOS does not composite an inline <video> nested inside a
`position: fixed` ancestor: the video decodes and plays but its layer
never paints. `.shorts-page` used `position: fixed; inset: 0`, trapping
every shorts video.

Make `.shorts-page` a normal-flow full-viewport block
(`position: relative; height: 100svh` with dvh/vh fallbacks) instead.
The immersive scroll lock is already provided by the component setting
html/body `overflow: hidden` on mount, so the look and behavior are
unchanged. Fixes the iOS black screen; desktop is unaffected.
2026-06-15 17:20:35 +08:00
nianzhibai 052e142520 chore: bump version to 0.1.8 2026-06-14 18:25:19 +08:00
nianzhibai f9351324c6 fix: show active preview generation status 2026-06-14 18:22:04 +08:00
nianzhibai bb83277d62 feat: add crawler preview generation toggle
Expose per-crawler teaser settings on crawler cards and persist them through the admin API.\n\nWhen preview generation is disabled, crawler imports still create thumbnails and fingerprints while marking previews disabled and allowing migration without waiting for teaser files.\n\nPreserve the latest teaser setting after crawler runs so stale crawl state cannot overwrite a user toggle.
2026-06-14 17:52:29 +08:00
nianzhibai aa856db1f6 fix: refine video UI loading and playback behavior 2026-06-14 16:59:41 +08:00
nianzhibai 7e5e67697e feat: add GuangYaPan drive support
Implement a new GuangYaPan cloud drive integration across the backend, admin UI, playback proxy, and Spider91 migration flow.

Backend changes:\n- Add a GuangYaPan drive driver with token refresh, QR/device login support, directory listing, stream link resolution, directory creation, rename/delete operations, OSS multipart upload, and upload task polling.\n- Register GuangYaPan as a supported storage kind in configuration, catalog normalization, admin APIs, public drive labels, and 302 playback redirects.\n- Allow Spider91 crawler uploads to target GuangYaPan through a dedicated migration adapter.\n- Add scan, thumbnail, preview, and fingerprint cooldown handling for GuangYaPan based on explicit HTTP status codes, Retry-After values, and structured provider codes instead of natural-language message matching.\n- Tighten existing provider cooldown detectors so OneDrive, Google Drive, 115, PikPak, 123pan, Wopan, and media workers avoid treating arbitrary response text as a rate-limit signal.\n- Keep large videos eligible for preview generation unless the user disables preview generation.

Admin and tooling changes:\n- Add GuangYaPan as a selectable drive type with QR login UI and token/root-path credential fields.\n- Add crawler upload target support for GuangYaPan in the admin UI.\n- Add drive branding, labels, metadata display, and docs/config examples for GuangYaPan.\n- Include a standalone GuangYaPan QR login helper script for manual credential acquisition.

Tests:\n- Add GuangYaPan driver, QR login, proxy, admin API, crawler upload target, fingerprint, cooldown, and form coverage.\n- Update rate-limit tests to assert that message-only throttling text no longer starts cooldowns.\n- Cover explicit HTTP status parsing through shared drive helper tests.
2026-06-14 15:44:50 +08:00
nianzhibai 9cc8e02bec feat: add sky theme and refresh themed UI
Add the sky theme across the frontend and backend theme APIs, including starfield assets and icon-only branding.

Refresh themed grid backgrounds, admin/login/sidebar styling, and theme-specific video/listing polish.
2026-06-14 11:53:07 +08:00
nianzhibai 139e63eef2 Update README to remove outdated features
Removed compatibility transcoding and video management features from the README.
2026-06-13 16:45:03 +08:00
nianzhibai b8388eba59 Prepare v0.1.7 release 2026-06-13 16:32:20 +08:00
nianzhibai 76782f3801 Refine shorts playback caching
Remove shorts recommendation preference, keep a six-video viewed cache window, preload the next two videos after healthy buffering, and avoid premature seen-list resets between sessions.
2026-06-13 16:26:36 +08:00
nianzhibai 1ae1408fb6 feat: refine video detail player controls
Remove the hide action from the video detail page and keep delete as the only management action.

Adjust mobile delete dialog and ArtPlayer settings UI, disable persisted player settings, and add a temporary loop option.
2026-06-13 15:18:20 +08:00
nianzhibai 738406162a feat: add video blacklist management
Add backend blacklist tombstone APIs and hidden-video migration support.

Update the admin video management UI with blacklist tabs, restore actions, alignment fixes, responsive layout polish, and regression coverage.
2026-06-13 14:34:00 +08:00
nianzhibai 0f111b846d feat: add opt-in toggle for local STRM targets outside the storage root
Local .strm files that pointed to a path outside the configured storage
root previously failed cover/preview/fingerprint generation and playback
with "strm target escapes root", breaking the common layout where the
strm library and the real media files (e.g. an rclone mount) live in
separate directories (issue #22 follow-up).

- localstorage driver gains STRMAllowOutsideRoot; when on, strm targets
  outside the root are allowed (still resolves symlinks and still rejects
  nested strm, so no new escape vector). Default off preserves the
  existing security boundary
- Toggle persisted as the strm_allow_outside_root credential; editing a
  localstorage drive now merges credentials per-key so leaving the path
  blank keeps the old value while flipping the toggle
- Saving a localstorage drive with the toggle on auto-re-enqueues
  previously-failed thumbnails/previews/fingerprints, so enabling it
  recovers without manually clicking the three retry buttons
- Drives API exposes strmAllowOutsideRoot for form echo-back; admin
  drive form adds a "允许指向目录外" select with a security warning
- Tests cover allow-outside-root on/off and that nested strm stays
  rejected even when the toggle is on

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 10:34:54 +08:00
nianzhibai 4dd9015bd7 feat: add per-storage manual transcode for browser-incompatible videos
Add a transcode control to each storage in the admin drives page,
modeled after the cover/preview generation controls:

- Manual start/stop button per storage; transcoding is off by default
  and never runs automatically (not triggered by scans or the nightly
  pipeline)
- New transcode worker probes candidates (non mp4/webm extensions)
  with ffprobe: already-compatible files are marked skipped; AVI with
  H.264 is remuxed losslessly; incompatible codecs (MPEG-4 Part 2,
  WMV, RMVB, HEVC...) are transcoded to H.264/AAC MP4 with +faststart
- Transcoded output is uploaded back to the same storage under a
  "91转码" directory which is auto-added to the drive's scan skip list
  so the scanner never re-imports the artifacts
- Playback source automatically prefers the transcoded file once
  ready, keeping the 302 direct-link mode for cloud drives
- videos table gains transcode_status/error/file_id/size columns via
  startup migration; counts and live task status surface in the
  admin drives API and generation panel UI
- Stop semantics: per-drive stop button, drive-level "stop all tasks"
  and global stop all include the transcode task; interrupted videos
  keep their candidate status and resume on next start

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 09:41:08 +08:00
nianzhibai 84fbb6f51c feat: improve shorts playback with buffer-aware preload and back-swipe cache
- Active video loads with priority; next video preloads only after the
  active one has 12s of forward buffer or is buffered to the end
- Add high/low watermark hysteresis (12s grant / 4s revoke) so the
  preload grant no longer thrashes around the threshold, discarding
  already-preloaded data
- Treat buffered-to-end as healthy to avoid revoking preload near the
  end of short clips
- Mark sources as cacheable on first canplay and keep src bound for
  cached adjacent slides in both directions, so swiping back (and
  forward again) reuses the browser buffer instead of reloading

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-13 09:00:33 +08:00
118 changed files with 9466 additions and 2947 deletions
+7 -3
View File
@@ -30,13 +30,17 @@ tmp/
# 91 爬虫脚本独立运行时的默认输出文件(backend 跑时会显式 --output 到 backend/data/spider91/,所以不会落在这里)
91porn_videos.json
91VideoSpider/91porn_videos.json
91VideoSpider/data/
91VideoSpider/__pycache__/
__pycache__/
*.pyc
# Local scratch images
/*.png
/*.jpg
/*.jpeg
/*.gif
/*.webp
/*.bmp
/*.ico
/image.jpg
/image003.jpg
/image004.jpg
-988
View File
@@ -1,988 +0,0 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
91porn 视频爬虫脚本
===================
爬取 https://www.91porn.com/v.php?category=top&viewtype=basic 下的所有视频信息:
- 视频名称
- 封面图直链
- 视频直链 (MP4)
依赖安装:
pip install requests beautifulsoup4 lxml PySocks
使用方法:
# 作为 video-site-91 通用爬虫脚本运行(后台会自动这样调用)
python spider_91porn.py --job /path/to/job.json
# 全量爬取(默认行为,从 page=1 一直爬到末尾,写到 OUTPUT_FILE
python spider_91porn.py
# 只爬指定页(单页模式,手动调试用)
python spider_91porn.py --page 1 --output /tmp/spider91_page1.json
# 凑够 N 个新视频模式(backend 凌晨任务用)
python spider_91porn.py --target-new 15 --seen-viewkeys-file /tmp/seen.txt --output /tmp/new.json
CLI 参数:
--job FILE crawler.v1 job JSON 路径;后台爬虫管理会使用此模式
--page N 只爬第 N 页,配合 --output 用于手动调试
--target-new N 从 page 1 起翻页直到凑够 N 个新视频(不在 seen 列表里的)
--seen-viewkeys-file FILE 每行一个已知 viewkey 或 mp4 源 ID,命中即跳过;与 --target-new 配合使用
--output FILE 输出 JSON 路径,覆盖默认的 OUTPUT_FILE
--no-resume 禁用断点续爬(单页/target-new 模式下自动禁用)
--quiet 压缩日志,每条视频只输出一行
-h / --help 帮助
配置说明 (编辑脚本内 "配置区域"):
- MIN_PAGE_DELAY / MAX_PAGE_DELAY : 列表页请求间隔 (默认 3-6 秒)
- MIN_DETAIL_DELAY / MAX_DETAIL_DELAY : 详情页请求间隔 (默认 2-5 秒)
- MAX_PAGES : 限制最大爬取页数 (None=不限, 如 5=只爬前5页)
- OUTPUT_FILE : 输出文件名
输出格式 (JSON):
--job 模式下 stdout 输出 crawler.v1 JSON Lines,日志全部写到 stderr。
手动运行模式仍会写传统 JSON 文件:
{
"videos": [
{
"title": "视频标题",
"thumb_url": "https://...thumb/xxxx.jpg",
"video_url": "https://...mp43/xxxx.mp4?st=...",
"viewkey": "abc123...",
"source_id": "xxxx",
"detail_url": "https://...view_video.php?viewkey=..."
},
...
]
}
注意:
1. 视频直链包含时效性token (e参数为过期时间戳),会过期,需定期重新爬取
2. 脚本已内置随机延时,请勿移除,避免对服务器造成压力
3. 网站有Cloudflare保护,如遇到403/5xx错误,可能需要使用带cookie的session
4. 本脚本仅供学习交流,请遵守当地法律法规
作者: OpenCode
日期: 2026-05-22
"""
import argparse
import requests
import re
import time
import random
import json
import os
import socket
import sys
import html
from urllib.parse import urljoin, unquote, urlparse
from datetime import datetime
try:
from bs4 import BeautifulSoup
except ImportError:
print("错误: 缺少依赖库 beautifulsoup4", file=sys.stderr)
print("请运行: pip install beautifulsoup4 lxml", file=sys.stderr)
sys.exit(1)
def prefer_ipv4_for_plain_socks5_proxy():
"""PySocks may pick IPv6 first for socks5://; some SOCKS5 servers only accept IPv4."""
proxy_envs = (
os.environ.get("HTTPS_PROXY", ""),
os.environ.get("HTTP_PROXY", ""),
os.environ.get("https_proxy", ""),
os.environ.get("http_proxy", ""),
)
uses_plain_socks5 = any(v.strip().lower().startswith("socks5://") for v in proxy_envs)
if not uses_plain_socks5 or getattr(socket, "_spider91_ipv4_first", False):
return
original_getaddrinfo = socket.getaddrinfo
def getaddrinfo_ipv4_first(*args, **kwargs):
infos = original_getaddrinfo(*args, **kwargs)
return sorted(infos, key=lambda info: 0 if info[0] == socket.AF_INET else 1)
socket.getaddrinfo = getaddrinfo_ipv4_first
socket._spider91_ipv4_first = True
# ===================== 配置区域 =====================
BASE_URL = "https://www.91porn.com/v.php"
LIST_PARAMS = {
"category": "top",
"viewtype": "basic"
}
# 请求头 (模拟真实浏览器)
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/125.0.0.0 Safari/537.36"
),
"Accept": (
"text/html,application/xhtml+xml,application/xml;"
"q=0.9,image/avif,image/webp,image/apng,*/*;"
"q=0.8,application/signed-exchange;v=b3;q=0.7"
),
"Accept-Language": "zh-CN,zh;q=0.9",
# 注意: 不要包含 "br" (brotli),除非安装了 brotli 库
# "Accept-Encoding": "gzip, deflate, br",
"Connection": "keep-alive",
"Upgrade-Insecure-Requests": "1",
"Sec-Fetch-Dest": "document",
"Sec-Fetch-Mode": "navigate",
"Sec-Fetch-Site": "none",
"Sec-Fetch-User": "?1",
}
# 延时配置 (秒) - 控制爬取频率,避免被封
MIN_PAGE_DELAY = 3.0 # 列表页之间最小延时
MAX_PAGE_DELAY = 6.0 # 列表页之间最大延时
MIN_DETAIL_DELAY = 2.0 # 详情页之间最小延时
MAX_DETAIL_DELAY = 5.0 # 详情页之间最大延时
# 重试配置
MAX_RETRIES = 3
RETRY_DELAY = 5.0
# 输出配置
OUTPUT_FILE = "91porn_videos.json"
MAX_PAGES = None # 设置为 None 爬取所有页,或设置整数如 5 只爬前5页
RESUME = True # 是否跳过输出文件中已存在的 viewkey (断点续爬)
MAX_EMPTY_PAGES = 2 # 连续空页数达到此值时停止爬取
CRAWLER_NAME = "91Porn"
CRAWLER_PROTOCOL = "crawler.v1"
# ===================================================
def crawler_source_id(raw: str) -> str:
"""Return a backend-safe source_id, preserving existing numeric 91 IDs."""
value = str(raw or "").strip()
if not value:
return ""
safe = re.sub(r"[^A-Za-z0-9_.-]+", "_", value).strip("._-")
return safe[:160]
def write_jsonl(event: dict):
print(json.dumps(event, ensure_ascii=False), flush=True)
class Porn91Spider:
def __init__(
self,
output_file: str = None,
start_page: int = 1,
max_pages: int = None,
resume: bool = None,
max_empty_pages: int = None,
quiet: bool = False,
target_new: int = None,
seen_viewkeys: list = None,
stream_output: bool = False,
stream_protocol: str = "legacy",
):
"""
构造函数。所有参数都有默认值,等同于使用脚本顶部的全局配置。
backend 调用时会传 output_file/seen_viewkeys/target_new,等价于:
"从第 1 页开始爬,跳过 seen_viewkeys 里的视频,凑够 target_new 个新视频后停止"
stream_output=True 时(backend 流水线用):
- 每凑齐一个 video 直链就把该 entry 作为一行 JSON 写到 stdout 并 flush
便于上层(Go crawler)边读边下载,不再等所有详情页处理完。
- 所有日志改走 stderr,避免与 stdout JSONL 流混合。
- --output 仍生效,作为离线归档用(脚本退出时一次性写完整 JSON)。
"""
self.session = requests.Session()
self.session.headers.update(HEADERS)
# 91porn 没有固定 mode cookie 时,详情页首次请求可能返回与列表卡片
# 不一致的视频源;固定桌面模式让列表页和详情页解析保持一致。
self.session.cookies.set("mode", "d")
# 解析后的实际配置;优先使用构造参数,回退到模块级配置
self.output_file = output_file if output_file is not None else OUTPUT_FILE
self.start_page = max(1, int(start_page or 1))
# max_pages=None 表示不限制;max_pages=N 表示从 start_page 起爬 N 页
self.max_pages = max_pages if max_pages is None or max_pages > 0 else None
# resume 默认跟模块配置;单页模式下调用方应该显式传 False
self.resume = RESUME if resume is None else bool(resume)
self.max_empty_pages = (
MAX_EMPTY_PAGES if max_empty_pages is None else int(max_empty_pages)
)
# target_new 是 backend 触发时的核心模式:累计处理这么多新源视频后退出。
self.target_new = target_new if target_new and target_new > 0 else None
self.quiet = bool(quiet)
# stream_output:每解析出一个 video 直链立即输出一行 JSON 到 stdout
# (配合 backend Go 端 bufio.Scanner 实时消费,下载一个就开始下一个)。
# 开启后所有 log 都走 stderr。
self.stream_output = bool(stream_output)
self.stream_protocol = stream_protocol or "legacy"
# 添加重试适配器
try:
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
retry_strategy = Retry(
total=MAX_RETRIES,
backoff_factor=1,
status_forcelist=[429, 500, 502, 503, 504],
)
adapter = HTTPAdapter(max_retries=retry_strategy)
self.session.mount("https://", adapter)
self.session.mount("http://", adapter)
except ImportError:
pass # urllib3 版本可能较低
self.results = []
self.pages_crawled = 0
self.processed_videos = 0
self.skipped_videos = 0
self.failed_videos = 0
self.skip_viewkeys = set()
# backend 通过 --seen-viewkeys-file 传进来一批已入库的历史 ID。
# 兼容旧名:文件里可能是 viewkey,也可能是新逻辑使用的 mp4 源 ID。
if seen_viewkeys:
for vk in seen_viewkeys:
if not vk:
continue
vk = vk.strip()
if vk:
self.skip_viewkeys.add(vk)
# 断点续爬:加载已有结果,跳过已处理的 viewkey
if self.resume and os.path.exists(self.output_file):
try:
with open(self.output_file, 'r', encoding='utf-8') as f:
existing_data = json.load(f)
existing_videos = existing_data.get('videos', [])
self.results = existing_videos
for v in existing_videos:
vk = v.get('viewkey', '')
if vk:
self.skip_viewkeys.add(vk)
self.processed_videos = existing_data.get('successful', 0)
self.failed_videos = existing_data.get('failed', 0)
self.log(f"加载已有数据: {len(self.results)} 个视频, 将跳过已处理项")
except Exception:
pass
def log(self, message: str):
"""带时间戳的日志输出。stream_output 模式下走 stderr,避免污染 stdout JSONL。"""
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
line = f"[{timestamp}] {message}"
if self.stream_output:
print(line, file=sys.stderr, flush=True)
else:
print(line)
def emit_stream_video(self, video: dict):
"""stream_output 模式下把单条 video entry 作为一行 JSON 写到 stdout 并立即刷盘。
Go 端 bufio.Scanner 按行读取,每收到一行就立即下载视频和封面。"""
if not self.stream_output:
return
try:
if self.stream_protocol == "crawler.v1":
source_id = crawler_source_id(video.get("source_id") or video.get("viewkey") or "")
item = {
"title": video.get("title") or "",
"detail_url": video.get("detail_url") or "",
"author": "91porn",
"tags": ["91porn"],
"media_url": video.get("video_url") or "",
"thumbnail_url": video.get("thumb_url") or "",
"headers": {
"Referer": video.get("detail_url") or BASE_URL,
},
}
if source_id:
item["source_id"] = source_id
event = {
"type": "item",
"item": item,
}
write_jsonl(event)
else:
print(json.dumps(video, ensure_ascii=False), flush=True)
except Exception as e:
# stdout 异常基本只在管道断开时发生(消费方进程死了);
# 写到 stderr 让 backend 看到,然后让 crawl 循环自己 break。
print(f"[stream] emit failed: {e}", file=sys.stderr, flush=True)
def random_sleep(self, min_sec: float, max_sec: float):
"""随机延时,模拟人类行为"""
delay = random.uniform(min_sec, max_sec)
if not self.quiet:
self.log(f" 随机延时 {delay:.2f} 秒...")
time.sleep(delay)
def fetch_page(self, url: str, description: str = "", referer: str = "") -> str:
"""
获取页面HTML内容,带错误处理和重试
"""
headers_extra = {}
if referer:
headers_extra["Referer"] = referer
for attempt in range(1, MAX_RETRIES + 1):
try:
self.log(f"正在请求: {description or url} (尝试 {attempt}/{MAX_RETRIES})")
response = self.session.get(url, timeout=30, headers=headers_extra)
# 检查是否被Cloudflare拦截 (需在 raise_for_status 之前)
if response.status_code == 403:
self.log("警告: 收到 403 Forbidden,可能被拦截")
if attempt < MAX_RETRIES:
self.random_sleep(RETRY_DELAY, RETRY_DELAY + 3)
continue
return ""
response.raise_for_status()
# 优先使用 content.decode('utf-8'),避免 requests 编码检测问题
try:
html_content = response.content.decode('utf-8', errors='replace')
except Exception:
html_content = response.text
# Cloudflare 挑战检测:如果页面主要内容只有挑战页面,而非正常内容
# 注意:网站本身会加载 challenge-platform 脚本,所以不能仅凭此判断
is_cf_challenge = (
"Just a moment" in html_content and
len(html_content) < 8000
)
if is_cf_challenge:
self.log("警告: 页面被Cloudflare挑战拦截,需要浏览器环境或正确cookie")
if attempt < MAX_RETRIES:
self.random_sleep(RETRY_DELAY, RETRY_DELAY + 5)
continue
return ""
return html_content
except requests.exceptions.HTTPError as e:
self.log(f"HTTP错误: {e}")
if attempt < MAX_RETRIES:
self.random_sleep(RETRY_DELAY, RETRY_DELAY + 3)
else:
return ""
except requests.exceptions.RequestException as e:
self.log(f"请求失败: {e}")
if attempt < MAX_RETRIES:
self.random_sleep(RETRY_DELAY, RETRY_DELAY + 3)
else:
self.log(f"达到最大重试次数,放弃: {url}")
return ""
return ""
def parse_list_page(self, html: str) -> list:
"""
解析列表页,提取视频基本信息
返回: [{title, detail_url, thumb_url, viewkey}, ...]
"""
videos = []
soup = BeautifulSoup(html, 'lxml')
# 只解析正常视频卡片。页面中还混有 col-lg-8 的异常大卡片,里面的标题、
# thumb、detail URL 会串到其它视频,不能作为入库来源。
video_cards = soup.select('div.col-xs-12.col-sm-4.col-md-3.col-lg-3')
seen_cards = set()
for card in video_cards:
link = card.find('a', href=re.compile(r'view_video\.php\?viewkey='))
if not link:
continue
href = link.get('href', '')
if not href:
continue
# 提取 viewkey
match = re.search(r'viewkey=([^&]+)', href)
if not match:
continue
viewkey = match.group(1)
detail_url = urljoin(BASE_URL, href)
# 提取标题
title = self._extract_title(link)
# 提取列表卡片来源 ID 和封面图 URL
thumb_url = ""
source_id = ""
overlay = link.find(id=re.compile(r'^playvthumb_\d+$'))
if overlay:
source_id = overlay.get('id', '').rsplit('_', 1)[-1]
img = link.find('img', class_=re.compile(r'img-responsive'))
if img:
thumb_url = img.get('src', '') or img.get('data-original', '')
if thumb_url:
thumb_url = urljoin(BASE_URL, thumb_url)
if not source_id and thumb_url:
source_id = self._extract_thumb_source_id(thumb_url)
card_key = source_id or detail_url
if card_key in seen_cards:
continue
seen_cards.add(card_key)
videos.append({
"title": title,
"detail_url": detail_url,
"thumb_url": thumb_url,
"viewkey": viewkey,
"source_id": source_id
})
return videos
def _extract_title(self, link) -> str:
"""
从视频链接标签中提取并清理标题
"""
# 优先从 span.video-title 获取 (已渲染的干净标题)
title_el = link.find('span', class_=re.compile(r'video-title'))
if title_el:
title = title_el.get_text(strip=True)
if title:
return html.unescape(title)
# 备用: 从 link 的 title 属性提取
title = link.get('title', '').strip()
if title:
return html.unescape(title)
# 最后手段: 从链接文本提取并清理前缀
text = link.get_text(separator=' ', strip=True)
# 去掉前缀: "HD" / "91" / 时间戳 "HH:MM:SS"
text = re.sub(r'^(HD\s+|91\s+)?\d{2}:\d{2}:\d{2}\s*', '', text)
text = re.sub(r'\s+', ' ', text).strip()
return html.unescape(text)[:120]
def parse_detail_page(self, html: str) -> dict:
"""
解析详情页,提取视频直链
返回: {"video_url": "...", "source_id": "...", "title": "..."} 或空字典
"""
result = {}
if not html:
return result
title = self._extract_detail_title(html)
if title:
result["title"] = title
# 方法1: 解码 strencode2 (主要方式, 页面通过 document.write 动态写入 video 标签)
# 格式: document.write(strencode2("%3c%73%6f..."));
strencode_match = re.search(r'strencode2\(["\']([^"\']+)["\']\)', html)
if strencode_match:
encoded = strencode_match.group(1)
try:
# strencode2 在JS中等价于 unescape / decodeURIComponent
decoded = unquote(encoded)
# 从解码后的 HTML 片段中提取 src
src_match = re.search(r"src=['\"]([^'\"]+)['\"]", decoded)
if src_match:
video_url = src_match.group(1)
# 规范化双斜杠 (如 https://host//path -> https://host/path)
video_url = re.sub(r'(https?://[^/]+)//+', r'\1/', video_url)
result["video_url"] = video_url
result["source_id"] = self._extract_source_id(video_url)
return result
except Exception as e:
self.log(f" 解码 strencode2 失败: {e}")
# 方法2: 通用正则匹配页面中的 mp4 链接 (备用, 过滤广告)
mp4_match = re.search(
r'https?://[^\s"\'<>]+\.mp4[^\s"\'<>]*',
html
)
if mp4_match:
url = mp4_match.group(0)
if 'kwai' not in url and 'ad-' not in url.lower():
result["video_url"] = url
result["source_id"] = self._extract_source_id(url)
return result
return result
def _extract_detail_title(self, html_text: str) -> str:
soup = BeautifulSoup(html_text, 'lxml')
title_el = soup.find('title')
if not title_el:
return ""
title = title_el.get_text(" ", strip=True)
title = re.sub(r'\s*-\s*91porn.*$', '', title, flags=re.IGNORECASE).strip()
return html.unescape(title)[:160]
def _extract_source_id(self, video_url: str) -> str:
path = urlparse(video_url or "").path
name = os.path.basename(path)
stem, ext = os.path.splitext(name)
if ext.lower() not in {".mp4", ".m4v", ".mov", ".webm", ".mkv", ".avi"}:
return ""
source_id = re.sub(r'[^0-9]+', '', stem)
if not source_id or source_id != stem:
return ""
return source_id
def _extract_thumb_source_id(self, thumb_url: str) -> str:
path = urlparse(thumb_url or "").path
match = re.search(r'/thumb/(\d+)\.[A-Za-z0-9]+$', path)
return match.group(1) if match else ""
def _thumb_url_for_source(self, thumb_url: str, source_id: str) -> str:
if not thumb_url or not source_id:
return thumb_url
parsed = urlparse(thumb_url)
match = re.search(r'/thumb/([^/?#]+)\.[A-Za-z0-9]+$', parsed.path)
if not match:
return thumb_url
current = match.group(1)
if current == source_id:
return thumb_url
path = re.sub(
r'/thumb/[^/?#]+\.[A-Za-z0-9]+$',
f'/thumb/{source_id}.jpg',
parsed.path,
)
return parsed._replace(path=path, query="", fragment="").geturl()
def crawl(self):
"""
主爬取流程。停止条件(任一满足即停):
- 达到 max_pages 配置
- 连续 max_empty_pages 页都没有视频
- target_new 模式下,已经累计处理 target_new 个新视频
"""
self.log("=" * 60)
self.log("91porn 视频爬虫启动")
self.log("=" * 60)
self.log(f"配置: 列表页延时 {MIN_PAGE_DELAY}-{MAX_PAGE_DELAY}s, 详情页延时 {MIN_DETAIL_DELAY}-{MAX_DETAIL_DELAY}s")
self.log(f"配置: 最大重试 {MAX_RETRIES} 次, 连续空页上限 {self.max_empty_pages}")
self.log(f"配置: 起始页 {self.start_page}, 最大爬取页数 {self.max_pages if self.max_pages else '不限'}")
if self.target_new:
self.log(f"配置: 目标新增视频数 {self.target_new}")
self.log(f"配置: 输出文件 {os.path.abspath(self.output_file)}")
if self.skip_viewkeys:
self.log(f"配置: 已跳过 {len(self.skip_viewkeys)} 个已知 viewkey")
self.log("")
page_num = self.start_page
consecutive_empty = 0
crawled_in_session = 0
while True:
if self.max_pages is not None and crawled_in_session >= self.max_pages:
self.log(f"达到配置的页数上限 {self.max_pages},停止")
break
if consecutive_empty >= self.max_empty_pages:
self.log(f"连续 {self.max_empty_pages} 页无结果,已达到末尾")
break
if self.target_new is not None and self.processed_videos >= self.target_new:
self.log(f"已累计 {self.processed_videos} 个新视频,达到目标 {self.target_new},停止")
break
if page_num == 1:
page_url = f"{BASE_URL}?category=top&viewtype=basic"
else:
page_url = f"{BASE_URL}?category=top&viewtype=basic&page={page_num}"
if crawled_in_session > 0:
self.log("")
self.random_sleep(MIN_PAGE_DELAY, MAX_PAGE_DELAY)
self.log(f"[页 {page_num}] 请求: {page_url}")
page_html = self.fetch_page(page_url, f"列表页 第{page_num}")
if not page_html:
self.log(f"[页 {page_num}] 获取失败,跳过")
consecutive_empty += 1
page_num += 1
crawled_in_session += 1
continue
page_videos = self.parse_list_page(page_html)
# 判断页面是否真的没有视频(而非全部已处理)
if not page_videos:
self.log(f"[页 {page_num}] 页面无视频,可能已到末尾")
consecutive_empty += 1
page_num += 1
crawled_in_session += 1
continue
consecutive_empty = 0
# 过滤已处理的 viewkey,只保留新视频
new_videos = [v for v in page_videos if v['viewkey'] not in self.skip_viewkeys]
skipped_on_page = len(page_videos) - len(new_videos)
if skipped_on_page > 0:
self.log(f"[页 {page_num}] 发现 {len(page_videos)} 个链接, 其中 {skipped_on_page} 个已处理, {len(new_videos)} 个新视频")
else:
self.log(f"[页 {page_num}] 发现 {len(new_videos)} 个视频")
if new_videos:
self._process_video_list(new_videos, referer=page_url)
self.pages_crawled += 1
page_num += 1
crawled_in_session += 1
self._save_results()
self._print_summary()
def _process_video_list(self, videos: list, referer: str = ""):
"""
处理一批视频列表,逐个获取详情页
"""
for idx, video in enumerate(videos, 1):
# target_new 模式下,凑够后立即停止,不再请求详情页
if self.target_new is not None and self.processed_videos >= self.target_new:
return
# 跳过已处理的 viewkey (断点续爬)
if video['viewkey'] in self.skip_viewkeys:
self.log(f" [SKIP] 已处理过: {video['viewkey']}")
self.skipped_videos += 1
continue
self.log(f" 处理视频 {idx}/{len(videos)}: {video['title'][:40]}...")
# 延时控制 (同一批次内第一个视频不延时)
if idx > 1:
self.random_sleep(MIN_DETAIL_DELAY, MAX_DETAIL_DELAY)
# 获取详情页
detail_html = self.fetch_page(video['detail_url'], f"详情页 viewkey={video['viewkey']}", referer=referer)
if not detail_html:
self.log(f" [FAIL] 详情页获取失败: {video['viewkey']}")
video["video_url"] = ""
self.results.append(video)
self.skip_viewkeys.add(video['viewkey'])
self.failed_videos += 1
continue
# 解析视频直链
detail_info = self.parse_detail_page(detail_html)
if detail_info.get("video_url"):
video["video_url"] = detail_info["video_url"]
if detail_info.get("title"):
video["title"] = detail_info["title"]
list_source_id = video.get("source_id", "")
detail_source_id = detail_info.get("source_id", "")
if list_source_id and detail_source_id and list_source_id != detail_source_id:
self.log(
f" [FAIL] 详情页视频源不匹配: list_source_id={list_source_id} "
f"detail_source_id={detail_source_id} viewkey={video['viewkey']}"
)
self.failed_videos += 1
self.skip_viewkeys.add(video['viewkey'])
continue
if not list_source_id and detail_source_id:
video["source_id"] = detail_source_id
if video.get("source_id"):
video["thumb_url"] = self._thumb_url_for_source(
video.get("thumb_url", ""),
video["source_id"],
)
if video["source_id"] in self.skip_viewkeys:
self.log(f" [SKIP] 已处理过 source_id: {video['source_id']}")
self.skipped_videos += 1
continue
self.results.append(video)
self.skip_viewkeys.add(video['viewkey'])
if video.get("source_id"):
self.skip_viewkeys.add(video["source_id"])
self.processed_videos += 1
self.log(f" [OK] 成功提取视频直链")
# 流式:立刻把这条 entry 交给 Go 端开始下载,不等本批余下视频
self.emit_stream_video(video)
else:
self.log(f" [FAIL] 未找到视频直链: {video['viewkey']}")
video["video_url"] = ""
self.results.append(video)
self.skip_viewkeys.add(video['viewkey'])
self.failed_videos += 1
def _save_results(self):
"""
保存结果到JSON文件
"""
output_data = {
"crawl_time": datetime.now().isoformat(),
"source_url": BASE_URL,
"pages_crawled": self.pages_crawled,
"total_videos": len(self.results),
"successful": self.processed_videos,
"skipped": self.skipped_videos,
"failed": self.failed_videos,
"videos": self.results
}
try:
# 保证父目录存在;写入临时文件后原子 rename,避免读到半截 JSON
out_path = self.output_file
parent = os.path.dirname(os.path.abspath(out_path))
if parent:
os.makedirs(parent, exist_ok=True)
tmp_path = out_path + ".part"
with open(tmp_path, 'w', encoding='utf-8') as f:
json.dump(output_data, f, ensure_ascii=False, indent=2)
os.replace(tmp_path, out_path)
self.log(f"结果已保存到: {os.path.abspath(out_path)}")
except Exception as e:
self.log(f"保存文件失败: {e}")
# 尝试输出到控制台作为备份
backup_out = sys.stderr if self.stream_output else sys.stdout
print("\n--- 备份输出 ---", file=backup_out, flush=True)
print(json.dumps(output_data, ensure_ascii=False, indent=2), file=backup_out, flush=True)
def _print_summary(self):
"""
打印爬取摘要
"""
self.log("")
self.log("=" * 60)
self.log("爬取完成!")
self.log("=" * 60)
self.log(f"爬取页数: {self.pages_crawled}")
self.log(f"总视频数: {len(self.results)}")
self.log(f"成功提取直链: {self.processed_videos}")
self.log(f"跳过(已处理): {self.skipped_videos}")
self.log(f"失败/缺失直链: {self.failed_videos}")
self.log(f"输出文件: {os.path.abspath(self.output_file)}")
self.log("=" * 60)
def print_help():
print("""
================================================
91porn 视频爬虫 v1.0
================================================
本脚本将爬取 91porn "本月最热" 分类下的所有视频信息:
- 视频名称
- 封面图直链
- 视频直链 (MP4)
依赖安装:
pip install requests beautifulsoup4 lxml PySocks
使用方法:
python spider_91porn.py
配置说明 (编辑脚本内 "配置区域"):
MIN_PAGE_DELAY / MAX_PAGE_DELAY : 列表页请求间隔 (默认 3-6 秒)
MIN_DETAIL_DELAY / MAX_DETAIL_DELAY : 详情页请求间隔 (默认 2-5 秒)
MAX_PAGES : 限制最大爬取页数 (None=不限, 如 5=只爬前5页)
OUTPUT_FILE : 输出文件名 (默认 91porn_videos.json)
按 Ctrl+C 可随时中断并保存已爬取的数据
注意:
1. 视频直链包含时效性token,会过期,需定期重新爬取
2. 脚本已内置随机延时,请勿移除,避免对服务器造成压力
3. 如遇到Cloudflare拦截,需要先通过浏览器获取Cookie
4. 本脚本仅供学习交流,请遵守当地法律法规
================================================
""")
def run_job(job_path: str):
"""Run as a crawler.v1 script plugin.
The Go host passes a job JSON file and expects stdout JSONL events. Logs go
to stderr so stdout stays machine-readable.
"""
with open(job_path, "r", encoding="utf-8") as f:
job = json.load(f)
if job.get("protocol") != CRAWLER_PROTOCOL:
raise ValueError(f"unsupported crawler protocol: {job.get('protocol')!r}")
if job.get("mode") not in ("", None, "crawl"):
raise ValueError(f"unsupported crawler mode: {job.get('mode')!r}")
try:
target_new = int(job.get("target_new") or 15)
except (TypeError, ValueError):
target_new = 15
if target_new <= 0:
target_new = 15
seen_file = job.get("seen_source_ids_file") or ""
output_dir = job.get("output_dir") or os.getcwd()
run_id = job.get("run_id") or datetime.utcnow().strftime("%Y%m%dT%H%M%SZ")
os.makedirs(output_dir, exist_ok=True)
output_file = os.path.join(output_dir, f"spider91-{run_id}.json")
network = job.get("network") if isinstance(job.get("network"), dict) else {}
proxy_url = str(network.get("proxy_url") or "").strip()
if proxy_url:
os.environ["HTTP_PROXY"] = proxy_url
os.environ["HTTPS_PROXY"] = proxy_url
os.environ["http_proxy"] = proxy_url
os.environ["https_proxy"] = proxy_url
os.environ["NO_PROXY"] = ""
os.environ["no_proxy"] = ""
seen_viewkeys = []
if seen_file:
try:
with open(seen_file, "r", encoding="utf-8") as f:
for line in f:
line = line.strip()
if line:
seen_viewkeys.append(line)
except FileNotFoundError:
print(f"警告: seen_source_ids_file 不存在: {seen_file}", file=sys.stderr, flush=True)
except Exception as e:
print(f"警告: 读取 seen_source_ids_file 失败: {e}", file=sys.stderr, flush=True)
prefer_ipv4_for_plain_socks5_proxy()
spider = Porn91Spider(
output_file=output_file,
start_page=1,
max_pages=None,
resume=False,
quiet=True,
target_new=target_new,
seen_viewkeys=seen_viewkeys,
stream_output=True,
stream_protocol="crawler.v1",
)
try:
spider.crawl()
done = {
"type": "done",
"stats": {
"emitted": spider.processed_videos,
"failed": spider.failed_videos,
"skipped": spider.skipped_videos,
},
}
write_jsonl(done)
except KeyboardInterrupt:
spider.log("\n用户中断,正在保存已爬取的数据...")
spider._save_results()
raise
def main():
if len(sys.argv) > 1 and sys.argv[1] in ('-h', '--help', 'help'):
print_help()
return
parser = argparse.ArgumentParser(
prog="spider_91porn.py",
description="91porn 视频元数据爬虫",
add_help=False, # 让 -h/--help 走 print_help() 中文版本
)
parser.add_argument("--page", type=int, default=None,
help="只爬指定页(单页模式,配合 --output 用于定时任务)")
parser.add_argument("--output", type=str, default=None,
help="输出 JSON 路径,覆盖默认 OUTPUT_FILE")
parser.add_argument("--max-pages", type=int, default=None,
help="单页模式下,从 --page 起最多再爬几页(默认 1)")
parser.add_argument("--no-resume", action="store_true",
help="禁用断点续爬(单页模式默认禁用)")
parser.add_argument("--quiet", action="store_true",
help="压缩日志,每条视频只输出关键事件")
parser.add_argument("--target-new", type=int, default=None,
help="目标新增模式:从 page 1 起翻页直到累计处理这么多新源视频后停止(backend 凌晨任务用)")
parser.add_argument("--seen-viewkeys-file", type=str, default=None,
help="文件路径,每行一个已处理过的 viewkey 或 mp4 源 ID;脚本会跳过这些视频")
parser.add_argument("--stream-output", action="store_true",
help="流式模式:每解析一条视频直链就立即把它作为一行 JSON 写到 stdout 并 flush"
"日志改走 stderr。配合 backend 边读边下载使用。")
parser.add_argument("--job", type=str, default=None,
help="crawler.v1 job JSON 路径;作为通用脚本爬虫运行。")
args, _ = parser.parse_known_args()
if args.job:
run_job(args.job)
return
cli_out = sys.stderr if args.stream_output else sys.stdout
prefer_ipv4_for_plain_socks5_proxy()
print("""
================================================
91porn 视频爬虫启动中...
================================================
按 Ctrl+C 可随时中断并保存进度
""", file=cli_out)
# 加载已知 ID(来自 backend 的 catalog 已入库列表;兼容旧参数名)
seen_viewkeys = []
if args.seen_viewkeys_file:
try:
with open(args.seen_viewkeys_file, 'r', encoding='utf-8') as f:
for line in f:
line = line.strip()
if line:
seen_viewkeys.append(line)
except FileNotFoundError:
print(f"警告: --seen-viewkeys-file 不存在: {args.seen_viewkeys_file}", file=cli_out)
except Exception as e:
print(f"警告: 读取 --seen-viewkeys-file 失败: {e}", file=cli_out)
# 决定运行模式
if args.target_new is not None:
# 凑够 N 个新视频模式:从 page 1 起翻页,直到累计 target_new 个新视频
spider = Porn91Spider(
output_file=args.output,
start_page=1,
max_pages=None,
resume=False, # 凑够 N 模式靠 seen_viewkeys 去重,不读 OUTPUT_FILE
quiet=args.quiet,
target_new=args.target_new,
seen_viewkeys=seen_viewkeys,
stream_output=args.stream_output,
)
elif args.page is not None:
# 单页模式(保留作手动调试用):start_page=N, max_pages=1
start_page = max(1, args.page)
max_pages = args.max_pages if args.max_pages and args.max_pages > 0 else 1
spider = Porn91Spider(
output_file=args.output,
start_page=start_page,
max_pages=max_pages,
resume=False,
quiet=args.quiet,
seen_viewkeys=seen_viewkeys,
stream_output=args.stream_output,
)
else:
# 全量模式(向后兼容):从 page 1 起爬到末尾
spider = Porn91Spider(
output_file=args.output,
resume=False if args.no_resume else None,
quiet=args.quiet,
seen_viewkeys=seen_viewkeys,
stream_output=args.stream_output,
)
try:
spider.crawl()
except KeyboardInterrupt:
spider.log("\n用户中断,正在保存已爬取的数据...")
spider._save_results()
spider._print_summary()
sys.exit(0)
except Exception as e:
spider.log(f"发生未预料的错误: {e}")
import traceback
traceback.print_exc()
spider._save_results()
raise
if __name__ == "__main__":
main()
+11 -3
View File
@@ -20,12 +20,11 @@
## 功能特性
- **多后端支持** — 兼容 115 云盘、PikPak 云盘、123网盘、联通网盘、OneDrive、Google Drive 和本地存储
- **低带宽播放** — 115 云盘、PikPak 云盘、123网盘、联通网盘、OneDrive 支持302模式,在线播放视频时,不占用服务器带宽,播放体验不受服务器带宽影响;Google Drive 不支持302模式,走服务器中转,观看体验会受服务器带宽影响
- **多后端支持** — 兼容 115 云盘、PikPak 云盘、123网盘、联通网盘、光鸭网盘、OneDrive、Google Drive 和本地存储
- **低带宽播放** — 115 云盘、PikPak 云盘、123网盘、联通网盘、光鸭网盘、OneDrive 支持302模式,在线播放视频时,不占用服务器带宽,播放体验不受服务器带宽影响;Google Drive 不支持302模式,走服务器中转,观看体验会受服务器带宽影响
- **封面 & 预览片段** — 自动为每个视频生成封面图和预览片段,首页快速选片
- **爬虫脚本** — 项目支持导入自定义脚本,但是有一些规范,具体可以参考 [SpiderFor91](https://github.com/Just-Spider/SpiderFor91),项目不再内置任何爬虫脚本
- **短视频模式** — 一键切换抖音风格,沉浸刷片
---
## 预览图
@@ -82,6 +81,14 @@ sudo bash install.sh
> `video-site-91` 为等效别名,两者可互换使用。
**已部署用户升级:**
```bash
91 update
```
升级会保留现有 `config.yaml`、数据库、封面、预览、上传文件和爬虫数据。脚本会自动安装或检查 `ffmpeg` / `ffprobe` 等运行依赖,并在新版本启动失败时回滚到升级前文件。
**自定义端口:**
```bash
@@ -153,6 +160,7 @@ docker compose up -d # 更新并重启
```
> 所有配置、数据库、封面、预览及上传文件均保存在 `./data/` 目录下。
> 从旧版本升级 Docker 部署时,执行 `docker compose pull && docker compose up -d` 即可;`./data/` 不会被镜像更新覆盖。
---
+5 -3
View File
@@ -2,7 +2,7 @@
视频聚合站的 Go 后端。提供三件事:
1. 多家网盘统一抽象(夸克 / 115 / PikPak / 联通网盘 / OneDrive / Google Drive / 本地存储)
1. 多家网盘统一抽象(夸克 / 115 / PikPak / 联通网盘 / 光鸭网盘 / OneDrive / Google Drive / 本地存储)
2. 视频元数据目录(SQLite)+ 扫描 + 预览视频预生成
3. REST API(前台)+ 管理后台 + 直链代理
4. 标签池、视频隐藏、按网盘统计和详情页来源网盘类型展示能力
@@ -20,6 +20,7 @@ internal/
p115/ 115(壳子 + SheltonZhu/115driver
pikpak/ PikPak(自己实现,参考 OpenList pikpak
wopan/ 联通网盘(壳子 + OpenListTeam/wopan-sdk-go
guangyapan/ 光鸭网盘(参考 AList GuangYaPan
onedrive/ OneDriveOpenList 在线续期 + Microsoft Graph 文件接口)
googledrive/ Google DriveOpenList 在线续期 + Google Drive API;播放走后端代理)
localstorage/ 本地目录扫描(服务器已有视频目录)
@@ -108,6 +109,7 @@ go run ./cmd/server 后端 9192
| p115 | `cookie`(形如 `UID=...; CID=...; SEID=...; KID=...` |
| pikpak | `username`、`password`(token、验证码和设备 ID 由服务端自动处理并保存) |
| wopan | `access_token`、`refresh_token`,可选 `family_id` |
| guangyapan | 推荐后台扫码登录自动写入 `access_token`、`refresh_token`;也可手工填写 token;可选 `root_path` |
| onedrive | `refresh_token` |
| googledrive | 默认只需 `refresh_token`;自建 OAuth 客户端模式还需 `use_online_api=false`、`client_id`、`client_secret` |
| localstorage | `path`(服务器上的已有视频目录,如 `/mnt/videos` |
@@ -154,9 +156,9 @@ Google Drive 默认按 OpenList 在线 API 调用 `https://api.oplist.org/google
## 管理能力
- `/admin/drives`:新增、编辑、删除网盘,触发扫描。
- `/admin/videos`:按网盘筛选视频,每页 100 条分页,查看各网盘预览视频统计,编辑标题/作者/分类/标签,单条或全量重生预览视频。
- `/admin/videos`:按网盘筛选视频,每页 100 条分页,查看各网盘预览视频统计,编辑标题/作者/分类/标签,单条或全量重生预览视频;拉黑视频页可查看被删除或被隐藏的视频,并支持移出黑名单后在下次扫盘重新入库
- `/admin/tags`:新增标签并用内置规则自动匹配已有视频;删除非系统标签时会从所有视频上同步移除该标签。
- 播放页视频信息会展示来源网盘类型;同时提供“不再展示”,点击后会把视频标记为全局隐藏。隐藏视频不会再出现在首页、列表、搜索、相关推荐和详情接口中。目前没有管理后台恢复入口,如需恢复可把数据库里对应视频的 `hidden` 字段改回 `0`
- 播放页视频信息会展示来源网盘类型,并提供删除入口。被删除或被隐藏视频会进入黑名单,不会再出现在首页、列表、搜索和详情接口中;在后台移出黑名单后,会在下次扫盘时重新发现并入库
## 预览视频生成
+506 -47
View File
@@ -26,6 +26,7 @@ import (
"github.com/video-site/backend/internal/config"
"github.com/video-site/backend/internal/drives"
"github.com/video-site/backend/internal/drives/googledrive"
"github.com/video-site/backend/internal/drives/guangyapan"
"github.com/video-site/backend/internal/drives/localstorage"
"github.com/video-site/backend/internal/drives/localupload"
"github.com/video-site/backend/internal/drives/onedrive"
@@ -43,6 +44,7 @@ import (
"github.com/video-site/backend/internal/proxy"
"github.com/video-site/backend/internal/scanner"
"github.com/video-site/backend/internal/spider91migrate"
"github.com/video-site/backend/internal/transcode"
)
const fingerprintReconcileInterval = time.Minute
@@ -130,6 +132,13 @@ func main() {
OnVideoUploaded: func(v *catalog.Video) {
app.enqueueUploadedVideo(ctx, v)
},
// 前台「不再展示」走拉黑逻辑:删记录 + 删本地封面/预览 + 写墓碑,
// 保留网盘源文件(deleteSource=false)。下次扫盘不再入库;如需恢复,
// 在后台「拉黑视频」移出黑名单即可,扫盘时会重新添加回来。
OnHideVideo: func(reqCtx context.Context, videoID string) error {
_, err := app.deleteVideo(reqCtx, videoID, false)
return err
},
GetTheme: func() string { return app.Theme() },
}
@@ -169,6 +178,14 @@ func main() {
return err
}
app.scheduleCrawlerUploadMigration(ctx, driveID)
// 本地存储开启 .strm 越root后,之前因 strm 指向目录外而失败的封面/
// 预览/指纹应自动重试,省得用户再手动点三个"重试失败"按钮。
if d.Kind == localstorage.Kind &&
parseBoolDefault(strings.TrimSpace(d.Credentials["strm_allow_outside_root"]), false) {
go app.regenFailedThumbnails(ctx, driveID)
go app.regenFailedPreviews(ctx, driveID)
go app.regenFailedFingerprints(ctx, driveID)
}
return nil
},
OnDriveDeleteCleanup: func(cleanupCtx context.Context, driveID string) (int, error) {
@@ -197,6 +214,9 @@ func main() {
}
return app.scheduleScan(ctx, driveID)
},
OnCrawlerUploadRequested: func(driveID string) (bool, string) {
return app.scheduleManualCrawlerUploadMigration(ctx, driveID)
},
OnStopDriveTasks: func(driveID string) bool {
return app.stopDriveTasks(ctx, driveID)
},
@@ -218,12 +238,21 @@ func main() {
OnRegenFailedFingerprints: func(driveID string) {
go app.regenFailedFingerprints(ctx, driveID)
},
OnStartDriveTranscode: func(driveID string) (bool, string) {
return app.startDriveTranscode(ctx, driveID)
},
OnStopDriveTranscode: func(driveID string) bool {
return app.stopDriveTranscode(driveID)
},
OnDeleteVideo: func(reqCtx context.Context, videoID string, deleteSource bool) (api.DeleteVideoResult, error) {
return app.deleteVideo(reqCtx, videoID, deleteSource)
},
GetDriveGenerationStatuses: func() map[string]api.DriveGenerationStatuses {
return app.driveGenerationStatuses()
},
GetPreviewGenerationVideoIDs: func() map[string]bool {
return app.previewGenerationVideoIDs()
},
OnTeaserEnabledChanged: func(driveID string, enabled bool) {
// 从关到开时立刻补扫该盘 pending 预览视频,行为对齐旧的"全局开关从关到开"。
// 关闭分支不需要做事 —— 入队前会重新查 catalog,新的 enqueue 自然停。
@@ -297,6 +326,7 @@ func main() {
}
}()
go app.attachExistingDrives(ctx)
go app.migrateHiddenVideosToTombstone(ctx)
// 等待退出信号
sigs := make(chan os.Signal, 1)
@@ -329,13 +359,13 @@ type App struct {
// 串行化可以避免启动后台挂载和手动扫盘按需挂载同一个 drive 时重复创建 worker。
driveAttachMu sync.Mutex
// 全站主题("dark" | "pink"),从 DB 读
// 全站主题("dark" | "pink" | "sky"),从 DB 读
theme string
// 显式指定的 spider91 上传目标 drive ID。
// 空字符串表示本地保存不上传,不再自动挑选 pikpak/p115/p123/onedrive/wopan drive。
// 空字符串表示本地保存不上传,不再自动挑选 pikpak/p115/p123/onedrive/wopan/guangyapan drive。
spider91UploadDriveID string
// spider91Migrator 把 spider91 视频上传到目标 drivePikPak、115、123、OneDrive、Google Drive 或联通网盘)。
// spider91Migrator 把 spider91 视频上传到目标 drivePikPak、115、123、OneDrive、Google Drive、联通网盘或光鸭网盘)。
spider91Migrator spider91MigrationRunner
// nightlyRunner 是凌晨流水线调度器:每天 cron_hour 串行跑扫盘 → 91 爬虫 → 迁移。
@@ -368,11 +398,19 @@ type App struct {
// uploadProgress 跟踪脚本爬虫迁移到云盘时的实时上传状态。
uploadProgressMu sync.Mutex
uploadProgress map[string]driveUploadProgress
// transcodeMu 保护 transcodeWorkers / transcodeCancels。
// 浏览器兼容性转码每盘最多一个任务,且只能由管理员手动开启
// (不随扫盘/夜间流水线自动运行),手动停止或处理完即从 map 清除。
transcodeMu sync.Mutex
transcodeWorkers map[string]*transcode.Worker
transcodeCancels map[string]context.CancelFunc
}
type driveScanProgress struct {
Scanned int
Added int
Scanned int
Added int
CooldownUntil time.Time
}
type driveUploadProgress struct {
@@ -421,7 +459,7 @@ func (a *App) Theme() string {
// SetTheme 切换并持久化主题;未知值会返回错误。
func (a *App) SetTheme(ctx context.Context, theme string) error {
if theme != "dark" && theme != "pink" {
if theme != "dark" && theme != "pink" && theme != "sky" {
return fmt.Errorf("unsupported theme %q", theme)
}
a.mu.Lock()
@@ -440,7 +478,7 @@ func (a *App) loadTheme(ctx context.Context) {
a.mu.Unlock()
return
}
if v != "pink" && v != "dark" {
if v != "pink" && v != "dark" && v != "sky" {
v = "dark"
}
a.mu.Lock()
@@ -449,7 +487,7 @@ func (a *App) loadTheme(ctx context.Context) {
}
// Spider91UploadDriveID 返回当前配置的 spider91 上传目标 drive ID。
// 空字符串表示本地保存不上传;只有管理员显式选择 pikpak/p115/p123/onedrive/googledrive/wopan drive 时才迁移上传。
// 空字符串表示本地保存不上传;只有管理员显式选择 pikpak/p115/p123/onedrive/googledrive/wopan/guangyapan drive 时才迁移上传。
func (a *App) Spider91UploadDriveID() string {
a.mu.Lock()
explicit := a.spider91UploadDriveID
@@ -466,7 +504,7 @@ func (a *App) Spider91UploadDriveID() string {
// SetSpider91UploadDriveID 设置 spider91 上传目标 drive ID 并持久化。
// 接受空字符串(本地保存不上传)。
// 设置一个不存在或 kind 不是 pikpak / p115 / p123 / onedrive / googledrive / wopan 的 drive 会返回错误。
// 设置一个不存在或 kind 不是 pikpak / p115 / p123 / onedrive / googledrive / wopan / guangyapan 的 drive 会返回错误。
func (a *App) SetSpider91UploadDriveID(ctx context.Context, driveID string) error {
driveID = strings.TrimSpace(driveID)
if driveID != "" {
@@ -475,7 +513,7 @@ func (a *App) SetSpider91UploadDriveID(ctx context.Context, driveID string) erro
return fmt.Errorf("drive %q not found", driveID)
}
if !isSpider91UploadKind(d.Kind()) {
return fmt.Errorf("drive %q kind=%s, only pikpak, p115, p123, onedrive, googledrive or wopan can be spider91 upload target", driveID, d.Kind())
return fmt.Errorf("drive %q kind=%s, only pikpak, p115, p123, onedrive, googledrive, wopan or guangyapan can be spider91 upload target", driveID, d.Kind())
}
}
a.mu.Lock()
@@ -508,7 +546,7 @@ func formatOptionalRFC3339(t time.Time) string {
// isSpider91UploadKind 是 spider91 迁移目标盘的 allowlist。
// 与 spider91migrate.adaptUploadTarget 的支持范围保持一致。
func isSpider91UploadKind(kind string) bool {
return kind == "pikpak" || kind == "p115" || kind == "p123" || kind == "onedrive" || kind == "googledrive" || kind == "wopan"
return kind == "pikpak" || kind == "p115" || kind == "p123" || kind == "onedrive" || kind == "googledrive" || kind == "wopan" || kind == guangyapan.Kind
}
// loadSpider91UploadDriveID 从 DB 读上传目标 drive ID 设置;不存在时使用空串。
@@ -557,18 +595,33 @@ func (a *App) driveGenerationStatuses() map[string]api.DriveGenerationStatuses {
}
a.mu.Unlock()
out := make(map[string]api.DriveGenerationStatuses, len(scanningDrives)+len(previewWorkers)+len(thumbWorkers)+len(fingerprintWorkers)+len(uploadProgresses))
a.transcodeMu.Lock()
transcodeWorkers := make(map[string]*transcode.Worker, len(a.transcodeWorkers))
for id, worker := range a.transcodeWorkers {
transcodeWorkers[id] = worker
}
a.transcodeMu.Unlock()
out := make(map[string]api.DriveGenerationStatuses, len(scanningDrives)+len(previewWorkers)+len(thumbWorkers)+len(fingerprintWorkers)+len(uploadProgresses)+len(transcodeWorkers))
now := time.Now()
for id, running := range scanningDrives {
if !running {
continue
}
progress := scanProgresses[id]
state := "scanning"
if progress.CooldownUntil.After(now) {
state = "cooling"
}
status := out[id]
status.Scan = api.GenerationStatus{
State: "scanning",
State: state,
ScannedCount: progress.Scanned,
AddedCount: progress.Added,
}
if !progress.CooldownUntil.IsZero() {
status.Scan.CooldownUntil = progress.CooldownUntil.Format(time.RFC3339)
}
out[id] = status
}
for id, worker := range previewWorkers {
@@ -601,6 +654,28 @@ func (a *App) driveGenerationStatuses() map[string]api.DriveGenerationStatuses {
}
out[id] = status
}
for id, worker := range transcodeWorkers {
status := out[id]
status.Transcode = generationStatusFromTranscode(worker.Status())
out[id] = status
}
return out
}
func (a *App) previewGenerationVideoIDs() map[string]bool {
a.mu.Lock()
previewWorkers := make([]*preview.Worker, 0, len(a.workers))
for _, worker := range a.workers {
previewWorkers = append(previewWorkers, worker)
}
a.mu.Unlock()
out := make(map[string]bool)
for _, worker := range previewWorkers {
for _, id := range worker.ActiveVideoIDs() {
out[id] = true
}
}
return out
}
@@ -687,6 +762,126 @@ func generationStatusFromFingerprint(status fingerprint.TaskStatus) api.Generati
return out
}
func generationStatusFromTranscode(status transcode.TaskStatus) api.GenerationStatus {
state := status.State
if state == "" {
state = "idle"
}
return api.GenerationStatus{
State: state,
CurrentTitle: status.CurrentTitle,
QueueLength: status.QueueLength,
DoneCount: status.DoneCount,
TotalCount: status.TotalCount,
}
}
// transcodeWorkDir 返回转码用的本地临时目录(下载原片 / 写产物),与
// localUploadDir 一样挂在数据目录下,避免 /tmp 空间不足。
func (a *App) transcodeWorkDir() string {
return filepath.Join(filepath.Dir(a.cfg.Storage.LocalPreviewDir), "transcode-tmp")
}
// startDriveTranscode 手动开启某盘的浏览器兼容性转码。
// 转码从不自动运行:扫盘、夜间流水线都不会触发,这里是唯一入口。
// 任务跑完候选列表后自然结束;中途可用 stopDriveTranscode / 停止所有任务中断。
func (a *App) startDriveTranscode(ctx context.Context, driveID string) (bool, string) {
driveID = strings.TrimSpace(driveID)
if driveID == "" {
return false, "缺少存储 ID"
}
drv, ok := a.registry.Get(driveID)
if !ok {
return false, "存储未挂载或不可用"
}
switch drv.Kind() {
case spider91.Kind, scriptcrawler.Kind:
return false, "爬虫存储不支持转码"
}
workDir := a.transcodeWorkDir()
if err := os.MkdirAll(workDir, 0o755); err != nil {
return false, "创建转码临时目录失败: " + err.Error()
}
a.transcodeMu.Lock()
if a.transcodeWorkers == nil {
a.transcodeWorkers = make(map[string]*transcode.Worker)
a.transcodeCancels = make(map[string]context.CancelFunc)
}
if existing := a.transcodeWorkers[driveID]; existing != nil {
a.transcodeMu.Unlock()
return false, "该存储的转码任务已在运行"
}
worker := transcode.NewWorker(transcode.Config{
FFmpegPath: a.cfg.Preview.FFmpegPath,
FFprobePath: a.cfg.Preview.FFprobePath,
WorkDir: workDir,
}, a.cat, drv)
taskCtx, done := a.registerDriveTaskContext(ctx, driveID)
runCtx, cancel := context.WithCancel(taskCtx)
a.transcodeWorkers[driveID] = worker
a.transcodeCancels[driveID] = cancel
a.transcodeMu.Unlock()
go func() {
defer func() {
cancel()
done()
a.transcodeMu.Lock()
if a.transcodeWorkers[driveID] == worker {
delete(a.transcodeWorkers, driveID)
delete(a.transcodeCancels, driveID)
}
a.transcodeMu.Unlock()
}()
candidates, err := a.cat.ListTranscodeCandidates(runCtx, driveID, 0)
if err != nil {
log.Printf("[transcode] list candidates drive=%s: %v", driveID, err)
return
}
if len(candidates) == 0 {
log.Printf("[transcode] drive=%s no candidates", driveID)
return
}
log.Printf("[transcode] drive=%s start, %d candidates", driveID, len(candidates))
worker.Run(runCtx, candidates)
}()
return true, ""
}
// stopAllDriveTranscodes 停掉所有盘的转码任务,返回被停的 driveID 列表。
func (a *App) stopAllDriveTranscodes() []string {
a.transcodeMu.Lock()
cancels := a.transcodeCancels
a.transcodeCancels = nil
a.transcodeWorkers = nil
a.transcodeMu.Unlock()
ids := make([]string, 0, len(cancels))
for id, cancel := range cancels {
if cancel != nil {
cancel()
}
ids = append(ids, id)
}
return ids
}
// stopDriveTranscode 手动停止某盘的转码任务。返回是否有任务被停。
func (a *App) stopDriveTranscode(driveID string) bool {
driveID = strings.TrimSpace(driveID)
a.transcodeMu.Lock()
cancel := a.transcodeCancels[driveID]
delete(a.transcodeCancels, driveID)
delete(a.transcodeWorkers, driveID)
a.transcodeMu.Unlock()
if cancel == nil {
return false
}
cancel()
log.Printf("[transcode] stop drive=%s", driveID)
return true
}
func (a *App) attachDrive(ctx context.Context, d *catalog.Drive) error {
a.driveAttachMu.Lock()
defer a.driveAttachMu.Unlock()
@@ -746,18 +941,20 @@ func (a *App) attachDriveUnlocked(ctx context.Context, d *catalog.Drive) error {
})
case "p115":
drv = p115.New(p115.Config{
ID: d.ID,
Cookie: d.Credentials["cookie"],
RootID: d.RootID,
ID: d.ID,
Cookie: d.Credentials["cookie"],
RootID: d.RootID,
UploadTempDir: a.uploadWorkDir("p115"),
})
case p123.Kind:
drv = p123.New(p123.Config{
ID: d.ID,
Username: d.Credentials["username"],
Password: d.Credentials["password"],
AccessToken: d.Credentials["access_token"],
Platform: d.Credentials["platform"],
RootID: d.RootID,
ID: d.ID,
Username: d.Credentials["username"],
Password: d.Credentials["password"],
AccessToken: d.Credentials["access_token"],
Platform: d.Credentials["platform"],
RootID: d.RootID,
UploadTempDir: a.uploadWorkDir(p123.Kind),
OnTokenUpdate: func(access string) {
if d.Credentials == nil {
d.Credentials = make(map[string]string)
@@ -778,6 +975,7 @@ func (a *App) attachDriveUnlocked(ctx context.Context, d *catalog.Drive) error {
DeviceID: d.Credentials["device_id"],
RootID: d.RootID,
DisableMediaLink: pikpak.ParseBoolDefault(d.Credentials["disable_media_link"], true),
UploadTempDir: a.uploadWorkDir("pikpak"),
OnTokenUpdate: func(access, refresh, captcha, deviceID string) {
d.Credentials["access_token"] = access
d.Credentials["refresh_token"] = refresh
@@ -788,17 +986,45 @@ func (a *App) attachDriveUnlocked(ctx context.Context, d *catalog.Drive) error {
})
case "wopan":
drv = wopan.New(wopan.Config{
ID: d.ID,
AccessToken: d.Credentials["access_token"],
RefreshToken: d.Credentials["refresh_token"],
FamilyID: d.Credentials["family_id"],
RootID: d.RootID,
ID: d.ID,
AccessToken: d.Credentials["access_token"],
RefreshToken: d.Credentials["refresh_token"],
FamilyID: d.Credentials["family_id"],
RootID: d.RootID,
UploadTempDir: a.uploadWorkDir("wopan"),
OnTokenUpdate: func(access, refresh string) {
d.Credentials["access_token"] = access
d.Credentials["refresh_token"] = refresh
_ = a.cat.UpsertDrive(ctx, d)
},
})
case guangyapan.Kind:
drv = guangyapan.New(guangyapan.Config{
ID: d.ID,
RootID: d.RootID,
RootPath: d.Credentials["root_path"],
PhoneNumber: d.Credentials["phone_number"],
CaptchaToken: d.Credentials["captcha_token"],
SendCode: parseBoolDefault(strings.TrimSpace(d.Credentials["send_code"]), false),
VerifyCode: d.Credentials["verify_code"],
VerificationID: d.Credentials["verification_id"],
AccessToken: d.Credentials["access_token"],
RefreshToken: d.Credentials["refresh_token"],
ClientID: d.Credentials["client_id"],
DeviceID: d.Credentials["device_id"],
PageSize: parseIntDefault(strings.TrimSpace(d.Credentials["page_size"]), 100),
OrderBy: parseIntDefault(strings.TrimSpace(d.Credentials["order_by"]), 3),
SortType: parseIntDefault(strings.TrimSpace(d.Credentials["sort_type"]), 1),
OnCredentialsUpdate: func(updated map[string]string) {
if d.Credentials == nil {
d.Credentials = make(map[string]string)
}
for k, v := range updated {
d.Credentials[k] = v
}
_ = a.cat.UpsertDrive(ctx, d)
},
})
case "onedrive":
drv = onedrive.New(onedrive.Config{
ID: d.ID,
@@ -841,8 +1067,9 @@ func (a *App) attachDriveUnlocked(ctx context.Context, d *catalog.Drive) error {
})
case localstorage.Kind:
drv = localstorage.New(localstorage.Config{
ID: d.ID,
RootPath: d.Credentials["path"],
ID: d.ID,
RootPath: d.Credentials["path"],
STRMAllowOutsideRoot: parseBoolDefault(strings.TrimSpace(d.Credentials["strm_allow_outside_root"]), false),
})
case scriptcrawler.Kind:
drv = scriptcrawler.New(scriptcrawler.Config{
@@ -918,7 +1145,7 @@ func generationCooldownForDrive(drv drives.Drive) time.Duration {
return 0
}
switch strings.ToLower(drv.Kind()) {
case "wopan":
case "wopan", "guangyapan":
return 10 * time.Minute
}
return 0
@@ -938,13 +1165,24 @@ func (a *App) localUploadDir() string {
return filepath.Join(filepath.Dir(a.cfg.Storage.LocalPreviewDir), "uploads")
}
func (a *App) uploadWorkDir(kind string) string {
if a == nil || a.cfg == nil || strings.TrimSpace(a.cfg.Storage.LocalPreviewDir) == "" {
return ""
}
kind = strings.Trim(strings.ToLower(strings.TrimSpace(kind)), string(filepath.Separator))
if kind == "" {
kind = "generic"
}
return filepath.Join(filepath.Dir(a.cfg.Storage.LocalPreviewDir), "upload-tmp", kind)
}
func fingerprintConfigForDrive(drv drives.Drive) fingerprint.Config {
cfg := fingerprint.Config{RateLimitCooldown: 5 * time.Minute}
if drv == nil {
return cfg
}
switch strings.ToLower(drv.Kind()) {
case "p115", "p123", "onedrive", "wopan":
case "p115", "p123", "onedrive", "wopan", "guangyapan":
cfg.RateLimitCooldown = 10 * time.Minute
case "pikpak":
cfg.RateLimitCooldown = 5 * time.Minute
@@ -1016,6 +1254,7 @@ func (a *App) attachScriptCrawler(d *catalog.Drive, drv *scriptcrawler.Driver) {
CommonThumbDir: a.commonThumbsDir(),
ProxyURL: proxyURL,
ConfigJSON: configJSON,
DisablePreview: !d.TeaserEnabled,
OnProgress: func(progress scriptcrawler.CrawlProgress) {
scanned := progress.Checked
if scanned < progress.TotalEntries {
@@ -1276,11 +1515,77 @@ func (a *App) updateDriveScanProgress(driveID string, scanned, added int) {
if a.scanProgress == nil {
a.scanProgress = make(map[string]driveScanProgress)
}
a.scanProgress[driveID] = driveScanProgress{Scanned: scanned, Added: added}
progress := a.scanProgress[driveID]
progress.Scanned = scanned
progress.Added = added
a.scanProgress[driveID] = progress
}
a.scanQueueMu.Unlock()
}
func (a *App) updateDriveScanCooldown(driveID string, until time.Time) {
driveID = strings.TrimSpace(driveID)
if driveID == "" {
return
}
a.scanQueueMu.Lock()
if a.scanQueued[driveID] {
if a.scanProgress == nil {
a.scanProgress = make(map[string]driveScanProgress)
}
progress := a.scanProgress[driveID]
progress.CooldownUntil = until
a.scanProgress[driveID] = progress
}
a.scanQueueMu.Unlock()
}
func (a *App) pauseDriveScanForRateLimit(ctx context.Context, driveID string, drv drives.Drive, err error) bool {
wait, ok := drives.RateLimitRetryAfter(err)
if !ok {
return false
}
if wait <= 0 {
wait = scanCooldownForDrive(drv)
}
if wait <= 0 {
wait = 5 * time.Minute
}
until := time.Now().Add(wait)
a.updateDriveScanCooldown(driveID, until)
log.Printf("[scan] drive=%s rate limited; cooling until=%s wait=%s: %v", driveID, until.Format(time.RFC3339), wait, err)
if !sleepDriveScanCooldown(ctx, wait) {
log.Printf("[scan] drive=%s cooldown canceled: %v", driveID, ctx.Err())
}
return true
}
func scanCooldownForDrive(drv drives.Drive) time.Duration {
if drv == nil {
return 5 * time.Minute
}
switch strings.ToLower(drv.Kind()) {
case "guangyapan":
return 10 * time.Minute
default:
return 5 * time.Minute
}
}
func sleepDriveScanCooldown(ctx context.Context, d time.Duration) bool {
if d <= 0 {
return true
}
timer := time.NewTimer(d)
defer timer.Stop()
select {
case <-ctx.Done():
return false
case <-timer.C:
return true
}
}
func (a *App) driveHasActiveWork(driveID string) bool {
driveID = strings.TrimSpace(driveID)
if driveID == "" {
@@ -1435,10 +1740,11 @@ func (a *App) stopDriveTasks(ctx context.Context, driveID string) bool {
queued := a.clearQueuedDriveTask(driveID)
fingerprintQueued := a.clearFingerprintQueueing(driveID)
uploading := a.clearCrawlerUploadProgress(driveID)
transcoding := a.stopDriveTranscode(driveID)
hadWorkers := a.resetDriveGenerationWorkers(ctx, driveID)
stopped := canceled > 0 || queued || fingerprintQueued || uploading || hadWorkers
log.Printf("[tasks] stop drive=%s stopped=%v canceled_tasks=%d queued=%v fingerprint_queue=%v uploading=%v workers=%v",
driveID, stopped, canceled, queued, fingerprintQueued, uploading, hadWorkers)
stopped := canceled > 0 || queued || fingerprintQueued || uploading || transcoding || hadWorkers
log.Printf("[tasks] stop drive=%s stopped=%v canceled_tasks=%d queued=%v fingerprint_queue=%v uploading=%v transcoding=%v workers=%v",
driveID, stopped, canceled, queued, fingerprintQueued, uploading, transcoding, hadWorkers)
return stopped
}
@@ -1459,6 +1765,9 @@ func (a *App) stopAllDriveTasks(ctx context.Context) int {
for _, id := range a.clearAllCrawlerUploadProgress() {
stoppedIDs[id] = struct{}{}
}
for _, id := range a.stopAllDriveTranscodes() {
stoppedIDs[id] = struct{}{}
}
for _, id := range a.resetAllDriveGenerationWorkers(ctx) {
stoppedIDs[id] = struct{}{}
}
@@ -1741,6 +2050,8 @@ func (a *App) runScanWithTaskContext(ctx context.Context, driveID string) {
if err != nil {
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
log.Printf("[scan] drive=%s canceled: %v", driveID, err)
} else if a.pauseDriveScanForRateLimit(ctx, driveID, drv, err) {
return
} else {
log.Printf("[scan] drive=%s error: %v", driveID, err)
}
@@ -1813,6 +2124,33 @@ func (a *App) cleanupMissingDriveVideos(ctx context.Context, driveID string, liv
return removed, nil
}
// migrateHiddenVideosToTombstone 把历史「隐藏」视频一次性迁移为黑名单墓碑。
// 隐藏机制已废弃——前台「不再展示」改走拉黑逻辑。迁移=删库记录 + 删本地
// 封面/预览 + 写墓碑,保留网盘源文件。迁移后无 hidden=1 记录,重复执行为空操作。
func (a *App) migrateHiddenVideosToTombstone(ctx context.Context) {
if a == nil || a.cat == nil {
return
}
hidden, err := a.cat.ListHiddenVideos(ctx)
if err != nil {
log.Printf("[migrate] list hidden videos: %v", err)
return
}
if len(hidden) == 0 {
return
}
log.Printf("[migrate] converting %d hidden video(s) to blacklist tombstones", len(hidden))
migrated := 0
for _, v := range hidden {
if _, err := a.deleteVideo(ctx, v.ID, false); err != nil {
log.Printf("[migrate] hidden->tombstone %s: %v", v.ID, err)
continue
}
migrated++
}
log.Printf("[migrate] hidden->tombstone done: %d/%d", migrated, len(hidden))
}
func (a *App) deleteVideo(ctx context.Context, videoID string, deleteSource bool) (api.DeleteVideoResult, error) {
if a == nil || a.cat == nil {
return api.DeleteVideoResult{}, sql.ErrNoRows
@@ -2844,18 +3182,7 @@ func (a *App) runScriptCrawlerCrawlWithTaskContext(ctx context.Context, driveID
driveID, res.TargetNew, res.CandidateBudget, res.TotalEntries, res.NewVideos, res.Skipped, res.Failed, res.SeenSnapshot)
}
if d.Credentials == nil {
d.Credentials = make(map[string]string)
}
d.Credentials["last_crawl_at"] = strconv.FormatInt(time.Now().Unix(), 10)
if runErr != nil {
d.Status = "error"
d.LastError = runErr.Error()
} else {
d.Status = "ok"
d.LastError = ""
}
if err := a.cat.UpsertDrive(ctx, d); err != nil {
if err := a.updateScriptCrawlerRunState(ctx, driveID, runErr); err != nil {
log.Printf("[scriptcrawler] drive=%s update last_crawl_at: %v", driveID, err)
}
if err := ctx.Err(); err != nil {
@@ -2873,6 +3200,25 @@ func (a *App) runScriptCrawlerCrawlWithTaskContext(ctx context.Context, driveID
return runErr == nil
}
func (a *App) updateScriptCrawlerRunState(ctx context.Context, driveID string, runErr error) error {
d, err := a.cat.GetDrive(ctx, driveID)
if err != nil {
return err
}
if d.Credentials == nil {
d.Credentials = make(map[string]string)
}
d.Credentials["last_crawl_at"] = strconv.FormatInt(time.Now().Unix(), 10)
if runErr != nil {
d.Status = "error"
d.LastError = runErr.Error()
} else {
d.Status = "ok"
d.LastError = ""
}
return a.cat.UpsertDrive(ctx, d)
}
func (a *App) runSpider91MigrationAfterManualCrawl(ctx context.Context, driveID string) {
a.runCrawlerMigrationAfterManualCrawl(ctx, driveID)
}
@@ -2957,6 +3303,108 @@ func (a *App) runCrawlerUploadMigrationAfterSave(ctx context.Context, driveID st
}
}
func (a *App) scheduleManualCrawlerUploadMigration(ctx context.Context, driveID string) (bool, string) {
driveID = strings.TrimSpace(driveID)
if driveID == "" || a == nil || a.cat == nil {
return false, "爬虫不存在"
}
if a.spider91Migrator == nil {
return false, "上传迁移器未初始化"
}
if a.driveHasActiveWork(driveID) {
return false, "当前爬虫有正在进行的任务,请稍后重试"
}
d, err := a.cat.GetDrive(ctx, driveID)
if err != nil || d == nil || d.Kind != scriptcrawler.Kind {
return false, "爬虫不存在"
}
targetDriveID := strings.TrimSpace(d.Credentials["upload_drive_id"])
if targetDriveID == "" {
return false, "请先配置上传网盘"
}
assets, err := a.cat.CountCrawlerAssets(ctx, driveID, crawlerCatalogVideoIDPrefixes(d))
if err != nil {
log.Printf("[scriptcrawler] drive=%s manual upload count assets: %v", driveID, err)
return false, "读取待上传视频失败"
}
if reason := crawlerUploadAssetBlockReason(d, assets); reason != "" {
return false, reason
}
if err := a.ensureDriveAttached(ctx, driveID); err != nil {
log.Printf("[scriptcrawler] drive=%s manual upload source attach: %v", driveID, err)
return false, "爬虫本地存储不可用"
}
if err := a.ensureDriveAttached(ctx, targetDriveID); err != nil {
log.Printf("[scriptcrawler] drive=%s manual upload target=%s attach: %v", driveID, targetDriveID, err)
return false, "上传网盘不可用:" + err.Error()
}
a.crawlerUploadMu.Lock()
if a.crawlerUploadRunning == nil {
a.crawlerUploadRunning = make(map[string]bool)
}
if a.crawlerUploadRunning[driveID] {
a.crawlerUploadMu.Unlock()
return false, "当前爬虫已有上传任务正在运行"
}
a.crawlerUploadRunning[driveID] = true
a.crawlerUploadMu.Unlock()
taskCtx, done := a.registerDriveTaskContext(ctx, driveID)
go func() {
defer func() {
done()
a.crawlerUploadMu.Lock()
delete(a.crawlerUploadRunning, driveID)
a.crawlerUploadMu.Unlock()
}()
a.runManualCrawlerUploadMigration(taskCtx, driveID, targetDriveID)
}()
return true, ""
}
func crawlerUploadAssetBlockReason(d *catalog.Drive, assets catalog.CrawlerAssetCounts) string {
if assets.Local <= 0 {
return "没有待上传的本地视频"
}
if assets.Fingerprint.Pending > 0 {
return "还有待生成的视频指纹"
}
if assets.Fingerprint.Failed > 0 {
return "存在指纹生成失败的视频,请先重试或处理失败项"
}
if d != nil && d.TeaserEnabled {
if assets.Teaser.Pending > 0 {
return "还有待生成的预览视频"
}
if assets.Teaser.Failed > 0 {
return "存在预览视频生成失败的视频,请先重试或处理失败项"
}
}
return ""
}
func crawlerCatalogVideoIDPrefixes(d *catalog.Drive) []string {
if d == nil {
return nil
}
return []string{
scriptcrawler.Kind + "-" + d.ID + "-",
spider91.Kind + "-" + d.ID + "-",
}
}
func (a *App) runManualCrawlerUploadMigration(ctx context.Context, driveID, targetDriveID string) {
if err := ctx.Err(); err != nil {
log.Printf("[scriptcrawler] drive=%s skip manual upload migration: %v", driveID, err)
return
}
log.Printf("[scriptcrawler] drive=%s running manual upload migration target=%s", driveID, targetDriveID)
if err := a.spider91Migrator.RunOnce(ctx); err != nil {
log.Printf("[scriptcrawler] drive=%s manual upload migration: %v", driveID, err)
}
}
func (a *App) runCrawlerMigrationAfterManualCrawl(ctx context.Context, driveID string) {
if err := ctx.Err(); err != nil {
log.Printf("[scriptcrawler] drive=%s skip post-crawl migration: %v", driveID, err)
@@ -3171,3 +3619,14 @@ func parseBoolDefault(raw string, def bool) bool {
}
return v
}
func parseIntDefault(raw string, def int) int {
if raw == "" {
return def
}
v, err := strconv.Atoi(raw)
if err != nil {
return def
}
return v
}
+6
View File
@@ -41,6 +41,7 @@ func TestSpider91UploadDriveIDDoesNotAutoSelectTarget(t *testing.T) {
reg.Set("p123-one", &spider91UploadTargetFakeDrive{id: "p123-one", kind: "p123"})
reg.Set("onedrive-one", &spider91UploadTargetFakeDrive{id: "onedrive-one", kind: "onedrive"})
reg.Set("wopan-one", &spider91UploadTargetFakeDrive{id: "wopan-one", kind: "wopan"})
reg.Set("guangyapan-one", &spider91UploadTargetFakeDrive{id: "guangyapan-one", kind: "guangyapan"})
app := &App{registry: reg}
if got := app.Spider91UploadDriveID(); got != "" {
@@ -67,6 +68,11 @@ func TestSpider91UploadDriveIDDoesNotAutoSelectTarget(t *testing.T) {
t.Fatalf("explicit wopan upload target = %q, want wopan-one", got)
}
app.spider91UploadDriveID = "guangyapan-one"
if got := app.Spider91UploadDriveID(); got != "guangyapan-one" {
t.Fatalf("explicit guangyapan upload target = %q, want guangyapan-one", got)
}
app.spider91UploadDriveID = "missing"
if got := app.Spider91UploadDriveID(); got != "" {
t.Fatalf("missing upload target = %q, want empty", got)
+200
View File
@@ -227,6 +227,53 @@ func TestRegisterPreviewWorkersBackfillsHistoricalFingerprints(t *testing.T) {
t.Fatalf("fingerprint status=%q sampled=%q, want ready with hash", got.FingerprintStatus, got.SampledSHA256)
}
func TestUpdateScriptCrawlerRunStatePreservesCurrentTeaserSwitch(t *testing.T) {
ctx := context.Background()
cat, err := catalog.Open(t.TempDir() + "/catalog.db")
if err != nil {
t.Fatalf("open catalog: %v", err)
}
t.Cleanup(func() {
if err := cat.Close(); err != nil {
t.Fatalf("close catalog: %v", err)
}
})
if err := cat.UpsertDrive(ctx, &catalog.Drive{
ID: "crawler-id",
Kind: scriptcrawler.Kind,
Name: "Crawler",
RootID: "/",
Credentials: map[string]string{
"script_path": "/tmp/crawler.py",
"target_new": "10",
},
TeaserEnabled: false,
}); err != nil {
t.Fatalf("seed crawler drive: %v", err)
}
if err := cat.SetDriveTeaserEnabled(ctx, "crawler-id", true); err != nil {
t.Fatalf("toggle teaser: %v", err)
}
app := &App{cat: cat}
if err := app.updateScriptCrawlerRunState(ctx, "crawler-id", nil); err != nil {
t.Fatalf("update run state: %v", err)
}
got, err := cat.GetDrive(ctx, "crawler-id")
if err != nil {
t.Fatalf("get crawler drive: %v", err)
}
if !got.TeaserEnabled {
t.Fatal("teaserEnabled = false after run state update, want preserved true")
}
if got.Status != "ok" || got.LastError != "" {
t.Fatalf("status=%q lastError=%q, want ok with no error", got.Status, got.LastError)
}
if got.Credentials["last_crawl_at"] == "" || got.Credentials["target_new"] != "10" {
t.Fatalf("credentials after run state update = %#v", got.Credentials)
}
}
func TestStopDriveTasksCancelsQueuedTasksAndReplacesWorkers(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
@@ -391,6 +438,37 @@ func TestDriveGenerationStatusIncludesScanState(t *testing.T) {
}
}
func TestDriveGenerationStatusIncludesScanCooldown(t *testing.T) {
until := time.Now().Add(time.Hour).Round(time.Second)
app := &App{
scanQueued: map[string]bool{"drive-id": true},
scanProgress: map[string]driveScanProgress{
"drive-id": {Scanned: 12, Added: 3, CooldownUntil: until},
},
}
status := app.driveGenerationStatuses()["drive-id"].Scan
if status.State != "cooling" {
t.Fatalf("scan status = %#v, want cooling", status)
}
if status.CooldownUntil != until.Format(time.RFC3339) {
t.Fatalf("cooldown until = %q, want %q", status.CooldownUntil, until.Format(time.RFC3339))
}
}
func TestGuangYaPanGenerationCooldowns(t *testing.T) {
drv := &serverFakeKindDrive{id: "gy", kind: "guangyapan"}
if got := generationCooldownForDrive(drv); got != 10*time.Minute {
t.Fatalf("generation cooldown = %s, want 10m", got)
}
if got := fingerprintConfigForDrive(drv).RateLimitCooldown; got != 10*time.Minute {
t.Fatalf("fingerprint cooldown = %s, want 10m", got)
}
if got := scanCooldownForDrive(drv); got != 10*time.Minute {
t.Fatalf("scan cooldown = %s, want 10m", got)
}
}
func TestRunSpider91MigrationAfterManualCrawlRequiresConfiguredUploadTarget(t *testing.T) {
ctx := context.Background()
registry := proxy.NewRegistry()
@@ -500,6 +578,128 @@ func TestScheduleCrawlerUploadMigrationSkipsWithoutUploadTarget(t *testing.T) {
}
}
func TestScheduleManualCrawlerUploadMigrationRunsWhenAssetsReady(t *testing.T) {
ctx := context.Background()
cat, err := catalog.Open(t.TempDir() + "/catalog.db")
if err != nil {
t.Fatalf("open catalog: %v", err)
}
t.Cleanup(func() {
if err := cat.Close(); err != nil {
t.Fatalf("close catalog: %v", err)
}
})
if err := cat.UpsertDrive(ctx, &catalog.Drive{
ID: "crawler-ready",
Kind: scriptcrawler.Kind,
Name: "Ready Crawler",
RootID: "/",
TeaserEnabled: true,
Credentials: map[string]string{
"script_path": "/tmp/ready.py",
"upload_drive_id": "pikpak-target",
},
}); err != nil {
t.Fatalf("seed crawler: %v", err)
}
if err := cat.UpsertVideo(ctx, &catalog.Video{
ID: scriptcrawler.BuildVideoID("crawler-ready", "source-1"),
DriveID: "crawler-ready",
FileID: "source-1.mp4",
FileName: "source-1.mp4",
Title: "Source 1",
Size: 123,
Ext: "mp4",
SampledSHA256: "sampled-source-1",
FingerprintStatus: "ready",
PreviewStatus: "ready",
PublishedAt: time.Now(),
CreatedAt: time.Now(),
UpdatedAt: time.Now(),
}); err != nil {
t.Fatalf("seed video: %v", err)
}
registry := proxy.NewRegistry()
registry.Set("crawler-ready", &serverFakeKindDrive{id: "crawler-ready", kind: scriptcrawler.Kind})
registry.Set("pikpak-target", &serverFakeKindDrive{id: "pikpak-target", kind: "pikpak"})
migrator := &serverFakeSpider91MigrationRunner{}
app := &App{
cat: cat,
registry: registry,
spider91Migrator: migrator,
workers: map[string]*preview.Worker{},
thumbWorkers: map[string]*preview.ThumbWorker{},
fingerprintWorkers: map[string]*fingerprint.Worker{},
}
accepted, message := app.scheduleManualCrawlerUploadMigration(ctx, "crawler-ready")
if !accepted {
t.Fatalf("accepted = false, message = %q", message)
}
deadline := time.After(time.Second)
for migrator.called == 0 {
select {
case <-deadline:
t.Fatalf("migration calls = %d, want 1", migrator.called)
case <-time.After(10 * time.Millisecond):
}
}
}
func TestScheduleManualCrawlerUploadMigrationRejectsPendingFingerprint(t *testing.T) {
ctx := context.Background()
cat, err := catalog.Open(t.TempDir() + "/catalog.db")
if err != nil {
t.Fatalf("open catalog: %v", err)
}
t.Cleanup(func() {
if err := cat.Close(); err != nil {
t.Fatalf("close catalog: %v", err)
}
})
if err := cat.UpsertDrive(ctx, &catalog.Drive{
ID: "crawler-pending",
Kind: scriptcrawler.Kind,
Name: "Pending Crawler",
RootID: "/",
TeaserEnabled: true,
Credentials: map[string]string{
"script_path": "/tmp/pending.py",
"upload_drive_id": "pikpak-target",
},
}); err != nil {
t.Fatalf("seed crawler: %v", err)
}
if err := cat.UpsertVideo(ctx, &catalog.Video{
ID: scriptcrawler.BuildVideoID("crawler-pending", "source-1"),
DriveID: "crawler-pending",
FileID: "source-1.mp4",
FileName: "source-1.mp4",
Title: "Source 1",
Size: 123,
Ext: "mp4",
PreviewStatus: "ready",
PublishedAt: time.Now(),
CreatedAt: time.Now(),
UpdatedAt: time.Now(),
}); err != nil {
t.Fatalf("seed video: %v", err)
}
migrator := &serverFakeSpider91MigrationRunner{}
app := &App{cat: cat, registry: proxy.NewRegistry(), spider91Migrator: migrator}
accepted, message := app.scheduleManualCrawlerUploadMigration(ctx, "crawler-pending")
if accepted {
t.Fatal("accepted = true, want false")
}
if !strings.Contains(message, "指纹") {
t.Fatalf("message = %q, want fingerprint reason", message)
}
if migrator.called != 0 {
t.Fatalf("migration calls = %d, want 0", migrator.called)
}
}
func TestDriveGenerationStatusUsesWorkerQueueNotPendingCatalogRows(t *testing.T) {
ctx := context.Background()
cat, err := catalog.Open(t.TempDir() + "/catalog.db")
+12 -1
View File
@@ -56,7 +56,7 @@ preview:
width: 480
# 盘列表。上线后请通过管理后台添加,本文件可留空。
# kind 支持 quark / p115 / p123 / pikpak / wopan / onedrive / googledrive / localstorage。
# kind 支持 quark / p115 / p123 / pikpak / wopan / guangyapan / onedrive / googledrive / localstorage。
# OneDrive 示例:
# - id: "my-onedrive"
# kind: "onedrive"
@@ -76,6 +76,17 @@ preview:
# # use_online_api: "false"
# # client_id: "..."
# # client_secret: "..."
# 光鸭网盘示例:
# - id: "my-guangyapan"
# kind: "guangyapan"
# name: "我的光鸭网盘"
# # 留空表示光鸭网盘根目录;也可以填写光鸭目录 fileId
# root_id: ""
# params:
# # 推荐在后台使用扫码登录自动写入 access_token / refresh_token。
# refresh_token: "..."
# # 可选:按路径解析扫描根目录,优先于 root_id
# # root_path: "影视/电影"
# 本地存储示例:
# - id: "local-media"
# kind: "localstorage"
+338 -26
View File
@@ -21,6 +21,7 @@ import (
"github.com/video-site/backend/internal/auth"
"github.com/video-site/backend/internal/catalog"
"github.com/video-site/backend/internal/drives/guangyapan"
"github.com/video-site/backend/internal/drives/p123"
"github.com/video-site/backend/internal/drives/scriptcrawler"
"github.com/video-site/backend/internal/drives/spider91"
@@ -48,27 +49,35 @@ type AdminServer struct {
// LocalPreviewDir is the local directory that stores generated preview videos and thumbs.
LocalPreviewDir string
// Hooks:外层注入实际执行者
OnDriveSaved func(driveID string) error
OnDriveDeleteCleanup func(ctx context.Context, driveID string) (int, error)
OnDriveRemoved func(driveID string)
OnScanRequested func(driveID string) bool
OnStopDriveTasks func(driveID string) bool
OnStopAllTasks func() int
OnRegenPreview func(videoID string)
OnRegenAllPreviews func()
OnRegenFailedPreviews func(driveID string)
OnRegenFailedThumbnails func(driveID string)
OnRegenFailedFingerprints func(driveID string)
OnDeleteVideo func(ctx context.Context, videoID string, deleteSource bool) (DeleteVideoResult, error)
GetDriveGenerationStatuses func() map[string]DriveGenerationStatuses
OnDriveSaved func(driveID string) error
OnDriveDeleteCleanup func(ctx context.Context, driveID string) (int, error)
OnDriveRemoved func(driveID string)
OnScanRequested func(driveID string) bool
OnCrawlerUploadRequested func(driveID string) (bool, string)
OnStopDriveTasks func(driveID string) bool
OnStopAllTasks func() int
OnRegenPreview func(videoID string)
OnRegenAllPreviews func()
OnRegenFailedPreviews func(driveID string)
OnRegenFailedThumbnails func(driveID string)
OnRegenFailedFingerprints func(driveID string)
// OnStartDriveTranscode 手动开启某盘的浏览器兼容性转码任务。
// 返回 (是否接受, 拒绝原因)。转码从不自动运行,只能在这里手动触发;
// 处理完候选列表后任务自然结束。
OnStartDriveTranscode func(driveID string) (bool, string)
// OnStopDriveTranscode 手动停止某盘正在进行的转码任务。返回是否有任务被停。
OnStopDriveTranscode func(driveID string) bool
OnDeleteVideo func(ctx context.Context, videoID string, deleteSource bool) (DeleteVideoResult, error)
GetDriveGenerationStatuses func() map[string]DriveGenerationStatuses
GetPreviewGenerationVideoIDs func() map[string]bool
// OnTeaserEnabledChanged 在 per-drive 预览视频开关被切换后调用。
// enabled=true 时上层应该重新把 pending 预览视频入队(类似旧的全局开关从关到开);
// enabled=false 时通常不用做事 —— worker 入队前会再次查 catalog,自然停止。
OnTeaserEnabledChanged func(driveID string, enabled bool)
// Theme 读写("dark" | "pink"
// Theme 读写("dark" | "pink" | "sky"
GetTheme func() string
SetTheme func(theme string) error
// Spider91 → 115/123/PikPak/OneDrive/Google Drive/联通网盘 上传目标 drive ID 读写
// Spider91 → 115/123/PikPak/OneDrive/Google Drive/联通网盘/光鸭网盘 上传目标 drive ID 读写
GetSpider91UploadDriveID func() string
SetSpider91UploadDriveID func(driveID string) error
// OnRunNightlyJob 触发一次完整的凌晨流水线(Phase1 扫盘 + Phase2 91 爬虫 +
@@ -88,6 +97,9 @@ type AdminServer struct {
// 联通网盘扫码登录接口测试注入;生产留空走官方 panservice.mail.wo.cn。
WopanQRAPIBaseURL string
WopanQRHTTPClient *http.Client
// 光鸭网盘扫码登录接口测试注入;生产留空走官方 account.guangyapan.com。
GuangYaPanAccountBaseURL string
GuangYaPanHTTPClient *http.Client
}
const (
@@ -118,6 +130,7 @@ type DriveGenerationStatuses struct {
Preview GenerationStatus `json:"preview"`
Fingerprint GenerationStatus `json:"fingerprint"`
Upload GenerationStatus `json:"upload"`
Transcode GenerationStatus `json:"transcode"`
}
type NightlyJobStatus struct {
@@ -160,6 +173,8 @@ func (a *AdminServer) Register(r chi.Router) {
r.Get("/drives/p123/qr/{uniID}", a.handleP123QRStatus)
r.Post("/drives/wopan/qr", a.handleWopanQRStart)
r.Get("/drives/wopan/qr/{uuid}", a.handleWopanQRStatus)
r.Post("/drives/guangyapan/qr", a.handleGuangYaPanQRStart)
r.Get("/drives/guangyapan/qr/status", a.handleGuangYaPanQRStatus)
r.Delete("/drives/{id}", a.handleDeleteDrive)
r.Post("/drives/{id}/rescan", a.handleRescan)
r.Post("/drives/{id}/tasks/stop", a.handleStopDriveTasks)
@@ -169,6 +184,8 @@ func (a *AdminServer) Register(r chi.Router) {
r.Post("/drives/{id}/previews/failed/regenerate", a.handleRegenFailedPreviews)
r.Post("/drives/{id}/thumbnails/failed/regenerate", a.handleRegenFailedThumbnails)
r.Post("/drives/{id}/fingerprints/failed/regenerate", a.handleRegenFailedFingerprints)
r.Post("/drives/{id}/transcode/start", a.handleStartDriveTranscode)
r.Post("/drives/{id}/transcode/stop", a.handleStopDriveTranscode)
// 爬虫
r.Get("/crawlers", a.handleListCrawlers)
@@ -178,14 +195,19 @@ func (a *AdminServer) Register(r chi.Router) {
r.Post("/crawlers/test-script", a.handleTestCrawlerScript)
r.Delete("/crawlers/{id}", a.handleDeleteCrawler)
r.Post("/crawlers/{id}/run", a.handleRunCrawler)
r.Post("/crawlers/{id}/upload", a.handleUploadCrawlerVideos)
r.Post("/crawlers/{id}/tasks/stop", a.handleStopCrawlerTasks)
// 视频
r.Get("/videos", a.handleAdminListVideos)
r.Get("/videos/stats", a.handleVideoStats)
r.Put("/videos/{id}", a.handleUpdateVideo)
r.Delete("/videos/{id}", a.handleDeleteVideo)
r.Post("/videos/regen-preview", a.handleRegenAllPreviews)
r.Post("/videos/{id}/regen-preview", a.handleRegenPreview)
// 黑名单(被拉黑/手动删除、扫盘不再入库的视频)
r.Get("/blacklist", a.handleListBlacklist)
r.Delete("/blacklist/{id}", a.handleRemoveBlacklist)
// 标签
r.Get("/tags", a.handleListTags)
@@ -431,6 +453,11 @@ func (a *AdminServer) handleListDrives(w http.ResponseWriter, r *http.Request) {
writeErr(w, http.StatusInternalServerError, err)
return
}
transcodeCounts, err := a.Catalog.CountTranscodesByDrive(r.Context())
if err != nil {
writeErr(w, http.StatusInternalServerError, err)
return
}
generationStatuses := map[string]DriveGenerationStatuses{}
if a.GetDriveGenerationStatuses != nil {
generationStatuses = a.GetDriveGenerationStatuses()
@@ -445,7 +472,8 @@ func (a *AdminServer) handleListDrives(w http.ResponseWriter, r *http.Request) {
Status string `json:"status"`
LastError string `json:"lastError,omitempty"`
HasCredential bool `json:"hasCredential"`
// TeaserEnabled 控制是否给本盘生成预览视频/封面。前端用它在网盘列表/编辑表单展示开关状态
// TeaserEnabled 控制是否给本盘生成预览视频封面生成不受影响
// 前端用它在网盘列表/编辑表单展示开关状态。
TeaserEnabled bool `json:"teaserEnabled"`
// SkipDirIDs 是用户在 admin 配置的"扫描跳过目录"集合(drive 侧目录 fileID)。
// 前端用它在"设置跳过目录"弹窗里回显已选项;JSON 字段名 camelCase 与
@@ -453,9 +481,12 @@ func (a *AdminServer) handleListDrives(w http.ResponseWriter, r *http.Request) {
SkipDirIDs []string `json:"skipDirIds"`
// LastCrawlAt 是 spider91 上次成功爬取的 unix 秒(来自 credentials.last_crawl_at)。
// 其它 kind 留 0;前端用它显示"上次抓取: N 小时前"。
Spider91Proxy string `json:"spider91Proxy,omitempty"`
LastCrawlAt int64 `json:"lastCrawlAt,omitempty"`
GoogleDriveUseOnlineAPI *bool `json:"googleDriveUseOnlineAPI,omitempty"`
Spider91Proxy string `json:"spider91Proxy,omitempty"`
LastCrawlAt int64 `json:"lastCrawlAt,omitempty"`
GoogleDriveUseOnlineAPI *bool `json:"googleDriveUseOnlineAPI,omitempty"`
GoogleDriveOpenListAPIURL string `json:"googleDriveOpenListApiUrl,omitempty"`
// STRMAllowOutsideRoot 是 localstorage 的 .strm 越root开关;其它 kind 省略。
STRMAllowOutsideRoot *bool `json:"strmAllowOutsideRoot,omitempty"`
ScanGenerationStatus GenerationStatus `json:"scanGenerationStatus"`
ThumbnailGenerationStatus GenerationStatus `json:"thumbnailGenerationStatus"`
PreviewGenerationStatus GenerationStatus `json:"previewGenerationStatus"`
@@ -470,6 +501,11 @@ func (a *AdminServer) handleListDrives(w http.ResponseWriter, r *http.Request) {
FingerprintReadyCount int `json:"fingerprintReadyCount"`
FingerprintPendingCount int `json:"fingerprintPendingCount"`
FingerprintFailedCount int `json:"fingerprintFailedCount"`
TranscodeGenerationStatus GenerationStatus `json:"transcodeGenerationStatus"`
TranscodePendingCount int `json:"transcodePendingCount"`
TranscodeReadyCount int `json:"transcodeReadyCount"`
TranscodeFailedCount int `json:"transcodeFailedCount"`
TranscodeSkippedCount int `json:"transcodeSkippedCount"`
}
list := make([]out, 0, len(drives))
for _, d := range drives {
@@ -479,6 +515,7 @@ func (a *AdminServer) handleListDrives(w http.ResponseWriter, r *http.Request) {
counts := teaserCounts[d.ID]
thumbCounts := thumbnailCounts[d.ID]
fingerprintCount := fingerprintCounts[d.ID]
transcodeCount := transcodeCounts[d.ID]
generation := generationStatuses[d.ID]
if generation.Scan.State == "" {
generation.Scan.State = "idle"
@@ -492,6 +529,9 @@ func (a *AdminServer) handleListDrives(w http.ResponseWriter, r *http.Request) {
if generation.Fingerprint.State == "" {
generation.Fingerprint.State = "idle"
}
if generation.Transcode.State == "" {
generation.Transcode.State = "idle"
}
// spider91 没有用户凭证概念;只要存在 drive 行就视为"已配置"。
// last_crawl_at 是后端自动写入的运行状态字段,不计入 hasCredential 判定。
hasCred := false
@@ -523,6 +563,8 @@ func (a *AdminServer) handleListDrives(w http.ResponseWriter, r *http.Request) {
Spider91Proxy: spider91ProxyForDrive(d),
LastCrawlAt: lastCrawlAt,
GoogleDriveUseOnlineAPI: googleDriveUseOnlineAPIForDrive(d),
GoogleDriveOpenListAPIURL: googleDriveOpenListAPIURLForDrive(d),
STRMAllowOutsideRoot: strmAllowOutsideRootForDrive(d),
ScanGenerationStatus: generation.Scan,
ThumbnailGenerationStatus: generation.Thumbnail,
PreviewGenerationStatus: generation.Preview,
@@ -537,6 +579,11 @@ func (a *AdminServer) handleListDrives(w http.ResponseWriter, r *http.Request) {
FingerprintReadyCount: fingerprintCount.Ready,
FingerprintPendingCount: fingerprintCount.Pending,
FingerprintFailedCount: fingerprintCount.Failed,
TranscodeGenerationStatus: generation.Transcode,
TranscodePendingCount: transcodeCount.Pending,
TranscodeReadyCount: transcodeCount.Ready,
TranscodeFailedCount: transcodeCount.Failed,
TranscodeSkippedCount: transcodeCount.Skipped,
})
}
writeJSON(w, http.StatusOK, list)
@@ -550,7 +597,7 @@ type upsertDriveReq struct {
// Deprecated: 扫描起点已固定为 rootId;保留字段只为兼容旧客户端请求体。
ScanRootID string `json:"scanRootId"`
Credentials map[string]string `json:"credentials"`
// TeaserEnabled 是 per-drive 预览视频/封面生成开关。
// TeaserEnabled 是 per-drive 预览视频生成开关;封面生成不受影响
// 用 *bool 区分 "未传" / "传了 false":未传时表示客户端不打算改这个字段,
// 沿用 catalog 现有值;新建时未传一律默认开启(true)。
TeaserEnabled *bool `json:"teaserEnabled,omitempty"`
@@ -587,6 +634,10 @@ func (a *AdminServer) handleUpsertDrive(w http.ResponseWriter, r *http.Request)
body.Credentials = credentials
} else if body.Kind == "googledrive" {
body.Credentials = mergeGoogleDriveCredentials(existing, body.Credentials)
} else if body.Kind == "localstorage" || body.Kind == "guangyapan" {
// 按键合并、空值沿用旧值:这些网盘的编辑表单允许只改某几个字段,
// 其它 token / 路径 / 开关字段应保留旧值。
body.Credentials = mergeNonEmptyCredentials(existing, body.Credentials)
} else if len(body.Credentials) == 0 && existing != nil && len(existing.Credentials) > 0 {
body.Credentials = existing.Credentials
}
@@ -647,6 +698,7 @@ type crawlerDTO struct {
Proxy string `json:"proxy,omitempty"`
TargetNew string `json:"targetNew,omitempty"`
UploadDriveID string `json:"uploadDriveId,omitempty"`
TeaserEnabled bool `json:"teaserEnabled"`
LastCrawlAt int64 `json:"lastCrawlAt,omitempty"`
ScanGenerationStatus GenerationStatus `json:"scanGenerationStatus"`
ThumbnailGenerationStatus GenerationStatus `json:"thumbnailGenerationStatus"`
@@ -674,6 +726,7 @@ type upsertCrawlerReq struct {
Proxy string `json:"proxy"`
TargetNew string `json:"targetNew"`
UploadDriveID string `json:"uploadDriveId"`
TeaserEnabled *bool `json:"teaserEnabled,omitempty"`
}
func (a *AdminServer) handleListCrawlers(w http.ResponseWriter, r *http.Request) {
@@ -735,6 +788,7 @@ func (a *AdminServer) crawlerDTOForDrive(d *catalog.Drive, assets catalog.Crawle
Proxy: strings.TrimSpace(d.Credentials["proxy"]),
TargetNew: strings.TrimSpace(d.Credentials["target_new"]),
UploadDriveID: strings.TrimSpace(d.Credentials["upload_drive_id"]),
TeaserEnabled: d.TeaserEnabled,
LastCrawlAt: lastCrawlAt,
ScanGenerationStatus: generation.Scan,
ThumbnailGenerationStatus: generation.Thumbnail,
@@ -821,6 +875,13 @@ func (a *AdminServer) handleUpsertCrawler(w http.ResponseWriter, r *http.Request
return
}
name := meta.Name
teaserEnabled := true
if existing != nil {
teaserEnabled = existing.TeaserEnabled
}
if body.TeaserEnabled != nil {
teaserEnabled = *body.TeaserEnabled
}
if id == "" {
generatedID, err := a.generateCrawlerID(r.Context(), name)
if err != nil {
@@ -836,15 +897,15 @@ func (a *AdminServer) handleUpsertCrawler(w http.ResponseWriter, r *http.Request
RootID: "/",
Credentials: merged,
Status: "disconnected",
TeaserEnabled: true,
}
if existing != nil {
d.TeaserEnabled = existing.TeaserEnabled
TeaserEnabled: teaserEnabled,
}
if err := a.Catalog.UpsertDrive(r.Context(), d); err != nil {
writeErr(w, http.StatusInternalServerError, err)
return
}
if existing != nil && existing.TeaserEnabled != teaserEnabled && a.OnTeaserEnabledChanged != nil {
a.OnTeaserEnabledChanged(id, teaserEnabled)
}
if a.OnDriveSaved != nil {
if err := a.OnDriveSaved(id); err != nil {
writeJSON(w, http.StatusOK, map[string]any{"ok": true, "id": id, "warning": err.Error()})
@@ -894,14 +955,14 @@ func (a *AdminServer) validateCrawlerUploadDrive(ctx context.Context, driveID st
return fmt.Errorf("上传目标网盘 %q 不存在", driveID)
}
if !isCrawlerUploadTargetKind(d.Kind) {
return fmt.Errorf("上传目标网盘 %q 类型为 %s,仅支持 115网盘、PikPak、123网盘、Google Drive、OneDrive、联通网盘", driveID, d.Kind)
return fmt.Errorf("上传目标网盘 %q 类型为 %s,仅支持 115网盘、PikPak、123网盘、Google Drive、OneDrive、联通网盘、光鸭网盘", driveID, d.Kind)
}
return nil
}
func isCrawlerUploadTargetKind(kind string) bool {
switch strings.TrimSpace(kind) {
case "p115", "pikpak", "p123", "googledrive", "onedrive", "wopan":
case "p115", "pikpak", "p123", "googledrive", "onedrive", "wopan", "guangyapan":
return true
default:
return false
@@ -1230,6 +1291,104 @@ func (a *AdminServer) handleRunCrawler(w http.ResponseWriter, r *http.Request) {
writeJSON(w, http.StatusAccepted, resp)
}
func (a *AdminServer) handleUploadCrawlerVideos(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
d, err := a.Catalog.GetDrive(r.Context(), id)
if err != nil || d == nil || !isConfiguredCrawlerDrive(d) {
http.Error(w, "crawler not found", http.StatusNotFound)
return
}
status := a.nightlyJobStatus()
if status.Running || status.Queued {
writeJSON(w, http.StatusAccepted, map[string]any{
"ok": true,
"accepted": false,
"message": fullScanBusyMessage,
"status": status,
})
return
}
assets, err := a.Catalog.CountCrawlerAssets(r.Context(), d.ID, crawlerVideoIDPrefixes(d))
if err != nil {
writeErr(w, http.StatusInternalServerError, err)
return
}
generation := DriveGenerationStatuses{}
if a.GetDriveGenerationStatuses != nil {
generation = a.GetDriveGenerationStatuses()[d.ID]
}
if reason := crawlerUploadBlockedReason(d, assets, generation); reason != "" {
writeJSON(w, http.StatusAccepted, map[string]any{
"ok": true,
"accepted": false,
"message": reason,
})
return
}
accepted := true
message := ""
if a.OnCrawlerUploadRequested != nil {
accepted, message = a.OnCrawlerUploadRequested(id)
}
resp := map[string]any{"ok": true, "accepted": accepted}
if !accepted {
if strings.TrimSpace(message) == "" {
message = driveTaskBusyMessage
}
resp["message"] = message
}
writeJSON(w, http.StatusAccepted, resp)
}
func crawlerUploadBlockedReason(d *catalog.Drive, assets catalog.CrawlerAssetCounts, generation DriveGenerationStatuses) string {
if d == nil || !isConfiguredCrawlerDrive(d) {
return "爬虫不存在"
}
if strings.TrimSpace(d.Credentials["upload_drive_id"]) == "" {
return "请先配置上传网盘"
}
if assets.Local <= 0 {
return "没有待上传的本地视频"
}
if crawlerGenerationBusy(generation) {
return "当前爬虫有正在进行的任务,请稍后重试"
}
if assets.Fingerprint.Pending > 0 {
return "还有待生成的视频指纹"
}
if assets.Fingerprint.Failed > 0 {
return "存在指纹生成失败的视频,请先重试或处理失败项"
}
if d.TeaserEnabled {
if assets.Teaser.Pending > 0 {
return "还有待生成的预览视频"
}
if assets.Teaser.Failed > 0 {
return "存在预览视频生成失败的视频,请先重试或处理失败项"
}
}
return ""
}
func crawlerGenerationBusy(g DriveGenerationStatuses) bool {
return generationBusy(g.Scan) ||
generationBusy(g.Thumbnail) ||
generationBusy(g.Preview) ||
generationBusy(g.Fingerprint) ||
generationBusy(g.Upload)
}
func generationBusy(g GenerationStatus) bool {
switch strings.TrimSpace(g.State) {
case "", "idle":
return false
default:
return true
}
}
func (a *AdminServer) handleStopCrawlerTasks(w http.ResponseWriter, r *http.Request) {
a.handleStopDriveTasks(w, r)
}
@@ -1322,6 +1481,21 @@ func spider91ProxyForDrive(d *catalog.Drive) string {
return strings.TrimSpace(d.Credentials["proxy"])
}
// strmAllowOutsideRootForDrive 返回 localstorage 的 .strm 越root开关;
// 其它 kind 返回 nilJSON 省略)。未配置时默认 false。
func strmAllowOutsideRootForDrive(d *catalog.Drive) *bool {
if d == nil || d.Kind != "localstorage" {
return nil
}
result := false
if d.Credentials != nil {
if v, err := strconv.ParseBool(strings.TrimSpace(d.Credentials["strm_allow_outside_root"])); err == nil {
result = v
}
}
return &result
}
func googleDriveUseOnlineAPIForDrive(d *catalog.Drive) *bool {
if d == nil || d.Kind != "googledrive" {
return nil
@@ -1342,7 +1516,25 @@ func googleDriveUseOnlineAPIForDrive(d *catalog.Drive) *bool {
return &result
}
func googleDriveOpenListAPIURLForDrive(d *catalog.Drive) string {
if d == nil || d.Kind != "googledrive" || d.Credentials == nil {
return ""
}
return strings.TrimSpace(d.Credentials["api_url_address"])
}
func mergeGoogleDriveCredentials(existing *catalog.Drive, incoming map[string]string) map[string]string {
merged := mergeNonEmptyCredentials(existing, incoming)
if _, ok := incoming["api_url_address"]; ok && strings.TrimSpace(incoming["api_url_address"]) == "" {
delete(merged, "api_url_address")
}
return merged
}
// mergeNonEmptyCredentials 逐键合并凭证:incoming 里非空的键覆盖旧值,
// 空值/缺失的键沿用旧值。googledrive、localstorage 和 guangyapan 的编辑表单都依赖
// 这个语义(留空 = 不修改)。
func mergeNonEmptyCredentials(existing *catalog.Drive, incoming map[string]string) map[string]string {
merged := map[string]string{}
if existing != nil {
for k, v := range existing.Credentials {
@@ -1547,6 +1739,35 @@ func (a *AdminServer) handleStopDriveTasks(w http.ResponseWriter, r *http.Reques
})
}
// handleStartDriveTranscode 手动开启某盘的浏览器兼容性转码。
// 转码默认不开启、从不自动运行;本接口是唯一入口。
func (a *AdminServer) handleStartDriveTranscode(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
if a.OnStartDriveTranscode == nil {
writeErr(w, http.StatusNotImplemented, errors.New("transcode not supported"))
return
}
accepted, message := a.OnStartDriveTranscode(id)
writeJSON(w, http.StatusAccepted, map[string]any{
"ok": true,
"accepted": accepted,
"message": message,
})
}
// handleStopDriveTranscode 手动停止某盘正在进行的转码任务。
func (a *AdminServer) handleStopDriveTranscode(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
stopped := false
if a.OnStopDriveTranscode != nil {
stopped = a.OnStopDriveTranscode(id)
}
writeJSON(w, http.StatusAccepted, map[string]any{
"ok": true,
"stopped": stopped,
})
}
func (a *AdminServer) p123QRClient() *p123.QRClient {
return p123.NewQRClient(p123.QRConfig{
UserAPIBaseURL: a.P123UserAPIBaseURL,
@@ -1612,6 +1833,38 @@ func (a *AdminServer) handleWopanQRStatus(w http.ResponseWriter, r *http.Request
writeJSON(w, http.StatusOK, status)
}
func (a *AdminServer) guangYaPanQRClient() *guangyapan.QRClient {
return guangyapan.NewQRClient(guangyapan.QRConfig{
AccountBaseURL: a.GuangYaPanAccountBaseURL,
HTTPClient: a.GuangYaPanHTTPClient,
})
}
func (a *AdminServer) handleGuangYaPanQRStart(w http.ResponseWriter, r *http.Request) {
session, err := a.guangYaPanQRClient().Generate(r.Context())
if err != nil {
writeErr(w, http.StatusBadGateway, err)
return
}
w.Header().Set("Cache-Control", "no-store")
writeJSON(w, http.StatusOK, session)
}
func (a *AdminServer) handleGuangYaPanQRStatus(w http.ResponseWriter, r *http.Request) {
deviceCode := r.URL.Query().Get("deviceCode")
if strings.TrimSpace(deviceCode) == "" {
http.Error(w, "deviceCode is required", http.StatusBadRequest)
return
}
status, err := a.guangYaPanQRClient().Poll(r.Context(), deviceCode)
if err != nil {
writeErr(w, http.StatusBadGateway, err)
return
}
w.Header().Set("Cache-Control", "no-store")
writeJSON(w, http.StatusOK, status)
}
// handleRunNightlyJob 触发一次完整的凌晨流水线(不论当前时间,不论今日是否已跑)。
// 立即返回 202;进度通过 backend 日志和下次 GET /admin/api/drives 的状态变化观察。
// 流水线已在跑或已排队时,Runner 会拒绝重复触发。
@@ -1798,6 +2051,14 @@ func (a *AdminServer) handleAdminListVideos(w http.ResponseWriter, r *http.Reque
writeErr(w, http.StatusInternalServerError, err)
return
}
if a.GetPreviewGenerationVideoIDs != nil {
generating := a.GetPreviewGenerationVideoIDs()
for _, item := range items {
if item != nil && generating[item.ID] {
item.PreviewStatus = "generating"
}
}
}
writeJSON(w, http.StatusOK, map[string]any{
"items": items,
"total": total,
@@ -1806,6 +2067,57 @@ func (a *AdminServer) handleAdminListVideos(w http.ResponseWriter, r *http.Reque
})
}
// handleVideoStats 返回后台视频管理两个标签页的计数(当前/拉黑)。
func (a *AdminServer) handleVideoStats(w http.ResponseWriter, r *http.Request) {
current, blacklisted, err := a.Catalog.VideoManagementCounts(r.Context())
if err != nil {
writeErr(w, http.StatusInternalServerError, err)
return
}
writeJSON(w, http.StatusOK, map[string]any{
"current": current,
"blacklisted": blacklisted,
})
}
// handleListBlacklist 分页返回黑名单(墓碑)视频。
func (a *AdminServer) handleListBlacklist(w http.ResponseWriter, r *http.Request) {
q := r.URL.Query()
page, _ := strconv.Atoi(q.Get("page"))
size, _ := strconv.Atoi(q.Get("size"))
if page <= 0 {
page = 1
}
if size <= 0 || size > 100 {
size = 100
}
items, total, err := a.Catalog.ListDeletedVideos(r.Context(), q.Get("keyword"), page, size)
if err != nil {
writeErr(w, http.StatusInternalServerError, err)
return
}
writeJSON(w, http.StatusOK, map[string]any{
"items": items,
"total": total,
"page": page,
"size": size,
})
}
// handleRemoveBlacklist 把视频移出黑名单(删除墓碑),下次扫盘会重新入库。
func (a *AdminServer) handleRemoveBlacklist(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
if err := a.Catalog.RemoveDeletedVideo(r.Context(), id); err != nil {
if errors.Is(err, sql.ErrNoRows) {
writeErr(w, http.StatusNotFound, err)
return
}
writeErr(w, http.StatusInternalServerError, err)
return
}
writeJSON(w, http.StatusOK, map[string]any{"ok": true})
}
func (a *AdminServer) handleListTags(w http.ResponseWriter, r *http.Request) {
tags, err := a.Catalog.ListTags(r.Context())
if err != nil {
+279 -23
View File
@@ -732,6 +732,31 @@ func TestHandleUpsertGoogleDriveMergesOAuthCredentials(t *testing.T) {
if got.Credentials["client_id"] != "google-client-id" || got.Credentials["client_secret"] != "google-client-secret" {
t.Fatalf("oauth client credentials = %#v, want saved", got.Credentials)
}
if got.Credentials["api_url_address"] != "https://api.oplist.org/googleui/renewapi" {
t.Fatalf("api_url_address = %q, want preserved", got.Credentials["api_url_address"])
}
clearReq := httptest.NewRequest(http.MethodPost, "/admin/api/drives", bytes.NewBufferString(`{
"id": "google-main",
"kind": "googledrive",
"name": "Google Drive",
"rootId": "root",
"credentials": {
"api_url_address": ""
}
}`))
clearRR := httptest.NewRecorder()
(&AdminServer{Catalog: cat}).handleUpsertDrive(clearRR, clearReq)
if clearRR.Code != http.StatusOK {
t.Fatalf("clear status = %d, body = %s", clearRR.Code, clearRR.Body.String())
}
cleared, err := cat.GetDrive(ctx, "google-main")
if err != nil {
t.Fatalf("get cleared drive: %v", err)
}
if _, ok := cleared.Credentials["api_url_address"]; ok {
t.Fatalf("api_url_address was not cleared: %#v", cleared.Credentials)
}
}
func TestHandleUpsertSpider91DriveIsRejected(t *testing.T) {
@@ -754,7 +779,7 @@ func TestHandleUpsertSpider91DriveIsRejected(t *testing.T) {
Credentials: map[string]string{
"last_crawl_at": "1800000000",
"proxy": "http://old-proxy.local:7890",
"script_path": "/opt/video-site-91/91VideoSpider/spider_91porn.py",
"script_path": "/opt/video-site-91/data/crawler-scripts/legacy-spider.py",
},
Status: "ok",
}); err != nil {
@@ -944,7 +969,8 @@ func TestHandleListCrawlersOnlyIncludesCrawlerPageScripts(t *testing.T) {
"script_path": scriptPath,
"upload_drive_id": "p115-target",
},
Status: "ok",
Status: "ok",
TeaserEnabled: false,
},
{
ID: "p115-target",
@@ -1027,6 +1053,7 @@ func TestHandleListCrawlersOnlyIncludesCrawlerPageScripts(t *testing.T) {
Kind string `json:"kind"`
Proxy string `json:"proxy"`
UploadDriveID string `json:"uploadDriveId"`
TeaserEnabled bool `json:"teaserEnabled"`
LastCrawlAt int64 `json:"lastCrawlAt"`
TotalCrawled int `json:"totalCrawledCount"`
LocalVideos int `json:"localVideoCount"`
@@ -1038,11 +1065,12 @@ func TestHandleListCrawlersOnlyIncludesCrawlerPageScripts(t *testing.T) {
if err := json.NewDecoder(rr.Body).Decode(&got); err != nil {
t.Fatalf("decode: %v", err)
}
byID := map[string]struct {
type crawlerListRow struct {
Name string
Kind string
Proxy string
UploadDriveID string
TeaserEnabled bool
LastCrawlAt int64
TotalCrawled int
LocalVideos int
@@ -1050,25 +1078,15 @@ func TestHandleListCrawlersOnlyIncludesCrawlerPageScripts(t *testing.T) {
ThumbnailReady int
TeaserReady int
FingerprintReady int
}{}
}
byID := map[string]crawlerListRow{}
for _, d := range got {
byID[d.ID] = struct {
Name string
Kind string
Proxy string
UploadDriveID string
LastCrawlAt int64
TotalCrawled int
LocalVideos int
MigratedVideo int
ThumbnailReady int
TeaserReady int
FingerprintReady int
}{
byID[d.ID] = crawlerListRow{
Name: d.Name,
Kind: d.Kind,
Proxy: d.Proxy,
UploadDriveID: d.UploadDriveID,
TeaserEnabled: d.TeaserEnabled,
LastCrawlAt: d.LastCrawlAt,
TotalCrawled: d.TotalCrawled,
LocalVideos: d.LocalVideos,
@@ -1096,6 +1114,9 @@ func TestHandleListCrawlersOnlyIncludesCrawlerPageScripts(t *testing.T) {
if byID["crawler-spider91"].UploadDriveID != "p115-target" {
t.Fatalf("uploadDriveId = %q, want p115-target", byID["crawler-spider91"].UploadDriveID)
}
if byID["crawler-spider91"].TeaserEnabled {
t.Fatal("teaserEnabled = true, want false from crawler drive")
}
if byID["crawler-spider91"].LastCrawlAt != 1800000000 {
t.Fatalf("lastCrawlAt = %d, want 1800000000", byID["crawler-spider91"].LastCrawlAt)
}
@@ -1171,7 +1192,8 @@ func TestHandleUpsertCrawlerRequiresScriptPath(t *testing.T) {
"id": "spider91-main",
"builtin": "spider91",
"scriptPath": "`+scriptPath+`",
"targetNew": "15"
"targetNew": "15",
"teaserEnabled": false
}`))
rr = httptest.NewRecorder()
srv.handleUpsertCrawler(rr, req)
@@ -1195,6 +1217,9 @@ func TestHandleUpsertCrawlerRequiresScriptPath(t *testing.T) {
if got.Credentials["script_path"] != scriptPath {
t.Fatalf("script_path = %q, want %q", got.Credentials["script_path"], scriptPath)
}
if got.TeaserEnabled {
t.Fatal("teaserEnabled = true, want false from request")
}
}
func TestHandleUpsertCrawlerGeneratesIDFromScriptName(t *testing.T) {
@@ -1271,18 +1296,28 @@ func TestHandleUpsertCrawlerPersistsAndValidatesUploadDrive(t *testing.T) {
for _, d := range []*catalog.Drive{
{ID: "p115-target", Kind: "p115", Name: "115", RootID: "0", Credentials: map[string]string{"cookie": "x"}},
{ID: "wopan-target", Kind: "wopan", Name: "沃盘", RootID: "0", Credentials: map[string]string{"access_token": "a", "refresh_token": "r"}},
{ID: "guangyapan-target", Kind: "guangyapan", Name: "光鸭", RootID: "", Credentials: map[string]string{"access_token": "a", "refresh_token": "r"}},
{ID: "local-target", Kind: "localstorage", Name: "Local", RootID: "/", Credentials: map[string]string{"path": tmp}},
} {
if err := cat.UpsertDrive(ctx, d); err != nil {
t.Fatalf("seed drive %s: %v", d.ID, err)
}
}
srv := &AdminServer{Catalog: cat}
var teaserCallbackID string
var teaserCallbackEnabled bool
srv := &AdminServer{
Catalog: cat,
OnTeaserEnabledChanged: func(id string, enabled bool) {
teaserCallbackID = id
teaserCallbackEnabled = enabled
},
}
req := httptest.NewRequest(http.MethodPost, "/admin/api/crawlers", strings.NewReader(`{
"id": "crawler-upload",
"scriptPath": "`+scriptPath+`",
"uploadDriveId": "p115-target"
"uploadDriveId": "p115-target",
"teaserEnabled": false
}`))
rr := httptest.NewRecorder()
srv.handleUpsertCrawler(rr, req)
@@ -1296,6 +1331,12 @@ func TestHandleUpsertCrawlerPersistsAndValidatesUploadDrive(t *testing.T) {
if got.Credentials["upload_drive_id"] != "p115-target" {
t.Fatalf("upload_drive_id = %q, want p115-target", got.Credentials["upload_drive_id"])
}
if got.TeaserEnabled {
t.Fatal("teaserEnabled = true, want false")
}
if teaserCallbackID != "" {
t.Fatalf("teaser callback on create = %q, want none", teaserCallbackID)
}
req = httptest.NewRequest(http.MethodPost, "/admin/api/crawlers", strings.NewReader(`{
"id": "crawler-upload",
@@ -1314,6 +1355,52 @@ func TestHandleUpsertCrawlerPersistsAndValidatesUploadDrive(t *testing.T) {
if got.Credentials["upload_drive_id"] != "wopan-target" {
t.Fatalf("upload_drive_id = %q, want wopan-target", got.Credentials["upload_drive_id"])
}
if got.TeaserEnabled {
t.Fatal("teaserEnabled after edit without field = true, want preserved false")
}
if teaserCallbackID != "" {
t.Fatalf("teaser callback after preserved edit = %q, want none", teaserCallbackID)
}
req = httptest.NewRequest(http.MethodPost, "/admin/api/crawlers", strings.NewReader(`{
"id": "crawler-upload",
"scriptPath": "`+scriptPath+`",
"uploadDriveId": "guangyapan-target"
}`))
rr = httptest.NewRecorder()
srv.handleUpsertCrawler(rr, req)
if rr.Code != http.StatusOK {
t.Fatalf("guangyapan target status = %d, body = %s", rr.Code, rr.Body.String())
}
got, err = cat.GetDrive(ctx, "crawler-upload")
if err != nil {
t.Fatalf("get crawler after guangyapan target: %v", err)
}
if got.Credentials["upload_drive_id"] != "guangyapan-target" {
t.Fatalf("upload_drive_id = %q, want guangyapan-target", got.Credentials["upload_drive_id"])
}
req = httptest.NewRequest(http.MethodPost, "/admin/api/crawlers", strings.NewReader(`{
"id": "crawler-upload",
"scriptPath": "`+scriptPath+`",
"uploadDriveId": "wopan-target",
"teaserEnabled": true
}`))
rr = httptest.NewRecorder()
srv.handleUpsertCrawler(rr, req)
if rr.Code != http.StatusOK {
t.Fatalf("enable teaser status = %d, body = %s", rr.Code, rr.Body.String())
}
got, err = cat.GetDrive(ctx, "crawler-upload")
if err != nil {
t.Fatalf("get crawler after teaser enable: %v", err)
}
if !got.TeaserEnabled {
t.Fatal("teaserEnabled after explicit enable = false, want true")
}
if teaserCallbackID != "crawler-upload" || !teaserCallbackEnabled {
t.Fatalf("teaser callback = %q/%v, want crawler-upload/true", teaserCallbackID, teaserCallbackEnabled)
}
req = httptest.NewRequest(http.MethodPost, "/admin/api/crawlers", strings.NewReader(`{
"id": "crawler-upload",
@@ -1704,6 +1791,94 @@ func TestHandleWopanQRStatus(t *testing.T) {
}
}
func TestHandleGuangYaPanQRStart(t *testing.T) {
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
if r.URL.Path != "/v1/auth/device/code" {
http.NotFound(w, r)
return
}
var body map[string]any
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
t.Fatalf("decode body: %v", err)
}
if body["scope"] != "user" {
t.Fatalf("scope = %#v, want user", body["scope"])
}
_ = json.NewEncoder(w).Encode(map[string]any{
"device_code": "device-1",
"verification_uri_complete": "https://account.guangyapan.example/device?code=abc",
"interval": 5,
"expires_in": 300,
})
}))
t.Cleanup(upstream.Close)
req := httptest.NewRequest(http.MethodPost, "/admin/api/drives/guangyapan/qr", nil)
rr := httptest.NewRecorder()
(&AdminServer{GuangYaPanAccountBaseURL: upstream.URL}).handleGuangYaPanQRStart(rr, req)
if rr.Code != http.StatusOK {
t.Fatalf("status = %d, body = %s", rr.Code, rr.Body.String())
}
var got struct {
DeviceCode string `json:"deviceCode"`
QRCodeURL string `json:"qrCodeUrl"`
QRImageDataURL string `json:"qrImageDataUrl"`
}
if err := json.NewDecoder(rr.Body).Decode(&got); err != nil {
t.Fatalf("decode: %v", err)
}
if got.DeviceCode != "device-1" || got.QRCodeURL != "https://account.guangyapan.example/device?code=abc" {
t.Fatalf("response = %#v", got)
}
if !strings.HasPrefix(got.QRImageDataURL, "data:image/png;base64,") {
t.Fatalf("qr image = %q", got.QRImageDataURL)
}
}
func TestHandleGuangYaPanQRStatus(t *testing.T) {
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
if r.URL.Path != "/v1/auth/token" {
http.NotFound(w, r)
return
}
var body map[string]any
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
t.Fatalf("decode body: %v", err)
}
if body["device_code"] != "device-1" {
t.Fatalf("device_code = %#v, want device-1", body["device_code"])
}
_ = json.NewEncoder(w).Encode(map[string]any{
"access_token": "access-1",
"refresh_token": "refresh-1",
"token_type": "Bearer",
})
}))
t.Cleanup(upstream.Close)
req := httptest.NewRequest(http.MethodGet, "/admin/api/drives/guangyapan/qr/status?deviceCode=device-1", nil)
rr := httptest.NewRecorder()
(&AdminServer{GuangYaPanAccountBaseURL: upstream.URL}).handleGuangYaPanQRStatus(rr, req)
if rr.Code != http.StatusOK {
t.Fatalf("status = %d, body = %s", rr.Code, rr.Body.String())
}
var got struct {
State string `json:"state"`
AccessToken string `json:"accessToken"`
RefreshToken string `json:"refreshToken"`
}
if err := json.NewDecoder(rr.Body).Decode(&got); err != nil {
t.Fatalf("decode: %v", err)
}
if got.State != "success" || got.AccessToken != "access-1" || got.RefreshToken != "refresh-1" {
t.Fatalf("response = %#v", got)
}
}
func TestHandleTestCrawlerScriptRunsImportedScript(t *testing.T) {
if _, err := exec.LookPath("python3"); err != nil {
t.Skip("python3 is required for crawler script dry-run")
@@ -1798,7 +1973,8 @@ func TestHandleListDrivesIncludesGoogleDriveOnlineAPIMode(t *testing.T) {
Name: "Google Legacy",
RootID: "root",
Credentials: map[string]string{
"refresh_token": "legacy-refresh",
"refresh_token": "legacy-refresh",
"api_url_address": "https://openlist-api.example/googleui/renewapi",
},
Status: "ok",
},
@@ -1829,15 +2005,18 @@ func TestHandleListDrivesIncludesGoogleDriveOnlineAPIMode(t *testing.T) {
}
var got []struct {
ID string `json:"id"`
GoogleDriveUseOnlineAPI bool `json:"googleDriveUseOnlineAPI"`
ID string `json:"id"`
GoogleDriveUseOnlineAPI bool `json:"googleDriveUseOnlineAPI"`
GoogleDriveOpenListAPIURL string `json:"googleDriveOpenListApiUrl"`
}
if err := json.NewDecoder(rr.Body).Decode(&got); err != nil {
t.Fatalf("decode: %v", err)
}
byID := map[string]bool{}
byAPIURL := map[string]string{}
for _, d := range got {
byID[d.ID] = d.GoogleDriveUseOnlineAPI
byAPIURL[d.ID] = d.GoogleDriveOpenListAPIURL
}
if !byID["google-legacy"] {
t.Fatalf("legacy google drive use_online_api = false, want true")
@@ -1845,6 +2024,9 @@ func TestHandleListDrivesIncludesGoogleDriveOnlineAPIMode(t *testing.T) {
if byID["google-oauth"] {
t.Fatalf("oauth google drive use_online_api = true, want false")
}
if byAPIURL["google-legacy"] != "https://openlist-api.example/googleui/renewapi" {
t.Fatalf("legacy google drive openlist api url = %q, want custom URL", byAPIURL["google-legacy"])
}
}
func TestHandleListDrivesIncludesTeaserCounts(t *testing.T) {
@@ -2373,6 +2555,80 @@ func TestHandleAdminListVideosPaginates(t *testing.T) {
}
}
func TestHandleAdminListVideosMarksActivePreviewGeneration(t *testing.T) {
ctx := context.Background()
cat, err := catalog.Open(t.TempDir() + "/catalog.db")
if err != nil {
t.Fatalf("open catalog: %v", err)
}
t.Cleanup(func() {
if err := cat.Close(); err != nil {
t.Fatalf("close catalog: %v", err)
}
})
now := time.Now()
for _, v := range []*catalog.Video{
{
ID: "active-video",
DriveID: "OneDrive",
FileID: "active-file",
Title: "Active video",
PreviewStatus: "ready",
PublishedAt: now,
CreatedAt: now,
UpdatedAt: now,
},
{
ID: "idle-video",
DriveID: "OneDrive",
FileID: "idle-file",
Title: "Idle video",
PreviewStatus: "ready",
PublishedAt: now.Add(-time.Hour),
CreatedAt: now,
UpdatedAt: now,
},
} {
if err := cat.UpsertVideo(ctx, v); err != nil {
t.Fatalf("seed video %s: %v", v.ID, err)
}
}
req := httptest.NewRequest(http.MethodGet, "/admin/api/videos?driveId=OneDrive", nil)
rr := httptest.NewRecorder()
(&AdminServer{
Catalog: cat,
GetPreviewGenerationVideoIDs: func() map[string]bool {
return map[string]bool{"active-video": true}
},
}).handleAdminListVideos(rr, req)
if rr.Code != http.StatusOK {
t.Fatalf("status = %d, body = %s", rr.Code, rr.Body.String())
}
var got struct {
Items []catalog.Video `json:"items"`
Total int `json:"total"`
}
if err := json.NewDecoder(rr.Body).Decode(&got); err != nil {
t.Fatalf("decode: %v", err)
}
if got.Total != 2 || len(got.Items) != 2 {
t.Fatalf("response total/items = %d/%d, want 2/2", got.Total, len(got.Items))
}
statusByID := map[string]string{}
for _, item := range got.Items {
statusByID[item.ID] = item.PreviewStatus
}
if statusByID["active-video"] != "generating" {
t.Fatalf("active status = %q, want generating", statusByID["active-video"])
}
if statusByID["idle-video"] != "ready" {
t.Fatalf("idle status = %q, want ready", statusByID["idle-video"])
}
}
func TestHandleRegenAllPreviewsInvokesHook(t *testing.T) {
called := false
server := &AdminServer{
+44 -22
View File
@@ -55,12 +55,16 @@ type Server struct {
LocalDir string
UploadDir string
OnVideoUploaded func(*catalog.Video)
// OnHideVideo 处理前台「不再展示」。隐藏机制已废弃,改走拉黑逻辑:
// 删除库中记录 + 本地封面/预览,保留网盘源文件,并写黑名单墓碑
// (扫盘不再入库)。未注入时回退为旧的 hidden 标记。
OnHideVideo func(ctx context.Context, videoID string) error
tagCacheMu sync.Mutex
tagCacheUntil time.Time
tagCache []TagDTO
// GetTheme 返回当前生效的主题("dark" | "pink")。前台 /api/settings/theme 用,
// GetTheme 返回当前生效的主题("dark" | "pink" | "sky")。前台 /api/settings/theme 用,
// 不需要登录。无注入时返回 "dark"。
GetTheme func() string
}
@@ -156,11 +160,11 @@ func (s *Server) RegisterRoutes(r chi.Router, a *auth.Authenticator) {
}
// handleGetTheme 返回当前生效的主题。无需登录。响应永远是
// {"theme": "dark"} 或 {"theme": "pink"},便于前端无脑解析。
// {"theme": "dark" | "pink" | "sky"},便于前端无脑解析。
func (s *Server) handleGetTheme(w http.ResponseWriter, r *http.Request) {
theme := "dark"
if s.GetTheme != nil {
if v := s.GetTheme(); v == "pink" || v == "dark" {
if v := s.GetTheme(); v == "pink" || v == "dark" || v == "sky" {
theme = v
}
}
@@ -526,11 +530,9 @@ func (s *Server) handleTags(w http.ResponseWriter, r *http.Request) {
}
// shortsNextReq 客户端把当前轮已看过的 video id 列表传上来。
// PreferredFromVideoID 来自短视频页最近一次点赞成功的视频,用于优先推荐相似标签。
type shortsNextReq struct {
SeenIDs []string `json:"seenIds"`
Count int `json:"count"`
PreferredFromVideoID string `json:"preferredFromVideoId"`
SeenIDs []string `json:"seenIds"`
Count int `json:"count"`
}
// ShortsItemDTO 是短视频流单条的精简结构。比 VideoDTO 多 videoSrc / poster
@@ -548,8 +550,8 @@ type ShortsItemDTO struct {
// - 服务器从未在 seenIds 中的可见视频里随机抽至多 count 条返回
// - 当返回数量 < count 且小于全库可见总数时,说明本轮即将结束,
// 返回 roundComplete=true,前端应在用户看完返回的这些后清空本地已看记录开新一轮
// - 当 seenIds 已经覆盖全库时,本接口直接返回新一轮的随机一批
// seenIds=[] 即可让客户端在轮次完成后重新开始
// - 当 seenIds 真实覆盖当前全部可见视频时,本接口直接返回新一轮的随机一批
// 不能仅看 seenIds 长度,里面可能有隐藏、删除或历史脏 ID
func (s *Server) handleShortsNext(w http.ResponseWriter, r *http.Request) {
var body shortsNextReq
if err := json.NewDecoder(r.Body).Decode(&body); err != nil && !errors.Is(err, io.EOF) {
@@ -570,22 +572,18 @@ func (s *Server) handleShortsNext(w http.ResponseWriter, r *http.Request) {
return
}
// 如果客户端已看记录已经 ≥ 全库,则视为新一轮,直接忽略 seenIds
exclude := body.SeenIDs
if total > 0 && len(exclude) >= total {
exclude = nil
}
var items []*catalog.Video
if strings.TrimSpace(body.PreferredFromVideoID) != "" {
items, err = s.Catalog.RandomVideosForPreferredVideoExcluding(r.Context(), body.PreferredFromVideoID, exclude, count)
} else {
items, err = s.Catalog.RandomVideosExcluding(r.Context(), exclude, count)
}
items, err := s.Catalog.RandomVideosExcluding(r.Context(), body.SeenIDs, count)
if err != nil {
writeErr(w, http.StatusInternalServerError, err)
return
}
if total > 0 && len(items) == 0 && len(body.SeenIDs) > 0 {
items, err = s.Catalog.RandomVideosExcluding(r.Context(), nil, count)
if err != nil {
writeErr(w, http.StatusInternalServerError, err)
return
}
}
// 注入 sourceLabel 以便前端展示来源网盘
driveLabels := make(map[string]string)
@@ -687,7 +685,14 @@ func (s *Server) handleView(w http.ResponseWriter, r *http.Request) {
func (s *Server) handleHideVideo(w http.ResponseWriter, r *http.Request) {
id := routeParam(r, "id")
if err := s.Catalog.HideVideo(r.Context(), id); err != nil {
var err error
if s.OnHideVideo != nil {
// 走拉黑逻辑:删记录 + 删本地封面/预览 + 写墓碑,保留网盘源文件。
err = s.OnHideVideo(r.Context(), id)
} else {
err = s.Catalog.HideVideo(r.Context(), id)
}
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
writeErr(w, http.StatusNotFound, err)
return
@@ -970,6 +975,15 @@ func thumbnailURL(v *catalog.Video) string {
return base + "?v=" + strconv.FormatInt(v.UpdatedAt.UnixMilli(), 10)
}
// transcodedSource 在视频有就绪的浏览器兼容性转码产物时返回产物的播放地址。
// 产物和原始文件在同一个 drive 上,走同一条 /p/stream 代理/302 链路。
func transcodedSource(v *catalog.Video) (string, bool) {
if v.TranscodeStatus == "ready" && v.TranscodedFileID != "" && v.DriveID != localUploadDriveID {
return fmt.Sprintf("/p/stream/%s/%s", pathSegment(v.DriveID), pathSegment(v.TranscodedFileID)), true
}
return "", false
}
func (s *Server) videoSource(v *catalog.Video) string {
if v.DriveID == localUploadDriveID {
return "/p/upload/" + pathSegment(v.ID)
@@ -982,6 +996,9 @@ func (s *Server) videoSource(v *catalog.Video) string {
}
}
}
if src, ok := transcodedSource(v); ok {
return src
}
return fmt.Sprintf("/p/stream/%s/%s", pathSegment(v.DriveID), pathSegment(v.FileID))
}
@@ -991,6 +1008,9 @@ func videoSource(v *catalog.Video) string {
if v.DriveID == localUploadDriveID {
return "/p/upload/" + pathSegment(v.ID)
}
if src, ok := transcodedSource(v); ok {
return src
}
return fmt.Sprintf("/p/stream/%s/%s", pathSegment(v.DriveID), pathSegment(v.FileID))
}
@@ -1048,6 +1068,8 @@ func driveKindLabel(kind string) string {
return "PikPak"
case "wopan":
return "联通网盘"
case "guangyapan":
return "光鸭网盘"
case "onedrive":
return "OneDrive"
case "googledrive":
+73 -6
View File
@@ -810,7 +810,7 @@ func TestHandleTagsReturnsUnifiedTagPool(t *testing.T) {
}
}
func TestHandleShortsNextUsesPreferredVideoLeastPopulatedTag(t *testing.T) {
func TestHandleShortsNextReturnsRandomBatchExcludingSeen(t *testing.T) {
ctx := context.Background()
cat, err := catalog.Open(t.TempDir() + "/catalog.db")
if err != nil {
@@ -834,7 +834,7 @@ func TestHandleShortsNextUsesPreferredVideoLeastPopulatedTag(t *testing.T) {
}
}
req := httptest.NewRequest(http.MethodPost, "/api/shorts/next", strings.NewReader(`{"seenIds":["current"],"count":3,"preferredFromVideoId":"current"}`))
req := httptest.NewRequest(http.MethodPost, "/api/shorts/next", strings.NewReader(`{"seenIds":["current"],"count":3}`))
rr := httptest.NewRecorder()
(&Server{Catalog: cat}).handleShortsNext(rr, req)
@@ -857,10 +857,7 @@ func TestHandleShortsNextUsesPreferredVideoLeastPopulatedTag(t *testing.T) {
t.Fatalf("total = %d, want 4", got.Total)
}
if got.RoundComplete {
t.Fatalf("roundComplete = true, want false with fallback-filled batch")
}
if !containsString(ids, "rare-1") {
t.Fatalf("ids = %#v, want rare-1 from least populated tag", ids)
t.Fatalf("roundComplete = true, want false with a full remaining batch")
}
if containsString(ids, "current") {
t.Fatalf("ids = %#v, should exclude current", ids)
@@ -868,6 +865,76 @@ func TestHandleShortsNextUsesPreferredVideoLeastPopulatedTag(t *testing.T) {
if len(ids) != 3 {
t.Fatalf("ids = %#v, want 3 items", ids)
}
for _, want := range []string{"common-1", "common-2", "rare-1"} {
if !containsString(ids, want) {
t.Fatalf("ids = %#v, want remaining id %s", ids, want)
}
}
}
func TestHandleShortsNextDoesNotResetForStaleSeenIDs(t *testing.T) {
ctx := context.Background()
cat, err := catalog.Open(t.TempDir() + "/catalog.db")
if err != nil {
t.Fatalf("open catalog: %v", err)
}
t.Cleanup(func() {
if err := cat.Close(); err != nil {
t.Fatalf("close catalog: %v", err)
}
})
now := time.Now()
for _, v := range []*catalog.Video{
{ID: "seen-1", DriveID: "drive", FileID: "f-seen-1", Title: "seen 1", PublishedAt: now, CreatedAt: now, UpdatedAt: now},
{ID: "fresh-1", DriveID: "drive", FileID: "f-fresh-1", Title: "fresh 1", PublishedAt: now, CreatedAt: now, UpdatedAt: now},
{ID: "fresh-2", DriveID: "drive", FileID: "f-fresh-2", Title: "fresh 2", PublishedAt: now, CreatedAt: now, UpdatedAt: now},
{ID: "hidden-1", DriveID: "drive", FileID: "f-hidden-1", Title: "hidden 1", PublishedAt: now, CreatedAt: now, UpdatedAt: now},
} {
if err := cat.UpsertVideo(ctx, v); err != nil {
t.Fatalf("seed %s: %v", v.ID, err)
}
}
if err := cat.HideVideo(ctx, "hidden-1"); err != nil {
t.Fatalf("hide hidden-1: %v", err)
}
req := httptest.NewRequest(http.MethodPost, "/api/shorts/next", strings.NewReader(`{"seenIds":["seen-1","hidden-1","deleted-stale"],"count":3}`))
rr := httptest.NewRecorder()
(&Server{Catalog: cat}).handleShortsNext(rr, req)
if rr.Code != http.StatusOK {
t.Fatalf("status = %d, body = %s", rr.Code, rr.Body.String())
}
var got struct {
Items []ShortsItemDTO `json:"items"`
Total int `json:"total"`
RoundComplete bool `json:"roundComplete"`
}
if err := json.NewDecoder(rr.Body).Decode(&got); err != nil {
t.Fatalf("decode: %v", err)
}
ids := make([]string, 0, len(got.Items))
for _, item := range got.Items {
ids = append(ids, item.ID)
}
if got.Total != 3 {
t.Fatalf("total = %d, want 3", got.Total)
}
if !got.RoundComplete {
t.Fatalf("roundComplete = false, want true after returning all unviewed visible videos")
}
if containsString(ids, "seen-1") || containsString(ids, "hidden-1") {
t.Fatalf("ids = %#v, should not reset and return seen or hidden videos", ids)
}
for _, want := range []string{"fresh-1", "fresh-2"} {
if !containsString(ids, want) {
t.Fatalf("ids = %#v, want %s", ids, want)
}
}
if len(ids) != 2 {
t.Fatalf("ids = %#v, want exactly the two unviewed visible videos", ids)
}
}
func TestHandleUpdateVideoTagsRejectsUnknownTags(t *testing.T) {
+263 -202
View File
@@ -51,38 +51,45 @@ func (c *Catalog) Close() error { return c.db.Close() }
// ---------- Video ----------
type Video struct {
ID string `json:"id"`
DriveID string `json:"driveId"`
FileID string `json:"fileId"`
FileName string `json:"fileName"`
ContentHash string `json:"contentHash"`
SampledSHA256 string `json:"sampledSha256"`
FingerprintStatus string `json:"fingerprintStatus"`
FingerprintError string `json:"fingerprintError"`
ParentID string `json:"parentId"`
Title string `json:"title"`
Author string `json:"author"`
Tags []string `json:"tags"`
DurationSeconds int `json:"durationSeconds"`
Size int64 `json:"size"`
Ext string `json:"ext"`
Quality string `json:"quality"`
ThumbnailURL string `json:"thumbnailUrl"`
PreviewFileID string `json:"previewFileId"`
PreviewLocal string `json:"previewLocal"`
PreviewStatus string `json:"previewStatus"`
Views int `json:"views"`
Favorites int `json:"favorites"`
Comments int `json:"comments"`
Likes int `json:"likes"`
Dislikes int `json:"dislikes"`
Category string `json:"category"`
Hidden bool `json:"hidden"`
Badges []string `json:"badges"`
Description string `json:"description"`
PublishedAt time.Time `json:"publishedAt"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
ID string `json:"id"`
DriveID string `json:"driveId"`
FileID string `json:"fileId"`
FileName string `json:"fileName"`
ContentHash string `json:"contentHash"`
SampledSHA256 string `json:"sampledSha256"`
FingerprintStatus string `json:"fingerprintStatus"`
FingerprintError string `json:"fingerprintError"`
ParentID string `json:"parentId"`
Title string `json:"title"`
Author string `json:"author"`
Tags []string `json:"tags"`
DurationSeconds int `json:"durationSeconds"`
Size int64 `json:"size"`
Ext string `json:"ext"`
Quality string `json:"quality"`
ThumbnailURL string `json:"thumbnailUrl"`
PreviewFileID string `json:"previewFileId"`
PreviewLocal string `json:"previewLocal"`
PreviewStatus string `json:"previewStatus"`
// TranscodeStatus:浏览器兼容性转码状态。
// ''=未检测 / pending=已入队 / ready=已转码 / skipped=无需转码 / failed=失败。
TranscodeStatus string `json:"transcodeStatus"`
TranscodeError string `json:"transcodeError"`
TranscodedFileID string `json:"transcodedFileId"`
TranscodedSize int64 `json:"transcodedSize"`
Views int `json:"views"`
LastViewedAt time.Time `json:"lastViewedAt"`
Favorites int `json:"favorites"`
Comments int `json:"comments"`
Likes int `json:"likes"`
Dislikes int `json:"dislikes"`
Category string `json:"category"`
Hidden bool `json:"hidden"`
Badges []string `json:"badges"`
Description string `json:"description"`
PublishedAt time.Time `json:"publishedAt"`
CreatedAt time.Time `json:"createdAt"`
UpdatedAt time.Time `json:"updatedAt"`
}
func (c *Catalog) UpsertVideo(ctx context.Context, v *Video) error {
@@ -106,13 +113,13 @@ INSERT INTO videos (
id, drive_id, file_id, file_name, content_hash, sampled_sha256, fingerprint_status, fingerprint_error, parent_id, title, author, tags,
duration_seconds, size_bytes, ext, quality, thumbnail_url, thumbnail_status,
preview_file_id, preview_local, preview_status,
views, favorites, comments, likes, dislikes,
views, last_viewed_at, favorites, comments, likes, dislikes,
category, hidden, badges, description, published_at, created_at, updated_at
) VALUES (
?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?, CASE WHEN COALESCE(?, '') != '' THEN 'ready' ELSE 'pending' END,
?, ?, ?,
?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?, ?
)
ON CONFLICT(id) DO UPDATE SET
@@ -163,7 +170,7 @@ ON CONFLICT(id) DO UPDATE SET
v.ID, v.DriveID, v.FileID, v.FileName, v.ContentHash, v.SampledSHA256, fingerprintStatus, v.FingerprintError, v.ParentID, v.Title, v.Author, string(tagsJSON),
v.DurationSeconds, v.Size, v.Ext, v.Quality, v.ThumbnailURL, v.ThumbnailURL,
v.PreviewFileID, v.PreviewLocal, nullableStatus(v.PreviewStatus),
v.Views, v.Favorites, v.Comments, v.Likes, v.Dislikes,
v.Views, unixMilliOrZero(v.LastViewedAt), v.Favorites, v.Comments, v.Likes, v.Dislikes,
v.Category, boolToInt(v.Hidden), string(badgesJSON), v.Description,
v.PublishedAt.UnixMilli(), v.CreatedAt.UnixMilli(), v.UpdatedAt.UnixMilli(),
)
@@ -190,6 +197,84 @@ func (c *Catalog) UpdatePreview(ctx context.Context, id, previewLocal, status st
return err
}
// transcodeCandidateWhereSQL 圈定"可能需要浏览器兼容性转码"的视频:
// mp4/webm/m4v 默认浏览器可播不进候选;strm 是远程引用没有本体。
// 其余扩展名都先入候选,由转码 worker probe 实际编码后决定转码还是跳过
// skipped)。failed 也保留在候选里,重新点开始转码时会自动重试。
const transcodeCandidateWhereSQL = `COALESCE(ext, '') NOT IN ('mp4', 'webm', 'm4v', 'strm')
AND COALESCE(transcode_status, '') IN ('', 'pending', 'failed')`
// ListTranscodeCandidates 列出某盘所有转码候选视频。limit<=0 表示不限制。
func (c *Catalog) ListTranscodeCandidates(ctx context.Context, driveID string, limit int) ([]*Video, error) {
query := `SELECT ` + allVideoCols + ` FROM videos
WHERE drive_id = ? AND ` + transcodeCandidateWhereSQL + `
ORDER BY created_at ASC, id ASC`
args := []any{driveID}
if limit > 0 {
query += ` LIMIT ?`
args = append(args, limit)
}
rows, err := c.db.QueryContext(ctx, query, args...)
if err != nil {
return nil, err
}
defer rows.Close()
var out []*Video
for rows.Next() {
v, err := scanVideo(rows)
if err != nil {
return nil, err
}
out = append(out, v)
}
return out, rows.Err()
}
// UpdateVideoTranscode 写回单条视频的转码结果。
// status=ready 时 transcodedFileID/transcodedSize 指向转码产物;
// 其它 status 调用方应传空值,本函数会按传入值原样覆盖。
func (c *Catalog) UpdateVideoTranscode(ctx context.Context, id, status, errMsg, transcodedFileID string, transcodedSize int64) error {
_, err := c.db.ExecContext(ctx,
`UPDATE videos SET transcode_status = ?, transcode_error = ?, transcoded_file_id = ?, transcoded_size = ?, updated_at = ? WHERE id = ?`,
status, errMsg, transcodedFileID, transcodedSize, time.Now().UnixMilli(), id)
return err
}
// DriveTranscodeCounts 是单盘的转码进度统计。
type DriveTranscodeCounts struct {
// Pending 是仍在候选集合里、还没有出结果的数量(含从未检测过的)。
Pending int
Ready int
Failed int
Skipped int
}
func (c *Catalog) CountTranscodesByDrive(ctx context.Context) (map[string]DriveTranscodeCounts, error) {
rows, err := c.db.QueryContext(ctx, `
SELECT drive_id,
COUNT(CASE WHEN COALESCE(ext, '') NOT IN ('mp4', 'webm', 'm4v', 'strm')
AND COALESCE(transcode_status, '') IN ('', 'pending') THEN 1 END) AS pending_count,
COUNT(CASE WHEN COALESCE(transcode_status, '') = 'ready' THEN 1 END) AS ready_count,
COUNT(CASE WHEN COALESCE(transcode_status, '') = 'failed' THEN 1 END) AS failed_count,
COUNT(CASE WHEN COALESCE(transcode_status, '') = 'skipped' THEN 1 END) AS skipped_count
FROM videos
GROUP BY drive_id`)
if err != nil {
return nil, err
}
defer rows.Close()
out := make(map[string]DriveTranscodeCounts)
for rows.Next() {
var driveID string
var counts DriveTranscodeCounts
if err := rows.Scan(&driveID, &counts.Pending, &counts.Ready, &counts.Failed, &counts.Skipped); err != nil {
return nil, err
}
out[driveID] = counts
}
return out, rows.Err()
}
func (c *Catalog) HideVideo(ctx context.Context, id string) error {
res, err := c.db.ExecContext(ctx,
`UPDATE videos SET hidden = 1, updated_at = ? WHERE id = ?`,
@@ -203,6 +288,27 @@ func (c *Catalog) HideVideo(ctx context.Context, id string) error {
return nil
}
// ListHiddenVideos 返回所有被标记隐藏(hidden=1)的视频。
// 仅用于一次性把历史「隐藏」视频迁移为黑名单墓碑——隐藏机制已废弃,
// 前台「不再展示」改走拉黑逻辑。
func (c *Catalog) ListHiddenVideos(ctx context.Context) ([]*Video, error) {
rows, err := c.db.QueryContext(ctx,
`SELECT `+allVideoCols+` FROM videos WHERE COALESCE(hidden, 0) = 1`)
if err != nil {
return nil, err
}
defer rows.Close()
var out []*Video
for rows.Next() {
v, err := scanVideo(rows)
if err != nil {
return nil, err
}
out = append(out, v)
}
return out, rows.Err()
}
// MigrateVideoToDrive 把 catalog 里 id=videoID 这条视频迁移到另一个 drive。
// 用于 spider91 → PikPak 的迁移:上传成功后改写 drive_id / file_id /
// content_hash,保留视频自身的 idspider91-<driveID>-<sourceID>),这样
@@ -318,9 +424,10 @@ func (c *Catalog) IncrementView(ctx context.Context, id string) (int, error) {
return 0, err
}
defer tx.Rollback()
now := time.Now().UnixMilli()
res, err := tx.ExecContext(ctx,
`UPDATE videos SET views = views + 1, updated_at = ? WHERE id = ?`,
time.Now().UnixMilli(), id)
`UPDATE videos SET views = views + 1, last_viewed_at = ?, updated_at = ? WHERE id = ?`,
now, now, id)
if err != nil {
return 0, err
}
@@ -346,6 +453,10 @@ type VideoMetaPatch struct {
Category string
ContentHash string
FileName string
Title string
TitleSet bool
Author string
AuthorSet bool
Tags []string
TagsSet bool
}
@@ -395,6 +506,14 @@ func (c *Catalog) UpdateVideoMeta(ctx context.Context, id string, p VideoMetaPat
parts = append(parts, "file_name = ?")
args = append(args, p.FileName)
}
if p.TitleSet {
parts = append(parts, "title = ?")
args = append(args, p.Title)
}
if p.AuthorSet {
parts = append(parts, "author = ?")
args = append(args, p.Author)
}
if p.TagsSet {
tagsJSON, _ := json.Marshal(p.Tags)
parts = append(parts, "tags = ?")
@@ -898,6 +1017,92 @@ func (c *Catalog) DeleteVideo(ctx context.Context, id string) error {
return tx.Commit()
}
// DeletedVideo 是黑名单(墓碑)表里的一条记录。原始视频行已删除,
// 这里只保留扫盘去重和后台展示需要的最小字段;没有 title/封面/作者。
type DeletedVideo struct {
ID string `json:"id"`
DriveID string `json:"driveId"`
FileID string `json:"fileId"`
FileName string `json:"fileName"`
Size int64 `json:"size"`
DeletedAt int64 `json:"deletedAt"` // unix 毫秒
}
// ListDeletedVideos 分页列出黑名单视频,按拉黑时间倒序。
// keyword 非空时按文件名模糊匹配。
func (c *Catalog) ListDeletedVideos(ctx context.Context, keyword string, page, size int) ([]*DeletedVideo, int, error) {
if size <= 0 {
size = 50
}
if page <= 0 {
page = 1
}
var where []string
var args []any
if kw := strings.TrimSpace(keyword); kw != "" {
where = append(where, "file_name LIKE ?")
args = append(args, "%"+kw+"%")
}
whereSQL := ""
if len(where) > 0 {
whereSQL = " WHERE " + strings.Join(where, " AND ")
}
var total int
if err := c.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM deleted_videos`+whereSQL, args...).Scan(&total); err != nil {
return nil, 0, err
}
offset := (page - 1) * size
rows, err := c.db.QueryContext(ctx,
`SELECT id, COALESCE(drive_id, ''), COALESCE(file_id, ''), COALESCE(file_name, ''), COALESCE(size_bytes, 0), deleted_at
FROM deleted_videos`+whereSQL+`
ORDER BY deleted_at DESC
LIMIT ? OFFSET ?`,
append(args, size, offset)...)
if err != nil {
return nil, 0, err
}
defer rows.Close()
var out []*DeletedVideo
for rows.Next() {
v := &DeletedVideo{}
if err := rows.Scan(&v.ID, &v.DriveID, &v.FileID, &v.FileName, &v.Size, &v.DeletedAt); err != nil {
return nil, 0, err
}
out = append(out, v)
}
return out, total, rows.Err()
}
// RemoveDeletedVideo 把视频移出黑名单(删除墓碑)。移除后该视频会在
// 下次扫盘/凌晨流水线时被重新发现并入库,本函数不主动触发扫描。
func (c *Catalog) RemoveDeletedVideo(ctx context.Context, id string) error {
res, err := c.db.ExecContext(ctx, `DELETE FROM deleted_videos WHERE id = ?`, id)
if err != nil {
return err
}
if rows, err := res.RowsAffected(); err == nil && rows == 0 {
return sql.ErrNoRows
}
return nil
}
// VideoManagementCounts 返回后台视频管理两个标签的计数:
// current=当前可见(与「当前视频」页一致的去重+在线盘+hidden=0 口径),
// blacklisted=黑名单墓碑总数。
func (c *Catalog) VideoManagementCounts(ctx context.Context) (current, blacklisted int, err error) {
currentSQL := `SELECT COUNT(*) FROM videos WHERE COALESCE(hidden, 0) = 0 AND ` + activeDriveWhereSQL + ` AND ` + uniqueVideoWhereSQL
if err = c.db.QueryRowContext(ctx, currentSQL).Scan(&current); err != nil {
return 0, 0, err
}
if err = c.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM deleted_videos`).Scan(&blacklisted); err != nil {
return 0, 0, err
}
return current, blacklisted, nil
}
func (c *Catalog) IsVideoDeleted(ctx context.Context, id string) (bool, error) {
id = strings.TrimSpace(id)
if id == "" {
@@ -1161,7 +1366,7 @@ type ListParams struct {
DriveID string
Tag string
Category string
Sort string // latest | hot | week | long
Sort string // latest | hot | recent
ThumbnailReadyOnly bool
PreferReadyThumbnails bool
SkipTotal bool
@@ -1216,10 +1421,8 @@ func (c *Catalog) ListVideos(ctx context.Context, p ListParams) ([]*Video, int,
case "hot":
// 热度 = 点赞数,点赞相同按最新
orderBy = " ORDER BY " + readyOrderPrefix + "likes DESC, published_at DESC"
case "week":
orderBy = " ORDER BY " + readyOrderPrefix + "likes DESC"
case "long":
orderBy = " ORDER BY " + readyOrderPrefix + "duration_seconds DESC"
case "recent":
orderBy = " ORDER BY " + readyOrderPrefix + "COALESCE(last_viewed_at, 0) DESC, published_at DESC"
}
var total int
@@ -1342,160 +1545,6 @@ func cleanVideoIDs(ids []string) []string {
return cleaned
}
func cleanTagLabels(labels []string) []string {
seen := make(map[string]struct{}, len(labels))
cleaned := make([]string, 0, len(labels))
for _, label := range labels {
label = strings.TrimSpace(label)
if label == "" {
continue
}
key := strings.ToLower(label)
if _, ok := seen[key]; ok {
continue
}
seen[key] = struct{}{}
cleaned = append(cleaned, label)
}
return cleaned
}
func (c *Catalog) LeastPopulatedVisibleUniqueTag(ctx context.Context, labels []string) (string, error) {
cleaned := cleanTagLabels(labels)
bestLabel := ""
bestCount := 0
for _, label := range cleaned {
var count int
if err := c.db.QueryRowContext(ctx,
`SELECT COUNT(*)
FROM videos
WHERE COALESCE(hidden, 0) = 0
AND `+activeDriveWhereSQL+`
AND `+uniqueVideoWhereSQL+`
AND EXISTS (
SELECT 1
FROM video_tags vt
JOIN tags t ON t.id = vt.tag_id
WHERE vt.video_id = videos.id
AND t.label = ? COLLATE NOCASE
)`,
label,
).Scan(&count); err != nil {
return "", err
}
if count == 0 {
continue
}
if bestLabel == "" || count < bestCount {
bestLabel = label
bestCount = count
}
}
return bestLabel, nil
}
func (c *Catalog) RandomVideosByTagExcluding(ctx context.Context, tag string, excludeIDs []string, limit int) ([]*Video, error) {
if limit <= 0 {
return nil, nil
}
tag = strings.TrimSpace(tag)
if tag == "" {
return nil, nil
}
cleaned := cleanVideoIDs(excludeIDs)
args := make([]any, 0, len(cleaned)+2)
args = append(args, tag)
whereSQL := `WHERE COALESCE(hidden, 0) = 0
AND ` + activeDriveWhereSQL + `
AND ` + uniqueVideoWhereSQL + `
AND EXISTS (
SELECT 1
FROM video_tags vt
JOIN tags t ON t.id = vt.tag_id
WHERE vt.video_id = videos.id
AND t.label = ? COLLATE NOCASE
)`
if len(cleaned) > 0 {
placeholders := strings.Repeat("?,", len(cleaned))
placeholders = placeholders[:len(placeholders)-1]
whereSQL += " AND id NOT IN (" + placeholders + ")"
for _, id := range cleaned {
args = append(args, id)
}
}
args = append(args, limit)
rows, err := c.db.QueryContext(ctx,
`SELECT `+allVideoCols+` FROM videos `+whereSQL+`
ORDER BY RANDOM() LIMIT ?`,
args...,
)
if err != nil {
return nil, err
}
defer rows.Close()
var out []*Video
for rows.Next() {
v, err := scanVideo(rows)
if err != nil {
return nil, err
}
out = append(out, v)
}
if err := rows.Err(); err != nil {
return nil, err
}
return out, nil
}
func (c *Catalog) RandomVideosForPreferredVideoExcluding(ctx context.Context, preferredVideoID string, excludeIDs []string, limit int) ([]*Video, error) {
if limit <= 0 {
return nil, nil
}
preferredVideoID = strings.TrimSpace(preferredVideoID)
if preferredVideoID == "" {
return c.RandomVideosExcluding(ctx, excludeIDs, limit)
}
preferredExclude := append([]string{}, excludeIDs...)
preferredExclude = append(preferredExclude, preferredVideoID)
preferred, err := c.GetVideo(ctx, preferredVideoID)
if err != nil || preferred == nil || preferred.Hidden || len(preferred.Tags) == 0 {
return c.RandomVideosExcluding(ctx, preferredExclude, limit)
}
tag, err := c.LeastPopulatedVisibleUniqueTag(ctx, preferred.Tags)
if err != nil {
return nil, err
}
if tag == "" {
return c.RandomVideosExcluding(ctx, preferredExclude, limit)
}
items, err := c.RandomVideosByTagExcluding(ctx, tag, preferredExclude, limit)
if err != nil {
return nil, err
}
if len(items) >= limit {
return items, nil
}
mergedExclude := make([]string, 0, len(preferredExclude)+len(items))
mergedExclude = append(mergedExclude, preferredExclude...)
for _, item := range items {
if item != nil {
mergedExclude = append(mergedExclude, item.ID)
}
}
fallback, err := c.RandomVideosExcluding(ctx, mergedExclude, limit-len(items))
if err != nil {
return nil, err
}
return append(items, fallback...), nil
}
type DriveTeaserCounts struct {
Ready int
Pending int
@@ -1900,7 +1949,7 @@ type Drive struct {
Credentials map[string]string `json:"credentials,omitempty"`
Status string `json:"status"`
LastError string `json:"lastError,omitempty"`
// TeaserEnabled 控制是否给本盘生成预览视频/封面。
// TeaserEnabled 控制是否给本盘生成预览视频封面生成不受影响
// 替代早期的全局 preview.enabled 开关;新建 drive 时 UpsertDrive 默认置 true。
TeaserEnabled bool `json:"teaserEnabled"`
// SkipDirIDs 是用户在管理后台为该盘选定的"扫描跳过目录"集合(网盘侧的目录 fileID)。
@@ -1955,7 +2004,7 @@ func normalizeDriveRootFields(d *Drive) {
func normalizeDriveRootID(kind, rootID string) string {
rootID = strings.TrimSpace(rootID)
switch kind {
case "pikpak":
case "pikpak", "guangyapan":
if rootID == "0" {
return ""
}
@@ -2033,7 +2082,7 @@ func (c *Catalog) DeleteDrive(ctx context.Context, id string) error {
return err
}
// SetDriveTeaserEnabled 切换某盘的预览视频/封面生成开关。
// SetDriveTeaserEnabled 切换某盘的预览视频生成开关。
//
// 与 UpsertDrive 的区别:只动 teaser_enabled + updated_at 一列,不要求调用方
// 重传 kind / name / credentials 等容易踩坑的字段。
@@ -2165,7 +2214,8 @@ COALESCE(sampled_sha256, ''), COALESCE(fingerprint_status, 'pending'), COALESCE(
COALESCE(parent_id, ''), title, COALESCE(author, ''), COALESCE(tags, '[]'),
duration_seconds, size_bytes, COALESCE(ext, ''), COALESCE(quality, ''), COALESCE(thumbnail_url, ''),
COALESCE(preview_file_id, ''), COALESCE(preview_local, ''), COALESCE(preview_status, 'pending'),
views, favorites, comments, likes, dislikes,
COALESCE(transcode_status, ''), COALESCE(transcode_error, ''), COALESCE(transcoded_file_id, ''), COALESCE(transcoded_size, 0),
views, COALESCE(last_viewed_at, 0), favorites, comments, likes, dislikes,
COALESCE(category, ''), COALESCE(hidden, 0), COALESCE(badges, '[]'), COALESCE(description, ''),
published_at, created_at, updated_at
`
@@ -2228,7 +2278,7 @@ type rowScanner interface {
func scanVideo(row rowScanner) (*Video, error) {
v := &Video{}
var tagsJSON, badgesJSON string
var publishedAt, createdAt, updatedAt int64
var publishedAt, createdAt, updatedAt, lastViewedAt int64
var hidden int
err := row.Scan(
&v.ID, &v.DriveID, &v.FileID, &v.FileName, &v.ContentHash,
@@ -2236,7 +2286,8 @@ func scanVideo(row rowScanner) (*Video, error) {
&v.ParentID, &v.Title, &v.Author, &tagsJSON,
&v.DurationSeconds, &v.Size, &v.Ext, &v.Quality, &v.ThumbnailURL,
&v.PreviewFileID, &v.PreviewLocal, &v.PreviewStatus,
&v.Views, &v.Favorites, &v.Comments, &v.Likes, &v.Dislikes,
&v.TranscodeStatus, &v.TranscodeError, &v.TranscodedFileID, &v.TranscodedSize,
&v.Views, &lastViewedAt, &v.Favorites, &v.Comments, &v.Likes, &v.Dislikes,
&v.Category, &hidden, &badgesJSON, &v.Description,
&publishedAt, &createdAt, &updatedAt,
)
@@ -2249,6 +2300,9 @@ func scanVideo(row rowScanner) (*Video, error) {
v.PublishedAt = time.UnixMilli(publishedAt)
v.CreatedAt = time.UnixMilli(createdAt)
v.UpdatedAt = time.UnixMilli(updatedAt)
if lastViewedAt > 0 {
v.LastViewedAt = time.UnixMilli(lastViewedAt)
}
return v, nil
}
@@ -2256,6 +2310,13 @@ func normalizeContentHash(hash string) string {
return strings.ToLower(strings.TrimSpace(hash))
}
func unixMilliOrZero(t time.Time) int64 {
if t.IsZero() {
return 0
}
return t.UnixMilli()
}
func boolToInt(v bool) int {
if v {
return 1
+1
View File
@@ -58,6 +58,7 @@ func TestUpsertDriveDefaultsRootIDByKind(t *testing.T) {
}{
{id: "p115", kind: "p115", want: "0"},
{id: "pikpak", kind: "pikpak", want: ""},
{id: "guangyapan", kind: "guangyapan", want: ""},
{id: "onedrive", kind: "onedrive", want: "root"},
{id: "googledrive", kind: "googledrive", want: "root"},
{id: "localstorage", kind: "localstorage", want: "/"},
@@ -0,0 +1,97 @@
package catalog
import (
"context"
"testing"
"time"
)
func TestIncrementViewStoresLastViewedAt(t *testing.T) {
ctx := context.Background()
cat, err := Open(t.TempDir() + "/catalog.db")
if err != nil {
t.Fatalf("open catalog: %v", err)
}
t.Cleanup(func() {
if err := cat.Close(); err != nil {
t.Fatalf("close catalog: %v", err)
}
})
now := time.Now()
if err := cat.UpsertVideo(ctx, &Video{
ID: "video-1",
DriveID: "drive",
FileID: "file-1",
Title: "Video 1",
PublishedAt: now,
CreatedAt: now,
UpdatedAt: now,
}); err != nil {
t.Fatalf("seed video: %v", err)
}
if _, err := cat.IncrementView(ctx, "video-1"); err != nil {
t.Fatalf("increment view: %v", err)
}
got, err := cat.GetVideo(ctx, "video-1")
if err != nil {
t.Fatalf("get video: %v", err)
}
if got.Views != 1 {
t.Fatalf("views = %d, want 1", got.Views)
}
if got.LastViewedAt.IsZero() {
t.Fatal("last viewed time was not stored")
}
}
func TestListVideosRecentSortUsesLastViewedAt(t *testing.T) {
ctx := context.Background()
cat, err := Open(t.TempDir() + "/catalog.db")
if err != nil {
t.Fatalf("open catalog: %v", err)
}
t.Cleanup(func() {
if err := cat.Close(); err != nil {
t.Fatalf("close catalog: %v", err)
}
})
now := time.Now()
for _, v := range []*Video{
{ID: "old-view", DriveID: "drive", FileID: "old-view", Title: "Old View", PublishedAt: now.Add(3 * time.Hour), CreatedAt: now, UpdatedAt: now},
{ID: "recent-view", DriveID: "drive", FileID: "recent-view", Title: "Recent View", PublishedAt: now, CreatedAt: now, UpdatedAt: now},
{ID: "unviewed", DriveID: "drive", FileID: "unviewed", Title: "Unviewed", PublishedAt: now.Add(4 * time.Hour), CreatedAt: now, UpdatedAt: now},
} {
if err := cat.UpsertVideo(ctx, v); err != nil {
t.Fatalf("seed %s: %v", v.ID, err)
}
}
if _, err := cat.db.ExecContext(ctx,
`UPDATE videos SET last_viewed_at = CASE id
WHEN 'old-view' THEN ?
WHEN 'recent-view' THEN ?
ELSE 0
END`,
now.Add(-time.Hour).UnixMilli(),
now.Add(time.Hour).UnixMilli(),
); err != nil {
t.Fatalf("seed last_viewed_at: %v", err)
}
items, _, err := cat.ListVideos(ctx, ListParams{Sort: "recent", Page: 1, PageSize: 3})
if err != nil {
t.Fatalf("list recent videos: %v", err)
}
if len(items) != 3 {
t.Fatalf("items = %d, want 3", len(items))
}
got := []string{items[0].ID, items[1].ID, items[2].ID}
want := []string{"recent-view", "old-view", "unviewed"}
for i := range want {
if got[i] != want[i] {
t.Fatalf("recent order = %#v, want %#v", got, want)
}
}
}
+8 -3
View File
@@ -21,8 +21,13 @@ CREATE TABLE IF NOT EXISTS videos (
thumbnail_failures INTEGER DEFAULT 0, -- consecutive transient thumbnail generation failures
preview_file_id TEXT, -- deprecated: 旧版回写网盘后的预览视频 file id
preview_local TEXT, -- 本地预览视频路径(兜底)
preview_status TEXT DEFAULT 'pending', -- pending / ready / failed
preview_status TEXT DEFAULT 'pending', -- pending / ready / failed / disabled
transcode_status TEXT DEFAULT '', -- '' / pending / ready / skipped / failed(浏览器兼容性转码)
transcode_error TEXT DEFAULT '',
transcoded_file_id TEXT DEFAULT '', -- 转码产物在同一 drive 上的 fileID,播放源优先用它
transcoded_size INTEGER DEFAULT 0,
views INTEGER DEFAULT 0,
last_viewed_at INTEGER DEFAULT 0,
favorites INTEGER DEFAULT 0,
comments INTEGER DEFAULT 0,
likes INTEGER DEFAULT 0,
@@ -110,14 +115,14 @@ CREATE INDEX IF NOT EXISTS idx_crawler_seen_sources_drive
-- 网盘账户
CREATE TABLE IF NOT EXISTS drives (
id TEXT PRIMARY KEY,
kind TEXT NOT NULL, -- quark / p115 / p123 / pikpak / wopan / onedrive / googledrive / localstorage / spider91
kind TEXT NOT NULL, -- quark / p115 / p123 / pikpak / wopan / guangyapan / onedrive / googledrive / localstorage / spider91
name TEXT NOT NULL,
root_id TEXT NOT NULL DEFAULT '0',
scan_root_id TEXT, -- deprecated: 扫描起点固定等于 root_id
credentials TEXT, -- JSON: cookie / refresh_token 等
status TEXT DEFAULT 'disconnected', -- disconnected / ok / error
last_error TEXT,
-- 是否给该盘生成预览视频/封面1 开 / 0 关。
-- 是否给该盘生成预览视频:1 开 / 0 关。封面生成不受影响。
-- 替代了早期的全局 preview.enabled 设置(保留旧 setting 行不再读)。
teaser_enabled INTEGER NOT NULL DEFAULT 1,
-- 扫描时要跳过的目录 ID 集合(JSON array of string)。命中其中任意一个的目录及其
-168
View File
@@ -165,171 +165,3 @@ func TestRandomVideosWithReadyThumbnailsExcluding(t *testing.T) {
}
}
}
func TestRandomVideosForPreferredVideoChoosesLeastPopulatedTag(t *testing.T) {
ctx := context.Background()
cat, err := Open(t.TempDir() + "/catalog.db")
if err != nil {
t.Fatalf("open catalog: %v", err)
}
t.Cleanup(func() { _ = cat.Close() })
now := time.Now()
for _, v := range []*Video{
{ID: "current", DriveID: "drive", FileID: "f-current", Title: "current", Tags: []string{"common", "rare"}, PublishedAt: now, CreatedAt: now, UpdatedAt: now},
{ID: "common-1", DriveID: "drive", FileID: "f-common-1", Title: "common 1", Tags: []string{"common"}, PublishedAt: now, CreatedAt: now, UpdatedAt: now},
{ID: "common-2", DriveID: "drive", FileID: "f-common-2", Title: "common 2", Tags: []string{"common"}, PublishedAt: now, CreatedAt: now, UpdatedAt: now},
{ID: "rare-1", DriveID: "drive", FileID: "f-rare-1", Title: "rare 1", Tags: []string{"rare"}, PublishedAt: now, CreatedAt: now, UpdatedAt: now},
} {
if err := cat.UpsertVideo(ctx, v); err != nil {
t.Fatalf("seed %s: %v", v.ID, err)
}
}
tag, err := cat.LeastPopulatedVisibleUniqueTag(ctx, []string{"common", "rare"})
if err != nil {
t.Fatalf("least populated tag: %v", err)
}
if tag != "rare" {
t.Fatalf("least populated tag = %q, want rare", tag)
}
got, err := cat.RandomVideosForPreferredVideoExcluding(ctx, "current", []string{"current"}, 1)
if err != nil {
t.Fatalf("random preferred: %v", err)
}
if len(got) != 1 || got[0].ID != "rare-1" {
t.Fatalf("preferred result = %#v, want rare-1", videoIDs(got))
}
got, err = cat.RandomVideosForPreferredVideoExcluding(ctx, "current", nil, 1)
if err != nil {
t.Fatalf("random preferred without explicit exclude: %v", err)
}
if len(got) != 1 || got[0].ID == "current" {
t.Fatalf("preferred result without explicit exclude = %#v, should not return current", videoIDs(got))
}
}
func TestRandomVideosForPreferredVideoFallsBackToFillBatch(t *testing.T) {
ctx := context.Background()
cat, err := Open(t.TempDir() + "/catalog.db")
if err != nil {
t.Fatalf("open catalog: %v", err)
}
t.Cleanup(func() { _ = cat.Close() })
now := time.Now()
for _, v := range []*Video{
{ID: "current", DriveID: "drive", FileID: "f-current", Title: "current", Tags: []string{"common", "rare"}, PublishedAt: now, CreatedAt: now, UpdatedAt: now},
{ID: "common-1", DriveID: "drive", FileID: "f-common-1", Title: "common 1", Tags: []string{"common"}, PublishedAt: now, CreatedAt: now, UpdatedAt: now},
{ID: "common-2", DriveID: "drive", FileID: "f-common-2", Title: "common 2", Tags: []string{"common"}, PublishedAt: now, CreatedAt: now, UpdatedAt: now},
{ID: "rare-1", DriveID: "drive", FileID: "f-rare-1", Title: "rare 1", Tags: []string{"rare"}, PublishedAt: now, CreatedAt: now, UpdatedAt: now},
{ID: "hidden-rare", DriveID: "drive", FileID: "f-hidden-rare", Title: "hidden rare", Tags: []string{"rare"}, PublishedAt: now, CreatedAt: now, UpdatedAt: now},
} {
if err := cat.UpsertVideo(ctx, v); err != nil {
t.Fatalf("seed %s: %v", v.ID, err)
}
}
if err := cat.HideVideo(ctx, "hidden-rare"); err != nil {
t.Fatalf("hide hidden-rare: %v", err)
}
got, err := cat.RandomVideosForPreferredVideoExcluding(ctx, "current", []string{"current"}, 3)
if err != nil {
t.Fatalf("random preferred: %v", err)
}
ids := videoIDs(got)
if len(ids) != 3 {
t.Fatalf("result ids = %#v, want 3 items", ids)
}
for _, excluded := range []string{"current", "hidden-rare"} {
if hasVideoID(ids, excluded) {
t.Fatalf("result ids = %#v, should not include %s", ids, excluded)
}
}
if !hasVideoID(ids, "rare-1") {
t.Fatalf("result ids = %#v, want rare-1 from least populated tag", ids)
}
if len(uniqueVideoIDs(ids)) != len(ids) {
t.Fatalf("result ids = %#v, want no duplicates", ids)
}
}
func TestRandomVideosForPreferredVideoFallbacksWhenPreferenceUnavailable(t *testing.T) {
ctx := context.Background()
cat, err := Open(t.TempDir() + "/catalog.db")
if err != nil {
t.Fatalf("open catalog: %v", err)
}
t.Cleanup(func() { _ = cat.Close() })
now := time.Now()
for _, v := range []*Video{
{ID: "untagged", DriveID: "drive", FileID: "f-untagged", Title: "untagged", PublishedAt: now, CreatedAt: now, UpdatedAt: now},
{ID: "visible-1", DriveID: "drive", FileID: "f-visible-1", Title: "visible 1", PublishedAt: now, CreatedAt: now, UpdatedAt: now},
{ID: "visible-2", DriveID: "drive", FileID: "f-visible-2", Title: "visible 2", PublishedAt: now, CreatedAt: now, UpdatedAt: now},
} {
if err := cat.UpsertVideo(ctx, v); err != nil {
t.Fatalf("seed %s: %v", v.ID, err)
}
}
got, err := cat.RandomVideosForPreferredVideoExcluding(ctx, "missing", []string{"untagged"}, 2)
if err != nil {
t.Fatalf("random missing preferred: %v", err)
}
if !sameVideoIDSet(videoIDs(got), []string{"visible-1", "visible-2"}) {
t.Fatalf("missing preferred ids = %#v, want visible fallback videos", videoIDs(got))
}
got, err = cat.RandomVideosForPreferredVideoExcluding(ctx, "untagged", []string{"untagged"}, 2)
if err != nil {
t.Fatalf("random untagged preferred: %v", err)
}
if !sameVideoIDSet(videoIDs(got), []string{"visible-1", "visible-2"}) {
t.Fatalf("untagged preferred ids = %#v, want visible fallback videos", videoIDs(got))
}
}
func videoIDs(videos []*Video) []string {
ids := make([]string, 0, len(videos))
for _, v := range videos {
ids = append(ids, v.ID)
}
return ids
}
func hasVideoID(ids []string, want string) bool {
for _, id := range ids {
if id == want {
return true
}
}
return false
}
func uniqueVideoIDs(ids []string) map[string]struct{} {
seen := make(map[string]struct{}, len(ids))
for _, id := range ids {
seen[id] = struct{}{}
}
return seen
}
func sameVideoIDSet(a, b []string) bool {
if len(a) != len(b) {
return false
}
seen := make(map[string]int, len(a))
for _, value := range a {
seen[value]++
}
for _, value := range b {
if seen[value] == 0 {
return false
}
seen[value]--
}
return true
}
+42
View File
@@ -66,6 +66,24 @@ func (c *Catalog) migrate(ctx context.Context) error {
if err := c.addColumnIfMissing(ctx, "videos", "thumbnail_failures", "INTEGER DEFAULT 0"); err != nil {
return err
}
if err := c.addColumnIfMissing(ctx, "videos", "last_viewed_at", "INTEGER DEFAULT 0"); err != nil {
return err
}
// videos.transcode_*:浏览器兼容性转码状态。
// status''=未检测 / pending=已入队 / ready=已转码 / skipped=检测后无需转码 / failed=失败。
// transcoded_file_id 指向转码产物在同一 drive 上的 fileID,播放源优先使用它。
if err := c.addColumnIfMissing(ctx, "videos", "transcode_status", "TEXT DEFAULT ''"); err != nil {
return err
}
if err := c.addColumnIfMissing(ctx, "videos", "transcode_error", "TEXT DEFAULT ''"); err != nil {
return err
}
if err := c.addColumnIfMissing(ctx, "videos", "transcoded_file_id", "TEXT DEFAULT ''"); err != nil {
return err
}
if err := c.addColumnIfMissing(ctx, "videos", "transcoded_size", "INTEGER DEFAULT 0"); err != nil {
return err
}
// drives.teaser_enabled:每盘预览视频开关,替代旧的全局 preview.enabled。
// 升级路径:直接让 ALTER TABLE 的 DEFAULT 1 兜底 —— 每个现存 drive 都默认开启,
// 不读旧的 settings.preview.enabled 字段。这样老用户即便之前关过全局开关,
@@ -109,6 +127,9 @@ CREATE TABLE IF NOT EXISTS deleted_videos (
if err := c.reconcileThumbnailStatusOnce(ctx); err != nil {
return err
}
if err := c.requeueSkippedPreviews(ctx); err != nil {
return err
}
if _, err := c.db.ExecContext(ctx, `CREATE INDEX IF NOT EXISTS idx_videos_content_hash ON videos(content_hash)`); err != nil {
return err
}
@@ -127,6 +148,9 @@ CREATE TABLE IF NOT EXISTS deleted_videos (
if _, err := c.db.ExecContext(ctx, `CREATE INDEX IF NOT EXISTS idx_videos_visible_pub ON videos(COALESCE(hidden, 0), published_at DESC)`); err != nil {
return err
}
if _, err := c.db.ExecContext(ctx, `CREATE INDEX IF NOT EXISTS idx_videos_last_viewed ON videos(last_viewed_at DESC)`); err != nil {
return err
}
if _, err := c.db.ExecContext(ctx, `CREATE INDEX IF NOT EXISTS idx_videos_file_name_size ON videos(file_name, size_bytes)`); err != nil {
return err
}
@@ -281,6 +305,24 @@ UPDATE videos
return nil
}
func (c *Catalog) requeueSkippedPreviews(ctx context.Context) error {
res, err := c.db.ExecContext(ctx, `
UPDATE videos
SET preview_file_id = '',
preview_local = '',
preview_status = 'pending',
updated_at = ?
WHERE COALESCE(preview_status, 'pending') = 'skipped'
`, time.Now().UnixMilli())
if err != nil {
return fmt.Errorf("requeue skipped previews: %w", err)
}
if affected, err := res.RowsAffected(); err == nil && affected > 0 {
log.Printf("[catalog] requeued %d skipped preview(s) for generation", affected)
}
return nil
}
func (c *Catalog) clearVolatileOneDriveThumbnails(ctx context.Context) error {
// 把 OneDrive 过期的 mediap.svc.ms thumb URL 清空,让 worker 重新抽帧生成本地封面。
// 同步把 thumbnail_status 重置为 'pending':清空后 url 是空的,本应进 worker 重做,
+64
View File
@@ -1539,6 +1539,70 @@ func TestReconcileThumbnailStatusOnce(t *testing.T) {
}
}
func TestRequeueSkippedPreviews(t *testing.T) {
ctx := context.Background()
cat, err := Open(t.TempDir() + "/catalog.db")
if err != nil {
t.Fatalf("open: %v", err)
}
t.Cleanup(func() { cat.Close() })
now := time.Now()
cases := []struct {
id string
status string
local string
fileID string
wantStatus string
wantLocal string
wantFileID string
}{
{"preview-skipped", "skipped", "/tmp/old-preview.mp4", "old-preview-file", "pending", "", ""},
{"preview-ready", "ready", "/tmp/ready-preview.mp4", "ready-preview-file", "ready", "/tmp/ready-preview.mp4", "ready-preview-file"},
{"preview-failed", "failed", "/tmp/failed-preview.mp4", "failed-preview-file", "failed", "/tmp/failed-preview.mp4", "failed-preview-file"},
}
for _, c := range cases {
if err := cat.UpsertVideo(ctx, &Video{
ID: c.id, DriveID: "d", FileID: "source-" + c.id, Title: c.id,
PreviewStatus: c.status, PreviewLocal: c.local, PreviewFileID: c.fileID,
PublishedAt: now, CreatedAt: now, UpdatedAt: now,
}); err != nil {
t.Fatalf("seed %s: %v", c.id, err)
}
}
if err := cat.requeueSkippedPreviews(ctx); err != nil {
t.Fatalf("requeue skipped previews: %v", err)
}
if err := cat.requeueSkippedPreviews(ctx); err != nil {
t.Fatalf("second requeue skipped previews: %v", err)
}
for _, c := range cases {
got, err := cat.GetVideo(ctx, c.id)
if err != nil {
t.Fatalf("get %s: %v", c.id, err)
}
if got.PreviewStatus != c.wantStatus {
t.Errorf("%s: preview status = %q, want %q", c.id, got.PreviewStatus, c.wantStatus)
}
if got.PreviewLocal != c.wantLocal {
t.Errorf("%s: preview local = %q, want %q", c.id, got.PreviewLocal, c.wantLocal)
}
if got.PreviewFileID != c.wantFileID {
t.Errorf("%s: preview file id = %q, want %q", c.id, got.PreviewFileID, c.wantFileID)
}
}
pending, err := cat.ListVideosByPreviewStatus(ctx, "d", "pending", 0)
if err != nil {
t.Fatalf("list pending previews: %v", err)
}
if len(pending) != 1 || pending[0].ID != "preview-skipped" {
t.Fatalf("pending previews = %#v, want only preview-skipped", pending)
}
}
// TestUpsertVideoSyncsThumbnailStatus 验证 scanner 创建/补回视频时
// thumbnail_status 跟随 thumbnail_url 自动设。这是历史 bug 的修复回归测试 ——
// 之前 UpsertVideo 的 SQL 不带 thumbnail_status 列,所有新视频都依赖
@@ -0,0 +1,133 @@
package catalog
import (
"context"
"testing"
"time"
)
// TestListHiddenVideosForMigration 验证:隐藏的视频不进可见列表,
// 但能被 ListHiddenVideos 拿到(供一次性迁移为墓碑)。
func TestListHiddenVideosForMigration(t *testing.T) {
ctx := context.Background()
cat, err := Open(t.TempDir() + "/catalog.db")
if err != nil {
t.Fatalf("open catalog: %v", err)
}
t.Cleanup(func() { _ = cat.Close() })
now := time.Now()
for _, id := range []string{"v1", "v2", "v3"} {
if err := cat.UpsertVideo(ctx, &Video{
ID: id, DriveID: "drive", FileID: "f-" + id, Title: id,
PublishedAt: now, CreatedAt: now, UpdatedAt: now,
}); err != nil {
t.Fatalf("seed %s: %v", id, err)
}
}
if err := cat.HideVideo(ctx, "v2"); err != nil {
t.Fatalf("hide v2: %v", err)
}
visible, total, err := cat.ListVideos(ctx, ListParams{Page: 1, PageSize: 50})
if err != nil {
t.Fatalf("list visible: %v", err)
}
if total != 2 || len(visible) != 2 {
t.Fatalf("visible total/len = %d/%d, want 2/2", total, len(visible))
}
for _, v := range visible {
if v.ID == "v2" {
t.Fatalf("hidden v2 leaked into visible list")
}
}
hidden, err := cat.ListHiddenVideos(ctx)
if err != nil {
t.Fatalf("list hidden: %v", err)
}
if len(hidden) != 1 || hidden[0].ID != "v2" {
t.Fatalf("ListHiddenVideos = %v, want only v2", hidden)
}
current, blacklisted, err := cat.VideoManagementCounts(ctx)
if err != nil {
t.Fatalf("counts: %v", err)
}
if current != 2 || blacklisted != 0 {
t.Fatalf("counts = current %d blacklisted %d, want 2/0", current, blacklisted)
}
}
// TestBlacklistListAndRemove 验证墓碑表的列出、关键字过滤和移除。
func TestBlacklistListAndRemove(t *testing.T) {
ctx := context.Background()
cat, err := Open(t.TempDir() + "/catalog.db")
if err != nil {
t.Fatalf("open catalog: %v", err)
}
t.Cleanup(func() { _ = cat.Close() })
now := time.Now()
seed := []struct{ id, file string }{
{"d1", "movie-alpha.avi"},
{"d2", "movie-beta.mp4"},
{"d3", "clip-gamma.wmv"},
}
for _, s := range seed {
if err := cat.UpsertVideo(ctx, &Video{
ID: s.id, DriveID: "drive", FileID: "f-" + s.id, FileName: s.file,
Title: s.id, PublishedAt: now, CreatedAt: now, UpdatedAt: now,
}); err != nil {
t.Fatalf("seed %s: %v", s.id, err)
}
if err := cat.DeleteVideoWithTombstone(ctx, s.id); err != nil {
t.Fatalf("tombstone %s: %v", s.id, err)
}
}
items, total, err := cat.ListDeletedVideos(ctx, "", 1, 50)
if err != nil {
t.Fatalf("list deleted: %v", err)
}
if total != 3 || len(items) != 3 {
t.Fatalf("deleted total/len = %d/%d, want 3/3", total, len(items))
}
// 关键字过滤
filtered, ftotal, err := cat.ListDeletedVideos(ctx, "movie", 1, 50)
if err != nil {
t.Fatalf("list deleted filtered: %v", err)
}
if ftotal != 2 || len(filtered) != 2 {
t.Fatalf("filtered total/len = %d/%d, want 2/2", ftotal, len(filtered))
}
// 移出黑名单
if err := cat.RemoveDeletedVideo(ctx, "d1"); err != nil {
t.Fatalf("remove d1: %v", err)
}
if deleted, err := cat.IsVideoDeleted(ctx, "d1"); err != nil || deleted {
t.Fatalf("d1 should no longer be blacklisted (deleted=%v err=%v)", deleted, err)
}
_, total, err = cat.ListDeletedVideos(ctx, "", 1, 50)
if err != nil {
t.Fatalf("list deleted after remove: %v", err)
}
if total != 2 {
t.Fatalf("deleted total after remove = %d, want 2", total)
}
if err := cat.RemoveDeletedVideo(ctx, "does-not-exist"); err == nil {
t.Fatalf("remove missing id should return error")
}
// counts: 删完一个还剩 2 个黑名单;可见视频已全部被墓碑删除
current, blacklisted, err := cat.VideoManagementCounts(ctx)
if err != nil {
t.Fatalf("counts: %v", err)
}
if current != 0 || blacklisted != 2 {
t.Fatalf("counts = current %d blacklisted %d, want 0/2", current, blacklisted)
}
}
+1 -1
View File
@@ -207,7 +207,7 @@ type Nightly struct {
// 这里保留 yaml 中的静态定义,用于启动时预置盘。生产建议只在 DB 里维护。
type Drive struct {
ID string `yaml:"id"`
Kind string `yaml:"kind"` // quark / p115 / p123 / pikpak / wopan / onedrive / googledrive / localstorage
Kind string `yaml:"kind"` // quark / p115 / p123 / pikpak / wopan / guangyapan / onedrive / googledrive / localstorage
Name string `yaml:"name"`
RootID string `yaml:"root_id"`
Params map[string]string `yaml:"params,omitempty"`
+4 -31
View File
@@ -647,7 +647,7 @@ func isGoogleUploadHTTPRateLimit(status int, header http.Header, body []byte, ap
if isGoogleRateLimit(nil, apiErr) {
return true
}
return googleLimitText(string(body))
return false
}
func googleUploadRateLimitError(status int, header http.Header, body []byte, message string) error {
@@ -910,7 +910,7 @@ func isGoogleRateLimit(res *resty.Response, body apiErrorBody) bool {
return true
}
for _, e := range body.Errors {
if googleLimitReason(e.Reason) || googleLimitText(e.Message) {
if googleLimitReason(e.Reason) {
return true
}
domain := compactGoogleLimitText(e.Domain)
@@ -918,7 +918,7 @@ func isGoogleRateLimit(res *resty.Response, body apiErrorBody) bool {
return true
}
}
return googleLimitText(body.Message)
return false
}
func isGoogleTokenRateLimit(res *resty.Response, out tokenResp) bool {
@@ -930,9 +930,7 @@ func isGoogleTokenRateLimit(res *resty.Response, out tokenResp) bool {
return true
}
}
return googleLimitText(out.Text) ||
googleLimitText(out.Error) ||
googleLimitText(out.ErrorDescription)
return googleLimitReason(out.Error)
}
func googleLimitReason(reason string) bool {
@@ -953,31 +951,6 @@ func googleLimitReason(reason string) bool {
}
}
func googleLimitText(text string) bool {
text = strings.ToLower(strings.TrimSpace(text))
if text == "" {
return false
}
compact := compactGoogleLimitText(text)
if strings.Contains(compact, "ratelimitexceeded") ||
strings.Contains(compact, "userratelimitexceeded") ||
strings.Contains(compact, "dailylimitexceeded") ||
strings.Contains(compact, "downloadquotaexceeded") ||
strings.Contains(compact, "sharingratelimitexceeded") ||
strings.Contains(compact, "quotaexceeded") ||
strings.Contains(compact, "toomanyrequests") {
return true
}
return strings.Contains(text, "rate limit") ||
strings.Contains(text, "too many requests") ||
strings.Contains(text, "quota exceeded") ||
strings.Contains(text, "download quota") ||
strings.Contains(text, "sharing rate") ||
strings.Contains(text, "daily limit") ||
strings.Contains(text, "user rate") ||
strings.Contains(text, "usage limit")
}
func compactGoogleLimitText(text string) string {
text = strings.ToLower(strings.TrimSpace(text))
replacer := strings.NewReplacer("_", "", "-", "", " ", "", ".", "", ":", "")
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,300 @@
package guangyapan
import (
"context"
"encoding/json"
"errors"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
"github.com/video-site/backend/internal/drives"
)
func TestDriverRefreshListAndStream(t *testing.T) {
var refreshed bool
var listedRoot bool
updates := map[string]string{}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/v1/auth/token":
refreshed = true
writeTestJSON(w, map[string]any{
"access_token": "new-access",
"refresh_token": "new-refresh",
})
case "/v1/user/me":
if got := r.Header.Get("Authorization"); got != "Bearer new-access" {
t.Fatalf("auth header = %q, want new access token", got)
}
writeTestJSON(w, map[string]any{"sub": "user-1"})
case "/userres/v1/file/get_file_list":
if got := r.Header.Get("Authorization"); got != "Bearer new-access" {
t.Fatalf("api auth header = %q, want new access token", got)
}
var body map[string]any
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
t.Fatalf("decode list body: %v", err)
}
if body["parentId"] != "" {
t.Fatalf("parentId = %#v, want root empty string", body["parentId"])
}
listedRoot = true
writeTestJSON(w, map[string]any{
"code": 0,
"msg": "success",
"data": map[string]any{
"total": 2,
"list": []map[string]any{
{"fileId": "dir-1", "parentId": "", "fileName": "Movies", "resType": 2},
{"fileId": "file-1", "parentId": "", "fileName": "clip.mp4", "fileSize": 123, "resType": 1, "utime": 1700000000},
},
},
})
case "/nd.bizuserres.s/v1/get_res_download_url":
writeTestJSON(w, map[string]any{
"code": 0,
"msg": "success",
"data": map[string]any{"signedURL": "https://cdn.example.test/clip.mp4"},
})
default:
t.Fatalf("unexpected path %s", r.URL.Path)
}
}))
defer srv.Close()
d := New(Config{
ID: "gy",
RefreshToken: "old-refresh",
AccountBaseURL: srv.URL,
APIBaseURL: srv.URL,
OnCredentialsUpdate: func(values map[string]string) {
for k, v := range values {
updates[k] = v
}
},
})
if err := d.Init(context.Background()); err != nil {
t.Fatalf("init: %v", err)
}
if !refreshed {
t.Fatal("refresh token endpoint was not called")
}
if updates["access_token"] != "new-access" || updates["refresh_token"] != "new-refresh" {
t.Fatalf("updates = %#v, want refreshed tokens", updates)
}
entries, err := d.List(context.Background(), "")
if err != nil {
t.Fatalf("list: %v", err)
}
if !listedRoot || len(entries) != 2 {
t.Fatalf("listedRoot=%v entries=%#v", listedRoot, entries)
}
if !entries[0].IsDir || entries[1].ID != "file-1" || entries[1].Size != 123 {
t.Fatalf("entries = %#v", entries)
}
link, err := d.StreamURL(context.Background(), "file-1")
if err != nil {
t.Fatalf("stream url: %v", err)
}
if link.URL != "https://cdn.example.test/clip.mp4" {
t.Fatalf("stream url = %q", link.URL)
}
}
func TestDriverResolvesRootPath(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/v1/user/me":
writeTestJSON(w, map[string]any{"sub": "user-1"})
case "/userres/v1/file/get_file_list":
var body map[string]any
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
t.Fatalf("decode list body: %v", err)
}
parent, _ := body["parentId"].(string)
switch parent {
case "":
writeTestJSON(w, listTestResponse([]map[string]any{
{"fileId": "folder-a", "parentId": "", "fileName": "影视", "resType": 2},
}))
case "folder-a":
writeTestJSON(w, listTestResponse([]map[string]any{
{"fileId": "folder-b", "parentId": "folder-a", "fileName": "电影", "resType": 2},
}))
case "folder-b":
writeTestJSON(w, listTestResponse([]map[string]any{
{"fileId": "file-1", "parentId": "folder-b", "fileName": "movie.mp4", "fileSize": 456, "resType": 1},
}))
default:
t.Fatalf("unexpected parent %q", parent)
}
default:
t.Fatalf("unexpected path %s", r.URL.Path)
}
}))
defer srv.Close()
d := New(Config{
ID: "gy",
RootID: "configured-root",
RootPath: "影视/电影",
AccessToken: "access",
AccountBaseURL: srv.URL,
APIBaseURL: srv.URL,
})
if err := d.Init(context.Background()); err != nil {
t.Fatalf("init: %v", err)
}
if d.RootID() != "folder-b" {
t.Fatalf("root id = %q, want folder-b", d.RootID())
}
entries, err := d.List(context.Background(), "")
if err != nil {
t.Fatalf("list resolved root: %v", err)
}
if len(entries) != 1 || entries[0].ID != "file-1" {
t.Fatalf("entries = %#v", entries)
}
}
func TestDriverSendSMSCodeUpdatesVerificationState(t *testing.T) {
updates := map[string]string{}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/v1/shield/captcha/init":
writeTestJSON(w, map[string]any{"captcha_token": "captcha-1"})
case "/v1/auth/verification":
writeTestJSON(w, map[string]any{"verification_id": "verify-1"})
default:
t.Fatalf("unexpected path %s", r.URL.Path)
}
}))
defer srv.Close()
d := New(Config{
ID: "gy",
PhoneNumber: "13800000000",
SendCode: true,
AccountBaseURL: srv.URL,
APIBaseURL: srv.URL,
OnCredentialsUpdate: func(values map[string]string) {
for k, v := range values {
updates[k] = v
}
},
})
err := d.Init(context.Background())
if err == nil || !strings.Contains(err.Error(), "验证码已发送") {
t.Fatalf("init err = %v, want verification prompt", err)
}
if updates["captcha_token"] != "captcha-1" || updates["verification_id"] != "verify-1" || updates["send_code"] != "false" {
t.Fatalf("updates = %#v, want sms state saved", updates)
}
if updates["device_id"] == "" {
t.Fatalf("updates = %#v, want generated device id saved", updates)
}
}
func TestListHTTP429ReturnsRateLimitError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/userres/v1/file/get_file_list" {
t.Fatalf("unexpected path %s", r.URL.Path)
}
w.Header().Set("Retry-After", "120")
w.WriteHeader(http.StatusTooManyRequests)
writeTestJSON(w, map[string]any{"code": 429, "msg": "操作频繁,请稍后重试"})
}))
defer srv.Close()
d := New(Config{
ID: "gy",
AccessToken: "access",
AccountBaseURL: srv.URL,
APIBaseURL: srv.URL,
})
_, err := d.List(context.Background(), "")
if err == nil {
t.Fatal("list succeeded, want rate limit error")
}
var rateLimit *drives.RateLimitError
if !errors.As(err, &rateLimit) {
t.Fatalf("error = %T %[1]v, want RateLimitError", err)
}
if rateLimit.RetryAfter != 2*time.Minute {
t.Fatalf("retry after = %s, want 2m", rateLimit.RetryAfter)
}
}
func TestListCode429ReturnsRateLimitError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/userres/v1/file/get_file_list" {
t.Fatalf("unexpected path %s", r.URL.Path)
}
writeTestJSON(w, map[string]any{"code": 429, "msg": "操作频繁,请稍后再试"})
}))
defer srv.Close()
d := New(Config{
ID: "gy",
AccessToken: "access",
AccountBaseURL: srv.URL,
APIBaseURL: srv.URL,
})
_, err := d.List(context.Background(), "")
if err == nil {
t.Fatal("list succeeded, want rate limit error")
}
var rateLimit *drives.RateLimitError
if !errors.As(err, &rateLimit) {
t.Fatalf("error = %T %[1]v, want RateLimitError", err)
}
}
func TestListInvalidToken403DoesNotReturnRateLimitError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/userres/v1/file/get_file_list" {
t.Fatalf("unexpected path %s", r.URL.Path)
}
w.WriteHeader(http.StatusForbidden)
writeTestJSON(w, map[string]any{"code": 401, "msg": "invalid access token"})
}))
defer srv.Close()
d := New(Config{
ID: "gy",
AccessToken: "access",
AccountBaseURL: srv.URL,
APIBaseURL: srv.URL,
})
_, err := d.List(context.Background(), "")
if err == nil {
t.Fatal("list succeeded, want auth error")
}
var rateLimit *drives.RateLimitError
if errors.As(err, &rateLimit) {
t.Fatalf("error = %T %[1]v, want non-rate-limit error", err)
}
}
func listTestResponse(items []map[string]any) map[string]any {
return map[string]any{
"code": 0,
"msg": "success",
"data": map[string]any{
"total": len(items),
"list": items,
},
}
}
func writeTestJSON(w http.ResponseWriter, v any) {
w.Header().Set("Content-Type", "application/json")
if err := json.NewEncoder(w).Encode(v); err != nil {
panic(err)
}
}
+244
View File
@@ -0,0 +1,244 @@
package guangyapan
import (
"context"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"net/http"
"strings"
"time"
"github.com/go-resty/resty/v2"
"github.com/skip2/go-qrcode"
)
const (
defaultQRScope = "user"
deviceCodeGrantType = "urn:ietf:params:oauth:grant-type:device_code"
defaultQRUserAgent = "GuangYaPan-Login/1.0"
)
type QRConfig struct {
AccountBaseURL string
HTTPClient *http.Client
Now func() time.Time
}
type QRClient struct {
accountBaseURL string
client *resty.Client
now func() time.Time
}
type QRCodeSession struct {
DeviceCode string `json:"deviceCode"`
QRCodeURL string `json:"qrCodeUrl"`
QRImageDataURL string `json:"qrImageDataUrl"`
IntervalSeconds int `json:"intervalSeconds"`
ExpiresAt string `json:"expiresAt,omitempty"`
}
type QRCodeStatus struct {
State string `json:"state"`
StatusText string `json:"statusText"`
IntervalSeconds int `json:"intervalSeconds,omitempty"`
AccessToken string `json:"accessToken,omitempty"`
RefreshToken string `json:"refreshToken,omitempty"`
TokenType string `json:"tokenType,omitempty"`
ExpiresIn int64 `json:"expiresIn,omitempty"`
}
type deviceCodeResp struct {
DeviceCode string `json:"device_code"`
VerificationURIComplete string `json:"verification_uri_complete"`
ShortURIComplete string `json:"short_uri_complete"`
Interval int `json:"interval"`
ExpiresIn int `json:"expires_in"`
Error string `json:"error"`
ErrorCode int `json:"error_code"`
ErrorDesc string `json:"error_description"`
}
type deviceTokenResp struct {
AccessToken string `json:"access_token"`
RefreshToken string `json:"refresh_token"`
TokenType string `json:"token_type"`
ExpiresIn int64 `json:"expires_in"`
Scope string `json:"scope"`
Error string `json:"error"`
ErrorCode int `json:"error_code"`
ErrorDesc string `json:"error_description"`
}
func NewQRClient(c QRConfig) *QRClient {
accountBaseURL := strings.TrimRight(strings.TrimSpace(c.AccountBaseURL), "/")
if accountBaseURL == "" {
accountBaseURL = defaultAccountBaseURL
}
httpClient := c.HTTPClient
if httpClient == nil {
httpClient = &http.Client{Timeout: 20 * time.Second}
}
now := c.Now
if now == nil {
now = time.Now
}
return &QRClient{
accountBaseURL: accountBaseURL,
client: resty.NewWithClient(httpClient).
SetTimeout(20*time.Second).
SetBaseURL(accountBaseURL).
SetHeader("User-Agent", defaultQRUserAgent).
SetHeader("Accept", "application/json").
SetHeader("Content-Type", "application/json"),
now: now,
}
}
func (c *QRClient) Generate(ctx context.Context) (QRCodeSession, error) {
var out deviceCodeResp
var errOut deviceCodeResp
resp, err := c.client.R().
SetContext(ctx).
SetBody(map[string]any{
"client_id": defaultClientID,
"scope": defaultQRScope,
}).
SetResult(&out).
SetError(&errOut).
Post("/v1/auth/device/code")
if err != nil {
return QRCodeSession{}, err
}
if resp.IsError() || out.Error != "" {
if out.Error == "" {
out = errOut
}
return QRCodeSession{}, fmt.Errorf("guangyapan qr: %s", deviceAPIError(out.ErrorDesc, out.Error, resp))
}
deviceCode := strings.TrimSpace(out.DeviceCode)
if deviceCode == "" {
return QRCodeSession{}, errors.New("guangyapan qr: empty device_code")
}
qrURL := strings.TrimSpace(out.VerificationURIComplete)
if qrURL == "" {
qrURL = strings.TrimSpace(out.ShortURIComplete)
}
if qrURL == "" {
return QRCodeSession{}, errors.New("guangyapan qr: empty verification uri")
}
interval := out.Interval
if interval <= 0 {
interval = 5
}
expiresIn := out.ExpiresIn
if expiresIn <= 0 {
expiresIn = 300
}
png, err := qrcode.Encode(qrURL, qrcode.Medium, 220)
if err != nil {
return QRCodeSession{}, err
}
return QRCodeSession{
DeviceCode: deviceCode,
QRCodeURL: qrURL,
QRImageDataURL: "data:image/png;base64," + base64.StdEncoding.EncodeToString(png),
IntervalSeconds: interval,
ExpiresAt: c.now().Add(time.Duration(expiresIn) * time.Second).Format(time.RFC3339),
}, nil
}
func (c *QRClient) Poll(ctx context.Context, deviceCode string) (QRCodeStatus, error) {
deviceCode = strings.TrimSpace(deviceCode)
if deviceCode == "" {
return QRCodeStatus{}, errors.New("deviceCode is required")
}
var out deviceTokenResp
var errOut deviceTokenResp
resp, err := c.client.R().
SetContext(ctx).
SetBody(map[string]any{
"client_id": defaultClientID,
"grant_type": deviceCodeGrantType,
"device_code": deviceCode,
}).
SetResult(&out).
SetError(&errOut).
Post("/v1/auth/token")
if err != nil {
return QRCodeStatus{}, err
}
if resp.IsError() && out.Error == "" {
out = errOut
}
if resp.IsError() && out.Error == "" {
_ = json.Unmarshal(resp.Body(), &out)
}
if out.Error != "" {
return qrStatusForDeviceError(out), nil
}
if resp.IsError() {
return QRCodeStatus{}, fmt.Errorf("guangyapan qr: status=%d body=%s", resp.StatusCode(), resp.String())
}
access := strings.TrimSpace(out.AccessToken)
refresh := strings.TrimSpace(out.RefreshToken)
if access == "" || refresh == "" {
return QRCodeStatus{}, errors.New("guangyapan qr: login succeeded but token response is incomplete")
}
tokenType := strings.TrimSpace(out.TokenType)
if tokenType == "" {
tokenType = "Bearer"
}
return QRCodeStatus{
State: "success",
StatusText: "登录成功",
AccessToken: access,
RefreshToken: refresh,
TokenType: tokenType,
ExpiresIn: out.ExpiresIn,
}, nil
}
func qrStatusForDeviceError(out deviceTokenResp) QRCodeStatus {
errCode := strings.TrimSpace(out.Error)
switch errCode {
case "authorization_pending":
return QRCodeStatus{State: "pending", StatusText: "等待扫码确认"}
case "slow_down":
return QRCodeStatus{State: "pending", StatusText: "等待扫码确认,已降低查询频率", IntervalSeconds: 10}
case "expired_token":
return QRCodeStatus{State: "expired", StatusText: "二维码已过期"}
case "access_denied":
return QRCodeStatus{State: "denied", StatusText: "用户拒绝了授权"}
default:
msg := strings.TrimSpace(out.ErrorDesc)
if msg == "" {
msg = errCode
}
if msg == "" {
msg = "未知错误"
}
return QRCodeStatus{State: "error", StatusText: msg}
}
}
func deviceAPIError(desc, short string, resp *resty.Response) string {
msg := strings.TrimSpace(desc)
if msg == "" {
msg = strings.TrimSpace(short)
}
if msg == "" && resp != nil {
msg = strings.TrimSpace(resp.String())
}
if msg == "" && resp != nil {
msg = fmt.Sprintf("status=%d", resp.StatusCode())
}
if msg == "" {
msg = "unknown error"
}
return msg
}
@@ -0,0 +1,102 @@
package guangyapan
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
)
func TestQRClientGenerate(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1/auth/device/code" {
t.Fatalf("path = %s, want device code endpoint", r.URL.Path)
}
var body map[string]any
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
t.Fatalf("decode body: %v", err)
}
if body["client_id"] != defaultClientID || body["scope"] != defaultQRScope {
t.Fatalf("body = %#v", body)
}
writeTestJSON(w, map[string]any{
"device_code": "device-1",
"verification_uri_complete": "https://account.guangyapan.com/device?code=abc",
"interval": 7,
"expires_in": 180,
})
}))
defer srv.Close()
client := NewQRClient(QRConfig{
AccountBaseURL: srv.URL,
Now: func() time.Time { return time.Unix(1700000000, 0) },
})
session, err := client.Generate(context.Background())
if err != nil {
t.Fatalf("generate: %v", err)
}
if session.DeviceCode != "device-1" || session.QRCodeURL != "https://account.guangyapan.com/device?code=abc" {
t.Fatalf("session = %#v", session)
}
if session.IntervalSeconds != 7 {
t.Fatalf("interval = %d, want 7", session.IntervalSeconds)
}
if session.ExpiresAt != time.Unix(1700000180, 0).Format(time.RFC3339) {
t.Fatalf("expiresAt = %q", session.ExpiresAt)
}
if !strings.HasPrefix(session.QRImageDataURL, "data:image/png;base64,") {
t.Fatalf("qr image = %q", session.QRImageDataURL)
}
}
func TestQRClientPollPendingAndSuccess(t *testing.T) {
var calls int
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1/auth/token" {
t.Fatalf("path = %s, want token endpoint", r.URL.Path)
}
var body map[string]any
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
t.Fatalf("decode body: %v", err)
}
if body["client_id"] != defaultClientID ||
body["grant_type"] != deviceCodeGrantType ||
body["device_code"] != "device-1" {
t.Fatalf("body = %#v", body)
}
calls++
if calls == 1 {
w.WriteHeader(http.StatusBadRequest)
writeTestJSON(w, map[string]any{"error": "authorization_pending"})
return
}
writeTestJSON(w, map[string]any{
"access_token": "access-1",
"refresh_token": "refresh-1",
"token_type": "Bearer",
"expires_in": 7200,
})
}))
defer srv.Close()
client := NewQRClient(QRConfig{AccountBaseURL: srv.URL})
pending, err := client.Poll(context.Background(), "device-1")
if err != nil {
t.Fatalf("poll pending: %v", err)
}
if pending.State != "pending" || pending.AccessToken != "" {
t.Fatalf("pending = %#v", pending)
}
success, err := client.Poll(context.Background(), "device-1")
if err != nil {
t.Fatalf("poll success: %v", err)
}
if success.State != "success" || success.AccessToken != "access-1" || success.RefreshToken != "refresh-1" {
t.Fatalf("success = %#v", success)
}
}
+129
View File
@@ -0,0 +1,129 @@
package guangyapan
import "time"
type tokenResp struct {
AccessToken string `json:"access_token"`
RefreshToken string `json:"refresh_token"`
Error string `json:"error"`
ErrorCode int `json:"error_code"`
ErrorDesc string `json:"error_description"`
}
type verificationResp struct {
VerificationID string `json:"verification_id"`
Error string `json:"error"`
ErrorCode int `json:"error_code"`
ErrorDesc string `json:"error_description"`
}
type captchaInitResp struct {
CaptchaToken string `json:"captcha_token"`
Error string `json:"error"`
ErrorCode int `json:"error_code"`
ErrorDesc string `json:"error_description"`
}
type verifyResp struct {
VerificationToken string `json:"verification_token"`
Error string `json:"error"`
ErrorCode int `json:"error_code"`
ErrorDesc string `json:"error_description"`
}
type userMeResp struct {
Sub string `json:"sub"`
}
type listResp struct {
Code int `json:"code"`
Msg string `json:"msg"`
Data struct {
Total int `json:"total"`
List []fileItem `json:"list"`
} `json:"data"`
}
type fileItem struct {
FileID string `json:"fileId"`
ParentID string `json:"parentId"`
FileName string `json:"fileName"`
FileSize int64 `json:"fileSize"`
ResType int `json:"resType"`
CTime int64 `json:"ctime"`
UTime int64 `json:"utime"`
}
type downloadResp struct {
Code int `json:"code"`
Msg string `json:"msg"`
Data struct {
SignedURL string `json:"signedURL"`
DownloadURL string `json:"downloadUrl"`
} `json:"data"`
}
type createDirResp struct {
Code int `json:"code"`
Msg string `json:"msg"`
Data struct {
FileID string `json:"fileId"`
FileName string `json:"fileName"`
ResType int `json:"resType"`
CTime int64 `json:"ctime"`
UTime int64 `json:"utime"`
} `json:"data"`
}
type deleteResp struct {
Code int `json:"code"`
Msg string `json:"msg"`
Data struct {
TaskID string `json:"taskId"`
} `json:"data"`
}
type taskStatusResp struct {
Code int `json:"code"`
Msg string `json:"msg"`
Data struct {
Status int `json:"status"`
} `json:"data"`
}
type uploadTokenResp struct {
Code int `json:"code"`
Msg string `json:"msg"`
Data uploadTokenData `json:"data"`
}
type uploadTokenData struct {
TaskID string `json:"taskId"`
ObjectPath string `json:"objectPath"`
BucketName string `json:"bucketName"`
EndPoint string `json:"endPoint"`
FullEndPoint string `json:"fullEndPoint"`
AccessKeyID string `json:"accessKeyID"`
SecretAccessKey string `json:"secretAccessKey"`
SessionToken string `json:"sessionToken"`
Creds struct {
AccessKeyID string `json:"accessKeyID"`
SecretAccessKey string `json:"secretAccessKey"`
SessionToken string `json:"sessionToken"`
} `json:"creds"`
}
type taskInfoResp struct {
Code int `json:"code"`
Msg string `json:"msg"`
Data struct {
FileID string `json:"fileId"`
} `json:"data"`
}
func unixOrZero(v int64) time.Time {
if v <= 0 {
return time.Time{}
}
return time.Unix(v, 0)
}
+42 -1
View File
@@ -5,12 +5,14 @@ import (
"errors"
"io"
"net/http"
"strconv"
"strings"
"time"
)
// Drive 是多家网盘统一抽象。上层不区分盘,只区分 Kind。
type Drive interface {
// Kind 返回驱动代号:"quark" / "p115" / "p123" / "pikpak" / "wopan" / "onedrive" / "googledrive" / "localstorage"
// Kind 返回驱动代号:"quark" / "p115" / "p123" / "pikpak" / "wopan" / "guangyapan" / "onedrive" / "googledrive" / "localstorage"
Kind() string
// ID 返回该盘在 catalog 中的唯一标识
@@ -119,3 +121,42 @@ func RateLimitRetryAfter(err error) (time.Duration, bool) {
}
return 0, false
}
// TextMentionsHTTPStatus only looks for explicit numeric HTTP status contexts
// in errors from tools that do not expose structured response metadata.
func TextMentionsHTTPStatus(text string, statuses ...int) bool {
text = strings.ToLower(strings.TrimSpace(text))
if text == "" {
return false
}
for _, status := range statuses {
if status <= 0 {
continue
}
code := strconv.Itoa(status)
if strings.HasPrefix(text, code+" ") ||
strings.Contains(text, "status="+code) ||
strings.Contains(text, "status: "+code) ||
strings.Contains(text, "status "+code) ||
strings.Contains(text, "status code "+code) ||
strings.Contains(text, "http "+code) ||
strings.Contains(text, "http status="+code) ||
strings.Contains(text, "http status: "+code) ||
strings.Contains(text, "http status "+code) ||
strings.Contains(text, "server returned "+code) ||
strings.Contains(text, "code="+code) ||
strings.Contains(text, "code: "+code) ||
strings.Contains(text, "error_code="+code) ||
strings.Contains(text, "error_code: "+code) {
return true
}
}
return false
}
func ErrorMentionsHTTPStatus(err error, statuses ...int) bool {
if err == nil {
return false
}
return TextMentionsHTTPStatus(err.Error(), statuses...)
}
+24
View File
@@ -0,0 +1,24 @@
package drives
import "testing"
func TestTextMentionsHTTPStatus(t *testing.T) {
tests := []struct {
name string
text string
want bool
}{
{name: "status context", text: "request failed with status: 429 Too Many Requests", want: true},
{name: "http context", text: "http 503 service unavailable", want: true},
{name: "server returned context", text: "Server returned 403 Forbidden", want: true},
{name: "message only", text: "操作频繁,请稍后重试", want: false},
{name: "unrelated number", text: "generated 429 bytes", want: false},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
if got := TextMentionsHTTPStatus(tc.text, 403, 429, 503); got != tc.want {
t.Fatalf("TextMentionsHTTPStatus(%q) = %v, want %v", tc.text, got, tc.want)
}
})
}
}
+13 -6
View File
@@ -23,17 +23,24 @@ const maxSTRMBytes = 64 * 1024
type Config struct {
ID string
RootPath string
// STRMAllowOutsideRoot 允许 .strm 指向存储根目录之外的本地路径。
// 默认关闭:strm 等于可以让 /p/stream 读到服务器上的任意文件,只有
// 管理员明确知道自己在做什么(例如 strm 库与 rclone 挂载目录分离)
// 时才应打开。
STRMAllowOutsideRoot bool
}
type Driver struct {
id string
rootPath string
id string
rootPath string
strmAllowOutsideRoot bool
}
func New(c Config) *Driver {
return &Driver{
id: c.ID,
rootPath: c.RootPath,
id: c.ID,
rootPath: c.RootPath,
strmAllowOutsideRoot: c.STRMAllowOutsideRoot,
}
}
@@ -230,8 +237,8 @@ func (d *Driver) localSTRMLink(strmPath, target string) (*drives.StreamLink, err
if err != nil {
return nil, err
}
if !within {
return nil, errors.New("localstorage: strm target escapes root")
if !within && !d.strmAllowOutsideRoot {
return nil, errors.New("localstorage: strm target escapes root (enable strm_allow_outside_root to allow)")
}
if strings.EqualFold(filepath.Ext(p), ".strm") || strings.EqualFold(filepath.Ext(realPath), ".strm") {
return nil, errors.New("localstorage: nested strm target is not supported")
@@ -195,6 +195,46 @@ func TestStreamURLRejectsSTRMTargetEscapingRootThroughSymlink(t *testing.T) {
}
}
func TestStreamURLAllowsSTRMTargetOutsideRootWhenEnabled(t *testing.T) {
root := t.TempDir()
outside := t.TempDir()
target := filepath.Join(outside, "movie.mp4")
writeLocalStorageTestFile(t, target, []byte("movie-data"))
writeLocalStorageTestFile(t, filepath.Join(root, "movie.strm"), []byte(target+"\n"))
// 默认关闭:根目录外的目标仍被拒绝
strict := New(Config{ID: "local", RootPath: root})
if _, err := strict.StreamURL(context.Background(), encodeRel("movie.strm")); err == nil || !strings.Contains(err.Error(), "strm target escapes root") {
t.Fatalf("default error = %v, want strm target escapes root", err)
}
// 开启 strm_allow_outside_root 后放行
relaxed := New(Config{ID: "local", RootPath: root, STRMAllowOutsideRoot: true})
link, err := relaxed.StreamURL(context.Background(), encodeRel("movie.strm"))
if err != nil {
t.Fatalf("StreamURL with allow-outside-root: %v", err)
}
resolved, err := filepath.EvalSymlinks(target)
if err != nil {
t.Fatalf("eval target: %v", err)
}
if link.URL != resolved {
t.Fatalf("link url = %q, want %q", link.URL, resolved)
}
}
func TestStreamURLAllowOutsideRootStillRejectsNestedSTRM(t *testing.T) {
root := t.TempDir()
outside := t.TempDir()
writeLocalStorageTestFile(t, filepath.Join(outside, "inner.strm"), []byte("http://example.com/v.mp4\n"))
writeLocalStorageTestFile(t, filepath.Join(root, "movie.strm"), []byte(filepath.Join(outside, "inner.strm")+"\n"))
drv := New(Config{ID: "local", RootPath: root, STRMAllowOutsideRoot: true})
if _, err := drv.StreamURL(context.Background(), encodeRel("movie.strm")); err == nil || !strings.Contains(err.Error(), "nested strm") {
t.Fatalf("error = %v, want nested strm rejection", err)
}
}
func TestStreamURLRejectsSymlinkFileIDEscapingRoot(t *testing.T) {
root := t.TempDir()
outside := t.TempDir()
+2 -14
View File
@@ -594,8 +594,8 @@ func (d *Driver) refresh(ctx context.Context) error {
return nil
}
func isRateLimitResponse(res *resty.Response, code, message string) bool {
if isRateLimitCode(code) || isRateLimitMessage(message) {
func isRateLimitResponse(res *resty.Response, code, _ string) bool {
if isRateLimitCode(code) {
return true
}
if res == nil {
@@ -632,18 +632,6 @@ func isRateLimitCode(code string) bool {
}
}
func isRateLimitMessage(message string) bool {
text := strings.ToLower(strings.TrimSpace(message))
if text == "" {
return false
}
return strings.Contains(text, "too many requests") ||
strings.Contains(text, "throttl") ||
strings.Contains(text, "rate limit") ||
strings.Contains(text, "activity limit") ||
strings.Contains(text, "temporarily blocked")
}
func onedriveRateLimitError(res *resty.Response, message string) error {
if strings.TrimSpace(message) == "" {
message = "onedrive rate limited"
@@ -214,7 +214,7 @@ func TestGraph429ReturnsRateLimitErrorWithRetryAfter(t *testing.T) {
}
}
func TestGraphThrottleMessageReturnsRateLimitError(t *testing.T) {
func TestGraphThrottleMessageDoesNotReturnRateLimitError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusForbidden)
@@ -238,11 +238,11 @@ func TestGraphThrottleMessageReturnsRateLimitError(t *testing.T) {
_, err := d.StreamURL(context.Background(), "file-id")
if err == nil {
t.Fatal("list succeeded, want rate limit error")
t.Fatal("list succeeded, want graph error")
}
var rateLimit *drives.RateLimitError
if !errors.As(err, &rateLimit) {
t.Fatalf("error = %T %[1]v, want RateLimitError", err)
if errors.As(err, &rateLimit) {
t.Fatalf("error = %T %[1]v, want non-rate-limit error", err)
}
}
+28 -29
View File
@@ -20,11 +20,12 @@ import (
)
type Driver struct {
id string
cookie string
rootID string
client *sdk.Pan115Client
ua string
id string
cookie string
rootID string
client *sdk.Pan115Client
ua string
uploadTempDir string
listMu sync.Mutex
lastListAt time.Time
@@ -32,10 +33,11 @@ type Driver struct {
}
type Config struct {
ID string
Cookie string // 形如 "UID=xxx; CID=xxx; SEID=xxx; KID=xxx"
RootID string // 默认 "0"
UA string // 默认 UA115Browser
ID string
Cookie string // 形如 "UID=xxx; CID=xxx; SEID=xxx; KID=xxx"
RootID string // 默认 "0"
UA string // 默认 UA115Browser
UploadTempDir string
}
func New(c Config) *Driver {
@@ -48,11 +50,12 @@ func New(c Config) *Driver {
ua = sdk.UA115Browser
}
return &Driver{
id: c.ID,
cookie: c.Cookie,
rootID: rootID,
ua: ua,
listInterval: 2 * time.Second,
id: c.ID,
cookie: c.Cookie,
rootID: rootID,
ua: ua,
uploadTempDir: strings.TrimSpace(c.UploadTempDir),
listInterval: 2 * time.Second,
}
}
@@ -87,7 +90,7 @@ func (d *Driver) List(ctx context.Context, dirID string) ([]drives.Entry, error)
// p115ListCooldown 是列目录触发疑似风控错误时的冷却时长。
//
// 历史上是 [30min × 3],3 次都失败就放弃;新策略改为 10 分钟无限重试 ——
// 只要错误仍属 transient429 / 405 / WAF / blocked / 安全威胁 / unexpected),
// 只要错误仍属明确 HTTP transient 状态429 / 405),
// 就持续等 10 分钟再发一次列目录请求,直到成功或 ctx 取消。这样即使 115
// 风控持续较长时间,扫描会自然延后到风控结束,不再丢半棵子树。
const p115ListCooldown = 10 * time.Minute
@@ -156,17 +159,7 @@ func isTransient115UpstreamError(err error) bool {
if err == nil {
return false
}
text := strings.ToLower(err.Error())
return strings.Contains(text, "405") ||
strings.Contains(text, "429") ||
strings.Contains(text, "too many request") ||
strings.Contains(text, "too many requests") ||
strings.Contains(text, "blocked") ||
strings.Contains(text, "security") ||
strings.Contains(text, "waf") ||
strings.Contains(text, "unexpected error") ||
strings.Contains(text, "访问被阻断") ||
strings.Contains(text, "安全威胁")
return drives.ErrorMentionsHTTPStatus(err, http.StatusMethodNotAllowed, http.StatusTooManyRequests)
}
// ListDirsOnly 只列指定目录的直接**子目录**,不返回文件条目。专为 admin 后台
@@ -357,7 +350,7 @@ func (d *Driver) UploadAndReportSha1(ctx context.Context, parentID, name string,
parentID = d.rootID
}
tmp, sha1Hex, written, err := bufferAndHashSha1(r, size)
tmp, sha1Hex, written, err := bufferAndHashSha1(d.uploadTempDir, r, size)
if err != nil {
return UploadResult{}, err
}
@@ -482,8 +475,14 @@ func (d *Driver) Remove(ctx context.Context, fileID string) error {
// 返回临时文件(位置在末尾,需调用方 Seek 回 0)、SHA1 hex 大写、实际字节数。
//
// 调用方负责 Close + Remove 临时文件。
func bufferAndHashSha1(r io.Reader, declaredSize int64) (*os.File, string, int64, error) {
tmp, err := os.CreateTemp("", "p115-upload-*.bin")
func bufferAndHashSha1(tempDir string, r io.Reader, declaredSize int64) (*os.File, string, int64, error) {
tempDir = strings.TrimSpace(tempDir)
if tempDir != "" {
if err := os.MkdirAll(tempDir, 0o755); err != nil {
return nil, "", 0, fmt.Errorf("p115 upload: create tmp dir: %w", err)
}
}
tmp, err := os.CreateTemp(tempDir, "p115-upload-*.bin")
if err != nil {
return nil, "", 0, fmt.Errorf("p115 upload: create tmp: %w", err)
}
+21 -7
View File
@@ -8,6 +8,7 @@ import (
"errors"
"io"
"os"
"path/filepath"
"strings"
"testing"
"time"
@@ -22,8 +23,9 @@ func TestIsTransient115ListError(t *testing.T) {
want bool
}{
{name: "nil", err: nil, want: false},
{name: "blocked html", err: errors.New(`<!doctype html><title>405</title>Sorry, your request has been blocked as it may cause potential threats to the server's security.`), want: true},
{name: "chinese waf", err: errors.New("很抱歉,由于您访问的URL有可能对网站造成安全威胁,您的访问被阻断。"), want: true},
{name: "blocked html without status context", err: errors.New(`<!doctype html><title>405</title>Sorry, your request has been blocked as it may cause potential threats to the server's security.`), want: false},
{name: "chinese waf", err: errors.New("很抱歉,由于您访问的URL有可能对网站造成安全威胁,您的访问被阻断。"), want: false},
{name: "status 405", err: errors.New("request failed with status: 405"), want: true},
{name: "rate limit", err: errors.New("429 too many requests"), want: true},
{name: "regular auth error", err: errors.New("invalid credential"), want: false},
}
@@ -43,10 +45,10 @@ func TestWrap115StreamTransientError(t *testing.T) {
err error
wantRateLimit bool
}{
{name: "unexpected", err: errors.New("unexpected error"), wantRateLimit: true},
{name: "unexpected", err: errors.New("unexpected error"), wantRateLimit: false},
{name: "405 blocked", err: errors.New("405 request has been blocked"), wantRateLimit: true},
{name: "429", err: errors.New("429 too many requests"), wantRateLimit: true},
{name: "blocked", err: errors.New("blocked by waf"), wantRateLimit: true},
{name: "blocked", err: errors.New("blocked by waf"), wantRateLimit: false},
{name: "auth", err: errors.New("invalid credential"), wantRateLimit: false},
}
@@ -85,7 +87,7 @@ func TestBufferAndHashSha1(t *testing.T) {
wantHex := strings.ToUpper(hex.EncodeToString(want[:]))
t.Run("declared size matches", func(t *testing.T) {
tmp, gotHex, n, err := bufferAndHashSha1(bytes.NewReader(body), int64(len(body)))
tmp, gotHex, n, err := bufferAndHashSha1("", bytes.NewReader(body), int64(len(body)))
if err != nil {
t.Fatalf("bufferAndHashSha1 returned error: %v", err)
}
@@ -110,14 +112,14 @@ func TestBufferAndHashSha1(t *testing.T) {
})
t.Run("declared size mismatch returns error", func(t *testing.T) {
_, _, _, err := bufferAndHashSha1(bytes.NewReader(body), int64(len(body))+1)
_, _, _, err := bufferAndHashSha1("", bytes.NewReader(body), int64(len(body))+1)
if err == nil {
t.Fatal("expected size mismatch error, got nil")
}
})
t.Run("declared size zero is unchecked", func(t *testing.T) {
tmp, gotHex, n, err := bufferAndHashSha1(bytes.NewReader(body), 0)
tmp, gotHex, n, err := bufferAndHashSha1("", bytes.NewReader(body), 0)
if err != nil {
t.Fatalf("bufferAndHashSha1 returned error: %v", err)
}
@@ -129,6 +131,18 @@ func TestBufferAndHashSha1(t *testing.T) {
t.Errorf("written = %d, want %d", n, len(body))
}
})
t.Run("uses configured temp dir", func(t *testing.T) {
tempDir := filepath.Join(t.TempDir(), "upload-tmp")
tmp, _, _, err := bufferAndHashSha1(tempDir, bytes.NewReader(body), int64(len(body)))
if err != nil {
t.Fatalf("bufferAndHashSha1 returned error: %v", err)
}
defer cleanup(tmp)
if gotDir := filepath.Dir(tmp.Name()); gotDir != tempDir {
t.Fatalf("tmp dir = %q, want %q", gotDir, tempDir)
}
})
}
// TestUploadAndReportSha1RejectsInvalidArgs 检查空 reader / 空 name / 负 size 在
+15 -32
View File
@@ -70,6 +70,7 @@ type Driver struct {
httpClient *http.Client
onTokenUpdate func(access string)
uploadTempDir string
tokenMu sync.RWMutex
@@ -90,6 +91,7 @@ type Config struct {
MainAPIBaseURL string
LoginAPIBaseURL string
UploadTempDir string
OnTokenUpdate func(access string)
}
@@ -123,6 +125,7 @@ func New(c Config) *Driver {
referer: defaultReferer,
userAgent: defaultUserAgent,
onTokenUpdate: c.OnTokenUpdate,
uploadTempDir: strings.TrimSpace(c.UploadTempDir),
client: resty.New().
SetTimeout(30*time.Second).
SetHeader("Accept", "application/json, text/plain, */*"),
@@ -289,7 +292,7 @@ func (d *Driver) UploadAndReportHash(ctx context.Context, parentID, name string,
parentID = d.rootID
}
tmp, md5Hex, actualSize, err := bufferAndHashMD5(r, size)
tmp, md5Hex, actualSize, err := bufferAndHashMD5(d.uploadTempDir, r, size)
if err != nil {
return UploadResult{}, err
}
@@ -754,8 +757,8 @@ func (d *Driver) request(ctx context.Context, endpoint, method string, configure
return nil, errors.New("123pan request: unauthorized")
}
func isP123RateLimitResponse(res *resty.Response, code int, message string) bool {
if code == http.StatusTooManyRequests || isP123RateLimitMessage(message) {
func isP123RateLimitResponse(res *resty.Response, code int, _ string) bool {
if code == http.StatusTooManyRequests {
return true
}
if res == nil {
@@ -764,7 +767,7 @@ func isP123RateLimitResponse(res *resty.Response, code int, message string) bool
return isP123RateLimitHTTPResponse(res.StatusCode(), res.Header().Get("Retry-After"), res.String())
}
func isP123RateLimitHTTPResponse(status int, retryAfter, body string) bool {
func isP123RateLimitHTTPResponse(status int, retryAfter, _ string) bool {
if status == http.StatusTooManyRequests {
return true
}
@@ -774,35 +777,9 @@ func isP123RateLimitHTTPResponse(status int, retryAfter, body string) bool {
return true
}
}
if isP123RateLimitMessage(body) {
return true
}
return false
}
func isP123RateLimitMessage(message string) bool {
text := strings.ToLower(strings.TrimSpace(message))
if text == "" {
return false
}
return strings.Contains(text, "请求太频繁") ||
strings.Contains(text, "请求过于频繁") ||
strings.Contains(text, "请求频繁") ||
strings.Contains(text, "操作频繁") ||
strings.Contains(text, "频率限制") ||
strings.Contains(text, "请求次数过多") ||
strings.Contains(text, "too many request") ||
strings.Contains(text, "too many requests") ||
strings.Contains(text, "rate limit") ||
strings.Contains(text, "rate-limit") ||
strings.Contains(text, "ratelimit") ||
strings.Contains(text, "throttl") ||
strings.Contains(text, "temporarily blocked") ||
strings.Contains(text, "request has been blocked") ||
strings.Contains(text, "blocked") ||
strings.Contains(text, "访问被阻断")
}
func p123RateLimitError(res *resty.Response, code int, message string) error {
if strings.TrimSpace(message) == "" {
message = "123pan rate limited"
@@ -1084,8 +1061,14 @@ func splitPath(p string) []string {
return strings.Split(p, "/")
}
func bufferAndHashMD5(r io.Reader, declaredSize int64) (*os.File, string, int64, error) {
tmp, err := os.CreateTemp("", "p123-upload-*.bin")
func bufferAndHashMD5(tempDir string, r io.Reader, declaredSize int64) (*os.File, string, int64, error) {
tempDir = strings.TrimSpace(tempDir)
if tempDir != "" {
if err := os.MkdirAll(tempDir, 0o755); err != nil {
return nil, "", 0, fmt.Errorf("123pan upload: create tmp dir: %w", err)
}
}
tmp, err := os.CreateTemp(tempDir, "p123-upload-*.bin")
if err != nil {
return nil, "", 0, fmt.Errorf("123pan upload: create tmp: %w", err)
}
@@ -11,6 +11,8 @@ import (
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"time"
@@ -458,6 +460,29 @@ func TestUploadPresignedPUT429ReturnsRateLimitError(t *testing.T) {
}
}
func TestBufferAndHashMD5UsesConfiguredTempDir(t *testing.T) {
body := []byte("hello-123-upload-test")
tempDir := filepath.Join(t.TempDir(), "upload-tmp")
tmp, gotHex, n, err := bufferAndHashMD5(tempDir, bytes.NewReader(body), int64(len(body)))
if err != nil {
t.Fatalf("bufferAndHashMD5 returned error: %v", err)
}
defer func() {
_ = tmp.Close()
_ = os.Remove(tmp.Name())
}()
if gotDir := filepath.Dir(tmp.Name()); gotDir != tempDir {
t.Fatalf("tmp dir = %q, want %q", gotDir, tempDir)
}
want := md5.Sum(body)
if gotHex != fmt.Sprintf("%x", want) {
t.Fatalf("md5 = %s, want %x", gotHex, want)
}
if n != int64(len(body)) {
t.Fatalf("written = %d, want %d", n, len(body))
}
}
func TestRenameSendsExpectedBody(t *testing.T) {
var renameRequest map[string]any
api := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+13 -19
View File
@@ -47,6 +47,7 @@ type Driver struct {
client *resty.Client
onTokenUpdate func(access, refresh, captcha, deviceID string)
uploadToOSSFunc func(context.Context, *s3Params, io.Reader) error
uploadTempDir string
// captchaMu serializes captcha-token refreshes triggered by 4002 / 9
// recovery in requestOnce. Without it, N concurrent callers all hitting
@@ -77,6 +78,7 @@ type Config struct {
DeviceID string
RootID string
DisableMediaLink bool
UploadTempDir string
OnTokenUpdate func(access, refresh, captcha, deviceID string)
}
@@ -109,6 +111,7 @@ func New(c Config) *Driver {
deviceID: deviceID,
disableMediaLink: c.DisableMediaLink,
onTokenUpdate: c.OnTokenUpdate,
uploadTempDir: strings.TrimSpace(c.UploadTempDir),
client: resty.New().
SetTimeout(30*time.Second).
SetHeader("Accept", "application/json, text/plain, */*"),
@@ -175,8 +178,8 @@ func (d *Driver) List(ctx context.Context, dirID string) ([]drives.Entry, error)
// pikpakListCooldown 是列目录触发疑似限流错误时的冷却时长。
//
// 与 p115 driver 的 listCooldown 同语义:只要错误属 transient
// error_code=10 / HTTP 429 / 5xx / 通用 "rate limit" 文本),就持续
// 与 p115 driver 的 listCooldown 同语义:只要错误属明确限流/临时状态
// 结构化 error_code=10 / HTTP 429 / 5xx),就持续
// 等 10 分钟再发一次列目录请求,直到成功或 ctx 取消。这样即使 PikPak
// 风控持续较长时间,扫描会自然延后到风控结束,不再丢半棵子树。
const pikpakListCooldown = 10 * time.Minute
@@ -242,7 +245,6 @@ func pikpakSleepContext(ctx context.Context, d time.Duration) error {
//
// - PikPak 业务码 error_code=10 ("操作频繁",见 OpenList drivers/pikpak/util.go)
// - HTTP 429 / 500 / 502 / 503 / 504 / 509rclone 也把这些归为 retry
// - 通用文本:rate limit / too many requests / blocked / temporarily unavailable
//
// 不包含 4122/4121/16access_token 过期)和 9/4002captcha 过期)—— 这些
// 由 requestOnce 内部已经做过一次自动恢复重试;如果恢复后仍然报这类错误,
@@ -259,22 +261,14 @@ func isTransientPikPakListError(err error) bool {
return true
}
}
text := strings.ToLower(err.Error())
return strings.Contains(text, "error_code=10") ||
strings.Contains(text, "429") ||
strings.Contains(text, "http 500") ||
strings.Contains(text, "http 502") ||
strings.Contains(text, "http 503") ||
strings.Contains(text, "http 504") ||
strings.Contains(text, "http 509") ||
strings.Contains(text, "too many request") ||
strings.Contains(text, "too many requests") ||
strings.Contains(text, "rate limit") ||
strings.Contains(text, "operation frequent") ||
strings.Contains(text, "操作频繁") ||
strings.Contains(text, "blocked") ||
strings.Contains(text, "temporarily unavailable") ||
strings.Contains(text, "service unavailable")
return drives.ErrorMentionsHTTPStatus(err,
http.StatusTooManyRequests,
http.StatusInternalServerError,
http.StatusBadGateway,
http.StatusServiceUnavailable,
http.StatusGatewayTimeout,
509,
)
}
func (d *Driver) Stat(ctx context.Context, fileID string) (*drives.Entry, error) {
+78 -14
View File
@@ -79,6 +79,20 @@ type UploadResult struct {
Size int64
}
type preparedUploadBody struct {
reader io.ReadSeeker
start int64
cleanup func()
}
func (b preparedUploadBody) rewind() error {
if b.reader == nil {
return errors.New("pikpak upload: nil upload body")
}
_, err := b.reader.Seek(b.start, io.SeekStart)
return err
}
// Upload 实现 drives.Drive 接口;只返回 fileID。
// 完整上传元数据见 UploadAndReportHash。
func (d *Driver) Upload(ctx context.Context, parentID, name string, r io.Reader, size int64) (string, error) {
@@ -125,15 +139,15 @@ func (d *Driver) UploadAndReportHash(ctx context.Context, parentID, name string,
parentID = d.rootID
}
// 1) 把 r 全量缓冲到临时文件,同时算 GCID。
tmp, gcidHex, actualSize, err := bufferAndHashGCID(r, size)
// 1) 算 GCID,并准备一个可重试读取的 body。爬虫迁移传入的是
// *os.File,可直接复用原文件,避免再占用一份视频大小的临时空间。
body, gcidHex, actualSize, err := d.prepareUploadBody(r, size)
if err != nil {
return UploadResult{}, err
}
defer func() {
_ = tmp.Close()
_ = os.Remove(tmp.Name())
}()
if body.cleanup != nil {
defer body.cleanup()
}
result := UploadResult{Hash: gcidHex, Size: actualSize}
var lastErr error
@@ -155,7 +169,7 @@ func (d *Driver) UploadAndReportHash(ctx context.Context, parentID, name string,
continue
}
out, err := d.completeUploadAttempt(ctx, tmp, parentID, name, result, resp)
out, err := d.completeUploadAttempt(ctx, body, parentID, name, result, resp)
if err == nil {
return out, nil
}
@@ -190,7 +204,7 @@ func (d *Driver) requestUploadSession(ctx context.Context, parentID, name string
return resp, nil
}
func (d *Driver) completeUploadAttempt(ctx context.Context, tmp *os.File, parentID, name string, result UploadResult, resp uploadTaskData) (UploadResult, error) {
func (d *Driver) completeUploadAttempt(ctx context.Context, body preparedUploadBody, parentID, name string, result UploadResult, resp uploadTaskData) (UploadResult, error) {
// 命中秒传:服务端已经知道这个 hash,直接返回新文件 ID。
if resp.Resumable == nil {
if resp.File.ID != "" {
@@ -207,10 +221,10 @@ func (d *Driver) completeUploadAttempt(ctx context.Context, tmp *os.File, parent
}
// 未命中秒传:把字节传到 S3 兼容存储。
if _, err := tmp.Seek(0, io.SeekStart); err != nil {
return UploadResult{}, fmt.Errorf("pikpak upload: seek tmp: %w", err)
if err := body.rewind(); err != nil {
return UploadResult{}, fmt.Errorf("pikpak upload: rewind body: %w", err)
}
if err := d.uploadToOSS(ctx, &resp.Resumable.Params, tmp); err != nil {
if err := d.uploadToOSS(ctx, &resp.Resumable.Params, body.reader); err != nil {
return UploadResult{}, fmt.Errorf("pikpak upload: oss put: %w", err)
}
@@ -279,12 +293,62 @@ func isRetryablePikPakUploadError(err error) bool {
strings.Contains(text, "service unavailable")
}
func (d *Driver) prepareUploadBody(r io.Reader, size int64) (preparedUploadBody, string, int64, error) {
if rs, ok := r.(io.ReadSeeker); ok {
gcidHex, actualSize, start, err := hashGCIDFromReadSeeker(rs, size)
if err != nil {
return preparedUploadBody{}, "", 0, err
}
return preparedUploadBody{reader: rs, start: start, cleanup: func() {}}, gcidHex, actualSize, nil
}
tmp, gcidHex, actualSize, err := bufferAndHashGCID(d.uploadTempDir, r, size)
if err != nil {
return preparedUploadBody{}, "", 0, err
}
return preparedUploadBody{
reader: tmp,
start: 0,
cleanup: func() {
_ = tmp.Close()
_ = os.Remove(tmp.Name())
},
}, gcidHex, actualSize, nil
}
func hashGCIDFromReadSeeker(r io.ReadSeeker, size int64) (string, int64, int64, error) {
start, err := r.Seek(0, io.SeekCurrent)
if err != nil {
return "", 0, 0, fmt.Errorf("pikpak upload: seek body: %w", err)
}
h := NewGCID(size)
written, copyErr := io.Copy(h, r)
_, seekErr := r.Seek(start, io.SeekStart)
if copyErr != nil {
return "", 0, start, fmt.Errorf("pikpak upload: hash body: %w", copyErr)
}
if seekErr != nil {
return "", 0, start, fmt.Errorf("pikpak upload: rewind body: %w", seekErr)
}
if size > 0 && written != size {
return "", 0, start, fmt.Errorf("pikpak upload: size mismatch: declared %d, copied %d", size, written)
}
return strings.ToUpper(hex.EncodeToString(h.Sum(nil))), written, start, nil
}
// bufferAndHashGCID 把 r 复制到一个临时文件,同时计算 GCID。
// 返回临时文件(位置在末尾,需要调用方 Seek 回 0)、GCID hex 大写、实际写入字节数。
// 返回临时文件(位置在末尾,需要调用方 Seek 回 start)、GCID hex 大写、实际写入字节数。
//
// 调用方负责 Close + Remove 临时文件。
func bufferAndHashGCID(r io.Reader, size int64) (*os.File, string, int64, error) {
tmp, err := os.CreateTemp("", "pikpak-upload-*.bin")
func bufferAndHashGCID(tempDir string, r io.Reader, size int64) (*os.File, string, int64, error) {
tempDir = strings.TrimSpace(tempDir)
if tempDir != "" {
if err := os.MkdirAll(tempDir, 0o755); err != nil {
return nil, "", 0, fmt.Errorf("pikpak upload: create tmp dir: %w", err)
}
}
tmp, err := os.CreateTemp(tempDir, "pikpak-upload-*.bin")
if err != nil {
return nil, "", 0, fmt.Errorf("pikpak upload: create tmp: %w", err)
}
+78 -2
View File
@@ -11,6 +11,8 @@ import (
"net"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
@@ -142,6 +144,80 @@ func TestUploadInstantSuccessReturnsFileID(t *testing.T) {
}
}
func TestUploadUsesReadSeekerWithoutTempCopy(t *testing.T) {
mux := http.NewServeMux()
mux.HandleFunc("/drive/v1/files", func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{
"upload_type": "UPLOAD_TYPE_RESUMABLE",
"resumable": null,
"file": {"id": "instant-file-id", "name": "test.mp4", "kind": "drive#file"}
}`))
})
server := httptest.NewServer(mux)
defer server.Close()
d := newTestDriver(t, server)
uploadTempDir := filepath.Join(t.TempDir(), "upload-tmp")
d.uploadTempDir = uploadTempDir
data := bytes.Repeat([]byte{0x31}, 1024)
path := filepath.Join(t.TempDir(), "video.bin")
if err := os.WriteFile(path, data, 0o644); err != nil {
t.Fatalf("write source: %v", err)
}
f, err := os.Open(path)
if err != nil {
t.Fatalf("open source: %v", err)
}
defer f.Close()
id, err := d.Upload(context.Background(), "parent-id", "test.mp4", f, int64(len(data)))
if err != nil {
t.Fatalf("upload: %v", err)
}
if id != "instant-file-id" {
t.Fatalf("file id = %q, want instant-file-id", id)
}
if _, err := os.Stat(uploadTempDir); !os.IsNotExist(err) {
t.Fatalf("upload temp dir stat err = %v, want not created for read seeker input", err)
}
}
func TestUploadBuffersNonSeekReaderInConfiguredTempDir(t *testing.T) {
mux := http.NewServeMux()
mux.HandleFunc("/drive/v1/files", func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{
"upload_type": "UPLOAD_TYPE_RESUMABLE",
"resumable": null,
"file": {"id": "instant-file-id", "name": "test.mp4", "kind": "drive#file"}
}`))
})
server := httptest.NewServer(mux)
defer server.Close()
d := newTestDriver(t, server)
uploadTempDir := filepath.Join(t.TempDir(), "upload-tmp")
d.uploadTempDir = uploadTempDir
data := bytes.Repeat([]byte{0x42}, 1024)
id, err := d.Upload(context.Background(), "parent-id", "test.mp4", bytes.NewBuffer(data), int64(len(data)))
if err != nil {
t.Fatalf("upload: %v", err)
}
if id != "instant-file-id" {
t.Fatalf("file id = %q, want instant-file-id", id)
}
entries, err := os.ReadDir(uploadTempDir)
if err != nil {
t.Fatalf("read upload temp dir: %v", err)
}
if len(entries) != 0 {
t.Fatalf("upload temp dir entries = %d, want cleaned", len(entries))
}
}
func TestUploadInstantSuccessFallsBackToListWhenFileIDMissing(t *testing.T) {
listCalled := false
mux := http.NewServeMux()
@@ -304,7 +380,7 @@ func TestUploadRejectsInvalidArguments(t *testing.T) {
func TestBufferAndHashGCIDDetectsSizeMismatch(t *testing.T) {
src := bytes.NewReader([]byte("hello"))
// 声明 size=10 但实际只有 5 字节
_, _, _, err := bufferAndHashGCID(src, 10)
_, _, _, err := bufferAndHashGCID("", src, 10)
if err == nil {
t.Fatal("expected size mismatch error")
}
@@ -315,7 +391,7 @@ func TestBufferAndHashGCIDDetectsSizeMismatch(t *testing.T) {
func TestBufferAndHashGCIDComputesCorrectHash(t *testing.T) {
data := bytes.Repeat([]byte{0x55}, 1024)
tmp, hex, written, err := bufferAndHashGCID(bytes.NewReader(data), int64(len(data)))
tmp, hex, written, err := bufferAndHashGCID("", bytes.NewReader(data), int64(len(data)))
if err != nil {
t.Fatalf("buffer: %v", err)
}
@@ -50,6 +50,7 @@ type CrawlerConfig struct {
CommonThumbDir string
ProxyURL string
ConfigJSON string
DisablePreview bool
HTTPClient *http.Client
DownloadTimeout time.Duration
OnProgress func(CrawlProgress)
@@ -562,6 +563,10 @@ func (c *Crawler) processItem(ctx context.Context, item Item) (bool, error) {
if quality == "" {
quality = "HD"
}
previewStatus := "pending"
if c.previewDisabled(ctx) {
previewStatus = "disabled"
}
v := &catalog.Video{
ID: videoID,
DriveID: c.cfg.Driver.ID(),
@@ -576,7 +581,7 @@ func (c *Crawler) processItem(ctx context.Context, item Item) (bool, error) {
Quality: quality,
Category: strings.TrimSpace(item.Category),
Description: strings.TrimSpace(item.Description),
PreviewStatus: "pending",
PreviewStatus: previewStatus,
PublishedAt: publishedAt,
CreatedAt: now,
UpdatedAt: now,
@@ -632,6 +637,18 @@ func (c *Crawler) processItem(ctx context.Context, item Item) (bool, error) {
return true, nil
}
func (c *Crawler) previewDisabled(ctx context.Context) bool {
if c == nil {
return false
}
if c.cfg.Catalog != nil && c.cfg.Driver != nil {
if d, err := c.cfg.Catalog.GetDrive(ctx, c.cfg.Driver.ID()); err == nil && d != nil {
return !d.TeaserEnabled
}
}
return c.cfg.DisablePreview
}
func (c *Crawler) materializeMedia(ctx context.Context, ref MediaRef, dst, referer string, required bool) (int64, error) {
if local := strings.TrimSpace(ref.LocalFile); local != "" {
return c.copyLocalOutput(local, dst)
@@ -114,6 +114,128 @@ func TestCrawlerRunOnceImportsLocalFileAndSkipsExisting(t *testing.T) {
}
}
func TestCrawlerRunOnceMarksPreviewDisabledWhenConfigured(t *testing.T) {
ctx := context.Background()
tmp := t.TempDir()
cat, err := catalog.Open(filepath.Join(tmp, "catalog.db"))
if err != nil {
t.Fatalf("open catalog: %v", err)
}
t.Cleanup(func() {
if err := cat.Close(); err != nil {
t.Fatalf("close catalog: %v", err)
}
})
drv := New(Config{ID: "demo", RootDir: filepath.Join(tmp, "crawler")})
if err := drv.Init(ctx); err != nil {
t.Fatalf("driver init: %v", err)
}
dummyScript := filepath.Join(tmp, "helper-script")
if err := os.WriteFile(dummyScript, []byte("helper"), 0o755); err != nil {
t.Fatalf("write dummy script: %v", err)
}
wrapper := filepath.Join(tmp, "helper-wrapper.sh")
wrapperScript := fmt.Sprintf("#!/bin/sh\nexec %q -test.run=TestScriptCrawlerHelperProcess \"$@\"\n", os.Args[0])
if err := os.WriteFile(wrapper, []byte(wrapperScript), 0o755); err != nil {
t.Fatalf("write helper wrapper: %v", err)
}
t.Setenv("GO_WANT_SCRIPTCRAWLER_HELPER", "1")
c := NewCrawler(CrawlerConfig{
Driver: drv,
Catalog: cat,
PythonPath: wrapper,
FFprobePath: writeScriptCrawlerFFprobeStub(t, tmp, true),
ScriptPath: dummyScript,
DisablePreview: true,
})
res, err := c.RunOnce(ctx, 1)
if err != nil {
t.Fatalf("run once: %v", err)
}
if res.NewVideos != 1 || res.Failed != 0 {
t.Fatalf("result = new:%d failed:%d, want 1/0", res.NewVideos, res.Failed)
}
v, err := cat.GetVideo(ctx, BuildVideoID("demo", "abc-123"))
if err != nil {
t.Fatalf("get video: %v", err)
}
if v.PreviewStatus != "disabled" {
t.Fatalf("preview status = %q, want disabled", v.PreviewStatus)
}
if v.FingerprintStatus != "ready" || v.SampledSHA256 == "" {
t.Fatalf("fingerprint status=%q sampled=%q, want ready and sampled hash", v.FingerprintStatus, v.SampledSHA256)
}
pending, err := cat.ListVideosByPreviewStatus(ctx, "demo", "pending", 0)
if err != nil {
t.Fatalf("list pending previews: %v", err)
}
if len(pending) != 0 {
t.Fatalf("pending previews = %d, want 0", len(pending))
}
}
func TestCrawlerRunOnceUsesCurrentDrivePreviewSwitch(t *testing.T) {
ctx := context.Background()
tmp := t.TempDir()
cat, err := catalog.Open(filepath.Join(tmp, "catalog.db"))
if err != nil {
t.Fatalf("open catalog: %v", err)
}
t.Cleanup(func() {
if err := cat.Close(); err != nil {
t.Fatalf("close catalog: %v", err)
}
})
drv := New(Config{ID: "demo", RootDir: filepath.Join(tmp, "crawler")})
if err := drv.Init(ctx); err != nil {
t.Fatalf("driver init: %v", err)
}
if err := cat.UpsertDrive(ctx, &catalog.Drive{
ID: drv.ID(),
Kind: Kind,
Name: "Demo",
RootID: "/",
Credentials: map[string]string{"script_path": "/tmp/crawler.py"},
TeaserEnabled: true,
}); err != nil {
t.Fatalf("seed drive: %v", err)
}
dummyScript := filepath.Join(tmp, "helper-script")
if err := os.WriteFile(dummyScript, []byte("helper"), 0o755); err != nil {
t.Fatalf("write dummy script: %v", err)
}
wrapper := filepath.Join(tmp, "helper-wrapper.sh")
wrapperScript := fmt.Sprintf("#!/bin/sh\nexec %q -test.run=TestScriptCrawlerHelperProcess \"$@\"\n", os.Args[0])
if err := os.WriteFile(wrapper, []byte(wrapperScript), 0o755); err != nil {
t.Fatalf("write helper wrapper: %v", err)
}
t.Setenv("GO_WANT_SCRIPTCRAWLER_HELPER", "1")
c := NewCrawler(CrawlerConfig{
Driver: drv,
Catalog: cat,
PythonPath: wrapper,
FFprobePath: writeScriptCrawlerFFprobeStub(t, tmp, true),
ScriptPath: dummyScript,
DisablePreview: true,
})
res, err := c.RunOnce(ctx, 1)
if err != nil {
t.Fatalf("run once: %v", err)
}
if res.NewVideos != 1 || res.Failed != 0 {
t.Fatalf("result = new:%d failed:%d, want 1/0", res.NewVideos, res.Failed)
}
v, err := cat.GetVideo(ctx, BuildVideoID("demo", "abc-123"))
if err != nil {
t.Fatalf("get video: %v", err)
}
if v.PreviewStatus != "pending" {
t.Fatalf("preview status = %q, want pending from current drive switch", v.PreviewStatus)
}
}
func TestCrawlerRunOnceUsesSourceKindNamespace(t *testing.T) {
ctx := context.Background()
tmp := t.TempDir()
+52 -33
View File
@@ -67,6 +67,55 @@ type DryRunResult struct {
DurationMs int64 `json:"durationMs"`
}
type dryRunLogTail struct {
mu sync.Mutex
lines []string
partial string
}
func newDryRunLogTail() *dryRunLogTail {
return &dryRunLogTail{lines: make([]string, 0, dryRunLogTailLines)}
}
func (t *dryRunLogTail) Write(p []byte) (int, error) {
t.mu.Lock()
defer t.mu.Unlock()
chunk := strings.ReplaceAll(string(p), "\r\n", "\n")
parts := strings.Split(t.partial+chunk, "\n")
t.partial = parts[len(parts)-1]
for _, line := range parts[:len(parts)-1] {
t.appendLocked(line)
}
return len(p), nil
}
func (t *dryRunLogTail) snapshot() []string {
t.mu.Lock()
defer t.mu.Unlock()
lines := append([]string{}, t.lines...)
if partial := strings.TrimSpace(t.partial); partial != "" {
lines = appendDryRunLogLine(lines, partial)
}
return lines
}
func (t *dryRunLogTail) appendLocked(line string) {
t.lines = appendDryRunLogLine(t.lines, line)
}
func appendDryRunLogLine(lines []string, line string) []string {
line = strings.TrimSpace(line)
if line == "" {
return lines
}
if len(lines) >= dryRunLogTailLines {
lines = lines[1:]
}
return append(lines, line)
}
func DryRun(ctx context.Context, cfg DryRunConfig) *DryRunResult {
started := time.Now()
result := &DryRunResult{Items: []DryRunItem{}}
@@ -169,41 +218,14 @@ func DryRun(ctx context.Context, cfg DryRunConfig) *DryRunResult {
result.Error = fmt.Sprintf("启动脚本失败: %v", err)
return result
}
stderr, err := cmd.StderrPipe()
if err != nil {
_ = stdout.Close()
result.Error = fmt.Sprintf("启动脚本失败: %v", err)
return result
}
logTail := newDryRunLogTail()
cmd.Stderr = logTail
if err := cmd.Start(); err != nil {
_ = stdout.Close()
_ = stderr.Close()
result.Error = fmt.Sprintf("启动脚本失败: %v", err)
return result
}
// stderr 是脚本日志,保留尾部若干行用于排错回显。
var logMu sync.Mutex
logTail := make([]string, 0, dryRunLogTailLines)
stderrDone := make(chan struct{})
go func() {
defer close(stderrDone)
scanner := bufio.NewScanner(stderr)
scanner.Buffer(make([]byte, 64*1024), 1024*1024)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
if line == "" {
continue
}
logMu.Lock()
if len(logTail) >= dryRunLogTailLines {
logTail = logTail[1:]
}
logTail = append(logTail, line)
logMu.Unlock()
}
}()
items := []DryRunItem{}
var firstMediaHeaders map[string]string
parseFailures := 0
@@ -264,11 +286,8 @@ func DryRun(ctx context.Context, cfg DryRunConfig) *DryRunResult {
_ = killDryRunProcess(cmd)
<-waitDone
}
<-stderrDone
logMu.Lock()
result.Log = append([]string{}, logTail...)
logMu.Unlock()
result.Log = logTail.snapshot()
result.Items = items
if len(items) == 0 {
@@ -48,6 +48,29 @@ echo '{"type":"done","stats":{"emitted":1}}'
}
}
func TestDryRunCapturesStderrWhenStoppingAfterFirstItem(t *testing.T) {
script := writeDryRunScript(t, `
echo '[log] first item ready' >&2
echo '{"type":"item","item":{"title":"Early Stop Video","media_url":"https://cdn.example.test/v.mp4","source_id":"early-stop"}}'
sleep 30
`)
start := time.Now()
result := DryRun(context.Background(), DryRunConfig{
PythonPath: "/bin/sh",
ScriptPath: script,
SkipMediaProbe: true,
})
if !result.OK {
t.Fatalf("ok = false, error = %q, log = %v", result.Error, result.Log)
}
if elapsed := time.Since(start); elapsed > 5*time.Second {
t.Fatalf("dry run took %s, script was not stopped after first item", elapsed)
}
if len(result.Log) == 0 || !strings.Contains(result.Log[0], "first item ready") {
t.Fatalf("log tail = %v, want stderr captured before early stop", result.Log)
}
}
func TestDryRunProbesMediaURL(t *testing.T) {
var gotRange, gotReferer string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+22 -42
View File
@@ -27,6 +27,7 @@ type Driver struct {
refreshToken string
client *sdk.WoClient
onTokenUpdate func(access, refresh string)
uploadTempDir string
listMu sync.Mutex
lastListAt time.Time
@@ -38,11 +39,12 @@ type Driver struct {
}
type Config struct {
ID string
AccessToken string
RefreshToken string
FamilyID string // 空则走个人空间,有值则走家庭空间
RootID string // 根目录 ID,默认 "0"
ID string
AccessToken string
RefreshToken string
FamilyID string // 空则走个人空间,有值则走家庭空间
RootID string // 根目录 ID,默认 "0"
UploadTempDir string
// 当 SDK 刷新 token 时回调,便于持久化
OnTokenUpdate func(access, refresh string)
}
@@ -59,6 +61,7 @@ func New(c Config) *Driver {
accessToken: c.AccessToken,
refreshToken: c.RefreshToken,
onTokenUpdate: c.OnTokenUpdate,
uploadTempDir: strings.TrimSpace(c.UploadTempDir),
listInterval: 800 * time.Millisecond,
listCooldown: 5 * time.Minute,
fidToID: make(map[string]string),
@@ -162,7 +165,12 @@ func (d *Driver) StreamURL(ctx context.Context, fileID string) (*drives.StreamLi
func (d *Driver) Upload(ctx context.Context, parentID, name string, r io.Reader, size int64) (string, error) {
// wopan SDK 要求 *os.File,先把流落到临时文件再上传
tmp, err := os.CreateTemp("", "wopan-upload-*.tmp")
if d.uploadTempDir != "" {
if err := os.MkdirAll(d.uploadTempDir, 0o755); err != nil {
return "", fmt.Errorf("wopan upload: create tmp dir: %w", err)
}
}
tmp, err := os.CreateTemp(d.uploadTempDir, "wopan-upload-*.tmp")
if err != nil {
return "", err
}
@@ -510,42 +518,14 @@ func isWopanRateLimitError(err error) bool {
if err == nil || errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
return false
}
text := strings.ToLower(strings.TrimSpace(err.Error()))
if text == "" {
return false
}
return strings.Contains(text, "status: 429") ||
strings.Contains(text, "status 429") ||
strings.Contains(text, "http status: 429") ||
strings.Contains(text, "status: 500") ||
strings.Contains(text, "status 500") ||
strings.Contains(text, "status: 502") ||
strings.Contains(text, "status 502") ||
strings.Contains(text, "status: 503") ||
strings.Contains(text, "status 503") ||
strings.Contains(text, "status: 504") ||
strings.Contains(text, "status 504") ||
strings.Contains(text, "status: 509") ||
strings.Contains(text, "status 509") ||
strings.Contains(text, "too many request") ||
strings.Contains(text, "too many requests") ||
strings.Contains(text, "rate limit") ||
strings.Contains(text, "rate-limit") ||
strings.Contains(text, "throttl") ||
strings.Contains(text, "blocked") ||
strings.Contains(text, "request has been blocked") ||
strings.Contains(text, "操作频繁") ||
strings.Contains(text, "请求频繁") ||
strings.Contains(text, "请求太频繁") ||
strings.Contains(text, "请求过于频繁") ||
strings.Contains(text, "频率限制") ||
strings.Contains(text, "请求次数过多") ||
strings.Contains(text, "系统繁忙") ||
strings.Contains(text, "服务繁忙") ||
strings.Contains(text, "稍后再试") ||
strings.Contains(text, "稍后重试") ||
strings.Contains(text, "访问被阻断") ||
strings.Contains(text, "风控")
return drives.ErrorMentionsHTTPStatus(err,
http.StatusTooManyRequests,
http.StatusInternalServerError,
http.StatusBadGateway,
http.StatusServiceUnavailable,
http.StatusGatewayTimeout,
509,
)
}
func guessMime(name string) string {
+14 -36
View File
@@ -372,37 +372,10 @@ func remoteRangeResponseLooksRateLimited(rawURL string, status int, body []byte)
status == 509) {
return true
}
text := strings.ToLower(strings.TrimSpace(string(body)))
compact := compactRemoteRangeErrorText(text)
if strings.Contains(text, "too many request") ||
strings.Contains(text, "too many requests") ||
strings.Contains(text, "rate limit") ||
strings.Contains(text, "quota exceeded") ||
strings.Contains(text, "操作频繁") ||
strings.Contains(text, "请求频繁") ||
strings.Contains(text, "请求太频繁") ||
strings.Contains(text, "请求过于频繁") ||
strings.Contains(text, "频率限制") ||
strings.Contains(text, "请求次数过多") ||
strings.Contains(text, "系统繁忙") ||
strings.Contains(text, "服务繁忙") ||
strings.Contains(text, "稍后再试") ||
strings.Contains(text, "稍后重试") ||
strings.Contains(text, "访问被阻断") ||
strings.Contains(text, "风控") ||
strings.Contains(text, "download quota") ||
strings.Contains(text, "sharing rate") ||
strings.Contains(text, "daily limit") ||
strings.Contains(text, "user rate") ||
strings.Contains(text, "usage limit") ||
strings.Contains(compact, "ratelimitexceeded") ||
strings.Contains(compact, "userratelimitexceeded") ||
strings.Contains(compact, "dailylimitexceeded") ||
strings.Contains(compact, "downloadquotaexceeded") ||
strings.Contains(compact, "sharingratelimitexceeded") ||
strings.Contains(compact, "quotaexceeded") ||
strings.Contains(compact, "toomanyrequests") ||
strings.Contains(compact, "usagelimits") {
if isGuangYaPanMediaURL(rawURL) && (status == http.StatusForbidden || status == http.StatusTooManyRequests ||
status == http.StatusInternalServerError || status == http.StatusBadGateway ||
status == http.StatusServiceUnavailable || status == http.StatusGatewayTimeout ||
status == 509) {
return true
}
if status == http.StatusForbidden && isGoogleDriveMediaURL(rawURL) {
@@ -424,6 +397,16 @@ func isWopanMediaURL(rawURL string) bool {
strings.Contains(path, "/openapi/download")
}
func isGuangYaPanMediaURL(rawURL string) bool {
u, err := url.Parse(rawURL)
if err != nil {
return false
}
host := strings.ToLower(u.Hostname())
return strings.HasSuffix(host, "guangyacdn.com") ||
strings.HasSuffix(host, "guangyapan.com")
}
func isGoogleDriveMediaURL(rawURL string) bool {
u, err := url.Parse(rawURL)
if err != nil {
@@ -434,11 +417,6 @@ func isGoogleDriveMediaURL(rawURL string) bool {
return strings.Contains(host, "googleapis.com") && strings.Contains(path, "/drive/")
}
func compactRemoteRangeErrorText(text string) string {
replacer := strings.NewReplacer("_", "", "-", "", " ", "", ".", "", ":", "")
return replacer.Replace(strings.ToLower(strings.TrimSpace(text)))
}
func parseRetryAfter(raw string) time.Duration {
raw = strings.TrimSpace(raw)
if raw == "" {
+28 -4
View File
@@ -86,16 +86,16 @@ func TestComputeRemoteUsesRangeSamples(t *testing.T) {
}
}
func TestComputeRemoteGoogleQuotaExceededReturnsRateLimit(t *testing.T) {
func TestComputeRemote429ReturnsRateLimit(t *testing.T) {
ctx := context.Background()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Retry-After", "60")
w.WriteHeader(http.StatusForbidden)
_, _ = w.Write([]byte(`{"error":{"code":403,"message":"The download quota for this file has been exceeded.","errors":[{"domain":"usageLimits","reason":"downloadQuotaExceeded","message":"The download quota for this file has been exceeded."}]}}`))
w.WriteHeader(http.StatusTooManyRequests)
_, _ = w.Write([]byte(`{"error":{"code":429}}`))
}))
defer srv.Close()
drv := &fakeDrive{paths: map[string]string{"remote": srv.URL + "/drive/v3/files/file-1?alt=media"}}
drv := &fakeDrive{paths: map[string]string{"remote": srv.URL + "/video.mp4"}}
_, err := Compute(ctx, drv, &catalog.Video{ID: "remote", FileID: "remote", Size: 1024 * 1024}, Config{
SampleSizeBytes: 4,
FullHashMaxSize: 8,
@@ -131,6 +131,30 @@ func TestWopanRemoteRangeErrorsLookRateLimited(t *testing.T) {
}
}
func TestGuangYaPanRemoteRangeErrorsLookRateLimited(t *testing.T) {
for _, tc := range []struct {
rawURL string
status int
}{
{rawURL: "https://txgz02-httpdown.guangyacdn.com/download/?fid=encoded", status: http.StatusForbidden},
{rawURL: "https://txgz02-httpdown.guangyacdn.com/download/?fid=encoded", status: http.StatusServiceUnavailable},
{rawURL: "https://txgz02-httpdown.guangyacdn.com/download/?fid=encoded", status: 509},
} {
if !remoteRangeResponseLooksRateLimited(tc.rawURL, tc.status, nil) {
t.Fatalf("remoteRangeResponseLooksRateLimited(%q, %d) = false, want true", tc.rawURL, tc.status)
}
}
if remoteRangeResponseLooksRateLimited("https://example.com/video.mp4", http.StatusForbidden, nil) {
t.Fatal("generic 403 should not be treated as guangyapan rate limit")
}
}
func TestGoogleDriveRemoteRangeForbiddenLooksRateLimitedByURL(t *testing.T) {
if !remoteRangeResponseLooksRateLimited("https://www.googleapis.com/drive/v3/files/file-1?alt=media", http.StatusForbidden, nil) {
t.Fatal("google drive media 403 should be treated as rate limit by URL and status")
}
}
type fakeDrive struct {
paths map[string]string
}
+32 -159
View File
@@ -952,15 +952,7 @@ func redactURLs(text string) string {
}
func ffmpegOutputLooksRateLimited(output []byte) bool {
text := strings.ToLower(string(output))
if !strings.Contains(text, "429") {
return false
}
return strings.Contains(text, "too many requests") ||
strings.Contains(text, "throttl") ||
strings.Contains(text, "rate limit") ||
strings.Contains(text, "rate-limit") ||
strings.Contains(text, "server returned 429")
return drives.TextMentionsHTTPStatus(string(output), http.StatusTooManyRequests)
}
// --- 本地落盘 ---
@@ -1064,12 +1056,10 @@ type ThumbWorker struct {
}
const (
defaultTransientMediaCooldown = 5 * time.Minute
defaultGenerationRateLimitCooldown = 5 * time.Minute
defaultThumbTransientMediaMaxFailures = 3
defaultWorkerQueueSize = 10000
maxPreviewTeaserSizeBytes int64 = 5 * 1024 * 1024 * 1024
previewStatusSkipped = "skipped"
defaultTransientMediaCooldown = 5 * time.Minute
defaultGenerationRateLimitCooldown = 5 * time.Minute
defaultThumbTransientMediaMaxFailures = 3
defaultWorkerQueueSize = 10000
)
type rateLimitState struct {
@@ -1124,6 +1114,19 @@ func (q *videoQueue) release(v *catalog.Video) {
q.mu.Unlock()
}
func (q *videoQueue) idsSnapshot() []string {
q.mu.Lock()
defer q.mu.Unlock()
if len(q.ids) == 0 {
return nil
}
out := make([]string, 0, len(q.ids))
for id := range q.ids {
out = append(out, id)
}
return out
}
func (q *videoQueue) lengthExcluding(currentID string) int {
q.mu.Lock()
defer q.mu.Unlock()
@@ -1251,6 +1254,13 @@ func (w *Worker) Status() TaskStatus {
return taskStatus(&w.activity, &w.rateLimit, w.queue.lengthExcluding(currentID))
}
func (w *Worker) ActiveVideoIDs() []string {
if w == nil {
return nil
}
return w.queue.idsSnapshot()
}
func (w *ThumbWorker) Status() TaskStatus {
if w == nil {
return TaskStatus{State: "idle"}
@@ -1518,145 +1528,21 @@ func driveErrorShouldCooldown(d drives.Drive, err error) bool {
}
switch d.Kind() {
case "p115":
text := strings.ToLower(err.Error())
return strings.Contains(text, "server returned 403") ||
strings.Contains(text, "403 forbidden") ||
strings.Contains(text, "server returned 405") ||
strings.Contains(text, "405 method") ||
strings.Contains(text, "access denied") ||
strings.Contains(text, "moov atom not found") ||
strings.Contains(text, "partial file") ||
strings.Contains(text, "request has been blocked") ||
strings.Contains(text, "访问被阻断")
return drives.ErrorMentionsHTTPStatus(err, http.StatusForbidden, http.StatusMethodNotAllowed, http.StatusTooManyRequests)
case "pikpak":
// PikPak 在预览视频 / 封面生成阶段(取链或拉直链字节)可能命中:
// - error_code=10 操作频繁
// - HTTP 429 / 5xx / 509 限流和服务端不可用
// - 通用文本:rate limit / too many requests / blocked
// 命中时让 worker 冷却 5 分钟,避免连续请求加重风控。
text := strings.ToLower(err.Error())
return strings.Contains(text, "error_code=10") ||
strings.Contains(text, "操作频繁") ||
strings.Contains(text, "429") ||
strings.Contains(text, "http 500") ||
strings.Contains(text, "http 502") ||
strings.Contains(text, "http 503") ||
strings.Contains(text, "http 504") ||
strings.Contains(text, "http 509") ||
strings.Contains(text, "too many request") ||
strings.Contains(text, "too many requests") ||
strings.Contains(text, "rate limit") ||
strings.Contains(text, "blocked") ||
strings.Contains(text, "partial file") ||
strings.Contains(text, "service unavailable")
return drives.ErrorMentionsHTTPStatus(err, http.StatusTooManyRequests, http.StatusInternalServerError, http.StatusBadGateway, http.StatusServiceUnavailable, http.StatusGatewayTimeout, 509)
case "p123":
// 123网盘直链解析 / ffmpeg 读取阶段可能返回 429、5xx,或 WAF 类
// blocked / 访问阻断文本。命中时冷却,避免封面和预览视频生成连续打接口。
text := strings.ToLower(err.Error())
return strings.Contains(text, "请求太频繁") ||
strings.Contains(text, "请求过于频繁") ||
strings.Contains(text, "请求频繁") ||
strings.Contains(text, "操作频繁") ||
strings.Contains(text, "频率限制") ||
strings.Contains(text, "请求次数过多") ||
strings.Contains(text, "429") ||
strings.Contains(text, "http 500") ||
strings.Contains(text, "http 502") ||
strings.Contains(text, "http 503") ||
strings.Contains(text, "http 504") ||
strings.Contains(text, "server returned 403") ||
strings.Contains(text, "403 forbidden") ||
strings.Contains(text, "too many request") ||
strings.Contains(text, "too many requests") ||
strings.Contains(text, "rate limit") ||
strings.Contains(text, "blocked") ||
strings.Contains(text, "访问被阻断") ||
strings.Contains(text, "service unavailable")
return drives.ErrorMentionsHTTPStatus(err, http.StatusForbidden, http.StatusTooManyRequests, http.StatusInternalServerError, http.StatusBadGateway, http.StatusServiceUnavailable, http.StatusGatewayTimeout)
case "wopan":
// 联通网盘的取链接口和下载直链都可能返回"操作频繁"、429、5xx
// 或 WAF 阻断文本。封面/预览失败时先冷却,避免持续触发风控。
text := strings.ToLower(err.Error())
return strings.Contains(text, "请求太频繁") ||
strings.Contains(text, "请求过于频繁") ||
strings.Contains(text, "请求频繁") ||
strings.Contains(text, "操作频繁") ||
strings.Contains(text, "频率限制") ||
strings.Contains(text, "请求次数过多") ||
strings.Contains(text, "系统繁忙") ||
strings.Contains(text, "服务繁忙") ||
strings.Contains(text, "稍后再试") ||
strings.Contains(text, "稍后重试") ||
strings.Contains(text, "429") ||
strings.Contains(text, "http 500") ||
strings.Contains(text, "http 502") ||
strings.Contains(text, "http 503") ||
strings.Contains(text, "http 504") ||
strings.Contains(text, "http 509") ||
strings.Contains(text, "server returned 403") ||
strings.Contains(text, "403 forbidden") ||
strings.Contains(text, "server returned 429") ||
strings.Contains(text, "server returned 500") ||
strings.Contains(text, "server returned 502") ||
strings.Contains(text, "server returned 503") ||
strings.Contains(text, "server returned 504") ||
strings.Contains(text, "too many request") ||
strings.Contains(text, "too many requests") ||
strings.Contains(text, "rate limit") ||
strings.Contains(text, "rate-limit") ||
strings.Contains(text, "throttl") ||
strings.Contains(text, "blocked") ||
strings.Contains(text, "request has been blocked") ||
strings.Contains(text, "访问被阻断") ||
strings.Contains(text, "风控") ||
strings.Contains(text, "service unavailable")
return drives.ErrorMentionsHTTPStatus(err, http.StatusForbidden, http.StatusTooManyRequests, http.StatusInternalServerError, http.StatusBadGateway, http.StatusServiceUnavailable, http.StatusGatewayTimeout, 509)
case "guangyapan":
return drives.ErrorMentionsHTTPStatus(err, http.StatusForbidden, http.StatusTooManyRequests, http.StatusInternalServerError, http.StatusBadGateway, http.StatusServiceUnavailable, http.StatusGatewayTimeout, 509)
case "googledrive":
// Google Drive 下载/取样阶段常把频控和配额问题包装成 403,
// 具体标识在 error.errors[].reason/message 里(OpenList 也按该结构解析)。
// ffmpeg/ffprobe 只能看到 stderr 文本时,按这些 reason/文本兜底冷却。
text := strings.ToLower(err.Error())
return googleDriveMediaErrorShouldCooldown(text)
return drives.ErrorMentionsHTTPStatus(err, http.StatusForbidden, http.StatusTooManyRequests, http.StatusInternalServerError, http.StatusBadGateway, http.StatusServiceUnavailable, http.StatusGatewayTimeout)
}
return false
}
func googleDriveMediaErrorShouldCooldown(text string) bool {
if text == "" {
return false
}
compact := compactGoogleDriveErrorText(text)
return strings.Contains(text, "server returned 403") ||
strings.Contains(text, "403 forbidden") ||
strings.Contains(text, "server returned 429") ||
strings.Contains(text, "http 429") ||
strings.Contains(text, "http 500") ||
strings.Contains(text, "http 502") ||
strings.Contains(text, "http 503") ||
strings.Contains(text, "http 504") ||
strings.Contains(text, "too many request") ||
strings.Contains(text, "too many requests") ||
strings.Contains(text, "rate limit") ||
strings.Contains(text, "quota exceeded") ||
strings.Contains(text, "download quota") ||
strings.Contains(text, "sharing rate") ||
strings.Contains(text, "daily limit") ||
strings.Contains(text, "user rate") ||
strings.Contains(text, "usage limit") ||
strings.Contains(text, "service unavailable") ||
strings.Contains(compact, "ratelimitexceeded") ||
strings.Contains(compact, "userratelimitexceeded") ||
strings.Contains(compact, "dailylimitexceeded") ||
strings.Contains(compact, "downloadquotaexceeded") ||
strings.Contains(compact, "sharingratelimitexceeded") ||
strings.Contains(compact, "quotaexceeded") ||
strings.Contains(compact, "toomanyrequests") ||
strings.Contains(compact, "usagelimits")
}
func compactGoogleDriveErrorText(text string) string {
replacer := strings.NewReplacer("_", "", "-", "", " ", "", ".", "", ":", "")
return replacer.Replace(strings.ToLower(strings.TrimSpace(text)))
}
func (w *ThumbWorker) process(ctx context.Context, v *catalog.Video) bool {
if w.skipIfRateLimited(v) {
return false
@@ -1806,15 +1692,6 @@ func localPreviewLink(v *catalog.Video) (*drives.StreamLink, bool) {
}
func (w *Worker) process(ctx context.Context, v *catalog.Video) {
if shouldSkipTeaser(v) {
removePreviousLocalTeaser(v.PreviewLocal, "")
if err := w.Catalog.UpdatePreview(ctx, v.ID, "", previewStatusSkipped); err != nil {
log.Printf("[preview] skip %s: update status: %v", v.Title, err)
return
}
log.Printf("[preview] skip %s: size=%d exceeds 5GiB teaser limit", v.Title, v.Size)
return
}
if w.skipIfRateLimited(v) {
return
}
@@ -1867,10 +1744,6 @@ func (w *Worker) process(ctx context.Context, v *catalog.Video) {
log.Printf("[preview] ready %s (duration=%.1fs)", v.Title, duration)
}
func shouldSkipTeaser(v *catalog.Video) bool {
return v != nil && v.Size > maxPreviewTeaserSizeBytes
}
func (w *Worker) generateTeaser(ctx context.Context, v *catalog.Video, link *drives.StreamLink, duration float64) (string, error) {
gen, ok := w.Gen.(refreshingTeaserGenerator)
if !ok || w.Drive == nil || w.Drive.Kind() != "p115" {
+56 -105
View File
@@ -349,42 +349,10 @@ func TestPreviewWorkerNeverCallsDriveUploadOrEnsureDir(t *testing.T) {
}
}
func TestPreviewWorkerSkipsTeaserForVideoLargerThanFiveGiB(t *testing.T) {
func TestPreviewWorkerGeneratesTeaserForLargeVideo(t *testing.T) {
ctx := context.Background()
cat, video := seedPreviewTestVideo(t, "preview-large-video")
video.Size = maxPreviewTeaserSizeBytes + 1
if err := cat.UpsertVideo(ctx, video); err != nil {
t.Fatalf("update video: %v", err)
}
gen := &fakeTeaserGenerator{}
drv := &previewFakeDrive{}
worker := NewWorker(gen, cat, drv)
worker.process(ctx, video)
got, err := cat.GetVideo(ctx, video.ID)
if err != nil {
t.Fatalf("get video: %v", err)
}
if got.PreviewStatus != previewStatusSkipped {
t.Fatalf("preview status = %q, want skipped", got.PreviewStatus)
}
if got.PreviewLocal != "" {
t.Fatalf("preview local = %q, want empty", got.PreviewLocal)
}
if drv.streamCalls != 0 {
t.Fatalf("stream calls = %d, want 0", drv.streamCalls)
}
if gen.generateCalls != 0 {
t.Fatalf("generate calls = %d, want 0", gen.generateCalls)
}
}
func TestPreviewWorkerGeneratesTeaserAtFiveGiBBoundary(t *testing.T) {
ctx := context.Background()
cat, video := seedPreviewTestVideo(t, "preview-five-gib-video")
video.Size = maxPreviewTeaserSizeBytes
video.Size = 6 * 1024 * 1024 * 1024
if err := cat.UpsertVideo(ctx, video); err != nil {
t.Fatalf("update video: %v", err)
}
@@ -485,9 +453,9 @@ func TestThumbWorkerRateLimitHonorsRetryAfter(t *testing.T) {
assertCooldownAround(t, worker.Status().CooldownUntil, before, 2*time.Hour)
}
func TestThumbWorkerP115TransientErrorFailsAfterRetryLimit(t *testing.T) {
func TestThumbWorkerP115MessageOnlyErrorFailsWithoutCooldown(t *testing.T) {
ctx := context.Background()
cat, video := seedPreviewTestVideo(t, "thumb-p115-transient")
cat, video := seedPreviewTestVideo(t, "thumb-p115-message-only")
gen := &fakeThumbGenerator{
generateErr: errors.New("ffmpeg thumb: exit status 183, stderr: partial file Cannot determine format of input 0:0 after EOF"),
@@ -495,69 +463,26 @@ func TestThumbWorkerP115TransientErrorFailsAfterRetryLimit(t *testing.T) {
drv := &previewFakeDrive{kind: "p115"}
worker := NewThumbWorker(gen, cat, drv)
for attempt := 1; attempt <= defaultThumbTransientMediaMaxFailures; attempt++ {
worker.rateLimit = rateLimitState{}
worker.process(ctx, video)
if attempt < defaultThumbTransientMediaMaxFailures {
pending, err := cat.ListVideosByThumbnailStatus(ctx, video.DriveID, "pending", 0)
if err != nil {
t.Fatalf("list pending thumbnails: %v", err)
}
if len(pending) != 1 || pending[0].ID != video.ID {
t.Fatalf("attempt %d pending thumbnails = %#v, want only %s", attempt, pending, video.ID)
}
missing, err := cat.CountVideosNeedingThumbnail(ctx, video.DriveID)
if err != nil {
t.Fatalf("count missing thumbnails: %v", err)
}
if missing != 1 {
t.Fatalf("attempt %d missing thumbnails = %d, want 1 before retry limit", attempt, missing)
}
continue
}
failed, err := cat.ListVideosByThumbnailStatus(ctx, video.DriveID, "failed", 0)
if err != nil {
t.Fatalf("list failed thumbnails: %v", err)
}
if len(failed) != 1 || failed[0].ID != video.ID {
t.Fatalf("failed thumbnails = %#v, want only %s", failed, video.ID)
}
missing, err := cat.CountVideosNeedingThumbnail(ctx, video.DriveID)
if err != nil {
t.Fatalf("count missing thumbnails: %v", err)
}
if missing != 0 {
t.Fatalf("missing thumbnails = %d, want 0 after retry limit marks failed", missing)
}
}
if gen.generateCalls != defaultThumbTransientMediaMaxFailures {
t.Fatalf("generate calls = %d, want %d", gen.generateCalls, defaultThumbTransientMediaMaxFailures)
}
if err := cat.UpdateVideoMeta(ctx, video.ID, catalog.VideoMetaPatch{
ThumbnailStatus: "pending",
ResetThumbnailFailures: true,
}); err != nil {
t.Fatalf("reset thumbnail status: %v", err)
}
worker.rateLimit = rateLimitState{}
worker.process(ctx, video)
pending, err := cat.ListVideosByThumbnailStatus(ctx, video.DriveID, "pending", 0)
failed, err := cat.ListVideosByThumbnailStatus(ctx, video.DriveID, "failed", 0)
if err != nil {
t.Fatalf("list pending thumbnails after reset: %v", err)
t.Fatalf("list failed thumbnails: %v", err)
}
if len(pending) != 1 || pending[0].ID != video.ID {
t.Fatalf("pending thumbnails after reset = %#v, want only %s", pending, video.ID)
if len(failed) != 1 || failed[0].ID != video.ID {
t.Fatalf("failed thumbnails = %#v, want only %s", failed, video.ID)
}
if !worker.Status().CooldownUntil.IsZero() {
t.Fatalf("cooldown until = %s, want no cooldown for message-only media error", worker.Status().CooldownUntil)
}
if gen.generateCalls != 1 {
t.Fatalf("generate calls = %d, want 1", gen.generateCalls)
}
}
func TestThumbWorkerRequeuesP115TransientErrorBeforeRetryLimit(t *testing.T) {
func TestThumbWorkerDoesNotRequeueP115MessageOnlyError(t *testing.T) {
ctx := context.Background()
cat, video := seedPreviewTestVideo(t, "thumb-p115-requeue")
cat, video := seedPreviewTestVideo(t, "thumb-p115-no-requeue")
gen := &fakeThumbGenerator{
generateErr: errors.New("ffmpeg thumb: partial file Cannot determine format of input 0:0 after EOF"),
@@ -569,11 +494,8 @@ func TestThumbWorkerRequeuesP115TransientErrorBeforeRetryLimit(t *testing.T) {
select {
case queued := <-worker.ch:
if queued.ID != video.ID {
t.Fatalf("requeued video id = %q, want %q", queued.ID, video.ID)
}
t.Fatalf("unexpected requeued video id = %q", queued.ID)
default:
t.Fatal("expected transient thumbnail failure to requeue the same video")
}
got, err := cat.GetVideo(ctx, video.ID)
@@ -581,14 +503,14 @@ func TestThumbWorkerRequeuesP115TransientErrorBeforeRetryLimit(t *testing.T) {
t.Fatalf("get video: %v", err)
}
if got.ThumbnailURL != "" {
t.Fatalf("thumbnail = %q, want empty after transient failure", got.ThumbnailURL)
t.Fatalf("thumbnail = %q, want empty after message-only failure", got.ThumbnailURL)
}
pending, err := cat.ListVideosByThumbnailStatus(ctx, video.DriveID, "pending", 0)
failed, err := cat.ListVideosByThumbnailStatus(ctx, video.DriveID, "failed", 0)
if err != nil {
t.Fatalf("list pending thumbnails: %v", err)
t.Fatalf("list failed thumbnails: %v", err)
}
if len(pending) != 1 || pending[0].ID != video.ID {
t.Fatalf("pending thumbnails = %#v, want only %s", pending, video.ID)
if len(failed) != 1 || failed[0].ID != video.ID {
t.Fatalf("failed thumbnails = %#v, want only %s", failed, video.ID)
}
}
@@ -649,13 +571,15 @@ func TestP123TransientErrorsShouldCooldown(t *testing.T) {
drv := &previewFakeDrive{kind: "p123"}
for _, err := range []error{
errors.New("Server returned 403 Forbidden"),
errors.New("请求太频繁"),
errors.New("http 503 service unavailable"),
} {
if !driveErrorShouldCooldown(drv, err) {
t.Fatalf("driveErrorShouldCooldown(%v) = false, want true", err)
}
}
if driveErrorShouldCooldown(drv, errors.New("请求太频繁")) {
t.Fatal("message-only throttling text should not trigger p123 cooldown")
}
if driveErrorShouldCooldown(drv, errors.New("invalid credential")) {
t.Fatal("invalid credential should not trigger p123 cooldown")
}
@@ -666,31 +590,58 @@ func TestWopanTransientErrorsShouldCooldown(t *testing.T) {
for _, err := range []error{
errors.New("ffmpeg: Server returned 403 Forbidden"),
errors.New("wopan download url: request failed with status: 429 Too Many Requests"),
errors.New("操作频繁,请稍后重试"),
errors.New("http 503 service unavailable"),
} {
if !driveErrorShouldCooldown(drv, err) {
t.Fatalf("driveErrorShouldCooldown(%v) = false, want true", err)
}
}
if driveErrorShouldCooldown(drv, errors.New("操作频繁,请稍后重试")) {
t.Fatal("message-only throttling text should not trigger wopan cooldown")
}
if driveErrorShouldCooldown(drv, errors.New("invalid access token")) {
t.Fatal("invalid access token should not trigger wopan cooldown")
}
}
func TestGuangYaPanTransientErrorsShouldCooldown(t *testing.T) {
drv := &previewFakeDrive{kind: "guangyapan"}
for _, err := range []error{
errors.New("ffmpeg: Server returned 403 Forbidden"),
errors.New("guangyapan api rate limited: status=429 msg=操作频繁,请稍后重试"),
errors.New("http 503 service unavailable"),
} {
if !driveErrorShouldCooldown(drv, err) {
t.Fatalf("driveErrorShouldCooldown(%v) = false, want true", err)
}
}
if driveErrorShouldCooldown(drv, errors.New("操作频繁,请稍后重试")) {
t.Fatal("message-only throttling text should not trigger guangyapan cooldown")
}
if driveErrorShouldCooldown(drv, errors.New("invalid access token")) {
t.Fatal("invalid access token should not trigger guangyapan cooldown")
}
}
func TestGoogleDriveMediaErrorsShouldCooldown(t *testing.T) {
drv := &previewFakeDrive{kind: "googledrive"}
for _, err := range []error{
errors.New("google drive api error: usageLimits userRateLimitExceeded"),
errors.New("ffmpeg: Server returned 403 Forbidden"),
errors.New("downloadQuotaExceeded: The download quota for this file has been exceeded"),
errors.New("sharingRateLimitExceeded"),
errors.New("http 503 service unavailable"),
} {
if !driveErrorShouldCooldown(drv, err) {
t.Fatalf("driveErrorShouldCooldown(%v) = false, want true", err)
}
}
for _, err := range []error{
errors.New("google drive api error: usageLimits userRateLimitExceeded"),
errors.New("downloadQuotaExceeded: The download quota for this file has been exceeded"),
errors.New("sharingRateLimitExceeded"),
} {
if driveErrorShouldCooldown(drv, err) {
t.Fatalf("message-only google drive error %v should not trigger cooldown", err)
}
}
if driveErrorShouldCooldown(drv, errors.New("invalid credentials")) {
t.Fatal("invalid credentials should not trigger googledrive cooldown")
}
+3 -1
View File
@@ -151,13 +151,15 @@ func (p *Proxy) ServeStream(w http.ResponseWriter, r *http.Request, driveID, fil
// 先解出最终 Location,浏览器可直接 302 到该短期地址
// - wopan:联通网盘 GetDownloadUrlV2 返回的是短期直链,OpenList 也是直接
// 将该 URL 交给客户端使用;不需要后端持续代传视频字节
// - guangyapan:光鸭 get_res_download_url 返回 signedURL / downloadUrl
// 浏览器可直接访问,不需要后端持续代传视频字节
//
// 其余网盘(如夸克等)仍走反代,因为它们的下载
// 链接通常需要随请求带上后端持有的 Cookie / Authorization / Range
// 的特殊处理,浏览器拿不到这些上下文。
func shouldRedirect(d drives.Drive) bool {
switch d.Kind() {
case "p115", "pikpak", "onedrive", "p123", "wopan":
case "p115", "pikpak", "onedrive", "p123", "wopan", "guangyapan":
return true
}
return false
+25
View File
@@ -226,6 +226,31 @@ func TestServeStreamRedirectsWopan(t *testing.T) {
}
}
func TestServeStreamRedirectsGuangYaPan(t *testing.T) {
reg := NewRegistry()
drv := &proxyFakeSimpleDrive{
kind: "guangyapan",
url: "https://cdn.guangyapan.example/video.mp4?sign=encoded",
}
reg.Set("guangyapan", drv)
p := New(reg)
req := httptest.NewRequest(http.MethodGet, "/p/stream/guangyapan/file-1", nil)
rr := httptest.NewRecorder()
p.ServeStream(rr, req, "guangyapan", "file-1")
if rr.Code != http.StatusFound {
t.Fatalf("status = %d, want %d", rr.Code, http.StatusFound)
}
if got := rr.Header().Get("Location"); got != "https://cdn.guangyapan.example/video.mp4?sign=encoded" {
t.Fatalf("Location = %q", got)
}
if drv.calls != 1 {
t.Fatalf("link calls = %d, want 1", drv.calls)
}
}
func TestServeStreamServesLocalFilePath(t *testing.T) {
path := filepath.Join(t.TempDir(), "video.mp4")
if err := os.WriteFile(path, []byte("0123456789"), 0o644); err != nil {
+6 -2
View File
@@ -206,15 +206,19 @@ func (s *Scanner) walk(ctx context.Context, dirID, dirName string, stats *Stats,
patch.ContentHash = e.Hash
existing.ContentHash = e.Hash
}
if e.Name != "" && existing.FileName == "" {
if e.Name != "" && existing.FileName != e.Name {
patch.FileName = e.Name
existing.FileName = e.Name
patch.Title = parsed.Title
patch.TitleSet = true
patch.Author = parsed.Author
patch.AuthorSet = true
}
// 已存在但轻量元数据空缺时,顺便补齐。
if existing.Category == "" && dirName != "" {
patch.Category = dirName
}
if patch.Category != "" || patch.ContentHash != "" || patch.FileName != "" {
if patch.Category != "" || patch.ContentHash != "" || patch.FileName != "" || patch.TitleSet || patch.AuthorSet {
_ = s.Catalog.UpdateVideoMeta(ctx, id, patch)
if err := ctx.Err(); err != nil {
return err
+61
View File
@@ -323,6 +323,67 @@ func TestRunDoesNotBackfillRemoteThumbnailForExistingVideo(t *testing.T) {
}
}
func TestRunSyncsRenamedExistingVideoMetadata(t *testing.T) {
ctx := context.Background()
cat, err := catalog.Open(t.TempDir() + "/catalog.db")
if err != nil {
t.Fatalf("open catalog: %v", err)
}
t.Cleanup(func() {
if err := cat.Close(); err != nil {
t.Fatalf("close catalog: %v", err)
}
})
now := time.Now()
if err := cat.UpsertVideo(ctx, &catalog.Video{
ID: "fake-drive-file-1",
DriveID: "drive",
FileID: "file-1",
FileName: "old-name - Old Author.mp4",
Title: "old-name",
Author: "Old Author",
PreviewStatus: "pending",
PublishedAt: now,
CreatedAt: now,
UpdatedAt: now,
}); err != nil {
t.Fatalf("seed video: %v", err)
}
drv := &scannerFakeDrive{
entries: []drives.Entry{{
ID: "file-1",
Name: "[4K] renamed clip.mp4",
Size: 123,
ModTime: now,
}},
}
sc := New(cat, drv, []string{".mp4"}, nil, nil)
stats, err := sc.Run(ctx, "")
if err != nil {
t.Fatalf("scan: %v", err)
}
if stats.Added != 0 {
t.Fatalf("added = %d, want existing video to be updated in place", stats.Added)
}
got, err := cat.GetVideo(ctx, "fake-drive-file-1")
if err != nil {
t.Fatalf("get video: %v", err)
}
if got.FileName != "[4K] renamed clip.mp4" {
t.Fatalf("file_name = %q, want remote name", got.FileName)
}
if got.Title != "renamed clip" {
t.Fatalf("title = %q, want parsed title from remote name", got.Title)
}
if got.Author != "" {
t.Fatalf("author = %q, want cleared author from remote name without author suffix", got.Author)
}
}
func TestRunReplacesExistingVideoTagsWithFixedFilenameTags(t *testing.T) {
ctx := context.Background()
cat, err := catalog.Open(t.TempDir() + "/catalog.db")
+54 -24
View File
@@ -1,5 +1,5 @@
// Package spider91migrate 周期性把 spider91 drive 下载到本地的视频
// 上传到一个指定的目标 drive 目录(PikPak、115、123、OneDrive、Google Drive 或联通网盘),上传成功后:
// 上传到一个指定的目标 drive 目录(PikPak、115、123、OneDrive、Google Drive、联通网盘或光鸭网盘),上传成功后:
//
// - 改写 catalog 行:drive_id / file_id / content_hash 改成目标盘的;
// 视频自身的 id 不变(仍是 spider91-<driveID>-<viewkey>),video_tags、
@@ -31,6 +31,7 @@ import (
"github.com/video-site/backend/internal/catalog"
"github.com/video-site/backend/internal/drives"
"github.com/video-site/backend/internal/drives/googledrive"
"github.com/video-site/backend/internal/drives/guangyapan"
"github.com/video-site/backend/internal/drives/onedrive"
"github.com/video-site/backend/internal/drives/p115"
"github.com/video-site/backend/internal/drives/p123"
@@ -42,7 +43,7 @@ import (
)
// uploadTarget 是 migrator 调用目标 drive 的最小接口。任何一种"接收 spider91 上传"的
// 网盘都要实现它;当前 PikPak、115、123、OneDrive、Google Drive 和联通网盘各自通过适配器满足。
// 网盘都要实现它;当前 PikPak、115、123、OneDrive、Google Drive、联通网盘和光鸭网盘各自通过适配器满足。
//
// 这一层抽象把"迁移调用方"和"具体盘的 SDK 协议"解耦:
// - PikPak 走 GCID + OSS PutObjectpikpak.UploadResult
@@ -51,6 +52,7 @@ import (
// - OneDrive 走 SHA1 + 小文件 PUT / 大文件 upload session
// - Google Drive 走 MD5 + resumable upload session
// - 联通网盘 走 SDK Upload2C,当前上游不返回内容 hash
// - 光鸭网盘 走 OSS 分片上传,当前上游不返回内容 hash
//
// 各家返回值都被归一成本地的 UploadResult,并在 catalog 改写阶段统一处理。
type uploadTarget interface {
@@ -76,7 +78,7 @@ type Spider91LocalSource interface {
// UploadResult 是 uploadTarget.UploadAndReportHash 的归一返回。
//
// FileID 目标盘上的新文件 ID;
// Hash GCIDPikPak)、MD5 HEX123 / Google Drive)或 SHA1 HEX115 / OneDrive),写入 catalog.content_hash 用于跨盘去重;联通网盘暂为空;
// Hash GCIDPikPak)、MD5 HEX123 / Google Drive)或 SHA1 HEX115 / OneDrive),写入 catalog.content_hash 用于跨盘去重;联通网盘和光鸭网盘暂为空;
// Size 实际上传字节数。
type UploadResult struct {
FileID string
@@ -99,18 +101,19 @@ const (
)
type migrationPlan struct {
source Spider91LocalSource
row *catalog.Drive
sourceKinds []string
targetDriveID string
target uploadTarget
uploadDir string
keepLatestN int
requireAssetsReady bool
legacyBackfill bool
source Spider91LocalSource
row *catalog.Drive
sourceKinds []string
targetDriveID string
target uploadTarget
uploadDir string
keepLatestN int
requireAssetsReady bool
requirePreviewReady bool
legacyBackfill bool
}
// pikpakAdapter / p115Adapter / p123Adapter / onedriveAdapter / googledriveAdapter / wopanAdapter 把具体 driver 包装成 uploadTarget。
// pikpakAdapter / p115Adapter / p123Adapter / onedriveAdapter / googledriveAdapter / wopanAdapter / guangyapanAdapter 把具体 driver 包装成 uploadTarget。
//
// 之所以不让 driver 直接实现 uploadTarget
//
@@ -243,6 +246,27 @@ func (a *wopanAdapter) Rename(ctx context.Context, fileID, newName string) error
return a.d.Rename(ctx, fileID, newName)
}
type guangyapanAdapter struct {
d *guangyapan.Driver
}
func (a *guangyapanAdapter) ID() string { return a.d.ID() }
func (a *guangyapanAdapter) Kind() string { return a.d.Kind() }
func (a *guangyapanAdapter) RootID() string { return a.d.RootID() }
func (a *guangyapanAdapter) EnsureDir(ctx context.Context, pathFromRoot string) (string, error) {
return a.d.EnsureDir(ctx, pathFromRoot)
}
func (a *guangyapanAdapter) UploadAndReportHash(ctx context.Context, parentID, name string, r io.Reader, size int64) (UploadResult, error) {
fileID, err := a.d.Upload(ctx, parentID, name, r, size)
if err != nil {
return UploadResult{}, err
}
return UploadResult{FileID: fileID, Size: size}, nil
}
func (a *guangyapanAdapter) Rename(ctx context.Context, fileID, newName string) error {
return a.d.Rename(ctx, fileID, newName)
}
// adaptUploadTarget 把通用 drive 包装成 uploadTarget。
// 不支持的盘 kind 返回 error;调用方静默跳过。
func adaptUploadTarget(d drives.Drive) (uploadTarget, error) {
@@ -259,6 +283,8 @@ func adaptUploadTarget(d drives.Drive) (uploadTarget, error) {
return &googledriveAdapter{d: v}, nil
case *wopan.Driver:
return &wopanAdapter{d: v}, nil
case *guangyapan.Driver:
return &guangyapanAdapter{d: v}, nil
case uploadTarget:
// 测试或自定义实现可以直接传入;优先使用具体类型分支以拿到适配器。
return v, nil
@@ -572,14 +598,15 @@ func (m *Migrator) migrationPlans(ctx context.Context) []migrationPlan {
continue
}
out = append(out, migrationPlan{
source: src,
row: row,
sourceKinds: crawlerSourceKindsForRow(row),
targetDriveID: resolvedID,
target: target,
uploadDir: scriptCrawlerUploadDir(row.ID),
keepLatestN: 0,
requireAssetsReady: true,
source: src,
row: row,
sourceKinds: crawlerSourceKindsForRow(row),
targetDriveID: resolvedID,
target: target,
uploadDir: scriptCrawlerUploadDir(row.ID),
keepLatestN: 0,
requireAssetsReady: true,
requirePreviewReady: row.TeaserEnabled,
})
case spider91.Kind:
if m.cfg.GetTargetDriveID == nil {
@@ -813,7 +840,7 @@ func (m *Migrator) migrateDrive(ctx context.Context, plan migrationPlan) (int, e
}
if plan.requireAssetsReady {
ready, err := m.crawlerVideoAssetsReady(ctx, v)
ready, err := m.crawlerVideoAssetsReady(ctx, v, plan.requirePreviewReady)
if err != nil {
log.Printf("[spider91migrate] %s check generated assets: %v", v.ID, err)
continue
@@ -889,7 +916,7 @@ func (m *Migrator) findVideoForLocalFile(ctx context.Context, plan migrationPlan
return nil
}
func (m *Migrator) crawlerVideoAssetsReady(ctx context.Context, v *catalog.Video) (bool, error) {
func (m *Migrator) crawlerVideoAssetsReady(ctx context.Context, v *catalog.Video, requirePreview bool) (bool, error) {
if v == nil {
return false, nil
}
@@ -897,6 +924,9 @@ func (m *Migrator) crawlerVideoAssetsReady(ctx context.Context, v *catalog.Video
if !fingerprintReady {
return false, nil
}
if !requirePreview {
return true, nil
}
if strings.EqualFold(strings.TrimSpace(v.PreviewStatus), "ready") {
return true, nil
}
@@ -1183,7 +1213,7 @@ func (m *Migrator) cleanupOldLocalVideos(ctx context.Context, plan migrationPlan
return deleted, nil
}
// backfillFileNames 扫描目标 drivePikPak、115、123、OneDrive、Google Drive 或联通网盘)下所有 spider91-* 起始 ID 的视频,
// backfillFileNames 扫描目标 drivePikPak、115、123、OneDrive、Google Drive、联通网盘或光鸭网盘)下所有 spider91-* 起始 ID 的视频,
// 对文件名不是 desiredPikPakName(...) 期望格式的,调 target.Rename 修正,
// 并把 catalog.file_name 同步到新名字。
//
@@ -365,11 +365,19 @@ func seedScriptCrawlerDrive(t *testing.T, cat *catalog.Catalog, d *scriptcrawler
"script_path": "/tmp/crawler.py",
"upload_drive_id": uploadDriveID,
},
TeaserEnabled: true,
}); err != nil {
t.Fatalf("seed scriptcrawler drive: %v", err)
}
}
func setScriptCrawlerTeaserEnabled(t *testing.T, cat *catalog.Catalog, driveID string, enabled bool) {
t.Helper()
if err := cat.SetDriveTeaserEnabled(context.Background(), driveID, enabled); err != nil {
t.Fatalf("set scriptcrawler teaser enabled: %v", err)
}
}
func writeScriptCrawlerVideo(t *testing.T, cat *catalog.Catalog, d *scriptcrawler.Driver, sourceID, ext string, content []byte, readyAssets bool) string {
t.Helper()
fileID := sourceID + ext
@@ -587,6 +595,47 @@ func TestRunOnceSkipsScriptCrawlerVideoUntilPreviewAndFingerprintReady(t *testin
}
}
func TestRunOnceMigratesScriptCrawlerVideoWithoutPreviewWhenTeaserDisabled(t *testing.T) {
cat := setupCatalog(t)
src := setupScriptCrawler(t, "crawler-no-preview")
pp := newFakePikPak("pikpak-target", "pikpak-root-id")
seedScriptCrawlerDrive(t, cat, src, pp.ID())
setScriptCrawlerTeaserEnabled(t, cat, src.ID(), false)
reg := newFakeRegistry()
reg.Add(src)
reg.Add(pp)
id := writeScriptCrawlerVideo(t, cat, src, "fingerprint-ready", ".mp4", []byte("script video bytes"), false)
if err := cat.UpdateVideoFingerprint(context.Background(), id, "sampled-fingerprint-ready", "ready", ""); err != nil {
t.Fatalf("mark fingerprint ready: %v", err)
}
if err := cat.UpdatePreview(context.Background(), id, "", "disabled"); err != nil {
t.Fatalf("mark preview disabled: %v", err)
}
m := New(Config{Catalog: cat, Registry: reg})
m.runOnce(context.Background())
if pp.uploadCalls != 1 {
t.Fatalf("upload calls = %d, want 1 when preview generation is disabled", pp.uploadCalls)
}
got, err := cat.GetVideo(context.Background(), id)
if err != nil {
t.Fatalf("get migrated video: %v", err)
}
if got.DriveID != pp.ID() {
t.Fatalf("drive_id = %q, want %q", got.DriveID, pp.ID())
}
if got.PreviewStatus != "disabled" || got.FingerprintStatus != "ready" || got.SampledSHA256 == "" {
t.Fatalf("asset status after migration = preview %q fingerprint %q sampled %q, want disabled/ready/non-empty", got.PreviewStatus, got.FingerprintStatus, got.SampledSHA256)
}
videoPath, _ := src.VideoPath("fingerprint-ready.mp4")
if _, err := os.Stat(videoPath); !os.IsNotExist(err) {
t.Fatalf("local scriptcrawler video still exists or stat error %v", err)
}
}
func TestRunOnceBindsScriptCrawlerDuplicateToExistingTargetWithoutUpload(t *testing.T) {
cat := setupCatalog(t)
src := setupScriptCrawler(t, "crawler-duplicate")
@@ -1464,7 +1513,7 @@ func TestAdaptUploadTargetSupportsWopanDriver(t *testing.T) {
}
}
// TestResolveTargetRejectsUnsupportedKind 验证当目标 drive 既不是 PikPak、115、123、OneDrive、Google Drive 也不是联通网盘时,
// TestResolveTargetRejectsUnsupportedKind 验证当目标 drive 既不是 PikPak、115、123、OneDrive、Google Drive、联通网盘也不是光鸭网盘时,
// resolveTarget 拒绝并返回 error,让 runOnce 静默跳过(不会做破坏性变更)。
func TestResolveTargetRejectsUnsupportedKind(t *testing.T) {
cat := setupCatalog(t)
+178
View File
@@ -0,0 +1,178 @@
// Package transcode 实现"浏览器兼容性转码":把网盘/本地存储中浏览器
// <video> 播不动的视频(AVI/WMV/FLV、MPEG-4 Part 2、RMVB 等)转成
// H.264 + AAC 的 MP4,并把产物上传回同一存储,播放源切到产物文件。
//
// 与封面/预览生成不同,转码不会自动运行——只能由管理员在网盘管理页
// 手动开启,也可以随时手动停止。
package transcode
import (
"context"
"encoding/json"
"fmt"
"os/exec"
"strings"
"time"
)
// MediaInfo 是 ffprobe 探测出来的、做兼容性判定所需的最小信息。
type MediaInfo struct {
// FormatName 是 ffprobe 的 format_name,逗号分隔的 demuxer 别名,
// 例如 "mov,mp4,m4a,3gp,3g2,mj2" / "avi" / "matroska,webm"。
FormatName string
VideoCodecs []string
AudioCodecs []string
}
// browserCompatibleVideoCodecs 是主流浏览器 <video> 普遍可解码的视频编码。
// HEVC/H.265 只有部分平台支持,保守起见不算兼容。
var browserCompatibleVideoCodecs = map[string]bool{
"h264": true,
"vp8": true,
"vp9": true,
"av1": true,
}
// browserCompatibleAudioCodecs 是主流浏览器普遍可解码的音频编码。
var browserCompatibleAudioCodecs = map[string]bool{
"aac": true,
"mp3": true,
"opus": true,
"vorbis": true,
"flac": true,
}
// NeedsTranscode 判断这个文件是否需要转码才能在浏览器里播放。
// ext 是 catalog 里记录的扩展名(小写、不带点),用来区分 mkv 和 webm
// (两者的 format_name 都是 "matroska,webm")。
func NeedsTranscode(info MediaInfo, ext string) bool {
if !containerCompatible(info.FormatName, ext) {
return true
}
for _, codec := range info.VideoCodecs {
if !browserCompatibleVideoCodecs[strings.ToLower(codec)] {
return true
}
}
for _, codec := range info.AudioCodecs {
if !browserCompatibleAudioCodecs[strings.ToLower(codec)] {
return true
}
}
return false
}
func containerCompatible(formatName, ext string) bool {
format := strings.ToLower(formatName)
for _, name := range strings.Split(format, ",") {
if name == "mp4" {
return true
}
}
// matroska,webm:只有真 .webm 信任为浏览器可播容器;.mkv 保守转码。
if strings.Contains(format, "webm") && strings.EqualFold(ext, "webm") {
return true
}
return false
}
// ProbeFile 用 ffprobe 探测本地文件的容器与音视频编码。
func ProbeFile(ctx context.Context, ffprobePath, path string) (MediaInfo, error) {
ctx2, cancel := context.WithTimeout(ctx, 60*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx2, ffprobePath,
"-v", "error",
"-show_entries", "format=format_name",
"-show_entries", "stream=codec_type,codec_name",
"-of", "json",
path,
)
out, err := cmd.Output()
if err != nil {
return MediaInfo{}, fmt.Errorf("transcode: ffprobe: %w", err)
}
var parsed struct {
Format struct {
FormatName string `json:"format_name"`
} `json:"format"`
Streams []struct {
CodecType string `json:"codec_type"`
CodecName string `json:"codec_name"`
} `json:"streams"`
}
if err := json.Unmarshal(out, &parsed); err != nil {
return MediaInfo{}, fmt.Errorf("transcode: parse ffprobe output: %w", err)
}
info := MediaInfo{FormatName: parsed.Format.FormatName}
for _, s := range parsed.Streams {
switch s.CodecType {
case "video":
info.VideoCodecs = append(info.VideoCodecs, s.CodecName)
case "audio":
info.AudioCodecs = append(info.AudioCodecs, s.CodecName)
}
}
return info, nil
}
// buildFFmpegArgs 按探测结果生成转码参数:
// - 编码本就兼容、只是容器不行(如 AVI 里装 H.264)→ 流拷贝 remux,零质量损失;
// - 否则视频转 H.264(裁到偶数尺寸 + yuv420p 保证兼容性)、音频转 AAC。
//
// 两种情况都加 +faststart 把 moov 提前,便于边下边播。
func buildFFmpegArgs(info MediaInfo, inPath, outPath string) []string {
args := []string{"-y", "-i", inPath}
videoOK := true
for _, codec := range info.VideoCodecs {
if !browserCompatibleVideoCodecs[strings.ToLower(codec)] {
videoOK = false
break
}
}
audioOK := true
for _, codec := range info.AudioCodecs {
if !browserCompatibleAudioCodecs[strings.ToLower(codec)] {
audioOK = false
break
}
}
if videoOK {
args = append(args, "-c:v", "copy")
} else {
args = append(args,
"-c:v", "libx264",
"-preset", "veryfast",
"-crf", "23",
"-vf", "scale=trunc(iw/2)*2:trunc(ih/2)*2",
"-pix_fmt", "yuv420p",
)
}
if len(info.AudioCodecs) == 0 {
args = append(args, "-an")
} else if audioOK {
args = append(args, "-c:a", "copy")
} else {
args = append(args, "-c:a", "aac", "-b:a", "128k")
}
args = append(args, "-movflags", "+faststart", "-f", "mp4", outPath)
return args
}
// TranscodeFile 把本地输入文件转成浏览器可播的 MP4 写到 outPath。
func TranscodeFile(ctx context.Context, ffmpegPath string, info MediaInfo, inPath, outPath string) error {
args := buildFFmpegArgs(info, inPath, outPath)
cmd := exec.CommandContext(ctx, ffmpegPath, args...)
out, err := cmd.CombinedOutput()
if err != nil {
return fmt.Errorf("transcode: ffmpeg: %w: %s", err, tailOf(string(out), 400))
}
return nil
}
func tailOf(s string, n int) string {
s = strings.TrimSpace(s)
if len(s) <= n {
return s
}
return s[len(s)-n:]
}
@@ -0,0 +1,125 @@
package transcode
import (
"strings"
"testing"
"github.com/video-site/backend/internal/catalog"
)
func TestNeedsTranscode(t *testing.T) {
cases := []struct {
name string
info MediaInfo
ext string
want bool
}{
{
name: "h264 aac mp4 is compatible",
info: MediaInfo{FormatName: "mov,mp4,m4a,3gp,3g2,mj2", VideoCodecs: []string{"h264"}, AudioCodecs: []string{"aac"}},
ext: "mp4",
want: false,
},
{
name: "mpeg4 in avi needs transcode",
info: MediaInfo{FormatName: "avi", VideoCodecs: []string{"mpeg4"}, AudioCodecs: []string{"mp3"}},
ext: "avi",
want: true,
},
{
name: "h264 in avi needs remux",
info: MediaInfo{FormatName: "avi", VideoCodecs: []string{"h264"}, AudioCodecs: []string{"aac"}},
ext: "avi",
want: true,
},
{
name: "hevc in mp4 needs transcode",
info: MediaInfo{FormatName: "mov,mp4,m4a,3gp,3g2,mj2", VideoCodecs: []string{"hevc"}, AudioCodecs: []string{"aac"}},
ext: "mp4",
want: true,
},
{
name: "vp9 opus webm is compatible",
info: MediaInfo{FormatName: "matroska,webm", VideoCodecs: []string{"vp9"}, AudioCodecs: []string{"opus"}},
ext: "webm",
want: false,
},
{
name: "h264 in mkv is conservative transcode",
info: MediaInfo{FormatName: "matroska,webm", VideoCodecs: []string{"h264"}, AudioCodecs: []string{"aac"}},
ext: "mkv",
want: true,
},
{
name: "pcm audio in mov needs transcode",
info: MediaInfo{FormatName: "mov,mp4,m4a,3gp,3g2,mj2", VideoCodecs: []string{"h264"}, AudioCodecs: []string{"pcm_s16le"}},
ext: "mov",
want: true,
},
{
name: "video only h264 mp4 is compatible",
info: MediaInfo{FormatName: "mov,mp4,m4a,3gp,3g2,mj2", VideoCodecs: []string{"h264"}},
ext: "mp4",
want: false,
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
if got := NeedsTranscode(tc.info, tc.ext); got != tc.want {
t.Fatalf("NeedsTranscode(%+v, %q) = %v, want %v", tc.info, tc.ext, got, tc.want)
}
})
}
}
func TestBuildFFmpegArgsRemuxWhenCodecsCompatible(t *testing.T) {
// AVI 里装 H.264+AAC:只需要换容器,应该走流拷贝
info := MediaInfo{FormatName: "avi", VideoCodecs: []string{"h264"}, AudioCodecs: []string{"aac"}}
args := strings.Join(buildFFmpegArgs(info, "in.avi", "out.mp4"), " ")
if !strings.Contains(args, "-c:v copy") {
t.Fatalf("expected video stream copy, got: %s", args)
}
if !strings.Contains(args, "-c:a copy") {
t.Fatalf("expected audio stream copy, got: %s", args)
}
if !strings.Contains(args, "+faststart") {
t.Fatalf("expected faststart flag, got: %s", args)
}
}
func TestBuildFFmpegArgsTranscodesIncompatibleCodecs(t *testing.T) {
info := MediaInfo{FormatName: "avi", VideoCodecs: []string{"mpeg4"}, AudioCodecs: []string{"wmav2"}}
args := strings.Join(buildFFmpegArgs(info, "in.avi", "out.mp4"), " ")
if !strings.Contains(args, "-c:v libx264") {
t.Fatalf("expected libx264 video encode, got: %s", args)
}
if !strings.Contains(args, "-c:a aac") {
t.Fatalf("expected aac audio encode, got: %s", args)
}
if !strings.Contains(args, "yuv420p") {
t.Fatalf("expected yuv420p pixel format, got: %s", args)
}
}
func TestBuildFFmpegArgsDropsAudioWhenNoAudioStream(t *testing.T) {
info := MediaInfo{FormatName: "avi", VideoCodecs: []string{"mpeg4"}}
args := strings.Join(buildFFmpegArgs(info, "in.avi", "out.mp4"), " ")
if !strings.Contains(args, "-an") {
t.Fatalf("expected -an for video without audio, got: %s", args)
}
}
func TestTranscodedName(t *testing.T) {
for _, tc := range []struct {
fileName, title, id, want string
}{
{"www.98T.la@167.avi", "www.98T.la@167", "p115-1", "www.98T.la@167.mp4"},
{"", "标题", "p115-2", "标题.mp4"},
{"a/b\\c.wmv", "", "p115-3", "a_b_c.mp4"},
} {
v := &catalog.Video{FileName: tc.fileName, Title: tc.title, ID: tc.id}
if got := transcodedName(v); got != tc.want {
t.Fatalf("transcodedName(%q,%q,%q) = %q, want %q", tc.fileName, tc.title, tc.id, got, tc.want)
}
}
}
+308
View File
@@ -0,0 +1,308 @@
package transcode
import (
"context"
"fmt"
"io"
"log"
"net/http"
"net/url"
"os"
"path/filepath"
"strings"
"sync"
"time"
"github.com/video-site/backend/internal/catalog"
"github.com/video-site/backend/internal/drives"
)
// DefaultTargetDirName 是转码产物在网盘上的存放目录(相对根目录)。
// worker 第一次上传前会 EnsureDir 并把该目录加进 drive 的扫描跳过列表,
// 避免 scanner 把转码产物当成新视频重复入库。
const DefaultTargetDirName = "91转码"
type Config struct {
FFmpegPath string
FFprobePath string
// WorkDir 是下载原始文件 / 写转码产物的本地临时目录。
WorkDir string
// TargetDirName 为空时用 DefaultTargetDirName。
TargetDirName string
}
// TaskStatus 与 preview/fingerprint worker 的状态结构对齐,供 admin 展示。
type TaskStatus struct {
State string
CurrentTitle string
QueueLength int
DoneCount int
TotalCount int
}
// Worker 串行处理一个 drive 的转码任务。生命周期与一次"开始转码"对应:
// Run 处理完整个候选列表(或 ctx 被取消)后即结束,不常驻。
type Worker struct {
cfg Config
cat *catalog.Catalog
drv drives.Drive
hc *http.Client
mu sync.Mutex
state string
currentTitle string
done int
total int
targetDirOnce sync.Once
targetDirID string
targetDirErr error
}
func NewWorker(cfg Config, cat *catalog.Catalog, drv drives.Drive) *Worker {
if cfg.FFmpegPath == "" {
cfg.FFmpegPath = "ffmpeg"
}
if cfg.FFprobePath == "" {
cfg.FFprobePath = "ffprobe"
}
if cfg.TargetDirName == "" {
cfg.TargetDirName = DefaultTargetDirName
}
if cfg.WorkDir == "" {
cfg.WorkDir = os.TempDir()
}
return &Worker{
cfg: cfg,
cat: cat,
drv: drv,
hc: &http.Client{Timeout: 0},
state: "idle",
}
}
func (w *Worker) Status() TaskStatus {
w.mu.Lock()
defer w.mu.Unlock()
queueLen := w.total - w.done
if w.state == "generating" && queueLen > 0 {
// 正在处理的那条不算"排队中"
queueLen--
}
if queueLen < 0 {
queueLen = 0
}
return TaskStatus{
State: w.state,
CurrentTitle: w.currentTitle,
QueueLength: queueLen,
DoneCount: w.done,
TotalCount: w.total,
}
}
// Run 串行转码整个候选列表。ctx 取消时停在当前条目边界(正在跑的 ffmpeg
// 会被 CommandContext 杀掉),未处理的候选保持原状态,下次开始时继续。
func (w *Worker) Run(ctx context.Context, videos []*catalog.Video) {
w.mu.Lock()
w.state = "generating"
w.total = len(videos)
w.done = 0
w.mu.Unlock()
defer func() {
w.mu.Lock()
w.state = "idle"
w.currentTitle = ""
w.mu.Unlock()
}()
for _, v := range videos {
if ctx.Err() != nil {
log.Printf("[transcode] drive=%s canceled after %d/%d", w.drv.ID(), w.doneCount(), len(videos))
return
}
w.mu.Lock()
w.currentTitle = v.Title
w.mu.Unlock()
if err := w.process(ctx, v); err != nil {
if ctx.Err() != nil {
// 取消导致的失败不要写 failed,保持候选状态便于下次继续
log.Printf("[transcode] drive=%s canceled while processing %s", w.drv.ID(), v.ID)
return
}
log.Printf("[transcode] drive=%s video=%s failed: %v", w.drv.ID(), v.ID, err)
if uerr := w.cat.UpdateVideoTranscode(context.WithoutCancel(ctx), v.ID, "failed", err.Error(), "", 0); uerr != nil {
log.Printf("[transcode] mark failed %s: %v", v.ID, uerr)
}
}
w.mu.Lock()
w.done++
w.mu.Unlock()
}
log.Printf("[transcode] drive=%s finished %d videos", w.drv.ID(), len(videos))
}
func (w *Worker) doneCount() int {
w.mu.Lock()
defer w.mu.Unlock()
return w.done
}
func (w *Worker) process(ctx context.Context, v *catalog.Video) error {
localPath, cleanup, err := w.fetchSource(ctx, v)
if err != nil {
return fmt.Errorf("fetch source: %w", err)
}
defer cleanup()
info, err := ProbeFile(ctx, w.cfg.FFprobePath, localPath)
if err != nil {
return err
}
if !NeedsTranscode(info, v.Ext) {
log.Printf("[transcode] drive=%s video=%s compatible (%s), skip", w.drv.ID(), v.ID, info.FormatName)
return w.cat.UpdateVideoTranscode(ctx, v.ID, "skipped", "", "", 0)
}
outPath := filepath.Join(w.cfg.WorkDir, sanitizeFileName(v.ID)+".transcoding.mp4")
defer os.Remove(outPath)
if err := TranscodeFile(ctx, w.cfg.FFmpegPath, info, localPath, outPath); err != nil {
return err
}
stat, err := os.Stat(outPath)
if err != nil {
return fmt.Errorf("stat transcoded output: %w", err)
}
dirID, err := w.ensureTargetDir(ctx)
if err != nil {
return fmt.Errorf("ensure target dir: %w", err)
}
f, err := os.Open(outPath)
if err != nil {
return err
}
defer f.Close()
fileID, err := w.drv.Upload(ctx, dirID, transcodedName(v), f, stat.Size())
if err != nil {
return fmt.Errorf("upload transcoded file: %w", err)
}
log.Printf("[transcode] drive=%s video=%s ready: file=%s size=%d", w.drv.ID(), v.ID, fileID, stat.Size())
return w.cat.UpdateVideoTranscode(ctx, v.ID, "ready", "", fileID, stat.Size())
}
// fetchSource 把原始文件准备成本地路径。本地存储直接复用源路径(cleanup
// 不删除源文件);云盘则整文件下载到 WorkDir。
func (w *Worker) fetchSource(ctx context.Context, v *catalog.Video) (string, func(), error) {
link, err := w.drv.StreamURL(ctx, v.FileID)
if err != nil {
return "", nil, err
}
u, err := url.Parse(link.URL)
if isLocal := err == nil && u.Scheme != "http" && u.Scheme != "https"; isLocal {
path := link.URL
if err == nil && u.Scheme == "file" {
path = u.Path
}
return path, func() {}, nil
}
tmpPath := filepath.Join(w.cfg.WorkDir, sanitizeFileName(v.ID)+".src.tmp")
cleanup := func() { os.Remove(tmpPath) }
if err := w.downloadTo(ctx, link, tmpPath); err != nil {
cleanup()
return "", nil, err
}
return tmpPath, cleanup, nil
}
func (w *Worker) downloadTo(ctx context.Context, link *drives.StreamLink, dst string) error {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, link.URL, nil)
if err != nil {
return err
}
for k, vals := range link.Headers {
for _, val := range vals {
req.Header.Add(k, val)
}
}
res, err := w.hc.Do(req)
if err != nil {
return err
}
defer res.Body.Close()
if res.StatusCode < 200 || res.StatusCode >= 300 {
return fmt.Errorf("download source: HTTP %d", res.StatusCode)
}
f, err := os.Create(dst)
if err != nil {
return err
}
defer f.Close()
if _, err := io.Copy(f, res.Body); err != nil {
return fmt.Errorf("download source: %w", err)
}
return f.Sync()
}
// ensureTargetDir 确保网盘上的转码产物目录存在,并把它写进 drive 的扫描
// 跳过列表(幂等),避免 scanner 把产物再当新视频收进库。
func (w *Worker) ensureTargetDir(ctx context.Context) (string, error) {
w.targetDirOnce.Do(func() {
dirID, err := w.drv.EnsureDir(ctx, w.cfg.TargetDirName)
if err != nil {
w.targetDirErr = err
return
}
w.targetDirID = dirID
if err := w.addDirToSkipList(ctx, dirID); err != nil {
// 跳过列表更新失败不阻塞转码,只记日志(最坏情况是 scanner
// 之后把产物扫成新视频,可手动加跳过目录修复)。
log.Printf("[transcode] drive=%s add skip dir %s: %v", w.drv.ID(), dirID, err)
}
})
return w.targetDirID, w.targetDirErr
}
func (w *Worker) addDirToSkipList(ctx context.Context, dirID string) error {
d, err := w.cat.GetDrive(ctx, w.drv.ID())
if err != nil {
return err
}
for _, existing := range d.SkipDirIDs {
if existing == dirID {
return nil
}
}
return w.cat.SetDriveSkipDirIDs(ctx, w.drv.ID(), append(d.SkipDirIDs, dirID))
}
// transcodedName 生成产物文件名:原文件名去掉扩展名 + .mp4。
func transcodedName(v *catalog.Video) string {
base := strings.TrimSpace(v.FileName)
if base == "" {
base = v.Title
}
if base == "" {
base = v.ID
}
if ext := filepath.Ext(base); ext != "" {
base = strings.TrimSuffix(base, ext)
}
return sanitizeFileName(base) + ".mp4"
}
// sanitizeFileName 把路径分隔符等危险字符替换掉,避免拼出意外路径。
func sanitizeFileName(name string) string {
replacer := strings.NewReplacer(
"/", "_", "\\", "_", ":", "_", "*", "_", "?", "_",
"\"", "_", "<", "_", ">", "_", "|", "_", "\x00", "_",
)
out := strings.TrimSpace(replacer.Replace(name))
if out == "" {
out = fmt.Sprintf("transcoded-%d", time.Now().UnixMilli())
}
return out
}
+283
View File
@@ -0,0 +1,283 @@
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
光鸭网盘 - 扫码登录脚本
========================
1. 调用 API 获取登录二维码
2. 保存二维码图片等待用户扫描
3. 扫描成功后保存用户凭证信息
"""
import io
import sys
# 修复 Windows 终端 GBK 编码问题
if sys.platform == 'win32':
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8', errors='replace')
sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding='utf-8', errors='replace')
import requests
import json
import time
import os
import sys
from datetime import datetime
# ========== 配置 ==========
API_ORIGIN = "https://account.guangyapan.com"
CLIENT_ID = "aMe-8VSlkrbQXpUR"
SCOPE = "user"
QR_IMAGE_PATH = "login_qr.png"
CREDENTIALS_PATH = "credentials.json"
# ========== 可选依赖 ==========
try:
import qrcode
HAS_QRCODE = True
except ImportError:
HAS_QRCODE = False
try:
from PIL import Image
HAS_PIL = True
except ImportError:
HAS_PIL = False
def generate_qr_image(url: str, path: str):
"""生成二维码图片"""
if HAS_QRCODE:
qr = qrcode.QRCode(
version=1,
error_correction=qrcode.constants.ERROR_CORRECT_M,
box_size=10,
border=4,
)
qr.add_data(url)
qr.make(fit=True)
img = qr.make_image(fill_color="black", back_color="white")
img.save(path)
print(f"[✓] 二维码已保存到: {path}")
else:
# Fallback: 使用 qrencode 命令行工具
import subprocess
try:
subprocess.run(["qrencode", "-o", path, url], check=True)
print(f"[✓] 二维码已保存到: {path}")
except FileNotFoundError:
print("[✗] 需要安装 qrcode 库: pip install qrcode[pil]")
print(f"[!] 请手动访问以下链接扫码:")
print(f" {url}")
return
# 尝试直接显示二维码到终端
try:
if HAS_PIL:
img = Image.open(path)
img.show()
print("[✓] 二维码已在图片查看器中打开")
except Exception:
pass
# 终端内显示小二维码
if HAS_QRCODE:
try:
qr.print_ascii(invert=True)
except Exception:
pass
def main():
session = requests.Session()
session.headers.update({
"User-Agent": "GuangYaPan-Login/1.0",
"Accept": "application/json",
"Content-Type": "application/json",
})
# ====== Step 1: 获取设备码和二维码链接 ======
print("=" * 60)
print("Step 1: 请求登录二维码...")
print("=" * 60)
device_code_url = f"{API_ORIGIN}/v1/auth/device/code"
device_payload = {
"client_id": CLIENT_ID,
"scope": SCOPE,
}
try:
resp = session.post(device_code_url, json=device_payload, timeout=30)
resp.raise_for_status()
device_data = resp.json()
except requests.exceptions.RequestException as e:
print(f"[✗] 请求失败: {e}")
if hasattr(e, 'response') and e.response is not None:
print(f" 响应内容: {e.response.text[:500]}")
sys.exit(1)
print(f"[✓] 设备码获取成功")
print(f" device_code: {device_data.get('device_code', 'N/A')[:30]}...")
print(f" interval: {device_data.get('interval', 5)}")
print(f" expires_in: {device_data.get('expires_in', 'N/A')}")
device_code = device_data["device_code"]
interval = int(device_data.get("interval", 5))
expires_in = int(device_data.get("expires_in", 300))
# 二维码链接
qr_url = device_data.get("verification_uri_complete") or device_data.get("short_uri_complete")
if not qr_url:
print("[✗] 响应中没有找到二维码链接")
print(f" 完整响应: {json.dumps(device_data, indent=2, ensure_ascii=False)}")
sys.exit(1)
print(f" qr_url: {qr_url}")
print()
# ====== Step 2: 生成并保存二维码 ======
print("=" * 60)
print("Step 2: 生成二维码图片...")
print("=" * 60)
generate_qr_image(qr_url, QR_IMAGE_PATH)
print()
print("!" * 60)
print("! 请使用「光鸭APP」扫描二维码登录")
print("!" * 60)
print()
# ====== Step 3: 轮询等待用户扫描 ======
print("=" * 60)
print("Step 3: 等待扫码授权...")
print("=" * 60)
token_url = f"{API_ORIGIN}/v1/auth/token"
token_payload = {
"client_id": CLIENT_ID,
"grant_type": "urn:ietf:params:oauth:grant-type:device_code",
"device_code": device_code,
}
start_time = time.time()
attempt = 0
while True:
attempt += 1
elapsed = time.time() - start_time
# 检查是否超时
if elapsed > expires_in:
print(f"\n[✗] 二维码已过期({expires_in}秒),请重新运行脚本")
sys.exit(1)
time.sleep(interval)
try:
resp = session.post(token_url, json=token_payload, timeout=30)
token_data = resp.json()
except requests.exceptions.RequestException as e:
print(f"\n[!] 网络错误: {e},重试中...")
continue
if "error" in token_data:
error = token_data["error"]
if error in ("authorization_pending", "slow_down"):
# 用户还未扫描或确认
dots = "." * ((attempt % 10) + 1)
print(f"\r 等待中{dots:<10} ({int(elapsed)}s / {expires_in}s)", end="", flush=True)
if error == "slow_down":
interval = min(interval * 2, 60)
continue
elif error == "expired_token":
print(f"\n[✗] 二维码已过期,请重新运行脚本")
sys.exit(1)
elif error == "access_denied":
print(f"\n[✗] 用户拒绝了授权")
sys.exit(1)
else:
print(f"\n[✗] 未知错误: {error}")
print(f" 完整响应: {json.dumps(token_data, indent=2, ensure_ascii=False)}")
sys.exit(1)
else:
# 成功!
print(f"\n[✓] 扫码授权成功!({int(elapsed)}s)")
break
# ====== Step 4: 保存凭证 ======
print()
print("=" * 60)
print("Step 4: 保存用户凭证...")
print("=" * 60)
# 保存完整 token 响应
credentials = {
"saved_at": datetime.now().isoformat(),
"api_origin": API_ORIGIN,
"client_id": CLIENT_ID,
"token_response": token_data,
"cookies": dict(session.cookies),
}
with open(CREDENTIALS_PATH, "w", encoding="utf-8") as f:
json.dump(credentials, f, indent=2, ensure_ascii=False)
print(f"[✓] 完整凭证已保存到: {CREDENTIALS_PATH}")
# 提取关键信息
access_token = token_data.get("access_token", "")
refresh_token = token_data.get("refresh_token", "")
id_token = token_data.get("id_token", "")
token_type = token_data.get("token_type", "Bearer")
expires_in = token_data.get("expires_in", 0)
print()
print("-" * 60)
print("凭证摘要:")
print("-" * 60)
print(f" access_token: {access_token[:50]}..." if access_token else " access_token: (无)")
print(f" refresh_token: {refresh_token[:50]}..." if refresh_token else " refresh_token: (无)")
print(f" id_token: {id_token[:50]}..." if id_token else " id_token: (无)")
print(f" token_type: {token_type}")
print(f" expires_in: {expires_in}")
print(f" scope: {token_data.get('scope', SCOPE)}")
print("-" * 60)
# 尝试获取用户信息
print()
print("=" * 60)
print("Step 5: 获取用户信息...")
print("=" * 60)
user_info_url = f"{API_ORIGIN}/v1/user/me"
try:
user_headers = {
"Authorization": f"{token_type} {access_token}",
}
user_resp = requests.get(user_info_url, headers=user_headers, timeout=15)
if user_resp.status_code == 200:
user_data = user_resp.json()
print("[✓] 用户信息获取成功:")
print(json.dumps(user_data, indent=2, ensure_ascii=False))
# 追加用户信息到凭证文件
credentials["user_info"] = user_data
with open(CREDENTIALS_PATH, "w", encoding="utf-8") as f:
json.dump(credentials, f, indent=2, ensure_ascii=False)
else:
print(f"[!] 获取用户信息返回 {user_resp.status_code}: {user_resp.text[:200]}")
except Exception as e:
print(f"[!] 获取用户信息失败: {e}")
print()
print("=" * 60)
print("完成!凭证文件: " + CREDENTIALS_PATH)
print("=" * 60)
if __name__ == "__main__":
main()
+9 -2
View File
@@ -3,8 +3,15 @@
<head>
<meta charset="UTF-8" />
<meta name="referrer" content="no-referrer" />
<link rel="icon" type="image/svg+xml" href="/favicon.svg" />
<link rel="icon" type="image/png" href="/icon.png" />
<link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png" />
<link rel="manifest" href="/manifest.webmanifest" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, viewport-fit=cover" />
<meta name="mobile-web-app-capable" content="yes" />
<meta name="apple-mobile-web-app-capable" content="yes" />
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
<meta name="apple-mobile-web-app-title" content="91" />
<meta name="theme-color" content="#000000" />
<meta name="description" content="91 视频站" />
<title>91</title>
<!-- Premium Fonts Preconnect & Links -->
@@ -19,7 +26,7 @@
(function () {
try {
var t = localStorage.getItem("video-site:theme");
if (t === "pink" || t === "dark") {
if (t === "pink" || t === "dark" || t === "sky") {
document.documentElement.setAttribute("data-theme", t);
} else {
document.documentElement.setAttribute("data-theme", "dark");
+2 -7
View File
@@ -194,7 +194,7 @@ backup_install_files() {
local backup="$1"
mkdir -p "$backup"
cp -a "$INSTALL_PATH/server" "$backup/server"
for item in dist config.example.yaml 91VideoSpider config.yaml .version; do
for item in dist config.example.yaml config.yaml .version; do
if [[ -e "$INSTALL_PATH/$item" ]]; then
cp -a "$INSTALL_PATH/$item" "$backup/$item"
fi
@@ -205,7 +205,7 @@ restore_install_files() {
local backup="$1"
mkdir -p "$INSTALL_PATH"
cp -a "$backup/server" "$INSTALL_PATH/server"
for item in dist config.example.yaml 91VideoSpider config.yaml .version; do
for item in dist config.example.yaml config.yaml .version; do
rm -rf "${INSTALL_PATH:?}/$item"
if [[ -e "$backup/$item" ]]; then
cp -a "$backup/$item" "$INSTALL_PATH/$item"
@@ -441,7 +441,6 @@ process_looks_like_app() {
[[ "$cmd" == *"VIDEO_FRONTEND_DIR=$INSTALL_PATH/dist"* ]] && return 0
[[ "$cmd" == *"VIDEO_CONFIG=$INSTALL_PATH/config.yaml"* ]] && return 0
[[ "$cmd" == *"video-site-91"* ]] && return 0
[[ "$cmd" == *"91VideoSpider"* ]] && return 0
return 1
}
@@ -595,10 +594,6 @@ fetch_and_unpack() {
rm -rf "$INSTALL_PATH/dist"
cp -R "$root/dist" "$INSTALL_PATH/dist"
cp "$root/config.example.yaml" "$INSTALL_PATH/config.example.yaml"
if [[ -d "$root/91VideoSpider" ]]; then
rm -rf "$INSTALL_PATH/91VideoSpider"
cp -R "$root/91VideoSpider" "$INSTALL_PATH/91VideoSpider"
fi
chmod +x "$INSTALL_PATH/server"
rm -rf "$tmp"
}
+2 -2
View File
@@ -1,12 +1,12 @@
{
"name": "video-site",
"version": "0.1.6",
"version": "0.2.1",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "video-site",
"version": "0.1.6",
"version": "0.2.1",
"license": "MIT",
"dependencies": {
"artplayer": "^5.4.0",
+1 -1
View File
@@ -2,7 +2,7 @@
"name": "video-site",
"private": true,
"license": "MIT",
"version": "0.1.6",
"version": "0.2.1",
"type": "module",
"scripts": {
"dev": "vite",
Binary file not shown.

After

Width:  |  Height:  |  Size: 21 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 136 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 114 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 19 KiB

-28
View File
@@ -1,28 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32">
<defs>
<!-- Background Gradient: Warm Orange to Sakura Pink, representing both themes -->
<linearGradient id="bg-grad" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" stop-color="#FF7E40" />
<stop offset="100%" stop-color="#FF4B91" />
</linearGradient>
<!-- Subtle drop shadow for the play button to give it depth -->
<filter id="shadow" x="-20%" y="-20%" width="140%" height="140%">
<feDropShadow dx="0" dy="1.5" stdDeviation="1" flood-opacity="0.25" />
</filter>
</defs>
<!-- Main Squircle Background -->
<rect x="2" y="2" width="28" height="28" rx="8" fill="url(#bg-grad)" />
<!-- Inner border for a premium, glassmorphic feel -->
<rect x="3" y="3" width="26" height="26" rx="7" fill="none" stroke="#ffffff" stroke-width="1" opacity="0.2" />
<!-- Stylized Play Button Icon, perfectly centered with rounded corners and drop shadow -->
<path d="M13 10.5 L21.5 16 L13 21.5 Z"
fill="#ffffff"
stroke="#ffffff"
stroke-width="2.5"
stroke-linejoin="round"
filter="url(#shadow)" />
</svg>

Before

Width:  |  Height:  |  Size: 1.1 KiB

BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 212 KiB

+30
View File
@@ -0,0 +1,30 @@
{
"name": "91",
"short_name": "91",
"start_url": "/",
"scope": "/",
"display": "standalone",
"display_override": ["fullscreen", "standalone"],
"background_color": "#000000",
"theme_color": "#000000",
"icons": [
{
"src": "/app-icon-192.png",
"sizes": "192x192",
"type": "image/png",
"purpose": "any"
},
{
"src": "/app-icon-512.png",
"sizes": "512x512",
"type": "image/png",
"purpose": "any"
},
{
"src": "/app-icon-maskable-512.png",
"sizes": "512x512",
"type": "image/png",
"purpose": "maskable"
}
]
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 864 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 855 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.3 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.5 MiB

+101 -73
View File
@@ -1,83 +1,111 @@
import { Suspense, lazy } from "react";
import { Navigate, Route, Routes } from "react-router-dom";
import HomePage from "@/pages/HomePage";
import ListingPage from "@/pages/ListingPage";
import ShortsPage from "@/pages/ShortsPage";
import UploadPage from "@/pages/UploadPage";
import VideoDetailPage from "@/pages/VideoDetailPage";
import { AdminLayout } from "@/admin/AdminLayout";
import { LoginPage } from "@/admin/LoginPage";
import { SkyStarfield } from "@/components/SkyStarfield";
import { RequireAuth } from "@/admin/RequireAuth";
import { DrivesPage } from "@/admin/DrivesPage";
import { CrawlersPage } from "@/admin/CrawlersPage";
import { VideosPage } from "@/admin/VideosPage";
import { TagsPage } from "@/admin/TagsPage";
import { ThemePage } from "@/admin/ThemePage";
const HomePage = lazy(() => import("@/pages/HomePage"));
const ListingPage = lazy(() => import("@/pages/ListingPage"));
const ShortsPage = lazy(() => import("@/pages/ShortsPage"));
const UploadPage = lazy(() => import("@/pages/UploadPage"));
const VideoDetailPage = lazy(() => import("@/pages/VideoDetailPage"));
const LoginPage = lazy(() =>
import("@/admin/LoginPage").then((module) => ({ default: module.LoginPage }))
);
const AdminLayout = lazy(() =>
import("@/admin/AdminLayout").then((module) => ({
default: module.AdminLayout,
}))
);
const DrivesPage = lazy(() =>
import("@/admin/DrivesPage").then((module) => ({ default: module.DrivesPage }))
);
const CrawlersPage = lazy(() =>
import("@/admin/CrawlersPage").then((module) => ({
default: module.CrawlersPage,
}))
);
const VideosPage = lazy(() =>
import("@/admin/VideosPage").then((module) => ({ default: module.VideosPage }))
);
const TagsPage = lazy(() =>
import("@/admin/TagsPage").then((module) => ({ default: module.TagsPage }))
);
const ThemePage = lazy(() =>
import("@/admin/ThemePage").then((module) => ({ default: module.ThemePage }))
);
export default function App() {
return (
<Routes>
<Route path="/login" element={<LoginPage />} />
<>
{/* 星空蓝主题的固定位置星星层,仅在 data-theme="sky" 下可见 */}
<SkyStarfield />
<Suspense fallback={null}>
<Routes>
<Route path="/login" element={<LoginPage />} />
{/* 主站需要登录 */}
<Route
path="/"
element={
<RequireAuth>
<HomePage />
</RequireAuth>
}
/>
<Route
path="/list"
element={
<RequireAuth>
<ListingPage />
</RequireAuth>
}
/>
<Route
path="/shorts"
element={
<RequireAuth>
<ShortsPage />
</RequireAuth>
}
/>
<Route
path="/upload"
element={
<RequireAuth>
<UploadPage />
</RequireAuth>
}
/>
<Route
path="/video/:id"
element={
<RequireAuth>
<VideoDetailPage />
</RequireAuth>
}
/>
{/* 主站需要登录 */}
<Route
path="/"
element={
<RequireAuth>
<HomePage />
</RequireAuth>
}
/>
<Route
path="/list"
element={
<RequireAuth>
<ListingPage />
</RequireAuth>
}
/>
<Route
path="/shorts"
element={
<RequireAuth>
<ShortsPage />
</RequireAuth>
}
/>
<Route
path="/upload"
element={
<RequireAuth>
<UploadPage />
</RequireAuth>
}
/>
<Route
path="/video/:id"
element={
<RequireAuth>
<VideoDetailPage />
</RequireAuth>
}
/>
{/* 管理后台也需要登录 */}
<Route
path="/admin"
element={
<RequireAuth>
<AdminLayout />
</RequireAuth>
}
>
<Route index element={<Navigate to="/admin/drives" replace />} />
<Route path="drives" element={<DrivesPage />} />
<Route path="crawlers" element={<CrawlersPage />} />
<Route path="videos" element={<VideosPage />} />
<Route path="tags" element={<TagsPage />} />
<Route path="theme" element={<ThemePage />} />
</Route>
{/* 管理后台也需要登录 */}
<Route
path="/admin"
element={
<RequireAuth>
<AdminLayout />
</RequireAuth>
}
>
<Route index element={<Navigate to="/admin/drives" replace />} />
<Route path="drives" element={<DrivesPage />} />
<Route path="crawlers" element={<CrawlersPage />} />
<Route path="videos" element={<VideosPage />} />
<Route path="tags" element={<TagsPage />} />
<Route path="theme" element={<ThemePage />} />
</Route>
<Route path="*" element={<Navigate to="/" replace />} />
</Routes>
<Route path="*" element={<Navigate to="/" replace />} />
</Routes>
</Suspense>
</>
);
}
-7
View File
@@ -4,7 +4,6 @@ import {
HardDrive,
Film,
LogOut,
Play,
Home,
Tags,
Palette,
@@ -71,12 +70,6 @@ export function AdminLayout() {
return (
<div className="admin-shell">
<aside className="admin-sidebar">
<div className="admin-sidebar__brand">
<span className="admin-sidebar__brand-mark">
<Play size={14} fill="#000" />
</span>
<span className="admin-sidebar__brand-text">91</span>
</div>
<nav className="admin-nav">
<div className="admin-nav__group admin-nav__group--home">
<span className="admin-nav__group-label"></span>
+71 -50
View File
@@ -18,6 +18,8 @@ import {
Link as LinkIcon,
Pencil,
Plus,
Power,
PowerOff,
RefreshCw,
TestTube,
Trash2,
@@ -33,7 +35,7 @@ import { SpiderIcon } from "./icons/SpiderIcon";
const BUSY_STATES = new Set(["scanning", "generating", "uploading", "queued"]);
const POLL_INTERVAL_MS = 5000;
const UPLOAD_TARGET_KINDS = new Set(["p115", "pikpak", "p123", "googledrive", "onedrive", "wopan"]);
const UPLOAD_TARGET_KINDS = new Set(["p115", "pikpak", "p123", "googledrive", "onedrive", "wopan", "guangyapan"]);
function statusBusy(status?: api.DriveGenerationStatus) {
return BUSY_STATES.has(status?.state ?? "");
@@ -55,7 +57,9 @@ export function CrawlersPage() {
const [loading, setLoading] = useState(true);
const [expandedId, setExpandedId] = useState("");
const [runningId, setRunningId] = useState("");
const [uploadingId, setUploadingId] = useState("");
const [stoppingId, setStoppingId] = useState("");
const [togglingTeaserId, setTogglingTeaserId] = useState("");
// undefined = 编辑器关闭;null = 新建;其余 = 编辑已有爬虫
const [editorTarget, setEditorTarget] = useState<api.AdminCrawler | null | undefined>(undefined);
const [deleteTarget, setDeleteTarget] = useState<api.AdminCrawler | null>(null);
@@ -123,6 +127,23 @@ export function CrawlersPage() {
}
}
async function uploadVideos(crawler: api.AdminCrawler) {
setUploadingId(crawler.id);
try {
const resp = await api.uploadCrawlerVideos(crawler.id);
if (!resp.accepted) {
show(resp.message || "当前爬虫暂不满足上传条件", "info");
return;
}
show("已触发上传任务", "success");
await refresh(true);
} catch (e) {
show(e instanceof Error ? e.message : "触发上传失败", "error");
} finally {
setUploadingId("");
}
}
async function stop(crawler: api.AdminCrawler) {
setStoppingId(crawler.id);
try {
@@ -136,6 +157,23 @@ export function CrawlersPage() {
}
}
async function toggleTeaser(crawler: api.AdminCrawler) {
const next = !crawler.teaserEnabled;
setTogglingTeaserId(crawler.id);
setList((prev) => prev.map((item) => (item.id === crawler.id ? { ...item, teaserEnabled: next } : item)));
try {
const resp = await api.setDriveTeaserEnabled(crawler.id, next);
setList((prev) => prev.map((item) => (item.id === crawler.id ? { ...item, teaserEnabled: resp.teaserEnabled } : item)));
show(resp.teaserEnabled ? `已开启「${crawler.name}」预览视频生成` : `已关闭「${crawler.name}」预览视频生成`, "success");
await refresh(true);
} catch (e) {
setList((prev) => prev.map((item) => (item.id === crawler.id ? { ...item, teaserEnabled: crawler.teaserEnabled } : item)));
show(e instanceof Error ? e.message : "切换预览视频失败", "error");
} finally {
setTogglingTeaserId("");
}
}
async function confirmDelete() {
if (!deleteTarget) return;
setDeleting(true);
@@ -213,10 +251,14 @@ export function CrawlersPage() {
crawler={crawler}
expanded={expandedId === crawler.id}
running={runningId === crawler.id}
uploading={uploadingId === crawler.id}
stopping={stoppingId === crawler.id}
togglingTeaser={togglingTeaserId === crawler.id}
onToggle={() => setExpandedId(expandedId === crawler.id ? "" : crawler.id)}
onRun={() => run(crawler)}
onUpload={() => uploadVideos(crawler)}
onStop={() => stop(crawler)}
onToggleTeaser={() => toggleTeaser(crawler)}
onEdit={() => setEditorTarget(crawler)}
onDelete={() => setDeleteTarget(crawler)}
/>
@@ -262,51 +304,37 @@ function CrawlerMetric({ label, value, icon, tone }: { label: string; value: num
);
}
type StageInfo = {
key: string;
label: string;
status?: api.DriveGenerationStatus;
};
function crawlerStages(crawler: api.AdminCrawler): StageInfo[] {
return [
{ key: "scan", label: "抓取", status: crawler.scanGenerationStatus },
{ key: "thumbnail", label: "封面", status: crawler.thumbnailGenerationStatus },
{ key: "preview", label: "预览", status: crawler.previewGenerationStatus },
{ key: "fingerprint", label: "指纹", status: crawler.fingerprintGenerationStatus },
{ key: "upload", label: "上传", status: crawler.uploadGenerationStatus },
];
}
function stageStateLabel(stage: StageInfo): string {
const state = stage.status?.state || "idle";
if (stage.key === "scan" && state === "scanning") return "抓取中";
if (stage.key === "upload" && state === "uploading") return "上传中";
return generationStateLabel(state);
}
function CrawlerRow({
crawler,
expanded,
running,
uploading,
stopping,
togglingTeaser,
onToggle,
onRun,
onUpload,
onStop,
onToggleTeaser,
onEdit,
onDelete,
}: {
crawler: api.AdminCrawler;
expanded: boolean;
running: boolean;
uploading: boolean;
stopping: boolean;
togglingTeaser: boolean;
onToggle: () => void;
onRun: () => void;
onUpload: () => void;
onStop: () => void;
onToggleTeaser: () => void;
onEdit: () => void;
onDelete: () => void;
}) {
const busy = crawlerBusy(crawler);
const uploadButtonTitle = uploading ? "上传请求处理中" : "上传本地爬虫视频到已配置的上传网盘";
return (
<div className={`admin-crawler-row ${expanded ? "is-expanded" : ""}`}>
<div className="admin-crawler-row__line">
@@ -320,29 +348,20 @@ function CrawlerRow({
{formatLastCrawl(crawler.lastCrawlAt)} · {crawler.targetNew || "10"} · {crawler.totalCrawledCount ?? 0}
</span>
</span>
<span className="admin-crawler-pipeline">
{crawlerStages(crawler).map((stage) => {
const state = stage.status?.state || "idle";
const active = BUSY_STATES.has(state) || state === "cooling";
return (
<span
key={stage.key}
className={`admin-crawler-stage is-${generationStateClass(state)}`}
title={`${stage.label}${stageStateLabel(stage)}`}
>
<span className="admin-crawler-stage__dot" />
{stage.label}
{active && <em>{stageStateLabel(stage)}</em>}
</span>
);
})}
</span>
<span className={`admin-status is-${crawler.status === "ok" ? "ok" : crawler.status === "error" ? "error" : "pending"}`}>
{crawlerStatusLabel(crawler)}
</span>
<ChevronDown size={16} className="admin-crawler-row__chevron" />
</button>
<div className="admin-crawler-row__actions">
<button
className="admin-btn admin-crawler-preview-card-toggle"
type="button"
onClick={onToggleTeaser}
disabled={togglingTeaser}
aria-pressed={crawler.teaserEnabled}
title={crawler.teaserEnabled ? "关闭后,该爬虫新爬取的视频不再生成预览视频" : "开启后,该爬虫新爬取的视频会生成预览视频"}
>
{crawler.teaserEnabled ? <Power size={13} /> : <PowerOff size={13} />}
<span>{crawler.teaserEnabled ? "预览:开" : "预览:关"}</span>
</button>
{busy ? (
<button className="admin-btn is-stop" type="button" onClick={onStop} disabled={stopping}>
<CircleStop size={13} /> {stopping ? "停止中..." : "停止"}
@@ -352,6 +371,14 @@ function CrawlerRow({
<Download size={13} /> {running ? "触发中..." : "立即抓取"}
</button>
)}
<button
className="admin-btn"
type="button"
onClick={onUpload}
title={uploadButtonTitle}
>
<Upload size={13} /> {uploading ? "上传中..." : "上传视频"}
</button>
<button className="admin-btn" type="button" onClick={onEdit}>
<Pencil size={13} />
</button>
@@ -1038,12 +1065,6 @@ function crawlerTestFailure(result: api.CrawlerDryRunResult) {
return result.error || result.mediaCheck?.error || "";
}
function crawlerStatusLabel(crawler: api.AdminCrawler) {
if (crawler.status === "ok") return "已就绪";
if (crawler.status === "error") return "错误";
return "未连接";
}
function formatLastCrawl(ts?: number) {
if (!ts) return "从未";
return new Date(ts * 1000).toLocaleString("zh-CN", {
+50 -4
View File
@@ -48,6 +48,7 @@ function isDriveBusy(d: api.AdminDrive) {
d.thumbnailGenerationStatus,
d.previewGenerationStatus,
d.fingerprintGenerationStatus,
d.transcodeGenerationStatus,
].some((status) => {
const state = status?.state || "idle";
return state !== "idle";
@@ -74,6 +75,7 @@ export function DrivesPage() {
const [regenFailedThumbId, setRegenFailedThumbId] = useState("");
const [regenFailedFingerprintId, setRegenFailedFingerprintId] = useState("");
const [togglingTeaserId, setTogglingTeaserId] = useState("");
const [togglingTranscodeId, setTogglingTranscodeId] = useState("");
const [scanningAll, setScanningAll] = useState(false);
const [stoppingAll, setStoppingAll] = useState(false);
const [trackingNightly, setTrackingNightly] = useState(false);
@@ -100,7 +102,8 @@ export function DrivesPage() {
d.kind === "p123" ||
d.kind === "onedrive" ||
d.kind === "googledrive" ||
d.kind === "wopan"
d.kind === "wopan" ||
d.kind === "guangyapan"
),
[list]
);
@@ -214,7 +217,12 @@ export function DrivesPage() {
d.kind === "spider91"
? { proxy: d.spider91Proxy ?? "" }
: d.kind === "googledrive"
? { use_online_api: (d.googleDriveUseOnlineAPI ?? true) ? "true" : "false" }
? {
use_online_api: (d.googleDriveUseOnlineAPI ?? true) ? "true" : "false",
api_url_address: d.googleDriveOpenListApiUrl ?? "",
}
: d.kind === "localstorage"
? { strm_allow_outside_root: (d.strmAllowOutsideRoot ?? false) ? "true" : "false" }
: {},
spider91UploadDriveId: settings?.spider91UploadDriveId ?? "",
};
@@ -499,6 +507,41 @@ export function DrivesPage() {
}
}
async function handleStartTranscode(d: api.AdminDrive) {
setTogglingTranscodeId(d.id);
try {
const resp = await api.startDriveTranscode(d.id);
if (resp.accepted) {
show(`已开始「${d.name || d.id}」的视频转码`, "success");
} else {
show(resp.message || "转码任务未能开启", "info");
}
refreshDriveList();
} catch (e) {
show(e instanceof Error ? e.message : "开启失败", "error");
} finally {
setTogglingTranscodeId("");
}
}
async function handleStopTranscode(d: api.AdminDrive) {
setTogglingTranscodeId(d.id);
try {
const resp = await api.stopDriveTranscode(d.id);
show(
resp.stopped
? `已停止「${d.name || d.id}」的视频转码`
: `${d.name || d.id}」没有正在运行的转码任务`,
"success"
);
refreshDriveList();
} catch (e) {
show(e instanceof Error ? e.message : "停止失败", "error");
} finally {
setTogglingTranscodeId("");
}
}
const selectedDrive = useMemo(() => {
return selectedDriveId ? list.find((d) => d.id === selectedDriveId) : null;
}, [selectedDriveId, list]);
@@ -592,7 +635,7 @@ export function DrivesPage() {
</button>
<button
type="button"
className="admin-btn is-stop"
className="admin-btn is-primary"
onClick={() => handleStopDriveTasks(d)}
disabled={!!stoppingDriveId}
title="停止此网盘当前的扫描、封面、预览视频和视频指纹生成任务。"
@@ -602,7 +645,7 @@ export function DrivesPage() {
</button>
</div>
{d.kind !== "spider91" && (
<button type="button" className="admin-btn" onClick={() => openEdit(d)}>
<button type="button" className="admin-btn is-primary" onClick={() => openEdit(d)}>
</button>
)}
@@ -634,10 +677,13 @@ export function DrivesPage() {
regenFailedThumbId={regenFailedThumbId}
regenFailedFingerprintId={regenFailedFingerprintId}
togglingTeaserId={togglingTeaserId}
togglingTranscodeId={togglingTranscodeId}
onToggleTeaser={() => handleToggleTeaser(d)}
onRegenFailed={() => handleRegenFailed(d)}
onRegenFailedThumbnails={() => handleRegenFailedThumbnails(d)}
onRegenFailedFingerprints={() => handleRegenFailedFingerprints(d)}
onStartTranscode={() => handleStartTranscode(d)}
onStopTranscode={() => handleStopTranscode(d)}
/>
<div className="admin-detail-card">
+5 -3
View File
@@ -79,9 +79,11 @@ export function LoginPage() {
return (
<div className="admin-login">
<form className="admin-login__card" onSubmit={handleSubmit}>
<h1 className="admin-login__title">
<Play size={18} fill="currentColor" /> {setupRequired ? "首次设置管理员" : "登录"}
</h1>
{setupRequired && (
<h1 className="admin-login__title">
<Play size={18} fill="currentColor" />
</h1>
)}
<div className="admin-form">
<div className="admin-form__row">
<label htmlFor="admin-login-username"></label>
+9 -2
View File
@@ -1,12 +1,12 @@
import { useEffect, useState } from "react";
import { Check, Loader2, Moon, Sparkles } from "lucide-react";
import { Check, Loader2, Moon, Sparkles, Star } from "lucide-react";
import * as api from "./api";
import type { Theme } from "./api";
import { useToast } from "./ToastContext";
import { applyTheme, getCurrentTheme } from "@/lib/theme";
function isTheme(value: unknown): value is Theme {
return value === "dark" || value === "pink";
return value === "dark" || value === "pink" || value === "sky";
}
type Option = {
@@ -32,6 +32,13 @@ const OPTIONS: Option[] = [
description: "柔和奶白底 + 樱花粉主色,清爽温柔,日间使用更舒适。",
icon: Sparkles,
},
{
id: "sky",
title: "星空蓝 + 暖星黄",
subtitle: "Starry Sky",
description: "浅天空蓝底 + 暖星黄主色,配上淡淡的网格与点点星光,顶级美感。",
icon: Star,
},
];
/**
+128 -20
View File
@@ -9,35 +9,125 @@ import {
} from "react";
type ToastKind = "info" | "success" | "error";
type Toast = { id: number; kind: ToastKind; text: string };
type Toast = { id: number; kind: ToastKind; text: string; copyable: boolean };
type Ctx = {
show: (text: string, kind?: ToastKind) => void;
};
const ToastCtx = createContext<Ctx | null>(null);
const TOAST_DISMISS_MS = 2500;
const TOAST_COPY_SUCCESS_TEXT = "已复制到剪贴板";
const TOAST_COPY_ERROR_TEXT = "复制失败,请手动复制";
async function copyTextToClipboard(text: string) {
try {
if (navigator.clipboard?.writeText) {
await navigator.clipboard.writeText(text);
return;
}
} catch {
// Fall back to the legacy copy command below.
}
if (!fallbackCopyText(text)) {
throw new Error("copy failed");
}
}
function fallbackCopyText(text: string) {
if (!document.body) return false;
const textarea = document.createElement("textarea");
textarea.value = text;
textarea.setAttribute("readonly", "");
textarea.style.position = "fixed";
textarea.style.left = "-9999px";
textarea.style.top = "0";
document.body.appendChild(textarea);
textarea.select();
try {
return document.execCommand("copy");
} finally {
document.body.removeChild(textarea);
}
}
export function ToastProvider({ children }: { children: ReactNode }) {
const [items, setItems] = useState<Toast[]>([]);
const timers = useRef<Record<string, ReturnType<typeof window.setTimeout>>>({});
const timers = useRef(new Map<number, ReturnType<typeof window.setTimeout>>());
const idsByText = useRef(new Map<string, number>());
const clearDismissTimer = useCallback((id: number) => {
const timer = timers.current.get(id);
if (!timer) return;
window.clearTimeout(timer);
timers.current.delete(id);
}, []);
const removeToast = useCallback(
(id: number, text: string) => {
clearDismissTimer(id);
if (idsByText.current.get(text) === id) {
idsByText.current.delete(text);
}
setItems((list) => list.filter((t) => t.id !== id));
},
[clearDismissTimer]
);
const scheduleDismiss = useCallback(
(id: number, text: string) => {
clearDismissTimer(id);
timers.current.set(
id,
window.setTimeout(() => removeToast(id, text), TOAST_DISMISS_MS)
);
},
[clearDismissTimer, removeToast]
);
const addToast = useCallback(
(text: string, kind: ToastKind = "info", copyable = true) => {
const existingID = idsByText.current.get(text);
if (existingID !== undefined) {
setItems((list) =>
list.map((t) => (t.id === existingID ? { ...t, kind, copyable } : t))
);
scheduleDismiss(existingID, text);
return;
}
const id = Date.now() + Math.random();
idsByText.current.set(text, id);
setItems((list) => [...list, { id, kind, text, copyable }]);
scheduleDismiss(id, text);
},
[scheduleDismiss]
);
// Deduplicate: same text won't stack, just resets the dismiss timer
const show = useCallback((text: string, kind: ToastKind = "info") => {
// Reset timer if duplicate
if (timers.current[text]) {
window.clearTimeout(timers.current[text]);
timers.current[text] = window.setTimeout(() => {
setItems((list) => list.filter((t) => t.text !== text));
delete timers.current[text];
}, 2600);
return;
}
const id = Date.now() + Math.random();
timers.current[text] = window.setTimeout(() => {
setItems((list) => list.filter((t) => t.id !== id));
delete timers.current[text];
}, 2600);
setItems((list) => [...list, { id, kind, text }]);
const show = useCallback(
(text: string, kind: ToastKind = "info") => {
addToast(text, kind, true);
},
[addToast]
);
const copyToastText = useCallback(
(text: string) => {
void copyTextToClipboard(text)
.then(() => addToast(TOAST_COPY_SUCCESS_TEXT, "success", false))
.catch(() => addToast(TOAST_COPY_ERROR_TEXT, "error", false));
},
[addToast]
);
useEffect(() => {
return () => {
for (const timer of timers.current.values()) {
window.clearTimeout(timer);
}
timers.current.clear();
idsByText.current.clear();
};
}, []);
return (
@@ -45,8 +135,26 @@ export function ToastProvider({ children }: { children: ReactNode }) {
{children}
<div className="admin-toast-stack" role="status" aria-live="polite">
{items.map((t) => (
<div key={t.id} className={`admin-toast is-${t.kind}`}>
{t.text}
<div
key={t.id}
className={`admin-toast is-${t.kind}${
t.copyable ? " is-copyable" : ""
}`}
role={t.copyable ? "button" : undefined}
tabIndex={t.copyable ? 0 : undefined}
aria-label={t.copyable ? `复制提示:${t.text}` : undefined}
onClick={t.copyable ? () => copyToastText(t.text) : undefined}
onKeyDown={
t.copyable
? (event) => {
if (event.key !== "Enter" && event.key !== " ") return;
event.preventDefault();
copyToastText(t.text);
}
: undefined
}
>
<span className="admin-toast__text">{t.text}</span>
</div>
))}
</div>
+586 -179
View File
@@ -1,5 +1,17 @@
import { useEffect, useId, useState } from "react";
import { ChevronDown, Edit, RefreshCw, Search, CheckSquare, Square, Image, Trash2 } from "lucide-react";
import { useSearchParams } from "react-router-dom";
import {
ChevronDown,
Edit,
RefreshCw,
Search,
CheckSquare,
Square,
Image,
Trash2,
Ban,
RotateCcw,
} from "lucide-react";
import * as api from "./api";
import { useToast } from "./ToastContext";
import { Modal } from "./Modal";
@@ -9,8 +21,94 @@ import { formatBytes } from "./storageFormat";
const DESKTOP_VIDEOS_PAGE_SIZE = 50;
const MOBILE_VIDEOS_PAGE_SIZE = 20;
const VIDEOS_MOBILE_QUERY = "(max-width: 640px)";
const REGEN_PREVIEW_STATUS = "generating";
const REGEN_PREVIEW_POLL_INTERVAL_MS = 2000;
const REGEN_PREVIEW_TRACK_TIMEOUT_MS = 30 * 60 * 1000;
type TabKey = "current" | "blacklist";
type RegenPreviewState = {
expiresAt: number;
originalUpdatedAt: number;
};
const TABS: { key: TabKey; label: string }[] = [
{ key: "current", label: "当前视频" },
{ key: "blacklist", label: "拉黑视频" },
];
/**
* / /
* URL ?tab= /videos/stats
*/
export function VideosPage() {
const [searchParams, setSearchParams] = useSearchParams();
const rawTab = searchParams.get("tab");
const activeTab: TabKey = rawTab === "blacklist" ? "blacklist" : "current";
const [stats, setStats] = useState<api.VideoStats | null>(null);
async function refreshStats() {
try {
setStats(await api.getVideoStats());
} catch {
// 计数仅用于标签徽标,失败不阻塞主流程。
}
}
useEffect(() => {
refreshStats();
}, []);
function selectTab(key: TabKey) {
setSearchParams(
(prev) => {
const next = new URLSearchParams(prev);
if (key === "current") next.delete("tab");
else next.set("tab", key);
return next;
},
{ replace: true }
);
}
const counts: Record<TabKey, number | undefined> = {
current: stats?.current,
blacklist: stats?.blacklisted,
};
return (
<section>
<header className="admin-page__header">
<h1 className="admin-page__title"></h1>
</header>
<div className="admin-video-tabs" role="tablist" aria-label="视频管理分类">
{TABS.map((t) => (
<button
key={t.key}
type="button"
role="tab"
aria-selected={activeTab === t.key}
className={`admin-video-tab ${activeTab === t.key ? "is-active" : ""}`}
onClick={() => selectTab(t.key)}
>
<span>{t.label}</span>
{counts[t.key] !== undefined && (
<span className="admin-video-tab__count">{counts[t.key]}</span>
)}
</button>
))}
</div>
{activeTab === "current" && <CurrentVideosTab onStatsChanged={refreshStats} />}
{activeTab === "blacklist" && <BlacklistTab onStatsChanged={refreshStats} />}
</section>
);
}
// ---------- 当前视频 ----------
function CurrentVideosTab({ onStatsChanged }: { onStatsChanged: () => void }) {
const [list, setList] = useState<api.AdminVideo[]>([]);
const [drives, setDrives] = useState<api.AdminDrive[]>([]);
const [loading, setLoading] = useState(true);
@@ -31,6 +129,7 @@ export function VideosPage() {
const [deleteTarget, setDeleteTarget] = useState<api.AdminVideo | null>(null);
const [deleting, setDeleting] = useState(false);
const [deleteSource, setDeleteSource] = useState(false);
const [regenPreviewById, setRegenPreviewById] = useState<Record<string, RegenPreviewState>>({});
const pageSize = useVideosPageSize();
const { show } = useToast();
@@ -57,6 +156,19 @@ export function VideosPage() {
}
}
async function refreshListOnly() {
try {
const r = await api.listVideos({ driveId, page, size: pageSize, keyword: searchKeyword });
setList(r.items ?? []);
setTotal(r.total ?? 0);
} catch {
// Polling is only used to clear optimistic preview-generation state.
}
}
const trackedRegenCount = Object.keys(regenPreviewById).length;
const hasGeneratingPreview = list.some((v) => v.previewStatus === REGEN_PREVIEW_STATUS);
useEffect(() => {
refresh();
}, [driveId, page, searchKeyword, pageSize]);
@@ -74,9 +186,34 @@ export function VideosPage() {
return () => window.clearTimeout(timer);
}, [keyword]);
const driveNameMap = new Map(
drives.map((d) => [d.id, d.name || d.id])
);
useEffect(() => {
if (trackedRegenCount === 0 && !hasGeneratingPreview) return;
const timer = window.setInterval(() => {
refreshListOnly();
}, REGEN_PREVIEW_POLL_INTERVAL_MS);
return () => window.clearInterval(timer);
}, [trackedRegenCount, hasGeneratingPreview, driveId, page, pageSize, searchKeyword]);
useEffect(() => {
if (trackedRegenCount === 0) return;
const now = Date.now();
setRegenPreviewById((current) => {
const next = { ...current };
let changed = false;
const byId = new Map(list.map((v) => [v.id, v]));
for (const [id, state] of Object.entries(current)) {
const video = byId.get(id);
const updatedAt = videoUpdatedAtMs(video);
if (!video || now >= state.expiresAt || updatedAt > state.originalUpdatedAt) {
delete next[id];
changed = true;
}
}
return changed ? next : current;
});
}, [list, trackedRegenCount]);
const driveNameMap = new Map(drives.map((d) => [d.id, d.name || d.id]));
const listItems = list;
const totalPages = Math.max(1, Math.ceil(total / pageSize));
@@ -89,6 +226,7 @@ export function VideosPage() {
async function handleRegen(v: api.AdminVideo) {
try {
await api.regenPreview(v.id);
trackRegeneratingPreview([v]);
show("已触发预览视频重生", "success");
} catch (e) {
show(e instanceof Error ? e.message : "触发失败", "error");
@@ -108,16 +246,24 @@ export function VideosPage() {
async function confirmBatchRegen() {
const ids = [...selectedIds];
const videoById = new Map(listItems.map((v) => [v.id, v]));
setBatchRegening(true);
let success = 0;
try {
const results = await Promise.allSettled(
ids.map((id) => api.regenPreview(id))
const results = await Promise.allSettled(ids.map((id) => api.regenPreview(id)));
const acceptedVideos: api.AdminVideo[] = [];
results.forEach((r, index) => {
if (r.status === "fulfilled") {
const video = videoById.get(ids[index]);
if (video) acceptedVideos.push(video);
success++;
}
});
trackRegeneratingPreview(acceptedVideos);
show(
`批量触发完成,成功 ${success} / ${ids.length}`,
success === ids.length ? "success" : "info"
);
for (const r of results) {
if (r.status === "fulfilled") success++;
}
show(`批量触发完成,成功 ${success} / ${ids.length}`, success === ids.length ? "success" : "info");
setSelectedIds(new Set());
setBatchRegenOpen(false);
} finally {
@@ -125,6 +271,25 @@ export function VideosPage() {
}
}
function trackRegeneratingPreview(videos: api.AdminVideo[]) {
if (videos.length === 0) return;
const startedAt = Date.now();
setRegenPreviewById((current) => {
const next = { ...current };
for (const v of videos) {
next[v.id] = {
expiresAt: startedAt + REGEN_PREVIEW_TRACK_TIMEOUT_MS,
originalUpdatedAt: videoUpdatedAtMs(v),
};
}
return next;
});
}
function isPreviewGenerating(v: api.AdminVideo) {
return !!regenPreviewById[v.id] || v.previewStatus === REGEN_PREVIEW_STATUS;
}
async function confirmDeleteVideo() {
if (!deleteTarget) return;
const target = deleteTarget;
@@ -139,6 +304,7 @@ export function VideosPage() {
return next;
});
show(result.deletedSource ? "已删除视频,并清理源文件" : "已删除视频", "success");
onStatsChanged();
if (listItems.length === 1 && page > 1) {
setPage((p) => Math.max(1, p - 1));
} else {
@@ -172,11 +338,15 @@ export function VideosPage() {
const extra = deletedSources > 0 ? `,其中 ${deletedSources} 个清理了源文件` : "";
show(`批量删除完成,成功 ${success}${extra}`, "success");
} else {
show(`批量删除完成,成功 ${success} / ${ids.length} 个,失败 ${failed}`, success > 0 ? "info" : "error");
show(
`批量删除完成,成功 ${success} / ${ids.length} 个,失败 ${failed}`,
success > 0 ? "info" : "error"
);
}
setSelectedIds(new Set());
setBatchDeleteOpen(false);
setBatchDeleteSource(false);
onStatsChanged();
if (success >= listItems.length && page > 1) {
setPage((p) => Math.max(1, p - 1));
} else {
@@ -191,7 +361,7 @@ export function VideosPage() {
if (selectedIds.size === listItems.length && listItems.length > 0) {
setSelectedIds(new Set());
} else {
setSelectedIds(new Set(listItems.map(v => v.id)));
setSelectedIds(new Set(listItems.map((v) => v.id)));
}
};
@@ -209,52 +379,21 @@ export function VideosPage() {
}
return (
<section>
<header className="admin-page__header">
<h1 className="admin-page__title"></h1>
<div className="admin-page__actions admin-videos-filter">
<div className="admin-videos-filter__select-wrap">
<select
className="admin-videos-filter__select"
value={driveId}
onChange={(e) => {
setDriveId(e.target.value);
setPage(1);
}}
>
<option value=""></option>
{drives.map((d) => (
<option key={d.id} value={d.id}>
{d.name || d.id} {d.teaserReadyCount ?? 0}{" "}
{d.teaserPendingCount ?? 0}
</option>
))}
</select>
<ChevronDown size={15} className="admin-videos-filter__select-icon" aria-hidden="true" />
</div>
<form className="admin-videos-filter__search" onSubmit={handleSearchSubmit}>
<Search size={14} className="admin-videos-filter__search-icon" />
<input
aria-label="搜索标题或作者"
value={keyword}
onChange={(e) => setKeyword(e.target.value)}
placeholder="搜索标题 / 作者"
/>
</form>
<button type="button" className="admin-btn" onClick={refresh}>
<RefreshCw size={13} />
</button>
</div>
</header>
<>
<div className="admin-page__actions admin-videos-filter">
<DriveFilter drives={drives} driveId={driveId} onChange={(id) => { setDriveId(id); setPage(1); }} withCounts />
<SearchBox keyword={keyword} onChange={setKeyword} onSubmit={handleSearchSubmit} />
<button type="button" className="admin-btn" onClick={refresh}>
<RefreshCw size={13} />
</button>
</div>
{!loading && (
<div className="admin-videos-list-toolbar">
<div className="admin-videos-summary">{listSummary}</div>
{selectedIds.size > 0 && (
<div className="admin-videos-bulk-actions">
<span className="admin-videos-bulk-actions__count">
{selectedIds.size}
</span>
<span className="admin-videos-bulk-actions__count"> {selectedIds.size} </span>
<button type="button" className="admin-btn is-primary admin-videos-bulk-actions__btn" onClick={handleBatchRegen}>
<RefreshCw size={13} />
</button>
@@ -267,18 +406,9 @@ export function VideosPage() {
)}
{loading ? (
<div className="admin-loading-state">
<RefreshCw size={20} className="admin-spin" />
<span>...</span>
</div>
<LoadingState />
) : loadError ? (
<div className="admin-error-state">
<strong></strong>
<span>{loadError}</span>
<button type="button" className="admin-btn" onClick={refresh}>
<RefreshCw size={13} />
</button>
</div>
<ErrorState message={loadError} onRetry={refresh} />
) : listItems.length === 0 ? (
<div className="admin-empty-state">
<div className="admin-empty-state__icon">
@@ -295,14 +425,22 @@ export function VideosPage() {
<table className="admin-table is-selectable admin-videos-table">
<thead>
<tr>
<th className="is-checkbox" style={{ width: '40px' }}>
<th className="is-checkbox" style={{ width: "40px" }}>
<button
type="button"
className="admin-table-checkbox-btn"
onClick={toggleSelectAll}
aria-label={selectedIds.size > 0 && selectedIds.size === listItems.length ? "清空当前页选择" : "选择当前页视频"}
aria-label={
selectedIds.size > 0 && selectedIds.size === listItems.length
? "清空当前页选择"
: "选择当前页视频"
}
>
{selectedIds.size > 0 && selectedIds.size === listItems.length ? <CheckSquare size={16} /> : <Square size={16} />}
{selectedIds.size > 0 && selectedIds.size === listItems.length ? (
<CheckSquare size={16} />
) : (
<Square size={16} />
)}
</button>
</th>
<th></th>
@@ -323,40 +461,20 @@ export function VideosPage() {
onClick={() => toggleSelect(v.id)}
aria-label={`${selectedIds.has(v.id) ? "取消选择" : "选择"}视频 ${v.title}`}
>
{selectedIds.has(v.id) ? <CheckSquare size={16} color="var(--accent)" /> : <Square size={16} color="var(--border-strong)" />}
{selectedIds.has(v.id) ? (
<CheckSquare size={16} color="var(--accent)" />
) : (
<Square size={16} color="var(--border-strong)" />
)}
</button>
</td>
<td data-label="标题">
<div className="admin-video-title-cell">
<div className="admin-video-thumb-wrap" aria-hidden="true">
{v.thumbnailUrl ? (
<img className="admin-video-thumb" src={v.thumbnailUrl} alt="" />
) : (
<div className="admin-video-thumb-placeholder">
<Image size={14} />
</div>
)}
</div>
<div className="admin-video-title-body">
<div className="admin-video-title">{v.title}</div>
{fileMeta(v) && (
<div className="admin-video-filemeta">{fileMeta(v)}</div>
)}
{(v.tags ?? []).length > 0 && (
<div className="admin-pills admin-video-title-tags">
{(v.tags ?? []).map((t) => (
<span key={t} className="admin-pill">{t}</span>
))}
</div>
)}
<VideoFileMetaPills video={v} />
</div>
</div>
<VideoTitleCell video={v} />
</td>
<td data-label="作者">{v.author || <span className="admin-text-faint"></span>}</td>
<td data-label="时长">{formatDur(v.durationSeconds)}</td>
<td data-label="预览视频">
<PreviewStatus s={v.previewStatus} />
<PreviewStatus s={isPreviewGenerating(v) ? REGEN_PREVIEW_STATUS : v.previewStatus} />
</td>
<td data-label="来源" className="admin-mono-cell">
{driveNameMap.get(v.driveId) ?? v.driveId}
@@ -365,8 +483,14 @@ export function VideosPage() {
<button type="button" className="admin-btn" onClick={() => setEditing(v)} title="编辑视频">
<Edit size={13} />
</button>{" "}
<button type="button" className="admin-btn" onClick={() => handleRegen(v)} title="重生预览视频">
<RefreshCw size={13} />
<button
type="button"
className="admin-btn"
onClick={() => handleRegen(v)}
disabled={isPreviewGenerating(v)}
title={isPreviewGenerating(v) ? "预览视频正在生成" : "重生预览视频"}
>
<RefreshCw size={13} className={isPreviewGenerating(v) ? "admin-spin" : undefined} />
</button>{" "}
<button
type="button"
@@ -384,43 +508,7 @@ export function VideosPage() {
))}
</tbody>
</table>
<div className="admin-table-pagination">
<button
type="button"
className="admin-btn"
onClick={() => setPage(1)}
disabled={page <= 1}
>
</button>
<button
type="button"
className="admin-btn"
onClick={() => setPage((p) => Math.max(1, p - 1))}
disabled={page <= 1}
>
</button>
<span className="admin-table-pagination__info">
{page} / {totalPages} {pageSize}
</span>
<button
type="button"
className="admin-btn"
onClick={() => setPage((p) => Math.min(totalPages, p + 1))}
disabled={page >= totalPages}
>
</button>
<button
type="button"
className="admin-btn"
onClick={() => setPage(totalPages)}
disabled={page >= totalPages}
>
</button>
</div>
<Pagination page={page} totalPages={totalPages} pageSize={pageSize} onPage={setPage} />
</>
)}
@@ -463,18 +551,7 @@ export function VideosPage() {
}}
onConfirm={confirmDeleteVideo}
>
<label className="admin-delete-source-option">
<input
type="checkbox"
checked={deleteSource}
disabled={deleting}
onChange={(e) => setDeleteSource(e.target.checked)}
/>
<span>
<strong></strong>
<small></small>
</span>
</label>
<DeleteSourceOption checked={deleteSource} disabled={deleting} onChange={setDeleteSource} note="开启后会先删除源文件,失败则不会删除管理库记录。" />
</ConfirmModal>
<ConfirmModal
open={batchDeleteOpen}
@@ -493,26 +570,354 @@ export function VideosPage() {
}}
onConfirm={confirmBatchDelete}
>
<label className="admin-delete-source-option">
<input
type="checkbox"
checked={batchDeleteSource}
disabled={batchDeleting}
onChange={(e) => setBatchDeleteSource(e.target.checked)}
/>
<span>
<strong></strong>
<small></small>
</span>
</label>
<DeleteSourceOption checked={batchDeleteSource} disabled={batchDeleting} onChange={setBatchDeleteSource} note="开启后会先删除源文件,失败的视频会保留管理库记录。" />
</ConfirmModal>
</section>
</>
);
}
// ---------- 拉黑视频 ----------
function BlacklistTab({ onStatsChanged }: { onStatsChanged: () => void }) {
const [list, setList] = useState<api.AdminDeletedVideo[]>([]);
const [drives, setDrives] = useState<api.AdminDrive[]>([]);
const [loading, setLoading] = useState(true);
const [loadError, setLoadError] = useState("");
const [keyword, setKeyword] = useState("");
const [searchKeyword, setSearchKeyword] = useState("");
const [page, setPage] = useState(1);
const [total, setTotal] = useState(0);
const [removeTarget, setRemoveTarget] = useState<api.AdminDeletedVideo | null>(null);
const [removing, setRemoving] = useState(false);
const pageSize = useVideosPageSize();
const { show } = useToast();
async function refresh() {
setLoading(true);
setLoadError("");
try {
const [r, driveList] = await Promise.all([
api.listBlacklist({ page, size: pageSize, keyword: searchKeyword }),
api.listDrives(),
]);
setList(r.items ?? []);
setTotal(r.total ?? 0);
setDrives(driveList ?? []);
} catch (e) {
const message = e instanceof Error ? e.message : "加载失败";
setLoadError(message);
show(message, "error");
} finally {
setLoading(false);
}
}
useEffect(() => {
refresh();
}, [page, searchKeyword, pageSize]);
useEffect(() => {
setPage(1);
}, [pageSize]);
useEffect(() => {
if (keyword === searchKeyword) return;
const timer = window.setTimeout(() => {
setSearchKeyword(keyword);
setPage(1);
}, 300);
return () => window.clearTimeout(timer);
}, [keyword]);
const driveNameMap = new Map(drives.map((d) => [d.id, d.name || d.id]));
const totalPages = Math.max(1, Math.ceil(total / pageSize));
async function confirmRemove() {
if (!removeTarget) return;
const target = removeTarget;
setRemoving(true);
try {
await api.removeBlacklist(target.id);
setRemoveTarget(null);
show("已移出黑名单,下次扫盘会重新入库", "success");
onStatsChanged();
if (list.length === 1 && page > 1) {
setPage((p) => Math.max(1, p - 1));
} else {
refresh();
}
} catch (e) {
show(e instanceof Error ? e.message : "操作失败", "error");
} finally {
setRemoving(false);
}
}
function handleSearchSubmit(e: React.FormEvent) {
e.preventDefault();
setSearchKeyword(keyword);
setPage(1);
}
return (
<>
<div className="admin-tab-intro">
</div>
<div className="admin-page__actions admin-videos-filter">
<SearchBox keyword={keyword} onChange={setKeyword} onSubmit={handleSearchSubmit} placeholder="搜索文件名" />
<button type="button" className="admin-btn" onClick={refresh}>
<RefreshCw size={13} />
</button>
</div>
{loading ? (
<LoadingState />
) : loadError ? (
<ErrorState message={loadError} onRetry={refresh} />
) : list.length === 0 ? (
<div className="admin-empty-state">
<div className="admin-empty-state__icon">
<Ban size={48} />
</div>
<div className="admin-empty-state__text"></div>
</div>
) : (
<>
<div className="admin-videos-list-toolbar">
<div className="admin-videos-summary"> {total} </div>
</div>
<table className="admin-table admin-blacklist-table">
<thead>
<tr>
<th></th>
<th></th>
<th></th>
<th></th>
<th className="is-actions"></th>
</tr>
</thead>
<tbody>
{list.map((v) => (
<tr key={v.id}>
<td data-label="文件名">
<span className="admin-blacklist-filename">{v.fileName || <span className="admin-text-faint"></span>}</span>
</td>
<td data-label="来源" className="admin-mono-cell">
{driveNameMap.get(v.driveId) ?? v.driveId}
</td>
<td data-label="大小">{v.size > 0 ? formatBytes(v.size) : <span className="admin-text-faint"></span>}</td>
<td data-label="拉黑时间">{formatDateTime(v.deletedAt)}</td>
<td className="is-actions" data-label="操作">
<button
type="button"
className="admin-btn admin-blacklist-restore-btn"
onClick={() => setRemoveTarget(v)}
title="移出黑名单"
>
<RotateCcw size={13} />
</button>
</td>
</tr>
))}
</tbody>
</table>
<Pagination page={page} totalPages={totalPages} pageSize={pageSize} onPage={setPage} />
</>
)}
<ConfirmModal
open={removeTarget !== null}
title="移出黑名单"
message={
removeTarget
? `确定把「${removeTarget.fileName || removeTarget.id}」移出黑名单吗?移出后它会在下次扫盘时被重新发现并入库。`
: ""
}
confirmText="移出黑名单"
centerMessage
loading={removing}
onCancel={() => {
if (!removing) setRemoveTarget(null);
}}
onConfirm={confirmRemove}
/>
</>
);
}
// ---------- 共享小组件 ----------
function DriveFilter({
drives,
driveId,
onChange,
withCounts = false,
}: {
drives: api.AdminDrive[];
driveId: string;
onChange: (id: string) => void;
withCounts?: boolean;
}) {
return (
<div className="admin-videos-filter__select-wrap">
<select
className="admin-videos-filter__select"
value={driveId}
onChange={(e) => onChange(e.target.value)}
>
<option value=""></option>
{drives.map((d) => (
<option key={d.id} value={d.id}>
{d.name || d.id}
{withCounts ? `(已生成 ${d.teaserReadyCount ?? 0},待生成 ${d.teaserPendingCount ?? 0}` : ""}
</option>
))}
</select>
<ChevronDown size={15} className="admin-videos-filter__select-icon" aria-hidden="true" />
</div>
);
}
function SearchBox({
keyword,
onChange,
onSubmit,
placeholder = "搜索标题 / 作者",
}: {
keyword: string;
onChange: (v: string) => void;
onSubmit: (e: React.FormEvent) => void;
placeholder?: string;
}) {
return (
<form className="admin-videos-filter__search" onSubmit={onSubmit}>
<Search size={14} className="admin-videos-filter__search-icon" />
<input
aria-label={placeholder}
value={keyword}
onChange={(e) => onChange(e.target.value)}
placeholder={placeholder}
/>
</form>
);
}
function Pagination({
page,
totalPages,
pageSize,
onPage,
}: {
page: number;
totalPages: number;
pageSize: number;
onPage: React.Dispatch<React.SetStateAction<number>>;
}) {
return (
<div className="admin-table-pagination">
<button type="button" className="admin-btn" onClick={() => onPage(() => 1)} disabled={page <= 1}>
</button>
<button type="button" className="admin-btn" onClick={() => onPage((p) => Math.max(1, p - 1))} disabled={page <= 1}>
</button>
<span className="admin-table-pagination__info">
{page} / {totalPages} {pageSize}
</span>
<button
type="button"
className="admin-btn"
onClick={() => onPage((p) => Math.min(totalPages, p + 1))}
disabled={page >= totalPages}
>
</button>
<button type="button" className="admin-btn" onClick={() => onPage(() => totalPages)} disabled={page >= totalPages}>
</button>
</div>
);
}
function LoadingState() {
return (
<div className="admin-loading-state">
<RefreshCw size={20} className="admin-spin" />
<span>...</span>
</div>
);
}
function ErrorState({ message, onRetry }: { message: string; onRetry: () => void }) {
return (
<div className="admin-error-state">
<strong></strong>
<span>{message}</span>
<button type="button" className="admin-btn" onClick={onRetry}>
<RefreshCw size={13} />
</button>
</div>
);
}
function DeleteSourceOption({
checked,
disabled,
onChange,
note,
}: {
checked: boolean;
disabled: boolean;
onChange: (v: boolean) => void;
note: string;
}) {
return (
<label className="admin-delete-source-option">
<input type="checkbox" checked={checked} disabled={disabled} onChange={(e) => onChange(e.target.checked)} />
<span>
<strong></strong>
<small>{note}</small>
</span>
</label>
);
}
function VideoTitleCell({ video: v }: { video: api.AdminVideo }) {
return (
<div className="admin-video-title-cell">
<div className="admin-video-thumb-wrap" aria-hidden="true">
{v.thumbnailUrl ? (
<img className="admin-video-thumb" src={v.thumbnailUrl} alt="" />
) : (
<div className="admin-video-thumb-placeholder">
<Image size={14} />
</div>
)}
</div>
<div className="admin-video-title-body">
<div className="admin-video-title">{v.title}</div>
{fileMeta(v) && <div className="admin-video-filemeta">{fileMeta(v)}</div>}
{(v.tags ?? []).length > 0 && (
<div className="admin-pills admin-video-title-tags">
{(v.tags ?? []).map((t) => (
<span key={t} className="admin-pill">
{t}
</span>
))}
</div>
)}
<VideoFileMetaPills video={v} />
</div>
</div>
);
}
function PreviewStatus({ s }: { s: string }) {
if (s === REGEN_PREVIEW_STATUS) return <span className="admin-status is-generating"></span>;
if (s === "ready") return <span className="admin-status is-ok"></span>;
if (s === "failed") return <span className="admin-status is-error"></span>;
if (s === "disabled") return <span className="admin-status"></span>;
if (s === "skipped") return <span className="admin-status"></span>;
return <span className="admin-status is-pending"></span>;
}
@@ -529,11 +934,7 @@ function VideoFileMetaPills({ video }: { video: api.AdminVideo }) {
{part}
</span>
))}
{category && (
<span className="admin-video-filemeta-pill is-category">
{category}
</span>
)}
{category && <span className="admin-video-filemeta-pill is-category">{category}</span>}
</div>
);
}
@@ -545,11 +946,23 @@ function formatDur(sec: number): string {
return `${m.toString().padStart(2, "0")}:${s.toString().padStart(2, "0")}`;
}
function formatDateTime(ms: number): string {
if (!ms) return "—";
const d = new Date(ms);
if (Number.isNaN(d.getTime())) return "—";
const pad = (n: number) => n.toString().padStart(2, "0");
return `${d.getFullYear()}-${pad(d.getMonth() + 1)}-${pad(d.getDate())} ${pad(d.getHours())}:${pad(d.getMinutes())}`;
}
function videoUpdatedAtMs(video?: api.AdminVideo): number {
if (!video?.updatedAt) return 0;
const value = Date.parse(video.updatedAt);
return Number.isFinite(value) ? value : 0;
}
function useVideosPageSize() {
const [pageSize, setPageSize] = useState(() =>
window.matchMedia(VIDEOS_MOBILE_QUERY).matches
? MOBILE_VIDEOS_PAGE_SIZE
: DESKTOP_VIDEOS_PAGE_SIZE
window.matchMedia(VIDEOS_MOBILE_QUERY).matches ? MOBILE_VIDEOS_PAGE_SIZE : DESKTOP_VIDEOS_PAGE_SIZE
);
useEffect(() => {
@@ -683,12 +1096,12 @@ function EditVideoModal({
<div className="admin-thumbnail-preview">
<input id={`${idPrefix}-video-thumbnail`} value={thumbnail} onChange={(e) => setThumbnail(e.target.value)} />
{thumbnail && (
<img
src={thumbnail}
alt="封面预览"
className="admin-thumbnail-img"
onError={(e) => (e.currentTarget.style.display = 'none')}
onLoad={(e) => (e.currentTarget.style.display = 'block')}
<img
src={thumbnail}
alt="封面预览"
className="admin-thumbnail-img"
onError={(e) => (e.currentTarget.style.display = "none")}
onLoad={(e) => (e.currentTarget.style.display = "block")}
/>
)}
</div>
@@ -730,11 +1143,7 @@ function fileMeta(v: api.AdminVideo): string {
}
function fileMetaParts(v: api.AdminVideo): string[] {
return [
normalizeExt(v.ext),
v.quality,
v.size > 0 ? formatBytes(v.size) : "",
].filter(Boolean);
return [normalizeExt(v.ext), v.quality, v.size > 0 ? formatBytes(v.size) : ""].filter(Boolean);
}
function normalizeExt(ext: string): string {
@@ -750,7 +1159,5 @@ function splitList(s: string): string[] {
}
function toggleTag(tags: string[], label: string): string[] {
return tags.includes(label)
? tags.filter((tag) => tag !== label)
: [...tags, label];
return tags.includes(label) ? tags.filter((tag) => tag !== label) : [...tags, label];
}
+116 -5
View File
@@ -78,13 +78,13 @@ export function checkUpdate() {
export type AdminDrive = {
id: string;
kind: "quark" | "p115" | "p123" | "pikpak" | "wopan" | "onedrive" | "googledrive" | "localstorage" | "spider91";
kind: "quark" | "p115" | "p123" | "pikpak" | "wopan" | "guangyapan" | "onedrive" | "googledrive" | "localstorage" | "spider91";
name: string;
rootId: string;
status: string;
lastError?: string;
hasCredential: boolean;
/** 当前是否给该盘生成预览视频/封面(per-drive 开关,替代旧的全局 preview.enabled)。 */
/** 当前是否给该盘生成预览视频(per-drive 开关,替代旧的全局 preview.enabled;封面不受影响)。 */
teaserEnabled: boolean;
/**
* admin "扫描跳过目录"drive fileID
@@ -98,6 +98,10 @@ export type AdminDrive = {
spider91Proxy?: string;
// Google Drive 是否使用 OpenList 在线续期 API;未配置时后端按 true 返回。
googleDriveUseOnlineAPI?: boolean;
// Google Drive OpenList 在线续期 API 地址;为空时后端使用驱动默认值。
googleDriveOpenListApiUrl?: string;
// localstorage 的 .strm 是否允许指向存储根目录之外;未配置时后端按 false 返回。
strmAllowOutsideRoot?: boolean;
scanGenerationStatus?: DriveGenerationStatus;
thumbnailGenerationStatus?: DriveGenerationStatus;
previewGenerationStatus?: DriveGenerationStatus;
@@ -112,6 +116,12 @@ export type AdminDrive = {
fingerprintReadyCount: number;
fingerprintPendingCount: number;
fingerprintFailedCount: number;
// 浏览器兼容性转码:候选(待处理)/已转码/失败/检测后无需转码 计数与任务状态。
transcodeGenerationStatus?: DriveGenerationStatus;
transcodePendingCount: number;
transcodeReadyCount: number;
transcodeFailedCount: number;
transcodeSkippedCount: number;
};
export type DriveGenerationStatus = {
@@ -147,7 +157,7 @@ export function getDriveStorage() {
export type UpsertDriveInput = {
id: string;
kind: "quark" | "p115" | "p123" | "pikpak" | "wopan" | "onedrive" | "googledrive" | "localstorage" | "spider91";
kind: "quark" | "p115" | "p123" | "pikpak" | "wopan" | "guangyapan" | "onedrive" | "googledrive" | "localstorage" | "spider91";
name: string;
rootId: string;
credentials: Record<string, string>;
@@ -204,6 +214,7 @@ export type AdminCrawler = {
proxy?: string;
targetNew?: string;
uploadDriveId?: string;
teaserEnabled: boolean;
lastCrawlAt?: number;
scanGenerationStatus?: DriveGenerationStatus;
thumbnailGenerationStatus?: DriveGenerationStatus;
@@ -306,6 +317,13 @@ export function runCrawler(id: string) {
);
}
export function uploadCrawlerVideos(id: string) {
return request<{ ok: boolean; accepted: boolean; message?: string; status?: NightlyJobStatus }>(
`/crawlers/${encodeURIComponent(id)}/upload`,
{ method: "POST" }
);
}
export function stopCrawlerTasks(id: string) {
return request<{ ok: boolean; stopped: boolean }>(
`/crawlers/${encodeURIComponent(id)}/tasks/stop`,
@@ -368,6 +386,33 @@ export function getWopanQRStatus(uuid: string) {
return request<WopanQRStatus>(`/drives/wopan/qr/${encodeURIComponent(uuid)}`);
}
export type GuangYaPanQRSession = {
deviceCode: string;
qrCodeUrl: string;
qrImageDataUrl: string;
intervalSeconds: number;
expiresAt?: string;
};
export type GuangYaPanQRStatus = {
state: "pending" | "success" | "expired" | "denied" | "error";
statusText: string;
intervalSeconds?: number;
accessToken?: string;
refreshToken?: string;
tokenType?: string;
expiresIn?: number;
};
export function startGuangYaPanQRLogin() {
return request<GuangYaPanQRSession>("/drives/guangyapan/qr", { method: "POST" });
}
export function getGuangYaPanQRStatus(deviceCode: string) {
const qs = new URLSearchParams({ deviceCode });
return request<GuangYaPanQRStatus>(`/drives/guangyapan/qr/status?${qs.toString()}`);
}
/**
* toggle
*
@@ -449,6 +494,26 @@ export function regenFailedFingerprints(id: string) {
);
}
/**
* AVI/WMV H.264 MP4
*
*
*/
export function startDriveTranscode(id: string) {
return request<{ ok: boolean; accepted: boolean; message?: string }>(
`/drives/${encodeURIComponent(id)}/transcode/start`,
{ method: "POST" }
);
}
/** 手动停止某存储正在进行的转码任务。 */
export function stopDriveTranscode(id: string) {
return request<{ ok: boolean; stopped: boolean }>(
`/drives/${encodeURIComponent(id)}/transcode/stop`,
{ method: "POST" }
);
}
// ---------- Videos ----------
export type AdminVideo = {
@@ -482,7 +547,9 @@ export type AdminVideoList = {
size: number;
};
export function listVideos(params: { driveId?: string; page?: number; size?: number; keyword?: string } = {}) {
export function listVideos(
params: { driveId?: string; page?: number; size?: number; keyword?: string } = {}
) {
const qs = new URLSearchParams();
if (params.driveId) qs.set("driveId", params.driveId);
if (params.page) qs.set("page", String(params.page));
@@ -492,6 +559,50 @@ export function listVideos(params: { driveId?: string; page?: number; size?: num
return request<AdminVideoList>(`/videos${suffix}`);
}
// 后台视频管理两个标签页的计数。
export type VideoStats = {
current: number;
blacklisted: number;
};
export function getVideoStats() {
return request<VideoStats>("/videos/stats");
}
// 黑名单(被拉黑/手动删除、扫盘不再入库的视频)。原始记录已删除,
// 只剩文件名/来源盘/大小/拉黑时间。
export type AdminDeletedVideo = {
id: string;
driveId: string;
fileId: string;
fileName: string;
size: number;
deletedAt: number;
};
export type AdminBlacklistList = {
items: AdminDeletedVideo[];
total: number;
page: number;
size: number;
};
export function listBlacklist(params: { page?: number; size?: number; keyword?: string } = {}) {
const qs = new URLSearchParams();
if (params.page) qs.set("page", String(params.page));
if (params.size) qs.set("size", String(params.size));
if (params.keyword) qs.set("keyword", params.keyword);
const suffix = qs.toString() ? `?${qs.toString()}` : "";
return request<AdminBlacklistList>(`/blacklist${suffix}`);
}
// 把视频移出黑名单(删除墓碑),下次扫盘会重新入库。
export function removeBlacklist(id: string) {
return request<{ ok: boolean }>(`/blacklist/${encodeURIComponent(id)}`, {
method: "DELETE",
});
}
export type UpdateVideoInput = Partial<{
title: string;
author: string;
@@ -558,7 +669,7 @@ export function deleteTag(id: number) {
// ---------- Settings ----------
export type Theme = "dark" | "pink";
export type Theme = "dark" | "pink" | "sky";
export type Settings = {
theme: Theme;
+47 -1
View File
@@ -1,4 +1,4 @@
import { PlayCircle, Power, PowerOff, RotateCcw } from "lucide-react";
import { CircleStop, PlayCircle, Power, PowerOff, RotateCcw, Wand2 } from "lucide-react";
import * as api from "../api";
import { formatBytes } from "../storageFormat";
import {
@@ -163,20 +163,26 @@ export function DriveGenerationPanel({
regenFailedThumbId,
regenFailedFingerprintId,
togglingTeaserId,
togglingTranscodeId,
onToggleTeaser,
onRegenFailed,
onRegenFailedThumbnails,
onRegenFailedFingerprints,
onStartTranscode,
onStopTranscode,
}: {
d: api.AdminDrive;
regenFailedId: string;
regenFailedThumbId: string;
regenFailedFingerprintId: string;
togglingTeaserId: string;
togglingTranscodeId: string;
onToggleTeaser: () => void;
onRegenFailed: () => void;
onRegenFailedThumbnails: () => void;
onRegenFailedFingerprints: () => void;
onStartTranscode: () => void;
onStopTranscode: () => void;
}) {
const canQueueThumbnails =
(d.thumbnailFailedCount ?? 0) > 0 ||
@@ -186,6 +192,12 @@ export function DriveGenerationPanel({
(d.teaserFailedCount ?? 0) > 0 || (d.teaserPendingCount ?? 0) > 0;
const canQueueFingerprints =
(d.fingerprintFailedCount ?? 0) > 0 || (d.fingerprintPendingCount ?? 0) > 0;
// 转码默认不运行,只能在这里手动开启/停止。
// 候选 = 还没出结果的不兼容格式视频 + 上次失败的(重新开始会自动重试)。
const transcodeRunning =
(d.transcodeGenerationStatus?.state || "idle") !== "idle";
const canStartTranscode =
(d.transcodePendingCount ?? 0) > 0 || (d.transcodeFailedCount ?? 0) > 0;
return (
<div className="admin-detail-card">
@@ -235,6 +247,13 @@ export function DriveGenerationPanel({
pending={d.fingerprintPendingCount}
failed={d.fingerprintFailedCount}
/>
<DriveGenCol
label="转码"
status={d.transcodeGenerationStatus}
ready={d.transcodeReadyCount}
pending={d.transcodePendingCount}
failed={d.transcodeFailedCount}
/>
</div>
<div className="admin-detail-actions">
@@ -262,6 +281,33 @@ export function DriveGenerationPanel({
<RotateCcw size={13} />
<span>{(d.fingerprintFailedCount ?? 0) > 0 ? "重试失败指纹" : "继续生成指纹"}</span>
</button>
{transcodeRunning ? (
<button
className="admin-btn is-stop"
disabled={togglingTranscodeId === d.id}
onClick={onStopTranscode}
title="停止当前的转码任务。未处理的视频保持原状态,下次开始时继续。"
>
<CircleStop size={13} />
<span>{togglingTranscodeId === d.id ? "停止中..." : "停止转码"}</span>
</button>
) : (
<button
className="admin-btn"
disabled={!canStartTranscode || togglingTranscodeId === d.id}
onClick={onStartTranscode}
title="把浏览器播放不了的视频(AVI/WMV/RMVB、MPEG-4 等老格式)转码成 H.264 MP4 并上传回本存储。转码不会自动运行,只能在这里手动开启。"
>
<Wand2 size={13} />
<span>
{togglingTranscodeId === d.id
? "开启中..."
: (d.transcodeFailedCount ?? 0) > 0 && (d.transcodePendingCount ?? 0) === 0
? "重试失败转码"
: "开始转码"}
</span>
</button>
)}
</div>
</div>
);
+17
View File
@@ -2,6 +2,7 @@ import { useId, useMemo, useState } from "react";
import { ArrowLeft, ChevronDown } from "lucide-react";
import { P123QRCodeLogin } from "./P123QRCodeLogin";
import { WopanQRCodeLogin } from "./WopanQRCodeLogin";
import { GuangYaPanQRCodeLogin } from "./GuangYaPanQRCodeLogin";
import { Spider91UploadTargetField } from "./Spider91UploadTargetField";
import {
FormState,
@@ -24,6 +25,7 @@ const DRIVE_OPTIONS: DriveOption[] = [
{ kind: "p115", label: "115 网盘", abbr: "115", desc: "302直链,不占带宽" },
{ kind: "p123", label: "123网盘", abbr: "123", desc: "扫码登录,302直链" },
{ kind: "pikpak", label: "PikPak", abbr: "Pk", desc: "302直链,稳定快速" },
{ kind: "guangyapan", label: "光鸭网盘", abbr: "GY", desc: "扫码登录,302直链" },
{ kind: "onedrive", label: "OneDrive", abbr: "OD", desc: "302直链,微软网盘" },
{ kind: "googledrive", label: "Google Drive", abbr: "GD", desc: "服务器中转模式" },
{ kind: "localstorage", label: "本地存储", abbr: "Lo", desc: "本机文件目录" },
@@ -194,6 +196,21 @@ export function DriveForm({
/>
)}
{form.kind === "guangyapan" && (
<GuangYaPanQRCodeLogin
onCredentials={(credentials) =>
onChange({
...form,
creds: {
...form.creds,
access_token: credentials.accessToken,
refresh_token: credentials.refreshToken,
},
})
}
/>
)}
{fields.map((f) => (
<div key={f.key} className="admin-form__row">
{f.type === "select" ? (
+150
View File
@@ -0,0 +1,150 @@
import { useEffect, useState } from "react";
import { QrCode } from "lucide-react";
import * as api from "../api";
import { useToast } from "../ToastContext";
function guangYaPanQRStatusClass(
status: api.GuangYaPanQRStatus | null,
completed: boolean,
error: string
): string {
if (completed || status?.state === "success") return "is-ok";
if (error || status?.state === "expired" || status?.state === "denied" || status?.state === "error")
return "is-error";
return "is-pending";
}
export function GuangYaPanQRCodeLogin({
onCredentials,
}: {
onCredentials: (credentials: {
accessToken: string;
refreshToken: string;
}) => void;
}) {
const { show } = useToast();
const [session, setSession] = useState<api.GuangYaPanQRSession | null>(null);
const [status, setStatus] = useState<api.GuangYaPanQRStatus | null>(null);
const [starting, setStarting] = useState(false);
const [pollingError, setPollingError] = useState("");
const [completed, setCompleted] = useState(false);
async function start() {
setStarting(true);
setPollingError("");
setCompleted(false);
setStatus(null);
try {
const next = await api.startGuangYaPanQRLogin();
setSession(next);
} catch (e) {
setSession(null);
show(e instanceof Error ? e.message : "生成二维码失败", "error");
} finally {
setStarting(false);
}
}
useEffect(() => {
if (!session || completed) return;
const activeSession = session;
let stopped = false;
let timer: number | undefined;
let delayMs = Math.max(1000, (activeSession.intervalSeconds || 5) * 1000);
async function poll() {
if (stopped) return;
try {
const next = await api.getGuangYaPanQRStatus(activeSession.deviceCode);
if (stopped) return;
setStatus(next);
setPollingError("");
if (next.intervalSeconds && next.intervalSeconds > 0) {
delayMs = Math.max(1000, next.intervalSeconds * 1000);
}
if (next.accessToken && next.refreshToken) {
stopped = true;
if (timer) window.clearTimeout(timer);
setCompleted(true);
onCredentials({
accessToken: next.accessToken,
refreshToken: next.refreshToken,
});
show("扫码成功,已填入 access_token 和 refresh_token,保存后生效", "success");
return;
}
if (next.state === "expired" || next.state === "denied" || next.state === "error") {
stopped = true;
if (timer) window.clearTimeout(timer);
return;
}
} catch (e) {
if (stopped) return;
setPollingError(e instanceof Error ? e.message : "查询扫码状态失败");
}
if (!stopped) {
timer = window.setTimeout(poll, delayMs);
}
}
poll();
return () => {
stopped = true;
if (timer) window.clearTimeout(timer);
};
}, [session, completed, onCredentials, show]);
const statusText = completed
? "已获取凭证"
: pollingError || status?.statusText || (session ? "等待扫码" : "未生成二维码");
const statusClass = guangYaPanQRStatusClass(status, completed, pollingError);
return (
<div className="admin-form__row">
<label></label>
<div className="admin-p123-qr">
<div className="admin-p123-qr__actions">
<button
type="button"
className="admin-btn"
onClick={start}
disabled={starting}
>
<QrCode size={14} />
{starting ? "生成中..." : session ? "重新生成二维码" : "生成二维码"}
</button>
<span className={`admin-status ${statusClass}`}>{statusText}</span>
</div>
{session && (
<div className="admin-p123-qr__body">
<img
className="admin-p123-qr__image"
src={session.qrImageDataUrl}
alt="光鸭网盘扫码登录二维码"
/>
<div className="admin-p123-qr__meta">
<div className="admin-form__help">
使 App access_token refresh_token
</div>
{session.expiresAt && (
<div className="admin-form__help">
{new Date(session.expiresAt).toLocaleTimeString("zh-CN", {
hour: "2-digit",
minute: "2-digit",
second: "2-digit",
})}
</div>
)}
{(status?.state === "expired" || status?.state === "denied") && (
<div className="admin-form__help">
{status.state === "denied" ? "已被拒绝" : "已过期"}
</div>
)}
</div>
</div>
)}
</div>
</div>
);
}
+57 -10
View File
@@ -1,4 +1,4 @@
export type Kind = "quark" | "p115" | "p123" | "pikpak" | "wopan" | "onedrive" | "googledrive" | "localstorage" | "spider91";
export type Kind = "quark" | "p115" | "p123" | "pikpak" | "wopan" | "guangyapan" | "onedrive" | "googledrive" | "localstorage" | "spider91";
export const kindAbbr: Record<string, string> = {
quark: "Qk",
@@ -6,6 +6,7 @@ export const kindAbbr: Record<string, string> = {
p123: "123",
pikpak: "Pk",
wopan: "Wo",
guangyapan: "GY",
onedrive: "OD",
googledrive: "GD",
localstorage: "Lo",
@@ -28,6 +29,7 @@ export const kindLabel: Record<string, string> = {
p123: "123网盘",
pikpak: "PikPak",
wopan: "联通网盘",
guangyapan: "光鸭网盘",
onedrive: "OneDrive",
googledrive: "Google Drive",
localstorage: "本地存储",
@@ -126,6 +128,7 @@ export function formatClock(value: string): string {
export function defaultRootId(kind: Kind): string {
if (kind === "pikpak") return "";
if (kind === "guangyapan") return "";
if (kind === "onedrive") return "root";
if (kind === "googledrive") return "root";
if (kind === "localstorage") return "/";
@@ -155,6 +158,8 @@ export function credentialHelp(kind: Kind, isEdit: boolean): string {
return `填写 PikPak 账号和密码即可。平台、设备 ID、验证码 token 和 refresh token 会由服务端自动处理并保存。${note}`;
case "wopan":
return `推荐使用扫码登录自动获取 access_token 和 refresh_token;也可以手工粘贴已有凭证。${note}`;
case "guangyapan":
return `推荐使用扫码登录自动获取 access_token 和 refresh_token;也可以手工粘贴已有 token。${note}`;
case "onedrive":
return `按 OpenList 默认应用在线挂载,只需要 refresh_token;保存时会自动刷新并保存 token。${note}`;
case "googledrive":
@@ -162,7 +167,7 @@ export function credentialHelp(kind: Kind, isEdit: boolean): string {
? "请参考OpenList文档中关于谷歌云盘的配置方法;如不修改凭证,留空即可,保存时会沿用旧值"
: "请参考OpenList文档中关于谷歌云盘的配置方法";
case "localstorage":
return `填写服务器可访问的本地目录绝对路径,例如 /mnt/videos。系统会扫描该目录及子目录中的视频文件和 .strm 文件;.strm 可指向 HTTP/HTTPS 直链,或指向本地存储根目录内的真实视频路径。Docker 部署时请填写容器内路径。${note}`;
return `填写服务器可访问的本地目录绝对路径,例如 /mnt/videos。系统会扫描该目录及子目录中的视频文件和 .strm 文件;.strm 可指向 HTTP/HTTPS 直链或本地视频路径(指向目录外需开启下方开关)。Docker 部署时请填写容器内路径。${note}`;
case "spider91":
return "91Spider 不再支持通过网盘添加或编辑。请到后台爬虫管理页面添加爬虫脚本。";
default:
@@ -272,6 +277,29 @@ export function credentialFields(kind: Kind, creds: Record<string, string> = {})
placeholder: "留空走个人空间",
},
];
case "guangyapan":
return [
{
key: "root_path",
label: "根目录路径(可选)",
placeholder: "例如:影视/电影;留空使用上方根目录 ID",
help: "如果填写 root_path,服务端会按路径解析光鸭目录,并优先作为扫描根目录。",
},
{
key: "refresh_token",
label: "refresh_token",
placeholder: "推荐填写,服务端会自动刷新 access_token",
multiline: true,
help: "扫码成功后会自动填入该字段。",
},
{
key: "access_token",
label: "access_token",
placeholder: "Bearer eyJ... 或直接粘贴 token",
multiline: true,
help: "扫码成功后会自动填入该字段;如果 token 过期,重新扫码后保存即可。",
},
];
case "onedrive":
return [
{
@@ -295,15 +323,15 @@ export function credentialFields(kind: Kind, creds: Record<string, string> = {})
{ value: "false", label: "自建 Google OAuth 客户端" },
],
},
{
key: "refresh_token",
label: "refresh_token",
placeholder: "OpenList Google Drive refresh_token",
multiline: true,
required: true,
},
...(googleDriveUsesOnlineAPI(creds)
? []
? [
{
key: "api_url_address",
label: "OpenList 在线 API URL",
placeholder: "默认:https://api.oplist.org/googleui/renewapi",
help: "留空时使用 OpenList 官方在线 API,填写后会使用自定义续期 API。",
},
]
: [
{
key: "client_id",
@@ -320,6 +348,13 @@ export function credentialFields(kind: Kind, creds: Record<string, string> = {})
help: "Google Cloud Console 中同一个 OAuth 客户端的 Client Secret",
},
]),
{
key: "refresh_token",
label: "refresh_token",
placeholder: "OpenList Google Drive refresh_token",
multiline: true,
required: true,
},
];
case "localstorage":
return [
@@ -330,6 +365,18 @@ export function credentialFields(kind: Kind, creds: Record<string, string> = {})
required: true,
help: "路径必须是后端服务器上的已有目录;保存后可手动重扫,系统会递归扫描支持的视频格式。",
},
{
key: "strm_allow_outside_root",
label: ".strm 允许指向目录外",
placeholder: "",
type: "select",
defaultValue: "false",
options: [
{ value: "false", label: "关闭(默认,仅允许目录内路径)" },
{ value: "true", label: "开启(允许任意本地路径)" },
],
help: "开启后 .strm 可指向本目录之外的本地文件(如 rclone 挂载点)。注意:等于允许通过 .strm 读取服务器上任意文件,请只在自己完全掌控媒体目录时开启。Docker 部署时路径必须是容器内路径。",
},
];
case "spider91":
return [
+1 -3
View File
@@ -3,7 +3,6 @@ import { NavLink } from "react-router-dom";
import {
Film,
Menu,
Play,
Settings,
Sparkles,
Upload,
@@ -25,9 +24,8 @@ export function MainNav() {
<div className="container main-nav__inner">
<NavLink to="/" className="main-nav__logo">
<span className="main-nav__logo-mark">
<Play size={16} fill="#000" />
<img src="/icon.png" alt="" className="main-nav__logo-img" />
</span>
<span className="main-nav__logo-text">91</span>
</NavLink>
<ul className="main-nav__list" role="menubar">
+116
View File
@@ -0,0 +1,116 @@
import type { CSSProperties } from "react";
/**
*
*
* vip.215.im GIF GIF
* CSS opacity
*
*
* - public/stickers/star-*.gif dist/stickers/
* - App
* - data-theme!=="sky" CSS display: none
* - aria-hidden + pointer-events: none访
* - / / DESKTOP_STARS / MOBILE_STARS
*/
const STICKERS = [
"/stickers/star-gold.gif",
"/stickers/star-pink.gif",
"/stickers/star-sparkle.gif",
"/stickers/star-mini.gif",
];
type StarSpec = {
/** 锚点用百分号写,CSS 直接当 top/left/right/bottom 用 */
top?: string;
bottom?: string;
left?: string;
right?: string;
/** 像素,控制 GIF 渲染尺寸 */
size: number;
};
/**
*
*
*/
const DESKTOP_STARS: StarSpec[] = [
{ top: "6%", left: "5%", size: 44 },
{ top: "4%", left: "24%", size: 26 },
{ top: "8%", right: "12%", size: 48 },
{ top: "17%", right: "31%", size: 30 },
{ top: "24%", left: "8%", size: 34 },
{ top: "28%", right: "5%", size: 38 },
{ top: "43%", left: "3%", size: 24 },
{ top: "49%", right: "9%", size: 28 },
{ top: "63%", left: "11%", size: 32 },
{ top: "66%", right: "18%", size: 44 },
{ bottom: "14%", left: "5%", size: 36 },
{ bottom: "10%", right: "6%", size: 42 },
{ bottom: "4%", left: "33%", size: 24 },
{ bottom: "6%", right: "34%", size: 28 },
{ top: "13%", left: "52%", size: 22 },
{ bottom: "24%", right: "41%", size: 22 },
];
/**
*
*/
const MOBILE_STARS: StarSpec[] = [
{ top: "7%", left: "6%", size: 30 },
{ top: "11%", right: "7%", size: 28 },
{ top: "24%", right: "3%", size: 22 },
{ top: "39%", left: "4%", size: 22 },
{ top: "57%", right: "6%", size: 26 },
{ bottom: "23%", left: "9%", size: 24 },
{ bottom: "12%", right: "12%", size: 30 },
{ bottom: "5%", left: "48%", size: 20 },
];
export function SkyStarfield() {
return (
<div className="sky-starfield" aria-hidden="true">
{DESKTOP_STARS.map((s, i) => {
const style: CSSProperties = {
top: s.top,
bottom: s.bottom,
left: s.left,
right: s.right,
width: s.size,
height: s.size,
};
const src = STICKERS[i % STICKERS.length];
return (
<img
key={`desktop-${i}`}
className="sky-star sky-star--desktop"
src={src}
alt=""
style={style}
/>
);
})}
{MOBILE_STARS.map((s, i) => {
const style: CSSProperties = {
top: s.top,
bottom: s.bottom,
left: s.left,
right: s.right,
width: s.size,
height: s.size,
};
const src = STICKERS[(i + 1) % STICKERS.length];
return (
<img
key={`mobile-${i}`}
className="sky-star sky-star--mobile"
src={src}
alt=""
style={style}
/>
);
})}
</div>
);
}
+1 -4
View File
@@ -13,10 +13,7 @@ type Props = {
const sortOptions: { key: SortKey; label: string }[] = [
{ key: "latest", label: "最新" },
{ key: "hot", label: "最热" },
{ key: "week", label: "本周" },
{ key: "long", label: "最长" },
{ key: "hd", label: "高清" },
{ key: "featured", label: "精选" },
{ key: "recent", label: "最近观看" },
];
export function SortToolbar({ sort, view, onSortChange, onViewChange }: Props) {
+2 -17
View File
@@ -1,13 +1,11 @@
import { useEffect, useState } from "react";
import { EyeOff, ThumbsDown, ThumbsUp, Trash2 } from "lucide-react";
import { ThumbsDown, ThumbsUp, Trash2 } from "lucide-react";
import type { VideoDetail } from "@/types";
import { formatCount } from "@/lib/format";
type Props = {
video: VideoDetail;
onHideVideo: () => void;
onDeleteVideo: () => void;
hideSaving?: boolean;
deleteSaving?: boolean;
};
@@ -15,7 +13,7 @@ type Props = {
*
* - 线"成体"
* - +
* - "不再显示" hover danger
* - hover danger
*
*
* - state
@@ -23,9 +21,7 @@ type Props = {
*/
export function VideoActions({
video,
onHideVideo,
onDeleteVideo,
hideSaving,
deleteSaving,
}: Props) {
const [likes, setLikes] = useState(video.likes ?? 0);
@@ -119,17 +115,6 @@ export function VideoActions({
</button>
</div>
<button
type="button"
className="vd-actions__btn vd-actions__hide"
onClick={onHideVideo}
disabled={hideSaving}
aria-label="不再显示这个视频"
>
<EyeOff size={16} />
<span>{hideSaving ? "处理中" : "不再显示"}</span>
</button>
<button
type="button"
className="vd-actions__btn vd-actions__delete"
+1
View File
@@ -295,6 +295,7 @@ function sourceKindFromLabel(label: string): string {
if (value.includes("123") || value.includes("p123")) return "p123";
if (value.includes("pikpak")) return "pikpak";
if (value.includes("沃盘") || value.includes("wopan") || value.includes("联通")) return "wopan";
if (value.includes("光鸭") || value.includes("guangyapan") || value.includes("guangya")) return "guangyapan";
if (value.includes("onedrive") || value.includes("one drive")) return "onedrive";
if (value.includes("本地") || value.includes("localstorage") || value.includes("local storage")) return "localstorage";
return "";
+2
View File
@@ -74,6 +74,8 @@ function sourceKindFromLabel(label: string): string {
if (value.includes("pikpak")) return "pikpak";
if (value.includes("沃盘") || value.includes("wopan") || value.includes("联通"))
return "wopan";
if (value.includes("光鸭") || value.includes("guangyapan") || value.includes("guangya"))
return "guangyapan";
if (value.includes("onedrive") || value.includes("one drive")) return "onedrive";
if (value.includes("本地") || value.includes("localstorage") || value.includes("local storage"))
return "localstorage";
+60 -89
View File
@@ -5,7 +5,7 @@ import {
type CSSProperties,
type MutableRefObject,
} from "react";
import Artplayer, { type Option } from "artplayer";
import Artplayer, { type Option, type SettingOption } from "artplayer";
import type Hls from "hls.js";
type Props = {
@@ -92,16 +92,28 @@ const LONG_PRESS_MS = 400;
const FAST_RATE = 2;
/** 默认倍速。 */
const NORMAL_RATE = 1;
/** ArtPlayer 内部播放失败自动重连次数。 */
const ARTPLAYER_RECONNECT_TIME_MAX = 3;
Artplayer.FAST_FORWARD_VALUE = FAST_RATE;
Artplayer.RECONNECT_TIME_MAX = ARTPLAYER_RECONNECT_TIME_MAX;
const SETTINGS_KEY = "video-site:player-settings";
const DEFAULT_SETTINGS: PlayerSettings = {
volume: 0.7,
muted: false,
playbackRate: 1,
brightness: 1,
};
const DEFAULT_SETTING_LAYOUT = {
width: Artplayer.SETTING_WIDTH,
itemWidth: Artplayer.SETTING_ITEM_WIDTH,
itemHeight: Artplayer.SETTING_ITEM_HEIGHT,
};
const COMPACT_SETTING_LAYOUT = {
width: 172,
itemWidth: 148,
itemHeight: 30,
};
const ORIENTATION_CONTROL_NAME = "orientationToggle";
const MANUAL_ORIENTATION_CLASS = "art-manual-orientation";
const FAST_RATE_CLASS = "art-fast-rate-active";
@@ -320,10 +332,12 @@ function mountArtPlayer({
onGestureHud: (label: string, duration?: number) => void;
}) {
const sourceType = inferSourceType(src);
const settings = readPlayerSettings();
const fastActiveRef = { current: false };
const loadHlsSource = createHlsSourceLoader(onError);
const enableOrientationControl = shouldEnableMobileOrientationControl();
configureArtPlayerSettingLayout(
shouldUseCompactPlayerSettings(mount, enableOrientationControl)
);
const option: Option = {
id: "91-detail-player",
container: mount,
@@ -331,8 +345,8 @@ function mountArtPlayer({
poster,
theme: "var(--video-player-progress)",
lang: "zh-cn",
volume: settings.volume,
muted: settings.muted,
volume: DEFAULT_SETTINGS.volume,
muted: DEFAULT_SETTINGS.muted,
autoplay: false,
autoSize: false,
playbackRate: true,
@@ -358,6 +372,7 @@ function mountArtPlayer({
preload: "metadata",
playsInline: true,
},
settings: [createLoopSetting()],
controls: enableOrientationControl ? [createOrientationControl()] : [],
contextmenu: [],
cssVar: {
@@ -377,8 +392,9 @@ function mountArtPlayer({
video.setAttribute("controlsList", "nodownload");
video.setAttribute("webkit-playsinline", "true");
video.disablePictureInPicture = false;
video.playbackRate = settings.playbackRate;
applyPlayerBrightness(art, settings.brightness);
video.loop = false;
video.playbackRate = DEFAULT_SETTINGS.playbackRate;
applyPlayerBrightness(art, DEFAULT_SETTINGS.brightness);
art.url = src;
function preventContextMenu(event: Event) {
@@ -414,21 +430,6 @@ function mountArtPlayer({
onFastChange(false);
}
function handleVolumeChange() {
writePlayerSettings({
volume: clamp(video.volume, 0, 1),
muted: video.muted,
});
}
function handleRateChange() {
if (fastActiveRef.current) return;
if (!Number.isFinite(video.playbackRate)) return;
writePlayerSettings({
playbackRate: clamp(video.playbackRate, 0.5, 3),
});
}
const handleFastChange = (active: boolean) => {
fastActiveRef.current = active;
setPlayerFastRateHint(art, active);
@@ -453,8 +454,6 @@ function mountArtPlayer({
: noop;
mount.addEventListener("contextmenu", preventContextMenu);
video.addEventListener("volumechange", handleVolumeChange);
video.addEventListener("ratechange", handleRateChange);
art.on("video:loadstart", handleLoadStart);
art.on("video:loadeddata", handleReady);
@@ -473,8 +472,6 @@ function mountArtPlayer({
unbindOrientationToggle();
setPlayerFastRateHint(art, false);
mount.removeEventListener("contextmenu", preventContextMenu);
video.removeEventListener("volumechange", handleVolumeChange);
video.removeEventListener("ratechange", handleRateChange);
destroyHls(video);
art.off("video:loadstart", handleLoadStart);
art.off("video:loadeddata", handleReady);
@@ -502,10 +499,42 @@ function shouldEnableMobileOrientationControl() {
return /Android|iPhone|iPad|iPod|Mobile/i.test(navigator.userAgent);
}
function shouldUseCompactPlayerSettings(
mount: HTMLElement,
mobileControls: boolean
) {
const narrowViewport =
window.matchMedia?.("(max-width: 640px)").matches ??
window.innerWidth <= 640;
return mobileControls || narrowViewport || mount.clientWidth <= 640;
}
function configureArtPlayerSettingLayout(compact: boolean) {
const layout = compact ? COMPACT_SETTING_LAYOUT : DEFAULT_SETTING_LAYOUT;
Artplayer.SETTING_WIDTH = layout.width;
Artplayer.SETTING_ITEM_WIDTH = layout.itemWidth;
Artplayer.SETTING_ITEM_HEIGHT = layout.itemHeight;
}
function shouldEnableMobileGestures() {
return shouldEnableMobileOrientationControl();
}
function createLoopSetting() {
return {
name: "mind-loop",
html: "洗脑循环",
tooltip: "关",
switch: false,
onSwitch(this: Artplayer, item: SettingOption) {
const next = !item.switch;
this.video.loop = next;
item.tooltip = next ? "开" : "关";
return next;
},
};
}
function isPlayerExpanded(art: Artplayer) {
return Boolean(
art.fullscreen || art.fullscreenWeb || getNativeFullscreenElement()
@@ -912,12 +941,10 @@ function getPlayerBrightness(art: Artplayer) {
"--video-player-brightness"
);
if (!raw.trim()) return DEFAULT_SETTINGS.brightness;
return clampNumber(
Number(raw),
DEFAULT_SETTINGS.brightness,
BRIGHTNESS_MIN,
BRIGHTNESS_MAX
);
const value = Number(raw);
return Number.isFinite(value)
? clamp(value, BRIGHTNESS_MIN, BRIGHTNESS_MAX)
: DEFAULT_SETTINGS.brightness;
}
function mobileGestureSeekSpan(duration: number) {
@@ -959,7 +986,7 @@ function createHlsSourceLoader(
destroyHls(target);
onError(null);
void import("hls.js")
void import("hls.js/light")
.then((hlsModule) => {
if (art.isDestroy || !video.isConnected) return;
loadHlsSourceWith(video, url, art, hlsModule.default, onError);
@@ -1321,15 +1348,6 @@ function bindMobilePlayerGestures(
);
}
}
} else if (state.mode === "brightness") {
writePlayerSettings({
brightness: getPlayerBrightness(art),
});
} else if (state.mode === "volume") {
writePlayerSettings({
volume: clamp(video.volume, 0, 1),
muted: video.muted,
});
}
resetGesture();
@@ -1401,25 +1419,6 @@ function bindProgressPreview(
};
}
function readPlayerSettings(): PlayerSettings {
const saved = safeGetJSON<Partial<PlayerSettings>>(SETTINGS_KEY) ?? {};
return {
volume: clampNumber(saved.volume, DEFAULT_SETTINGS.volume, 0, 1),
muted: typeof saved.muted === "boolean" ? saved.muted : DEFAULT_SETTINGS.muted,
playbackRate: clampNumber(saved.playbackRate, DEFAULT_SETTINGS.playbackRate, 0.5, 3),
brightness: clampNumber(
saved.brightness,
DEFAULT_SETTINGS.brightness,
BRIGHTNESS_MIN,
BRIGHTNESS_MAX
),
};
}
function writePlayerSettings(patch: Partial<PlayerSettings>) {
safeSetJSON(SETTINGS_KEY, { ...readPlayerSettings(), ...patch });
}
function mediaErrorMessage(error: MediaError | null) {
switch (error?.code) {
case MediaError.MEDIA_ERR_ABORTED:
@@ -1464,34 +1463,6 @@ function fallbackCopyText(text: string) {
}
}
function safeGetJSON<T>(key: string): T | null {
try {
const raw = localStorage.getItem(key);
return raw ? (JSON.parse(raw) as T) : null;
} catch {
return null;
}
}
function safeSetJSON(key: string, value: unknown) {
try {
localStorage.setItem(key, JSON.stringify(value));
} catch {
// ignore
}
}
function clampNumber(
value: unknown,
fallback: number,
min: number,
max: number
) {
return typeof value === "number" && Number.isFinite(value)
? clamp(value, min, max)
: fallback;
}
function clamp(n: number, min: number, max: number) {
return n < min ? min : n > max ? max : n;
}
+3 -5
View File
@@ -132,19 +132,17 @@ export type ShortsNextResponse = {
/**
* video id
* count preferredFromVideoId
*
* count
*
* + roundComplete=false
*/
export function fetchShortsNext(
seenIds: string[],
count: number,
preferredFromVideoId?: string
count: number
): Promise<ShortsNextResponse> {
return apiJSON<ShortsNextResponse>("/api/shorts/next", {
method: "POST",
body: JSON.stringify({ seenIds, count, preferredFromVideoId }),
body: JSON.stringify({ seenIds, count }),
}).catch(() => ({ items: [], total: 0, roundComplete: false }));
}
+4
View File
@@ -0,0 +1,4 @@
declare module "hls.js/light" {
export { default } from "hls.js";
export * from "hls.js";
}
+3 -3
View File
@@ -10,13 +10,13 @@
// 公开端点 /api/settings/theme 不需要登录,原因见 backend/internal/api/api.go 中
// 的注释——登录页本身就要在用户登录之前正确显示主题。
export type Theme = "dark" | "pink";
export type Theme = "dark" | "pink" | "sky";
export const THEMES: Theme[] = ["dark", "pink"];
export const THEMES: Theme[] = ["dark", "pink", "sky"];
const STORAGE_KEY = "video-site:theme";
function isTheme(value: unknown): value is Theme {
return value === "dark" || value === "pink";
return value === "dark" || value === "pink" || value === "sky";
}
/**
+1 -4
View File
@@ -226,9 +226,6 @@ function isSortKey(value: unknown): value is SortKey {
return (
value === "latest" ||
value === "hot" ||
value === "week" ||
value === "long" ||
value === "hd" ||
value === "featured"
value === "recent"
);
}
+533 -133
View File
@@ -7,11 +7,8 @@ import {
Minimize,
Volume2,
VolumeX,
Play,
Pause,
EyeOff,
Info,
Loader2,
Sparkles,
AlertCircle,
} from "lucide-react";
@@ -32,9 +29,21 @@ const BATCH_SIZE = 5;
// 当队列里"还没看过的视频"少于这个数时,提前请求下一批。
const PREFETCH_THRESHOLD = 2;
// 距离 activeIndex 多少屏内的视频会被 mount 真实 <video>
// =1 表示上一屏 / 当前 / 下一屏 都加载,这样切换时几乎无空白。
const MOUNT_RADIUS = 1;
// 当前视频至少有这么多秒的前向缓冲后,才允许后续视频开始预加载
const ACTIVE_PRELOAD_BUFFER_SECONDS = 12;
// 当前视频流畅播放后,向后预加载多少条视频。
const PRELOAD_AHEAD_COUNT = 2;
// 预加载授权一旦发出,只有当前视频前向缓冲跌破这个秒数(或发生 stall)
// 才收回。高低水位之间不动作,避免缓冲量在 12s 附近波动时
// 反复绑定/剥离后续视频的 src、丢弃已预加载的数据。
const ACTIVE_PRELOAD_KEEP_SECONDS = 4;
// 维护一个固定大小的视频窗口:窗口内才 mount 真实 <video> 壳。
// 当前屏先绑定 src;后续预加载要等当前屏缓冲健康后才开始。
// 窗口内只要已经产生过可复用缓冲,就保留 src 复用浏览器缓存。
const VIDEO_WINDOW_SIZE = 6;
function loadSeenIds(): string[] {
try {
@@ -77,16 +86,34 @@ export default function ShortsPage() {
}, 1500);
}, []);
const stopHeaderControlPropagation = useCallback((e: React.SyntheticEvent) => {
e.stopPropagation();
}, []);
const handleVolumeButtonClick = useCallback(() => {
const activeVideo = videoRefs.current.get(activeIndex);
const canResumeActiveVideo = () =>
Boolean(activeVideo) &&
videoRefs.current.get(activeIndexRef.current) === activeVideo &&
userPausedIndexRef.current !== activeIndexRef.current;
const wasPlaying = Boolean(activeVideo) && canResumeActiveVideo() && !activeVideo?.paused;
setMuted((v) => {
const next = !v;
if (activeVideo) {
normalizeVideoPlaybackRate(activeVideo);
applyVideoAudioState(activeVideo, next, volume);
stabilizeVideoAfterAudioToggle(
activeVideo,
() => wasPlaying && canResumeActiveVideo()
);
}
showHud(
next ? "已静音" : "音量已开启",
next ? <VolumeX size={16} /> : <Volume2 size={16} />
);
return next;
});
}, [showHud]);
}, [activeIndex, showHud, volume]);
const handleVolumeSliderChange = useCallback((e: React.ChangeEvent<HTMLInputElement>) => {
const val = parseFloat(e.target.value);
@@ -99,8 +126,19 @@ export default function ShortsPage() {
// Update active video volume directly
const activeVideo = videoRefs.current.get(activeIndex);
if (activeVideo) {
activeVideo.volume = val;
activeVideo.muted = val === 0;
normalizeVideoPlaybackRate(activeVideo);
applyVideoAudioState(activeVideo, val === 0, val);
const wasPlaying =
videoRefs.current.get(activeIndexRef.current) === activeVideo &&
userPausedIndexRef.current !== activeIndexRef.current &&
!activeVideo.paused;
stabilizeVideoAfterAudioToggle(
activeVideo,
() =>
wasPlaying &&
videoRefs.current.get(activeIndexRef.current) === activeVideo &&
userPausedIndexRef.current !== activeIndexRef.current
);
}
}, [activeIndex]);
@@ -120,7 +158,6 @@ export default function ShortsPage() {
// seenIds 用 ref 维护,方便在异步 callback 里读到最新值
const seenIdsRef = useRef<string[]>(loadSeenIds());
const preferredFromVideoIdRef = useRef<string | null>(null);
const containerRef = useRef<HTMLDivElement | null>(null);
// 整个页面根元素,用于 requestFullscreen
@@ -128,15 +165,24 @@ export default function ShortsPage() {
// index → video element,用来精确控制播放/暂停
const videoRefs = useRef<Map<number, HTMLVideoElement>>(new Map());
const activeIndexRef = useRef(0);
const userPausedIndexRef = useRef<number | null>(null);
const ignoreIntersectionUntilRef = useRef(0);
const fullscreenRestoreTimersRef = useRef<number[]>([]);
const fullscreenPointerHandledRef = useRef(false);
const [activeReadyForPreload, setActiveReadyForPreload] = useState(false);
const [userPausedIndex, setUserPausedIndexState] = useState<number | null>(null);
const [cacheableSourceIds, setCacheableSourceIds] = useState<Set<string>>(
() => new Set()
);
const [cacheWindowHighIndex, setCacheWindowHighIndex] = useState(-1);
// 当前是否处在浏览器全屏(Fullscreen API)状态。
// iOS Safari 不支持元素级 Fullscreen API,这里会一直保持 false
// 全屏按钮在那种环境下点了也无效(按钮仍展示"进入全屏"图标)。
// iPhone Safari 不支持网页元素级全屏;那种环境下改用页面滚动让浏览器栏随刷动收起。
const useDocumentScroll = shouldUseDocumentScrollForShorts();
const [canRequestFullscreen, setCanRequestFullscreen] = useState(() =>
supportsElementFullscreenAPI()
);
const [isFullscreen, setIsFullscreen] = useState(false);
// 自动尝试进入全屏只做一次,避免反复打扰用户
const autoFullscreenAttemptedRef = useRef(false);
// 本次会话内已经点过赞的视频 id 集合。
// 与后端的真实 likes 字段同步——后端是单纯计数器,前端在这里防重避免连发。
@@ -147,6 +193,61 @@ export default function ShortsPage() {
activeIndexRef.current = activeIndex;
}, [activeIndex]);
useEffect(() => {
const page = pageRef.current;
if (page && supportsElementFullscreenAPI(page)) {
setCanRequestFullscreen(true);
}
}, []);
const updateUserPausedIndex = useCallback((index: number | null) => {
userPausedIndexRef.current = index;
setUserPausedIndexState(index);
}, []);
const setUserPausedForIndex = useCallback(
(index: number, isPaused: boolean) => {
if (isPaused) {
updateUserPausedIndex(index);
} else if (userPausedIndexRef.current === index) {
updateUserPausedIndex(null);
}
},
[updateUserPausedIndex]
);
const isVideoPausedByUser = useCallback(
(index: number) => userPausedIndexRef.current === index,
[]
);
useEffect(() => {
updateUserPausedIndex(null);
}, [activeIndex, updateUserPausedIndex]);
const handleActiveReadyForPreload = useCallback((index: number) => {
if (index === activeIndexRef.current) {
setActiveReadyForPreload(true);
}
}, []);
const handleActiveNeedsPriority = useCallback((index: number) => {
if (index === activeIndexRef.current) {
setActiveReadyForPreload(false);
}
}, []);
// 标记某条视频"浏览器里已有可复用的缓冲"。之后只要它还在缓存窗口内,
// 就保留 src 不剥离,回滑/再前滑时直接续用已缓冲数据,秒开不卡顿。
const handleSourceCached = useCallback((videoId: string) => {
setCacheableSourceIds((prev) => {
if (prev.has(videoId)) return prev;
const next = new Set(prev);
next.add(videoId);
return next;
});
}, []);
/**
*
* - liked=true POST /api/video/:id/like
@@ -171,11 +272,6 @@ export default function ShortsPage() {
);
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const data = (await res.json()) as { likes?: number };
if (liked) {
preferredFromVideoIdRef.current = videoId;
} else if (preferredFromVideoIdRef.current === videoId) {
preferredFromVideoIdRef.current = null;
}
return typeof data.likes === "number" ? data.likes : null;
} catch {
// 请求失败:回滚集合,让 Slide 自己回滚 UI
@@ -204,11 +300,7 @@ export default function ShortsPage() {
setLoading(true);
try {
const seen = seenIdsRef.current;
const resp = await fetchShortsNext(
seen,
BATCH_SIZE,
preferredFromVideoIdRef.current ?? undefined
);
const resp = await fetchShortsNext(seen, BATCH_SIZE);
if (resp.items.length === 0) {
setEmpty((prev) => prev || true /* 维持 true 即可 */);
setRoundComplete(true);
@@ -242,6 +334,8 @@ export default function ShortsPage() {
const active = items[activeIndex];
if (!active) return;
setCacheWindowHighIndex((prev) => Math.max(prev, activeIndex));
if (!seenIdsRef.current.includes(active.id)) {
seenIdsRef.current = [...seenIdsRef.current, active.id];
saveSeenIds(seenIdsRef.current);
@@ -250,8 +344,10 @@ export default function ShortsPage() {
const remaining = items.length - 1 - activeIndex;
if (remaining < PREFETCH_THRESHOLD && !loading) {
if (roundComplete) {
// 上一次后端说"本轮已耗尽",且当前已经看到队列接近末尾。
// 清空 localStorage 后再请求即可开新一轮。
// 上一次后端说"本轮已耗尽"时,必须等用户真正滑到当前队列最后一条
// 清空已看记录开新一轮。否则退出后重新进入会把未完成轮次提前重置,
// 导致刚刷过的视频再次出现在下一次会话里。
if (remaining > 0) return;
seenIdsRef.current = [];
saveSeenIds([]);
setRoundComplete(false);
@@ -260,7 +356,8 @@ export default function ShortsPage() {
}
}, [activeIndex, items, loading, roundComplete, loadMore]);
// 用 IntersectionObserver 找出当前进入视口的 item
// 用 IntersectionObserver 找出当前进入视口的 item
// root 直接用 viewport:普通模式和 iPhone 页面滚动模式都能正确观测。
useEffect(() => {
const root = containerRef.current;
if (!root) return;
@@ -280,10 +377,14 @@ export default function ShortsPage() {
if (!Number.isNaN(idx)) bestIndex = idx;
}
}
if (bestIndex >= 0) setActiveIndex(bestIndex);
if (bestIndex >= 0 && bestIndex !== activeIndexRef.current) {
activeIndexRef.current = bestIndex;
setActiveReadyForPreload(false);
setActiveIndex(bestIndex);
}
},
{
root,
root: null,
threshold: [0.6, 0.85],
}
);
@@ -293,31 +394,28 @@ export default function ShortsPage() {
return () => observer.disconnect();
}, [items.length]);
// 控制每个 video 的播放状态与音量:只有 activeIndex 对应的在播
// 控制每个 video 的播放状态:只有 activeIndex 对应的在播
// 声音切换不要进入这里,否则移动端切换 muted 时可能额外触发 play/pause。
useEffect(() => {
videoRefs.current.forEach((video, idx) => {
if (idx === activeIndex) {
video.muted = muted;
video.volume = volume;
if (video.paused) {
// 切到这个视频时从头开始播
try {
video.currentTime = 0;
} catch {
// ignore
}
if (userPausedIndex === idx) {
if (!video.paused) video.pause();
} else if (video.paused) {
video.play().catch(() => undefined);
}
} else {
if (!video.paused) video.pause();
try {
video.currentTime = 0;
} catch {
// ignore
}
}
});
}, [activeIndex, muted, volume, items.length]);
}, [activeIndex, items.length, userPausedIndex]);
// 单独同步音频属性。这里不做 play/pause,避免手机端切换静音时打断播放节奏。
useEffect(() => {
videoRefs.current.forEach((video) => {
applyVideoAudioState(video, muted, volume);
});
}, [muted, volume, items.length]);
// 键盘快捷键监听
useEffect(() => {
@@ -349,12 +447,15 @@ export default function ShortsPage() {
e.preventDefault();
const activeVideo = videoRefs.current.get(activeIndex);
if (activeVideo) {
if (activeVideo.paused) {
const shouldResume =
userPausedIndexRef.current === activeIndex ||
(activeVideo.paused && activeVideo.readyState >= 3);
if (shouldResume) {
setUserPausedForIndex(activeIndex, false);
activeVideo.play().catch(() => undefined);
showHud("播放", <Play size={16} fill="currentColor" />);
} else {
setUserPausedForIndex(activeIndex, true);
activeVideo.pause();
showHud("暂停", <Pause size={16} fill="currentColor" />);
}
}
} else if (e.key === "m" || e.key === "M") {
@@ -390,7 +491,7 @@ export default function ShortsPage() {
window.addEventListener("keydown", handleKeyDown);
return () => window.removeEventListener("keydown", handleKeyDown);
}, [activeIndex, items, toggleFullscreen, showHud, handleVolumeButtonClick]);
}, [activeIndex, items, toggleFullscreen, showHud, handleVolumeButtonClick, setUserPausedForIndex]);
// 页面卸载时暂停所有
useEffect(() => {
@@ -417,16 +518,21 @@ export default function ShortsPage() {
document.title = "短视频 · 91";
}, []);
// 沉浸式:进入页面后锁住 body 滚动 + 把主题色改黑(Android Chrome 状态栏会变黑)
// 沉浸式:默认锁住 body 滚动;iPhone 浏览器里放开根页面滚动,让 Safari 工具栏能随刷动收起。
useEffect(() => {
const html = document.documentElement;
const body = document.body;
const prevHtmlOverflow = html.style.overflow;
const prevBodyOverflow = body.style.overflow;
const prevBodyBg = body.style.background;
html.style.overflow = "hidden";
body.style.overflow = "hidden";
body.style.background = "#000";
if (useDocumentScroll) {
html.classList.add("shorts-document-scroll");
body.classList.add("shorts-document-scroll");
} else {
html.style.overflow = "hidden";
body.style.overflow = "hidden";
body.style.background = "#000";
}
let prevThemeColor: string | null = null;
let themeMeta = document.querySelector<HTMLMetaElement>(
@@ -443,6 +549,8 @@ export default function ShortsPage() {
themeMeta.content = "#000000";
return () => {
html.classList.remove("shorts-document-scroll");
body.classList.remove("shorts-document-scroll");
html.style.overflow = prevHtmlOverflow;
body.style.overflow = prevBodyOverflow;
body.style.background = prevBodyBg;
@@ -454,7 +562,7 @@ export default function ShortsPage() {
}
}
};
}, []);
}, [useDocumentScroll]);
function clearFullscreenRestoreTimers() {
for (const timer of fullscreenRestoreTimersRef.current) {
@@ -516,29 +624,8 @@ export default function ShortsPage() {
};
}, []);
// 进入页面后第一次任意触摸时尝试自动进入全屏。
// 浏览器要求 requestFullscreen 必须在用户手势内调用;进页面时直接调
// 一定会被拒绝,所以挂在 pointerdown 上利用第一次手势。
// iOS Safari 不支持元素级 Fullscreen API,这里 catch 后保持原样,
// 退化为已经做的 100svh 沉浸样式。
useEffect(() => {
const page = pageRef.current;
if (!page) return;
function onFirstPointer() {
if (autoFullscreenAttemptedRef.current) return;
autoFullscreenAttemptedRef.current = true;
requestPageFullscreen();
}
page.addEventListener("pointerdown", onFirstPointer, {
once: true,
passive: true,
});
return () => {
page.removeEventListener("pointerdown", onFirstPointer);
};
}, []);
function requestPageFullscreen() {
if (!canRequestFullscreen) return;
const page = pageRef.current;
if (!page) return;
// eslint-disable-next-line @typescript-eslint/no-explicit-any
@@ -577,8 +664,33 @@ export default function ShortsPage() {
function toggleFullscreen() {
scheduleFullscreenActiveRestore();
if (isFullscreen) exitPageFullscreen();
else requestPageFullscreen();
if (canRequestFullscreen) {
if (isFullscreen) exitPageFullscreen();
else requestPageFullscreen();
return;
}
if (useDocumentScroll) {
restoreActiveSlideIntoView();
}
}
function handleFullscreenButtonPointerDown(
e: React.PointerEvent<HTMLButtonElement>
) {
e.preventDefault();
e.stopPropagation();
fullscreenPointerHandledRef.current = true;
toggleFullscreen();
}
function handleFullscreenButtonClick(e: React.MouseEvent<HTMLButtonElement>) {
e.preventDefault();
e.stopPropagation();
if (fullscreenPointerHandledRef.current) {
fullscreenPointerHandledRef.current = false;
return;
}
toggleFullscreen();
}
const handleHideSuccess = useCallback((idx: number) => {
@@ -594,8 +706,13 @@ export default function ShortsPage() {
}
}, [items.length, showHud]);
const videoWindow = getVideoWindowBounds(cacheWindowHighIndex, items.length);
return (
<div className="shorts-page" ref={pageRef}>
<div
className={`shorts-page${useDocumentScroll ? " is-document-scroll" : ""}`}
ref={pageRef}
>
<header className="shorts-header">
<Link to="/" className="shorts-header__back" aria-label="返回首页">
<ChevronLeft size={22} />
@@ -605,7 +722,8 @@ export default function ShortsPage() {
type="button"
className="shorts-header__icon-btn"
aria-label={isFullscreen ? "退出全屏" : "进入全屏"}
onClick={toggleFullscreen}
onPointerDown={handleFullscreenButtonPointerDown}
onClick={handleFullscreenButtonClick}
>
{isFullscreen ? <Minimize size={20} /> : <Maximize size={20} />}
</button>
@@ -627,7 +745,16 @@ export default function ShortsPage() {
type="button"
className="shorts-header__icon-btn"
aria-label={muted ? "取消静音" : "静音"}
onClick={handleVolumeButtonClick}
onPointerDownCapture={stopHeaderControlPropagation}
onTouchStartCapture={stopHeaderControlPropagation}
onMouseDownCapture={stopHeaderControlPropagation}
onPointerDown={stopHeaderControlPropagation}
onTouchStart={stopHeaderControlPropagation}
onMouseDown={stopHeaderControlPropagation}
onClick={(e) => {
e.stopPropagation();
handleVolumeButtonClick();
}}
>
{muted || volume === 0 ? <VolumeX size={20} /> : <Volume2 size={20} />}
</button>
@@ -652,33 +779,51 @@ export default function ShortsPage() {
</div>
)}
{items.map((item, index) => (
<ShortsSlide
key={item.id}
item={item}
index={index}
isActive={index === activeIndex}
// 距离 active 在 MOUNT_RADIUS 之内才挂载真正的 <video>
// 其它槽位用海报占位以节省内存和带宽
shouldMount={Math.abs(index - activeIndex) <= MOUNT_RADIUS}
muted={muted}
volume={volume}
setMuted={setMuted}
setVolume={setVolume}
videoRef={setVideoRef(index)}
onLikeToggle={handleLikeToggle}
hasLiked={hasLiked}
onHideSuccess={handleHideSuccess}
showHud={showHud}
/>
))}
{!empty && items.length > 0 && loading && (
<div className="shorts-loading">
<Loader2 size={16} className="shorts-slide__buffering-icon" />
<span></span>
</div>
)}
{items.map((item, index) => {
const isActiveSlide = index === activeIndex;
const isInCacheWindow =
index >= videoWindow.start && index <= videoWindow.end;
const preloadOffset = index - activeIndex;
const shouldPreload =
activeReadyForPreload &&
preloadOffset > 0 &&
preloadOffset <= PRELOAD_AHEAD_COUNT;
const shouldMount = isActiveSlide || isInCacheWindow || shouldPreload;
// 视频窗口内已经缓冲过的视频保留 src:
// 在窗口内来回切换时,直接复用浏览器已缓冲数据。
const shouldRetainCached =
isInCacheWindow && !isActiveSlide && cacheableSourceIds.has(item.id);
const shouldLoad = isActiveSlide || shouldPreload || shouldRetainCached;
const shouldEagerLoad = isActiveSlide || shouldPreload;
return (
<ShortsSlide
key={item.id}
item={item}
index={index}
isActive={isActiveSlide}
// 固定 6 条视频窗口内才挂载 <video> 壳;
// 当前屏先绑定 src;后两个视频等当前屏缓冲健康后再预加载;
// 已缓冲过的窗口内视频保留 src,便于来回切换复用缓存。
shouldMount={shouldMount}
shouldLoad={shouldLoad}
shouldEagerLoad={shouldEagerLoad}
muted={muted}
volume={volume}
setMuted={setMuted}
setVolume={setVolume}
videoRef={setVideoRef(index)}
onLikeToggle={handleLikeToggle}
hasLiked={hasLiked}
onHideSuccess={handleHideSuccess}
onActiveReadyForPreload={handleActiveReadyForPreload}
onActiveNeedsPriority={handleActiveNeedsPriority}
onSourceCached={handleSourceCached}
onUserPausedChange={setUserPausedForIndex}
isVideoPausedByUser={isVideoPausedByUser}
showHud={showHud}
/>
);
})}
</div>
</div>
);
@@ -689,6 +834,8 @@ type SlideProps = {
index: number;
isActive: boolean;
shouldMount: boolean;
shouldLoad: boolean;
shouldEagerLoad: boolean;
muted: boolean;
volume: number;
setMuted: (muted: boolean) => void;
@@ -702,6 +849,12 @@ type SlideProps = {
/** 父组件查询某 id 是否已经在本次会话内点过赞 */
hasLiked: (videoId: string) => boolean;
onHideSuccess: (index: number) => void;
onActiveReadyForPreload: (index: number) => void;
onActiveNeedsPriority: (index: number) => void;
/** 本条视频在浏览器里已有可复用缓冲,之后在视频窗口内保留 src */
onSourceCached: (videoId: string) => void;
onUserPausedChange: (index: number, isPaused: boolean) => void;
isVideoPausedByUser: (index: number) => boolean;
showHud: (text: string, icon?: React.ReactNode) => void;
};
@@ -717,6 +870,8 @@ function ShortsSlide({
index,
isActive,
shouldMount,
shouldLoad,
shouldEagerLoad,
muted,
volume,
setMuted,
@@ -725,6 +880,11 @@ function ShortsSlide({
onLikeToggle,
hasLiked,
onHideSuccess,
onActiveReadyForPreload,
onActiveNeedsPriority,
onSourceCached,
onUserPausedChange,
isVideoPausedByUser,
showHud,
}: SlideProps) {
const localRef = useRef<HTMLVideoElement | null>(null);
@@ -733,8 +893,6 @@ function ShortsSlide({
// 视频缓冲状态
const [isBuffering, setIsBuffering] = useState(false);
// 单击播放暂停的瞬间 HUD 动效
const [playPauseHud, setPlayPauseHud] = useState<{ id: number; type: "play" | "pause" } | null>(null);
// 是否已经被隐藏/拉黑
const [isMarkedHidden, setIsMarkedHidden] = useState(false);
@@ -778,6 +936,23 @@ function ShortsSlide({
[videoRef]
);
// 非当前屏/后续预加载/视频窗口内缓存视频不保留媒体源,确保离开窗口后浏览器中止原始网盘流。
useEffect(() => {
if (shouldLoad) return;
const video = localRef.current;
if (!video) return;
try {
video.pause();
video.removeAttribute("src");
video.load();
} catch {
// ignore
}
setDuration(0);
setCurrentTime(0);
setIsBuffering(false);
}, [shouldLoad, item.id]);
// 离开活跃后清掉本地的暂停状态,避免回来时 UI 还显示着 paused
useEffect(() => {
if (!isActive) {
@@ -785,7 +960,6 @@ function ShortsSlide({
setScrubbing(false);
scrubbingRef.current = false;
setIsBuffering(false);
setPlayPauseHud(null);
}
}, [isActive]);
@@ -793,8 +967,7 @@ function ShortsSlide({
useEffect(() => {
const video = localRef.current;
if (video && isActive) {
video.muted = muted;
video.volume = volume;
applyVideoAudioState(video, muted, volume);
}
}, [muted, volume, isActive]);
@@ -810,7 +983,8 @@ function ShortsSlide({
}, [isMarkedHidden]);
// 监听 video 的时长 / 进度 / 缓冲状态 / 音量物理键变化。
// MOUNT_RADIUS 会让第三屏以后的 slide 先以海报占位,之后才挂载 video;
// VIDEO_WINDOW_SIZE 会让窗口外的 slide 先以海报占位,之后才挂载 video
// 只有 shouldLoad=true 的当前屏/后续预加载/缓存窗口视频会绑定 src,因此不会一次拉完整队列。
// 因此这里必须跟随 shouldMount 重新绑定,否则后续视频没有 timeupdate 事件。
useEffect(() => {
if (!shouldMount) {
@@ -832,14 +1006,38 @@ function ShortsSlide({
const handleTime = () => {
// 拖动期间不要被 timeupdate 覆盖 UI
if (!scrubbingRef.current) setCurrentTime(video.currentTime);
syncActivePreloadReadiness(video);
};
const handleWaiting = () => {
if (video.paused || isVideoPausedByUser(index)) {
setIsBuffering(false);
return;
}
setIsBuffering(true);
if (isActive) onActiveNeedsPriority(index);
};
const handlePlayingOrCanPlay = () => {
// 已经能解码播放,说明浏览器里有了值得复用的数据。
if (shouldLoad) onSourceCached(item.id);
if (isActive && isVideoPausedByUser(index)) {
video.pause();
setPaused(true);
setIsBuffering(false);
return;
}
setIsBuffering(false);
syncActivePreloadReadiness(video);
};
const handleProgress = () => {
syncActivePreloadReadiness(video);
// 窗口内视频只要已经产生缓冲,就标记为可复用;
// 之后预加载授权被收回时不再丢弃它的 src 和已缓冲数据。
if (shouldLoad && videoHasBufferedData(video)) {
onSourceCached(item.id);
}
};
const handleVolumeChange = () => {
if (!isActive) return;
// 当检测到 video 自身的 mute 状态或 volume 改变时,同步更新 React 状态。
// 这可以在移动端浏览器支持物理音量键调整时,自动反向取消静音并展示音量 HUD。
if (video.muted !== muted) {
@@ -849,6 +1047,32 @@ function ShortsSlide({
setVolume(video.volume);
}
};
const handlePlay = () => {
if (!isActive) return;
if (isVideoPausedByUser(index)) {
video.pause();
setPaused(true);
setIsBuffering(false);
return;
}
setPaused(false);
};
const handlePause = () => {
if (!isActive || video.ended) return;
setPaused(true);
setIsBuffering(false);
};
function syncActivePreloadReadiness(currentVideo: HTMLVideoElement) {
if (!isActive) return;
if (videoHasComfortableBuffer(currentVideo)) {
onActiveReadyForPreload(index);
} else if (videoBufferIsCritical(currentVideo)) {
// 高低水位滞回:只有缓冲真正告急才收回预加载授权,
// 在两个水位之间维持现状,避免阈值附近来回抖动。
onActiveNeedsPriority(index);
}
}
handleLoaded();
handleTime();
@@ -858,7 +1082,10 @@ function ShortsSlide({
video.addEventListener("waiting", handleWaiting);
video.addEventListener("playing", handlePlayingOrCanPlay);
video.addEventListener("canplay", handlePlayingOrCanPlay);
video.addEventListener("progress", handleProgress);
video.addEventListener("volumechange", handleVolumeChange);
video.addEventListener("play", handlePlay);
video.addEventListener("pause", handlePause);
// 挂载时如果已经在播放但是状态不到 ready 则置 buffering
if (video.readyState < 3 && !video.paused) {
@@ -872,9 +1099,12 @@ function ShortsSlide({
video.removeEventListener("waiting", handleWaiting);
video.removeEventListener("playing", handlePlayingOrCanPlay);
video.removeEventListener("canplay", handlePlayingOrCanPlay);
video.removeEventListener("progress", handleProgress);
video.removeEventListener("volumechange", handleVolumeChange);
video.removeEventListener("play", handlePlay);
video.removeEventListener("pause", handlePause);
};
}, [shouldMount, item.id, muted, volume, setMuted, setVolume]);
}, [shouldMount, shouldLoad, item.id, index, isActive, muted, volume, setMuted, setVolume, onActiveReadyForPreload, onActiveNeedsPriority, onSourceCached, isVideoPausedByUser]);
// 长按 2 倍速:直接绑原生事件
useEffect(() => {
@@ -939,16 +1169,18 @@ function ShortsSlide({
function togglePlayInternal() {
const video = localRef.current;
if (!video) return;
if (video.paused) {
const shouldResume =
isVideoPausedByUser(index) || (video.paused && paused && !isBuffering);
if (shouldResume) {
onUserPausedChange(index, false);
video.play().catch(() => undefined);
setPaused(false);
setPlayPauseHud({ id: Date.now(), type: "play" });
setTimeout(() => setPlayPauseHud(null), 450);
if (video.readyState < 3) setIsBuffering(true);
} else {
onUserPausedChange(index, true);
video.pause();
setPaused(true);
setPlayPauseHud({ id: Date.now(), type: "pause" });
setTimeout(() => setPlayPauseHud(null), 450);
setIsBuffering(false);
}
}
@@ -1175,9 +1407,9 @@ function ShortsSlide({
<video
ref={setRef}
className="shorts-slide__video"
src={item.videoSrc}
src={shouldLoad ? item.videoSrc : undefined}
poster={item.poster}
preload="auto"
preload={shouldLoad ? (shouldEagerLoad ? "auto" : "metadata") : "none"}
playsInline
loop
muted={muted}
@@ -1203,23 +1435,16 @@ function ShortsSlide({
{paused && isActive && !scrubbing && !playPauseHud && (
{paused && isActive && !scrubbing && (
<div className="shorts-slide__paused" aria-hidden="true">
</div>
)}
{/* 视频加载/缓冲旋转器 */}
{isBuffering && isActive && shouldMount && !isMarkedHidden && (
{isBuffering && isActive && shouldLoad && !isMarkedHidden && (
<div className="shorts-slide__buffering" aria-hidden="true">
<Loader2 size={30} className="shorts-slide__buffering-icon" />
</div>
)}
{/* 播放暂停瞬间 HUD 动效 */}
{playPauseHud && isActive && (
<div key={playPauseHud.id} className="shorts-slide__hud-pulse" aria-hidden="true">
{playPauseHud.type === "play" ? <Play size={30} fill="currentColor" /> : <Pause size={30} fill="currentColor" />}
<ShortsLoadingSpinner size={30} />
</div>
)}
@@ -1309,7 +1534,7 @@ function ShortsSlide({
)}
{/* 进度条 */}
{shouldMount && !isMarkedHidden && (
{isActive && shouldLoad && !isMarkedHidden && (
<div
className={`shorts-slide__progress ${
scrubbing ? "is-scrubbing" : ""
@@ -1343,10 +1568,184 @@ function ShortsSlide({
);
}
function ShortsLoadingSpinner({ size }: { size: number }) {
const ref = useRef<HTMLSpanElement | null>(null);
useEffect(() => {
let frame = 0;
const startedAt = performance.now();
const tick = (now: number) => {
const spinner = ref.current;
if (spinner) {
const rotation = ((now - startedAt) / 800) * 360;
spinner.style.transform = `rotate(${rotation}deg)`;
}
frame = window.requestAnimationFrame(tick);
};
frame = window.requestAnimationFrame(tick);
return () => window.cancelAnimationFrame(frame);
}, []);
return (
<span
ref={ref}
className="shorts-slide__loading-spinner"
style={{
"--shorts-spinner-size": `${size}px`,
} as React.CSSProperties}
aria-hidden="true"
/>
);
}
function applyVideoAudioState(
video: HTMLVideoElement,
nextMuted: boolean,
nextVolume: number
) {
const safeVolume = clamp(nextVolume, 0, 1);
const syncVolume = () => {
try {
if (Math.abs(video.volume - safeVolume) > 0.001) {
video.volume = safeVolume;
}
} catch {
// Some mobile browsers expose volume as effectively read-only.
}
};
if (!nextMuted) syncVolume();
try {
if (video.muted !== nextMuted) {
video.muted = nextMuted;
}
} catch {
// ignore
}
if (nextMuted) syncVolume();
}
function normalizeVideoPlaybackRate(video: HTMLVideoElement) {
try {
if (video.defaultPlaybackRate !== 1) {
video.defaultPlaybackRate = 1;
}
if (video.playbackRate !== 1) {
video.playbackRate = 1;
}
} catch {
// ignore
}
}
function stabilizeVideoAfterAudioToggle(
video: HTMLVideoElement,
shouldResume: () => boolean
) {
const stabilize = () => {
normalizeVideoPlaybackRate(video);
if (shouldResume() && video.paused && !video.ended) {
video.play().catch(() => undefined);
}
};
stabilize();
for (const delay of [80, 240, 600]) {
window.setTimeout(stabilize, delay);
}
}
function shouldUseDocumentScrollForShorts() {
return isIPhoneBrowserShell();
}
function isIPhoneBrowserShell() {
if (typeof window === "undefined" || typeof navigator === "undefined") {
return false;
}
const ua = navigator.userAgent || "";
return /\biPhone\b|\biPod\b/.test(ua) && !isStandaloneDisplayMode();
}
function isStandaloneDisplayMode() {
if (typeof window === "undefined" || typeof navigator === "undefined") {
return false;
}
const nav = navigator as Navigator & { standalone?: boolean };
return (
nav.standalone === true ||
window.matchMedia?.("(display-mode: standalone)").matches === true ||
window.matchMedia?.("(display-mode: fullscreen)").matches === true
);
}
function supportsElementFullscreenAPI(target?: Element | null) {
if (typeof document === "undefined") return false;
const el = (target ?? document.documentElement) as HTMLElement & {
webkitRequestFullscreen?: () => Promise<void> | void;
};
return (
typeof el.requestFullscreen === "function" ||
typeof el.webkitRequestFullscreen === "function"
);
}
function clamp(n: number, min: number, max: number) {
return n < min ? min : n > max ? max : n;
}
function getVideoWindowBounds(highestViewedIndex: number, itemCount: number) {
const size = Math.min(VIDEO_WINDOW_SIZE, itemCount);
if (size <= 0 || highestViewedIndex < 0) return { start: 0, end: -1 };
const end = clamp(highestViewedIndex, 0, itemCount - 1);
const start = Math.max(0, end - size + 1);
return { start, end };
}
/** 已经缓冲到片尾(含误差余量),不会再因网络卡顿 */
function videoBufferedToEnd(video: HTMLVideoElement) {
const duration = Number.isFinite(video.duration) ? video.duration : 0;
if (duration <= 0) return false;
const remaining = Math.max(0, duration - (video.currentTime || 0));
return bufferedAheadSeconds(video) >= remaining - 0.25;
}
function videoHasBufferedData(video: HTMLVideoElement) {
for (let i = 0; i < video.buffered.length; i += 1) {
if (video.buffered.end(i) > video.buffered.start(i)) {
return true;
}
}
return false;
}
/** 前向缓冲健康(达到高水位或已缓冲到结尾),可以放心预加载后续视频 */
function videoHasComfortableBuffer(video: HTMLVideoElement) {
if (video.readyState < 3) return false;
if (videoBufferedToEnd(video)) return true;
return bufferedAheadSeconds(video) >= ACTIVE_PRELOAD_BUFFER_SECONDS;
}
/** 前向缓冲告急(跌破低水位且没缓冲到结尾),应收回预加载授权 */
function videoBufferIsCritical(video: HTMLVideoElement) {
if (video.readyState < 3) return true;
if (videoBufferedToEnd(video)) return false;
return bufferedAheadSeconds(video) < ACTIVE_PRELOAD_KEEP_SECONDS;
}
function bufferedAheadSeconds(video: HTMLVideoElement) {
const current = video.currentTime || 0;
for (let i = 0; i < video.buffered.length; i += 1) {
const start = video.buffered.start(i);
const end = video.buffered.end(i);
if (start <= current + 0.25 && end > current) {
return Math.max(0, end - current);
}
}
return 0;
}
function formatClock(seconds: number) {
if (!Number.isFinite(seconds) || seconds < 0) return "00:00";
const total = Math.floor(seconds);
@@ -1372,6 +1771,7 @@ function getDriveShortName(source: string): string {
if (s.includes("quark") || s.includes("夸克")) return "Quak";
if (s.includes("onedrive")) return "OneDrive";
if (s.includes("wopan") || s.includes("沃盘")) return "沃盘";
if (s.includes("guangyapan") || s.includes("guangya") || s.includes("光鸭")) return "光鸭";
if (s.includes("localstorage") || s.includes("本地")) return "本地";
if (s.includes("spider") || s.includes("爬虫")) return "爬虫";
return source.substring(0, 4);
-17
View File
@@ -10,7 +10,6 @@ import {
deleteVideo,
fetchTags,
fetchVideoDetail,
hideVideo,
recordView,
updateVideoTags,
} from "@/data/videos";
@@ -23,7 +22,6 @@ export default function VideoDetailPage() {
const [tags, setTags] = useState<TagItem[]>([]);
const [loading, setLoading] = useState(true);
const [tagSaving, setTagSaving] = useState(false);
const [hideSaving, setHideSaving] = useState(false);
const [deleteOpen, setDeleteOpen] = useState(false);
const [deleteSource, setDeleteSource] = useState(false);
const [deleteSaving, setDeleteSaving] = useState(false);
@@ -68,19 +66,6 @@ export default function VideoDetailPage() {
}
}
async function handleHideVideo() {
if (!detail || hideSaving) return;
if (!window.confirm("确定以后不再展示这个视频吗?")) return;
setHideSaving(true);
try {
await hideVideo(detail.id);
navigate("/list", { replace: true });
} catch {
setHideSaving(false);
window.alert("隐藏失败,请稍后重试");
}
}
function handleOpenDelete() {
if (!detail || deleteSaving) return;
setDeleteSource(false);
@@ -233,9 +218,7 @@ export default function VideoDetailPage() {
<VideoActions
video={detail}
onHideVideo={handleHideVideo}
onDeleteVideo={handleOpenDelete}
hideSaving={hideSaving}
deleteSaving={deleteSaving}
/>
</section>

Some files were not shown because too many files have changed in this diff Show More