ci: deploy.yml 按分支分流(dev→srv1563674 / main·master→sf5-gpu-01)

prod 也接入自动部署:push main/master 在 prod GPU 机重建 content-harvester 容器。
对应在 sf5-gpu-01 上新增了实例级 gitea runner(label sf5-gpu-01)。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

.gitea/workflows/deploy.yml

@@ -0,0 +1,92 @@
+name: Deploy
+
+# push dev   → dev 开发机(srv1563674)重建 content-harvester 容器
+# push main/master → prod GPU 机(sf5-gpu-01)重建 content-harvester 容器
+# (与主站 AIautomation 的 deploy.yml 一致的分支→机器分流)
+on:
+  push:
+    branches:
+      - dev
+      - main
+      - master
+  workflow_dispatch:
+
+jobs:
+  deploy:
+    name: Build and deploy
+    runs-on: ${{ github.ref == 'refs/heads/dev' && 'srv1563674' || 'sf5-gpu-01' }}
+
+    env:
+      IMAGE: content-harvester
+      CONTAINER: content-harvester
+      HOST_PORT: "8019"               # 主机端口(主要给本机调试用)
+      DOCKER_NETWORK: ai-admin-cuiyue_default   # 跟主站 backend 同一个 docker 网络
+      PROFILE_VOLUME: content-harvester-profile  # 持久浏览器资料目录(过 Cloudflare 用)
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Prepare env file
+        env:
+          CH_ENV_FILE: ${{ secrets.CH_ENV_FILE }}
+        run: |
+          set -euo pipefail
+          if [ -n "${CH_ENV_FILE:-}" ]; then
+            printf '%s\n' "$CH_ENV_FILE" > .env
+          fi
+          test -f .env || {
+            echo "Missing .env — 请在该仓库的 Gitea Actions secrets 里加一个 'CH_ENV_FILE',内容形如:"
+            echo "  HARVESTER_TOKEN=..."
+            echo "  MINIO_ENDPOINT=https://minio.aidfxn.com"
+            echo "  MINIO_BUCKET=clawlerdata"
+            echo "  MINIO_ACCESS_KEY=..."
+            echo "  MINIO_SECRET_KEY=..."
+            exit 1
+          }
+          # 只打 key 名,不打值
+          echo "env keys: $(grep -oE '^[A-Za-z_][A-Za-z0-9_]*' .env | paste -sd, -)"
+
+      - name: Show docker info
+        run: docker --version
+
+      - name: Build image
+        run: docker build -t "$IMAGE:latest" .
+
+      - name: Recreate container
+        run: |
+          set -euo pipefail
+          docker rm -f "$CONTAINER" >/dev/null 2>&1 || true
+          docker run -d \
+            --name "$CONTAINER" \
+            --restart unless-stopped \
+            --network "$DOCKER_NETWORK" \
+            -p "${HOST_PORT}:8000" \
+            --env-file .env \
+            -v "${PROFILE_VOLUME}:/data/browser_profile" \
+            --memory=2g --cpus=2 \
+            "$IMAGE:latest"
+
+      - name: Health check
+        run: |
+          set -e
+          for i in $(seq 1 20); do
+            if docker exec "$CONTAINER" python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8000/healthz', timeout=3).read()" >/dev/null 2>&1; then
+              echo "healthz ok"
+              docker ps --filter "name=$CONTAINER" --format '{{.Names}} | {{.Status}} | {{.Ports}}'
+              exit 0
+            fi
+            echo "waiting for healthz... ($i/20)"
+            sleep 3
+          done
+          echo "health check failed"
+          docker logs --tail 80 "$CONTAINER" || true
+          exit 1
+
+      - name: Cleanup env file
+        if: always()
+        run: rm -f .env
+
+      - name: Prune dangling images
+        if: always()
+        run: docker image prune -f >/dev/null 2>&1 || true

CLAUDE.md

@@ -1,23 +1,24 @@
 # CLAUDE.md — content-harvester
 
-内容采集服务(FastAPI + Playwright headed-Chrome/xvfb)。给一个来源页 URL,逐个打开内容详情页抓 **标题 / 详情页URL / 封面图 / 媒体地址**;若调用方下发了对象存储(MinIO/S3)配置,则把媒体(目前是 m3u8 视频)拉成 mp4 + 封面一起上传,`video_url`/`cover_image` 改成对象存储稳定地址(很多站点的媒体地址带 `expires` 签名、约 1h 失效)。以 `resources` 列表回传。给「启源 AI 平台」主站「多内容数据采集 / 来源=内容网站」任务(lobster 执行器)调用,主站把结果写进资源库。当前内置一个 xchina.co 的 scraper(`core/scraper.py`);加新来源就再加一个 scraper 模块。
+内容采集服务(FastAPI + Playwright headed-Chrome/xvfb)。给一个来源页 URL,逐个打开内容详情页抓元数据;若调用方下发了对象存储(MinIO/S3)配置,把媒体 + 封面一起上传,媒体地址改成对象存储稳定地址(很多站点的媒体地址带 `expires` 签名、约 1h 失效)。以 `resources` 列表回传。给「启源 AI 平台」主站「多内容数据采集 / 来源=内容网站」任务(lobster 执行器)调用,主站把结果写进资源库。**两类内容**:视频(m3u8→yt-dlp+ffmpeg 转 mp4,`source_type=video`)和图集(套图详情页几百~上千张图逐张下载上传,`source_type=image`/`content_type=article`/回传 `images[]`,主站对 `images` 自动跑图片 OCR 审核)——按 `content_url` 路径(`/videos/`·`/video/` ↔ video,`/photos/`·`/photo/` ↔ photo)或载荷 `media_kind` 自动选 scraper。当前内置 xchina.co 的视频 scraper(`core/scraper.py`)+ 图集 scraper(`core/scraper_photos.py`);加新来源就再加一个 scraper 模块。
 
 > 前身是 Tkinter GUI 下载器,已删除。无单元测试。详细设计见 `PRD-minio-upload.md`。
 
 ## 结构
 
 - `core/browser.py` —— `browser_context(headless)` async ctxmgr,`launch_persistent_context`(资料目录 `BROWSER_PROFILE_DIR`)。`HEADLESS=0` + `BROWSER_CHANNEL`(x86_64 可设 `chrome`)用真实有头浏览器过 Cloudflare;容器里靠 `docker-entrypoint.sh` 起 Xvfb。
-- `core/scraper.py` —— xchina 专属 scraper。`XChinaScraper(context, is_cancelled, log)`,`run(series_url,start,end) -> ScrapeResult`;`is_detail_url`、`_page_url_candidates`、`_collect_video_links`(`.item.video` + 正则兜底)、`scrape_detail`(`_extract_title` / `_find_cover_url`(截屏tab→og:image/JSON-LD/poster→最大图)/ `_find_video_url`(扫 `page.content()` 的 `.m3u8` + 网络嗅探))、`_handle_interstitial_ad`(广告中转页)、`_assert_not_blocked`(Cloudflare 质询页等其自动放行 ~24s 否则抛)。`ScrapeCancelled` 取消信号。加别的站就新建 `core/scraper_<site>.py`。
-- `core/media.py` —— `MinioConfig`(`from_payload` 从 invoke 载荷读 `minio_endpoint/access_key/secret_key/bucket`,任一缺则 `is_complete=False`;`safe_repr` 不露 secret)、`download_video`(yt-dlp 并发分片 + ffmpeg 合并 → `<videoid>.mp4`,`max_mb` 超限抛)、`download_cover`(requests 流式)、`upload_to_minio`(`minio.Minio.fput_object` → 返回 `endpoint/bucket/key`)、`extract_video_id`(从详情页 URL 的 `id-(\w+)\.html`)。
-- `core/models.py` —— `ScrapedItem`(`source_url/title/source_name/cover_image/video_url/page_no/video_id/extra`,`to_ingest_dict()` → `source_type:"video"/content_type:"video"/extra_data:{referer,page_no,collector,video_id,**extra}`;`extra` 里放 `original_m3u8/original_cover/minio_object_video/minio_object_cover/video_bytes/upload_skipped_reason`)、`ScrapeResult`。
-- `server/jobs.py` —— 内存级 job 表 + `asyncio.create_task` 后台跑;`_RUN_LOCK` 串行化(单并发)。`_run`:① 浏览器开着 `scraper.run()` 抓元数据 → 关浏览器;② 若 `job.minio_cfg.is_complete` 则 `_process_uploads` 逐条 `asyncio.to_thread` 下载视频+封面、上传 MinIO、回写 URL(失败的条目 `extra.upload_skipped_reason=...` 降级保留原媒体地址);③ 临时目录 `/tmp/content-harvester/<jobid>/` 用完即删。`MAX_VIDEO_MB`/`CONCURRENT_FRAGMENTS` env。**不要把 secret_key 传进 `_log`**。
-- `server/app.py` —— FastAPI。`POST /agents/{id}/invoke`(body=主站调用载荷:`content_url/platform_name/source_kind/start_page/end_page` + `minio_*`;`MinioConfig.from_payload(payload)` → 传给 `jobs.submit`;返回 `{"task_id","status"}`)、`GET /tasks/{id}`、`POST /tasks/{id}/cancel`、`GET /healthz`。Bearer 鉴权 `HARVESTER_TOKEN`(空则不校验)、SSRF 白名单 `ALLOWED_HOSTS`(默认 `xchina.co`)。
+- `core/scraper.py` —— xchina 视频 scraper。`XChinaScraper(context, is_cancelled, log)`,`run(series_url,start,end) -> ScrapeResult`;`is_detail_url`、`_page_url_candidates`、`_collect_video_links`(`.item.video` + 正则兜底)、`scrape_detail`(`_extract_title` / `_find_cover_url`(截屏tab→og:image/JSON-LD/poster→最大图)/ `_find_video_url`(扫 `page.content()` 的 `.m3u8` + 网络嗅探))、`_handle_interstitial_ad`(广告中转页)、`_assert_not_blocked`(Cloudflare 质询页等其自动放行 ~24s 否则抛)。`ScrapeCancelled` 取消信号。**导航/反爬 helper(`_visit`/`_assert_not_blocked`/`_looks_blocked`/`_page_url_candidates`/`_check_cancel`)被 `scraper_photos.py` 继承复用。**
+- `core/scraper_photos.py` —— xchina 图集 scraper。`XChinaPhotoScraper(context, is_cancelled, log, max_images=100)` 继承 `XChinaScraper`;`run` 同签名,`is_photo_series_url`/`is_photo_detail_url`/`_canon_detail_url`(`/photo/id-X/3.html`→`/photo/id-X.html`) 模块函数。流程:系列页 `.item.photo` 收 `/photo/id-` 链接(带 title hint)→ `scrape_photo_detail`:① `_photo_bg_images`(读 `div.img[role=img]` 的 `background-image` URL)② `_parse_img_template`(推 `.../{detailId}/{NNNN}{_WxH}.{ext}` 的 `.format(n=)` 模板,实测 xchina 是 `https://img.xchina.io/photos2/{id}/{n:04d}_600x0.webp`,旧套图前缀可能是 `photos/` 不带 2 → 不硬编码)③ 总张数 = `(\d+)P` 文本 ∪ `.pager` 最大页码×14 ∪ 第 1 页渲染数 ④ `images = [模板.format(n=1..min(total,max_images))]`,`head_ok` 校验末张;不通过 → `_scrape_all_pages_images` 逐页加载 `/photo/id-{id}/{N}.html` 收 bg-image ⑤ `_extract_photo_meta`(模特/系列/日期/标签,系列名优先取面包屑/`.subs` 里的避免误命中侧边栏)→ `_compose_content`(标题 + 模特/系列/日期/共N张/标签;保证非空)。返回 `ScrapedItem(media_kind="photo", images=[...], image_total, image_capped, ...)`。
+- `core/media.py` —— `MinioConfig`(`from_payload` 从 invoke 载荷读 `minio_endpoint/access_key/secret_key/bucket`,任一缺则 `is_complete=False`;载荷找不到从 `MINIO_*` env 兜底;`safe_repr` 不露 secret)、`download_video`(yt-dlp 并发分片 + ffmpeg 合并 → `<id>.mp4`,`max_mb` 超限抛)、`download_cover`/`download_image`(requests 流式;失败返回 None,不致命)、`head_ok`(HEAD 探活,405 退 GET)、`upload_to_minio`(`minio.Minio.fput_object` → 返回 `endpoint/bucket/key`)、`extract_video_id`(=`extract_detail_id`;从 `/(video|videos|photo|photos)/id-(\w+)(/\d+)?\.html`)。
+- `core/models.py` —— `ScrapedItem`:`media_kind`(`video`|`photo`,默认 video)+ 视频字段(`video_url`)+ 图集字段(`images: list[str]`/`image_total`/`image_capped`)。`to_ingest_dict()` 按 `media_kind` 分流:video → `source_type:"video"/content_type:"video"/video_url`;photo → `source_type:"image"/content_type:"article"/images:[...]/video_url:None`(`content_type` 是 `article` 因为主站 `ContentTypeEnum` 只有 article/video)。`extra_data` 里 video 放 `original_m3u8/original_cover/minio_object_video/minio_object_cover/video_bytes`;photo 放 `image_total/image_count/image_capped/base_prefix/detail_id/img_template/minio_objects_images/images_uploaded/images_failed/original_images_count/original_cover`;两者都可能有 `upload_skipped_reason`。
+- `server/jobs.py` —— 内存级 job 表 + `asyncio.create_task` 后台跑;`_RUN_LOCK` 串行化(单并发)。`Job` 加 `media_kind`/`max_images`。`_run`:① `_resolve_media_kind`(载荷给了用载荷,否则按 URL 判)+ `_make_scraper`(photo→`XChinaPhotoScraper`,否则 `XChinaScraper`)→ 浏览器开着 `scraper.run()` → 关浏览器;② 若 `minio_cfg.is_complete` 则 `_process_uploads`(按 `item.media_kind` 调 `_process_video_item`:yt-dlp 下视频+封面上传;或 `_process_photo_item`:逐张 `download_image`→`upload_to_minio`→ `images[]` 换成 MinIO URL,边下边传边删不囤盘,单张失败保留原图床地址,封面=已上传第一张)③ 临时目录 `/tmp/content-harvester/<jobid>/` 用完即删。`MAX_VIDEO_MB`/`CONCURRENT_FRAGMENTS` env。**不要把 secret_key 传进 `_log`**。
+- `server/app.py` —— FastAPI。`POST /agents/{id}/invoke`(body=主站调用载荷:`content_url/platform_name/source_kind/start_page/end_page/media_kind?/max_images?` + `minio_*`)解析 `media_kind`(载荷 → URL 路径 → `agent_id` 兜底)与 `max_images`(默认 `DEFAULT_MAX_IMAGES`=100,clamp 到 `MAX_IMAGES_CAP`=2000),`MinioConfig.from_payload(payload)`,→ `jobs.submit(...media_kind, max_images)`;返回 `{"task_id","status"}`。`GET /tasks/{id}`、`POST /tasks/{id}/cancel`、`GET /healthz`。Bearer 鉴权 `HARVESTER_TOKEN`(空则不校验)、SSRF 白名单 `ALLOWED_HOSTS`(默认 `xchina.co`,图集视频同域)。
 - `Dockerfile` —— base `mcr.microsoft.com/playwright/python`;apt 装 `xvfb`(有头)+`ffmpeg`(yt-dlp 合并);`HEADLESS=0`;挂卷 `/data/browser_profile`;ENTRYPOINT=`docker-entrypoint.sh`(起 Xvfb)→ uvicorn。x86_64 上可改 `playwright install chromium chrome` + `BROWSER_CHANNEL=chrome`。
 - `requirements.txt` —— playwright / fastapi / uvicorn / requests / yt-dlp / minio。
 
 ## 对接契约(主站 lobster 执行器)
 
-主站 `app/services/lobster_client.py::call_lobster_agent`:POST `{api_url}/agents/{agent_id}/invoke`(body=`payload`)→ 取 `task_id` → 轮询 GET `{api_url}/tasks/{task_id}` 直到 `status∈{succeeded,failed}` → 用 `body["output"]`/`body["error"]`。本服务 `output` 成功时 = `{"resources":[ResourceIngestItem字段...], "pages_scanned":n, "item_errors":[...]}`,每条 `resources[i]` 的 `video_url`/`cover_image` 若上传成功是对象存储地址(`extra_data.original_m3u8` 留原始签名URL),否则是原始来源地址(`extra_data.upload_skipped_reason`)。
+主站 `app/services/lobster_client.py::call_lobster_agent`:POST `{api_url}/agents/{agent_id}/invoke`(body=`payload`)→ 取 `task_id` → 轮询 GET `{api_url}/tasks/{task_id}` 直到 `status∈{succeeded,failed}` → 用 `body["output"]`/`body["error"]`。本服务 `output` 成功时 = `{"resources":[ResourceIngestItem字段...], "pages_scanned":n, "item_errors":[...]}`,`resources[i]` 是视频条目(`source_type=video`,`video_url`/`cover_image` 上传成功即对象存储地址、原始签名 URL 进 `extra_data.original_m3u8`)或图集条目(`source_type=image`/`content_type=article`,`images[]` 上传成功即对象存储 URL 数组、`cover_image`=已上传第一张);上传失败则相应地址保留原始来源 + `extra_data.upload_skipped_reason`。主站对 `images[]` 数组会自动逐张排图片 OCR 审核(命中 ad/minor/non_frontal 剔除;一条资源 content/images/video 全空 → soft delete)。
 
 主站侧配套改动(`agent-system-backend/app/services/task_service.py`):① 非批次 lobster 改 `asyncio.create_task(_run_lobster_task_bg(...))` 后台跑;② `_run_lobster_task_bg` 成功后 `_maybe_ingest_to_resource_pool`;③ `_maybe_ingest_to_resource_pool` 兼容从 `result["output"]["resources"]` 取;④ `_dispatch_lobster` 把 `_resolve_skill_variables` 解出的 Skill 变量并入 payload(让远端拿到 `minio_*`);⑤ crawl 类用 `CRAWL_LOBSTER_TIMEOUT`(env,默认 7200s)。配置侧:给那个采集 Skill 绑定 `minio_endpoint/minio_access_key/minio_secret_key/minio_bucket` 4 个 Skill 变量(实际值由运维在主站配置,本服务不持有任何密钥)。
 

DEPLOY.md

@@ -8,27 +8,46 @@
 - 代码:`git clone -b dev https://git.aidfxn.com/admin/content-harvester.git`,以后 `git pull origin dev`。
 - 对接的主站 Skill:「内容网站采集(content-harvester)」(`executor_type=lobster`,`lobster_api_url=http://content-harvester:8000`,`lobster_api_key` = `.env.dev` 里的 `HARVESTER_TOKEN`)。
 
-## 重新部署 / 更新(改了代码、push 了 dev 之后)
+## 重新部署 / 更新
 
-本仓库**没有自动部署 workflow**,要手动:
+### dev:自动部署(push `dev` 分支即触发)
+
+`.gitea/workflows/deploy.yml` —— **push 到 `dev` 分支**(或在 gitea 上手动 `workflow_dispatch`)→ dev 开发机的 gitea runner(标签 `srv1563674`,跟主站 AIautomation 的 deploy.yml 用同一个)自动:`actions/checkout` → `docker build -t content-harvester .` → `docker rm -f content-harvester` → `docker run -d ...`(env 走 `--env-file .env`)→ healthz 检查 → `docker image prune`。**只管 dev**;`master`/prod 不在 `on.push.branches` 里,不会被这个工作流碰(prod 仍走下面的手动流程)。
+
+**前置条件(一次性)**:在该仓库的 Gitea Actions secrets 里加一个 **`CH_ENV_FILE`**,内容就是要喂给容器的 env 文件全文(等价于 `.env.dev`),至少:
+
+```
+HARVESTER_TOKEN=<和主站采集 Skill 的 lobster_api_key 一致>
+MINIO_ENDPOINT=https://minio.aidfxn.com
+MINIO_BUCKET=clawlerdata
+MINIO_ACCESS_KEY=<...>
+MINIO_SECRET_KEY=<...>
+# 可选:HEADLESS=0 / BROWSER_CHANNEL=chrome / MAX_PAGES=... / DEFAULT_MAX_IMAGES=... 等
+```
+
+没配 `CH_ENV_FILE` 的话工作流会在「Prepare env file」那步报错退出(不会拿一份残缺的 env 去跑)。工作流里硬编码的部署参数(非密钥):host 端口 `8019`、docker 网络 `ai-admin-cuiyue_default`、卷 `content-harvester-profile:/data/browser_profile`、`--memory=2g --cpus=2`、`--restart unless-stopped` —— 要改这些就改 `deploy.yml` 顶部的 `env:` 块。
+
+> 工作流在 runner 的 workspace 里 build(`actions/checkout` 取的那份代码),**不动 `/opt/content-harvester`** —— 那个目录会逐渐过时,「现在 dev 跑的是哪份代码」要看运行中的容器(`docker exec content-harvester ...`),别看 `/opt/content-harvester` 的 git。
+
+### 手动部署(prod;或 dev 想临时绕过工作流)
 
 ```bash
 cd /opt/content-harvester
-git pull -q origin dev
+git pull -q origin <dev|master>          # prod 用 master
 docker build -t content-harvester .
 docker rm -f content-harvester
 docker run -d --name content-harvester \
   --restart unless-stopped \
-  --network <主站 backend 所在的 docker 网络> \
+  --network ai-admin-cuiyue_default \    # 跟主站 backend 同一个 docker 网络
   -p 8019:8000 \
-  --env-file /opt/content-harvester/.env.dev \   # 这个文件要先放到服务器上(本地 scp 过去,或在服务器上照样写一份)
+  --env-file /opt/content-harvester/.env.dev \   # 这个文件要先放到服务器上(本地 scp 过去,或在服务器上照样写一份;内容同 CH_ENV_FILE)
   -v content-harvester-profile:/data/browser_profile \
   --memory=2g --cpus=2 \
   content-harvester
 sleep 6 && curl -s localhost:8019/healthz   # 期望 {"ok":true,"headless":false,...}
 ```
 
-> 注:`.env.dev` 被 `.gitignore` 排除,`git pull` 不会带它过去。第一次在服务器上部署时手动把它放到 `/opt/content-harvester/.env.dev`(从本地 scp,或照本地内容写一份)。或者也可以不用 env-file、把那几个 `-e KEY=VALUE` 直接展开写在 `docker run` 里(但 token/凭据就会出现在 shell history / `docker inspect`)。
+> 注:`.env.dev` 被 `.gitignore` 排除,`git pull` 不会带它过去。第一次在服务器上手动部署时把它放到 `/opt/content-harvester/.env.dev`(从本地 scp,或照本地内容写一份)。或者也可以不用 env-file、把那几个 `-e KEY=VALUE` 直接展开写在 `docker run` 里(但 token/凭据就会出现在 shell history / `docker inspect`)。
 
 如果一时没法在服务器上 build,也可以本地 `docker build` 后 `docker save content-harvester | gzip | ssh ... 'gunzip | docker load'` —— 但**注意 CPU 架构**:Mac 一般是 arm64,服务器多是 x86_64,本地构建的 arm64 镜像在服务器跑不了。所以一般还是在服务器上 build。
 

README.md

@@ -1,10 +1,12 @@
 # content-harvester
 
-内容采集服务。给一个来源页 URL,用真实有头浏览器逐个打开内容详情页,抓出每条内容的元数据(标题 / 详情页 URL / 封面图 / 媒体地址);若调用方下发了对象存储(MinIO/S3)配置,顺带把媒体(目前是 m3u8 视频)拉成 mp4、连同封面一起上传,把媒体地址换成对象存储上的稳定 URL(很多站点的媒体地址带 `expires` 签名、一小时左右就失效)。结果以 `resources` 列表回传。
+内容采集服务。给一个来源页 URL,用真实有头浏览器逐个打开内容详情页,抓出每条内容的元数据(标题 / 详情页 URL / 封面图 / 媒体地址);若调用方下发了对象存储(MinIO/S3)配置,顺带把媒体拉下来、连同封面一起上传,把媒体地址换成对象存储上的稳定 URL(很多站点的媒体地址带 `expires` 签名、一小时左右就失效)。结果以 `resources` 列表回传。
+
+支持两类内容:**视频**(m3u8 → yt-dlp+ffmpeg 转 mp4 上传,`source_type=video`)和 **图集**(套图详情页里成百上千张图,逐张下载上传,`source_type=image`、`content_type=article`、回传 `images[]` 数组——主站对 `images` 数组会自动跑图片 OCR 审核)。按 `content_url` 路径(`/videos/`·`/video/` ↔ 视频,`/photos/`·`/photo/` ↔ 图集)或载荷里的 `media_kind` 自动选 scraper。
 
 主要给「启源 AI 平台」主站调用:主站「多内容数据采集」任务里来源大类选「内容网站」时,把作业 POST 给本服务,本服务返回 `resources`,主站写进资源库。
 
-当前内置一个 **xchina.co** 的 scraper(`core/scraper.py`);接别的来源就再加一个 scraper 模块。
+当前内置 **xchina.co** 的视频 scraper(`core/scraper.py`)和图集 scraper(`core/scraper_photos.py`);接别的来源就再加一个 scraper 模块。
 
 > 前身是个 Tkinter GUI 视频下载器,已废弃改造成本服务。详细设计见 `PRD-minio-upload.md`,实现速查见 `CLAUDE.md`。
 
@@ -15,11 +17,16 @@
 **采集流程(一次作业)**
 
 1. 收到 `POST /agents/{id}/invoke`(载荷来自主站),校验 token、`source_kind`、`content_url` host 白名单,从载荷取 MinIO 配置,生成 `job_id`,**立即返回** `{"task_id": job_id, "status":"running"}`,作业转后台异步跑(进程内单作业串行,防风控)。
-2. 用有头 Chromium(容器里靠 Xvfb 提供虚拟显示)打开来源页:
-   - `content_url` 是系列页 → 取页内 `.item.video` 卡片里的详情链接,按 `start_page..end_page` 翻页(默认只爬给定页);
-   - `content_url` 是详情页 → 直接采这一条。
-3. 逐个打开详情页:处理广告中转页;遇 Cloudflare JS 质询页等它自动放行(最多 ~24s);抓 **标题**、**简介/正文**(meta/og/jsonld 描述 + 详情面板文本 + 标签;实在没有就用标题——主站资源库对空正文会拒收)、**封面图 URL**(截屏 tab 图 → og:image/JSON-LD thumbnail/`<video poster>` → 页面最大图,排除 logo/占位)、**视频地址**(扫页面 HTML 里的 `.m3u8`/`.mp4` + 嗅探网络响应)。
-4. 若调用方给了完整 MinIO 配置:逐条用 **yt-dlp**(并发分片下载)+ **ffmpeg**(合并/转封装,不重编码、无损、很快)把 m3u8 拉成 `<videoid>.mp4` → 上传 `xchina/<videoid>.mp4` 到桶;封面图也下载上传 `xchina/<videoid>.<ext>`;把该条的 `video_url`/`cover_image` 改成对象存储 URL,原始地址留进 `extra_data`(`original_m3u8`/`original_cover`),并记 `minio_object_video`/`minio_object_cover`/`video_bytes`。临时文件用完即删,不长期落盘。
+2. 按 `content_url` 路径(或载荷 `media_kind`)选 scraper:视频(`/videos/`·`/video/`)走 `XChinaScraper`,图集(`/photos/`·`/photo/`)走 `XChinaPhotoScraper`。用有头 Chromium(容器里靠 Xvfb 提供虚拟显示)打开来源页:
+   - 系列页 → 取页内卡片(视频:`.item.video`;图集:`.item.photo`)里的详情链接,按 `start_page..end_page` 翻页(默认只爬给定页);
+   - 详情页 → 直接采这一条。
+3. 逐个打开详情页:处理广告中转页;遇 Cloudflare JS 质询页等它自动放行(最多 ~24s);抓 **标题**、**简介/正文**(实在没有就用标题——主站资源库对空正文会拒收)、**封面图 URL**:
+   - **视频**:简介=meta/og/jsonld 描述 + 详情面板文本 + 标签;封面=截屏 tab 图 → og:image/JSON-LD thumbnail/`<video poster>` → 页面最大图(排除 logo/占位);**视频地址**=扫页面 HTML 里的 `.m3u8`/`.mp4` + 嗅探网络响应。
+   - **图集**:从详情页 `div.img[role=img]` 的 `background-image` 解析图片 URL 模板(实测 `https://img.xchina.io/photos2/{detailId}/{NNNN}_600x0.webp`,NNNN 四位补零序号),从「601P」文本 / 分页器估出总张数 `total`,构造 `images = [模板.format(n=1..min(total, max_images))]`(对末张做 HEAD 校验,不通过则回退逐页加载 `/photo/id-{detailId}/{N}.html` 抓 bg-image);封面=`images[0]`;正文 = 标题 + 模特/系列/日期/标签/共N张图。
+4. 若调用方给了完整 MinIO 配置:
+   - **视频**:用 **yt-dlp**(并发分片)+ **ffmpeg**(无损 remux)把 m3u8 拉成 `xchina/<id>.mp4` 上传,封面上传 `xchina/<id>.<ext>`,`video_url`/`cover_image` 换成对象存储 URL,原始地址留 `extra_data.original_m3u8`/`original_cover`,记 `minio_object_video`/`minio_object_cover`/`video_bytes`。
+   - **图集**:逐张下载(边下边传边删,不囤盘)上传 `xchina/<detailId>/{NNNN}.<ext>`,`images[]` 整体换成对象存储 URL(单张失败则该项保留原图床地址),封面=已上传的第一张,记 `extra_data` 的 `minio_objects_images`/`images_uploaded`/`images_failed`/`original_images_count`/`original_cover`(`image_total`/`image_count`/`image_capped` 在 `to_ingest_dict` 时写)。
+   临时文件用完即删,不长期落盘。
 5. 作业结束:`status=succeeded`(只要至少抓到一条)/ `failed`(0 条且有抓取错误)。主站轮询 `GET /tasks/{job_id}` 拿结果。
 
 **降级(任一环节出问题不让整作业白跑)**
@@ -45,13 +52,14 @@
 
 ```
 core/
-  browser.py    # Playwright 持久化 BrowserContext(资料目录可挂卷),反自动化参数 / zh-CN / Asia-Shanghai
-  scraper.py    # xchina.co 专属 scraper:系列页翻页、详情页解析(标题/简介/封面/视频)、广告中转页处理、Cloudflare 质询等待
-  media.py      # MinioConfig(从载荷/env 解析)、download_video(yt-dlp+ffmpeg)、download_cover、upload_to_minio、extract_video_id
-  models.py     # ScrapedItem(字段对齐主站 ResourceIngestItem)、ScrapeResult
+  browser.py        # Playwright 持久化 BrowserContext(资料目录可挂卷),反自动化参数 / zh-CN / Asia-Shanghai
+  scraper.py        # xchina.co 视频 scraper:系列页翻页、详情页解析(标题/简介/封面/m3u8)、广告中转页、Cloudflare 质询等待;导航/反爬 helper 也给图集 scraper 复用
+  scraper_photos.py # xchina.co 图集 scraper(继承 XChinaScraper 复用导航 helper):系列页 .item.photo 翻页、详情页解析图片 URL 模板 + 总张数 + 模特/系列/日期/标签,construct-or-paginate 收集 images[]
+  media.py          # MinioConfig(从载荷/env 解析)、download_video(yt-dlp+ffmpeg)、download_cover、download_image、head_ok、upload_to_minio、extract_video_id(=extract_detail_id)
+  models.py         # ScrapedItem(media_kind=video|photo;to_ingest_dict 按 media_kind 出 source_type=video|image;字段对齐主站 ResourceIngestItem)、ScrapeResult
 server/
-  app.py        # FastAPI:/agents/{id}/invoke、/tasks/{id}、/tasks/{id}/cancel、/healthz;Bearer 鉴权;SSRF host 白名单
-  jobs.py       # 内存 job 表 + 后台执行 + 单并发串行 + 下载/上传/降级 + 临时目录清理
+  app.py        # FastAPI:/agents/{id}/invoke(按 content_url/agent_id/media_kind 选视频或图集)、/tasks/{id}、/tasks/{id}/cancel、/healthz;Bearer 鉴权;SSRF host 白名单
+  jobs.py       # 内存 job 表 + 后台执行 + 单并发串行;_process_uploads 按 media_kind 分流(视频:yt-dlp/ffmpeg;图集:逐张下载上传)+ 降级 + 临时目录清理
 Dockerfile          # base mcr.microsoft.com/playwright/python;apt 装 xvfb + ffmpeg;ENTRYPOINT 起 Xvfb → uvicorn
 docker-entrypoint.sh
 requirements.txt    # playwright / fastapi / uvicorn / requests / yt-dlp / minio
@@ -59,7 +67,7 @@ PRD-minio-upload.md # 设计文档
 CLAUDE.md           # 给 AI / 维护者的实现速查
 ```
 
-加新来源:新建 `core/scraper_<site>.py`(实现 `run(url, start, end) -> ScrapeResult`),在 `server/jobs.py` 里按 `content_url` host 或载荷里的某字段选用对应 scraper。`core/media.py`、`server/` 不用动。
+加新来源:新建 `core/scraper_<site>.py`(实现 `run(url, start, end) -> ScrapeResult`),在 `server/jobs.py::_make_scraper` 里按 `media_kind` / `content_url` host 选用对应 scraper。`core/media.py`、`server/app.py` 基本不用动(图集类记得 `ScrapedItem.media_kind="photo"`、填 `images`)。
 
 ---
 
@@ -75,6 +83,8 @@ CLAUDE.md           # 给 AI / 维护者的实现速查
 |---|---|---|
 | `content_url` | 是 | 来源页 / 详情页 URL;host 必须在 `ALLOWED_HOSTS` 白名单(默认 `xchina.co`) |
 | `source_kind` | 否 | 非 `website` 直接 400 |
+| `media_kind` | 否 | `video` / `photo`;不给则按 `content_url` 路径(`/videos/`·`/video/` ↔ video,`/photos/`·`/photo/` ↔ photo)或 `agent_id`(含 `photo` → photo)自动判断 |
+| `max_images` | 否 | 图集每个套图最多抓多少张,默认 `DEFAULT_MAX_IMAGES`(100),上限 `MAX_IMAGES_CAP`(2000);仅 photo 生效 |
 | `platform_name` | 否 | 作为 `source_name`,默认 `xchina` |
 | `start_page` / `end_page` | 否 | 默认 `1`/`start_page`(即只爬给定页/这一条);区间超 `MAX_PAGES` 会被截断 |
 | `minio_endpoint` / `minio_access_key` / `minio_secret_key` / `minio_bucket` | 否 | 四者齐全才启用上传;缺则降级。也可改由本服务 env 提供(`MINIO_*`),载荷优先 |
@@ -89,23 +99,40 @@ CLAUDE.md           # 给 AI / 维护者的实现速查
   "status": "running" | "succeeded" | "failed",
   "output": {                       // succeeded 时
     "resources": [
+      // —— 视频条目 ——
       {
-        "source_type": "video",
+        "source_type": "video", "content_type": "video",
         "source_name": "xchina",
         "source_url": "https://xchina.co/video/id-XXXX.html",   // 主站资源库按它去重
-        "content_type": "video",
-        "title": "...",
-        "summary": "...",            // 简介(可空)
+        "title": "...", "summary": "...",
         "content": "...",            // 正文(保证非空,实在没有就是标题)——主站对空正文拒收
         "cover_image": "<对象存储URL 或 原图床URL>",
         "video_url":   "<对象存储URL 或 原始m3u8URL>",
         "extra_data": {
           "referer": "...", "page_no": 1, "collector": "xchina", "video_id": "XXXX",
-          // 上传成功还有:
           "original_m3u8": "...", "original_cover": "...",
-          "minio_object_video": "xchina/XXXX.mp4", "minio_object_cover": "xchina/XXXX.jpg",
-          "video_bytes": 78167166
-          // 没上传则有:"upload_skipped_reason": "minio config absent | download failed: ... | video too large: ... | minio upload failed: ..."
+          "minio_object_video": "xchina/XXXX.mp4", "minio_object_cover": "xchina/XXXX.jpg", "video_bytes": 78167166
+          // 没上传则有:"upload_skipped_reason": "minio config absent | ... | video too large: ... | minio upload failed: ..."
+        }
+      },
+      // —— 图集条目 ——
+      {
+        "source_type": "image", "content_type": "article",      // 注意:content_type 是 article(主站枚举只有 article/video)
+        "source_name": "xchina",
+        "source_url": "https://xchina.co/photo/id-XXXX.html",
+        "title": "...", "summary": "模特 · 系列 · N张",
+        "content": "标题\n模特: .. · 系列: .. · 日期: .. · 共 N 张图 · 标签: ..",
+        "cover_image": "<已上传第一张的对象存储URL,或原图床URL>",
+        "images": ["<对象存储URL>", ...],                        // 主站对 images 数组会自动跑图片 OCR 审核(命中 ad/minor/non_frontal 的会被剔除;全空则资源 soft delete)
+        "video_url": null,
+        "extra_data": {
+          "referer": "...", "page_no": 1, "collector": "xchina", "video_id": "XXXX",
+          "image_total": 601, "image_count": 100, "image_capped": true,   // image_total=站点上的总张数,image_count=本次回传的张数(受 max_images 限)
+          "base_prefix": "https://img.xchina.io/photos2/XXXX", "detail_id": "XXXX",
+          "img_template": "https://img.xchina.io/photos2/XXXX/{n:04d}_600x0.webp",
+          "minio_objects_images": ["xchina/XXXX/0001.webp", ...], "images_uploaded": 100, "original_cover": "...",
+          "original_images_count": 100, "images_failed": 0           // images_failed 仅当有单张失败时出现
+          // 没上传则有:"upload_skipped_reason": "minio config absent | ..."
         }
       }
       // ...
@@ -139,6 +166,8 @@ CLAUDE.md           # 给 AI / 维护者的实现速查
 | `BROWSER_PROFILE_DIR` | `/data/browser_profile` | 持久浏览器资料目录(挂卷;可预置过了 Cloudflare 的 cookie 作为反爬退路) |
 | `ALLOWED_HOSTS` | `xchina.co` | `content_url` host 白名单(SSRF 防护),逗号分隔 |
 | `MAX_PAGES` | `50` | 单作业最多翻页数 |
+| `DEFAULT_MAX_IMAGES` | `100` | 图集每个套图默认抓多少张(载荷 `max_images` 覆盖) |
+| `MAX_IMAGES_CAP` | `2000` | 单套图抓图张数硬上限(载荷给再大也截到这) |
 | `CONCURRENT_FRAGMENTS` | `8` | yt-dlp 并发分片数 |
 | `MAX_VIDEO_MB` | `0` | 单视频大小上限(0=不限),超过则跳过上传、降级保留原媒体地址 |
 | `TMP_DIR` | `/tmp` | 临时下载目录(`<TMP_DIR>/content-harvester/<jobid>/`,用完即删) |
@@ -172,16 +201,8 @@ curl -s localhost:8019/healthz   # {"ok":true,"headless":false,...}
 
 ### 更新
 
-本仓库**没有 gitea 自动部署 workflow**,要手动:
-
-```bash
-cd <部署目录> && git pull origin dev
-docker build -t content-harvester .
-docker rm -f content-harvester
-docker run -d ... content-harvester    # 同上面那串参数
-```
-
-> 各环境的具体服务器、目录、容器、token / 凭据见 `DEPLOY.md` 与本地运维记录(不入库)。
+- **dev**:有自动部署 —— `.gitea/workflows/deploy.yml`,**push 到 `dev` 分支**(或 gitea 上 `workflow_dispatch`)→ dev 开发机的 runner 自动 `docker build` + 重建 `content-harvester` 容器 + healthz 检查。前置:在该仓库的 Gitea Actions secrets 里配一个 `CH_ENV_FILE`(等价于 `.env.dev` 全文,`HARVESTER_TOKEN` + 4 个 `MINIO_*` + 可选项)。
+- **prod / 手动**:`master`/prod 不走那个工作流,仍手动:`cd <部署目录> && git pull origin <dev|master> && docker build -t content-harvester . && docker rm -f content-harvester && docker run -d ... content-harvester`(完整命令、各环境的服务器/目录/容器/凭据见 `DEPLOY.md` 与本地运维记录,不入库)。
 
 ---
 
@@ -226,9 +247,10 @@ content-harvester:抓页面 → 下 m3u8 转 mp4 → 传 MinIO → 回传 output
 ### 已知行为 / 注意点
 
 - **主站资源库对空 `content` 直接拒收**(`reason:"正文为空,不予入库"`,且 `_maybe_ingest_to_resource_pool` 只记 log 不抛错 → 任务仍 SUCCEEDED 但资源没进库)。所以本服务保证每条 `content` 非空(抓不到描述就用标题兜底)。
-- 入库成功后主站会自动派 AI 改标题任务,资源库里的 `title` 会被润色掉(如「0144 小雪」→「小雪:怀旧风格丝袜摄影大赏」),属预期。文本内容审核、AI 正文润色也一并跑(和 TG 采集同一套);**视频封面图目前没走图片 OCR 审核**(那套只对 `images` 数组里的 URL 触发,而视频资源只有单个 `cover_image`)。
-- m3u8 媒体地址带 `expires` 签名约 1h 失效——所以才在采集时就转 mp4 上传对象存储;不上传的话资源库里那个 URL 一会儿就 403。
-- 性能:每条 ~15s 抓页 + 几十秒~几分钟下载 + 几秒~几十秒上传;一页 11 条实测 ~4 分钟(服务器带宽好的话)。`CRAWL_LOBSTER_TIMEOUT` 默认 7200s 够用。多页/大量视频会很慢,属预期。
+- 入库成功后主站会自动派 AI 改标题任务,资源库里的 `title` 会被润色掉(如「0144 小雪」→「小雪:怀旧风格丝袜摄影大赏」),属预期。文本内容审核、AI 正文润色也一并跑(和 TG 采集同一套)。
+- **图片 OCR 审核**:图集回传的 `images[]` 数组主站会自动逐张排 OCR 审核任务,命中 `BLOCK_TAGS`(ad/minor/non_frontal)的图会被从 `images` 里剔除;若一条资源最后 content/images/video 全空 → 主站把它 soft delete。所以图集抓得多 = OCR 任务多、且不少图会被判 minor/non_frontal 被剔——这是预期,`max_images` 就是用来控这个量级的。视频资源只有单个 `cover_image`(不在 `images` 里),不走这套 OCR。
+- m3u8 媒体地址带 `expires` 签名约 1h 失效——所以才在采集时就转 mp4 上传对象存储;不上传的话资源库里那个 URL 一会儿就 403。图集图 URL(`img.xchina.io/photos2/...`)不带签名、较稳定,但仍统一转存对象存储(脱离对源站可用性的依赖)。
+- 性能:视频每条 ~15s 抓页 + 几十秒~几分钟下载 + 上传;图集每个套图 ~5-10s 抓页 + 逐张下载上传(600px webp 每张几十 KB,100 张约 1-2 分钟)。`CRAWL_LOBSTER_TIMEOUT` 默认 7200s 够用。多页/大量内容会很慢,属预期(单作业串行)。
 
 ---
 

core/media.py

@@ -90,7 +90,7 @@ _VIDEO_EXTS = (".mp4", ".mkv", ".webm", ".mov", ".ts", ".m4v")
 
 
 def extract_video_id(detail_url: str) -> str:
-    m = re.search(r"/video[s]?/id-([A-Za-z0-9_-]+)\.html", detail_url)
+    m = re.search(r"/(?:video|videos|photo|photos)/id-([A-Za-z0-9_-]+)(?:/\d+)?\.html", detail_url)
     if m:
         return m.group(1)
     # 兜底:用 url 的 hash
@@ -98,6 +98,54 @@ def extract_video_id(detail_url: str) -> str:
     return "u" + hashlib.sha1(detail_url.encode("utf-8")).hexdigest()[:16]
 
 
+# 兼容旧名;图集详情页用同一个 id 提取逻辑
+extract_detail_id = extract_video_id
+
+
+_IMAGE_EXTS = (".jpg", ".jpeg", ".png", ".webp", ".gif", ".bmp")
+
+
+def download_image(image_url: str, referer: str, dest_dir: Path, name: str) -> Optional[Path]:
+    """流式下载一张图 → dest_dir/<name><ext>。失败返回 None(单张失败不致命)。"""
+    import requests
+
+    try:
+        ext = os.path.splitext(urlparse(image_url).path)[1].lower()
+        if ext not in _IMAGE_EXTS:
+            ext = ".webp"
+        dest_dir.mkdir(parents=True, exist_ok=True)
+        path = dest_dir / f"{name}{ext}"
+        with requests.get(image_url, headers={"Referer": referer, "User-Agent": _UA}, stream=True, timeout=30) as r:
+            r.raise_for_status()
+            with open(path, "wb") as f:
+                for chunk in r.iter_content(chunk_size=64 * 1024):
+                    if chunk:
+                        f.write(chunk)
+        if path.stat().st_size == 0:
+            path.unlink(missing_ok=True)
+            return None
+        return path
+    except Exception as exc:  # noqa: BLE001
+        logger.warning("image download failed %s: %s", image_url, exc)
+        return None
+
+
+def head_ok(url: str, referer: str = "", timeout: int = 15) -> bool:
+    """HEAD 探测一个 URL 是否可达(2xx)。失败/异常返回 False。"""
+    import requests
+
+    try:
+        r = requests.head(url, headers={"Referer": referer or url, "User-Agent": _UA},
+                          allow_redirects=True, timeout=timeout)
+        if r.status_code == 405:  # 有些 CDN 不支持 HEAD,退而用 GET 流
+            r = requests.get(url, headers={"Referer": referer or url, "User-Agent": _UA},
+                             stream=True, timeout=timeout)
+            r.close()
+        return 200 <= r.status_code < 300
+    except Exception:  # noqa: BLE001
+        return False
+
+
 def download_video(media_url: str, referer: str, dest_dir: Path, video_id: str,
                    concurrent_fragments: int = 8, max_mb: int = 0) -> Path:
     """用 yt-dlp 下载 m3u8/mp4 → dest_dir/<video_id>.mp4(尽量)。返回最终文件路径。

core/models.py

@@ -16,10 +16,17 @@ class ScrapedItem:
     title: str
     # 来源站点名,默认 "xchina"(可被 invoke 载荷里的 platform_name 覆盖)
     source_name: str = "xchina"
+    # 资源类型: "video"(视频) | "photo"(图集)
+    media_kind: str = "video"
     # 封面图 URL(上传 MinIO 后改为 MinIO 地址,否则是 xchina 图床地址)
     cover_image: str | None = None
-    # 视频 URL(上传 MinIO 后改为 MinIO 地址,否则是 xchina 限时签名 m3u8)
+    # 视频 URL(media_kind=video 时;上传 MinIO 后改为 MinIO 地址,否则是 xchina 限时签名 m3u8)
     video_url: str | None = None
+    # 图集图片 URL 列表(media_kind=photo 时;上传 MinIO 后逐个改为 MinIO 地址)
+    images: list[str] = field(default_factory=list)
+    # 图集原始总张数(images 可能因 max_images 被截断)
+    image_total: int = 0
+    image_capped: bool = False
     # 简介 / 正文(主站资源库要求 content 非空才入库;为空时回退用 title)
     summary: str | None = None
     content: str | None = None
@@ -38,9 +45,28 @@ class ScrapedItem:
         }
         if self.video_id:
             extra_data["video_id"] = self.video_id
+        if self.media_kind == "photo":
+            extra_data["image_total"] = self.image_total or len(self.images)
+            extra_data["image_count"] = len(self.images)
+            extra_data["image_capped"] = bool(self.image_capped)
         extra_data.update(self.extra or {})
         # 主站 ingest 对空 content 直接拒收 —— 兜底用 title,确保非空
         content = (self.content or "").strip() or (self.title or "").strip() or self.source_url
+        if self.media_kind == "photo":
+            return {
+                # 主站 SourceTypeEnum: image;ContentTypeEnum 只有 article/video,图集归 article
+                "source_type": "image",
+                "source_name": self.source_name,
+                "source_url": self.source_url,
+                "content_type": "article",
+                "title": self.title,
+                "cover_image": self.cover_image or (self.images[0] if self.images else None),
+                "summary": (self.summary or None),
+                "content": content,
+                "images": list(self.images),
+                "video_url": None,
+                "extra_data": extra_data,
+            }
         return {
             "source_type": "video",
             "source_name": self.source_name,

core/scraper.py

@@ -464,46 +464,46 @@ class XChinaScraper:
     # ---- description / content -------------------------------------------
 
     async def _extract_description(self, page, title: str) -> tuple[str | None, str]:
-        """返回 (summary, content)。content 保证非空(实在没有就用 title)——
-        主站资源库 ingest 对空 content 直接拒收。"""
+        """返回 (summary, content)。content 保证非空(实在没有就用 title)——主站资源库 ingest 对空 content 直接拒收。
+
+        xchina 视频详情页**没有真正的简介文本**:`.video-detail` 面板就是 标题 / 模特 / 分类 / 系列名,
+        再加一整块「推广：...」广告位(里面是 `a.cpc-text` → xchina.click/prepare.html 的联盟跳转,常含
+        VPN/楼凤/约炮乃至「幼女」「学生妹」这类违规词,会被主站内容审核 block)。`meta description` 也只是
+        站点级 boilerplate。所以这里只抓干净的结构化字段:模特名 + 分类/系列名(限同源 xchina.co,排除 cpc-text /
+        xchina.click / prepare.html),拼成 content;不要面板 innerText、不要 boilerplate meta/jsonld。
+        """
         data = await page.evaluate(
             """() => {
                 function clean(t){ return (t || '').replace(/\\s+/g,' ').trim(); }
-                const out = {};
-                const m = document.querySelector('meta[name="description"], meta[property="og:description"], meta[name="twitter:description"]');
-                out.meta_desc = clean(m && m.getAttribute('content'));
-                for (const s of document.querySelectorAll('script[type="application/ld+json"]')) {
-                    try { const d = JSON.parse(s.textContent||'null'); const rows = Array.isArray(d)?d:[d];
-                        for (const r of rows) { if (r && r.description) { out.jsonld_desc = clean(r.description); break; } } } catch(_){}
+                const out = {model: '', cats: []};
+                const ma = document.querySelector('.video-detail a[href*="/model/id-"], a.model-item, .model-container a[href*="/model/id-"]');
+                out.model = clean(ma && ma.innerText);
+                const seen = new Set();
+                for (const a of document.querySelectorAll('.video-detail a, .video-info a, .breadcrumb a')) {
+                    if (a.classList && a.classList.contains('cpc-text')) continue;       // 广告位链接
+                    let h; try { h = new URL(a.href, location.href); } catch(e){ continue; }
+                    if (!h.host.endsWith('xchina.co')) continue;                          // 排除 xchina.click 等广告域
+                    if (h.pathname.indexOf('prepare.html') >= 0) continue;
+                    const ok = h.pathname.indexOf('/videos/series-') >= 0 || h.pathname.indexOf('/photos/series-') >= 0
+                        || h.pathname.indexOf('/tags/') === 0 || h.pathname.indexOf('/tag/') === 0;
+                    if (!ok) continue;
+                    const t = clean(a.innerText);
+                    if (t && t.length <= 24 && !seen.has(t)) { seen.add(t); out.cats.push(t); }
                 }
-                // 影片信息 / 详情面板里的文本(键值对、简介等)
-                const panel = document.querySelector('.video-detail') || document.querySelector('.tab-content.active') || document.querySelector('.tab-content');
-                out.panel = panel ? clean(panel.innerText).slice(0, 4000) : '';
-                // 标签
-                const tags = [];
-                for (const a of document.querySelectorAll('.tags a, .tag a, a[href*="/tags/"], .video-detail .item a')) {
-                    const t = clean(a.innerText); if (t && t.length<=20 && !tags.includes(t)) tags.push(t);
-                }
-                out.tags = tags.slice(0, 30);
+                out.cats = out.cats.slice(0, 8);
                 return out;
             }"""
         )
-        meta_desc = (data.get("meta_desc") or "").strip()
-        jsonld_desc = (data.get("jsonld_desc") or "").strip()
-        panel = (data.get("panel") or "").strip()
-        tags = data.get("tags") or []
-        # content: 优先详情面板文本(信息量最大),否则 og/jsonld 描述,再否则 标签拼接,最后 title
+        model = (data.get("model") or "").strip()
+        cats = data.get("cats") or []
         parts: list[str] = []
-        body = panel or jsonld_desc or meta_desc
-        if body:
-            parts.append(body)
-        if tags:
-            parts.append("标签: " + " ".join(tags))
-        content = "\n".join(parts).strip()
-        if not content:
-            content = title
-        # summary: og/jsonld 描述,或 content 头一截
-        summary = (meta_desc or jsonld_desc or content)[:300].strip() or None
+        if model:
+            parts.append(f"模特: {model}")
+        if cats:
+            parts.append("分类: " + " / ".join(cats))
+        content = ("\n".join([title, " · ".join(parts)]).strip() if parts else (title or "").strip()) or (title or "")
+        bits = [b for b in [model, *cats[:3]] if b]
+        summary = " · ".join(bits) if bits else None
         return summary, content
 
     # ---- cover -----------------------------------------------------------

core/scraper_photos.py

@@ -0,0 +1,406 @@
+"""xchina.co 图集(photos series)采集 —— 抓系列页里的套图详情页,提取标题/封面/全部图片 URL。
+
+图片 URL 模板(实测):https://img.xchina.io/photos2/{detailId}/{NNNN}_600x0.webp
+  - NNNN: 四位补零的 1-based 序号
+  - _600x0: 600px 宽缩放变体 —— 就用它,体积小(几十 KB),不追原图
+  - 旧套图前缀可能是 .../photos/{id}/...(不带 2) —— 从页面渲染出的实际 bg-image 取,不硬编码
+详情页本身分页:/photo/id-{detailId}/{N}.html,每页约 14 张。
+
+复用 XChinaScraper 的导航/反爬 helper(_visit / _assert_not_blocked / _page_url_candidates / _check_cancel)。
+"""
+from __future__ import annotations
+
+import math
+import re
+from urllib.parse import urlparse
+
+from playwright.async_api import TimeoutError as PlaywrightTimeoutError
+
+from .media import extract_detail_id, head_ok
+from .models import ScrapedItem, ScrapeResult
+from .scraper import ScrapeCancelled, XChinaScraper
+
+# 详情页一页大约多少张图(用于从分页数估总数)
+_PER_DETAIL_PAGE = 14
+# 解析图片 URL:.../{NNNN}{_WxH}?.{ext}
+_IMG_URL_RE = re.compile(
+    r"^(?P<prefix>https?://[^\s'\"]+?)/(?P<seg>\d{2,6})(?P<size>_\d+x\d+)?\.(?P<ext>webp|jpe?g|png)(?:\?[^\s'\"]*)?$",
+    re.I,
+)
+_BAD_IMG_PARTS = ("/images/sites/", "/model/", "/media/", "/css/avatars", "/upload/", "logo", "banner", "icon", "avatar")
+
+
+def is_photo_detail_url(url: str) -> bool:
+    path = urlparse(url).path.lower()
+    return path.endswith(".html") and re.search(r"/photo/id-[a-z0-9_-]+(?:/\d+)?\.html$", path) is not None
+
+
+def is_photo_series_url(url: str) -> bool:
+    return "/photos/series-" in urlparse(url).path.lower()
+
+
+def _canon_detail_url(url: str) -> str:
+    """/photo/id-XXX/3.html -> /photo/id-XXX.html"""
+    return re.sub(r"(/photo/id-[A-Za-z0-9_-]+)/\d+\.html$", r"\1.html", url.split("#", 1)[0])
+
+
+class XChinaPhotoScraper(XChinaScraper):
+    def __init__(self, context, is_cancelled=lambda: False, log=lambda _m: None,
+                 items_per_page: int = 14, max_images: int = 100):
+        super().__init__(context, is_cancelled=is_cancelled, log=log, items_per_page=items_per_page)
+        self.max_images = max(1, int(max_images or 100))
+
+    # ---- top level -------------------------------------------------------
+
+    async def run(self, series_url: str, start_page: int = 1, end_page: int = 1) -> ScrapeResult:
+        result = ScrapeResult()
+        if is_photo_detail_url(series_url):
+            self.log(f"输入是单个图集详情页,直接采集: {series_url}")
+            item = await self._scrape_photo_safe(_canon_detail_url(series_url), 1, result)
+            if item:
+                result.resources.append(item)
+            result.pages_scanned = 1
+            return result
+
+        list_page = await self.context.new_page()
+        try:
+            for page_no in range(start_page, end_page + 1):
+                self._check_cancel()
+                ok = await self._go_series_page_photos(list_page, series_url, page_no)
+                if not ok:
+                    self.log(f"第 {page_no} 页不存在或无图集,停止翻页。")
+                    break
+                result.pages_scanned += 1
+                links = await self._collect_photo_links(list_page)
+                if not links:
+                    self.log(f"第 {page_no} 页没识别到图集链接。")
+                    continue
+                self.log(f"第 {page_no} 页识别到 {len(links)} 个图集链接。")
+                for detail_url, hint_title in links:
+                    self._check_cancel()
+                    item = await self._scrape_photo_safe(detail_url, page_no, result, hint_title=hint_title)
+                    if item:
+                        result.resources.append(item)
+            return result
+        finally:
+            await list_page.close()
+
+    async def _scrape_photo_safe(self, detail_url, page_no, result, hint_title=None):
+        try:
+            return await self.scrape_photo_detail(detail_url, page_no, hint_title=hint_title)
+        except ScrapeCancelled:
+            raise
+        except Exception as exc:  # noqa: BLE001
+            self.log(f"图集详情页采集失败,跳过: {detail_url} -> {exc}")
+            result.item_errors.append({"source_url": detail_url, "reason": f"{type(exc).__name__}: {exc}"})
+            return None
+
+    # ---- list page -------------------------------------------------------
+
+    async def _go_series_page_photos(self, page, series_url: str, page_no: int) -> bool:
+        for candidate in self._page_url_candidates(series_url, page_no):
+            try:
+                await self._visit(page, candidate)
+                await page.wait_for_timeout(300)
+                await self._assert_not_blocked(page)
+                if await self._collect_photo_links(page):
+                    return True
+            except RuntimeError:
+                raise
+            except Exception as exc:  # noqa: BLE001
+                self.log(f"访问候选分页失败: {candidate} -> {exc}")
+        return False
+
+    async def _collect_photo_links(self, page) -> list[tuple[str, str]]:
+        await self._assert_not_blocked(page)
+        current = page.url
+        rows = await page.evaluate(
+            """() => {
+                const out = [];
+                for (const item of document.querySelectorAll('.item.photo, .list .item')) {
+                    const a = item.querySelector('a[href*="/photo/id-"]');
+                    if (!a) continue;
+                    const href = new URL(a.getAttribute('href'), location.href).href;
+                    let t = '';
+                    const te = item.querySelector('.title a') || item.querySelector('.title');
+                    if (te) t = te.innerText || '';
+                    if (!t) t = a.getAttribute('title') || '';
+                    out.push({href: href, title: t.replace(/\\s+/g, ' ').trim()});
+                }
+                return out;
+            }"""
+        )
+        seen, links = set(), []
+        for row in rows:
+            href = _canon_detail_url(row.get("href") or "")
+            if not href or href in seen or href == current or not is_photo_detail_url(href):
+                continue
+            seen.add(href)
+            links.append((href, row.get("title") or ""))
+            if len(links) >= self.items_per_page:
+                break
+        return links
+
+    # ---- detail page -----------------------------------------------------
+
+    async def scrape_photo_detail(self, detail_url: str, page_no: int, hint_title: str | None = None) -> ScrapedItem:
+        detail_url = _canon_detail_url(detail_url)
+        page = await self.context.new_page()
+        try:
+            await self._visit(page, detail_url)
+            await self._assert_not_blocked(page)
+            await self._wait_photo_detail(page, detail_url)
+
+            title = await self._extract_photo_title(page, hint_title)
+            meta = await self._extract_photo_meta(page)            # {model, series, date, tags, count}
+            bg_first = await self._photo_bg_images(page)           # 第 1 页渲染出的图片 URL(按 DOM 顺序)
+            detail_id = extract_detail_id(detail_url)
+
+            template = self._parse_img_template(bg_first, detail_id)
+            total = int(meta.get("count") or 0)
+            if not total:
+                total = await self._estimate_total_from_pager(page) or len(bg_first)
+            total = max(total, len(bg_first))
+
+            wanted = min(total, self.max_images) if total else self.max_images
+            images: list[str] = []
+            if template and wanted > 0:
+                images = [template.format(n=i) for i in range(1, wanted + 1)]
+                if not head_ok(images[-1], referer=detail_url):
+                    self.log(f"图集 {detail_id}: 构造 URL 末张不可达,回退逐页抓取")
+                    images = []
+            if not images:
+                images = await self._scrape_all_pages_images(page, detail_url, limit=wanted or self.max_images)
+            if not images:
+                images = bg_first[: self.max_images]
+
+            capped = bool(total) and total > len(images)
+            cover = images[0] if images else (bg_first[0] if bg_first else None)
+            eff_total = total or len(images)
+            content = self._compose_content(title, meta, eff_total)
+            summary = self._compose_summary(meta, eff_total) or None
+
+            self.log(
+                f"图集采集: {title} | 图 {len(images)}/{total or '?'}"
+                f"{' (已截断)' if capped else ''} | cover={'有' if cover else '无'}"
+            )
+            return ScrapedItem(
+                source_url=detail_url,
+                title=title,
+                media_kind="photo",
+                cover_image=cover,
+                images=images,
+                image_total=eff_total,
+                image_capped=capped,
+                summary=summary,
+                content=content,
+                page_no=page_no,
+                video_id=detail_id,
+                extra={
+                    "base_prefix": (template.split("/{", 1)[0] if template else ""),
+                    "detail_id": detail_id,
+                    "img_template": template or "",
+                },
+            )
+        finally:
+            await page.close()
+
+    async def _wait_photo_detail(self, page, expected_url: str) -> None:
+        """图集详情页没有 .video-detail/.tab-nav;等图片容器出现,顺手处理中转广告页。"""
+        sel = ".list.photo-items, .item.photo-image, div.img[role='img']"
+        want_id = (re.search(r"/photo/id-([A-Za-z0-9_-]+)", expected_url) or [None, ""])[1]
+        for _ in range(3):
+            self._check_cancel()
+            await page.wait_for_timeout(900)
+            parsed = urlparse(page.url)
+            cur_id = (re.search(r"/photo/id-([A-Za-z0-9_-]+)", page.url) or [None, ""])[1]
+            on_detail = parsed.netloc.endswith("xchina.co") and cur_id == want_id and bool(want_id)
+            if on_detail:
+                try:
+                    if await page.locator(sel).count():
+                        return
+                except Exception:
+                    pass
+            else:
+                # 可能被弹到广告/中转页
+                try:
+                    await page.go_back(wait_until="domcontentloaded", timeout=20000)
+                    await page.wait_for_timeout(1000)
+                except Exception:
+                    await self._visit(page, expected_url)
+        cur_id = (re.search(r"/photo/id-([A-Za-z0-9_-]+)", page.url) or [None, ""])[1]
+        if cur_id != want_id:
+            await self._visit(page, expected_url)
+        try:
+            await page.wait_for_selector(sel, timeout=10000)
+        except PlaywrightTimeoutError:
+            pass
+
+    async def _photo_bg_images(self, page) -> list[str]:
+        """页面里所有套图缩略 div(div.img[role=img] 等)的 background-image URL,按 DOM 顺序去重。"""
+        urls = await page.evaluate(
+            """() => {
+                const out = [];
+                const els = document.querySelectorAll("div.img[role='img'], .item.photo-image .img, .photo-items .img");
+                for (const el of els) {
+                    let s = (getComputedStyle(el).backgroundImage || el.style.backgroundImage || '');
+                    const m = s.match(/url\\((["']?)(.*?)\\1\\)/);
+                    if (m && m[2]) { try { out.push(new URL(m[2], location.href).href); } catch (e) {} }
+                }
+                return out;
+            }"""
+        )
+        seen, res = set(), []
+        for u in urls:
+            if not u or u in seen:
+                continue
+            low = u.lower()
+            if not any(d in low for d in ("xchina.io", "xchina.co", "xchina.click")):
+                continue
+            if any(w in low for w in _BAD_IMG_PARTS):
+                continue
+            seen.add(u)
+            res.append(u)
+        return res
+
+    def _parse_img_template(self, urls: list[str], detail_id: str) -> str | None:
+        """从一批图片 URL 推 .../{NNNN}{size}.{ext} 模板,返回带 {n} 占位的 .format 串。"""
+        best = None
+        for u in urls:
+            m = _IMG_URL_RE.match(u)
+            if not m:
+                continue
+            prefix, seg, size, ext = m.group("prefix"), m.group("seg"), m.group("size") or "", m.group("ext")
+            width = len(seg)
+            tmpl = f"{prefix}/{{n:0{width}d}}{size}.{ext}"
+            if detail_id and detail_id in prefix:
+                return tmpl
+            best = best or tmpl
+        return best
+
+    async def _estimate_total_from_pager(self, page) -> int:
+        try:
+            nums = await page.eval_on_selector_all(
+                ".pager a.pager-num, .pager .pager-num, [class*=pager] a",
+                "els => els.map(e => parseInt((e.innerText || '').trim(), 10)).filter(n => !isNaN(n))",
+            )
+            if nums:
+                return max(nums) * _PER_DETAIL_PAGE
+        except Exception:
+            pass
+        return 0
+
+    async def _scrape_all_pages_images(self, page, detail_url: str, limit: int) -> list[str]:
+        """逐页加载 /photo/id-{id}/{N}.html 收集 bg-image,直到够 limit 张或没有新图。"""
+        base = re.sub(r"(/photo/id-[A-Za-z0-9_-]+)(?:/\d+)?\.html$", r"\1", detail_url)
+        collected: list[str] = list(await self._photo_bg_images(page))  # 第 1 页就是当前 page
+        max_pages = max(1, math.ceil(max(limit, 1) / _PER_DETAIL_PAGE)) + 2
+        n = 2
+        while len(collected) < limit and n <= max_pages:
+            self._check_cancel()
+            url = f"{base}/{n}.html"
+            try:
+                await self._visit(page, url)
+                await self._assert_not_blocked(page)
+                await page.wait_for_timeout(400)
+            except Exception as exc:  # noqa: BLE001
+                self.log(f"图集翻页失败 {url}: {exc}")
+                break
+            new = [u for u in await self._photo_bg_images(page) if u not in collected]
+            if not new:
+                break
+            collected.extend(new)
+            n += 1
+        seen, res = set(), []
+        for u in collected:
+            if u not in seen:
+                seen.add(u)
+                res.append(u)
+        return res[:limit]
+
+    # ---- title / meta / content -----------------------------------------
+
+    async def _extract_photo_title(self, page, hint_title: str | None) -> str:
+        t = await page.evaluate(
+            """() => {
+                function clean(s){ return (s || '').replace(/\\s+/g, ' ').trim(); }
+                const h = document.querySelector('h1, .photo-title, .video-detail .text');
+                const ht = clean(h && h.innerText);
+                if (ht && ht.length >= 2 && ht.length <= 200) return ht;
+                const og = document.querySelector('meta[property="og:title"], meta[name="twitter:title"]');
+                const ot = clean(og && og.getAttribute('content'));
+                if (ot) return ot;
+                return clean(document.title);
+            }"""
+        )
+        t = self._strip_title_suffix(t)
+        bad = ("cloudflare", "attention required", "you have been blocked")
+        if not (t and len(t.strip()) >= 2) or any(b in t.lower() for b in bad):
+            t = self._strip_title_suffix(hint_title or "")
+        if not (t and len(t.strip()) >= 2):
+            m = re.search(r"/photo/id-([A-Za-z0-9_-]+)", page.url)
+            t = f"photos-{m.group(1)}" if m else "未命名图集"
+        return t[:500]
+
+    @staticmethod
+    def _strip_title_suffix(t: str) -> str:
+        # "标题 - 系列 - 中国工作室 - 小黄书 xChina - 第 2 页" -> "标题"
+        t = (t or "").strip()
+        t = re.sub(r"\s*-\s*第\s*\d+\s*页\s*$", "", t)
+        t = re.split(r"\s*-\s*(小黄书|xChina|XChina)\b.*$", t)[0]
+        parts = [p.strip() for p in t.split(" - ") if p.strip()]
+        return (parts[0] if parts else t).strip()
+
+    async def _extract_photo_meta(self, page) -> dict:
+        return await page.evaluate(
+            """() => {
+                function clean(s){ return (s || '').replace(/\\s+/g, ' ').trim(); }
+                const out = {model: '', series: '', date: '', tags: [], count: 0};
+                const ma = document.querySelector('.model-container a, a.model-item, a[href*="/model/id-"]');
+                out.model = clean(ma && ma.innerText);
+                // 系列名优先取面包屑 / .subs 里的(侧边栏分类也是 /photos/series- 链接,会误命中)
+                const sa = document.querySelector(
+                    '.breadcrumb a[href*="/photos/series-"], .subs a[href*="/photos/series-"], '
+                    + '.path a[href*="/photos/series-"], .nav-path a[href*="/photos/series-"]')
+                    || document.querySelector('a[href*="/photos/series-"]');
+                out.series = clean(sa && sa.innerText);
+                const tags = [];
+                for (const a of document.querySelectorAll('a[href*="/tags/"], .tags a, .tag a')) {
+                    if (a.classList && a.classList.contains('cpc-text')) continue;       // 广告位链接
+                    let h; try { h = new URL(a.href, location.href); } catch(e){ continue; }
+                    if (!h.host.endsWith('xchina.co') || h.pathname.indexOf('prepare.html') >= 0) continue;
+                    const t = clean(a.innerText);
+                    if (t && t.length <= 20 && tags.indexOf(t) < 0) tags.push(t);
+                }
+                out.tags = tags.slice(0, 30);
+                const bt = clean(document.body.innerText);
+                let m = bt.match(/(\\d{4})[.\\-/](\\d{1,2})[.\\-/](\\d{1,2})/);
+                if (m) out.date = m[1] + '.' + ('0' + m[2]).slice(-2) + '.' + ('0' + m[3]).slice(-2);
+                m = bt.match(/(\\d{1,5})\\s*P\\b/);
+                if (m) out.count = parseInt(m[1], 10);
+                return out;
+            }"""
+        ) or {}
+
+    @staticmethod
+    def _compose_content(title: str, meta: dict, total: int) -> str:
+        parts = []
+        if meta.get("model"):
+            parts.append(f"模特: {meta['model']}")
+        if meta.get("series"):
+            parts.append(f"系列: {meta['series']}")
+        if meta.get("date"):
+            parts.append(f"日期: {meta['date']}")
+        if total:
+            parts.append(f"共 {total} 张图")
+        if meta.get("tags"):
+            parts.append("标签: " + " ".join(meta["tags"]))
+        body = " · ".join(parts)
+        return (f"{title}\n{body}".strip() if body else (title or "").strip()) or (title or "")
+
+    @staticmethod
+    def _compose_summary(meta: dict, total: int) -> str:
+        bits = [b for b in (meta.get("model"), meta.get("series")) if b]
+        if total:
+            bits.append(f"{total}张")
+        return " · ".join(bits)

server/app.py

@@ -29,6 +29,8 @@ ALLOWED_HOSTS = tuple(
     h.strip().lower() for h in os.environ.get("ALLOWED_HOSTS", "xchina.co").split(",") if h.strip()
 )
 MAX_PAGES = int(os.environ.get("MAX_PAGES", "50"))
+DEFAULT_MAX_IMAGES = int(os.environ.get("DEFAULT_MAX_IMAGES", "100"))
+MAX_IMAGES_CAP = int(os.environ.get("MAX_IMAGES_CAP", "2000"))
 
 app = FastAPI(title="content-harvester", version="1.0")
 
@@ -91,14 +93,34 @@ async def invoke(agent_id: str, payload: dict, authorization: str | None = Heade
 
     source_name = str(payload.get("platform_name") or "xchina")[:100]
 
+    # 媒体类型:可由 payload.media_kind 显式指定(video/photo);否则按 content_url 自动判断;
+    # 再否则按 agent_id 兜底(含 photo → photo)。
+    media_kind = str(payload.get("media_kind") or "").strip().lower()
+    if media_kind not in ("video", "photo"):
+        path_low = urlparse(content_url).path.lower()
+        if "/photos/" in path_low or "/photo/id-" in path_low:
+            media_kind = "photo"
+        elif "/videos/" in path_low or "/video/id-" in path_low:
+            media_kind = "video"
+        elif "photo" in agent_id.lower():
+            media_kind = "photo"
+        else:
+            media_kind = "auto"
+    # 图集每个套图最多抓多少张
+    max_images = _as_int(payload.get("max_images"), DEFAULT_MAX_IMAGES)
+    max_images = max(1, min(max_images, MAX_IMAGES_CAP))
+
     # MinIO 配置由主站经 lobster payload 下发(主站把绑定的 minio_* Skill 变量并入 payload);
-    # 缺失则降级:不上传,video_url 保留原始 m3u8 地址
+    # 缺失则降级:不上传,媒体地址保留原始来源地址
     minio_cfg = MinioConfig.from_payload(payload)
 
-    job = jobs.submit(content_url, source_name, start_page, end_page, headless=HEADLESS, minio_cfg=minio_cfg)
+    job = jobs.submit(
+        content_url, source_name, start_page, end_page,
+        headless=HEADLESS, minio_cfg=minio_cfg, media_kind=media_kind, max_images=max_images,
+    )
     logger.info(
-        "invoke agent=%s job=%s url=%s pages=%s-%s minio=%s",
-        agent_id, job.id, content_url, start_page, end_page,
+        "invoke agent=%s job=%s kind=%s url=%s pages=%s-%s max_images=%s minio=%s",
+        agent_id, job.id, media_kind, content_url, start_page, end_page, max_images,
         "on" if minio_cfg.is_complete else "off",
     )
     return {"task_id": job.id, "status": job.status}

server/jobs.py

@@ -22,6 +22,7 @@ from core.browser import browser_context
 from core.media import (
     MinioConfig,
     download_cover,
+    download_image,
     download_video,
     guess_content_type,
     upload_to_minio,
@@ -45,6 +46,8 @@ class Job:
     start_page: int
     end_page: int
     minio_cfg: MinioConfig = field(default_factory=MinioConfig)
+    media_kind: str = "auto"          # auto | video | photo
+    max_images: int = 100             # 图集每个套图最多抓多少张(media_kind=photo 时)
     status: str = "running"           # running | succeeded | failed
     output: Optional[dict[str, Any]] = None
     error: Optional[str] = None
@@ -92,8 +95,7 @@ def _bytes_h(n: int) -> str:
 
 
 async def _process_uploads(job: Job, result, tmp_dir: Path) -> None:
-    """对 result.resources 逐条:下载视频+封面 → 上传 MinIO → 回写 video_url/cover_image。失败降级。"""
-    cfg = job.minio_cfg
+    """对 result.resources 逐条:下载媒体 → 上传 MinIO → 回写地址。失败降级保留原地址。"""
     total = len(result.resources)
     for idx, item in enumerate(result.resources, 1):
         if job._cancelled:
@@ -101,69 +103,147 @@ async def _process_uploads(job: Job, result, tmp_dir: Path) -> None:
         vid = item.video_id or f"item{idx}"
         item_dir = tmp_dir / vid
         try:
-            if not item.video_url:
-                item.extra["upload_skipped_reason"] = "no video url"
-                _log(job, f"[{idx}/{total}] {item.title}: 无视频地址,跳过上传")
-                continue
-            _log(job, f"[{idx}/{total}] {item.title}: 下载视频 …")
-            mp4_path = await asyncio.to_thread(
-                download_video, item.video_url, item.source_url, item_dir, vid,
-                CONCURRENT_FRAGMENTS, MAX_VIDEO_MB,
-            )
-            size = mp4_path.stat().st_size
-            video_key = f"xchina/{vid}{mp4_path.suffix.lower() or '.mp4'}"
-            _log(job, f"[{idx}/{total}] {item.title}: 视频 {_bytes_h(size)},上传 MinIO {video_key} …")
-            video_public = await asyncio.to_thread(
-                upload_to_minio, mp4_path, video_key, "video/mp4", cfg,
-            )
-            item.extra["original_m3u8"] = item.video_url
-            item.extra["minio_object_video"] = video_key
-            item.extra["video_bytes"] = size
-            item.video_url = video_public
-
-            # 封面(失败不致命)
-            if item.cover_image:
-                cover_path = await asyncio.to_thread(
-                    download_cover, item.cover_image, item.source_url, item_dir, vid,
-                )
-                if cover_path is not None:
-                    cover_key = f"xchina/{vid}{cover_path.suffix.lower() or '.jpg'}"
-                    try:
-                        cover_public = await asyncio.to_thread(
-                            upload_to_minio, cover_path, cover_key, guess_content_type(cover_path, "image/jpeg"), cfg,
-                        )
-                        item.extra["original_cover"] = item.cover_image
-                        item.extra["minio_object_cover"] = cover_key
-                        item.cover_image = cover_public
-                    except Exception as exc:  # noqa: BLE001
-                        _log(job, f"[{idx}/{total}] {item.title}: 封面上传失败({exc}),保留原图床地址")
-            _log(job, f"[{idx}/{total}] {item.title}: OK -> {item.video_url}")
+            if getattr(item, "media_kind", "video") == "photo":
+                await _process_photo_item(job, item, idx, total, item_dir)
+            else:
+                await _process_video_item(job, item, idx, total, vid, item_dir)
         except ScrapeCancelled:
             raise
         except Exception as exc:  # noqa: BLE001
             reason = f"{type(exc).__name__}: {exc}"
             item.extra["upload_skipped_reason"] = reason
-            _log(job, f"[{idx}/{total}] {item.title}: 下载/上传失败,降级保留原 m3u8 地址 -> {reason}")
+            _log(job, f"[{idx}/{total}] {item.title}: 下载/上传失败,降级保留原地址 -> {reason}")
         finally:
             shutil.rmtree(item_dir, ignore_errors=True)
 
 
+async def _process_video_item(job: Job, item, idx: int, total: int, vid: str, item_dir: Path) -> None:
+    cfg = job.minio_cfg
+    if not item.video_url:
+        item.extra["upload_skipped_reason"] = "no video url"
+        _log(job, f"[{idx}/{total}] {item.title}: 无视频地址,跳过上传")
+        return
+    _log(job, f"[{idx}/{total}] {item.title}: 下载视频 …")
+    mp4_path = await asyncio.to_thread(
+        download_video, item.video_url, item.source_url, item_dir, vid,
+        CONCURRENT_FRAGMENTS, MAX_VIDEO_MB,
+    )
+    size = mp4_path.stat().st_size
+    video_key = f"xchina/{vid}{mp4_path.suffix.lower() or '.mp4'}"
+    _log(job, f"[{idx}/{total}] {item.title}: 视频 {_bytes_h(size)},上传 MinIO {video_key} …")
+    video_public = await asyncio.to_thread(upload_to_minio, mp4_path, video_key, "video/mp4", cfg)
+    item.extra["original_m3u8"] = item.video_url
+    item.extra["minio_object_video"] = video_key
+    item.extra["video_bytes"] = size
+    item.video_url = video_public
+
+    if item.cover_image:
+        cover_path = await asyncio.to_thread(download_cover, item.cover_image, item.source_url, item_dir, vid)
+        if cover_path is not None:
+            cover_key = f"xchina/{vid}{cover_path.suffix.lower() or '.jpg'}"
+            try:
+                cover_public = await asyncio.to_thread(
+                    upload_to_minio, cover_path, cover_key, guess_content_type(cover_path, "image/jpeg"), cfg,
+                )
+                item.extra["original_cover"] = item.cover_image
+                item.extra["minio_object_cover"] = cover_key
+                item.cover_image = cover_public
+            except Exception as exc:  # noqa: BLE001
+                _log(job, f"[{idx}/{total}] {item.title}: 封面上传失败({exc}),保留原图床地址")
+    _log(job, f"[{idx}/{total}] {item.title}: OK -> {item.video_url}")
+
+
+async def _process_photo_item(job: Job, item, idx: int, total: int, item_dir: Path) -> None:
+    """逐张下载图集图片 → 上传 MinIO → 把 item.images 换成 MinIO 地址(边下边传边删,不囤盘)。"""
+    cfg = job.minio_cfg
+    detail_id = item.video_id or f"item{idx}"
+    src_images = list(item.images or [])
+    if not src_images:
+        item.extra["upload_skipped_reason"] = "no images"
+        _log(job, f"[{idx}/{total}] {item.title}: 图集无图片地址,跳过上传")
+        return
+    n = len(src_images)
+    _log(job, f"[{idx}/{total}] {item.title}: 图集 {n} 张,逐张下载并上传 MinIO …")
+    out_urls: list[str] = []
+    out_keys: list[str] = []
+    ok_cnt = fail_cnt = 0
+    cover_public: str | None = None
+    for i, src in enumerate(src_images, 1):
+        if job._cancelled:
+            raise ScrapeCancelled()
+        path = await asyncio.to_thread(download_image, src, item.source_url, item_dir, f"{i:04d}")
+        if path is None:
+            out_urls.append(src)  # 单张失败:保留原图床地址(img.xchina.io 链接较稳定,不签名)
+            fail_cnt += 1
+            continue
+        ext = path.suffix.lower() or ".webp"
+        key = f"xchina/{detail_id}/{i:04d}{ext}"
+        try:
+            public = await asyncio.to_thread(
+                upload_to_minio, path, key, guess_content_type(path, "image/webp"), cfg,
+            )
+            out_urls.append(public)
+            out_keys.append(key)
+            if cover_public is None and item.cover_image and item.cover_image == src:
+                cover_public = public
+            ok_cnt += 1
+        except Exception as exc:  # noqa: BLE001
+            out_urls.append(src)
+            fail_cnt += 1
+            _log(job, f"[{idx}/{total}] {item.title}: 第 {i} 张上传失败({exc}),保留原图床地址")
+        finally:
+            path.unlink(missing_ok=True)
+        if i % 25 == 0:
+            _log(job, f"[{idx}/{total}] {item.title}: 进度 {i}/{n}(成功 {ok_cnt} / 失败 {fail_cnt})")
+    item.images = out_urls
+    item.extra["original_images_count"] = n
+    item.extra["minio_objects_images"] = out_keys
+    item.extra["images_uploaded"] = ok_cnt
+    if fail_cnt:
+        item.extra["images_failed"] = fail_cnt
+    # 封面:优先用上传成功的第一张
+    if cover_public is None:
+        first_minio = next((u for u in out_urls if "://" in u and u not in src_images), None)
+        cover_public = first_minio or (out_urls[0] if out_urls else item.cover_image)
+    item.extra["original_cover"] = item.cover_image
+    item.cover_image = cover_public
+    _log(job, f"[{idx}/{total}] {item.title}: OK 图集 {ok_cnt}/{n} 张已上传 MinIO")
+
+
+def _resolve_media_kind(job: Job) -> str:
+    kind = (job.media_kind or "auto").lower()
+    if kind in ("video", "photo"):
+        return kind
+    from core.scraper_photos import is_photo_detail_url, is_photo_series_url
+    url = job.series_url
+    if is_photo_series_url(url) or is_photo_detail_url(url):
+        return "photo"
+    return "video"
+
+
+def _make_scraper(kind: str, ctx, job: Job):
+    if kind == "photo":
+        from core.scraper_photos import XChinaPhotoScraper
+        return XChinaPhotoScraper(
+            ctx, is_cancelled=lambda: job._cancelled, log=lambda m: _log(job, m), max_images=job.max_images,
+        )
+    return XChinaScraper(ctx, is_cancelled=lambda: job._cancelled, log=lambda m: _log(job, m))
+
+
 async def _run(job: Job, headless: bool) -> None:
     tmp_dir = TMP_ROOT / job.id
     try:
         async with _RUN_LOCK:
-            _log(job, f"开始采集: {job.series_url} (page {job.start_page}-{job.end_page})")
+            kind = _resolve_media_kind(job)
+            extra = f", max_images={job.max_images}" if kind == "photo" else ""
+            _log(job, f"开始采集[{kind}]: {job.series_url} (page {job.start_page}-{job.end_page}{extra})")
             if job.minio_cfg.is_complete:
                 _log(job, f"启用 MinIO 上传: {job.minio_cfg.safe_repr()}")
             else:
-                _log(job, "未提供完整 MinIO 配置,本次不上传(video_url 保留原始 m3u8 地址)")
+                _log(job, "未提供完整 MinIO 配置,本次不上传(媒体地址保留原始来源地址)")
 
             async with browser_context(headless=headless) as ctx:
-                scraper = XChinaScraper(
-                    ctx,
-                    is_cancelled=lambda: job._cancelled,
-                    log=lambda m: _log(job, m),
-                )
+                scraper = _make_scraper(kind, ctx, job)
                 result = await scraper.run(job.series_url, job.start_page, job.end_page)
             # 浏览器已关闭;开始下载+上传(若配置完整)
             if job.minio_cfg.is_complete:
@@ -184,7 +264,10 @@ async def _run(job: Job, headless: bool) -> None:
                 )
             else:
                 job.status = "succeeded"
-            uploaded = sum(1 for r in output["resources"] if r.get("extra_data", {}).get("minio_object_video"))
+            uploaded = sum(
+                1 for r in output["resources"]
+                if r.get("extra_data", {}).get("minio_object_video") or r.get("extra_data", {}).get("minio_objects_images")
+            )
             _log(job, f"完成: {len(output['resources'])} 条(其中 {uploaded} 条已上传 MinIO), 页数 {output['pages_scanned']}, 抓取失败 {len(output['item_errors'])}")
     except ScrapeCancelled:
         job.status = "failed"
@@ -200,7 +283,8 @@ async def _run(job: Job, headless: bool) -> None:
 
 
 def submit(series_url: str, source_name: str, start_page: int, end_page: int,
-           headless: bool = True, minio_cfg: Optional[MinioConfig] = None) -> Job:
+           headless: bool = True, minio_cfg: Optional[MinioConfig] = None,
+           media_kind: str = "auto", max_images: int = 100) -> Job:
     job = Job(
         id=uuid.uuid4().hex,
         series_url=series_url,
@@ -208,6 +292,8 @@ def submit(series_url: str, source_name: str, start_page: int, end_page: int,
         start_page=max(1, start_page),
         end_page=max(start_page, end_page),
         minio_cfg=minio_cfg or MinioConfig(),
+        media_kind=(media_kind or "auto"),
+        max_images=max(1, int(max_images or 100)),
     )
     _JOBS[job.id] = job
     job._task = asyncio.create_task(_run(job, headless))