init: content-harvester —— 内容采集服务

FastAPI + Playwright(headed Chrome/xvfb 过 Cloudflare)抓内容详情页元数据;
可选把 m3u8 视频转 mp4 + 封面上传对象存储(配置由调用方下发,本服务不持密钥);
走主站 lobster 执行器契约(invoke/poll),回传 ResourceIngestItem 形式的 resources。
当前内置 xchina.co 的 scraper(core/scraper.py)。

.gitignore

@@ -0,0 +1,21 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+
+# venv
+.venv/
+venv/
+env/
+
+# 运行时浏览器资料(含 cookie/缓存,绝不提交)
+browser_profile/
+
+# 临时下载目录(若 TMP_DIR 指到工程内)
+tmp/
+
+# 编辑器 / OS
+.idea/
+.vscode/
+.DS_Store
+*.log

CLAUDE.md

@@ -0,0 +1,42 @@
+# CLAUDE.md — content-harvester
+
+内容采集服务(FastAPI + Playwright headed-Chrome/xvfb)。给一个来源页 URL,逐个打开内容详情页抓 **标题 / 详情页URL / 封面图 / 媒体地址**;若调用方下发了对象存储(MinIO/S3)配置,则把媒体(目前是 m3u8 视频)拉成 mp4 + 封面一起上传,`video_url`/`cover_image` 改成对象存储稳定地址(很多站点的媒体地址带 `expires` 签名、约 1h 失效)。以 `resources` 列表回传。给「启源 AI 平台」主站「多内容数据采集 / 来源=内容网站」任务(lobster 执行器)调用,主站把结果写进资源库。当前内置一个 xchina.co 的 scraper(`core/scraper.py`);加新来源就再加一个 scraper 模块。
+
+> 前身是 Tkinter GUI 下载器,已删除。无单元测试。详细设计见 `PRD-minio-upload.md`。
+
+## 结构
+
+- `core/browser.py` —— `browser_context(headless)` async ctxmgr,`launch_persistent_context`(资料目录 `BROWSER_PROFILE_DIR`)。`HEADLESS=0` + `BROWSER_CHANNEL`(x86_64 可设 `chrome`)用真实有头浏览器过 Cloudflare;容器里靠 `docker-entrypoint.sh` 起 Xvfb。
+- `core/scraper.py` —— xchina 专属 scraper。`XChinaScraper(context, is_cancelled, log)`,`run(series_url,start,end) -> ScrapeResult`;`is_detail_url`、`_page_url_candidates`、`_collect_video_links`(`.item.video` + 正则兜底)、`scrape_detail`(`_extract_title` / `_find_cover_url`(截屏tab→og:image/JSON-LD/poster→最大图)/ `_find_video_url`(扫 `page.content()` 的 `.m3u8` + 网络嗅探))、`_handle_interstitial_ad`(广告中转页)、`_assert_not_blocked`(Cloudflare 质询页等其自动放行 ~24s 否则抛)。`ScrapeCancelled` 取消信号。加别的站就新建 `core/scraper_<site>.py`。
+- `core/media.py` —— `MinioConfig`(`from_payload` 从 invoke 载荷读 `minio_endpoint/access_key/secret_key/bucket`,任一缺则 `is_complete=False`;`safe_repr` 不露 secret)、`download_video`(yt-dlp 并发分片 + ffmpeg 合并 → `<videoid>.mp4`,`max_mb` 超限抛)、`download_cover`(requests 流式)、`upload_to_minio`(`minio.Minio.fput_object` → 返回 `endpoint/bucket/key`)、`extract_video_id`(从详情页 URL 的 `id-(\w+)\.html`)。
+- `core/models.py` —— `ScrapedItem`(`source_url/title/source_name/cover_image/video_url/page_no/video_id/extra`,`to_ingest_dict()` → `source_type:"video"/content_type:"video"/extra_data:{referer,page_no,collector,video_id,**extra}`;`extra` 里放 `original_m3u8/original_cover/minio_object_video/minio_object_cover/video_bytes/upload_skipped_reason`)、`ScrapeResult`。
+- `server/jobs.py` —— 内存级 job 表 + `asyncio.create_task` 后台跑;`_RUN_LOCK` 串行化(单并发)。`_run`:① 浏览器开着 `scraper.run()` 抓元数据 → 关浏览器;② 若 `job.minio_cfg.is_complete` 则 `_process_uploads` 逐条 `asyncio.to_thread` 下载视频+封面、上传 MinIO、回写 URL(失败的条目 `extra.upload_skipped_reason=...` 降级保留原媒体地址);③ 临时目录 `/tmp/content-harvester/<jobid>/` 用完即删。`MAX_VIDEO_MB`/`CONCURRENT_FRAGMENTS` env。**不要把 secret_key 传进 `_log`**。
+- `server/app.py` —— FastAPI。`POST /agents/{id}/invoke`(body=主站调用载荷:`content_url/platform_name/source_kind/start_page/end_page` + `minio_*`;`MinioConfig.from_payload(payload)` → 传给 `jobs.submit`;返回 `{"task_id","status"}`)、`GET /tasks/{id}`、`POST /tasks/{id}/cancel`、`GET /healthz`。Bearer 鉴权 `HARVESTER_TOKEN`(空则不校验)、SSRF 白名单 `ALLOWED_HOSTS`(默认 `xchina.co`)。
+- `Dockerfile` —— base `mcr.microsoft.com/playwright/python`;apt 装 `xvfb`(有头)+`ffmpeg`(yt-dlp 合并);`HEADLESS=0`;挂卷 `/data/browser_profile`;ENTRYPOINT=`docker-entrypoint.sh`(起 Xvfb)→ uvicorn。x86_64 上可改 `playwright install chromium chrome` + `BROWSER_CHANNEL=chrome`。
+- `requirements.txt` —— playwright / fastapi / uvicorn / requests / yt-dlp / minio。
+
+## 对接契约(主站 lobster 执行器)
+
+主站 `app/services/lobster_client.py::call_lobster_agent`:POST `{api_url}/agents/{agent_id}/invoke`(body=`payload`)→ 取 `task_id` → 轮询 GET `{api_url}/tasks/{task_id}` 直到 `status∈{succeeded,failed}` → 用 `body["output"]`/`body["error"]`。本服务 `output` 成功时 = `{"resources":[ResourceIngestItem字段...], "pages_scanned":n, "item_errors":[...]}`,每条 `resources[i]` 的 `video_url`/`cover_image` 若上传成功是对象存储地址(`extra_data.original_m3u8` 留原始签名URL),否则是原始来源地址(`extra_data.upload_skipped_reason`)。
+
+主站侧配套改动(`agent-system-backend/app/services/task_service.py`):① 非批次 lobster 改 `asyncio.create_task(_run_lobster_task_bg(...))` 后台跑;② `_run_lobster_task_bg` 成功后 `_maybe_ingest_to_resource_pool`;③ `_maybe_ingest_to_resource_pool` 兼容从 `result["output"]["resources"]` 取;④ `_dispatch_lobster` 把 `_resolve_skill_variables` 解出的 Skill 变量并入 payload(让远端拿到 `minio_*`);⑤ crawl 类用 `CRAWL_LOBSTER_TIMEOUT`(env,默认 7200s)。配置侧:给那个采集 Skill 绑定 `minio_endpoint/minio_access_key/minio_secret_key/minio_bucket` 4 个 Skill 变量(实际值由运维在主站配置,本服务不持有任何密钥)。
+
+## 跑
+
+```bash
+python3 -m venv .venv && source .venv/bin/activate
+pip install -r requirements.txt && python -m playwright install chromium
+# 本地无 Xvfb 可设 HEADLESS=1(但那样过不了 Cloudflare 详情页);minio 配置走 payload 传
+HEADLESS=1 uvicorn server.app:app --reload   # :8000
+```
+
+Docker:`docker build -t content-harvester . && docker run -d -p 8000:8000 -v harvester_profile:/data/browser_profile -e HARVESTER_TOKEN=xxx --memory=2g --cpus=2 content-harvester`
+
+## 坑
+
+- 详情页被 Cloudflare 挡:必须有头浏览器(`HEADLESS=0` + Xvfb,或真实 Chrome)才能自动过 JS 质询;无头/headless-shell 过不了。站点策略变了可能又挂 → 退路是预置一份过了验证的 `browser_profile` 卷。
+- 媒体地址(m3u8)常带 `expires` 签名(~1h 失效)→ 所以要在采集时就转 mp4 上传对象存储;不传的话资源库里那个 URL 一会儿就 403。
+- 下载+上传很慢:每条 ~15s 抓 + 几十秒~几分钟下载 + 上传;一页 11 条估 30min~1h+。主站对 crawl lobster 任务给了 7200s 轮询超时。
+- 单并发是有意的;并发跑多个浏览器作业容易触发风控。
+- 站点结构变了改对应的 `core/scraper*.py` 的选择器/正则/JS 片段;媒体下载/上传逻辑改 `core/media.py`。
+- `secret_key` 等敏感值别打日志(`_log` / logger 都注意)。

Dockerfile

@@ -0,0 +1,32 @@
+FROM mcr.microsoft.com/playwright/python:v1.44.0-jammy
+
+WORKDIR /app
+
+# xvfb: 让我们能跑「有头」Chrome —— Cloudflare 的 JS 质询对真实 headed 浏览器才会自动放行,
+#       无头 chromium-headless-shell 会被一眼识破
+# ffmpeg: yt-dlp 合并 m3u8 分片成 mp4 用
+RUN apt-get update && apt-get install -y --no-install-recommends xvfb ffmpeg \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY requirements.txt .
+# 注:Google Chrome 在 Linux arm64 上没有官方包,这里只装 Playwright 自带 Chromium。
+# 在 x86_64 服务器上可改成 `playwright install chromium chrome` 并设 BROWSER_CHANNEL=chrome,
+# 用真实 Chrome 通道对绕 Cloudflare 更有效。
+RUN pip install --no-cache-dir -r requirements.txt \
+    && python -m playwright install chromium
+
+COPY core/ core/
+COPY server/ server/
+COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
+RUN chmod +x /usr/local/bin/docker-entrypoint.sh
+
+# 持久浏览器资料目录(挂卷;可预置过了 Cloudflare 的 cf_clearance cookie)
+ENV BROWSER_PROFILE_DIR=/data/browser_profile \
+    HEADLESS=0 \
+    BROWSER_CHANNEL= \
+    PYTHONUNBUFFERED=1
+VOLUME ["/data/browser_profile"]
+
+EXPOSE 8000
+ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]
+CMD ["uvicorn", "server.app:app", "--host", "0.0.0.0", "--port", "8000"]

PRD-minio-upload.md

@@ -0,0 +1,282 @@
+# PRD —— 采集服务下载视频并上传 MinIO,资源库存 MinIO 地址
+
+状态:已实现 · 范围:本工程(content-harvester) + 主站 `agent-system-backend` 少量改动 · 实现说明见 `CLAUDE.md`
+
+## 1. 背景
+
+现状:主站「多内容数据采集 / 来源=内容网站」任务 → 走 lobster 执行器调本采集服务 → 服务用无头 Chrome(xvfb)抓 xchina 详情页,回传每条视频的 `{title, source_url(详情页), cover_image(图床URL), video_url(m3u8 签名URL)}` → 主站 `_maybe_ingest_to_resource_pool` 写入 `ResourcePool`。
+
+问题:`video_url` 现在存的是 xchina CDN 上**带 `expires` + `md5` 的限时签名 m3u8 地址**,大约 1 小时后 xchina 自己返回 403。资源库里的视频地址会失效,下游(转码 / 搬运 / 发布到 smnqae)无法稳定使用。
+
+目标:采集时就把 m3u8 拉成 mp4、连同封面图一起传到主站现有的 MinIO(`<bucket>` 桶),资源库 `video_url` / `cover_image` 改存 MinIO 上的稳定地址。MinIO 配置不在采集服务里硬编码,而是由主站通过 lobster payload 下发(复用主站已有的 `minio_*` Skill 变量),采集服务保持"不持有密钥"。
+
+## 2. 范围
+
+**In scope**
+- 采集服务:m3u8 → mp4 下载(yt-dlp 并发分片 + ffmpeg 合并)、封面图下载、上传 MinIO、`video_url`/`cover_image` 改为 MinIO 地址。
+- 采集服务:从 lobster payload 读取 MinIO 配置(`minio_endpoint/minio_access_key/minio_secret_key/minio_bucket`);缺失时降级(见 §9)。
+- 主站:`_dispatch_lobster` 把解析后的 Skill 变量并入发给采集服务的 payload(目前只有 script 执行器这么做);把 crawl 类 lobster 任务的轮询超时调到足够长(下载转码耗时)。
+- 配置:给 xchina 那个 Skill 绑定 4 个 `minio_*` Skill 变量(运维操作,非代码)。
+
+**Out of scope**
+- 不改主站 `ResourcePool` 表结构、不改 `ResourceIngestItem` 字段(仍是 `video_url`/`cover_image` 两个字符串,只是值变了)。
+- 不在主站后端加 ffmpeg/yt-dlp/minio(媒体重活全在采集服务)。
+- 不做断点续传、不做 MinIO 去重(同 source_url 重复采集会覆盖同名对象,可接受)。
+- 不动 OCR / 人工复核 / 发布流程。
+
+## 3. 参与系统
+
+| 系统 | 角色 |
+|---|---|
+| 主站后端 `agent-system-backend` | 创建任务、派发 lobster 调用、轮询、把回传 resources 入库;持有 MinIO 凭据(作为 Skill 变量) |
+| 本采集服务 `content-harvester` | 抓 xchina 元数据 + 下载 m3u8→mp4 + 下载封面 + 上传 MinIO + 回传 resources |
+| xchina.co | 数据源(列表页开放,详情页 Cloudflare JS 质询,媒体 m3u8 限时签名) |
+| 对象存储(MinIO/S3) | 视频/封面对象存储,产出稳定可访问 URL;endpoint/bucket/凭据由主站作为 Skill 变量提供 |
+
+## 4. 端到端流程
+
+```
+用户在主站创建任务(选 xchina Skill,来源大类=内容网站,填系列页/详情页 URL)
+        │  task.input_params = { source_kind:"website", content_url:"...", platform_name?, start_page?, end_page?, _crawl_config:{...} }
+        ▼
+主站 create() → 后台 _run_lobster_task_bg → _dispatch_lobster
+        │  ① 解析该 Skill 绑定的 Skill 变量(含 minio_endpoint/access_key/secret_key/bucket)→ var_env
+        │  ② payload = { ...task.input_params, ...var_env }      (Skill 变量同名覆盖 input_params,与 script 执行器一致)
+        │  ③ POST {lobster_api_url}/agents/{agent_id}/invoke   body=payload   Authorization: Bearer {lobster_api_key}
+        ▼
+采集服务 /agents/{id}/invoke
+        │  校验 token、source_kind=website、content_url host 白名单
+        │  从 payload 取 minio 配置(4 个 key) + start_page/end_page/platform_name
+        │  生成 job_id,后台 asyncio 任务开跑,立即返回 {"task_id": job_id, "status":"running"}
+        ▼
+后台采集作业(单并发串行)
+   for 每页(start..end):
+     打开列表页(headed Chrome/xvfb,过 Cloudflare)→ 取 .item.video 卡片里的详情链接
+     for 每个详情链接:
+        打开详情页(过 Cloudflare、处理广告中转页)
+        抓 title / cover_image(图床URL) / video_url(m3u8 签名URL,带 referer)
+        ┌─ 若拿到 minio 配置且拿到 m3u8:
+        │    yt-dlp(并发分片, 带 Referer 头)下载 → ffmpeg 合并 → /tmp/<jobid>/<videoid>.mp4
+        │    minio.fput_object(bucket, "xchina/<videoid>.mp4", mp4, content_type="video/mp4")
+        │    requests 下载封面图 → minio.fput_object(bucket, "xchina/<videoid>.jpg", img, content_type=...)
+        │    video_url = "{endpoint}/{bucket}/xchina/<videoid>.mp4"
+        │    cover_image = "{endpoint}/{bucket}/xchina/<videoid>.jpg"
+        │    extra_data += { original_m3u8: <原签名URL>, original_cover: <原图床URL>, minio_object_video, minio_object_cover, video_bytes }
+        │    删 /tmp/<jobid>/<videoid>.mp4
+        └─ 否则(无 minio 配置 / 下载失败 / 超大):降级 —— 保留原 m3u8 / 原图床 URL,extra_data += { upload_skipped_reason: "..." }
+        累积一条 ScrapedItem
+   作业结束:job.status = succeeded(只要至少抓到 1 条)/ failed(0 条且有错误)
+        ▼
+主站轮询 GET {lobster_api_url}/tasks/{job_id}  直到 status ∈ {succeeded, failed}
+        │  succeeded → body.output = { resources:[...], pages_scanned, item_errors }
+        ▼
+主站 _dispatch_lobster 写回 task.result(={success,output,error,duration_ms});task.status=SUCCEEDED
+        │  _run_lobster_task_bg 再调 _maybe_ingest_to_resource_pool(task)
+        │    取 task.result.output.resources(已兼容)→ 逐条 ResourceIngestItem.model_validate
+        │    组 ResourceIngestRequest(task_id/skill_id/created_by/product_id + _crawl_config 里的 dedupe_strategy/default_status/visibility)
+        │    ResourcePoolService.ingest_resources() → 写 ResourcePool 表(每条 video_url/cover_image 已是 MinIO 地址)
+        ▼
+完成。资源库列表里出现这些视频,video_url 指向 MinIO,长期有效。
+```
+
+## 5. 数据流转(字段级)
+
+### 5.1 主站 → 采集服务:`POST /agents/{agent_id}/invoke` 请求体(= 合并后的 payload)
+
+| 字段 | 来源 | 说明 |
+|---|---|---|
+| `source_kind` | 任务表单(CrawlerTaskBlock) | 必须 `"website"`,否则 400 |
+| `content_url` | 任务表单「地址」 | 系列页或单个详情页 URL;host 必须在采集服务白名单(默认 `xchina.co`) |
+| `platform_name` | 任务表单 | 可选 → 作为 `source_name`,默认 `"xchina"` |
+| `start_page` / `end_page` | 任务表单(可选,Skill input_schema 里声明) | 默认 `1` / `start_page` —— 即只爬给定 URL 这一页/这一条 |
+| `_crawl_config` | 主站任务创建页注入 | `{dedupe_strategy, default_status, visibility}`,采集服务**不用**,原样回传给主站也行;实际由主站 ingest 用 |
+| `minio_endpoint` | Skill 变量(主站 `_dispatch_lobster` 注入) | 形如 `<minio-endpoint>` |
+| `minio_access_key` / `minio_secret_key` | Skill 变量 | MinIO 鉴权 |
+| `minio_bucket` | Skill 变量 | 形如 `<bucket>` |
+
+> 注:Skill 变量同名覆盖 `input_params`(与 script 执行器 `_resolve_skill_variables` 后的合并优先级一致)。`minio_secret_key` 是敏感值,采集服务日志里**不打印**。
+
+### 5.2 采集服务内部:`ScrapedItem`(每条视频)
+
+| 字段 | 上传前 | 上传后 |
+|---|---|---|
+| `source_url` | xchina 详情页 URL(不变) | 同 |
+| `title` | 抓到的标题 | 同 |
+| `source_name` | `platform_name` 或 `"xchina"` | 同 |
+| `cover_image` | xchina 图床 URL(`img.xchina.io/...`) | `{endpoint}/{bucket}/xchina/<videoid>.jpg`(上传成功时) |
+| `video_url` | xchina m3u8 签名 URL | `{endpoint}/{bucket}/xchina/<videoid>.mp4`(上传成功时) |
+| `video_id` | 从 `source_url` 的 `id-(\w+)\.html` 提取 | 同;作 MinIO object key 前缀 |
+| `page_no` | 列表页页码 | 同 |
+
+### 5.3 采集服务 → 主站:`GET /tasks/{job_id}` 响应里的 `output`
+
+```jsonc
+{
+  "resources": [
+    {
+      "source_type": "video",
+      "source_name": "xchina",
+      "source_url": "https://xchina.co/video/id-XXXXXXXXXXXXX.html",
+      "content_type": "video",
+      "title": "0145 依依",
+      "cover_image": "<minio-endpoint>/<bucket>/xchina/XXXXXXXXXXXXX.jpg",
+      "video_url":   "<minio-endpoint>/<bucket>/xchina/XXXXXXXXXXXXX.mp4",
+      "extra_data": {
+        "referer": "https://xchina.co/video/id-XXXXXXXXXXXXX.html",
+        "page_no": 1,
+        "collector": "xchina",
+        "original_m3u8": "https://video.xchina.download/m3u8/XXXXXXXXXXXXX/720.m3u8?expires=...&md5=...",
+        "original_cover": "https://img.xchina.io/photos/.../00060.jpg",
+        "minio_object_video": "xchina/XXXXXXXXXXXXX.mp4",
+        "minio_object_cover": "xchina/XXXXXXXXXXXXX.jpg",
+        "video_bytes": 41234567
+      }
+    }
+    // ...
+  ],
+  "pages_scanned": 1,
+  "item_errors": [ { "source_url": "...", "reason": "..." } ]
+}
+```
+
+每条 `resources[i]` 的顶层字段就是主站 `ResourceIngestItem` 的字段集,主站可直接 `model_validate`。
+
+### 5.4 主站入库:`ResourcePool` 行(节选)
+
+`source_type=video`, `content_type=video`, `source_url`(去重键), `title`, `cover_image`(MinIO URL), `video_url`(MinIO URL), `extra_data`(含 original_m3u8 等), `task_id/skill_id/created_by/product_id`(主站补), `status`= `_crawl_config.default_status`(`ready`/`pending`), `visibility`= `_crawl_config.visibility`(普通用户强制 private)。
+
+## 6. 各步骤逻辑
+
+### 6.1 主站 —— Skill 配置(运维一次性)
+- 「Skill 管理」里那个采集 Skill(`executor_type=LOBSTER`、`skill_category=CRAWL`、`lobster_api_url` 指向采集服务):在变量关联里勾上 `minio_endpoint`、`minio_access_key`、`minio_secret_key`、`minio_bucket` 这 4 个 Skill 变量(它们的实际值由运维在主站配置,本服务不持有)。`input_schema` 仍需含 `source_kind` 字段(前端才渲染采集表单),并声明 `content_url`(必填),可选 `start_page`/`end_page`。
+
+### 6.2 主站 —— `_dispatch_lobster` 注入 Skill 变量(代码,~5–8 行)
+- 在 `call_lobster_agent` 之前:`var_env, var_files_dir = await self._resolve_skill_variables(skill, task)`;`payload = {**(task.input_params or {}), **var_env}`;把 `payload` 传给 `call_lobster_agent`(原来传 `task.input_params`)。`var_files_dir` 若产生则 finally 里 `shutil.rmtree`(本场景 minio 变量是 TEXT,通常不产生 FILE 目录)。
+- crawl 类任务轮询超时:`timeout_sec = CRAWL_LOBSTER_TIMEOUT(默认 7200) if skill.skill_category == "crawl" else 30`(下载转码 + 上传可能几十分钟到小时级;`call_lobster_agent` 在 `asyncio.create_task` 后台跑,长时持有连接可接受)。
+
+### 6.3 采集服务 —— `/agents/{id}/invoke`
+- 鉴权(Bearer == `HARVESTER_TOKEN`,未配则不校验)。
+- 校验 `source_kind`(若有)== `website`;`content_url` 非空且 host 在 `ALLOWED_HOSTS`。
+- 从 payload 取 MinIO 配置:`minio_endpoint/minio_access_key/minio_secret_key/minio_bucket` 四者**齐全**才启用上传;任一缺失 → 本次作业不上传(降级)。也支持从采集服务自身 env 兜底(可选,默认不配)。
+- 取 `start_page/end_page`(默认 1..1,上限 `MAX_PAGES`)、`source_name=platform_name or "xchina"`。
+- `jobs.submit(...)` → 后台跑;立即返回 `{"task_id": job_id}`。
+
+### 6.4 采集服务 —— 抓取(不变,已实现)
+- headed Chromium + Xvfb 过 Cloudflare(质询页等其自动放行,最多 ~24s);列表页取 `.item.video` 详情链接;详情页处理广告中转页;抓 title / cover(`截屏tab图 → og:image/JSON-LD thumbnail/<video poster> → 页面最大图`,排除 logo/占位)/ video(扫 `page.content()` 的 `.m3u8`/`.mp4` + 网络响应嗅探)。
+
+### 6.5 采集服务 —— 下载 + 上传(新增)
+- `video_id` = `re.search(r"/video/id-([A-Za-z0-9_-]+)\.html", source_url)`;取不到则用 url hash 兜底。
+- **下载视频**:m3u8 → 用 yt-dlp Python API(`format` 默认、`concurrent_fragments=N`(env `CONCURRENT_FRAGMENTS`,默认 8)、`http_headers={"Referer": source_url, "User-Agent": ...}`、`outtmpl=/tmp/<jobid>/<videoid>.%(ext)s`)→ 落 mp4(yt-dlp 内部调 ffmpeg 合并)。直链 mp4 则直接流式下载。
+  - 大小保护:env `MAX_VIDEO_MB`(默认 0=不限);超过则跳过上传、降级保留 m3u8,记 `extra_data.upload_skipped_reason="video too large: Xmb"`。
+- **下载封面**:`requests.get(cover_image, headers={Referer:..., UA:...}, stream=True)` → 落 `/tmp/<jobid>/<videoid>.<ext>`(从 URL 后缀猜,默认 `.jpg`)。失败不致命:`cover_image` 保留原图床 URL。
+- **上传 MinIO**:`Minio(host, access_key, secret_key, secure=endpoint以https开头).fput_object(bucket, "xchina/<videoid>.mp4", path, content_type="video/mp4")`;封面同理 `content_type` 按扩展名。公开 URL = `endpoint.rstrip('/') + '/' + bucket + '/' + object_key`(与脚本侧 `upload_file_to_minio` 的 `public_base/relative_path` 拼法一致)。
+- **回写**:`video_url`/`cover_image` 改 MinIO URL;`extra_data` 加 `original_m3u8 / original_cover / minio_object_video / minio_object_cover / video_bytes`。
+- **清理**:无论成败,作业结束(或每条处理完)删 `/tmp/<jobid>/`。采集服务容器**不长期落盘**。
+
+### 6.6 采集服务 —— 组装回传 & 主站入库(不变,已实现)
+- `output = {resources:[ScrapedItem.to_ingest_dict()...], pages_scanned, item_errors}`;`job.status` succeeded/failed。
+- 主站 `_run_lobster_task_bg` 成功后 `_maybe_ingest_to_resource_pool`(已兼容 `result.output.resources`)→ `ResourcePoolService.ingest_resources` 写库。
+
+## 7. 接口契约(汇总)
+
+| 接口 | 方向 | 关键约定 |
+|---|---|---|
+| `POST /agents/{agent_id}/invoke` | 主站→采集 | body=合并后 payload(§5.1);返回 `{"task_id": str, "status":"running"}`;鉴权 `Authorization: Bearer <token>` |
+| `GET /tasks/{task_id}` | 主站→采集 | `{"status":"running"|"succeeded"|"failed", "output": {...}|null, "error": str|null, "log_tail":[...]}`;`output` 见 §5.3 |
+| `POST /tasks/{task_id}/cancel` | 主站→采集 | 标记取消(仅 running) |
+| `GET /healthz` | 任意 | 健康检查 |
+
+与主站 `app/services/lobster_client.py::call_lobster_agent` 完全对齐(POST `/agents/{id}/invoke` 取 `task_id` → 轮询 `/tasks/{id}` 直到 `status∈{succeeded,failed}` → 用 `output`/`error`)。
+
+## 8. MinIO 对象命名约定
+
+- 桶:由 payload `minio_bucket` 指定(默认 `<bucket>`)。
+- object key:`xchina/<videoid>.mp4`、`xchina/<videoid>.jpg`(或封面原扩展名)。`<videoid>` = xchina 详情页 URL 里的 id 段,稳定且唯一。
+- 同一视频重复采集 → 覆盖同名对象(幂等;不另起版本)。
+- 公开 URL:`<minio_endpoint>/<bucket>/<key>`(假定该桶对象可匿名 GET —— 与现有 clawler 脚本写入的对象同桶同策略)。
+
+## 9. 异常与降级
+
+| 情况 | 行为 |
+|---|---|
+| payload 里 MinIO 4 个 key 不齐 | 本次作业整体不上传;`video_url` 保留原 m3u8,`extra_data.upload_skipped_reason="minio config absent"`;作业仍可 succeeded |
+| 某条详情页被 Cloudflare 卡住(等不出来) | 该条进 `item_errors`,不计入 `resources`;不影响其它条 |
+| m3u8 下载/合并失败 | 该条降级:`video_url` 保留原 m3u8,`extra_data.upload_skipped_reason="download failed: ..."`;仍作为一条 resource 回传 |
+| 视频超过 `MAX_VIDEO_MB` | 同上,reason="video too large" |
+| 封面下载失败 | `cover_image` 保留原图床 URL;不致命 |
+| MinIO 上传失败(网络/鉴权/桶不存在) | 该条降级保留原 URL,reason="minio upload failed: ..."；连续多条失败可考虑整作业 failed(可选,先不做) |
+| 整作业:0 条 resources 且有 item_errors | `job.status="failed"`,`error` 汇总前几条原因 |
+| 主站轮询超时(超过 `CRAWL_LOBSTER_TIMEOUT`) | 主站 `LobsterResult(success=False, error="polling timeout")` → task FAILED;采集服务那边作业可能还在跑(无害,结果丢弃);可调大超时 |
+
+降级策略的意义:即使 MinIO 这条链路出问题,也不要让整个采集白跑 —— 至少把元数据 + 原始(限时)URL 入库,人工/后续可补救。
+
+## 10. 配置项
+
+### 采集服务(env)
+| 变量 | 默认 | 说明 |
+|---|---|---|
+| `HARVESTER_TOKEN` | 空 | Bearer token(= 主站 Skill 的 `lobster_api_key`) |
+| `HEADLESS` | `0` | 0=有头(容器内靠 Xvfb);过 Cloudflare 必须 |
+| `BROWSER_CHANNEL` | 空 | x86_64 服务器可设 `chrome` 用真实 Chrome(arm64 无 Chrome 包) |
+| `BROWSER_PROFILE_DIR` | `/data/browser_profile` | 持久浏览器资料目录(挂卷) |
+| `ALLOWED_HOSTS` | `xchina.co` | content_url host 白名单(SSRF) |
+| `MAX_PAGES` | `50` | 单作业最多翻页数 |
+| `CONCURRENT_FRAGMENTS` | `8` | yt-dlp 并发分片数 |
+| `MAX_VIDEO_MB` | `0` | 单视频大小上限(0=不限),超过则跳过上传降级 |
+| `TMP_DIR` | `/tmp` | 临时下载目录(用完即删) |
+| `MINIO_ENDPOINT` / `MINIO_ACCESS_KEY` / `MINIO_SECRET_KEY` / `MINIO_BUCKET` | 空 | 仅作 payload 缺失时的兜底;生产默认不配,由主站下发 |
+
+### 主站(已有 Skill 变量,运维勾选关联即可)
+`minio_endpoint`、`minio_bucket`、`minio_access_key`、`minio_secret_key`(实际值由运维在主站配置,本仓库不含)。
+
+### 主站(新增 env / 常量,可选)
+`CRAWL_LOBSTER_TIMEOUT`(默认 7200s):crawl 类 lobster 任务的轮询超时。
+
+## 11. 性能 / 时延
+
+- 每条:抓元数据 ~15s + 下载 m3u8→mp4(随视频大小,几十秒~几分钟)+ 上传 MinIO(几秒~几十秒)。一页 11 条估计 30 分钟 ~ 1 小时+。
+- 单并发(进程内串行),避免 xchina 风控;多页/大量视频会很慢,属预期。
+- 主站侧靠 `asyncio.create_task` 后台跑 + 轮询,不阻塞任何用户请求。
+
+## 12. 安全
+
+- `minio_secret_key` 等敏感值:采集服务**不写日志**(log_tail / logger 都过滤);只在内存里用。
+- SSRF:`content_url` host 白名单;封面图下载也只允许 xchina/img.xchina.io 域(或宽松些但限制 scheme=https)。
+- token 鉴权:`/agents/*` `/tasks/*` 都校验 Bearer。
+- 采集服务容器不长期落盘媒体文件(临时目录用完即删);浏览器 profile 卷里可能含 Cloudflare cookie,按敏感卷管理。
+
+## 13. 改动清单(文件级)
+
+### 采集服务(本工程)
+- `requirements.txt`:加 `yt-dlp`、`minio`、`requests`。
+- `Dockerfile`:`apt-get install -y ffmpeg`(与 xvfb 一起);其余不变。
+- `core/media.py`(新):`download_video(m3u8_or_mp4_url, referer, dest_dir, video_id, max_mb, concurrent_fragments) -> Path`、`download_cover(url, referer, dest_dir, video_id) -> Path | None`、`upload_to_minio(local_path, object_key, content_type, cfg) -> public_url`、`build_public_url(endpoint, bucket, key)`、`MinioConfig` dataclass + `from_payload(payload)` / `is_complete`。
+- `core/models.py`:`ScrapedItem` 加 `video_id` 字段;`to_ingest_dict` 的 `extra_data` 支持附加 `original_m3u8/original_cover/minio_object_*/video_bytes/upload_skipped_reason`。
+- `server/jobs.py`:`Job` 带 `minio_cfg`;`_run` 里每条 `scrape_detail` 后,若 `minio_cfg.is_complete` 则 download+upload+回写,失败降级;作业结束清临时目录;敏感值不入 `log_lines`。
+- `server/app.py`:`invoke` 从 payload 解析 `MinioConfig.from_payload(payload)`,传给 `jobs.submit`。
+- `README.md` / `CLAUDE.md`:更新(新增依赖、env、行为、对接说明)。
+
+### 主站 `agent-system-backend`
+- `app/services/task_service.py::_dispatch_lobster`:① `var_env = await self._resolve_skill_variables(skill, task)` 后 `payload = {**(task.input_params or {}), **var_env}` 传给 `call_lobster_agent`;② crawl 类用 `CRAWL_LOBSTER_TIMEOUT`(默认 7200)。(`_run_lobster_task_bg` 成功后调 `_maybe_ingest_to_resource_pool` 与 `_maybe_ingest_to_resource_pool` 读 `output.resources` 已在前面做过。)
+- `app/config.py`(可选):加 `CRAWL_LOBSTER_TIMEOUT`。
+
+### 配置(非代码)
+- 主站「Skill 管理」给 xchina Skill 绑定 4 个 `minio_*` Skill 变量。
+- 采集服务部署到 x86_64 时:Dockerfile 改 `playwright install chromium chrome` + 设 `BROWSER_CHANNEL=chrome`(更稳过 Cloudflare)。
+
+## 14. 验收
+
+采集服务本地(docker):
+1. `docker build` + `docker run`(`HEADLESS=0`),`/healthz` 返回 `{"ok":true,"headless":false}`。
+2. `curl -X POST /agents/xchina/invoke -d '{"source_kind":"website","content_url":"<xchina系列页URL>","minio_endpoint":"<minio-endpoint>","minio_access_key":"...","minio_secret_key":"...","minio_bucket":"<bucket>"}'` → 拿 task_id。
+3. 轮询 `/tasks/{id}` 到 `succeeded`:`output.resources` 每条 `video_url` 形如 `<minio-endpoint>/<bucket>/xchina/<id>.mp4`,`cover_image` 形如 `.../xchina/<id>.jpg`,`extra_data` 含 `original_m3u8`、`minio_object_video`、`video_bytes`。
+4. `curl -I <那个 mp4 URL>` 返回 200(对象确实上传成功且可访问);随便取一个验证能播放。
+5. 不带 minio 字段再跑一次 → `video_url` 仍是 m3u8,`extra_data.upload_skipped_reason="minio config absent"`,作业仍 succeeded。
+6. 负向:错误 token → 401;`source_kind:"platform"` → 400;非白名单 host → 400。
+
+主站联调(dev 环境):
+7. 给 xchina Skill 绑定 4 个 minio 变量;创建任务(选该 Skill、来源大类=内容网站、地址填 xchina 系列页、入库后状态=ready)。
+8. 等任务 `SUCCEEDED`;`GET /resource-pool` 出现这些视频,详情里 `video_url` 是 MinIO 地址、`cover_image` 是 MinIO 地址、`extra_data` 含 `original_m3u8`、`task_id/skill_id/created_by` 正确、`status=ready`。
+9. `curl -I` 资源里的 `video_url` → 200,且过 1 小时后再 `curl -I` 仍 200(证明不再依赖 xchina 限时签名)。
+10. 同 URL 再跑一次,`dedupe_strategy=skip` 时资源库不重复入库;MinIO 上同名对象被覆盖。
+11. UI/e2e 步骤丢给子 agent 跑,主线程只收结论摘要。

README.md

@@ -0,0 +1,77 @@
+# content-harvester
+
+内容采集服务(FastAPI + Playwright)。给一个来源页 URL,用有头 Chrome(容器内靠 Xvfb,能过 Cloudflare)逐个打开内容详情页,抓出每条内容的 **标题 / 详情页 URL / 封面图 / 媒体地址**;如果调用方下发了对象存储(MinIO/S3)配置,就把媒体(目前是 m3u8 视频)拉成 mp4、连同封面一起上传,把 `video_url`/`cover_image` 换成对象存储上的稳定地址(很多站点的媒体地址是带签名的短时 URL)。结果以 `resources` 列表回传。
+
+当前内置一个 **xchina.co** 的 scraper(`core/scraper.py`);以后接别的来源就再加一个 scraper 模块。
+
+设计文档见 `PRD-minio-upload.md`,实现细节见 `CLAUDE.md`。
+
+## 用法定位
+
+对接「启源 AI 平台」主站的 **lobster 执行器**:主站「多内容数据采集」任务里选「来源大类 = 内容网站」时,把作业 POST 给本服务,本服务返回 `resources`,主站写进资源库。
+
+## HTTP 接口
+
+- `POST /agents/{agent_id}/invoke` — body 是主站的调用载荷:`content_url`(必填,来源页/详情页 URL)、`platform_name`(可选 → `source_name`,默认 `xchina`)、`source_kind`(可选,非 `website` 直接 400)、`start_page`/`end_page`(可选,默认只爬 `content_url` 这一页/这一条)、`minio_endpoint`/`minio_access_key`/`minio_secret_key`/`minio_bucket`(可选,主站把对应 Skill 变量并进来)。立即返回 `{"task_id":"...","status":"running"}`,作业后台跑。
+- `GET /tasks/{task_id}` — `{"status":"running"|"succeeded"|"failed","output":{...}|null,"error":str|null,"log_tail":[...]}`。
+  - 成功时 `output = {"resources":[...], "pages_scanned":n, "item_errors":[{"source_url","reason"},...]}`。`resources` 每项字段对齐主站 `ResourceIngestItem`:`source_type:"video" / source_name / source_url / content_type:"video" / title / cover_image / video_url / extra_data`。
+  - `extra_data` 含 `referer / page_no / collector / video_id`;上传成功还有 `original_m3u8 / original_cover / minio_object_video / minio_object_cover / video_bytes`;没上传则有 `upload_skipped_reason`(如 `minio config absent` / `download failed: ...` / `video too large: ...`)。
+- `POST /tasks/{task_id}/cancel` — 标记取消(仅 running)。`GET /healthz` — 健康检查。
+
+鉴权:设了 `HARVESTER_TOKEN` 则要求 `Authorization: Bearer <token>`(= 主站 Skill 的 `lobster_api_key`);没设则不鉴权(仅内网/本地)。
+
+并发:进程内单作业串行(防风控)。作业状态只在内存,重启即丢。
+
+## 环境变量
+
+| 变量 | 默认 | 说明 |
+| --- | --- | --- |
+| `HARVESTER_TOKEN` | 空 | Bearer token;空=不鉴权 |
+| `HEADLESS` | `0` | `0`=有头(容器内靠 Xvfb);过 Cloudflare 必须;`1`=无头(本地调试,过不了详情页) |
+| `BROWSER_CHANNEL` | 空 | x86_64 服务器可设 `chrome`(需 Dockerfile 里 `playwright install chromium chrome`);arm64 无 Chrome 包 |
+| `BROWSER_PROFILE_DIR` | `/data/browser_profile` | 持久浏览器资料目录(挂卷;可预置过了 Cloudflare 的 cookie) |
+| `ALLOWED_HOSTS` | `xchina.co` | `content_url` host 白名单(SSRF) |
+| `MAX_PAGES` | `50` | 单作业最多翻页数 |
+| `CONCURRENT_FRAGMENTS` | `8` | yt-dlp 并发分片数 |
+| `MAX_VIDEO_MB` | `0` | 单视频大小上限(0=不限),超过则跳过上传、降级保留原媒体地址 |
+| `TMP_DIR` | `/tmp` | 临时下载目录(`<TMP_DIR>/content-harvester/<jobid>/`,用完即删) |
+| `MINIO_ENDPOINT` / `MINIO_ACCESS_KEY` / `MINIO_SECRET_KEY` / `MINIO_BUCKET` | 空 | 仅作 payload 缺失时的兜底;生产默认不配,由主站下发 |
+
+## 本地跑
+
+```bash
+python3 -m venv .venv && source .venv/bin/activate
+pip install -r requirements.txt
+python -m playwright install chromium
+# 本地没 Xvfb 用 HEADLESS=1(但详情页会被 Cloudflare 拦)
+HEADLESS=1 uvicorn server.app:app --reload   # http://localhost:8000
+
+curl -s -X POST localhost:8000/agents/xchina/invoke -H 'Content-Type: application/json' \
+  -d '{"source_kind":"website","content_url":"https://xchina.co/videos/series-XXXX.html","minio_endpoint":"<MINIO_ENDPOINT>","minio_access_key":"<KEY>","minio_secret_key":"<SECRET>","minio_bucket":"<BUCKET>"}'
+curl -s localhost:8000/tasks/<task_id>
+```
+
+## Docker(推荐)
+
+```bash
+docker build -t content-harvester .
+docker run -d --name content-harvester -p 8000:8000 \
+  -e HARVESTER_TOKEN=xxxxx \
+  -v harvester_profile:/data/browser_profile \
+  --memory=2g --cpus=2 \
+  content-harvester
+```
+
+镜像里已装 `xvfb`(跑有头浏览器)+ `ffmpeg`(yt-dlp 合并 m3u8 分片)。部署到主站所在 VPS 时,把这个容器加进主站 docker compose 同一网络,主站 Skill 的 `lobster_api_url` 填 `http://content-harvester:8000`。
+
+## 主站侧对接
+
+1. 「Skill 管理」新建/编辑那个采集 Skill:`executor_type=LOBSTER`、`skill_category=CRAWL`、`lobster_api_url=http://content-harvester:8000`、`lobster_api_key=<HARVESTER_TOKEN>`、`lobster_agent_id=xchina`(随意);`input_schema` 含 `source_kind` 字段(前端才渲染采集表单)+ 声明 `content_url`(必填),可选 `start_page`/`end_page`;**变量关联里勾上 `minio_endpoint`/`minio_access_key`/`minio_secret_key`/`minio_bucket` 这 4 个 Skill 变量**(它们的实际值由运维在主站配置,本服务不持有任何密钥)。
+2. 主站后端配套改动:非批次 lobster 改后台跑、成功后自动 `_maybe_ingest_to_resource_pool`、ingest 兼容 `output.resources`、`_dispatch_lobster` 把 Skill 变量并入 payload、crawl 类用 `CRAWL_LOBSTER_TIMEOUT`(env,默认 7200s)。
+3. 创建任务:选该 Skill → 来源大类「内容网站」→「地址」填来源页/详情页 URL → 设可见性/去重/入库后状态 → 提交。任务成功后资源库出现这些内容,`video_url` 指向对象存储,长期有效。
+
+## 反爬 / 限制
+
+- 详情页常被 Cloudflare 挡,只有真实有头浏览器(`HEADLESS=0` + Xvfb,或真实 Chrome)才能自动过 JS 质询;`_assert_not_blocked` 会等质询自动放行(最多 ~24s),仍卡住则该条进 `item_errors`。站点策略变化可能导致采集失效,退路是预置一份过了验证的 `browser_profile` 卷。
+- 下载+转码+上传慢:每条 ~15s 抓 + 几十秒~几分钟下载 + 上传;多页/大量视频会很慢,属预期。
+- 任一条目下载/上传失败 → 该条降级:`video_url` 保留原始媒体地址(`extra_data.upload_skipped_reason` 记原因),不让整作业白跑。

core/browser.py

@@ -0,0 +1,58 @@
+"""Playwright 浏览器上下文 —— 无头、持久化资料目录。
+
+持久目录(BROWSER_PROFILE_DIR,默认 ./browser_profile)用于复用登录态 / Cookie /
+过验证后的状态。无头服务器上若 xchina 弹 Cloudflare,没有人工过验证的退路 ——
+运维需一次性在桌面浏览器里登录并把 profile 目录拷进该路径(容器里挂卷)。
+"""
+from __future__ import annotations
+
+import os
+from contextlib import asynccontextmanager
+from pathlib import Path
+
+from playwright.async_api import async_playwright
+
+DEFAULT_USER_AGENT = (
+    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
+    "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36"
+)
+
+PROFILE_DIR = Path(os.environ.get("BROWSER_PROFILE_DIR", Path(__file__).resolve().parent.parent / "browser_profile"))
+# 用真实 Google Chrome 通道(BROWSER_CHANNEL=chrome) + 有头(HEADLESS=0,容器里靠 xvfb)
+# 对绕 Cloudflare 的 JS 质询效果最好;留空则用 Playwright 自带 Chromium
+BROWSER_CHANNEL = os.environ.get("BROWSER_CHANNEL", "").strip() or None
+
+_LAUNCH_ARGS = [
+    "--disable-blink-features=AutomationControlled",
+    "--disable-dev-shm-usage",
+    "--no-first-run",
+    "--no-default-browser-check",
+    "--no-sandbox",
+]
+
+
+@asynccontextmanager
+async def browser_context(headless: bool = True):
+    """打开一个持久化 BrowserContext,用完自动关闭。"""
+    PROFILE_DIR.mkdir(parents=True, exist_ok=True)
+    pw = await async_playwright().start()
+    context = None
+    try:
+        kwargs = dict(
+            headless=headless,
+            user_agent=DEFAULT_USER_AGENT,
+            viewport={"width": 1366, "height": 900},
+            locale="zh-CN",
+            timezone_id="Asia/Shanghai",
+            accept_downloads=False,
+            args=_LAUNCH_ARGS,
+        )
+        if BROWSER_CHANNEL:
+            kwargs["channel"] = BROWSER_CHANNEL
+        context = await pw.chromium.launch_persistent_context(str(PROFILE_DIR), **kwargs)
+        context.set_default_timeout(20000)
+        yield context
+    finally:
+        if context is not None:
+            await context.close()
+        await pw.stop()

core/media.py

@@ -0,0 +1,172 @@
+"""下载 m3u8/mp4 视频 + 封面图,上传到 MinIO。
+
+MinIO 配置由调用方(主站经 lobster payload)传入,本模块不读全局配置。
+敏感值(secret_key)不打印。
+"""
+from __future__ import annotations
+
+import logging
+import mimetypes
+import os
+import re
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Optional
+from urllib.parse import urlparse
+
+logger = logging.getLogger("harvester.media")
+
+_UA = (
+    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
+    "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36"
+)
+
+
+# --------------------------------------------------------------------------
+# MinIO 配置
+# --------------------------------------------------------------------------
+
+@dataclass
+class MinioConfig:
+    endpoint: str = ""        # 形如 https://minio.example.com
+    access_key: str = ""
+    secret_key: str = ""
+    bucket: str = ""
+
+    @classmethod
+    def from_payload(cls, payload: dict) -> "MinioConfig":
+        def g(*keys: str) -> str:
+            for k in keys:
+                v = payload.get(k)
+                if v:
+                    return str(v).strip()
+            return ""
+        cfg = cls(
+            endpoint=g("minio_endpoint", "MINIO_ENDPOINT") or os.environ.get("MINIO_ENDPOINT", "").strip(),
+            access_key=g("minio_access_key", "MINIO_ACCESS_KEY") or os.environ.get("MINIO_ACCESS_KEY", "").strip(),
+            secret_key=g("minio_secret_key", "MINIO_SECRET_KEY") or os.environ.get("MINIO_SECRET_KEY", "").strip(),
+            bucket=g("minio_bucket", "MINIO_BUCKET") or os.environ.get("MINIO_BUCKET", "").strip(),
+        )
+        return cfg
+
+    @property
+    def is_complete(self) -> bool:
+        return bool(self.endpoint and self.access_key and self.secret_key and self.bucket)
+
+    def safe_repr(self) -> str:
+        return f"MinioConfig(endpoint={self.endpoint!r}, bucket={self.bucket!r}, access_key={self.access_key[:4]+'***' if self.access_key else ''!r}, secret_key=***)"
+
+
+def _endpoint_host_secure(endpoint: str) -> tuple[str, bool]:
+    """https://minio.example.com -> ('minio.aidfxn.com', True);裸 host 也兼容。"""
+    e = endpoint.strip()
+    if e.startswith("http://"):
+        return e[len("http://"):].strip("/"), False
+    if e.startswith("https://"):
+        return e[len("https://"):].strip("/"), True
+    # 没带 scheme,默认 https
+    return e.strip("/"), True
+
+
+def build_public_url(endpoint: str, bucket: str, object_key: str) -> str:
+    return f"{endpoint.rstrip('/')}/{bucket.strip('/')}/{object_key.lstrip('/')}"
+
+
+def upload_to_minio(local_path: Path, object_key: str, content_type: str, cfg: MinioConfig) -> str:
+    """上传一个文件到 MinIO,返回公开 URL。失败抛异常。"""
+    from minio import Minio  # 延迟 import,缺依赖时报错更清晰
+
+    host, secure = _endpoint_host_secure(cfg.endpoint)
+    client = Minio(host, access_key=cfg.access_key, secret_key=cfg.secret_key, secure=secure)
+    client.fput_object(cfg.bucket, object_key, str(local_path), content_type=content_type or "application/octet-stream")
+    return build_public_url(cfg.endpoint, cfg.bucket, object_key)
+
+
+# --------------------------------------------------------------------------
+# 下载
+# --------------------------------------------------------------------------
+
+_VIDEO_EXTS = (".mp4", ".mkv", ".webm", ".mov", ".ts", ".m4v")
+
+
+def extract_video_id(detail_url: str) -> str:
+    m = re.search(r"/video[s]?/id-([A-Za-z0-9_-]+)\.html", detail_url)
+    if m:
+        return m.group(1)
+    # 兜底:用 url 的 hash
+    import hashlib
+    return "u" + hashlib.sha1(detail_url.encode("utf-8")).hexdigest()[:16]
+
+
+def download_video(media_url: str, referer: str, dest_dir: Path, video_id: str,
+                   concurrent_fragments: int = 8, max_mb: int = 0) -> Path:
+    """用 yt-dlp 下载 m3u8/mp4 → dest_dir/<video_id>.mp4(尽量)。返回最终文件路径。
+
+    max_mb > 0 时,下完发现超过则删文件并抛 ValueError("video too large: ...")。
+    """
+    import yt_dlp
+
+    dest_dir.mkdir(parents=True, exist_ok=True)
+    outtmpl = str(dest_dir / f"{video_id}.%(ext)s")
+    opts = {
+        "outtmpl": outtmpl,
+        "format": "best",
+        "merge_output_format": "mp4",
+        "concurrent_fragment_downloads": max(1, int(concurrent_fragments)),
+        "http_headers": {"Referer": referer, "User-Agent": _UA},
+        "quiet": True,
+        "no_warnings": True,
+        "noprogress": True,
+        "retries": 3,
+        "fragment_retries": 3,
+        "noplaylist": True,
+    }
+    with yt_dlp.YoutubeDL(opts) as ydl:
+        ydl.download([media_url])
+
+    # 找产物:dest_dir/<video_id>.<ext>
+    candidates = sorted(
+        (p for p in dest_dir.glob(f"{video_id}.*") if p.suffix.lower() in _VIDEO_EXTS),
+        key=lambda p: p.stat().st_size,
+        reverse=True,
+    )
+    if not candidates:
+        raise RuntimeError("yt-dlp 未产出视频文件")
+    path = candidates[0]
+    size_mb = path.stat().st_size / (1024 * 1024)
+    if max_mb and size_mb > max_mb:
+        try:
+            path.unlink()
+        except OSError:
+            pass
+        raise ValueError(f"video too large: {size_mb:.0f}mb > {max_mb}mb")
+    return path
+
+
+def download_cover(cover_url: str, referer: str, dest_dir: Path, video_id: str) -> Optional[Path]:
+    """流式下载封面图 → dest_dir/<video_id><ext>。失败返回 None(非致命)。"""
+    import requests
+
+    try:
+        ext = os.path.splitext(urlparse(cover_url).path)[1].lower()
+        if ext not in (".jpg", ".jpeg", ".png", ".webp", ".gif"):
+            ext = ".jpg"
+        dest_dir.mkdir(parents=True, exist_ok=True)
+        path = dest_dir / f"{video_id}{ext}"
+        with requests.get(cover_url, headers={"Referer": referer, "User-Agent": _UA}, stream=True, timeout=30) as r:
+            r.raise_for_status()
+            with open(path, "wb") as f:
+                for chunk in r.iter_content(chunk_size=64 * 1024):
+                    if chunk:
+                        f.write(chunk)
+        if path.stat().st_size == 0:
+            path.unlink(missing_ok=True)
+            return None
+        return path
+    except Exception as exc:  # noqa: BLE001
+        logger.warning("cover download failed %s: %s", cover_url, exc)
+        return None
+
+
+def guess_content_type(path: Path, default: str = "application/octet-stream") -> str:
+    return mimetypes.guess_type(str(path))[0] or default

core/models.py

@@ -0,0 +1,62 @@
+"""采集结果的数据结构。
+
+ScrapedItem 的字段刻意与主站 `ResourceIngestItem`(app/schemas/resource_pool.py)
+对齐,这样主站 `_maybe_ingest_to_resource_pool` 能直接 `model_validate` 每一条。
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any
+
+
+@dataclass
+class ScrapedItem:
+    # 详情页 URL —— 主站资源库按它去重
+    source_url: str
+    title: str
+    # 来源站点名,默认 "xchina"(可被 invoke 载荷里的 platform_name 覆盖)
+    source_name: str = "xchina"
+    # 封面图 URL(上传 MinIO 后改为 MinIO 地址,否则是 xchina 图床地址)
+    cover_image: str | None = None
+    # 视频 URL(上传 MinIO 后改为 MinIO 地址,否则是 xchina 限时签名 m3u8)
+    video_url: str | None = None
+    # 列表页页码(单详情页直采时为 1)
+    page_no: int = 1
+    # xchina 详情页里的 id 段;作 MinIO object key 前缀
+    video_id: str = ""
+    # 附加进 extra_data 的字段(original_m3u8 / minio_object_video / video_bytes / upload_skipped_reason ...)
+    extra: dict[str, Any] = field(default_factory=dict)
+
+    def to_ingest_dict(self) -> dict[str, Any]:
+        extra_data: dict[str, Any] = {
+            "referer": self.source_url,
+            "page_no": self.page_no,
+            "collector": "xchina",
+        }
+        if self.video_id:
+            extra_data["video_id"] = self.video_id
+        extra_data.update(self.extra or {})
+        return {
+            "source_type": "video",
+            "source_name": self.source_name,
+            "source_url": self.source_url,
+            "content_type": "video",
+            "title": self.title,
+            "cover_image": self.cover_image,
+            "video_url": self.video_url,
+            "extra_data": extra_data,
+        }
+
+
+@dataclass
+class ScrapeResult:
+    resources: list[ScrapedItem] = field(default_factory=list)
+    pages_scanned: int = 0
+    item_errors: list[dict[str, str]] = field(default_factory=list)
+
+    def to_output(self) -> dict[str, Any]:
+        return {
+            "resources": [it.to_ingest_dict() for it in self.resources],
+            "pages_scanned": self.pages_scanned,
+            "item_errors": self.item_errors,
+        }

core/scraper.py

@@ -0,0 +1,579 @@
+"""xchina.co 详情页采集 —— 无头、只抓元数据(标题 / 封面图 URL / m3u8 URL),不下载文件。
+
+逻辑大体从原 GUI `xchina_gui_downloader.py` 的 Downloader 里抽出并精简:
+- 去掉 yt-dlp / ffmpeg / 文件落盘 / GUI 回调 / 跳过记录
+- stop/skip 的 threading.Event 换成一个 asyncio cancel 检查
+- 无头环境下遇到 Cloudflare/验证码直接抛错(没有人工过验证的退路)
+"""
+from __future__ import annotations
+
+import asyncio
+import re
+from pathlib import Path
+from typing import Callable, Optional
+from urllib.parse import urljoin, urlparse
+
+from playwright.async_api import BrowserContext
+from playwright.async_api import Error as PlaywrightError
+from playwright.async_api import TimeoutError as PlaywrightTimeoutError
+
+from .models import ScrapedItem, ScrapeResult
+
+# 真正的 Cloudflare 质询页 body 很短(几十~几百字),且含这些特征串之一。
+# 不用 "verify" / "验证" 这种泛词 —— 正常详情页(年龄确认、表单等)也会命中,会误判。
+_BLOCK_MARKERS = [
+    "just a moment", "checking your browser", "cloudflare",
+    "attention required", "you have been blocked", "verify you are human",
+    "请稍候", "正在验证您的浏览器", "你已被屏蔽",
+]
+_BLOCK_BODY_MAXLEN = 800
+
+
+class ScrapeCancelled(Exception):
+    pass
+
+
+class XChinaScraper:
+    def __init__(
+        self,
+        context: BrowserContext,
+        is_cancelled: Callable[[], bool] = lambda: False,
+        log: Callable[[str], None] = lambda _msg: None,
+        items_per_page: int = 12,
+    ):
+        self.context = context
+        self.is_cancelled = is_cancelled
+        self.log = log
+        self.items_per_page = items_per_page
+
+    # ---- cancel ----------------------------------------------------------
+
+    def _check_cancel(self) -> None:
+        if self.is_cancelled():
+            raise ScrapeCancelled()
+
+    # ---- top level -------------------------------------------------------
+
+    async def run(self, series_url: str, start_page: int = 1, end_page: int = 1) -> ScrapeResult:
+        result = ScrapeResult()
+        series_page = await self.context.new_page()
+        try:
+            if self.is_detail_url(series_url):
+                self.log(f"输入是单个详情页,直接采集: {series_url}")
+                item = await self._scrape_detail_safe(series_url, 1, result)
+                if item:
+                    result.resources.append(item)
+                result.pages_scanned = 1
+                return result
+
+            for page_no in range(start_page, end_page + 1):
+                self._check_cancel()
+                ok = await self._go_series_page(series_page, series_url, page_no)
+                if not ok:
+                    self.log(f"第 {page_no} 页不存在或无视频,停止翻页。")
+                    break
+                result.pages_scanned += 1
+                links = await self._collect_video_links(series_page)
+                if not links:
+                    self.log(f"第 {page_no} 页没识别到详情链接。")
+                    continue
+                self.log(f"第 {page_no} 页识别到 {len(links)} 个详情链接。")
+                for detail_url in links:
+                    self._check_cancel()
+                    item = await self._scrape_detail_safe(detail_url, page_no, result)
+                    if item:
+                        result.resources.append(item)
+            return result
+        finally:
+            await series_page.close()
+
+    async def _scrape_detail_safe(self, detail_url: str, page_no: int, result: ScrapeResult) -> Optional[ScrapedItem]:
+        try:
+            return await self.scrape_detail(detail_url, page_no)
+        except ScrapeCancelled:
+            raise
+        except Exception as exc:  # noqa: BLE001
+            self.log(f"详情页采集失败,跳过: {detail_url} -> {exc}")
+            result.item_errors.append({"source_url": detail_url, "reason": f"{type(exc).__name__}: {exc}"})
+            return None
+
+    # ---- url helpers -----------------------------------------------------
+
+    @staticmethod
+    def is_detail_url(url: str) -> bool:
+        path = urlparse(url).path.lower()
+        if not path.endswith(".html"):
+            return False
+        if any(part in path for part in ["/series-", "/category", "/tags", "/search"]):
+            return False
+        return "/video/" in path or "/videos/" in path
+
+    def _page_url_candidates(self, current_url: str, page_no: int) -> list[str]:
+        parsed = urlparse(current_url)
+        base = current_url.split("#", 1)[0].split("?", 1)[0]
+        candidates: list[str] = []
+        if base.endswith(".html"):
+            m = re.match(r"(.*/series-[^/]+)(?:/\d+)?\.html$", base)
+            if m:
+                candidates.append(f"{m.group(1)}.html" if page_no == 1 else f"{m.group(1)}/{page_no}.html")
+            else:
+                stem = base[:-5]
+                if page_no == 1:
+                    candidates.append(base)
+                else:
+                    candidates.extend([
+                        f"{stem}/{page_no}.html",
+                        f"{stem}-{page_no}.html",
+                        f"{stem}_{page_no}.html",
+                        f"{stem}/page/{page_no}.html",
+                    ])
+        candidates.extend([f"{base}?page={page_no}", f"{base}?p={page_no}"])
+        root = f"{parsed.scheme}://{parsed.netloc}"
+        try:
+            series_part = Path(parsed.path).stem.split("series-")[-1]
+            candidates.append(urljoin(root, f"/videos/series-{series_part}-{page_no}.html"))
+        except Exception:
+            pass
+        unique: list[str] = []
+        for url in candidates:
+            if url not in unique:
+                unique.append(url)
+        return unique
+
+    def _extract_detail_urls_from_text(self, text: str, base_url: str) -> list[str]:
+        if not text:
+            return []
+        text = text.replace("\\/", "/")
+        parsed_base = urlparse(base_url)
+        site_root = f"{parsed_base.scheme}://{parsed_base.netloc}"
+        patterns = [
+            r"https?://[^'\"<>\s]+/video/[A-Za-z0-9_-]+\.html",
+            r"https?://[^'\"<>\s]+/videos/[A-Za-z0-9_-]+\.html",
+            r"/video/[A-Za-z0-9_-]+\.html",
+            r"/videos/[A-Za-z0-9_-]+\.html",
+            r"video/[A-Za-z0-9_-]+\.html",
+            r"videos/[A-Za-z0-9_-]+\.html",
+        ]
+        results: list[str] = []
+        for pattern in patterns:
+            for match in re.findall(pattern, text, flags=re.I):
+                if match.startswith(("video/", "videos/")):
+                    match = "/" + match
+                url = urljoin(site_root + "/", match)
+                if self.is_detail_url(url) and url not in results:
+                    results.append(url)
+        return results
+
+    def _extract_media_urls_from_text(self, text: str) -> list[str]:
+        if not text:
+            return []
+        text = text.replace("\\/", "/").replace("&amp;", "&")
+        patterns = [
+            r"https?://[^'\"<>\s]+\.m3u8(?:\?[^'\"<>\s]+)?",
+            r"https?://[^'\"<>\s]+\.mp4(?:\?[^'\"<>\s]+)?",
+            r"https?://[^'\"<>\s]+\.webm(?:\?[^'\"<>\s]+)?",
+            r"https?://[^'\"<>\s]+\.mov(?:\?[^'\"<>\s]+)?",
+        ]
+        urls: list[str] = []
+        for pattern in patterns:
+            for match in re.findall(pattern, text, flags=re.I):
+                if "upload.xchina.io/media/" in match:
+                    continue
+                if match not in urls:
+                    urls.append(match)
+        return urls
+
+    # ---- navigation ------------------------------------------------------
+
+    async def _visit(self, page, url: str) -> None:
+        last_error = None
+        attempt = 0
+        for attempt in range(1, 4):
+            self._check_cancel()
+            try:
+                await page.goto(url, wait_until="domcontentloaded", timeout=60000)
+                break
+            except PlaywrightError as exc:
+                last_error = exc
+                txt = str(exc)
+                retryable = any(m in txt for m in [
+                    "ERR_CONNECTION_CLOSED", "ERR_CONNECTION_RESET", "ERR_TIMED_OUT", "Timeout",
+                ])
+                if attempt >= 3 or not retryable:
+                    raise
+                await page.wait_for_timeout(2500 * attempt)
+        if last_error and attempt >= 3:
+            raise last_error
+        try:
+            await page.wait_for_load_state("networkidle", timeout=15000)
+        except PlaywrightTimeoutError:
+            pass
+
+    async def _looks_blocked(self, page) -> Optional[str]:
+        """返回命中的特征串(说明当前停在 Cloudflare 质询页),否则 None。"""
+        try:
+            body = await page.locator("body").inner_text(timeout=8000)
+        except Exception:
+            body = ""
+        try:
+            title = await page.title()
+        except Exception:
+            title = ""
+        low = body.lower()
+        low_t = title.lower()
+        hit = next((w for w in _BLOCK_MARKERS if w in low or w in low_t), None)
+        # 真正的质询页内容很少;内容丰富的正常页即使含某串也放过
+        if hit and len(body.strip()) <= _BLOCK_BODY_MAXLEN:
+            return hit
+        return None
+
+    async def _assert_not_blocked(self, page) -> None:
+        """若停在 Cloudflare 质询页,等它的 JS 质询自动放行(通常几秒);仍卡住才报错。"""
+        hit = await self._looks_blocked(page)
+        if not hit:
+            return
+        self.log(f"检测到 Cloudflare 质询页(命中 {hit!r}),等待其自动放行…")
+        for _ in range(8):  # 最多 ~24s
+            self._check_cancel()
+            await page.wait_for_timeout(3000)
+            try:
+                await page.wait_for_load_state("networkidle", timeout=5000)
+            except PlaywrightTimeoutError:
+                pass
+            if not await self._looks_blocked(page):
+                self.log("Cloudflare 质询已通过。")
+                return
+        raise RuntimeError(
+            f"被 Cloudflare 质询拦住未能自动放行(命中 {hit!r};无头环境过不了交互式验证,"
+            f"请用桌面浏览器登录/过一次验证后把 browser_profile 目录预置进容器卷)"
+        )
+
+    async def _go_series_page(self, page, series_url: str, page_no: int) -> bool:
+        for candidate in self._page_url_candidates(series_url, page_no):
+            try:
+                await self._visit(page, candidate)
+                await page.wait_for_timeout(300)
+                await self._assert_not_blocked(page)
+                if await self._collect_video_links(page):
+                    return True
+            except RuntimeError:
+                raise
+            except Exception as exc:  # noqa: BLE001
+                self.log(f"访问候选分页失败: {candidate} -> {exc}")
+        return False
+
+    # ---- list page -------------------------------------------------------
+
+    async def _collect_video_links(self, page) -> list[str]:
+        await self._assert_not_blocked(page)
+        current = page.url
+        structured = await page.evaluate(
+            """() => {
+                const rows = [];
+                for (const item of document.querySelectorAll('.item.video')) {
+                    const a = item.querySelector('a[href*="/video/"]');
+                    if (!a) continue;
+                    const href = new URL(a.getAttribute('href'), location.href).href;
+                    rows.push({href});
+                }
+                return rows;
+            }"""
+        )
+        seen: set[str] = set()
+        links: list[str] = []
+        for row in structured:
+            href = row.get("href", "")
+            if href in seen or href == current or not self.is_detail_url(href):
+                continue
+            seen.add(href)
+            links.append(href)
+            if len(links) >= self.items_per_page:
+                break
+        if links:
+            return links
+
+        # 兜底:从整页 HTML 里正则抠 /video/*.html
+        try:
+            html = await page.content()
+        except Exception:
+            html = ""
+        for href in self._extract_detail_urls_from_text(html, current):
+            if href in seen or href == current:
+                continue
+            seen.add(href)
+            links.append(href)
+            if len(links) >= self.items_per_page:
+                break
+        return links
+
+    # ---- detail page -----------------------------------------------------
+
+    async def scrape_detail(self, detail_url: str, page_no: int) -> ScrapedItem:
+        page = await self.context.new_page()
+        media_urls: list[str] = []
+
+        def remember_response(response):
+            url = response.url
+            lower = url.lower().split("?", 1)[0]
+            if any(lower.endswith(ext) for ext in (".mp4", ".m3u8", ".mov", ".webm")):
+                media_urls.append(url)
+
+        page.on("response", remember_response)
+        try:
+            await self._visit(page, detail_url)
+            await self._assert_not_blocked(page)
+            await self._handle_interstitial_ad(page, detail_url)
+            title = await self._extract_title(page)
+            cover_url = await self._find_cover_url(page)
+            video_url = await self._find_video_url(page, media_urls)
+            self.log(f"采集: {title} | cover={'有' if cover_url else '无'} | video={video_url or '无'}")
+            from .media import extract_video_id
+            return ScrapedItem(
+                source_url=detail_url,
+                title=title,
+                cover_image=cover_url,
+                video_url=video_url,
+                page_no=page_no,
+                video_id=extract_video_id(detail_url),
+            )
+        finally:
+            await page.close()
+
+    # ---- ad / overlay handling (无头时只是尽力,不卡死) ----------------------
+
+    async def _has_ad_overlay(self, page) -> bool:
+        try:
+            return bool(await page.evaluate(
+                """() => {
+                    const bodyText = document.body.innerText || '';
+                    if (bodyText.includes('请点击广告后继续浏览')) return true;
+                    const nodes = Array.from(document.querySelectorAll('div, a, img'));
+                    return nodes.some(el => {
+                        const rect = el.getBoundingClientRect();
+                        const style = getComputedStyle(el);
+                        const z = Number.parseInt(style.zIndex || '0', 10) || 0;
+                        const text = el.innerText || el.alt || '';
+                        const visible = rect.width > 100 && rect.height > 40 &&
+                            style.display !== 'none' && style.visibility !== 'hidden' &&
+                            Number(style.opacity || '1') > 0.2;
+                        return visible && z >= 100 && /请点击广告|继续浏览|点击后当天不再出现/.test(text);
+                    });
+                }"""
+            ))
+        except Exception:
+            return False
+
+    async def _handle_interstitial_ad(self, page, expected_url: str) -> None:
+        expected_path = urlparse(expected_url).path
+        for _ in range(3):
+            self._check_cancel()
+            await page.wait_for_timeout(1000)
+            parsed = urlparse(page.url)
+            on_detail = parsed.netloc.endswith("xchina.co") and parsed.path == expected_path
+            if on_detail and not await self._has_ad_overlay(page):
+                try:
+                    if await page.locator(".tab-nav, .video-detail").count():
+                        return
+                except Exception:
+                    return
+            # 跳到了广告/中转页 -> 后退或重开
+            parsed = urlparse(page.url)
+            if not (parsed.netloc.endswith("xchina.co") and parsed.path == expected_path):
+                try:
+                    await page.go_back(wait_until="domcontentloaded", timeout=20000)
+                    await page.wait_for_timeout(1200)
+                except Exception:
+                    await self._visit(page, expected_url)
+        parsed = urlparse(page.url)
+        if not (parsed.netloc.endswith("xchina.co") and parsed.path == expected_path):
+            await self._visit(page, expected_url)
+        try:
+            await page.wait_for_selector(".video-detail, .tab-nav", timeout=10000)
+        except PlaywrightTimeoutError:
+            pass
+
+    async def _click_tab_by_text(self, page, text: str) -> None:
+        for selector in (
+            f"text={text}", f"a:has-text('{text}')", f"button:has-text('{text}')",
+            f"li:has-text('{text}')", f"div:has-text('{text}')",
+        ):
+            try:
+                loc = page.locator(selector).first
+                if await loc.count() and await loc.is_visible():
+                    await loc.click(timeout=5000)
+                    return
+            except Exception:
+                continue
+
+    # ---- title -----------------------------------------------------------
+
+    async def _extract_title(self, page) -> str:
+        title = ""
+        for _ in range(4):
+            await self._click_tab_by_text(page, "影片信息")
+            await page.wait_for_timeout(600)
+            title = await page.evaluate(
+                """() => {
+                    function clean(t){ return (t || '').replace(/\\s+/g, ' ').trim(); }
+                    const infoTitle = document.querySelector('.video-detail .item .text');
+                    const infoText = clean(infoTitle && infoTitle.innerText);
+                    if (infoText && infoText.length >= 2 && infoText.length <= 180) return infoText;
+                    for (const script of document.querySelectorAll('script[type="application/ld+json"]')) {
+                        try {
+                            const data = JSON.parse(script.textContent || 'null');
+                            const rows = Array.isArray(data) ? data : [data];
+                            for (const row of rows) {
+                                if (row && row['@type'] === 'VideoObject' && row.name) return clean(row.name);
+                            }
+                        } catch (_) {}
+                    }
+                    const og = document.querySelector('meta[property="og:title"], meta[name="twitter:title"]');
+                    const ogText = clean(og && og.getAttribute('content'));
+                    if (ogText) return ogText;
+                    const h = document.querySelector('h1, h2, .video-title, .movie-title');
+                    const hText = clean(h && h.innerText);
+                    if (hText) return hText;
+                    return clean(document.title);
+                }"""
+            )
+            title = self._clean_title(title)
+            if self._is_good_title(title):
+                break
+            await page.wait_for_timeout(1200)
+        if not self._is_good_title(title):
+            m = re.search(r"/video/id-([A-Za-z0-9_-]+)\.html", page.url)
+            title = f"video-{m.group(1)}" if m else "未命名"
+        return title[:500]
+
+    @staticmethod
+    def _clean_title(title: str) -> str:
+        title = re.split(r"\s*[-_|]\s*(XChina|xchina|AV视频|成人视频|小黄书).*$", title or "", flags=re.I)[0]
+        return re.sub(r"^【[^】]+】", "", title).strip()
+
+    @staticmethod
+    def _is_good_title(title: str) -> bool:
+        if not title or len(title.strip()) < 2:
+            return False
+        bad = ["广告", "推广", "澳门", "新葡京", "blocked", "cloudflare", "attention required", "未命名"]
+        low = title.lower()
+        return not any(w.lower() in low for w in bad)
+
+    # ---- cover -----------------------------------------------------------
+
+    # 明显不是封面的图: 站点 logo / 占位图 / 图标 / 广告
+    _COVER_BLOCKLIST = ("logo", "avatar", "icon", "banner", "ads", "adserver", "/images/sites/", "/static/", "placeholder")
+
+    def _bad_cover(self, src: str) -> bool:
+        if not src or src.startswith("data:"):
+            return True
+        low = src.lower()
+        return any(w in low for w in self._COVER_BLOCKLIST)
+
+    async def _find_cover_url(self, page) -> str | None:
+        # 1) 截屏 tab 里的截图
+        await self._click_tab_by_text(page, "截屏")
+        await page.wait_for_timeout(800)
+        screenshot_cover = await page.evaluate(
+            """() => {
+                const img = document.querySelector('.tab-content.active .screenshot-container img, .screenshot-container img');
+                if (!img) return null;
+                const src = img.currentSrc || img.src || img.getAttribute('data-src') || img.getAttribute('data-original') || '';
+                return src ? new URL(src, location.href).href : null;
+            }"""
+        )
+        if screenshot_cover and not self._bad_cover(screenshot_cover):
+            return screenshot_cover
+
+        # 2) og:image / twitter:image / JSON-LD thumbnailUrl / <video poster> —— 通常就是真封面
+        meta_cover = await page.evaluate(
+            """() => {
+                function abs(u){ try { return u ? new URL(u, location.href).href : null; } catch(_) { return null; } }
+                const og = document.querySelector('meta[property="og:image"], meta[name="og:image"], meta[name="twitter:image"], meta[property="twitter:image"]');
+                if (og && og.getAttribute('content')) return abs(og.getAttribute('content'));
+                for (const s of document.querySelectorAll('script[type="application/ld+json"]')) {
+                    try {
+                        const data = JSON.parse(s.textContent || 'null');
+                        const rows = Array.isArray(data) ? data : [data];
+                        for (const r of rows) {
+                            if (!r) continue;
+                            let t = r.thumbnailUrl || r.thumbnail || (r.image && (r.image.url || r.image));
+                            if (Array.isArray(t)) t = t[0];
+                            if (typeof t === 'string') return abs(t);
+                        }
+                    } catch(_){}
+                }
+                const v = document.querySelector('video[poster]');
+                if (v && v.getAttribute('poster')) return abs(v.getAttribute('poster'));
+                return null;
+            }"""
+        )
+        if meta_cover and not self._bad_cover(meta_cover):
+            return meta_cover
+
+        # 3) 兜底: 页面里最靠上/最大的一张正常图
+        images = await page.evaluate(
+            """() => {
+                const rows = [];
+                for (const img of document.querySelectorAll('img')) {
+                    const rect = img.getBoundingClientRect();
+                    const style = getComputedStyle(img);
+                    const src = img.currentSrc || img.src || img.getAttribute('data-src') || img.getAttribute('data-original') || '';
+                    if (!src || style.display === 'none' || style.visibility === 'hidden') continue;
+                    rows.push({
+                        src: new URL(src, location.href).href,
+                        width: img.naturalWidth || rect.width,
+                        height: img.naturalHeight || rect.height,
+                        x: rect.x, y: rect.y,
+                        area: Math.max(rect.width,1) * Math.max(rect.height,1)
+                    });
+                }
+                return rows;
+            }"""
+        )
+        candidates = [
+            img for img in images
+            if not self._bad_cover(img.get("src", "")) and img.get("width", 0) >= 180 and img.get("height", 0) >= 120
+        ]
+        if not candidates:
+            return meta_cover or screenshot_cover or None
+        candidates.sort(key=lambda x: (x.get("y", 99999), x.get("x", 99999), -x.get("area", 0)))
+        return candidates[0]["src"]
+
+    # ---- video url -------------------------------------------------------
+
+    async def _find_video_url(self, page, media_urls: list[str]) -> str | None:
+        await self._click_tab_by_text(page, "下载")
+        await page.wait_for_timeout(800)
+        try:
+            page_source = await page.content()
+        except Exception:
+            page_source = ""
+        for url in self._extract_media_urls_from_text(page_source):
+            return url
+        direct = await page.evaluate(
+            """() => {
+                const attrs = ['href', 'src', 'data-url', 'data-src', 'data-href'];
+                const nodes = Array.from(document.querySelectorAll('a, button, video, source, [data-url], [data-src], [data-href]'));
+                const good = [];
+                for (const node of nodes) {
+                    for (const attr of attrs) {
+                        const raw = node.getAttribute(attr);
+                        if (!raw) continue;
+                        let url; try { url = new URL(raw, location.href).href; } catch (_) { continue; }
+                        const lower = url.toLowerCase();
+                        if (/(\\.mp4|\\.m3u8|\\.webm|\\.mov)(\\?|$)/.test(lower) || /download|downurl/.test(lower)) good.push(url);
+                    }
+                }
+                return good;
+            }"""
+        )
+        for url in direct:
+            if url:
+                return url
+        # 网络请求里嗅到的媒体
+        if media_urls:
+            return media_urls[-1]
+        await page.wait_for_timeout(2000)
+        if media_urls:
+            return media_urls[-1]
+        return None

docker-entrypoint.sh

@@ -0,0 +1,17 @@
+#!/bin/sh
+set -e
+# 有头模式(HEADLESS=0)需要一个 X server。手动起 Xvfb(比 xvfb-run 省事、不依赖 xauth)。
+if [ "${HEADLESS:-0}" = "0" ] || [ "${HEADLESS:-0}" = "false" ]; then
+  : "${DISPLAY:=:99}"
+  export DISPLAY
+  rm -f /tmp/.X99-lock 2>/dev/null || true
+  Xvfb "$DISPLAY" -screen 0 "${XVFB_SCREEN:-1366x900x24}" -nolisten tcp >/tmp/xvfb.log 2>&1 &
+  XVFB_PID=$!
+  # 等 Xvfb 起来
+  for _ in 1 2 3 4 5 6 7 8 9 10; do
+    if [ -e /tmp/.X99-lock ] || xdpyinfo -display "$DISPLAY" >/dev/null 2>&1; then break; fi
+    sleep 0.3
+  done
+  trap 'kill "$XVFB_PID" 2>/dev/null || true' EXIT INT TERM
+fi
+exec "$@"

requirements.txt

@@ -0,0 +1,6 @@
+playwright>=1.44.0
+fastapi>=0.110.0
+uvicorn[standard]>=0.29.0
+requests>=2.31.0
+yt-dlp>=2024.1.1
+minio>=7.2.0

server/app.py

@@ -0,0 +1,122 @@
+"""content-harvester —— 内容采集服务,实现主站 lobster 执行器的调用契约。
+
+当前内置一个 xchina.co 的 scraper(core/scraper.py),以后可加更多来源。
+对接 app/services/lobster_client.py::call_lobster_agent:
+  POST /agents/{agent_id}/invoke   body = 主站合并后的 payload, 返回 {"task_id": ...}
+  GET  /tasks/{task_id}            返回 {"status": running|succeeded|failed, "output": ..., "error": ...}
+
+鉴权: 主站带 `Authorization: Bearer <lobster_api_key>`,需与本服务 env HARVESTER_TOKEN 一致。
+SSRF 防护: 只接受 ALLOWED_HOSTS(默认 xchina.co 及其子域) 的 content_url。
+"""
+from __future__ import annotations
+
+import logging
+import os
+from urllib.parse import urlparse
+
+from fastapi import FastAPI, Header, HTTPException
+from pydantic import BaseModel
+
+from core.media import MinioConfig
+from server import jobs
+
+logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s %(message)s")
+logger = logging.getLogger("harvester.app")
+
+SERVICE_TOKEN = os.environ.get("HARVESTER_TOKEN", "")
+HEADLESS = os.environ.get("HEADLESS", "1") not in ("0", "false", "False")
+ALLOWED_HOSTS = tuple(
+    h.strip().lower() for h in os.environ.get("ALLOWED_HOSTS", "xchina.co").split(",") if h.strip()
+)
+MAX_PAGES = int(os.environ.get("MAX_PAGES", "50"))
+
+app = FastAPI(title="content-harvester", version="1.0")
+
+
+def _check_auth(authorization: str | None) -> None:
+    if not SERVICE_TOKEN:
+        # 未配置 token 时不鉴权(仅适合内网/本地);生产务必设 HARVESTER_TOKEN
+        return
+    expected = f"Bearer {SERVICE_TOKEN}"
+    if authorization != expected:
+        raise HTTPException(status_code=401, detail="invalid token")
+
+
+def _host_allowed(url: str) -> bool:
+    try:
+        host = (urlparse(url).hostname or "").lower()
+    except Exception:
+        return False
+    if not host:
+        return False
+    return any(host == h or host.endswith("." + h) for h in ALLOWED_HOSTS)
+
+
+class TaskStatusResponse(BaseModel):
+    status: str
+    output: dict | None = None
+    error: str | None = None
+
+
+@app.get("/healthz")
+async def healthz():
+    return {"ok": True, "headless": HEADLESS, "allowed_hosts": list(ALLOWED_HOSTS)}
+
+
+@app.post("/agents/{agent_id}/invoke")
+async def invoke(agent_id: str, payload: dict, authorization: str | None = Header(default=None)):
+    _check_auth(authorization)
+
+    # payload 即主站 task.input_params:含 content_url / platform_name / source_kind / start_page / end_page / _crawl_config ...
+    source_kind = payload.get("source_kind")
+    if source_kind is not None and str(source_kind) != "website":
+        raise HTTPException(status_code=400, detail=f"unsupported source_kind: {source_kind}(本服务只处理 website)")
+
+    content_url = (payload.get("content_url") or payload.get("series_url") or payload.get("url") or "").strip()
+    if not content_url:
+        raise HTTPException(status_code=400, detail="missing content_url")
+    if not _host_allowed(content_url):
+        raise HTTPException(status_code=400, detail=f"host not allowed: {content_url}")
+
+    def _as_int(v, default):
+        try:
+            return int(v)
+        except (TypeError, ValueError):
+            return default
+
+    start_page = max(1, _as_int(payload.get("start_page"), 1))
+    end_page = max(start_page, _as_int(payload.get("end_page"), start_page))
+    if end_page - start_page + 1 > MAX_PAGES:
+        end_page = start_page + MAX_PAGES - 1
+
+    source_name = str(payload.get("platform_name") or "xchina")[:100]
+
+    # MinIO 配置由主站经 lobster payload 下发(主站把绑定的 minio_* Skill 变量并入 payload);
+    # 缺失则降级:不上传,video_url 保留原始 m3u8 地址
+    minio_cfg = MinioConfig.from_payload(payload)
+
+    job = jobs.submit(content_url, source_name, start_page, end_page, headless=HEADLESS, minio_cfg=minio_cfg)
+    logger.info(
+        "invoke agent=%s job=%s url=%s pages=%s-%s minio=%s",
+        agent_id, job.id, content_url, start_page, end_page,
+        "on" if minio_cfg.is_complete else "off",
+    )
+    return {"task_id": job.id, "status": job.status}
+
+
+@app.get("/tasks/{task_id}")
+async def get_task(task_id: str, authorization: str | None = Header(default=None)):
+    _check_auth(authorization)
+    job = jobs.get_job(task_id)
+    if not job:
+        raise HTTPException(status_code=404, detail="task not found")
+    return job.to_dict()
+
+
+@app.post("/tasks/{task_id}/cancel")
+async def cancel_task(task_id: str, authorization: str | None = Header(default=None)):
+    _check_auth(authorization)
+    ok = jobs.cancel(task_id)
+    if not ok:
+        raise HTTPException(status_code=409, detail="not cancellable")
+    return {"ok": True}

server/jobs.py

@@ -0,0 +1,222 @@
+"""内存级作业注册表 + 后台执行。
+
+- 单并发:同一时刻只跑一个浏览器作业(xchina 怕风控);用 asyncio.Lock 串行化。
+- 作业状态只存进程内存,重启即丢 —— 主站靠轮询 GET /tasks/{id} 拿结果,够用。
+- 流程:① headed Chrome/xvfb 抓元数据 ② 若给了完整 MinIO 配置则逐条下载 m3u8→mp4 + 封面,
+  上传 MinIO,把 video_url/cover_image 改成 MinIO 地址;失败的条目降级保留原 URL。
+- 临时下载目录用完即删,不长期落盘。
+"""
+from __future__ import annotations
+
+import asyncio
+import logging
+import os
+import shutil
+import time
+import uuid
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any, Optional
+
+from core.browser import browser_context
+from core.media import (
+    MinioConfig,
+    download_cover,
+    download_video,
+    guess_content_type,
+    upload_to_minio,
+)
+from core.scraper import ScrapeCancelled, XChinaScraper
+
+logger = logging.getLogger("harvester.jobs")
+
+_RUN_LOCK = asyncio.Lock()
+
+TMP_ROOT = Path(os.environ.get("TMP_DIR", "/tmp")) / "content-harvester"
+CONCURRENT_FRAGMENTS = int(os.environ.get("CONCURRENT_FRAGMENTS", "8"))
+MAX_VIDEO_MB = int(os.environ.get("MAX_VIDEO_MB", "0"))  # 0 = 不限
+
+
+@dataclass
+class Job:
+    id: str
+    series_url: str
+    source_name: str
+    start_page: int
+    end_page: int
+    minio_cfg: MinioConfig = field(default_factory=MinioConfig)
+    status: str = "running"           # running | succeeded | failed
+    output: Optional[dict[str, Any]] = None
+    error: Optional[str] = None
+    created_at: float = field(default_factory=time.time)
+    finished_at: Optional[float] = None
+    log_lines: list[str] = field(default_factory=list)
+    _cancelled: bool = False
+    _task: Optional[asyncio.Task] = None
+
+    def to_dict(self) -> dict[str, Any]:
+        return {
+            "task_id": self.id,
+            "status": self.status,
+            "output": self.output,
+            "error": self.error,
+            "created_at": self.created_at,
+            "finished_at": self.finished_at,
+            "log_tail": self.log_lines[-80:],
+        }
+
+
+_JOBS: dict[str, Job] = {}
+
+
+def get_job(job_id: str) -> Optional[Job]:
+    return _JOBS.get(job_id)
+
+
+def _log(job: Job, msg: str) -> None:
+    # 注意:不要把 secret_key 等敏感值传进来
+    line = f"{time.strftime('%H:%M:%S')} {msg}"
+    job.log_lines.append(line)
+    if len(job.log_lines) > 800:
+        del job.log_lines[: len(job.log_lines) - 800]
+    logger.info("[job %s] %s", job.id, msg)
+
+
+def _bytes_h(n: int) -> str:
+    f = float(n)
+    for unit in ("B", "KB", "MB", "GB"):
+        if f < 1024 or unit == "GB":
+            return f"{f:.1f}{unit}"
+        f /= 1024
+    return f"{f:.1f}GB"
+
+
+async def _process_uploads(job: Job, result, tmp_dir: Path) -> None:
+    """对 result.resources 逐条:下载视频+封面 → 上传 MinIO → 回写 video_url/cover_image。失败降级。"""
+    cfg = job.minio_cfg
+    total = len(result.resources)
+    for idx, item in enumerate(result.resources, 1):
+        if job._cancelled:
+            raise ScrapeCancelled()
+        vid = item.video_id or f"item{idx}"
+        item_dir = tmp_dir / vid
+        try:
+            if not item.video_url:
+                item.extra["upload_skipped_reason"] = "no video url"
+                _log(job, f"[{idx}/{total}] {item.title}: 无视频地址,跳过上传")
+                continue
+            _log(job, f"[{idx}/{total}] {item.title}: 下载视频 …")
+            mp4_path = await asyncio.to_thread(
+                download_video, item.video_url, item.source_url, item_dir, vid,
+                CONCURRENT_FRAGMENTS, MAX_VIDEO_MB,
+            )
+            size = mp4_path.stat().st_size
+            video_key = f"xchina/{vid}{mp4_path.suffix.lower() or '.mp4'}"
+            _log(job, f"[{idx}/{total}] {item.title}: 视频 {_bytes_h(size)},上传 MinIO {video_key} …")
+            video_public = await asyncio.to_thread(
+                upload_to_minio, mp4_path, video_key, "video/mp4", cfg,
+            )
+            item.extra["original_m3u8"] = item.video_url
+            item.extra["minio_object_video"] = video_key
+            item.extra["video_bytes"] = size
+            item.video_url = video_public
+
+            # 封面(失败不致命)
+            if item.cover_image:
+                cover_path = await asyncio.to_thread(
+                    download_cover, item.cover_image, item.source_url, item_dir, vid,
+                )
+                if cover_path is not None:
+                    cover_key = f"xchina/{vid}{cover_path.suffix.lower() or '.jpg'}"
+                    try:
+                        cover_public = await asyncio.to_thread(
+                            upload_to_minio, cover_path, cover_key, guess_content_type(cover_path, "image/jpeg"), cfg,
+                        )
+                        item.extra["original_cover"] = item.cover_image
+                        item.extra["minio_object_cover"] = cover_key
+                        item.cover_image = cover_public
+                    except Exception as exc:  # noqa: BLE001
+                        _log(job, f"[{idx}/{total}] {item.title}: 封面上传失败({exc}),保留原图床地址")
+            _log(job, f"[{idx}/{total}] {item.title}: OK -> {item.video_url}")
+        except ScrapeCancelled:
+            raise
+        except Exception as exc:  # noqa: BLE001
+            reason = f"{type(exc).__name__}: {exc}"
+            item.extra["upload_skipped_reason"] = reason
+            _log(job, f"[{idx}/{total}] {item.title}: 下载/上传失败,降级保留原 m3u8 地址 -> {reason}")
+        finally:
+            shutil.rmtree(item_dir, ignore_errors=True)
+
+
+async def _run(job: Job, headless: bool) -> None:
+    tmp_dir = TMP_ROOT / job.id
+    try:
+        async with _RUN_LOCK:
+            _log(job, f"开始采集: {job.series_url} (page {job.start_page}-{job.end_page})")
+            if job.minio_cfg.is_complete:
+                _log(job, f"启用 MinIO 上传: {job.minio_cfg.safe_repr()}")
+            else:
+                _log(job, "未提供完整 MinIO 配置,本次不上传(video_url 保留原始 m3u8 地址)")
+
+            async with browser_context(headless=headless) as ctx:
+                scraper = XChinaScraper(
+                    ctx,
+                    is_cancelled=lambda: job._cancelled,
+                    log=lambda m: _log(job, m),
+                )
+                result = await scraper.run(job.series_url, job.start_page, job.end_page)
+            # 浏览器已关闭;开始下载+上传(若配置完整)
+            if job.minio_cfg.is_complete:
+                await _process_uploads(job, result, tmp_dir)
+            else:
+                for item in result.resources:
+                    item.extra.setdefault("upload_skipped_reason", "minio config absent")
+
+            # 统一覆盖 source_name(invoke 里指定的 platform_name)
+            for item in result.resources:
+                item.source_name = job.source_name
+            output = result.to_output()
+            job.output = output
+            if not output["resources"] and output["item_errors"]:
+                job.status = "failed"
+                job.error = "no resources scraped; " + "; ".join(
+                    e.get("reason", "") for e in output["item_errors"][:3]
+                )
+            else:
+                job.status = "succeeded"
+            uploaded = sum(1 for r in output["resources"] if r.get("extra_data", {}).get("minio_object_video"))
+            _log(job, f"完成: {len(output['resources'])} 条(其中 {uploaded} 条已上传 MinIO), 页数 {output['pages_scanned']}, 抓取失败 {len(output['item_errors'])}")
+    except ScrapeCancelled:
+        job.status = "failed"
+        job.error = "cancelled"
+        _log(job, "已取消")
+    except Exception as exc:  # noqa: BLE001
+        job.status = "failed"
+        job.error = f"{type(exc).__name__}: {exc}"
+        _log(job, f"失败: {job.error}")
+    finally:
+        shutil.rmtree(tmp_dir, ignore_errors=True)
+        job.finished_at = time.time()
+
+
+def submit(series_url: str, source_name: str, start_page: int, end_page: int,
+           headless: bool = True, minio_cfg: Optional[MinioConfig] = None) -> Job:
+    job = Job(
+        id=uuid.uuid4().hex,
+        series_url=series_url,
+        source_name=source_name or "xchina",
+        start_page=max(1, start_page),
+        end_page=max(start_page, end_page),
+        minio_cfg=minio_cfg or MinioConfig(),
+    )
+    _JOBS[job.id] = job
+    job._task = asyncio.create_task(_run(job, headless))
+    return job
+
+
+def cancel(job_id: str) -> bool:
+    job = _JOBS.get(job_id)
+    if not job or job.status != "running":
+        return False
+    job._cancelled = True
+    return True