Banners, additional images, and files upload to Dropbox.
Dropbox file IDs are saved with the article in Supabase. Banners appear in thumbnails and behind the article title/subtitle. Additional images and files appear below the article text.
This commit is contained in:
@@ -77,12 +77,31 @@ Use **+ Section** in the editor to add a section heading and opening paragraph.
|
|||||||
|
|
||||||
The media picker supports drag-and-drop or file browsing, a live banner preview with the article title and subtitle, file-size and format checks, and duplicate detection by file name and size. Uploads have individual progress, cancel, and retry controls; a failed file does not stop the remaining queue. Retry or remove pending/failed uploads before publishing. Use the up/down controls to set the order of additional photos and files. Replacing a banner keeps the previous image until the replacement uploads successfully.
|
The media picker supports drag-and-drop or file browsing, a live banner preview with the article title and subtitle, file-size and format checks, and duplicate detection by file name and size. Uploads have individual progress, cancel, and retry controls; a failed file does not stop the remaining queue. Retry or remove pending/failed uploads before publishing. Use the up/down controls to set the order of additional photos and files. Replacing a banner keeps the previous image until the replacement uploads successfully.
|
||||||
|
|
||||||
The publisher accepts a banner image (JPEG, PNG, WebP, or GIF, up to 8 MB) and up to ten additional photos or files (20 MB each). The banner appears in the journal thumbnail and behind the article title and subtitle with a readability overlay; additional photos and downloadable files appear below the body. Uploads require an admin session. File contents are saved in `backend/data/uploads` in both article storage modes; set `JOURNAL_UPLOAD_DIR` to a persistent mounted directory in production and include it in backups. Uploaded files are publicly accessible by their generated URLs. Removing a selection from an unpublished draft does not delete its stored upload.
|
The publisher accepts a banner image (JPEG, PNG, WebP, or GIF, up to 8 MB) and up to ten additional photos or files (20 MB each). The banner appears in the journal thumbnail and behind the article title and subtitle with a readability overlay; additional photos and downloadable files appear below the body. Uploads require an admin session. New uploads use Dropbox by default. The Supabase article's `banner` and each item in `attachments` contain `storage: "dropbox"` and `dropbox_file_id: "id:..."`, alongside their names, sizes, media types, and stable website URLs.
|
||||||
|
|
||||||
|
The `article_media` table stores the upload-ID-to-Dropbox-ID mapping, including uploads not yet attached to a published article. The public `/api/uploads/{upload_id}` route retrieves bytes from Dropbox by ID, so article rows never contain expiring temporary links or access tokens. Original local uploads continue to work until migrated. Removing an attachment or deleting an article does not delete its stored file.
|
||||||
|
|
||||||
For an existing Supabase project, run `bash scripts/setup_supabase.sh --schema-only` to add the nullable `banner` and default-empty `attachments` columns before running the updated backend. Existing articles remain compatible.
|
For an existing Supabase project, run `bash scripts/setup_supabase.sh --schema-only` to add the nullable `banner` and default-empty `attachments` columns before running the updated backend. Existing articles remain compatible.
|
||||||
|
|
||||||
Verify article publishing and uploads with `.venv/Scripts/python.exe -m unittest backend.test_articles`.
|
Verify article publishing and uploads with `.venv/Scripts/python.exe -m unittest backend.test_articles`.
|
||||||
|
|
||||||
|
## Connect the owner's Dropbox account
|
||||||
|
|
||||||
|
This website uses its own server-side Dropbox API app, independently of journal admin sign-in or any chat connector. A personal Dropbox account is sufficient; it must authorize the website's developer app once.
|
||||||
|
|
||||||
|
1. In the [Dropbox App Console](https://www.dropbox.com/developers/apps), create a **Scoped access** app with **App folder** access. Enable `files.content.write` and `files.content.read` on its Permissions tab and click **Submit** to save. Keep this app's permissions limited to those needed for the website.
|
||||||
|
2. Set `DROPBOX_APP_KEY` and `DROPBOX_APP_SECRET` in `backend/.env` using the app's Settings tab. Keep these values server-side.
|
||||||
|
3. Run `.venv/Scripts/python.exe -m backend.connect_dropbox`. The helper opens the authorization URL in your browser (and prints it as a fallback). Approve access using the owner's personal account, and paste the returned code into the terminal. It uses the app's saved permissions and verifies that both required file scopes were granted before writing the refresh token directly into the ignored `backend/.env` file without printing it. Setup gives credentials in this file priority over shell variables.
|
||||||
|
4. Apply the media table with `bash scripts/setup_supabase.sh --schema-only`, if not already applied, then restart FastAPI with `--env-file backend/.env`.
|
||||||
|
|
||||||
|
If an old authorization link shows **No scope requested can be granted for this app**, rerun the updated helper and use its newly opened page. It no longer sends an explicit `scope` parameter. If setup reports missing permissions after approval, compare the App key printed by the helper with the app you edited in App Console; enable the listed permissions on that exact app, save with Submit, and rerun setup. Old links are not changed by editing the helper.
|
||||||
|
|
||||||
|
The SDK refreshes access tokens automatically using `DROPBOX_REFRESH_TOKEN`. See the [Dropbox OAuth guide](https://developers.dropbox.com/oauth-guide) for offline access. `DROPBOX_ACCESS_TOKEN` is supported for initial testing, but a short-lived token alone will eventually need replacing. New uploads fail clearly if Dropbox is not connected; they do not silently fall back to local storage.
|
||||||
|
|
||||||
|
To move existing published media to Dropbox, preview with `.venv/Scripts/python.exe -m backend.migrate_media_dropbox`, then run it with `--apply`. It uploads referenced files, updates the article rows, and preserves website image URLs and the original local files. Run the migration while article editing is paused. It can resume after a partial failure without re-uploading successfully recorded files.
|
||||||
|
|
||||||
|
For offline development only, set `JOURNAL_MEDIA_STORAGE=local`. Local files and legacy uploads use `JOURNAL_UPLOAD_DIR` (default `backend/data/uploads`). Dropbox uploads with Supabase configured do not require local file storage. If running without Supabase, upload metadata remains in that local directory and needs persistent storage.
|
||||||
|
|
||||||
## Supabase article storage
|
## Supabase article storage
|
||||||
|
|
||||||
The runtime backend needs two values from **Supabase Dashboard → Settings → API Keys**:
|
The runtime backend needs two values from **Supabase Dashboard → Settings → API Keys**:
|
||||||
|
|||||||
@@ -21,6 +21,14 @@ JOURNAL_TOKEN_SECRET=replace-with-a-long-random-secret
|
|||||||
# Mount durable storage here when deploying to a host with an ephemeral filesystem.
|
# Mount durable storage here when deploying to a host with an ephemeral filesystem.
|
||||||
# JOURNAL_UPLOAD_DIR=/var/lib/portfolio/uploads
|
# JOURNAL_UPLOAD_DIR=/var/lib/portfolio/uploads
|
||||||
|
|
||||||
|
# Article uploads go to Dropbox; use local only for offline development.
|
||||||
|
JOURNAL_MEDIA_STORAGE=dropbox
|
||||||
|
DROPBOX_APP_KEY=
|
||||||
|
DROPBOX_APP_SECRET=
|
||||||
|
DROPBOX_REFRESH_TOKEN=
|
||||||
|
# Optional short-lived token for initial testing; refresh tokens are preferred.
|
||||||
|
DROPBOX_ACCESS_TOKEN=
|
||||||
|
|
||||||
# Server-only Supabase article storage. Prefer the new sb_secret_ key.
|
# Server-only Supabase article storage. Prefer the new sb_secret_ key.
|
||||||
SUPABASE_URL=https://your-project-ref.supabase.co
|
SUPABASE_URL=https://your-project-ref.supabase.co
|
||||||
SUPABASE_SECRET_KEY=sb_secret_replace_me
|
SUPABASE_SECRET_KEY=sb_secret_replace_me
|
||||||
|
|||||||
@@ -0,0 +1,64 @@
|
|||||||
|
"""Run python -m backend.connect_dropbox to authorize the owner's Dropbox once."""
|
||||||
|
import getpass
|
||||||
|
import os
|
||||||
|
import webbrowser
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from dotenv import dotenv_values, set_key
|
||||||
|
from dropbox import DropboxOAuth2FlowNoRedirect
|
||||||
|
|
||||||
|
REQUIRED_SCOPES = frozenset({"files.content.write", "files.content.read"})
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
env_path = Path(__file__).with_name(".env")
|
||||||
|
# Setup edits this file, so a stale shell variable must not select another app.
|
||||||
|
settings = {**os.environ, **dotenv_values(env_path)}
|
||||||
|
key = (settings.get("DROPBOX_APP_KEY") or "").strip()
|
||||||
|
secret = (settings.get("DROPBOX_APP_SECRET") or "").strip()
|
||||||
|
if not key or not secret:
|
||||||
|
raise SystemExit("Add DROPBOX_APP_KEY and DROPBOX_APP_SECRET to backend/.env first. Create the app at https://www.dropbox.com/developers/apps (App Folder access).")
|
||||||
|
flow = DropboxOAuth2FlowNoRedirect(
|
||||||
|
key, secret, token_access_type="offline",
|
||||||
|
# Omitting scope asks Dropbox for the permissions saved in App Console.
|
||||||
|
# Check the actual grant below before persisting credentials.
|
||||||
|
timeout=30,
|
||||||
|
)
|
||||||
|
url = flow.start()
|
||||||
|
print(f"Using App key: {key} (compare with your Dropbox app's Settings tab).")
|
||||||
|
print("Requesting the permissions saved in that app's Permissions tab.")
|
||||||
|
print("Enable files.content.read and files.content.write there, then click Submit.")
|
||||||
|
print("Opening Dropbox. Review the permissions and approve using the owner's account.")
|
||||||
|
print("If the browser does not open, copy this entire URL:")
|
||||||
|
print(url)
|
||||||
|
try:
|
||||||
|
webbrowser.open(url)
|
||||||
|
except webbrowser.Error:
|
||||||
|
pass # The printed URL also works on servers without a browser.
|
||||||
|
code = getpass.getpass("Paste the authorization code here (hidden): ").strip()
|
||||||
|
if not code:
|
||||||
|
raise SystemExit("No authorization code entered; no credentials were saved.")
|
||||||
|
try:
|
||||||
|
result = flow.finish(code)
|
||||||
|
except Exception as exc:
|
||||||
|
raise SystemExit(f"Dropbox authorization failed ({type(exc).__name__}). Try again; no credentials were saved.") from None
|
||||||
|
if not result.refresh_token:
|
||||||
|
raise SystemExit("Dropbox did not issue a refresh token. Run setup again.")
|
||||||
|
granted_scopes = result.scope or []
|
||||||
|
if isinstance(granted_scopes, str):
|
||||||
|
granted_scopes = granted_scopes.split()
|
||||||
|
missing = REQUIRED_SCOPES - set(granted_scopes)
|
||||||
|
if missing:
|
||||||
|
raise SystemExit(
|
||||||
|
"Dropbox authorization succeeded, but the app did not grant: "
|
||||||
|
+ ", ".join(sorted(missing))
|
||||||
|
+ f". In App Console, select the app with App key {key}, enable these "
|
||||||
|
"permissions, click Submit, and rerun setup. No credentials were saved."
|
||||||
|
)
|
||||||
|
set_key(str(env_path), "DROPBOX_REFRESH_TOKEN", result.refresh_token)
|
||||||
|
set_key(str(env_path), "JOURNAL_MEDIA_STORAGE", "dropbox")
|
||||||
|
print("Dropbox connected. Refresh token saved in backend/.env. Restart FastAPI, then migrate existing article media if needed.")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
"""Server-only Dropbox access. Browser clients only receive media identifiers."""
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
from functools import lru_cache
|
||||||
|
|
||||||
|
import dropbox
|
||||||
|
from dropbox.exceptions import AuthError, DropboxException
|
||||||
|
from fastapi import HTTPException
|
||||||
|
from requests.exceptions import RequestException
|
||||||
|
|
||||||
|
|
||||||
|
@lru_cache(maxsize=1)
|
||||||
|
def get_dropbox():
|
||||||
|
refresh = os.getenv("DROPBOX_REFRESH_TOKEN", "").strip()
|
||||||
|
access = os.getenv("DROPBOX_ACCESS_TOKEN", "").strip()
|
||||||
|
key = os.getenv("DROPBOX_APP_KEY", "").strip()
|
||||||
|
secret = os.getenv("DROPBOX_APP_SECRET", "").strip()
|
||||||
|
if not (refresh and key) and not access:
|
||||||
|
raise HTTPException(503, "Dropbox uploads are not connected. Configure the server's Dropbox app credentials.")
|
||||||
|
return dropbox.Dropbox(
|
||||||
|
oauth2_refresh_token=refresh or None, oauth2_access_token=access or None,
|
||||||
|
app_key=key or None, app_secret=secret or None,
|
||||||
|
timeout=30, max_retries_on_error=0, max_retries_on_rate_limit=0,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def upload(data: bytes, upload_id: str, name: str) -> str:
|
||||||
|
# A UUID prevents clashes. With App Folder access this path is inside the app's folder.
|
||||||
|
safe_name = re.sub(r'[\\/:*?"<>|\x00-\x1f]', "_", name)[:120]
|
||||||
|
try:
|
||||||
|
result = get_dropbox().files_upload(
|
||||||
|
data, f"/article-{upload_id}-{safe_name}",
|
||||||
|
mode=dropbox.files.WriteMode.add, autorename=False, mute=True,
|
||||||
|
)
|
||||||
|
return result.id
|
||||||
|
except AuthError as exc:
|
||||||
|
# Never return 401 here: Dropbox authorization is separate from journal sign-in.
|
||||||
|
raise HTTPException(503, "Dropbox needs to be reconnected by the site owner.") from exc
|
||||||
|
except (DropboxException, RequestException) as exc:
|
||||||
|
raise HTTPException(502, "Dropbox could not store this file. Please retry.") from exc
|
||||||
|
|
||||||
|
|
||||||
|
def download(file_id: str, limit: int) -> bytes:
|
||||||
|
try:
|
||||||
|
metadata, response = get_dropbox().files_download(file_id)
|
||||||
|
try:
|
||||||
|
if metadata.size > limit:
|
||||||
|
raise HTTPException(502, "The stored file exceeds the upload limit.")
|
||||||
|
data = bytearray()
|
||||||
|
for chunk in response.iter_content(64 * 1024):
|
||||||
|
data.extend(chunk)
|
||||||
|
if len(data) > limit:
|
||||||
|
raise HTTPException(502, "The stored file exceeds the upload limit.")
|
||||||
|
return bytes(data)
|
||||||
|
finally:
|
||||||
|
response.close()
|
||||||
|
except AuthError as exc:
|
||||||
|
raise HTTPException(503, "Dropbox needs to be reconnected by the site owner.") from exc
|
||||||
|
except (DropboxException, RequestException) as exc:
|
||||||
|
raise HTTPException(502, "This file is temporarily unavailable from Dropbox. Please retry.") from exc
|
||||||
+61
-11
@@ -12,6 +12,7 @@ import threading
|
|||||||
import time
|
import time
|
||||||
import io
|
import io
|
||||||
from uuid import uuid4
|
from uuid import uuid4
|
||||||
|
from urllib.parse import quote
|
||||||
from datetime import date
|
from datetime import date
|
||||||
from email.message import EmailMessage
|
from email.message import EmailMessage
|
||||||
from functools import lru_cache
|
from functools import lru_cache
|
||||||
@@ -21,11 +22,12 @@ from typing import Any, Literal
|
|||||||
from fastapi import Depends, FastAPI, Header, HTTPException, Query, Request, status
|
from fastapi import Depends, FastAPI, Header, HTTPException, Query, Request, status
|
||||||
from PIL import Image, UnidentifiedImageError
|
from PIL import Image, UnidentifiedImageError
|
||||||
from fastapi.middleware.cors import CORSMiddleware
|
from fastapi.middleware.cors import CORSMiddleware
|
||||||
from fastapi.responses import FileResponse
|
from fastapi.responses import FileResponse, Response
|
||||||
from pydantic import BaseModel, EmailStr, Field
|
from pydantic import BaseModel, EmailStr, Field
|
||||||
from starlette.concurrency import run_in_threadpool
|
from starlette.concurrency import run_in_threadpool
|
||||||
from httpx import TransportError
|
from httpx import TransportError
|
||||||
from supabase import Client, ClientOptions, PostgrestAPIError, create_client
|
from supabase import Client, ClientOptions, PostgrestAPIError, create_client
|
||||||
|
from backend import dropbox_storage
|
||||||
|
|
||||||
|
|
||||||
BASE_DIR = Path(__file__).resolve().parent
|
BASE_DIR = Path(__file__).resolve().parent
|
||||||
@@ -46,6 +48,8 @@ class ArticleMedia(BaseModel):
|
|||||||
name: str = Field(min_length=1, max_length=200)
|
name: str = Field(min_length=1, max_length=200)
|
||||||
media_type: str = Field(max_length=100)
|
media_type: str = Field(max_length=100)
|
||||||
size: int = Field(gt=0, le=MAX_UPLOAD_BYTES)
|
size: int = Field(gt=0, le=MAX_UPLOAD_BYTES)
|
||||||
|
storage: Literal["dropbox"] | None = None
|
||||||
|
dropbox_file_id: str | None = Field(default=None, pattern=r"^id:[A-Za-z0-9_-]+$", max_length=200)
|
||||||
|
|
||||||
|
|
||||||
app = FastAPI(
|
app = FastAPI(
|
||||||
@@ -318,9 +322,44 @@ def uploaded_media(upload_id: str) -> dict[str, Any]:
|
|||||||
if not re.fullmatch(r"[a-f0-9]{32}", upload_id):
|
if not re.fullmatch(r"[a-f0-9]{32}", upload_id):
|
||||||
raise HTTPException(status_code=404, detail="File not found.")
|
raise HTTPException(status_code=404, detail="File not found.")
|
||||||
metadata = UPLOAD_DIR / f"{upload_id}.json"
|
metadata = UPLOAD_DIR / f"{upload_id}.json"
|
||||||
if not metadata.is_file() or not (UPLOAD_DIR / upload_id).is_file():
|
if metadata.is_file():
|
||||||
|
local = json.loads(metadata.read_text(encoding="utf-8"))
|
||||||
|
if local.get("storage") == "dropbox" or (UPLOAD_DIR / upload_id).is_file():
|
||||||
|
return local
|
||||||
|
client = get_supabase()
|
||||||
|
if client is not None:
|
||||||
|
result = read_article_query(client.table("article_media").select("metadata").eq("upload_id", upload_id).limit(1))
|
||||||
|
if result.data:
|
||||||
|
return result.data[0]["metadata"]
|
||||||
raise HTTPException(status_code=404, detail="File not found.")
|
raise HTTPException(status_code=404, detail="File not found.")
|
||||||
return json.loads(metadata.read_text(encoding="utf-8"))
|
|
||||||
|
|
||||||
|
def save_media_metadata(upload_id: str, metadata: dict[str, Any]) -> None:
|
||||||
|
client = get_supabase()
|
||||||
|
if client is not None:
|
||||||
|
try:
|
||||||
|
result = client.table("article_media").upsert({"upload_id": upload_id, "metadata": metadata}).execute()
|
||||||
|
if not result.data:
|
||||||
|
raise RuntimeError("Media record was not returned.")
|
||||||
|
except Exception as exc:
|
||||||
|
raise HTTPException(502, "The file uploaded, but its record could not be saved. Please retry.") from exc
|
||||||
|
else:
|
||||||
|
UPLOAD_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
(UPLOAD_DIR / f"{upload_id}.json").write_text(json.dumps(metadata), encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def store_upload(data: bytes, upload_id: str, metadata: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
storage = os.getenv("JOURNAL_MEDIA_STORAGE", "dropbox").strip().lower()
|
||||||
|
if storage == "dropbox":
|
||||||
|
metadata = {**metadata, "storage": "dropbox", "dropbox_file_id": dropbox_storage.upload(data, upload_id, metadata["name"])}
|
||||||
|
save_media_metadata(upload_id, metadata)
|
||||||
|
elif storage == "local":
|
||||||
|
UPLOAD_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
(UPLOAD_DIR / upload_id).write_bytes(data)
|
||||||
|
(UPLOAD_DIR / f"{upload_id}.json").write_text(json.dumps(metadata), encoding="utf-8")
|
||||||
|
else:
|
||||||
|
raise HTTPException(500, "Invalid journal media storage setting.")
|
||||||
|
return metadata
|
||||||
|
|
||||||
|
|
||||||
@app.post("/api/uploads", status_code=201)
|
@app.post("/api/uploads", status_code=201)
|
||||||
@@ -349,15 +388,20 @@ async def upload_media(
|
|||||||
raise HTTPException(status_code=422, detail="Choose a valid JPEG, PNG, WebP, or GIF banner.")
|
raise HTTPException(status_code=422, detail="Choose a valid JPEG, PNG, WebP, or GIF banner.")
|
||||||
upload_id = uuid4().hex
|
upload_id = uuid4().hex
|
||||||
metadata = {"url": f"/api/uploads/{upload_id}", "name": name.replace("\\", "/").split("/")[-1] or "attachment", "media_type": media_type, "size": len(data)}
|
metadata = {"url": f"/api/uploads/{upload_id}", "name": name.replace("\\", "/").split("/")[-1] or "attachment", "media_type": media_type, "size": len(data)}
|
||||||
UPLOAD_DIR.mkdir(parents=True, exist_ok=True)
|
return await run_in_threadpool(store_upload, bytes(data), upload_id, metadata)
|
||||||
(UPLOAD_DIR / upload_id).write_bytes(data)
|
|
||||||
(UPLOAD_DIR / f"{upload_id}.json").write_text(json.dumps(metadata), encoding="utf-8")
|
|
||||||
return metadata
|
|
||||||
|
|
||||||
|
|
||||||
@app.get("/api/uploads/{upload_id}")
|
@app.get("/api/uploads/{upload_id}", response_model=None)
|
||||||
def download_media(upload_id: str) -> FileResponse:
|
def download_media(upload_id: str):
|
||||||
media = uploaded_media(upload_id)
|
media = uploaded_media(upload_id)
|
||||||
|
if media.get("storage") == "dropbox":
|
||||||
|
data = dropbox_storage.download(media["dropbox_file_id"], MAX_UPLOAD_BYTES)
|
||||||
|
disposition = "inline" if media["media_type"].startswith("image/") else "attachment"
|
||||||
|
return Response(data, media_type=media["media_type"], headers={
|
||||||
|
"Content-Disposition": f"{disposition}; filename*=UTF-8''{quote(media['name'], safe='')}",
|
||||||
|
"X-Content-Type-Options": "nosniff",
|
||||||
|
"Cache-Control": "public, max-age=3600",
|
||||||
|
})
|
||||||
return FileResponse(
|
return FileResponse(
|
||||||
UPLOAD_DIR / upload_id,
|
UPLOAD_DIR / upload_id,
|
||||||
media_type=media["media_type"],
|
media_type=media["media_type"],
|
||||||
@@ -377,9 +421,15 @@ def validated_article(payload: NewPostPayload, slug: str) -> dict[str, Any]:
|
|||||||
article = payload.model_dump(mode="json")
|
article = payload.model_dump(mode="json")
|
||||||
article["slug"] = slug
|
article["slug"] = slug
|
||||||
article["tags"] = clean_tags
|
article["tags"] = clean_tags
|
||||||
for media in [article["banner"], *article["attachments"]]:
|
def canonical_media(media):
|
||||||
if media is not None and uploaded_media(media["url"].rsplit("/", 1)[-1]) != media:
|
if media is None:
|
||||||
|
return None
|
||||||
|
stored = uploaded_media(media["url"].rsplit("/", 1)[-1])
|
||||||
|
if ArticleMedia.model_validate(stored).model_dump() != media:
|
||||||
raise HTTPException(status_code=422, detail="An attachment is invalid. Please upload it again.")
|
raise HTTPException(status_code=422, detail="An attachment is invalid. Please upload it again.")
|
||||||
|
return stored
|
||||||
|
article["banner"] = canonical_media(article["banner"])
|
||||||
|
article["attachments"] = [canonical_media(media) for media in article["attachments"]]
|
||||||
if article["banner"] and not article["banner"]["media_type"].startswith("image/"):
|
if article["banner"] and not article["banner"]["media_type"].startswith("image/"):
|
||||||
raise HTTPException(status_code=422, detail="The banner must be an image.")
|
raise HTTPException(status_code=422, detail="The banner must be an image.")
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,59 @@
|
|||||||
|
"""Migrate referenced local article media; originals are retained as backups."""
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
|
||||||
|
from backend import main, dropbox_storage
|
||||||
|
|
||||||
|
|
||||||
|
def migrate(apply=False):
|
||||||
|
records = main.article_records()
|
||||||
|
media = {item["url"]: item for post in records for item in [post.get("banner"), *(post.get("attachments") or [])] if item}
|
||||||
|
local = [item for item in media.values() if item.get("storage") != "dropbox"]
|
||||||
|
print(f"{len(local)} local media references across {len(records)} articles.")
|
||||||
|
if not apply:
|
||||||
|
print("Preview only. Use --apply to upload files and update article metadata.")
|
||||||
|
return
|
||||||
|
dropbox_storage.get_dropbox() # Fail before modifying anything if not configured.
|
||||||
|
converted = {}
|
||||||
|
for item in local:
|
||||||
|
upload_id = item["url"].rsplit("/", 1)[-1]
|
||||||
|
stored = main.uploaded_media(upload_id)
|
||||||
|
if stored.get("storage") != "dropbox":
|
||||||
|
data = (main.UPLOAD_DIR / upload_id).read_bytes()
|
||||||
|
stored = {**stored, "storage": "dropbox", "dropbox_file_id": dropbox_storage.upload(data, upload_id, stored["name"])}
|
||||||
|
main.save_media_metadata(upload_id, stored)
|
||||||
|
# Keep old URLs working on this server too; retain the original bytes.
|
||||||
|
(main.UPLOAD_DIR / f"{upload_id}.json").write_text(json.dumps(stored), encoding="utf-8")
|
||||||
|
converted[item["url"]] = stored
|
||||||
|
client = main.get_supabase()
|
||||||
|
for post in records:
|
||||||
|
banner = post.get("banner")
|
||||||
|
attachments = post.get("attachments") or []
|
||||||
|
replacement = {
|
||||||
|
"banner": converted.get(banner["url"], banner) if banner else None,
|
||||||
|
"attachments": [converted.get(item["url"], item) for item in attachments],
|
||||||
|
}
|
||||||
|
if replacement == {"banner": banner, "attachments": attachments}:
|
||||||
|
continue
|
||||||
|
if client is not None:
|
||||||
|
result = client.table(main.articles_table()).update(replacement).eq("slug", post["slug"]).execute()
|
||||||
|
if not result.data:
|
||||||
|
raise RuntimeError("An article could not be updated. Retry the migration.")
|
||||||
|
else:
|
||||||
|
with main.POSTS_LOCK:
|
||||||
|
latest = json.loads(main.POSTS_PATH.read_text(encoding="utf-8"))
|
||||||
|
for current in latest:
|
||||||
|
if current["slug"] == post["slug"]:
|
||||||
|
current.update(replacement)
|
||||||
|
main.save_local_posts(latest)
|
||||||
|
print(f"Updated {len(converted)} media references. Original local files were retained.")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--apply", action="store_true")
|
||||||
|
arguments = parser.parse_args()
|
||||||
|
load_dotenv(main.BASE_DIR / ".env")
|
||||||
|
migrate(arguments.apply)
|
||||||
@@ -1,5 +1,6 @@
|
|||||||
fastapi
|
fastapi
|
||||||
httpx
|
httpx
|
||||||
|
dropbox==12.2.1
|
||||||
Pillow
|
Pillow
|
||||||
uvicorn[standard]
|
uvicorn[standard]
|
||||||
email-validator
|
email-validator
|
||||||
|
|||||||
@@ -25,3 +25,14 @@ alter table public.articles enable row level security;
|
|||||||
-- service_role and bypasses RLS; browser roles receive no table privileges.
|
-- service_role and bypasses RLS; browser roles receive no table privileges.
|
||||||
revoke all on table public.articles from anon, authenticated;
|
revoke all on table public.articles from anon, authenticated;
|
||||||
grant select, insert, update, delete on table public.articles to service_role;
|
grant select, insert, update, delete on table public.articles to service_role;
|
||||||
|
|
||||||
|
-- Upload records also exist before an article is published. The article's banner
|
||||||
|
-- and attachments JSON include storage='dropbox' and the Dropbox file ID.
|
||||||
|
create table if not exists public.article_media (
|
||||||
|
upload_id text primary key check (upload_id ~ '^[a-f0-9]{32}$'),
|
||||||
|
metadata jsonb not null,
|
||||||
|
created_at timestamptz not null default now()
|
||||||
|
);
|
||||||
|
alter table public.article_media enable row level security;
|
||||||
|
revoke all on table public.article_media from anon, authenticated;
|
||||||
|
grant select, insert, update, delete on table public.article_media to service_role;
|
||||||
|
|||||||
@@ -10,6 +10,7 @@ from fastapi.testclient import TestClient
|
|||||||
from PIL import Image
|
from PIL import Image
|
||||||
from httpx import ReadTimeout, RemoteProtocolError
|
from httpx import ReadTimeout, RemoteProtocolError
|
||||||
from supabase import PostgrestAPIError
|
from supabase import PostgrestAPIError
|
||||||
|
from dropbox.exceptions import AuthError
|
||||||
|
|
||||||
from backend import main
|
from backend import main
|
||||||
|
|
||||||
@@ -25,7 +26,7 @@ class ArticleMediaTests(unittest.TestCase):
|
|||||||
patch.object(main, "POSTS_PATH", root / "posts.json"),
|
patch.object(main, "POSTS_PATH", root / "posts.json"),
|
||||||
patch.object(main, "UPLOAD_DIR", root / "uploads"),
|
patch.object(main, "UPLOAD_DIR", root / "uploads"),
|
||||||
patch.object(main, "get_supabase", return_value=None),
|
patch.object(main, "get_supabase", return_value=None),
|
||||||
patch.dict(os.environ, {"JOURNAL_ADMIN_PASSWORD": "test-password", "JOURNAL_TOKEN_SECRET": "test-secret"}),
|
patch.dict(os.environ, {"JOURNAL_ADMIN_PASSWORD": "test-password", "JOURNAL_TOKEN_SECRET": "test-secret", "JOURNAL_MEDIA_STORAGE": "local"}),
|
||||||
):
|
):
|
||||||
mocked.start()
|
mocked.start()
|
||||||
self.addCleanup(mocked.stop)
|
self.addCleanup(mocked.stop)
|
||||||
@@ -174,6 +175,84 @@ class ArticleMediaTests(unittest.TestCase):
|
|||||||
self.assertEqual(self.client.get("/api/posts").json(), [])
|
self.assertEqual(self.client.get("/api/posts").json(), [])
|
||||||
self.assertEqual(self.client.get("/api/posts/missing").status_code, 404)
|
self.assertEqual(self.client.get("/api/posts/missing").status_code, 404)
|
||||||
|
|
||||||
|
def test_dropbox_upload_publish_and_download_use_file_id(self):
|
||||||
|
image = io.BytesIO()
|
||||||
|
Image.new("RGB", (20, 20), "blue").save(image, format="PNG")
|
||||||
|
data = image.getvalue()
|
||||||
|
dropbox = MagicMock()
|
||||||
|
dropbox.files_upload.return_value.id = "id:test_photo"
|
||||||
|
stream = MagicMock()
|
||||||
|
stream.iter_content.return_value = [data]
|
||||||
|
dropbox.files_download.return_value = (MagicMock(size=len(data)), stream)
|
||||||
|
with patch.dict(os.environ, {"JOURNAL_MEDIA_STORAGE": "dropbox"}), patch.object(main.dropbox_storage, "get_dropbox", return_value=dropbox):
|
||||||
|
result = self.upload(data, "banner")
|
||||||
|
self.assertEqual(result.status_code, 201, result.text)
|
||||||
|
media = result.json()
|
||||||
|
self.assertEqual(media["dropbox_file_id"], "id:test_photo")
|
||||||
|
self.assertEqual(media["storage"], "dropbox")
|
||||||
|
upload_id = media["url"].rsplit("/", 1)[-1]
|
||||||
|
self.assertFalse((main.UPLOAD_DIR / upload_id).exists())
|
||||||
|
payload = {**self.article_payload(), "banner": media, "attachments": [media]}
|
||||||
|
article = self.client.post("/api/posts", headers=self.headers, json=payload)
|
||||||
|
self.assertEqual(article.status_code, 201, article.text)
|
||||||
|
self.assertEqual(article.json()["banner"]["dropbox_file_id"], "id:test_photo")
|
||||||
|
saved = json.loads(main.POSTS_PATH.read_text())[0]
|
||||||
|
self.assertEqual(saved["attachments"][0]["dropbox_file_id"], "id:test_photo")
|
||||||
|
downloaded = self.client.get(media["url"])
|
||||||
|
self.assertEqual(downloaded.content, data)
|
||||||
|
self.assertEqual(downloaded.headers["content-type"], "image/png")
|
||||||
|
dropbox.files_download.assert_called_once_with("id:test_photo")
|
||||||
|
stream.close.assert_called_once()
|
||||||
|
payload["banner"]["dropbox_file_id"] = "id:someone_else"
|
||||||
|
self.assertEqual(self.client.put("/api/posts/an-editable-article", headers=self.headers, json=payload).status_code, 422)
|
||||||
|
|
||||||
|
def test_dropbox_media_metadata_survives_without_local_files_in_supabase(self):
|
||||||
|
client = MagicMock()
|
||||||
|
media_table = MagicMock()
|
||||||
|
article_table = MagicMock()
|
||||||
|
client.table.side_effect = lambda name: media_table if name == "article_media" else article_table
|
||||||
|
metadata = {"url": "/api/uploads/" + "a" * 32, "name": "image.png", "size": 10,
|
||||||
|
"media_type": "image/png", "storage": "dropbox", "dropbox_file_id": "id:remote_photo"}
|
||||||
|
media_query = media_table.select.return_value.eq.return_value.limit.return_value
|
||||||
|
media_query.retry.return_value.execute.return_value.data = [{"metadata": metadata}]
|
||||||
|
article_table.select.return_value.eq.return_value.limit.return_value.execute.return_value.data = []
|
||||||
|
article_table.insert.side_effect = lambda record: MagicMock(execute=lambda: MagicMock(data=[record]))
|
||||||
|
with patch.object(main, "get_supabase", return_value=client):
|
||||||
|
main.save_media_metadata("a" * 32, metadata)
|
||||||
|
media_table.upsert.assert_called_once_with({"upload_id": "a" * 32, "metadata": metadata})
|
||||||
|
self.assertEqual(main.uploaded_media("a" * 32), metadata)
|
||||||
|
self.assertFalse(main.UPLOAD_DIR.exists())
|
||||||
|
result = self.client.post("/api/posts", headers=self.headers, json={**self.article_payload(), "banner": metadata})
|
||||||
|
self.assertEqual(result.status_code, 201, result.text)
|
||||||
|
self.assertEqual(article_table.insert.call_args.args[0]["banner"]["dropbox_file_id"], "id:remote_photo")
|
||||||
|
|
||||||
|
def test_dropbox_auth_failure_does_not_expire_journal_session(self):
|
||||||
|
dropbox = MagicMock()
|
||||||
|
dropbox.files_upload.side_effect = AuthError("test-request", "expired_access_token")
|
||||||
|
with patch.dict(os.environ, {"JOURNAL_MEDIA_STORAGE": "dropbox"}), patch.object(main.dropbox_storage, "get_dropbox", return_value=dropbox):
|
||||||
|
self.assertEqual(self.upload(b"notes", name="notes.txt").status_code, 503)
|
||||||
|
self.assertEqual(self.client.get("/api/auth/session", headers=self.headers).status_code, 200)
|
||||||
|
self.assertFalse(main.UPLOAD_DIR.exists())
|
||||||
|
|
||||||
|
def test_dropbox_migration_preserves_urls_originals_and_can_resume(self):
|
||||||
|
from backend.migrate_media_dropbox import migrate
|
||||||
|
media = self.upload(b"original notes", name="notes.txt").json()
|
||||||
|
result = self.client.post("/api/posts", headers=self.headers, json={**self.article_payload(), "attachments": [media]})
|
||||||
|
self.assertEqual(result.status_code, 201)
|
||||||
|
dropbox = MagicMock()
|
||||||
|
dropbox.files_upload.return_value.id = "id:migrated_file"
|
||||||
|
with patch.object(main.dropbox_storage, "get_dropbox", return_value=dropbox):
|
||||||
|
migrate(apply=False)
|
||||||
|
dropbox.files_upload.assert_not_called()
|
||||||
|
migrate(apply=True)
|
||||||
|
migrate(apply=True)
|
||||||
|
dropbox.files_upload.assert_called_once()
|
||||||
|
post = self.client.get("/api/posts/an-editable-article").json()
|
||||||
|
migrated = post["attachments"][0]
|
||||||
|
self.assertEqual(migrated["url"], media["url"])
|
||||||
|
self.assertEqual(migrated["dropbox_file_id"], "id:migrated_file")
|
||||||
|
self.assertEqual((main.UPLOAD_DIR / media["url"].rsplit("/", 1)[-1]).read_bytes(), b"original notes")
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main()
|
unittest.main()
|
||||||
|
|||||||
@@ -0,0 +1,89 @@
|
|||||||
|
"""OAuth setup must not persist a grant that cannot support article media."""
|
||||||
|
import io
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
from contextlib import ExitStack, redirect_stdout
|
||||||
|
from pathlib import Path
|
||||||
|
from types import SimpleNamespace
|
||||||
|
from unittest.mock import patch
|
||||||
|
from urllib.parse import parse_qs, urlparse
|
||||||
|
|
||||||
|
from dotenv import dotenv_values
|
||||||
|
from dropbox import DropboxOAuth2FlowNoRedirect
|
||||||
|
|
||||||
|
from backend import connect_dropbox
|
||||||
|
|
||||||
|
|
||||||
|
class DropboxSetupTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.stack = ExitStack()
|
||||||
|
self.addCleanup(self.stack.close)
|
||||||
|
folder = Path(self.stack.enter_context(tempfile.TemporaryDirectory()))
|
||||||
|
self.env = folder / ".env"
|
||||||
|
self.env.write_text(
|
||||||
|
"DROPBOX_APP_KEY=file-key\nDROPBOX_APP_SECRET=file-secret\n"
|
||||||
|
"DROPBOX_REFRESH_TOKEN=existing-token\n", encoding="utf-8"
|
||||||
|
)
|
||||||
|
self.stack.enter_context(patch.object(connect_dropbox, "__file__", str(folder / "connect_dropbox.py")))
|
||||||
|
self.stack.enter_context(patch.dict("os.environ", {
|
||||||
|
"DROPBOX_APP_KEY": "stale-shell-key", "DROPBOX_APP_SECRET": "stale-shell-secret"
|
||||||
|
}))
|
||||||
|
self.stack.enter_context(patch.object(connect_dropbox.getpass, "getpass", return_value="auth-code"))
|
||||||
|
self.browser = self.stack.enter_context(patch.object(connect_dropbox.webbrowser, "open"))
|
||||||
|
self.output = self.stack.enter_context(redirect_stdout(io.StringIO()))
|
||||||
|
# Exercise real SDK URL generation, but never exchange a real code.
|
||||||
|
self.finish = self.stack.enter_context(patch.object(DropboxOAuth2FlowNoRedirect, "finish"))
|
||||||
|
self.finish.return_value = SimpleNamespace(
|
||||||
|
refresh_token="new-token", scope=["files.content.read", "files.content.write"]
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_opens_default_permissions_offline_url_for_env_file_app(self):
|
||||||
|
connect_dropbox.main()
|
||||||
|
url = urlparse(self.browser.call_args.args[0])
|
||||||
|
self.assertEqual(url.hostname, "www.dropbox.com")
|
||||||
|
self.assertEqual(parse_qs(url.query), {
|
||||||
|
"client_id": ["file-key"], "response_type": ["code"], "token_access_type": ["offline"]
|
||||||
|
})
|
||||||
|
settings = dotenv_values(self.env)
|
||||||
|
self.assertEqual(settings["DROPBOX_REFRESH_TOKEN"], "new-token")
|
||||||
|
self.assertEqual(settings["JOURNAL_MEDIA_STORAGE"], "dropbox")
|
||||||
|
for secret in ("new-token", "existing-token", "file-secret", "auth-code"):
|
||||||
|
self.assertNotIn(secret, self.output.getvalue())
|
||||||
|
|
||||||
|
def test_missing_permissions_preserves_existing_credentials(self):
|
||||||
|
original = self.env.read_bytes()
|
||||||
|
self.finish.return_value.scope = ["account_info.read"]
|
||||||
|
with self.assertRaisesRegex(SystemExit, "files.content.read, files.content.write"):
|
||||||
|
connect_dropbox.main()
|
||||||
|
self.assertEqual(self.env.read_bytes(), original)
|
||||||
|
|
||||||
|
def test_unknown_permissions_preserves_existing_credentials(self):
|
||||||
|
original = self.env.read_bytes()
|
||||||
|
self.finish.return_value.scope = None
|
||||||
|
with self.assertRaisesRegex(SystemExit, "did not grant"):
|
||||||
|
connect_dropbox.main()
|
||||||
|
self.assertEqual(self.env.read_bytes(), original)
|
||||||
|
|
||||||
|
def test_string_scopes_are_accepted(self):
|
||||||
|
self.finish.return_value.scope = "files.content.read files.content.write"
|
||||||
|
connect_dropbox.main()
|
||||||
|
self.assertEqual(dotenv_values(self.env)["DROPBOX_REFRESH_TOKEN"], "new-token")
|
||||||
|
|
||||||
|
def test_exchange_failure_does_not_print_secret_or_overwrite_token(self):
|
||||||
|
original = self.env.read_bytes()
|
||||||
|
self.finish.side_effect = RuntimeError("sensitive-response-token")
|
||||||
|
with self.assertRaises(SystemExit) as error:
|
||||||
|
connect_dropbox.main()
|
||||||
|
self.assertNotIn("sensitive-response-token", str(error.exception))
|
||||||
|
self.assertEqual(self.env.read_bytes(), original)
|
||||||
|
|
||||||
|
def test_missing_refresh_token_preserves_existing_credentials(self):
|
||||||
|
original = self.env.read_bytes()
|
||||||
|
self.finish.return_value.refresh_token = None
|
||||||
|
with self.assertRaisesRegex(SystemExit, "did not issue a refresh token"):
|
||||||
|
connect_dropbox.main()
|
||||||
|
self.assertEqual(self.env.read_bytes(), original)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
Reference in New Issue
Block a user