From 6a14ac8ae6c78ebc2ef97d7e51a0f467e0d2cbfe Mon Sep 17 00:00:00 2001 From: Tom Moor Date: Sat, 11 Jul 2026 01:05:08 +0200 Subject: [PATCH] feat: Slab importer (#12859) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Add Slab importer backed by the Markdown import pipeline Slab exports a zip of Markdown documents that is structurally identical to the Outline Markdown export, with one difference: images are referenced as remote signed URLs rather than files inside the archive. Rather than build a parallel importer, this reuses the Markdown task and processor and teaches the shared import pipeline to download remote images: - Enable per-page attachment upload for the Markdown task. The base APIImportTask already downloads remote image/video/attachment URLs and rewrites them to internal redirect URLs; the Markdown task previously opted out because its attachments live in the zip. - Make the base upload step skip URLs that are already internal, so local zip attachments (resolved to redirect URLs during rewriteMarkdown and uploaded from the archive in onAllTasksCompleted) pass through untouched while remote URLs are fetched and re-hosted. Also guards the URL rewrite against nodes not present in the download map. - Introduce IntegrationService.Slab as a distinct, importable service that routes through the Markdown task/processor, so imports are tracked and labelled as "Slab" with no duplicated import logic. - Wire the create API (schema + route), the import settings UI (Slab card and dialog), and a placeholder logo asset. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01AC5Ho2SHBjNpxEAMr5uL6o * Add shell SlabAPIImportTask as a Slab-specific seam Slab imports previously ran through MarkdownAPIImportTask directly. Introduce a thin SlabAPIImportTask subclass so Slab gets its own task name (for scheduling, tracing, and retries) and a dedicated place to override behavior as Slab exports diverge from the generic Markdown shape. The subclass inherits all conversion/attachment/persistence logic and only overrides scheduleNextTask to keep the whole import chain on the Slab class. MarkdownImportsProcessor now selects the task implementation by the import's service when scheduling the first task. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01AC5Ho2SHBjNpxEAMr5uL6o * Unwrap single root "slab" directory in Slab imports Slab exports wrap the entire workspace in a single top-level directory named "slab". Without handling this, the import produces one "slab" collection containing everything, instead of mapping each workspace area to its own collection. Add a `resolveCollectionRootNodes` seam on the Markdown bootstrap phase (default: pass entries through unchanged) and override it in SlabAPIImportTask to descend into a lone, case-insensitive "slab" root directory so its child directories become collections. Paths are left intact, so the attachment manifest, completion re-walk, and internal-link resolution stay consistent. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01AC5Ho2SHBjNpxEAMr5uL6o * Use the filename, not the first heading, as the title for Slab imports In Slab a document's leading heading is real content, not its title — the filename is authoritative. Add an `extractTitle` option to DocumentConverter.convert (default true, preserving existing callers) that, when false, skips lifting a leading H1 into the title and leaves it in the body. Expose it through a `shouldExtractTitleFromHeading` seam on the Markdown task and override it in SlabAPIImportTask. With no extracted title, the page falls back to the filename-derived title. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01AC5Ho2SHBjNpxEAMr5uL6o * Update logo * Narrow MarkdownAPIImportTask generic back to Markdown Slab imports now run through SlabAPIImportTask, so the base Markdown task no longer handles the Slab service directly and its generic can return to Markdown-only. SlabAPIImportTask keeps a Markdown | Slab union on its scheduleNextTask override, which is required so the override stays a valid (contravariant) override of the base's Markdown-typed parameter. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01AC5Ho2SHBjNpxEAMr5uL6o * Optimised images with calibre/image-actions * reorder * Move Slab importer into its own plugin Restructure the Slab importer to live in plugins/slab, mirroring the Notion plugin, instead of being threaded through the core Markdown importer. - plugins/slab/server registers SlabImportsProcessor (Hook.Processor) and SlabAPIImportTask (Hook.Task). The processor extends MarkdownImportsProcessor to claim the Slab service and schedule the Slab task; the task keeps its Slab-specific overrides (remote image download, root "slab" dir unwrap, filename-as-title). - plugins/slab/client registers the Slab import card (Hook.Imports) and owns the import dialog. - Revert MarkdownImportsProcessor to Markdown-only and drop the hardcoded Slab card from the Import settings screen. The Slab service enum, import schema, and create route remain in core, as the Notion plugin's equivalents do. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01AC5Ho2SHBjNpxEAMr5uL6o * Generalize DropToImport to accept an importable service DropToImport previously branched on `format` to map MarkdownZip/JSON/Slab to specific imports.create calls, with a dead collections.import fallback. Replace this with a single `service` prop and one generic imports.create call. Core no longer hardcodes any service name (notably the Slab plugin service), and the Markdown, JSON, and Slab dialogs each pass their service. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01AC5Ho2SHBjNpxEAMr5uL6o --------- Co-authored-by: Claude Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> --- .../Settings/components/DropToImport.tsx | 36 +++----- .../Settings/components/ImportJSONDialog.tsx | 4 +- .../components/ImportMarkdownDialog.tsx | 4 +- plugins/slab/client/Imports.tsx | 24 ++++++ .../client/components/ImportSlabDialog.tsx | 30 +++++++ plugins/slab/client/index.tsx | 19 +++++ plugins/slab/plugin.json | 5 ++ plugins/slab/server/index.ts | 17 ++++ .../server/processors/SlabImportsProcessor.ts | 24 ++++++ .../server/tasks/SlabAPIImportTask.test.ts | 80 ++++++++++++++++++ .../slab/server/tasks/SlabAPIImportTask.ts | 55 ++++++++++++ public/images/slab.png | Bin 0 -> 1195 bytes server/queues/tasks/APIImportTask.ts | 23 ++++- .../tasks/MarkdownAPIImportTask.test.ts | 12 +++ server/queues/tasks/MarkdownAPIImportTask.ts | 43 +++++++++- server/routes/api/imports/imports.ts | 1 + server/routes/api/imports/schema.ts | 5 ++ server/utils/DocumentConverter.test.ts | 14 +++ server/utils/DocumentConverter.ts | 19 +++-- shared/i18n/locales/en_US/translation.json | 3 + shared/schema.ts | 17 ++-- shared/types.ts | 3 + 22 files changed, 390 insertions(+), 48 deletions(-) create mode 100644 plugins/slab/client/Imports.tsx create mode 100644 plugins/slab/client/components/ImportSlabDialog.tsx create mode 100644 plugins/slab/client/index.tsx create mode 100644 plugins/slab/plugin.json create mode 100644 plugins/slab/server/index.ts create mode 100644 plugins/slab/server/processors/SlabImportsProcessor.ts create mode 100644 plugins/slab/server/tasks/SlabAPIImportTask.test.ts create mode 100644 plugins/slab/server/tasks/SlabAPIImportTask.ts create mode 100644 public/images/slab.png diff --git a/app/scenes/Settings/components/DropToImport.tsx b/app/scenes/Settings/components/DropToImport.tsx index 7d943f8c74..ab048e5697 100644 --- a/app/scenes/Settings/components/DropToImport.tsx +++ b/app/scenes/Settings/components/DropToImport.tsx @@ -10,8 +10,7 @@ import { s } from "@shared/styles"; import { AttachmentPreset, CollectionPermission, - FileOperationFormat, - IntegrationService, + type ImportableIntegrationService, } from "@shared/types"; import { bytesToHumanReadable } from "@shared/utils/files"; import Button from "~/components/Button"; @@ -25,15 +24,16 @@ import { uploadFile } from "~/utils/files"; type Props = { children: JSX.Element; - format?: string; + /** The importable service to create an import for. */ + service: ImportableIntegrationService; disabled?: boolean; activeClassName?: string; onSubmit: () => void; }; -function DropToImport({ disabled, onSubmit, children, format }: Props) { +function DropToImport({ disabled, onSubmit, children, service }: Props) { const { t } = useTranslation(); - const { collections, imports } = useStores(); + const { imports } = useStores(); const [file, setFile] = useState(null); const [isImporting, setImporting] = useState(false); const [uploadProgress, setUploadProgress] = useState(0); @@ -63,25 +63,13 @@ function DropToImport({ disabled, onSubmit, children, format }: Props) { onProgress: (progress) => setUploadProgress(progress), }); - if (format === FileOperationFormat.MarkdownZip) { - await imports.create( - { service: IntegrationService.Markdown }, - { - attachmentId: attachment.id, - permission: permission ?? undefined, - } - ); - } else if (format === FileOperationFormat.JSON) { - await imports.create( - { service: IntegrationService.JSON }, - { - attachmentId: attachment.id, - permission: permission ?? undefined, - } - ); - } else { - await collections.import(attachment.id, { format, permission }); - } + await imports.create( + { service }, + { + attachmentId: attachment.id, + permission: permission ?? undefined, + } + ); onSubmit(); toast.message(file.name, { diff --git a/app/scenes/Settings/components/ImportJSONDialog.tsx b/app/scenes/Settings/components/ImportJSONDialog.tsx index d83d055174..e27b0219be 100644 --- a/app/scenes/Settings/components/ImportJSONDialog.tsx +++ b/app/scenes/Settings/components/ImportJSONDialog.tsx @@ -1,5 +1,5 @@ import { Trans } from "react-i18next"; -import { FileOperationFormat } from "@shared/types"; +import { IntegrationService } from "@shared/types"; import env from "~/env"; import useStores from "~/hooks/useStores"; import DropToImport from "./DropToImport"; @@ -22,7 +22,7 @@ function ImportJSONDialog() { Drag and drop the zip file from the JSON export option in{" "} diff --git a/app/scenes/Settings/components/ImportMarkdownDialog.tsx b/app/scenes/Settings/components/ImportMarkdownDialog.tsx index 081fb47105..8e78e62e91 100644 --- a/app/scenes/Settings/components/ImportMarkdownDialog.tsx +++ b/app/scenes/Settings/components/ImportMarkdownDialog.tsx @@ -1,5 +1,5 @@ import { Trans } from "react-i18next"; -import { FileOperationFormat } from "@shared/types"; +import { IntegrationService } from "@shared/types"; import env from "~/env"; import useStores from "~/hooks/useStores"; import DropToImport from "./DropToImport"; @@ -21,7 +21,7 @@ function ImportMarkdownDialog() { Drag and drop the zip file from the Markdown export option in{" "} diff --git a/plugins/slab/client/Imports.tsx b/plugins/slab/client/Imports.tsx new file mode 100644 index 0000000000..e1915b35fd --- /dev/null +++ b/plugins/slab/client/Imports.tsx @@ -0,0 +1,24 @@ +import { observer } from "mobx-react"; +import { useCallback } from "react"; +import { useTranslation } from "react-i18next"; +import Button from "~/components/Button"; +import useStores from "~/hooks/useStores"; +import { ImportSlabDialog } from "./components/ImportSlabDialog"; + +export const Slab = observer(() => { + const { t } = useTranslation(); + const { dialogs } = useStores(); + + const handleClick = useCallback(() => { + dialogs.openModal({ + title: t("Import data"), + content: , + }); + }, [t, dialogs]); + + return ( + + ); +}); diff --git a/plugins/slab/client/components/ImportSlabDialog.tsx b/plugins/slab/client/components/ImportSlabDialog.tsx new file mode 100644 index 0000000000..a613ff5c60 --- /dev/null +++ b/plugins/slab/client/components/ImportSlabDialog.tsx @@ -0,0 +1,30 @@ +import { Trans } from "react-i18next"; +import { IntegrationService } from "@shared/types"; +import Text from "@shared/components/Text"; +import useStores from "~/hooks/useStores"; +import DropToImport from "~/scenes/Settings/components/DropToImport"; + +export function ImportSlabDialog() { + const { dialogs } = useStores(); + + return ( + <> + + + Import a zip file of Markdown documents exported from Slab – + collections, posts, and images will be imported. In Slab, open the + admin settings and use the Export option to download an + archive of your content. + + + + + Drag and drop the zip file exported from Slab, or click to upload + + + + ); +} diff --git a/plugins/slab/client/index.tsx b/plugins/slab/client/index.tsx new file mode 100644 index 0000000000..7f85355365 --- /dev/null +++ b/plugins/slab/client/index.tsx @@ -0,0 +1,19 @@ +import { t } from "i18next"; +import * as React from "react"; +import { cdnPath } from "@shared/utils/urls"; +import { Hook, PluginManager } from "~/utils/PluginManager"; +import config from "../plugin.json"; +import { Slab } from "./Imports"; + +PluginManager.add([ + { + ...config, + type: Hook.Imports, + value: { + title: "Slab", + subtitle: t("Import a zip file of Markdown documents exported from Slab"), + icon: , + action: , + }, + }, +]); diff --git a/plugins/slab/plugin.json b/plugins/slab/plugin.json new file mode 100644 index 0000000000..2785fac078 --- /dev/null +++ b/plugins/slab/plugin.json @@ -0,0 +1,5 @@ +{ + "id": "slab", + "name": "Slab", + "description": "Adds a Slab integration for importing data." +} diff --git a/plugins/slab/server/index.ts b/plugins/slab/server/index.ts new file mode 100644 index 0000000000..9dd94b8c42 --- /dev/null +++ b/plugins/slab/server/index.ts @@ -0,0 +1,17 @@ +import { Hook, PluginManager } from "@server/utils/PluginManager"; +import config from "../plugin.json"; +import { SlabImportsProcessor } from "./processors/SlabImportsProcessor"; +import SlabAPIImportTask from "./tasks/SlabAPIImportTask"; + +PluginManager.add([ + { + ...config, + type: Hook.Processor, + value: SlabImportsProcessor, + }, + { + ...config, + type: Hook.Task, + value: SlabAPIImportTask, + }, +]); diff --git a/plugins/slab/server/processors/SlabImportsProcessor.ts b/plugins/slab/server/processors/SlabImportsProcessor.ts new file mode 100644 index 0000000000..d182b82db6 --- /dev/null +++ b/plugins/slab/server/processors/SlabImportsProcessor.ts @@ -0,0 +1,24 @@ +import { IntegrationService } from "@shared/types"; +import type { Import, ImportTask } from "@server/models"; +import MarkdownImportsProcessor from "@server/queues/processors/MarkdownImportsProcessor"; +import SlabAPIImportTask from "../tasks/SlabAPIImportTask"; + +// Supertype of the base processor's generic so the overrides below remain +// valid (contravariant) overrides of MarkdownImportsProcessor's Markdown-typed +// methods while still being able to inspect the Slab service value. +type Service = IntegrationService.Markdown | IntegrationService.Slab; + +/** + * Processes Slab imports. Slab shares the Markdown zip pipeline, so collection + * and document persistence is inherited from {@link MarkdownImportsProcessor}; + * this subclass only claims the Slab service and schedules the Slab task. + */ +export class SlabImportsProcessor extends MarkdownImportsProcessor { + protected canProcess(importModel: Import): boolean { + return importModel.service === IntegrationService.Slab; + } + + protected async scheduleTask(importTask: ImportTask): Promise { + await new SlabAPIImportTask().schedule({ importTaskId: importTask.id }); + } +} diff --git a/plugins/slab/server/tasks/SlabAPIImportTask.test.ts b/plugins/slab/server/tasks/SlabAPIImportTask.test.ts new file mode 100644 index 0000000000..24ad9f943c --- /dev/null +++ b/plugins/slab/server/tasks/SlabAPIImportTask.test.ts @@ -0,0 +1,80 @@ +import type { ZipTreeNode } from "@server/utils/ZipHelper"; +import SlabAPIImportTask from "./SlabAPIImportTask"; + +class TestSlabAPIImportTask extends SlabAPIImportTask { + public resolve(nodes: ZipTreeNode[]): ZipTreeNode[] { + return this.resolveCollectionRootNodes(nodes); + } + + public extractsTitleFromHeading(): boolean { + return this.shouldExtractTitleFromHeading(); + } +} + +const dir = ( + title: string, + children: ZipTreeNode[] = [], + pathInZip = title +): ZipTreeNode => ({ + name: title, + title, + pathInZip, + children, +}); + +const file = (name: string, pathInZip = name): ZipTreeNode => ({ + name, + title: name.replace(/\.[^.]+$/, ""), + pathInZip, + children: [], +}); + +describe("SlabAPIImportTask#resolveCollectionRootNodes", () => { + const task = new TestSlabAPIImportTask(); + + it("unwraps a single root 'slab' directory into its children", () => { + const eng = dir( + "Engineering", + [file("doc.md", "slab/Engineering/doc.md")], + "slab/Engineering" + ); + const product = dir( + "Product", + [file("spec.md", "slab/Product/spec.md")], + "slab/Product" + ); + const root = dir("slab", [eng, product], "slab"); + + expect(task.resolve([root])).toEqual([eng, product]); + }); + + it("is case-insensitive on the wrapper directory name", () => { + const child = dir("Team", [file("a.md", "Slab/Team/a.md")], "Slab/Team"); + const root = dir("Slab", [child], "Slab"); + + expect(task.resolve([root])).toEqual([child]); + }); + + it("leaves multiple root directories untouched", () => { + const a = dir("Engineering", [file("a.md", "Engineering/a.md")]); + const b = dir("slab", [file("b.md", "slab/b.md")]); + + expect(task.resolve([a, b])).toEqual([a, b]); + }); + + it("leaves a single non-'slab' root directory untouched", () => { + const a = dir("Engineering", [file("doc.md", "Engineering/doc.md")]); + + expect(task.resolve([a])).toEqual([a]); + }); + + it("does not unwrap an empty 'slab' directory", () => { + const root = dir("slab", []); + + expect(task.resolve([root])).toEqual([root]); + }); + + it("does not derive the title from a document's leading heading", () => { + expect(task.extractsTitleFromHeading()).toBe(false); + }); +}); diff --git a/plugins/slab/server/tasks/SlabAPIImportTask.ts b/plugins/slab/server/tasks/SlabAPIImportTask.ts new file mode 100644 index 0000000000..abb8c66fad --- /dev/null +++ b/plugins/slab/server/tasks/SlabAPIImportTask.ts @@ -0,0 +1,55 @@ +import type { IntegrationService } from "@shared/types"; +import type { ImportTask } from "@server/models"; +import MarkdownAPIImportTask from "@server/queues/tasks/MarkdownAPIImportTask"; +import type { ZipTreeNode } from "@server/utils/ZipHelper"; + +// Supertype of the base task's generic so the `scheduleNextTask` override +// remains a valid override of MarkdownAPIImportTask's Markdown-typed method. +type Service = IntegrationService.Markdown | IntegrationService.Slab; + +/** + * Imports a Slab workspace export. + * + * Slab exports the same zip-of-Markdown structure as Outline's own Markdown + * export, so all conversion, attachment, and persistence logic is inherited + * from {@link MarkdownAPIImportTask}. This subclass exists as a dedicated + * seam for Slab-specific behavior (e.g. quirks in how Slab references remote + * assets) and so Slab imports are scheduled, traced, and retried under their + * own task name rather than the generic Markdown one. + */ +export default class SlabAPIImportTask extends MarkdownAPIImportTask { + protected async scheduleNextTask( + importTask: ImportTask + ): Promise { + await new SlabAPIImportTask().schedule({ importTaskId: importTask.id }); + } + + /** + * Slab exports wrap the entire workspace in a single root directory named + * "slab". When present, descend into it so the directories one level down + * are imported as collections rather than a single "slab" collection. + * + * @param nodes The archive's top-level tree nodes. + * @returns The nodes to import as collections. + */ + protected resolveCollectionRootNodes(nodes: ZipTreeNode[]): ZipTreeNode[] { + if ( + nodes.length === 1 && + nodes[0].children.length > 0 && + nodes[0].title.toLowerCase() === "slab" + ) { + return nodes[0].children; + } + return nodes; + } + + /** + * Slab does not treat a document's first heading as its title — the + * filename is authoritative. Keep the leading heading as body content. + * + * @returns false so the leading heading is preserved in the document body. + */ + protected shouldExtractTitleFromHeading(): boolean { + return false; + } +} diff --git a/public/images/slab.png b/public/images/slab.png new file mode 100644 index 0000000000000000000000000000000000000000..8fb3f472d3ae38bd800e53b96b5ebd5311044aee GIT binary patch literal 1195 zcmeAS@N?(olHy`uVBq!ia0vp^KR}p+8AyKPzikPmq5^zETzxVN4+dDj%`hzy@%Y!M z^WV|=?-tQgQICM*_hXKn4>)!QC=Mh8j@<=H1snx&|7;Qc4+NcEfA<;vKY!u>_8s@O zTmH|>U!vl6!^CC5-iCi~j{a}&_`Z%c2WSglNswPKgQRnI(~av76kf%yn6$sfL#PHQ z!&%@FSLQDO z>eNXq-(PGu*}44ig*wr#@-@@C7w$g3Yp+pR-`kZ-sxtDs-X4qHbXryA_o8R!YB&2{ zc&At|SpL)KX7S6WH+dQ9$G@H9oImZf?{`fzw#&6v*RN08RqeOjJ>c1Mzx^)-x)+{# zukqbR?R&(#?Ah;DKfSo~c!YFmymJ5Lzgum>zK4IRxI5o}hfdu0ih0_1uSdN%+UwB2Opleeq|D5!<_aEw9;+lG)y3RkB?F9A`^weCa>G<%asNkWZHb@rbZO_(x3#B&qdDs(^LK7+nSXrt+Kb^*{dgH?6-RuO}JDo~Zr9W@>o3en+?ct*Afolhd0%ICAZqoA*(nV5OLSU8xT1 z!?h7TAN=_KX|Tup&wDiSu=%17j(q#(#>6*Nw_Gp!aQDeQF{XQ2Eywq4kmE0^E)RPB zfzP7g!GT6*c0QROhaNKKr9;>t!MO+Ot$4~x9~|&Kbb4z-$06a9416*m3FS8S9dj9h zIv`rhnfO0c&Wx3{_^{)lEX**tsTLnrJe>5=*PsDp1ws_0&Yf2uD4Hj@@BdT1d19>P zKf>f-3T)1t`t?9(-nTcepzirl@xWJAe389I$o;2G79SesPd7c(9ODgkI8f8(M+|qa zKY!Z(chQ53?_qih?9A>7vKBnxTDNrn-J@~Jd&KwjKlfz5BVywn_N%Y(f!ssSNiVI> zG9Bnv@w~dVf}_H2-|-bXhc*k|Td{i)OT%*Inx*-PU;kC@tf;G+lDYZ|!$f)c*wAl! zk8cXzlh`v$HaT_6rO8`1g}r?FGBq`M^QF{hRr+?fTAuA{Yw_+em*o`SvBE>mW^aYX zy&Kl2Q-anlK74q2jAwg$e9{*cn>PveFFY%5Ov#>^voLtwQw?t0`&GWJavN?`$F|C? zyJ714Wx~Ky!Bmo@2%%PhVL5d z*UY(p&);%Nq4Jijj8EWf0Bv$*Pa?fP)>0duI2d6Q6&dsb{qA?h>dl+`0;w`x!i4{an^LB{Ts5IayL# literal 0 HcmV?d00001 diff --git a/server/queues/tasks/APIImportTask.ts b/server/queues/tasks/APIImportTask.ts index e2037b7793..16b03a6342 100644 --- a/server/queues/tasks/APIImportTask.ts +++ b/server/queues/tasks/APIImportTask.ts @@ -21,6 +21,7 @@ import { ImportTaskState, } from "@shared/types"; import { toError } from "@shared/utils/error"; +import { isExternalUrl } from "@shared/utils/urls"; import { createContext } from "@server/context"; import { schema } from "@server/editor"; import Logger from "@server/logging/Logger"; @@ -365,7 +366,14 @@ export default abstract class APIImportTask< return { url, name: name.length !== 0 ? name : node.type.name }; }), "url" - ); + ).filter((item) => isExternalUrl(item.url)); + + // Nothing remote to download — content already points at internal + // attachments (e.g. a Markdown zip's local files resolved to redirect + // URLs), so leave the doc untouched. + if (!attachmentsData.length) { + return doc; + } await sequelize.transaction(async (transaction) => { const dbPromises = attachmentsData.map(async (item) => { @@ -436,14 +444,23 @@ export default abstract class APIImportTask< const attrs = json.attrs ?? {}; if (node.type.name === "attachment") { - const attachmentModel = urlToAttachment[attrs.href as string]; // attachment node uses 'href' attribute. + const attachmentModel = urlToAttachment[attrs.href as string]; + // Nodes already pointing at internal attachments aren't in the map; + // leave them untouched. + if (!attachmentModel) { + return node; + } attrs.href = attachmentModel.redirectUrl; // attachment node can have id. attrs.id = attachmentModel.id; } else if (node.type.name === "image" || node.type.name === "video") { // image & video nodes use 'src' attribute. - attrs.src = urlToAttachment[attrs.src as string].redirectUrl; + const attachmentModel = urlToAttachment[attrs.src as string]; + if (!attachmentModel) { + return node; + } + attrs.src = attachmentModel.redirectUrl; } json.attrs = attrs; diff --git a/server/queues/tasks/MarkdownAPIImportTask.test.ts b/server/queues/tasks/MarkdownAPIImportTask.test.ts index 5767f340b3..ad832dc50b 100644 --- a/server/queues/tasks/MarkdownAPIImportTask.test.ts +++ b/server/queues/tasks/MarkdownAPIImportTask.test.ts @@ -66,6 +66,18 @@ describe("rewriteAttachmentPaths", () => { ]); expect(out).toBe("![x](https://example.com/a.png)"); }); + + it("leaves remote signed URLs untouched so the base task can download them", () => { + // Slab exports reference images as remote signed URLs rather than files + // in the zip; these aren't in the manifest and must survive rewriting so + // the per-page attachment upload step can fetch and re-host them. + const signedUrl = + "https://uploads.slab.com/posts/abc/image.png?Signature=xyz&Expires=123"; + const out = rewriteAttachmentPaths(`![x](${signedUrl})`, [ + { id: "id-a", pathInZip: "C/attachments/local.png" }, + ]); + expect(out).toBe(`![x](${signedUrl})`); + }); }); describe("rewriteInternalLinks", () => { diff --git a/server/queues/tasks/MarkdownAPIImportTask.ts b/server/queues/tasks/MarkdownAPIImportTask.ts index 0f125ed97a..1350bc7140 100644 --- a/server/queues/tasks/MarkdownAPIImportTask.ts +++ b/server/queues/tasks/MarkdownAPIImportTask.ts @@ -153,7 +153,26 @@ export function rewriteInternalLinks( export default class MarkdownAPIImportTask extends APIImportTask { protected shouldUploadAttachmentsPerPage(): boolean { - return false; + // Per-page upload downloads remote image/attachment URLs referenced in + // the markdown (e.g. Slab's signed export URLs) and rewrites them to + // internal redirect URLs. The base step skips URLs that are already + // internal, so local zip attachments — rewritten to redirect URLs during + // `rewriteMarkdown` and uploaded from the archive in + // `onAllTasksCompleted` — pass through untouched. + return true; + } + + /** + * Whether a document's leading H1 heading should be lifted out as its title + * (and removed from the body). Outline's own Markdown export writes the + * title as a leading H1, so this defaults to true. Sources where the + * filename is authoritative and the first heading is real content (e.g. + * Slab) override this to keep the heading in the body. + * + * @returns true to derive the title from a leading H1 heading. + */ + protected shouldExtractTitleFromHeading(): boolean { + return true; } protected async scheduleNextTask(importTask: ImportTask) { @@ -262,14 +281,16 @@ export default class MarkdownAPIImportTask extends APIImportTask { } ); - if (tree.children.length === 0) { + const rootNodes = this.resolveCollectionRootNodes(tree.children); + + if (rootNodes.length === 0) { throw new Error("Could not find valid content in zip file"); } const collections: DiscoveredCollection[] = []; const manifest: MarkdownAttachmentManifestItem[] = []; - for (const node of tree.children) { + for (const node of rootNodes) { if (node.children.length === 0) { Logger.debug("task", `Unhandled file in zip: ${node.pathInZip}`, { importTaskId: importTask.id, @@ -420,7 +441,8 @@ export default class MarkdownAPIImportTask extends APIImportTask { const { doc, title, icon } = await DocumentConverter.convert( transformedMarkdown, path.basename(item.path), - "text/markdown" + "text/markdown", + { extractTitle: this.shouldExtractTitleFromHeading() } ); taskOutput.push({ @@ -499,6 +521,19 @@ export default class MarkdownAPIImportTask extends APIImportTask { }); } + /** + * Resolves the archive's top-level entries into the nodes that should be + * treated as collections. The base implementation uses the entries as-is; + * subclasses can override to unwrap a known wrapper directory before the + * bootstrap phase maps each node to a collection. + * + * @param nodes The archive's top-level tree nodes. + * @returns The nodes to import as collections. + */ + protected resolveCollectionRootNodes(nodes: ZipTreeNode[]): ZipTreeNode[] { + return nodes; + } + /** * Detects folders containing only attachments (no markdown documents). * Recursively considers nested folders; mirrors the legacy heuristic. diff --git a/server/routes/api/imports/imports.ts b/server/routes/api/imports/imports.ts index 3e0c98c378..a28025904b 100644 --- a/server/routes/api/imports/imports.ts +++ b/server/routes/api/imports/imports.ts @@ -34,6 +34,7 @@ router.post( if ( body.service === IntegrationService.Markdown || + body.service === IntegrationService.Slab || body.service === IntegrationService.JSON ) { const attachment = await Attachment.findByPk(body.attachmentId, { diff --git a/server/routes/api/imports/schema.ts b/server/routes/api/imports/schema.ts index 70c99f7d4c..86e2204277 100644 --- a/server/routes/api/imports/schema.ts +++ b/server/routes/api/imports/schema.ts @@ -43,6 +43,11 @@ export const ImportsCreateSchema = BaseSchema.extend({ attachmentId: z.uuid(), permission: z.enum(CollectionPermission).optional(), }), + z.object({ + service: z.literal(IntegrationService.Slab), + attachmentId: z.uuid(), + permission: z.enum(CollectionPermission).optional(), + }), z.object({ service: z.literal(IntegrationService.JSON), attachmentId: z.uuid(), diff --git a/server/utils/DocumentConverter.test.ts b/server/utils/DocumentConverter.test.ts index ed66816797..bd65f3ab90 100644 --- a/server/utils/DocumentConverter.test.ts +++ b/server/utils/DocumentConverter.test.ts @@ -149,6 +149,20 @@ Jane,24,`; expect(result.text).toContain("Subtitle"); }); + it("should keep the leading H1 in the body when extractTitle is false", async () => { + const md = "# My Title\n\nContent here"; + const result = await DocumentConverter.convert( + md, + "test.md", + "text/markdown", + { extractTitle: false } + ); + + expect(result.title).toEqual(""); + expect(result.text).toContain("My Title"); + expect(result.text).toContain("Content here"); + }); + it("should convert frontmatter to yaml codeblock", async () => { const md = `--- title: Test Document diff --git a/server/utils/DocumentConverter.ts b/server/utils/DocumentConverter.ts index e6fd3cd75a..575e8762ca 100644 --- a/server/utils/DocumentConverter.ts +++ b/server/utils/DocumentConverter.ts @@ -29,13 +29,20 @@ export class DocumentConverter { * @param content The content of the file. * @param fileName The name of the file, including extension. * @param mimeType The mime type of the file. + * @param options Conversion options. + * @param options.extractTitle Whether a leading H1 heading should be lifted + * out as the document title and removed from the body. Defaults to true; + * set false for sources where the filename is authoritative and the first + * heading must remain part of the content (e.g. Slab). * @returns The converted document with text, data, title, and icon. */ public static async convert( content: Buffer | string, fileName: string, - mimeType: string + mimeType: string, + options: { extractTitle?: boolean } = {} ): Promise { + const { extractTitle = true } = options; let doc: Node; // Route to appropriate conversion method @@ -53,10 +60,12 @@ export class DocumentConverter { // Extract title from first H1 heading let title = ""; - const headings = ProsemirrorHelper.getHeadings(doc); - if (headings.length > 0 && headings[0].level === 1) { - title = headings[0].title; - doc = ProsemirrorHelper.removeFirstHeading(doc); + if (extractTitle) { + const headings = ProsemirrorHelper.getHeadings(doc); + if (headings.length > 0 && headings[0].level === 1) { + title = headings[0].title; + doc = ProsemirrorHelper.removeFirstHeading(doc); + } } // Extract emoji from start of document diff --git a/shared/i18n/locales/en_US/translation.json b/shared/i18n/locales/en_US/translation.json index 1ade8b3c25..6e233f840c 100644 --- a/shared/i18n/locales/en_US/translation.json +++ b/shared/i18n/locales/en_US/translation.json @@ -1266,6 +1266,8 @@ "{{ count }} document imported_other": "{{ count }} document imported", "You can import a zip file that was previously exported from an Outline installation – collections, documents, and images will be imported. In Outline, open Export in the Settings sidebar and click on Export Data.": "You can import a zip file that was previously exported from an Outline installation – collections, documents, and images will be imported. In Outline, open Export in the Settings sidebar and click on Export Data.", "Drag and drop the zip file from the Markdown export option in {{appName}}, or click to upload": "Drag and drop the zip file from the Markdown export option in {{appName}}, or click to upload", + "Import a zip file of Markdown documents exported from Slab – collections, posts, and images will be imported. In Slab, open the admin settings and use the <1>Export option to download an archive of your content.": "Import a zip file of Markdown documents exported from Slab – collections, posts, and images will be imported. In Slab, open the admin settings and use the <1>Export option to download an archive of your content.", + "Drag and drop the zip file exported from Slab, or click to upload": "Drag and drop the zip file exported from Slab, or click to upload", "Configure": "Configure", "Role": "Role", "Guest": "Guest", @@ -1346,6 +1348,7 @@ "Import a JSON data file exported from another {{ appName }} instance": "Import a JSON data file exported from another {{ appName }} instance", "Import pages from a Confluence instance": "Import pages from a Confluence instance", "Enterprise": "Enterprise", + "Import a zip file of Markdown documents exported from Slab": "Import a zip file of Markdown documents exported from Slab", "Quickly transfer your existing documents, pages, and files from other tools and services into {{appName}}. You can also drag and drop any HTML, Markdown, and text documents directly into Collections in the app.": "Quickly transfer your existing documents, pages, and files from other tools and services into {{appName}}. You can also drag and drop any HTML, Markdown, and text documents directly into Collections in the app.", "Recent imports": "Recent imports", "Configure a variety of integrations with third-party services.": "Configure a variety of integrations with third-party services.", diff --git a/shared/schema.ts b/shared/schema.ts index b55f8d2e99..82124e685d 100644 --- a/shared/schema.ts +++ b/shared/schema.ts @@ -41,7 +41,7 @@ export type JSONImportInput = z.infer[]; export type ImportInput = T extends IntegrationService.Notion ? NotionImportInput - : T extends IntegrationService.Markdown + : T extends IntegrationService.Markdown | IntegrationService.Slab ? MarkdownImportInput : T extends IntegrationService.JSON ? JSONImportInput @@ -130,12 +130,13 @@ export interface JSONImportScratch { * isn't part of any single task's input. Cleared when the import flips to * `Processed`. */ -export type ImportScratch = - T extends IntegrationService.Markdown - ? MarkdownImportScratch - : T extends IntegrationService.JSON - ? JSONImportScratch - : never; +export type ImportScratch = T extends + | IntegrationService.Markdown + | IntegrationService.Slab + ? MarkdownImportScratch + : T extends IntegrationService.JSON + ? JSONImportScratch + : never; /** * Per-page task input. Generated by the bootstrap task and consumed by @@ -213,7 +214,7 @@ export type JSONImportTaskInput = ( export type ImportTaskInput = T extends IntegrationService.Notion ? NotionImportTaskInput - : T extends IntegrationService.Markdown + : T extends IntegrationService.Markdown | IntegrationService.Slab ? MarkdownImportTaskInput : T extends IntegrationService.JSON ? JSONImportTaskInput diff --git a/shared/types.ts b/shared/types.ts index 1e22cd23ad..63b55f784d 100644 --- a/shared/types.ts +++ b/shared/types.ts @@ -168,6 +168,7 @@ export enum IntegrationService { Figma = "figma", Notion = "notion", Markdown = "markdown", + Slab = "slab", JSON = "json", } @@ -175,12 +176,14 @@ export type ImportableIntegrationService = Extract< IntegrationService, | IntegrationService.Notion | IntegrationService.Markdown + | IntegrationService.Slab | IntegrationService.JSON >; export const ImportableIntegrationService = { Notion: IntegrationService.Notion, Markdown: IntegrationService.Markdown, + Slab: IntegrationService.Slab, JSON: IntegrationService.JSON, } as const;