ImageGen
Technical details and background
Supplement to the README: how the parts are built, why, and what was measured. Measurements come from a
Ryzen AI Max+ 395 (iGPU Radeon 8060S, gfx1151, 128 GB unified memory, of which up to 108 GiB for the GPU).
The workflow titles below are the internal (German) names; the English interface shows them translated, e.g.
“Realistisch - Qwen 2.1” as “Realistic - Qwen 2.1” or “Video aus Bild - Wan 2.2” as “Image to video - Wan 2.2”.
#Structure
comfyui/Dockerfile: AMD's rocm/pytorch (ROCm 7.2, PyTorch 2.10) + ComfyUI in a fixed version. torch is pinned
against replacement by pip: normal PyTorch wheels have no gfx1151 kernels and silently compute on the CPU.
--disable-mmap, because mmap of large models is very slow under ROCm on gfx1151. The own extensions from
comfyui/custom_nodes/ are part of the image and copied into the custom_nodes folder at start.webapp/: FastAPI + one HTML page per view, SQLite for users, jobs and image ownership. At start the app builds its
workflows from build_workflows.py (in the image) plus the extensions in /extras (webapp/start.sh) and puts the
token templates into ComfyUI's input folder.- With the OllamaProxy the web app registers as a module (
/proxy-module): the proxy shows how much RAM and GPU
ComfyUI holds and frees it after idle time (WEBAPP_IDLE_FREE_MINUTES, default 2) so that language models have room
again.
#Data (IMAGEGEN_DATA)
models/ checkpoints/ diffusion_models/ loras/ vae/ text_encoders/ ... (see MODELS.md)
output/ finished images and videos
input/ input images (uploads, imports, token templates)
custom_nodes/ ComfyUI extensions
extras/ your own workflow extensions (*.py)
comfy-user/ ComfyUI settings; default/workflows/ = workflow menu
webapp-data/ webapp.db (users, jobs) – belongs in the backup
cache/ MIOpen/Triton kernel caches
#Workflows (workflows/)
build_workflows.py creates all workflows from one definition, in two formats: .<wbr>json</code> (UI format, for
ComfyUI's workflow menu: copy to <code>comfy-<wbr>user/<wbr>default/<wbr>workflows/<wbr></code>) and <code>api/<wbr><name>.<wbr>json</code> (API format for Open WebUI /
scripts) plus <code>api/<wbr>openwebui_<wbr>mapping.<wbr>json</code> (which node for prompt, model, size, seed …) and <code>api/<wbr>index.<wbr>json</code> (list for
the web app). Every workflow has one text field (node 10); quality and style additions are appended by the workflow
itself (<code>StringConcatenate</code>) so that they stay even if a client only inserts the subject.</p>
<pre><code class="language-bash">python3 workflows/build_workflows.py http://127.0.0.1:8188 workflows # UI + API, checks nodes against ComfyUI
python3 workflows/build_workflows.py --api-only /tmp/wf # API only, without ComfyUI
python3 workflows/run_workflow.py workflows/api/<name>.json "Prompt" [seed] [url] [--image file.png]
</code></pre>
<p><strong>Extensions:</strong> every <code>*.<wbr>py</code> in <code>extras/<wbr></code> is loaded; its function <code>apply(b)</code> may add workflows, options and LoRAs
(<a href="/en/docs/imagegen/erweiterungen.html">workflows/extras/README.en.md</a>). That way your own or delicate additions (e.g. LoRAs
for adult content) stay outside the repository.</p>
<div class="tablewrap"><table><thead><tr><th>Workflow</th><th>Model</th><th>Time</th><th>Prompt</th></tr></thead><tbody><tr><td>Anime - Anima</td><td>Anima Turbo v1.1</td><td>~13 s</td><td>sentences or tags, <code>safe</code>/<code>nsfw</code>/<code>explicit</code></td></tr><tr><td>Realistisch - Qwen 2.1</td><td>Qwen-Image 2.1 Turbo</td><td>~31 s (1024²)</td><td>sentences, also long/several people</td></tr><tr><td>Gezeichnet - D&D</td><td>Qwen-Image 2.1 Turbo + D&D style sentence</td><td>~31 s (1344×768)</td><td>sentences</td></tr><tr><td>Bearbeiten - Qwen 2.1 (edit)</td><td>Qwen-Image 2.1 Turbo</td><td>~70 s</td><td>instruction + image(s)</td></tr><tr><td>Charakter - Qwen 2.1</td><td>Qwen-Image 2.1 Turbo</td><td>~65 s</td><td>reference image of the character + new scene</td></tr><tr><td>D&D - Token aus Bild (token from image)</td><td>Qwen-Image 2.1 Turbo + frame</td><td>~65 s</td><td>image of the character, text optional</td></tr><tr><td>Test - Flux 3 Klein</td><td>FLUX 3 Image Klein (distilled, 16 steps)</td><td>~30 s (1024²)</td><td>sentences, style in the text</td></tr><tr><td>Video - Wan 2.2</td><td>Wan 2.2 T2V 14B fp8 + Lightx2v (4 steps)</td><td>~6 min (5 s)</td><td>scene + motion</td></tr><tr><td>Video aus Bild - Wan 2.2 (image to video)</td><td>Wan 2.2 I2V 14B fp8 + Lightx2v</td><td>~6 min (5 s)</td><td>start image + motion</td></tr><tr><td>Video aus Bild (Dynamisch) - Wan 2.2</td><td>Wan 2.2 I2V 14B fp8, 16 steps, CFG 3.5</td><td>~21 min (3 s)</td><td>start image + body/camera motion</td></tr></tbody></table></div>
<p>The models do not filter content. The web app only shows hints and options for adult content to users with NSFW
access.</p>
<p><strong>Qwen-Image 2.1</strong> (7B, generating and editing in one model, up to 10 reference images as <code><image1></code> …). Chain after
the official ComfyUI template “Qwen Image 2.1 Turbo Image Edit” without the prompt enhancer: <code>TextEncodeQwenImage21</code>
(images + VAE) → <code>QwenImage21Cache</code> → CFG 1, euler, 8 steps via <code>ManualSigmas</code>. The “Load Image” node of
Bearbeiten/Charakter shows the placeholder <code>bild_<wbr>hier_<wbr>hochladen.<wbr>png</code> (ComfyUI requires a file there).</p>
<p><strong>Editing was slow (~180 s), now ~70 s.</strong> Sampling takes ~35 s; the rest is in the vision part of the Qwen3-VL encoder,
which deliberately computes in fp32 (fp16/bf16 give NaN). Two brakes:</p>
<ul>
<li>hipBLASLt uses a fallback kernel (~1 TFLOPS) for fp32 GEMM on gfx1151. Remedy: <code>TORCH_<wbr>BLAS_<wbr>PREFER_<wbr>HIPBLASLT=0</code>
(rocBLAS; fp32 2.7×, bf16/fp16 ~30 % faster).</li>
<li><code>Conv3d</code> in the patch embedding: MIOpen searches ~125 s for the best kernel for every new image size. The extension
<code>conv3d_<wbr>patch_<wbr>embed</code> computes it as an equivalent matrix multiplication (kernel = stride).</li>
</ul>
<p><strong>Memory on unified-memory systems</strong> (<code>comfyui/<wbr>custom_<wbr>nodes/<wbr>proxy_<wbr>memory</code>): with <code>-<wbr>-<wbr>disable-<wbr>mmap</code> and the RAM pressure
cache (ComfyUI ≥ 0.42), unloaded models stay in process RAM (~22 GB), while <code>/<wbr>system_<wbr>stats</code> only reports the GPU part
and <code>/<wbr>free</code> does not return the RAM to the system. The extension</p>
<ul>
<li>adds <code>GET /<wbr>proxy/<wbr>memory</code> (RSS + GPU + loaded models) and <code>POST /<wbr>proxy/<wbr>free</code> (waits for unloading and returns the RAM
via <code>gc.<wbr>collect()</code>, <code>_<wbr>host_<wbr>emptyCache()</code> and <code>malloc_<wbr>trim(0)</code>),</li>
<li>caps <code>get_<wbr>free_<wbr>memory</code> at <code>min(GTT free, MemAvailable)</code> and empties the cache when RAM is short, so that Wan 2.2 (two
14B experts) does not run into a HIP OOM even next to a large language model,</li>
<li>sets <code>mallopt(M_<wbr>MMAP_<wbr>THRESHOLD, 128 KB)</code>: otherwise tensors end up in glibc arenas whose pages ROCm/<code>amdkfd</code> pulls
back into RAM via a <code>userptr</code> lock right after <code>malloc_<wbr>trim</code> (only <code>MADV_<wbr>DONTNEED</code>); with its own <code>mmap()</code> per tensor,
<code>free()</code> really releases them via <code>munmap()</code>.</li>
</ul>
<p><strong>Videos (Wan 2.2)</strong>: two experts (high noise levels = coarse motion, low = details), each with a Lightx2v 4-step LoRA,
CFG 1, euler/simple, shift 5, 16 fps, <code>SaveVideo</code> (MP4/H.264). Length = seconds × 16 + 1 (Wan is trained on ~5 s). For
image to video, “like the image” picks landscape/portrait/square per start image (dimensions from the file header),
otherwise the image would be cropped. Measured (5 s, 832×480): ~6 min; ~78 s of pure computation per sampling step,
VAE decode ~37 s. No reloading between the experts. The web app scales its estimate linearly with length and pixel
count.</p>
<p><strong>Motion</strong> (5 levels, <code>WAN_<wbr>MOTION</code> / <code>WAN_<wbr>MOTION_<wbr>HQ</code>): Wan 2.2 has no control like <code>motion_<wbr>bucket_<wbr>id</code> (SVD). Each level
adds a phrase to the prompt + shift 4/4.5/5/6.5/8 (fast) or 5/6.5/8/9.5/11 (dynamic). With Lightx2v, image to video is
strongly bound to the first image and tends to mouth/jaw twitching; that is what <strong>“Video aus Bild (Dynamisch)”</strong> is
for: native dual model without the 4-step LoRA, 16 steps with CFG 3.5 and sgm_uniform, mouth movements suppressed in
the negative prompt. CFG means double computation per step (measured 1229–1297 s for 3 s). Besides the shift, the
levels also set the expert switch to match sigma 0.9 (step 6/7/8/9/9).</p>
<p><strong>Prompt expansion</strong> (button at the prompt): a language model (best via the OllamaProxy, so that context and
<code>keep_<wbr>alive</code> are fixed) turns a short idea into a detailed English prompt (~3 s); placeholders <code>[.<wbr>.<wbr>]</code> are kept. Rules per
type (image, edit, video, image to video with start image) plus <code>ENHANCE_<wbr>HINT</code> per workflow. Example with the same
seed: “A female tiefling bard in a tavern” → without: an ordinary woman with a staff; expanded: a tiefling woman with
horns, tail and lute. Without the D&D rule the model invented candlelight → dark, brown images; so the rule asks for
bright, colourful light. For videos: no negations, describe motion concretely.</p>
<p><strong>D&D types</strong> (own tab, sub-choice “type” via <code>VARIANTS</code> → <code>group</code>/<code>variant</code>/<code>tab</code> in <code>index.<wbr>json</code>):</p>
<ul>
<li>Illustration: Qwen 2.1 + style sentence <code>DND_<wbr>STYLE</code> (colourful, painted illustration like the 5e books). “oil
painting” + “warm lighting” gave gold-brown, almost monochrome images → explicitly ask for a colourful palette and
mixed light. The model only half understands German → English (or prompt expansion).</li>
<li>Location: <code>DND_<wbr>LOCATION_<wbr>STYLE</code> (wide shot without a main character).</li>
<li>Token: bust, then a ring on top (<code>token_<wbr>rahmen.<wbr>png</code>, <code>ImageCompositeMasked</code> with inverted mask) and cut out round
(<code>token_<wbr>maske.<wbr>png</code>, <code>JoinImageWithAlpha</code>) → PNG with alpha, fixed 1024×1024. “Token from image” repaints the character
from your own image (same identity, D&D style).</li>
<li>Battle map and location map: Anima Aesthetic v1.1 (30 steps, CFG 4.5, er_sde, negative prompt) + Mapcraft LoRA, style
like Dungeondraft/Forgotten Adventures or Inkarnate: muted colours, ink outlines, flat light without lighting effects
and without a grid (the VTT software does both). Qwen 2.1 paints places nicely but at an angle/isometric; Anima Turbo
was too colourful.</li>
</ul>
<p><strong>Character</strong>: reference image as <code><image1></code>, prefix <code>CHARACTER_<wbr>PREFIX</code> (deliberately general – a fixed “horns” gave
characters without horns some) asks for the same identity but a new setting. Objects from the reference come along →
“her hands are empty”.</p>
<p><strong>Collapsed encoders</strong>: in the UI the text field of an encoder stays visible even if its text is connected; typing there
breaks the connection (style sentence gone). <code>build_<wbr>workflows.<wbr>py</code> therefore collapses such nodes.</p>
<p>Pure diffusion models (Anima, Qwen, Flux, Wan) load with “Load Diffusion Model” (<code>UNETLoader</code>, <code>diffusion_<wbr>models/<wbr></code>),
text encoder and VAE separately.</p>
<h2 id="web-app"><a class="anchor" href="#web-app" aria-hidden="true">#</a>Web app</h2>
<p><strong>Interface:</strong> tabs Generate / Image as input / D&D, below them workflow, prompt, size.
Save the result, keep editing it right away, repeat with a new seed or delete; below that “My images” (loads more while
scrolling; large view with workflow, date, prompt and “use prompt”).</p>
<p><strong>Editing several images:</strong> upload several and/or select them in “My images” → “Edit (n)”. Each input image is
edited separately with the same prompt. First all images are checked (permissions, content), then uploaded – on an
error nothing is left half done. At most 100 input images per job.</p>
<p><strong>Variables & series:</strong></p>
<ul>
<li>In the prompt <code>[red hair|blue hair]</code> → one option per image. Named variables, one per line:
<code>hair = red hair | blue hair</code>, in the prompt <code>[hair]</code>; the same value everywhere in the same image. Values may contain
variables themselves (any depth; circular references give an error message).</li>
<li>Count without a fixed limit, each with its own seed. Rolled evenly: every option comes up before one repeats. Rolling
happens in the app; ComfyUI only gets the finished prompt.</li>
<li><strong>Cancel:</strong> waiting jobs leave the queue, the running one is interrupted specifically; finished images stay.</li>
</ul>
<p><strong>Users:</strong> a setup wizard creates the admin. The admin creates users, sets passwords, grants admin rights and <strong>NSFW
access</strong> (without it, options with <code>"nsfw": true</code> are missing and the server rejects other levels). Passwords
PBKDF2-SHA256 (600k rounds), session 30 days (cookie httponly, SameSite=Lax, Secure behind HTTPS). Forgot the admin
password: move <code>webapp.<wbr>db</code> away → the wizard appears again (image ownership is lost).</p>
<p><strong>Images and jobs:</strong> everyone only sees and deletes their own images. ZIP download (own images, or all as admin). File
names get a time stamp, image addresses a version, so that no old image appears from the browser cache after deleting.
Job status: open → done / error / lost (ComfyUI restarted) / cancelled. The server reconciles open jobs itself every
20 s.</p>
<p><strong>API for other programs:</strong> key per user via “API” in the app (only the SHA-256 hash is stored).
Header <code>Authorization: Bearer ig_<wbr>.<wbr>.<wbr>.<wbr></code>; rights like the account, but no administration, no keys, no password. The same
workflows apply including style sentences, options and prompt expansion (<code>enhance</code>, on by default; if the language model
is gone, your own text passes through with a hint).</p>
<ul>
<li><code>GET /<wbr>api/<wbr>v1/<wbr>workflows</code> – list as in the app (options, sizes, lengths)</li>
<li><code>POST /<wbr>api/<wbr>v1/<wbr>generate</code> (JSON): <code>workflow</code>, <code>prompt</code>, <code>enhance</code>, <code>image</code>/<code>images</code> (Base64 or data: URL), <code>width</code>,
<code>height</code>, <code>seconds</code>, <code>motion</code>, <code>action</code>, <code>seed</code>, <code>count</code>, <code>variables</code>, <code>wait</code> (seconds; then with <code>results</code> including
image <code>url</code>, at most <code>WEBAPP_<wbr>API_<wbr>WAIT_<wbr>MAX</code> = 1800)</li>
<li><code>GET /<wbr>api/<wbr>v1/<wbr>jobs/<wbr><id></code>; images via the <code>url</code> with the same key</li>
<li>OpenAI-compatible: <code>GET /<wbr>v1/<wbr>models</code> (image workflows only), <code>POST /<wbr>v1/<wbr>images/<wbr>generations</code> (<code>model</code> = workflow file or
title, <code>dall-<wbr>e*</code>/<code>gpt-<wbr>image*</code> = first workflow, <code>n</code> up to 10, <code>size</code>, <code>response_<wbr>format</code> b64_json/url) and
<code>POST /<wbr>v1/<wbr>images/<wbr>edits</code> (multipart <code>image</code>). Response with <code>revised_<wbr>prompt</code>. Open WebUI: Images → OpenAI, base URL
<code>http:/<wbr>/<wbr><host>:8190/<wbr>v1</code>.</li>
</ul>
<p><strong>Module for the OllamaProxy:</strong> <code>GET /<wbr>proxy-<wbr>module</code>, <code>GET /<wbr>proxy-<wbr>module/<wbr>memory</code>, <code>POST /<wbr>proxy-<wbr>module/<wbr>free</code>, optionally
protected with <code>WEBAPP_<wbr>MODULE_<wbr>TOKEN</code>.</p>
<p><strong>Security:</strong> meant for your own network; to the outside only behind a reverse proxy with HTTPS (that sets
<code>X-<wbr>Forwarded-<wbr>For</code>) or via VPN. ComfyUI (:8188) stays in the LAN; the app talks to it internally.</p>
<ul>
<li>Login lockout: 10 failed attempts per IP and 20 per name within 10 min, 1 s waiting time per failed attempt; unknown
names cost the same time.</li>
<li><code>X-<wbr>Forwarded-<wbr>*</code> is only trusted from trusted proxies (<code>WEBAPP_<wbr>TRUSTED_<wbr>PROXIES</code>) and read from the right.</li>
<li>Changing requests only from the app's own page (origin check), protection against embedding, <code>nosniff</code>.</li>
<li>Uploads: at most 30 MB and only real PNG/JPG/WebP (content checked); uploaded input images are removed after the job.</li>
</ul>
<p><strong>Tests:</strong> <code>webapp/<wbr>run_<wbr>tests.<wbr>sh</code> (only needs Docker) builds the image, creates the workflows with the test extension
<code>webapp/<wbr>tests/<wbr>extras/<wbr></code> and runs <code>webapp/<wbr>tests/<wbr>test_<wbr>app.<wbr>py</code> against a simulated ComfyUI (<code>tests/<wbr>fake_<wbr>comfy.<wbr>py</code>). Covers
setup, login protection, permissions, generating, uploads, jobs, variables, series, cancelling, ZIP, videos, prompt
expansion, options, D&D types, API and the module interface (69 tests).</p>
<h2 id="operation"><a class="anchor" href="#operation" aria-hidden="true">#</a>Operation</h2>
<pre><code class="language-bash">docker compose --env-file ~/imagegen/imagegen.env up -d --build # start/update
docker compose --env-file ~/imagegen/imagegen.env logs -f comfyui
docker compose --env-file ~/imagegen/imagegen.env down # stop
</code></pre>
<p>Updating ComfyUI: raise <code>COMFYUI_<wbr>VERSION</code> in <code>imagegen.<wbr>env</code> and rebuild. Your own ComfyUI extensions in <code>custom_<wbr>nodes/<wbr></code>
are executable code – only trustworthy ones, and no packages that pull in <code>torch</code>.</p>
<h2 id="measured"><a class="anchor" href="#measured" aria-hidden="true">#</a>Measured</h2>
<ul>
<li>PyTorch in the container: <code>gfx1151</code>, 108 GiB for the GPU, matmul fp16/bf16 ~26 TFLOPS</li>
<li>SDXL Base 1.0, 1024×1024, 25 steps: 21.6 s per image</li>
<li>runs fine in parallel to a 26B language model with 128k context</li>
</ul>
<p class="dfoot"><a href="https://github.com/Poldion/ImageGen/blob/main/docs/technik.en.md">View on GitHub →</a> · <a href="/projekte/en/imagegen.html">Project page →</a></p>
</article>
</div>
</main>
<footer class="foot"><div class="wrap">
<span>Pold – All projects are under free licenses and run on your own hardware.</span>
<span class="footlinks"><a href="https://github.com/Poldion"><svg viewBox="0 0 16 16" width="18" height="18" aria-hidden="true" fill="currentColor"><path d="M8 0C3.58 0 0 3.58 0 8c0 3.54 2.29 6.53 5.47 7.59.4.07.55-.17.55-.38 0-.19-.01-.82-.01-1.49-2.01.37-2.53-.49-2.69-.94-.09-.23-.48-.94-.82-1.13-.28-.15-.68-.52-.01-.53.63-.01 1.08.58 1.23.82.72 1.21 1.87.87 2.33.66.07-.52.28-.87.51-1.07-1.78-.2-3.64-.89-3.64-3.95 0-.87.31-1.59.82-2.15-.08-.2-.36-1.02.08-2.12 0 0 .67-.21 2.2.82.64-.18 1.32-.27 2-.27.68 0 1.36.09 2 .27 1.53-1.04 2.2-.82 2.2-.82.44 1.1.16 1.92.08 2.12.51.56.82 1.27.82 2.15 0 3.07-1.87 3.75-3.65 3.95.29.25.54.73.54 1.48 0 1.07-.01 1.93-.01 2.2 0 .21.15.46.55.38A8.013 8.013 0 0 0 16 8c0-4.42-3.58-8-8-8z"/></svg> github.com/Poldion</a></span>
<a class="btn primary donate" data-donate href="#">Support</a>
</div></footer>
<script src="/projekte/site.js"></script>
<script src="/docs.js"></script>
</body>
</html>