A worked example

Heimdallm's own reference provider lists eleven served entries across eight rails: chat, image generation, video generation, embeddings, rerank, decisions, speech and transcription. The video card is listed twice because it renders through two pipelines. Three Laya decision engines share GPU 4 with the reranker. Behind its origin sit ten engines, five vLLM processes, two ComfyUI graph runners and three Laya processes, and in front of them one small provider service that publishes the model list and forwards inference. The service lives in the repository's examples/provider-vllm/ and is configured by a single YAML file.

If you already have one origin serving an OpenAI-compatible API for all your models, you do not need it. Serve the JSON document yourself and skip to going live.

The shape of the problem#

Heimdallm talks to one base URL per source, and for an OpenAI-compatible source that base URL is the one an OpenAI SDK takes as base_url — it ends in /v1. The model list is <base URL>/models, and every inference request goes to the rail's own path under it: <base URL>/chat/completions, <base URL>/embeddings, <base URL>/rerank, <base URL>/systemone, <base URL>/videos/generations and the rest. Little serves that shape unaided.

• An inference engine such as vLLM or Laya serves one model per process. Ten engines use ten ports.
• A diffusion engine does not speak the dialect at all. ComfyUI takes a workflow graph and answers with the files a run produced; there is no /v1/images/generations on it to forward to.

The provider service is the adapter for both cases. It publishes the model list, authenticates Heimdallm's calls, and forwards each chat, embeddings or transcription request to the engine serving that model, swapping your marketplace model id for the engine's own name on the way through. For the two rendering rails it does more than forward: it builds a ComfyUI workflow from the request, queues it, and reads the rendered files back out.

Heimdallm
  │  pulls /v1/models
  │  sends inference
  ▼
provider service ──┬─▶ four vLLM engines, one model each
(one origin, TLS)  ├─▶ two ComfyUI graph runners
                   ├─▶ one SGLang chat engine
                   └─▶ three Laya decision engines

The mesh companion shares card 1 with HiDream under its own Compose profile. Both mesh routes and the card 0 chat route remain listed in MODELS_SERVED while their engines answer. examples/provider-vllm/impcc/compose.yml reserves GPU 4 for the reranker and three Laya processes. The reranker uses a 0.80 allocator budget and each Laya process caps itself at 3 GiB. The mesh companion and HiDream share card 1; the other primary engines each reserve a card. A card driven to within a few tens of MB of its capacity stops accepting work until the host reboots. The *_device names inside a comfyui_video block are still the container's own, so cuda:0 there is whichever card the deployment reserved for that service — always the only card the service sees.

The ComfyUI runners share one image built from examples/provider-vllm/impcc/comfyui.Dockerfile, which is ComfyUI at a pinned release on top of the same torch and CUDA build the vLLM engines run. Installing a second torch beside a working one is how a card that served stops serving, so ComfyUI's own torch requirements are stripped and everything else is installed against the pair already there.

The rendering rails#

POST /v1/images/generations, POST /v1/videos/generations and POST /v1/meshes/generations arrive at the service in the same OpenAI dialect as everything else, and leave it as a ComfyUI graph. provider-service/src/comfyui.ts is the whole of that translation for stills, with comfyui-video.ts and comfyui-mesh.ts beside it; queueing, waiting and reading the files back is shared. Three properties of the translation are deliberate:

• The graph is built per request from the endpoint's own configuration, so a caller cannot reach past the parameters the endpoint declares and drive the engine directly.
• The resolution comes from the request and is checked against the envelope the endpoint publishes. Nothing in the translation widens it.
• steps is honoured but capped. Steps are compute the caller does not pay for, since the bill counts the output rather than the effort, so the ceiling lives in the service.

The output modality's supported_parameters names the request fields each renderer reads. Engine settings such as upstream.comfyui.cfg, sampler and scheduler stay in the upstream configuration unless the route also accepts them from a request.

The models.yaml#

One file drives everything. service.api_keys lists the credentials you accept, and an env:NAME entry reads the secret from the environment so it never lives in the file:

service:
  api_keys:
    - env:PROVIDER_API_KEY

Each models entry pairs a native v2.4 declaration in doc with the upstream engine and its concurrent request capacity.total. The service publishes the declaration directly at /v1/models, adding its live slot occupancy. It validates declared fields at boot without fetching Hugging Face or inventing missing metadata. The marketplace fills omitted repository facts and then validates the completed listing. A successful provider startup alone does not prove admission when the linked repository is unavailable or lacks a required fact.

Leave checkpoint names, release dates, descriptions and other repository metadata out of the configuration. Declare identity, the exact weights repository, prices and the deployment's supported parameters, limits and policies. A model's price belongs to its native input/output modality, so an explicit modality object is needed when that object carries a price or serving constraint.

The examples below are the entries from the reference configuration, each with its own models envelope. Combine their entries under one models key and supply the service block above. Keep an explicit weight format when the engine selects a particular build from a repository containing several formats or the Hub metadata cannot identify the loaded format.

A text model#

This entry serves text chat at a 131,072-token context limit. The 2B BF16 checkpoint uses card 0 with an 85 percent SGLang static memory fraction. Eight shorter chats can run together; full-context requests use more cache and can reduce that concurrency. The engine disables the model's thinking trace by default. top_k: -1 disables top-k filtering. stop and stop_regex each accept a string or an array of strings. Stop-token IDs are tokenizer IDs, and no_stop_trim includes the matched stop text. Repository metadata supplies the checkpoint's name, release date and description.

models:
  - doc:
      schema_version: '2.4'
      id: openbmb/minicpm5-2b
      hugging_face_id: openbmb/MiniCPM5-2B
      quantization: bf16
      input_modalities:
        - type: text
          supported_inputs:
            max_context_length:
              value: 131072
              unit: token
          pricing:
            - type: prompt
              unit: token
              cost_usd: '0.00000003'
            - type: cached_prompt
              unit: token
              cost_usd: '0.00000001'
      output_modalities:
        - type: text
          max_length:
            value: 131072
            unit: token
          streaming: true
          supported_parameters:
            temperature:
              type: range
              min: 0
              max: 2
              default: 1
            top_p:
              type: range
              exclusive_min: 0
              max: 1
              default: 0.95
            top_k:
              type: integer
              min: -1
              max: 200
              default: -1
            min_p:
              type: range
              min: 0
              max: 1
              default: 0
            frequency_penalty:
              type: range
              min: -2
              max: 2
              default: 0
            presence_penalty:
              type: range
              min: -2
              max: 2
              default: 0
            repetition_penalty:
              type: range
              exclusive_min: 0
              max: 2
              default: 1
            stop:
              type: one_of
              options:
                - type: string
                - type: array
                  items:
                    type: string
            stop_token_ids:
              type: array
              items:
                type: integer
                min: 0
            stop_regex:
              type: one_of
              options:
                - type: string
                - type: array
                  items:
                    type: string
            no_stop_trim:
              type: boolean
              default: false
            ignore_eos:
              type: boolean
              default: false
            skip_special_tokens:
              type: boolean
              default: true
            max_tokens:
              type: integer
              min: 1
              max: 131072
              unit: token
            max_completion_tokens:
              type: integer
              min: 1
              max: 131072
              unit: token
            logit_bias:
              type: unknown
            n:
              type: integer
              min: 1
            response_format:
              type: unknown
            structured_outputs:
              type: boolean
            logprobs:
              type: boolean
            top_logprobs:
              type: integer
              min: 0
              max: 20
            tools:
              type: array
              items:
                type: unknown
            tool_choice:
              type: unknown
            parallel_tool_calls:
              type: boolean
              default: true
            reasoning_effort:
              type: enum
              values: [none, medium]
          pricing:
            - type: completion
              unit: token
              cost_usd: '0.0000001'
      is_ready: true
      datacenters:
        - country_code: CA
      compliance:
        zdr: true
        hipaa: false
      heimdall:
        serving_id: minicpm5
        data_policy:
          training: false
          retention_days: 0
    upstream:
      base_url: http://127.0.0.1:8098/v1
      model: minicpm5
    capacity:
      total: 8
  - doc:
      schema_version: '2.4'
      id: hidream-ai/hidream-o1-image
      hugging_face_id: Comfy-Org/HiDream-O1-Image
      input_modalities:
        - type: text
          supported_inputs:
            max_context_length:
              value: 8192
              unit: token
        - type: image
          supported_inputs:
            sources: { type: enum, values: [base64] }
            formats: { type: enum, values: [image/png, image/jpeg, image/webp] }
      output_modalities:
        - type: image
          streaming: false
          supported_parameters:
            input_references:
              type: array
              max_items: 1
            negative_prompt:
              type: unknown
            size:
              type: unknown
            n:
              type: integer
              min: 1
              max: 4
            steps:
              type: integer
              min: 1
              max: 50
              default: 40
            seed:
              type: integer
              min: 0
              max: 2147483647
            guidance_scale:
              type: range
              min: 0
              max: 100
              default: 5
            sampler:
              type: enum
              values:
                - euler
                - euler_cfg_pp
                - euler_ancestral
                - euler_ancestral_cfg_pp
                - heun
                - heunpp2
                - exp_heun_2_x0
                - exp_heun_2_x0_sde
                - dpm_2
                - dpm_2_ancestral
                - lms
                - dpm_fast
                - dpm_adaptive
                - dpmpp_2s_ancestral
                - dpmpp_2s_ancestral_cfg_pp
                - dpmpp_sde
                - dpmpp_sde_gpu
                - dpmpp_2m
                - dpmpp_2m_cfg_pp
                - dpmpp_2m_sde
                - dpmpp_2m_sde_gpu
                - dpmpp_2m_sde_heun
                - dpmpp_2m_sde_heun_gpu
                - dpmpp_3m_sde
                - dpmpp_3m_sde_gpu
                - ddpm
                - lcm
                - ipndm
                - ipndm_v
                - deis
                - cfgpp_ud10_ab
                - res_multistep
                - res_multistep_cfg_pp
                - res_multistep_ancestral
                - res_multistep_ancestral_cfg_pp
                - gradient_estimation
                - gradient_estimation_cfg_pp
                - er_sde
                - seeds_2
                - seeds_3
                - sa_solver
                - sa_solver_pece
                - ddim
                - uni_pc
                - uni_pc_bh2
              default: dpmpp_2m_sde_gpu
            scheduler:
              type: enum
              values:
                [
                  simple,
                  sgm_uniform,
                  karras,
                  exponential,
                  ddim_uniform,
                  beta,
                  normal,
                  linear_quadratic,
                  kl_optimal,
                ]
              default: normal
            denoise:
              type: range
              min: 0
              max: 1
              default: 1
            seam_smoothing_enabled:
              type: boolean
              default: true
            seam_smoothing:
              type: object
              properties:
                start_percent: { type: range, min: 0, max: 1, default: 0.8 }
                end_percent: { type: range, min: 0, max: 1, default: 1 }
                pattern: { type: enum, values: [single_shift, symmetric], default: single_shift }
                passes: { type: enum, values: ['2', '4', ramp_2_4, ramp_2_4_8], default: ramp_2_4 }
                blend: { type: enum, values: [average, window, median], default: median }
                strength: { type: range, min: 0, max: 1, default: 1 }
          pricing:
            - type: completion
              unit: image
              cost_usd: '0.029360128'
      is_ready: true
      datacenters:
        - country_code: CA
      compliance:
        zdr: true
        hipaa: false
      heimdall:
        reference_inputs:
          image:
            description: >-
              One image guides an edit; describe the desired changes in the
              prompt. Reference images must contain at most 16,777,216 pixels
              and are fitted to the output shape with padding. Edits sample at
              native resolution before resizing to the requested output size, so
              they take longer than small text-only renders.
        serving_id: hidream
        base_model_hugging_face_id: HiDream-ai/HiDream-O1-Image
        data_policy:
          training: false
          retention_days: 0
        image_envelope:
          max_megapixels: 4.194304
          min_edge: 1024
          max_edge: 3104
          edge_multiple: 32
      quantization: bf16
    upstream:
      base_url: http://127.0.0.1:8099
      model: hidream
      comfyui:
        reference_inputs: true
        checkpoint: hidream_o1_image_bf16.safetensors
        width: 2048
        height: 2048
        steps: 40
        max_steps: 50
        cfg: 5
        noise_scale: 8
        sampler: dpmpp_2m_sde_gpu
        scheduler: normal
        seam_smoothing:
          start_percent: 0.8
          end_percent: 1
          pattern: single_shift
          passes: ramp_2_4
          blend: median
          strength: 1
        output_dir: /mnt/render-output/hidream
    capacity:
      total: 1

A video model#

A clip is frames, so the envelope carries a third axis. heimdall.video_envelope.frame is an image_envelope read by the same rules, max_seconds and fps bound that axis, and frame_step is the temporal grid the frame count sits on, which for this engine's causal VAE is one keyframe plus groups of eight.

Both grids are declared because the engine enforces neither: a request for 1280x720 comes back 1280x704 and one for 96 frames comes back 89, floored with no error and no warning. Billing the request as asked rather than as rendered would overcharge by 5.4% every time, so declaring the grids is what makes Heimdallm refuse an off-grid ask instead.

The video output's completion price is $0.01 per second of delivered video across the supported shapes and frame rates. The default 1280×704 clip contains 97 frames at 25 fps: 3.88 seconds, costing $0.0388. The frame grid determines which durations the engine can produce; it does not create a per-frame charge. The declared generate_audio default includes native sound; false selects silent output at the same bundled rate. The audio VAE decodes the jointly sampled audio latent and the MP4 contains the resulting soundtrack. Image, audio and video references condition the output through input_references; supplied audio is preserved as the soundtrack.

roles is where the served entry says what each kind of reference is for, and it publishes no video reference: the in-context control adapter reads a rendered depth, edges or pose pass, one per generation, and control_sets lists the three singles. Neither is written in this file. The service projects both from the engine — a control role exists for an endpoint exactly where a control_lora is loaded to read one — so the file carries the description and the promise follows the adapter rather than sitting beside it. An image naming no role is the frame the clip starts on for a caller reaching this service directly; through the marketplace a role-less item arrives as reference, which this entry does not take, so a customer names the role or sends the picture as a frame_images item. reference_strength controls how closely the clip follows a control pass, from 0.01 to 1 with a default of 0.7. The output declares it in v2.4 supported_parameters, so the marketplace publishes its bounds and the video job forwards the selected value. Sending it without a control pass is refused with HTTP 400. See video references for input limits and combinations.

models:
  - doc:
      schema_version: '2.4'
      id: lightricks/ltx-2.5
      hugging_face_id: Lightricks/LTX-2.5
      input_modalities:
        - type: text
          supported_inputs:
            max_context_length:
              value: 1024
              unit: token
        - type: image
          supported_inputs:
            sources: { type: enum, values: [base64] }
            formats: { type: enum, values: [image/png, image/jpeg, image/webp] }
        - type: video
          supported_inputs:
            sources: { type: enum, values: [base64] }
            formats: { type: enum, values: [video/mp4, video/webm] }
        - type: audio
          supported_inputs:
            sources: { type: enum, values: [base64] }
            formats:
              { type: enum, values: [audio/wav, audio/mpeg, audio/flac, audio/ogg, audio/mp4, audio/webm] }
      output_modalities:
        - type: video
          streaming: false
          supported_parameters:
            input_references:
              type: array
              max_items: 3
            generate_audio:
              type: boolean
              default: true
            size:
              type: unknown
            seconds:
              type: unknown
            duration:
              type: enum
              values:
                - 1
                - 9
            fps:
              type: enum
              values:
                - 24
                - 25
            seed:
              type: integer
              min: 0
              max: 2147483647
            reference_strength:
              type: range
              min: 0.01
              max: 1
              default: 0.7
            guidance_scale: &ltx_guidance
              type: range
              min: 0
              max: 20
              default: 1
            sampler: &ltx_sampler
              type: enum
              values:
                - euler
                - euler_cfg_pp
                - euler_ancestral
                - euler_ancestral_cfg_pp
                - heun
                - heunpp2
                - exp_heun_2_x0
                - exp_heun_2_x0_sde
                - dpm_2
                - dpm_2_ancestral
                - lms
                - dpm_fast
                - dpm_adaptive
                - dpmpp_2s_ancestral
                - dpmpp_2s_ancestral_cfg_pp
                - dpmpp_sde
                - dpmpp_sde_gpu
                - dpmpp_2m
                - dpmpp_2m_cfg_pp
                - dpmpp_2m_sde
                - dpmpp_2m_sde_gpu
                - dpmpp_2m_sde_heun
                - dpmpp_2m_sde_heun_gpu
                - dpmpp_3m_sde
                - dpmpp_3m_sde_gpu
                - ddpm
                - lcm
                - ipndm
                - ipndm_v
                - deis
                - cfgpp_ud10_ab
                - res_multistep
                - res_multistep_cfg_pp
                - res_multistep_ancestral
                - res_multistep_ancestral_cfg_pp
                - gradient_estimation
                - gradient_estimation_cfg_pp
                - er_sde
                - seeds_2
                - seeds_3
                - sa_solver
                - sa_solver_pece
                - ddim
                - uni_pc
                - uni_pc_bh2
              default: euler_ancestral
            sigmas: &ltx_sigmas
              type: array
              items: { type: range, min: 0, max: 1 }
              max_items: 9
            crf: &ltx_crf
              type: integer
              min: 0
              max: 51
              default: 23
          pricing:
            - type: completion
              unit: second
              cost_usd: '0.01'
      is_ready: true
      datacenters:
        - country_code: CA
      compliance:
        zdr: true
        hipaa: false
      heimdall:
        reference_inputs:
          video:
            description: >-
              Combine at most three items in one generation: one or two images,
              the frame the clip starts on and the frame it ends on; one control
              video; and one audio file. The video is an already rendered depth,
              edge or pose pass whose structure the clip keeps, one pass per
              generation, read at an attention of 0.7 and never noised. Audio
              supplies the soundtrack and requires audio output enabled; sound
              from the video file is not used. Audio and video inputs must be at
              most 60 seconds; images and video frames must contain at most
              16,777,216 pixels. Visual references are scaled to cover the
              output frame and centre-cropped to it, so a pass at another aspect
              loses its edges rather than gaining bars that read as far depth.
              Clips are trimmed or extended to the selected output length: video
              repeats its last frame and audio pads with silence.
        serving_id: ltx-2-5
        data_policy:
          training: false
          retention_days: 0
        video_envelope:
          frame:
            max_megapixels: 0.90112
            min_edge: 256
            max_edge: 1280
            edge_multiple: 64
          max_seconds: 15
          fps:
            - 24
            - 25
          frame_step: 8
          audio: true
      quantization: int8
    upstream:
      base_url: http://127.0.0.1:8100
      model: ltx-2-5
      comfyui_video:
        reference_inputs: true
        unet: LTX-2.5-Distilled-Q8_0.gguf
        clip: gemma4-12b-with-proj-ltx-2.5-comfy-int8-convrot.safetensors
        clip_type: ltxv
        vae: ltx-2.5-video-vae-bf16.safetensors
        audio_vae: ltx-2.5-audio-vae-bf16.safetensors
        control_lora: ltx-2.3-22b-ic-lora-union-control-ref0.5.safetensors
        decode_tile: 768
        generate_audio: true
        width: 1280
        height: 704
        seconds: 3.88
        fps: 25
        sigmas:
          - 1
          - 0.99375
          - 0.9875
          - 0.98125
          - 0.975
          - 0.909375
          - 0.725
          - 0.421875
          - 0
        cfg: 1
        sampler: euler_ancestral
        codec: h264
        crf: 23
        output_dir: /mnt/render-output/ltx
    capacity:
      total: 1

The same model in two stages#

lightricks/ltx-2.5-2x is that checkpoint on that engine, listed a second time because it renders through a second pipeline and costs what that pipeline costs. A customer selects it by naming it, nothing routes to it implicitly, and the entry above is unchanged by its presence. Listing a pipeline as its own model is what lets a caller choose the cost: a price is per entry, and a parameter that tripled the bill on the entry above would be a surprise on a rate the caller already agreed to. It is also the one entry in the document that carries its own name and description: the catalogue names an unnamed entry from its repository, and this entry shares the repository above.

Stage one is the graph above at half the asked frame, with the same schedule, guidance, sampler and references. LTXVLatentUpsampler, loaded from the checkpoint two_stage.upscaler names, doubles the latent it produces on both spatial axes, and a second sampler finishes it at 0.85, 0.725, 0.4219, 0 — three steps starting below 1, because stage one already laid the scene down. A frame the clip starts on is prepared once and laid in twice, at first_frame_strength while stage one composes and at 1 on the doubled latent, since the upscaler returns a latent carrying no noise mask. A control pass conditions stage one only: it rides as extra latent frames that come off before the upscaler sees the latent. A request carrying a first_frame image may set first_frame_strength and first_frame_compression on this two-stage entry; the configured values are its defaults. Requests naming either control without that image are refused.

The grid doubles with the frame. The control adapter reads its pass at half the latent's resolution, so stage one's frame is on a multiple of 64 and the frame the caller asks for is on 128, which is what frame.edge_multiple declares. The service refuses a two_stage block beside any other edge multiple, and refuses a configured default frame off the same grid, since that default is the one frame a request reaches without being checked against the envelope. On that grid the entry publishes 2048x1152, 2.6 times the area of the base entry's 1280x704.

A length is a frame count before it is a number of seconds, and both entries render 8n + 1 frames. POST /videos takes duration as a whole number of seconds, so a length a customer can name is one whose fps x seconds lands on that grid: 24 fps never does, and 25 fps reaches the seconds congruent to 1 modulo 8. Both entries admit 1 and 9 seconds, and both declare those under duration, which is what GET /videos/models publishes as supported_durations. A request naming no length renders the endpoint's own default of 3.88 seconds. An entry that declared no duration would publish supported_durations: null and leave a customer to find the lengths it refuses one 400 at a time.

The completion price is $0.03 per second of delivered video against the base entry's $0.01. A two-stage render is a full stage-one pass over a quarter of the tokens plus a three-step refine over all of them, and the refine is most of what the figure buys.

A checkpoint the engine holds is not a setting the file carries, so it is checked against the engine. The service reads the node classes a verdict needs once per engine, since a class describes the engine rather than the entry — LTXVLatentUpsampler and the named upscaler file for a two_stage entry, GetICLoRAParameters and the control_lora file for an entry publishing control roles — and withholds the entry from /v1/models while any of them goes unreported, naming the missing one in the log. ComfyUI validates a prompt at submission and answers 400 for a node it has no class for or a file it does not hold, so nothing renders stage one and fails late; what the check buys is that routed traffic never meets a promise the deployment cannot keep, and no endpoint has its own uptime counted down for one. An engine that could not answer at all — one still loading, one refusing the connection — has said nothing about any part, so it is asked again on the next pull rather than left withheld until someone restarts the service. That read runs beside the pull rather than in front of it, so the entry appears on the pull after the one that asked and no list waits on an engine slow to answer; an entry admitted that way is warmed as it is admitted, so the first render routed to it is not the one that loads the weights. That warm holds the card the way a customer's render does, reads busy on every entry queueing on it, and ends on the same ceiling.

Capacity is declared per entry and both LTX entries name one engine, so each declares one slot and Heimdallm can hold a slot on each at the same time. The service runs video generations one at a time per engine base URL, so the second queues behind the first on the card rather than meeting it there. An engine two entries share is an engine that queues them, not one that refuses the second, and what each entry reports in live_capacity.current is what the card holds rather than what the entry holds, so a card mid-render reads busy on both listings rather than one of them reading idle and then waiting the other render out. MODELS_SERVED names the serving ids a host runs, and the reference host lists both LTX entries, so it renders them on the one card in turn. The entry names base_model, which files it under the base model's family beside the single-stage entry rather than as a second model carrying the same weights.

models:
  - doc:
      schema_version: '2.4'
      id: lightricks/ltx-2.5-2x
      # Named in the document: the catalogue names an unnamed entry from its repository, and this
      # entry shares its repository with the single-stage one.
      name: 'Lightricks: LTX-2.5 2x'
      description: >-
        LTX-2.5 rendered in two stages: the clip is sampled at half the
        requested frame, the latent is upsampled 2x by the LTX-2.5 latent
        spatial upscaler, and a short refine pass finishes it at the requested
        frame, up to 2.36 megapixels on a 128 grid with no edge over 2432
        pixels. Clips come to a multiple of eight frames plus one, so at 25 fps
        the whole-second lengths are 1 and 9; an omitted length renders 3.88
        seconds at 2048x1152.
      hugging_face_id: Lightricks/LTX-2.5
      input_modalities:
        - type: text
          supported_inputs:
            max_context_length:
              value: 1024
              unit: token
        - type: image
          supported_inputs:
            sources: { type: enum, values: [base64] }
            formats: { type: enum, values: [image/png, image/jpeg, image/webp] }
        - type: video
          supported_inputs:
            sources: { type: enum, values: [base64] }
            formats: { type: enum, values: [video/mp4, video/webm] }
        - type: audio
          supported_inputs:
            sources: { type: enum, values: [base64] }
            formats:
              { type: enum, values: [audio/wav, audio/mpeg, audio/flac, audio/ogg, audio/mp4, audio/webm] }
      output_modalities:
        - type: video
          streaming: false
          supported_parameters:
            input_references:
              type: array
              max_items: 3
            generate_audio:
              type: boolean
              default: true
            size:
              type: unknown
            seconds:
              type: unknown
            duration:
              type: enum
              values:
                - 1
                - 9
            fps:
              type: enum
              values:
                - 24
                - 25
            seed:
              type: integer
              min: 0
              max: 2147483647
            reference_strength:
              type: range
              min: 0.01
              max: 1
              default: 0.7
            guidance_scale: *ltx_guidance
            sampler: *ltx_sampler
            sigmas: *ltx_sigmas
            crf: *ltx_crf
            refine_guidance_scale: *ltx_guidance
            refine_sampler: *ltx_sampler
            refine_sigmas:
              type: array
              items: { type: range, min: 0, max: 1 }
              max_items: 4
            first_frame_strength:
              type: range
              min: 0
              max: 1
              default: 0.7
            first_frame_compression:
              type: integer
              min: 0
              max: 100
              default: 18
          pricing:
            - type: completion
              unit: second
              cost_usd: '0.03'
      is_ready: true
      datacenters:
        - country_code: CA
      compliance:
        zdr: true
        hipaa: false
      heimdall:
        serving_id: ltx-2-5-2x
        base_model: lightricks/ltx-2.5
        data_policy:
          training: false
          retention_days: 0
        video_envelope:
          frame:
            max_megapixels: 2.359296
            min_edge: 512
            max_edge: 2432
            edge_multiple: 128
          max_seconds: 9
          fps:
            - 24
            - 25
          frame_step: 8
          audio: true
        reference_inputs:
          video:
            description: 'Combine at most three items in one generation: one or two images,
              the frame the clip starts on and the frame it ends on; one control
              video; and one audio file. The video is an already rendered depth,
              edge or pose pass whose structure the clip keeps, one pass per
              generation, read at an attention of 0.7 and never noised. Audio
              supplies the soundtrack and requires audio output enabled; sound
              from the video file is not used. Audio and video inputs must be at
              most 60 seconds; images and video frames must contain at most
              16,777,216 pixels. Visual references are scaled to cover the
              output frame and centre-cropped to it, so a pass at another aspect
              loses its edges rather than gaining bars that read as far depth.
              Clips are trimmed or extended to the selected output length: video
              repeats its last frame and audio pads with silence. The clip is
              rendered in two stages, composed at half the output frame and
              refined at the full one, which reaches twice the frame
              lightricks/ltx-2.5 serves; every reference conditions the
              composing stage.'
      quantization: int8
    upstream:
      base_url: http://127.0.0.1:8100
      model: ltx-2-5-2x
      comfyui_video:
        reference_inputs: true
        unet: LTX-2.5-Distilled-Q8_0.gguf
        clip: gemma4-12b-with-proj-ltx-2.5-comfy-int8-convrot.safetensors
        clip_type: ltxv
        vae: ltx-2.5-video-vae-bf16.safetensors
        audio_vae: ltx-2.5-audio-vae-bf16.safetensors
        control_lora: ltx-2.3-22b-ic-lora-union-control-ref0.5.safetensors
        decode_tile: 768
        generate_audio: true
        width: 2048
        height: 1152
        seconds: 3.88
        fps: 25
        sigmas:
          - 1
          - 0.99375
          - 0.9875
          - 0.98125
          - 0.975
          - 0.909375
          - 0.725
          - 0.421875
          - 0
        cfg: 1
        sampler: euler_ancestral
        codec: h264
        crf: 23
        output_dir: /mnt/render-output/ltx
        two_stage:
          upscaler: ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0.safetensors
          sigmas:
            - 0.85
            - 0.725
            - 0.4219
            - 0
          first_frame_strength: 0.7
          first_frame_compression: 18
    capacity:
      total: 1

An embeddings model#

The output modality is embeddings. The text input carries the per-token prompt price, and the input list also declares images and video clips (base64 video/mp4 or video/webm). Interleaved text-and-image items embed together, and a clip is embedded as a content item. Standalone file parts are not declared because the serving path cannot embed them. No generated-token price is published because this rail returns vectors. Only parameters the serving engine accepts belong in the output's parameter map.

models:
  - doc:
      schema_version: '2.4'
      id: tencent/wemm-embedding-2b
      hugging_face_id: tencent/WeMM-Embedding-2B
      # A declared description replaces the one read from the repository. The repository cannot say how
      # the engine reads an input, so the sentences after its summary state the two bases.
      description: >-
        A 2.7B-parameter multimodal embedding model built on a Qwen3.5
        backbone, reading text, images, short video clips and interleaved text-and-image
        documents, and returning a 2,048-dimension L2-normalized vector. Served
        through the embeddings API; it has no generative path. It has two input
        bases, and their vectors are not interchangeable: a string, a list of
        strings or token ids is embedded as the text it stands for, while a
        content item, { "content": [...] }, a text-only one included, is
        wrapped in the model's chat template with its role turns and the
        <embedding> token. The same short text embeds 0.1189 apart (cosine
        distance) through the two. Embed images and mixed items as content
        items and query them the same way.
      input_modalities:
        - type: text
          supported_inputs:
            max_context_length:
              value: 32768
              unit: token
          pricing:
            - type: prompt
              unit: token
              cost_usd: '0.00000003'
        - type: image
        - type: video
          supported_inputs:
            sources:
              type: enum
              values:
                - base64
            formats:
              type: enum
              values:
                - video/mp4
                - video/webm
      output_modalities:
        - type: embeddings
          supported_parameters:
            encoding_format:
              type: enum
              values:
                - float
                - base64
              default: float
            dimensions:
              type: enum
              values:
                - 64
                - 128
                - 256
                - 512
                - 1024
                - 2048
              default: 2048
      is_ready: true
      datacenters:
        - country_code: CA
      compliance:
        zdr: true
        hipaa: false
      heimdall:
        serving_id: wemm
        data_policy:
          training: false
          retention_days: 0
    upstream:
      base_url: http://127.0.0.1:8101/v1
      model: wemm
    capacity:
      total: 8

The entry declares a description, which replaces the one the marketplace would read from the repository and is the prose the model page shows. A multimodal embeddings engine can read a plain string and a content item through different prompts, which the repository cannot say and a caller cannot see in the response, so the description is where this entry states which input shapes use which basis and that their vectors are not interchangeable.

A rerank model#

Output modality rerank, priced per token like the embeddings entry above. The endpoint accepts text, image and video content, and declares ranking, truncation, activation, cache-salt and template instruction controls. query and documents are required content, not declared parameters, for the same reason no chat entry declares its own required messages. The reply reports the whole batch scored — the query and every document — under usage.total_tokens, a real measured count the gateway bills a rerank call against directly.

models:
  - doc:
      schema_version: '2.4'
      id: qwen/qwen3-vl-reranker-8b
      hugging_face_id: Qwen/Qwen3-VL-Reranker-8B
      input_modalities:
        - type: text
          supported_inputs:
            max_context_length:
              value: 32768
              unit: token
          pricing:
            - type: prompt
              unit: token
              cost_usd: '0.00000002'
        - type: image
        - type: video
      output_modalities:
        - type: rerank
          supported_parameters:
            top_n:
              type: integer
              min: 1
            return_documents:
              type: boolean
              default: false
            instruction:
              type: string
            truncate_prompt_tokens:
              type: integer
              min: 1
            truncation_side:
              type: enum
              values:
                - left
                - right
            max_tokens_per_query:
              type: integer
              min: 0
              default: 0
            max_tokens_per_doc:
              type: integer
              min: 0
              default: 0
            use_activation:
              type: boolean
              default: true
            cache_salt:
              type: string
            chat_template_kwargs:
              type: object
              properties:
                instruction:
                  type: string
                instruct:
                  type: string
      is_ready: true
      datacenters:
        - country_code: CA
      compliance:
        zdr: true
        hipaa: false
      heimdall:
        serving_id: reranker
        data_policy:
          training: false
          retention_days: 0
    upstream:
      base_url: http://127.0.0.1:8102/v1
      model: reranker
    capacity:
      total: 8

Decisions models#

The three Laya checkpoints each run in a pinned laya-serve container on GPU 4. The reranker reserves 0.80 of the 64 GiB card; each Laya process caps its allocator at 3 GiB. A decision request has one encoder row per question. The max_context_length is the state plus one question and its options, and a guard rejects any state that would be truncated. decisions_envelope declares the question types, per-request count and option budget. Each answer uses measured prompt tokens only. Each Laya endpoint declares one concurrent slot because its process runs one forward pass at a time.

English

models:
  - doc:
      schema_version: '2.4'
      id: convaiinnovations/laya
      created: 1789707955
      name: 'Convai: Laya'
      hugging_face_id: convaiinnovations/laya
      quantization: fp16
      input_modalities:
        - type: text
          supported_inputs:
            max_context_length: { value: 512, unit: token }
          pricing:
            - { type: prompt, unit: token, cost_usd: '0.000000042' }
      output_modalities:
        - type: decisions
          supported_parameters: {}
      is_ready: true
      datacenters: [{ country_code: CA }]
      compliance: { zdr: true, hipaa: false }
      heimdall:
        serving_id: laya
        license: apache-2.0
        parameter_count: 421000000
        data_policy: { training: false, retention_days: 0 }
        decisions_envelope:
          question_types: [choice, score, noul]
          max_questions: 64
          option_token_budget: 192
    upstream:
      base_url: http://127.0.0.1:8106/v1
      model: english
    capacity:
      total: 1

multilingual

models:
  - doc:
      schema_version: '2.4'
      id: convaiinnovations/laya-multilingual
      created: 1789788214
      name: 'Convai: Laya Multilingual'
      hugging_face_id: convaiinnovations/laya-multilingual
      quantization: fp16
      description: Answers typed questions across more than 100 languages.
      input_modalities:
        - type: text
          supported_inputs:
            max_context_length: { value: 1024, unit: token }
          pricing:
            - { type: prompt, unit: token, cost_usd: '0.000000042' }
      output_modalities:
        - type: decisions
          supported_parameters: {}
      is_ready: true
      datacenters: [{ country_code: CA }]
      compliance: { zdr: true, hipaa: false }
      heimdall:
        serving_id: laya-multilingual
        license: apache-2.0
        parameter_count: 322000000
        data_policy: { training: false, retention_days: 0 }
        decisions_envelope:
          question_types: [choice, score, noul]
          max_questions: 64
          option_token_budget: 256
    upstream:
      base_url: http://127.0.0.1:8107/v1
      model: multilingual
    capacity:
      total: 1

typed decisions

models:
  - doc:
      schema_version: '2.4'
      id: convaiinnovations/laya-typed-decisions
      created: 1789753514
      name: 'Convai: Laya Typed Decisions'
      hugging_face_id: convaiinnovations/laya-typed-decisions
      quantization: fp16
      input_modalities:
        - type: text
          supported_inputs:
            max_context_length: { value: 1024, unit: token }
          pricing:
            - { type: prompt, unit: token, cost_usd: '0.000000042' }
      output_modalities:
        - type: decisions
          supported_parameters: {}
      is_ready: true
      datacenters: [{ country_code: CA }]
      compliance: { zdr: true, hipaa: false }
      heimdall:
        serving_id: laya-typed-decisions
        license: apache-2.0
        parameter_count: 421000000
        data_policy: { training: false, retention_days: 0 }
        decisions_envelope:
          question_types: [choice, score, noul]
          max_questions: 64
          option_token_budget: 256
    upstream:
      base_url: http://127.0.0.1:8108/v1
      model: typed-decisions
    capacity:
      total: 1

A speech model#

Text in, encoded audio out, and two declarations no other rail carries. The speech output's voice enum and response_format enum are the set of voices and the set of codecs this endpoint answers to, and routing matches a request's voice and response_format against them before sending anything: a codec outside the set is refused naming what is declared, rather than served in a container the caller did not ask for and is billed for. heimdall.speech_envelope states the same facts in the marketplace's spelling and is read in preference where a document declares both. A voice is the one part of the ask the marketplace will stand in for, and only where standing in is not a choice: an endpoint declaring exactly one voice, or declaring one spelled default, is sent that voice when a request names one it never loaded, and the reply says so in X-Heimdall-Voice. Spelling a voice default is therefore how an endpoint that loads several says which of them stands in; without it, a request naming an undeclared voice is refused with a 400 listing what is declared. Its speed range, or min_speed and max_speed in the envelope, gives the playback rates it serves, and it names its generation token limit in supported_parameters. The speech output's completion price is USD per character of submitted text, counted off the request before it is sent. A voice sample takes precedence over a named voice; the provider translates the inline reference and optional transcript into VoxCPM conditioning without creating a saved speaker.

models:
  - doc:
      schema_version: '2.4'
      id: openbmb/voxcpm2
      hugging_face_id: openbmb/VoxCPM2
      input_modalities:
        - type: text
          supported_inputs:
            max_context_length:
              value: 8192
              unit: token
        - type: audio
      output_modalities:
        - type: speech
          streaming: true
          supported_parameters:
            input_references:
              type: array
              max_items: 1
            response_format:
              type: enum
              values:
                - wav
                - mp3
                - opus
                - flac
                - pcm
            voice:
              type: enum
              values:
                - default
            speed:
              type: range
              min: 0.25
              max: 4
              default: 1
            max_new_tokens:
              type: integer
              min: 3
              max: 4096
          pricing:
            - type: completion
              unit: character
              cost_usd: '0.000015'
      is_ready: true
      datacenters:
        - country_code: CA
      compliance:
        zdr: true
        hipaa: false
      heimdall:
        reference_inputs:
          speech:
            description: >-
              One audio sample chooses a custom voice instead of the named
              voice. Samples must total at most 5 MiB. A sample takes precedence
              over voice and does not create a saved speaker profile.
        serving_id: voxcpm
        data_policy:
          training: false
          retention_days: 0
        speech_envelope:
          voices:
            - default
          formats:
            - wav
            - mp3
            - opus
            - flac
            - pcm
          min_speed: 0.25
          max_speed: 4
    upstream:
      base_url: http://127.0.0.1:8103/v1
      speech_reference_format: voxcpm
      model: voxcpm
    capacity:
      total: 4

A transcription model#

Audio is the input and text is the output. The audio input's prompt price is USD per second of audio. The declaration preserves the served 448-token text-output ceiling and the 25 MiB audio file limit its engine enforces. Token prices are omitted because this endpoint charges for audio duration. The transcript token limit can be selected from 1 to 448. This marketplace route returns a buffered transcript, so its listing does not advertise streaming even when the underlying engine can emit chunks. The pinned ASR engine returns JSON for a response_format: text request, so this endpoint only declares json as an honoured response format.

models:
  - doc:
      schema_version: '2.4'
      id: qwen/qwen3-asr-1.7b
      hugging_face_id: Qwen/Qwen3-ASR-1.7B
      input_modalities:
        - type: audio
          supported_inputs:
            max_content_size_bytes:
              value: 26214400
              unit: byte
          pricing:
            - type: prompt
              unit: second
              cost_usd: '0.00002'
      output_modalities:
        - type: text
          max_length:
            value: 448
            unit: token
          streaming: false
          supported_parameters:
            max_completion_tokens:
              type: integer
              min: 1
              max: 448
            response_format:
              type: enum
              values:
                - json
            temperature:
              type: range
              min: 0
              max: 1
              default: 0
            top_p:
              type: range
              exclusive_min: 0
              max: 1
              default: 1
            top_k:
              type: integer
              min: -1
              default: 0
            min_p:
              type: range
              min: 0
              max: 1
              default: 0
            seed:
              type: integer
              min: -9007199254740991
              max: 9007199254740991
            frequency_penalty:
              type: range
              min: -2
              max: 2
              default: 0
            presence_penalty:
              type: range
              min: -2
              max: 2
              default: 0
            repetition_penalty:
              type: range
              exclusive_min: 0
              default: 1
      is_ready: true
      datacenters:
        - country_code: CA
      compliance:
        zdr: true
        hipaa: false
      heimdall:
        serving_id: asr
        data_policy:
          training: false
          retention_days: 0
      quantization: bf16
    upstream:
      base_url: http://127.0.0.1:8104/v1
      model: asr
    capacity:
      total: 8

What the entries have in common#

schema_version: "2.4" identifies the published contract. id names the marketplace model, hugging_face_id names the actual weights, and heimdall.serving_id names the route accepted by the provider when it differs from the marketplace id. Before forwarding inference, the service replaces that name with upstream.model.

Prices are exact decimal strings in USD per declared unit. Tokens, characters, images, video seconds, audio seconds and megapixels describe different meters. Capacity is an inference-slot count, independent of those prices. Set it to the number of simultaneous requests the engine can serve.

Repository metadata cannot establish a deployment's serving limits. MiniCPM5 explicitly declares a 131,072-token input/output window and eight slots. The image and video envelopes describe the geometry the loaded engines accept. The speech envelope names available voices and formats. These remain provider declarations even when the checkpoint's name and other metadata come from Hugging Face.

The published document#

GET /v1/models serves a data array of the declared entries. This is the chat entry with no requests currently in flight. Omitted metadata remains absent until the marketplace fills it; the provider does not need a copy of that metadata to serve inference.

{
  "schema_version": "2.4",
  "id": "openbmb/minicpm5-2b",
  "hugging_face_id": "openbmb/MiniCPM5-2B",
  "quantization": "bf16",
  "input_modalities": [
    {
      "type": "text",
      "supported_inputs": {
        "max_context_length": {
          "value": 131072,
          "unit": "token"
        }
      },
      "pricing": [
        {
          "type": "prompt",
          "unit": "token",
          "cost_usd": "0.00000003"
        },
        {
          "type": "cached_prompt",
          "unit": "token",
          "cost_usd": "0.00000001"
        }
      ]
    }
  ],
  "output_modalities": [
    {
      "type": "text",
      "max_length": {
        "value": 131072,
        "unit": "token"
      },
      "streaming": true,
      "supported_parameters": {
        "temperature": {
          "type": "range",
          "min": 0,
          "max": 2,
          "default": 1
        },
        "top_p": {
          "type": "range",
          "exclusive_min": 0,
          "max": 1,
          "default": 0.95
        },
        "top_k": {
          "type": "integer",
          "min": -1,
          "max": 200,
          "default": -1
        },
        "min_p": {
          "type": "range",
          "min": 0,
          "max": 1,
          "default": 0
        },
        "frequency_penalty": {
          "type": "range",
          "min": -2,
          "max": 2,
          "default": 0
        },
        "presence_penalty": {
          "type": "range",
          "min": -2,
          "max": 2,
          "default": 0
        },
        "repetition_penalty": {
          "type": "range",
          "exclusive_min": 0,
          "max": 2,
          "default": 1
        },
        "stop": {
          "type": "one_of",
          "options": [{ "type": "string" }, { "type": "array", "items": { "type": "string" } }]
        },
        "stop_token_ids": {
          "type": "array",
          "items": {
            "type": "integer",
            "min": 0
          }
        },
        "stop_regex": {
          "type": "one_of",
          "options": [{ "type": "string" }, { "type": "array", "items": { "type": "string" } }]
        },
        "no_stop_trim": {
          "type": "boolean",
          "default": false
        },
        "ignore_eos": {
          "type": "boolean",
          "default": false
        },
        "skip_special_tokens": {
          "type": "boolean",
          "default": true
        },
        "max_tokens": {
          "type": "integer",
          "min": 1,
          "max": 131072,
          "unit": "token"
        },
        "max_completion_tokens": {
          "type": "integer",
          "min": 1,
          "max": 131072,
          "unit": "token"
        },
        "logit_bias": {
          "type": "unknown"
        },
        "n": {
          "type": "integer",
          "min": 1
        },
        "response_format": {
          "type": "unknown"
        },
        "structured_outputs": {
          "type": "boolean"
        },
        "logprobs": {
          "type": "boolean"
        },
        "top_logprobs": {
          "type": "integer",
          "min": 0,
          "max": 20
        },
        "tools": {
          "type": "array",
          "items": {
            "type": "unknown"
          }
        },
        "tool_choice": {
          "type": "unknown"
        },
        "parallel_tool_calls": {
          "type": "boolean",
          "default": true
        },
        "reasoning_effort": {
          "type": "enum",
          "values": ["none", "medium"]
        }
      },
      "pricing": [
        {
          "type": "completion",
          "unit": "token",
          "cost_usd": "0.0000001"
        }
      ]
    }
  ],
  "is_ready": true,
  "datacenters": [
    {
      "country_code": "CA"
    }
  ],
  "compliance": {
    "zdr": true,
    "hipaa": false
  },
  "heimdall": {
    "serving_id": "minicpm5",
    "data_policy": {
      "training": false,
      "retention_days": 0
    },
    "live_capacity": {
      "current": 0,
      "total": 8
    }
  }
}

Use the provider dashboard's pull result to confirm which fields were filled, which facts remain unknown and whether the model was admitted. A repository outage is not a reason to substitute a plausible name, context window or price.

Running it#

Each engine runs bound to loopback; the service is the only thing exposed, behind TLS:

docker build -f deploy/Dockerfile.provider-example -t heimdall-provider-example .
docker run -d --restart unless-stopped --network host \
  --log-driver=none \
  -e MODELS_CONFIG=/etc/provider/models.yaml \
  -e PROVIDER_API_KEY=<the credential you register with Heimdallm> \
  -v /etc/provider:/etc/provider:ro \
  heimdall-provider-example

--log-driver=none because these endpoints declare heimdall.data_policy.retention_days: 0. The service never logs a request body, since it sets disableRequestLogging, but the container's stdout would otherwise land in Docker's default json-file driver, and a file of your service's output is a log store whatever happens to be in it. If you publish a retention promise, make the runtime keep it. The cost is real: a container that dies on startup will tell you only that it died. The renderers take the same setting in Compose, as logging: { driver: none }.

One file can describe every model you have an engine for while a host runs some of them. MODELS_SERVED names the ones this host runs, as a comma-separated list of the names requests are routed on: each entry's heimdall.serving_id, or its marketplace id where none is declared. The service validates the whole file and lists that subset; Heimdallm withdraws the rest on its next pull. A name the file does not hold stops the service from booting, and unset, every model in the file is served. The reference provider names all eleven of its own serving ids this way: minicpm5,hidream,ltx-2-5,ltx-2-5-2x,wemm,reranker,voxcpm,asr,laya,laya-multilingual,laya-typed-decisions.

Check it answers the way Heimdallm will call it:

curl -H "Authorization: Bearer $PROVIDER_API_KEY" https://your-origin.example/v1/models

More than one box#

One base URL is one source, and a source is what routing sees. An endpoint is a provider, a model id and a quantization, with no room in it for a machine. Several replicas behind that URL are therefore the supported way to grow, and they are invisible to the marketplace by design: put a balancer on the origin, give every replica the same engines, and the listing does not change.

What does change is what the capacity report has to mean, and what you have to build yourself.

What capacity reports once there are replicas#

The report's total — capacity.total in the configuration, heimdall.live_capacity.total as published — is the concurrent request slots the source can serve for the model across every replica, and current the slots in use across every replica. Four replicas of an engine started with --max-num-seqs 4 report total: 16. Each replica answering with its own total: 4 would under-report the fleet by a factor of its size, and cost you three quarters of the traffic you can serve.

The reference service counts requests in flight in its own process, which is the right counter for one box and the wrong one for four. Behind replicas the count has to come from wherever the fleet is already aggregated — the balancer's own active-connection gauge, or a counter the replicas share — and be published by whatever answers /v1/models.

The poll that reads it is its own: separate from the full pull, every fifteen seconds by default, with a ten-second timeout, and it takes nothing from the document but heimdall.live_capacity. A report older than a minute counts as no report at all, so a /v1/models that goes slow or quiet decays to unknown rather than steering routing on stale numbers. The field itself is documented with the rest of the schema in Listing your models, and the honest-implementation rule in Operating a source.

An honest total earns traffic#

Routing partitions eligible endpoints by what they report before it sorts them by price or latency: known room first, then no report, then the last reserved quarter, then known-full. It orders rather than excludes, so a request with no better option still reaches a saturated endpoint. Both directions of a dishonest number cost you:

• Under-reporting hands requests to endpoints with less to give while your cards idle. An endpoint reporting room is tried ahead of every endpoint reporting nothing, whatever the sort in effect, so the report is worth more than a small price advantage.
• Over-reporting wins the request and then queues it upstream. That wait is on the customer's clock and inside your measured time to first token and throughput, which are published beside your price.
• total: 0 declares no slots, which is not a measurement of occupancy: nothing can be in use out of nothing. It reads as no report at all and the endpoint is ordered with those that report nothing rather than behind those that are full. The pull records a warning against it.

Two mechanics are worth designing a fleet around:

• A quarter of the slots is held back. An endpoint whose free slots are down to the last quarter is treated as reserved and ordered below endpoints with room to spare. The share is floored, so an endpoint declaring one or two slots reserves nothing. The reserve keeps a lane open for requests that have nowhere else to go, and one with no alternative still takes it.
• A report is adjusted for what arrived after it. Between polls Heimdallm adds the requests it started against the endpoint since the report landed, so a fifteen-second-old current is not read as fifteen seconds of idleness. Reporting promptly still pays, because the estimate errs towards more load rather than less.

Where the marketplace stops and your balancer starts#

Which replica serves a request is yours to decide, and Heimdallm hands you nothing to decide it with. What reaches your origin is your credential, the model, and the caller's body. There is no machine hint, no conversation id and no routing header.

Session affinity is the closest thing, and it stops one level above the replica. X-Session-Id pins a conversation to an endpoint for an hour past its last request, and an endpoint is your whole source. It brings a conversation back to you; it does not bring it back to the box whose KV cache holds the prefix, and the header is not forwarded.

So KV-cache-aware routing across your replicas is your own load balancer's job, and the marketplace has nothing to say about how you do it. What your balancer has to work with is the request itself, which is what a prefix-aware balancer wants anyway: the prompt arrives in the body, and a hash of its leading messages is the ordinary key. The one stable per-caller token on the wire is user, an opaque end-user identifier forwarded to you as sent — it is not a conversation id, and it is absent unless the caller chose to send one.

The boundary is worth stating plainly. Heimdallm routes to a source and measures what the source did. Everything inside the source is the provider's: replicas, cache placement, queueing, autoscaling and which card answers. The one lever a fleet has on inbound load is the capacity report, which is why it is worth wiring to something real.

Two consequences follow for how a fleet is registered:

• Replicas in more than one country are more than one listing. datacenters is a claim about where inference runs and it is published per endpoint, so a single entry cannot describe replicas on two continents. Register a source per region, each with its own base URL.
• Different hardware behind one entry averages into one measurement. Uptime, time to first token and throughput are measured per endpoint, so a slow replica and a fast one publish as one figure. Splitting them into two sources is what makes either legible.

Going live#

In the provider dashboard, add a source with base URL https://your-origin.example/v1 and the same credential. Heimdallm pulls <base URL>/models, each entry becomes a pending_health endpoint, and probes call it on the rail its modalities name: the chat model with a chat probe, the embeddings model with an embeddings probe, the rerank model with a rerank probe, the transcription model with a transcription probe, the speech model with one fixed sentence to say, and the two renderers with a render. After two successful probes out of three each endpoint goes live on the marketplace.

From then on the loop is: edit the YAML, restart the service, and the next pull applies it. A source pulls every sixty seconds by default, and "Pull now" in the dashboard applies a change at once, at most once per source per minute.

3D asset generation#

tencentarc/pixal3d accepts up to three views of one object in front, left, back order. microsoft/trellis-2 accepts one view. Both return textured GLB assets; a text-only request first renders an image using the configured HiDream stage. The default target is 20,000 faces. Textured assets cost $0.04 per mesh and untextured assets cost $0.025 per mesh. The face budget limits geometry, independently of texture resolution. Lower budgets can lose silhouette detail; inspect the exported mesh and texture together.

mesh is a Heimdallm extension to the 2.4 modality and pricing enums. These entries must be omitted when publishing to a consumer that only supports the upstream schema.

models:
  - doc:
      schema_version: '2.4'
      id: tencentarc/pixal3d
      created: 1777544329
      name: 'TencentARC: Pixal3D'
      description: 'One photograph of an object becomes a textured GLB mesh, or up to
        three views taken ninety degrees apart on one orbit and sent in front,
        left, back order. A prompt with no view is rendered into a view first
        and the mesh is built from that. `target_faces` sets the face budget,
        from 2,000 to 500,000, and a request naming none is decimated to 20,000.
        The budget is a maximum, not a promised face count or quality score.
        Lower budgets can lose silhouette detail; textures remain available.
        Use `generate_texture: false` when only geometry is needed.'
      hugging_face_id: TencentARC/Pixal3D
      input_modalities:
        - type: text
          supported_inputs:
            max_context_length:
              value: 8192
              unit: token
        - type: image
          supported_inputs:
            sources:
              type: enum
              values:
                - base64
            formats:
              type: enum
              values:
                - image/png
                - image/jpeg
                - image/webp
      output_modalities:
        - type: mesh
          supported_parameters:
            input_references:
              type: array
              max_items: 3
            target_faces:
              type: integer
              min: 2000
              max: 500000
              default: 20000
            voxel_resolution:
              type: enum
              values: [1024, 1536]
              default: 1536
            texture_size:
              type: enum
              values: [1024, 2048, 4096]
              default: 4096
            camera_fov:
              type: range
              min: 1
              max: 170
            negative_prompt:
              type: unknown
            generate_texture:
              type: boolean
              default: true
            steps:
              type: integer
              min: 8
              max: 50
              default: 20
            guidance_scale:
              type: range
              min: 0
              max: 20
              default: 7.5
            seed:
              type: integer
              min: 0
              max: 2147483647
          pricing:
            - type: completion
              unit: mesh
              cost_usd: '0.04'
              overrides:
                - when:
                    generate_texture:
                      equals: false
                  cost_usd: '0.025'
      is_ready: true
      datacenters:
        - country_code: CA
      compliance:
        zdr: true
        hipaa: false
      heimdall:
        reference_inputs:
          mesh:
            roles:
              image:
                reference: 3
            description: One view of the object, or up to three taken ninety degrees apart
              on one orbit and sent in front, left, back order. Each view shows
              the object on a plain background at the same scale, and the first
              one sent is the front the mesh is posed to. A prompt with no view
              is rendered by the text-to-image stage first. A prompt beside a
              view is refused because the mesh model cannot read both.
        serving_id: pixal3d
        data_policy:
          training: false
          retention_days: 0
        mesh_envelope:
          max_faces: 500000
          min_faces: 2000
          texture: true
      quantization: int8
    upstream:
      base_url: http://127.0.0.1:8105
      model: pixal3d
      comfyui_mesh:
        family: pixal3d
        diffusion_model: pixal3d_int8_convrot.safetensors
        multiview_model: pixal3d_multiview_int8_convrot.safetensors
        shape_vae: trellis_2_shape_vae_bf16.safetensors
        texture_vae: trellis_2_texture_vae_bf16.safetensors
        clip_vision: dino_v3_L_naf_fp32.safetensors
        background_removal: birefnet.safetensors
        text_to_image: hidream
        output_dir: /mnt/render-output/mesh
    capacity:
      total: 1

models:
  - doc:
      schema_version: '2.4'
      id: microsoft/trellis-2
      created: 1764555940
      name: 'Microsoft: TRELLIS.2'
      description: 'One photograph of an object becomes a textured GLB mesh. The
        encoder reads a single view and places no second camera, so one view is
        what this entry takes; a prompt with no view is rendered into a view
        first and the mesh is built from that. `target_faces` sets the face
        budget, from 2,000 to 500,000, and a request naming none is decimated to
        20,000. The budget is a maximum, not a promised face count or quality score.
        Lower budgets can lose silhouette detail; textures remain available.
        Use `generate_texture: false` when only geometry is needed.'
      hugging_face_id: microsoft/TRELLIS.2-4B
      input_modalities:
        - type: text
          supported_inputs:
            max_context_length:
              value: 8192
              unit: token
        - type: image
          supported_inputs:
            sources:
              type: enum
              values:
                - base64
            formats:
              type: enum
              values:
                - image/png
                - image/jpeg
                - image/webp
      output_modalities:
        - type: mesh
          supported_parameters:
            input_references:
              type: array
              max_items: 1
            target_faces:
              type: integer
              min: 2000
              max: 500000
              default: 20000
            voxel_resolution:
              type: enum
              values: [1024, 1536]
              default: 1536
            texture_size:
              type: enum
              values: [1024, 2048, 4096]
              default: 4096
            negative_prompt:
              type: unknown
            generate_texture:
              type: boolean
              default: true
            steps:
              type: integer
              min: 8
              max: 50
              default: 20
            guidance_scale:
              type: range
              min: 0
              max: 20
              default: 7.5
            seed:
              type: integer
              min: 0
              max: 2147483647
          pricing:
            - type: completion
              unit: mesh
              cost_usd: '0.04'
              overrides:
                - when:
                    generate_texture:
                      equals: false
                  cost_usd: '0.025'
      is_ready: true
      datacenters:
        - country_code: CA
      compliance:
        zdr: true
        hipaa: false
      heimdall:
        reference_inputs:
          mesh:
            roles:
              image:
                reference: 1
            description: One view of the object on a plain background. A prompt with no view
              is rendered by the text-to-image stage first. A prompt beside a
              view is refused because the mesh model cannot read both.
        serving_id: trellis-2
        data_policy:
          training: false
          retention_days: 0
        mesh_envelope:
          max_faces: 500000
          min_faces: 2000
          texture: true
      quantization: int8
    upstream:
      base_url: http://127.0.0.1:8105
      model: trellis-2
      comfyui_mesh:
        family: trellis2
        diffusion_model: trellis_2_int8_convrot.safetensors
        shape_vae: trellis_2_shape_vae_bf16.safetensors
        texture_vae: trellis_2_texture_vae_bf16.safetensors
        clip_vision: dino_v3_L_naf_fp32.safetensors
        background_removal: birefnet.safetensors
        text_to_image: hidream
        output_dir: /mnt/render-output/mesh
    capacity:
      total: 1