master
1205 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 budiony
|
d81d0d8fb9 |
Multi-column on Intel XMX matrix engines refactored (#29864)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* Multi-column on Intel XMX matrix engines refactored Multi-column (2 to 80 activation columns) mul_mat_vec_q for reordered K-quant and Q8_0 weights on the Intel XMX matrix engines (int8 DPAS, ESIMD), for the SYCL backend. It targets speculative decoding verification (2 to 8 tokens per step) and short prompt batches (9 to 80 tokens). Three types also use it for a single column (Q2_K, Q3_K, Q8_0). * Fixed regression bug real bug found: the first version produced NaN whenever the row count wasn't a multiple of 4 * Removed GGML_TYPE_Q8_0 activation * rewritten xmx_mul_mat 64-byte alignment and 16-byte pitch check on the activations * Implemented fused gate/up kernel One launch computes up * act(gate) for both FFN weights, reading the activations once and applying SWIGLU or GEGLU in the epilogue. It covers up to 16 columns and works for all six XMX types. * Removed the temporary XMX_NOGLU environment switch * Q2_K min-sum optimization Q2_K weight is d·scl·q − dmin·mc, where scl (the 4-bit scale) and mc (the 4-bit min code) apply to each half of a 32-weight group * Added dispatch check and GRF sweep tg128 on Q4_K_M goes from 26.67 to 28.23 tok/s on the B70 (+5.9%). Q6_K is unchanged. MUL_MAT passes 1727/1727 and MUL_MAT_VEC_FUSION 1361/1361. Perplexity at batch 1 is 5.2064 with XMX and 5.2058 without. * Renamed variables * Added 2 environment variables GGML_SYCL_XMX_MAX_COLS (default 80, any value 0 or higher) and GGML_SYCL_XMX_GLU_MAX_COLS (default 16, range 0 to 16): the largest batch for the fused gate/up kernel. It is clamped to 16 because that is the most the fused kernel is compiled for, so the variable can only lower it. * Update esimd.hpp 1. All five quant structs were repeating the same boilerplate, so I moved it into shared pieces. 2. A new helper xmx_term does one DPAS, converts it to float and scales it per row. The four branches are now one or two lines each, and the shared Cb += z * d8r is written once. The refactor passes all three op tests. MUL_MAT_VEC_FUSION is 1369/1369, MUL_MAT is 2281/2281 (the count includes your uncommitted f32 tests) and MUL_MAT_ID is 937/937. |
||
|
tkran2
|
d88040c704 |
vocab : add clean_spaces GGUF key (#30940)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
|
||
|
alexey-rsi
|
a37cc46104 |
models : add Flint model (FlintForCausalLM) (#30928)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* Add Flint model (FlintForCausalLM). Assisted by Codex * cleanup * one more --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> |
||
|
Tarek Dakhran
|
99fb997184 |
jinja : fix namespace attributes shadowed by dict methods (#30941)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
|
||
|
Maphist0
|
7775ffb7e4 |
model: Add support for SenseNova U1 model (#28919)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* model : add SenseNova U1 understanding support Add support for the dense SenseNova U1 understanding branch, including the U1.5-8B-MoT model. - Add HF-to-GGUF conversion for NEOChatModel, excluding MoT generation tensors and concatenating temporal and spatial Q/K normalization weights. - Register the sensenova_u1 GGUF architecture and custom spatial RoPE metadata. - Implement temporal/spatial Q/K normalization with 4-axis RoPE. - Keep U1-specific text position handling in the model implementation. - Add synthetic architecture coverage for sensenova_u1. Only the dense understanding branch is supported; generation branches are intentionally rejected during conversion. * conversion : treat one-expert SenseNova U1 configs as dense * mtmd : add SenseNova U1 vision support Add conversion support for the SenseNova U1 vision encoder and implement its CPU mtmd projector graph. The graph applies patch embedding, GELU, adjacent-pair 2D RoPE, spatial downsampling, and dense projection for dynamic-resolution image inputs. * mtmd : fix SenseNova U1 vision token layout * model : clean up SenseNova U1 RoPE metadata Replace the hard-coded spatial RoPE key with shared GGUF and LLM_KV definitions used by conversion, model loading, and architecture tests. Clarify why SenseNova U1 retains its custom text-position input: U1 uses [t, 0, 0, 0], while the generic M-RoPE input uses [t, t, t, 0]. * mtmd : simplify SenseNova U1 projector integration Map the SenseNova U1 dense projection to the standard mm.0 weight and bias tensors and reuse the shared projector fields in the loader and graph. Read vision parameters from the checkpoint configuration and remove the U1-specific tensor names and model fields while preserving its strided projection, bias, and row-major token layout. * mtmd : fix SenseNova U1 image positions and attention * fix spacing * mtmd : restore generic test input chunks * mtmd : restore SenseNova U1 integration after upstream merge * Update conversion/sensenova_u1.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Update src/models/sensenova-u1.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * model : remove SenseNova U1 BF16 activation casts Use standard backend precision handling for QKV, FFN, and output projections instead of casting inputs based on the output tensor type. Assisted-by: Codex * Update src/models/sensenova-u1.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> |
||
|
Aman Gupta
|
d454a204ef |
llama: create new fused op for QSA mask (#30640)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* llama: create new fused op for QSA scoring * enroll glm5.3-next too * tests : initialize the remaining tensors in test_kpool_mask The harness sentinels were left uninitialized, so leftover memory could read as NaN and fail the NaN check. * ggml: generalize the k-pool mask op into ggml_top_k_mask The k-pool mask op only covered the pooled QSA selection of qwen4exp and glm5-next, while the DSA models and the DeepSeek V4 CSA build the same top-k mask by hand with fill, set_rows and add. ggml_top_k_mask(mask, top_k, score, blocks, extra) keeps the mask cells of the live picks and the extra cells, and sets the rest to -inf. Every row tensor is 4D in the layout of top_k, and the mask is broadcast to those rows, so a selection can also be made per head. score, blocks and extra are optional: without score every pick is live, without blocks a pick is a cell, and extra holds the cells a row keeps without a pick. llm_graph_context::build_fused_top_k_mask views the row tensors in the layout of kq_mask and registers the fused node. The k-pool models, the DSA attention of build_attn and the DeepSeek V4 CSA mask use it when the probe enables the op, and keep their current path on backends without it. --------- Co-authored-by: Pascal <admin@serveurperso.com> |
||
|
alexey-rsi
|
9febe63d5a |
jinja: support hexadecimal string escapes (#30929)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
|
||
|
bosh
|
479a52779f |
ggml-cpu : compute f32 gelu_quick and geglu_quick without the fp16 table (#30737)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* ggml-cpu : compute f32 gelu_quick and geglu_quick without the fp16 table * ggml-cpu : keep the GGML_GELU_QUICK_FP16 table path * ggml-cpu : add the VXE gelu_quick path * tests : add test-gelu-quick for the f32 CPU kernels * tests : drop inf/nan inputs from test-gelu-quick |
||
|
Anant Shrivastava
|
493253fe5a |
Sycl iq3 crash fix (#30604)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
|
||
|
Yuri Khrustalev
|
4a45de4f2c |
mtmd : lfm2 tiles and layout match transformers (#30261)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* mtmd : lfm2 tiles stretch the image to the grid Lfm2VlImageProcessor.crop_image_to_patches resizes the image straight to (512 * rows, 512 * cols) before it cuts the tiles. llama.cpp kept the aspect ratio and added black bars for LFM2 (only d1-omni stretched), so the tiles of d1-3B and every LFM2-VL / LFM2.5-VL model differed from transformers unless the image had the aspect ratio of the grid. With PAD_NONE the tiles are the same as in transformers. * mtmd : lfm2 layout with Python rounding in double precision Lfm2VlImageProcessor.smart_resize rounds with Python's round (half to even) and computes beta and the tile grid aspect ratio in double. llama.cpp used std::round and float, so 31926 of the 16.7M sizes in [1,4096]^2 got another thumbnail size or grid than transformers (and d1-omni vision.py), e.g. 400x600 -> 416x608 instead of 384x608 and 720x1600 -> 2x5 tiles instead of 2x4. The lfm2 smart resize and grid choice now follow the Python code and match it on every size of that range. * fix --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> |
||
|
Tarek Dakhran
|
39da130fcd |
ggml-cuda : support ssm_conv kernel size 7 (#30873)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
|
||
|
Tarek Dakhran
|
36140dbee2 |
mtmd : read LFM2 tiling params from GGUF metadata (#30840)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* mtmd : read LFM2 tiling params from GGUF metadata Write min_tiles, max_tiles, tile_size and max_pixels_tolerance from processor_config.json when converting LFM2-VL mmproj, read them in clip.cpp with the previous constants as defaults. do_image_splitting=false maps to min_tiles=max_tiles=1 (no tiling). Rebased version of #25524, original implementation by Michael DM (@michael-dm). Co-authored-by: Michael DM <usable_siren180@simplelogin.com> * fix test --------- Co-authored-by: Michael DM <usable_siren180@simplelogin.com> |
||
|
Georgi Gerganov
|
df11702136 |
metal : optimize argmax and pool_1d (#30862)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* metal : optimize argmax and pool_1d Add a two-pass partial reduction for large-row argmax and use cooperative threadgroup reductions for pool_1d. Allocate scratch space for the argmax partials through the Metal buffer allocator. Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB) * tests : add pool_1d perf cases Add representative POOL_1D max cases for the MoE down-projection amax shapes, reducing the whole intermediate dim. Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB) * cont : minor fix |
||
|
Georgi Gerganov
|
7879491cf4 |
tests : make MoE router selection independent of the backend noise (#30818)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* tests : make MoE router selection independent of the backend noise
The router scores are in the range [0, 1), so a random draw can leave the
last expert (or expert group) still selected and the first one dropped tied
within the noise between the backends. They then route a token to different
experts and the logits differ by 4e-3 instead of the usual 5e-7.
Beam is the only arch here with a real top-k (8 experts, 2 used) and a group
top-k (4 groups, 2 used), so it is the only one that hits this: 2 of 20 seeds
failed on an M2 Ultra, the other MoE archs select 2 of 2 and cannot tie.
Fill the selection bias with a ramp of step 2 to give every expert a score
range of its own, which turns the top-k into a function of the expert index
alone. The bias affects only the selection, the weights still come from the
unbiased probs, so the MoE output keeps its magnitude. Beam now matches the
CPU reference within 4e-7 for all of 160 seeds.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
* tests : tell the CPU and the GPU of an Apple SoC apart in the device table
Both devices report the same description ("Apple M2 Ultra" here), so the
device column of the test table had two indistinguishable rows and a failing
row did not say whether the Metal device or the host CPU produced it.
Add the device name, which is also what -b takes, unless the description
already is the name.
Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB)
|
||
|
Sam Murdoch
|
3bb63cbcbf |
sycl : fix get_rows on reordered quantized weights (#30430)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
Co-authored-by: Sam <sam@garagellm.tail7dd858.ts.net> |
||
|
bosh
|
f711c8f303 |
mtmd : match the transformers Qwen-VL resize (#30736)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
|
||
|
Aldehir Rojas
|
0ce8cd1141 |
jinja : implement unique filter (#30819)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* jinja : implement `unique` filter * cont : inline key_of * cont : fix alignment |
||
|
Aaron Teo
|
fda00f18cd |
tests: add big-endian conversion tests (#30705)
* ci: add quantize to big-endian tests Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * ci: remove incoherent data types from testing Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * tests: add verified hashes Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * tests: rename Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * tests: add result table Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * tests: swap hash and result column, add colours Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * tests: switch to perplexity test Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * tests: update output table Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> --------- Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> |
||
|
Georgi Gerganov
|
27af58168a |
tests : update Metal fusion baseline for beam (#30764)
* tests: add beam fusion rows to MTL baseline Regenerate tests/fusion/MTL.csv with test-fusion --record. The newly added beam architecture has no baseline rows, so test-fusion fails with 2 failing rows (MUL+ADD x6, RMS_NORM+MUL x25). All other arch counts are unchanged. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * skills: add update-fusion-table skill Add a skill for regenerating a per-device test-fusion baseline. It keeps the workflow from tests/fusion/README.md and the parts that are easy to get wrong: diff the recorded file before applying it, the distinct meanings of a FAIL row, and never recording a partial run. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) |
||
|
Andrea Rossetti
|
0878f7c304 |
ggml: support non-contiguous SUM (CPU, CUDA, Vulkan, WebGPU) (#28851)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* ggml: support non-contiguous SUM (CPU, CUDA) * tests: add non-contiguous SUM cases * tests: SUM over non-contiguous views * vulkan: support non-contiguous SUM * webgpu: support non-contiguous SUM |
||
|
Todd Malsbary
|
547208dca2 |
Restore ESIMD kernels on FFN path (#30239)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* Reenable ESIMD on the FFN path Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * Extend ESIMD fusion on FFN path to include Q8_0 Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * Remove comment explicitly referencing ESIMD in test-backend-ops Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> --------- Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> |
||
|
Aleksander Grygier
|
c0fd786264 |
ui : Discover & Download Models from HF Hub (#29584)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* common : resolve <quant>-<sidecar> download tags and list cached sidecars A Q4_0-mtp style tag now resolves the sidecar file when no model file matches it, so a solo draft or mmproj download actually pulls the file. Cached sidecar files list as their own entries so the state survives a restart, and removing such a tag deletes only the sidecar. Assisted-by: pi:zai-org/GLM-5.3 * ui : optional sanitized raw HTML in markdown Add an allowHtml prop to MarkdownContent: raw HTML found in the markdown is rendered after DOMPurify sanitization instead of being escaped as literal text. Default stays escaped. Assisted-by: pi:GLM-5.3-Flash * ui : add the discover models components The Discover surface: the curated-catalog list with its search and skeletons, the details pane with its readme, metadata, Hub stats and download options, and the standalone download progress bar that replaces the plain one. The quant download button sizes its select with a new xs trigger, so the select gains that size and the muted box look. The store stays incomplete when a repo fails, so the next mount retries it. Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash * ui : put the discover view in the models manager The manager dialog gains its second view: the Discover Models call to action fades the table out, slides the title with its compass mark, and fades the discover surface in; the arrow button returns to the table. Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash # Conflicts: # tools/ui/src/lib/components/ui/button/button.svelte * ui : add the Enable Discover Models setting The flag gates browsing and downloading HuggingFace GGUF models, so it is declared on the discover layer; the repo-tree lookup in the manager layer above reads it from here. Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash * ui : resume model downloads from the pause point A stopped download keeps its partial file instead of deleting it, so re-posting the tag resumes from disk. The first progress record now carries the resume offset, the paused snapshot survives into the resumed entry, and a pause settles as soon as it is requested. Assisted-by: pi * ui : lead the discover list with downloads The discover sidebar lists running downloads above the suggested models, and steps aside while a query is active. A paused download stays in the manager table, where it is resumed or dropped. Assisted-by: pi * ui : make resuming a paused download work Re-posting the tag of a paused download hit the router's "already exists" check, because a paused download parks its entry as DOWNLOADED until the next reload: a download entry that is not running is now replaced and resumed. The UI keeps the paused row when the request still fails, instead of dropping it and looking finished. Assisted-by: pi * ui : remember unfinished model downloads A partial file is named after its content hash, so nothing on disk says which model it belongs to once the router restarts. The router now keeps the tags of the downloads it started, clears a tag when the download finishes or is deleted, and lists what is left as still downloading without starting anything. The UI adopts those tags as paused on every list fetch, so a refresh or a fresh open shows an incomplete download and can resume it from the files already on disk. Assisted-by: pi * ui : show a paused download's state on its chip A paused quant chip reads as paused at rest, the way a running one shows its spinner, and offers the resume action on hover. The selector's download row gets a View in Discover action beside its cancel. Assisted-by: pi * ui : collapse the download options by default The copyable download command starts collapsed so the option rows lead; expanding it stays one click away. Assisted-by: pi * ui : keep decision models out of the hub lists Decision models answer with calibrated probabilities instead of text, so llama-ui cannot run them: the hub data layer drops them from the catalog and every search, keyed by the decision-model tag or a text-classification pipeline. Assisted-by: pi * ui : read download tags as quant and sidecar A `<repo>:<quant>-<sidecar>` tag now parses into its quant and sidecar instead of one opaque quant token, and ModelsService.draftSidecarsByRepo collects the router listing's sidecar-only entries per repo, replacing the inline copy in the models store. Assisted-by: pi * ui : cluster draft downloads under their main model Tracked downloads group by repo: the main weights lead and the draft sidecars ride inside their item as compact sub-lines, on the selector, the discover list and the manager table. The main model's controls pause, resume and cancel the whole cluster, and the manager's download rows select into the configuration pane. Assisted-by: pi * ui : act on tracked downloads from the table rows The table passes its clusters' draft sub-lines to the model rows. A row's actions cascade pause and resume, an in-flight row offers Cancel download while a paused one keeps Delete from disk, and the confirm dialog's copy follows the state. Assisted-by: pi * ui : give downloads a configuration pane of their own Selecting a tracked download opens the pane in a download mode: live progress with pause, resume and cancel, and See in Discover, instead of metadata and load actions that a model without files cannot offer. Assisted-by: pi * ui : make the discover details self-fetching and responsive ModelsDiscoverDetailsContainer owns the details, tree and readme fetches so hosts outside the discover view can render the pane, and the details layout adapts to narrow panes with container queries instead of viewport breakpoints. Assisted-by: pi * ui : download a matching draft with the main model A main quant download now pulls the quant-matched draft of the most preferred kind the repo ships (DFlash before MTP), fire-and-forget, so speculative decoding is ready when the main lands. Manual draft chips stay available; the options row also collapses to one column in narrow panes. Assisted-by: pi * ui : keep discover off by default and couple it with the hub setting Discover Models starts unchecked. Enabling it turns the Hugging Face Hub setting on and disabling it turns it back off, so the checkbox never lies; with discover off the hub setting stays available on its own for the display data. Stored and reset state are corrected the same way on load. Assisted-by: pi * ui : open the manage models dialog on the manager view The dialog now opens on the manager instead of returning to the last view, discover entries still land on the discover view while the setting allows it, and the discover surfaces follow the setting's state. Assisted-by: pi * ui : give the discover search room for its focus ring The sticky search bar insets from the panel edges so the focus ring stays inside the scroll container, and rows scroll behind an opaque bar instead of through it. Assisted-by: pi * common: settle sidecar names on the listing and drop unfinished partials An infix or trailing sidecar token is ambiguous: draft heads use it (Model-MTP-BF16.gguf), so do full models with embedded heads (Model-Q8D-MTP.gguf). Such a name is a sidecar only next to token-free main weights of the same family, while mmproj or imatrix anywhere in the name, or a leading token, decides alone (mmproj_f16.gguf). A quant tag always carries its bit count, so a sidecar named after no quant (mtp-Model-it.gguf) gets the bare tag: the listing, the plan and remove all use it. An imatrix tag resolves into a plan slot of its own, downloaded but never wired into the params. remove resolves the tag the way a download does and deletes the partial files of every planned blob, so cancelling a running download or dropping one left unfinished by an earlier run leaves nothing on disk. The router passes its token and releases its lock around the call, and records the unfinished downloads with an atomic write. llama download succeeds on a sidecar-only plan and prints its paths. * ui : move dialog views and draft states into enums and constants The models dialog's views become the ModelsDialogView enum, the draft subline's states join ModelRowDownloadState, the view fade and the copy-feedback reset move to ui constants, and the dialog's compass marks come from DISCOVER_ICON. Assisted-by: pi * ui : drop the unused discover surface The dead openDiscoverModels twin, the commented-out command divider, the details pane's unused onClose and its header's close button, and a redundant type cast leave; the command's copy feedback joins the shared reset constant. Assisted-by: pi * ui : trim the download option comments and highlight the chat template The comment verbosity the options panel and its chips carried shrinks to the non-obvious why, and the chat template dialog renders through SyntaxHighlightedCode as jinja instead of a plain pre block. Assisted-by: pi * common: only a .gguf file is a sidecar, and mmproj/imatrix as whole words A non-GGUF file named like a sidecar (quantization-mmproj-Q8_0.json) is picked as the mmproj, and a sidecar word inside a longer word of a model name (Phi-2-OmniMatrix.i1-Q4_K_M.gguf) drops the model. * ui: dedupe draft sidecar badges per quant The draft sidecar badges deduped by kind and repo without the quant, so with a draft cached at two quants only one badge survived, and a delete of the other main quant found no badge to remove its sidecar with. Also restores the getDownloadEntries doc lost when the clusters method landed above it. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: adopt the incomplete download status and drop the paused-ids registry The store inferred an unfinished download from a downloading entry it held no local state for, which misread a download running in another tab as paused and offered a resume the server rejects. The listing carries the "incomplete" status now, so the adoption keys off it directly. The paused ids no longer need to survive in localStorage: the server remembers them across restarts, and the progress snapshot was stale after a reload anyway. The stale localStorage key is removed. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: quant-match the pause and cancel cascades Pause, resume and cancel cascaded to every draft sidecar of the repo while delete only took the quant-matched ones, so canceling a Q4_0 download also killed a Q8_0 draft that belongs to another quant's main. Every cascade acts on the quant-matched drafts now, matching what a delete takes with it. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: collapse the duplicated draft subline markup The downloaded and in-flight branches repeated the identical connector, label and badge markup; one block with a conditional tail keeps the two states in sync. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: scrub positioned styles out of sanitized model card html The style attribute stays allowed for the KaTeX output, which let a third party card draw a fixed overlay over the app. The sanitize hook drops position:fixed and position:absolute values and leaves the KaTeX layout styles alone. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: run the discover mount fetch once and reuse the shared repoOf helper fetch() toggles its loading flag synchronously inside the mount effect, which re-ran the fetch on every toggle and retried failed catalog repos endlessly; the effect now untracks the call. The local repoOf copy and the duplicate services import give way to the shared helpers. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: narrow the spec-type map to the draft sidecars SPEC_TYPE carried empty entries for the aux sidecars it is never indexed with; the map and the command component now type the badge as a draft sidecar through a draftBadgeOf helper instead of a cast. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: name the decision-model tags and markdown cache keys The decision-model hub tag and pipeline tag move into the huggingface constants next to the other pipeline tags, and the markdown transform cache prefixes get names instead of bare one-letter tokens. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: read the discover setting through one truthiness check Three places read the setting with !== false and one with Boolean(); all four read it the same way now. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: compare the models dialog view against its enum The template compared the view state against the 'manage' literal in three spots while the script used the enum. Assisted-by: pi (zai-org/GLM-5.3-Flash) * nit: Remove `+` before draft sidecar badge * server: list unfinished downloads as incomplete A download that never finished was reported as "downloading", the same status a live download carries, so a client could not tell a running download from a dead one. The tags remembered in unfinished-downloads now list under a dedicated "incomplete" status, without an instance. Assisted-by: pi (zai-org/GLM-5.3-Flash) * server: download a model with its sidecars as -hf does Resolution goes back to master. POST /models takes the sidecar kinds to pull with the model (mmproj by default) and the download child gets them as -hf flags, so the server picks the files. /models lists real models only, each with the sidecars it has on disk, and incomplete downloads keep their sidecars across restarts. A stopped download keeps its partial files for the next attempt, and DELETE purges them through the same plan the download follows. * ui: download sidecars as parameters of their model The download options keep their chips, but a chip no longer posts a sidecar tag: the store maps it to its model and the sidecar kinds to pull, the quant chip takes its companion draft in the same download, and draft badges come from the sidecars each cached model lists. The server adds the sidecars a cached model lacks when it is posted again, and DELETE removes a single sidecar of a cached model. * ui : mirror the server's sidecar lookup in draft badges The server resolves a sidecar by the case-sensitive `<kind>-` token in the same directory as the weights. The UI matched the token loosely, so a badge could offer a draft the server never pulls. Assisted-by: DeepSeek-V4.1-Flash * ui : post the model's sidecar kinds with its download The store keeps one download per model: the chip's kinds ride along with the model's own sidecars, and a download row no longer expands into one row per sidecar. Draft badges carry no quant of their own yet, so a sidecar of a cached model is never labelled with the model's quant. Assisted-by: DeepSeek-V4.1-Flash * ui : join a quant's draft into its download chip One chip per quant, its draft badged inside it and its size covering both files. The draft is the file the server would pick: the same directory, the exact quant, else the closest bit depth, so a 4-bit model can pair with a 16-bit head. Assisted-by: DeepSeek-V4.1-Flash * ui : drop the draft download rows from the manager and selector A download is one row per model again: the drafts ride with it, and the rows read the sidecars on disk off the listing instead of pairing sidecar entries with their model. Pause, resume and cancel act on the model's own download. Assisted-by: DeepSeek-V4.1-Flash * ui : keep the discover download list one row per model The list leads with one row per downloading model, its sidecars as sub-lines. A row badges only the draft kinds the server can resolve, so a badge never outlives the file its lookup would pick. Assisted-by: DeepSeek-V4.1-Flash * ui : keep the discover chips readable in dark mode The metadata chips, the chat template button and the View on Hugging Face link carried a faint border and no surface over the dialog background, so they were near invisible. They take the tertiary button surface now, and their icons size with size-* so the variant's base rule leaves them alone. Assisted-by: DeepSeek-V4.1-Flash * ui : carry the draft through the download options One chip per main quant, its draft badged on it and the chip's downloaded state covering both files. The drafts sit behind an opt-in checkbox with an inline type pick, and each row opens its own command: it pins the row's quant and only names the draft type, so -hfd is gone. Assisted-by: DeepSeek-V4.1-Flash * ui : line the details skeleton up with the loaded pane Same root padding and container context as the loaded view, so the container queries resolve alike. The download options card pads its blocks instead of itself, the readme gets its card, and a description line stands in for the catalog paragraph. Assisted-by: DeepSeek-V4.1-Flash * common : drop the partials of a removed sidecar Removing a sidecar alone left the .downloadInProgress files of the sidecars a download had planned in blobs/. The removal purges them through the same plan the download follows, and the test covers a finished draft beside an unfinished one, whose partial goes while the model keeps its files. Assisted-by: DeepSeek-V4.1-Flash * server : name why a download is refused A model that is already downloading, or that is running, answered "already exists". It now says which: "is already downloading", or "is running, unload it first", both before the resumable check. The router test covers the two. Assisted-by: DeepSeek-V4.1-Flash * ui : post a download once per click Two quick clicks on the same chip posted twice, and the second answer was "already exists". The store tracks the model as pending, returns early while it is pending or in flight, and reads it as in flight before the server answers, so the chip shows the download at once. The new client test pins the flows: one POST per double click, the chip reading downloaded once the list reports the model, a resume keeping its sidecars, a cancel dropping the whole download, and a reload restoring the paused state. Assisted-by: DeepSeek-V4.1-Flash * ui : keep a plain download row the height of the others The sub-lines wrapper of a download row was rendered even with nothing in it, so its padding made the row taller than a plain one and the overlaid progress bar sat in that empty strip. It renders only with sub-lines now, in the selector and in the discover list, and the selector row takes the phone padding its neighbours have. Assisted-by: DeepSeek-V4.1-Flash * ui : badge a repo's draft once The args of a loaded model and the listing of a cached one report the same draft in different words, one with the file's quant and one with the kind alone, so the merge kept both and a repo row showed the same kind twice. The badges merge per kind now, the args one winning as it knows the quant, and the test pins both sources of a real repo. Assisted-by: DeepSeek-V4.1-Flash * server : report the sidecars of a preset or a download on its way A model the preset also names is registered as a preset, so its drafts were never reported and no row could badge them. A download on its way reported none either, though the record of the kinds it pulls is right there: those win over the plan now, the plan resolves through the preset's hf-repo, and a models dir stays out of it. Assisted-by: DeepSeek-V4.1-Flash * ui : badge a download's drafts and open its model from the row A draft still coming down badges the model id now, the way a landed one does, while an on-disk draft keeps its checked sub-line. The row itself opens the model, the status slot beside it pauses or resumes, and the manager resolves a focus on a download row so its panel opens. Assisted-by: DeepSeek-V4.1-Flash * common : report a removed sidecar that only had a partial Removing a sidecar alone dropped the result of the partial purge, so a sidecar that only ever landed as a .downloadInProgress blob answered false and the UI had no reason to refetch. The purge result is folded into the return now, and the test covers a partial-only removal. Assisted-by: DeepSeek-V4.1-Flash * ui : ignore a pause while the download POST is still on its way The chip reads as downloading before the server answers, so a quick second click unloaded a model that had not started and left the store neither in flight nor paused. A pause returns early while the POST is pending, before it touches the feed or the paused marker. The client test holds the POST open and pins it. Assisted-by: DeepSeek-V4.1-Flash * ui : offer a missing draft as its own chip A model already on disk without its draft had no way to add it: the quant chip carried the draft, so it read as a download of both. The quant chip keeps to its own state now, its joined draft shows only while it rides along or is on disk, and a downloaded quant missing its draft gets a chip of its own - badge, size, download - that posts the model with that kind, so the server keeps the files already there and pulls the draft. Assisted-by: DeepSeek-V4.1-Flash * server : list a paused sidecar download by the files on disk A model on disk whose sidecar download stops keeps its unfinished record, and the listing reported the kinds that record plans as if they were on disk: the UI checked a draft that never landed and offered no way to add it. The planned kinds now stand in only while the download runs. * chore: Clean up comments * server : list sidecars only for a model that resolves from a HF repo A preset on a local file has no HF repo, yet the listing resolved its name as one: the plan threw on the invalid repo format and GET /models failed for every model. The sidecars now come from the repo a preset names or a cached model is listed under, and a local file lists without them. * common : fix download cancel regression Signed-off-by: Adrien Gallouët <angt@huggingface.co> * common : preserve model partials when removing sidecars Signed-off-by: Adrien Gallouët <angt@huggingface.co> * server : one sidecar table and an unfinished downloads class Address review from @ggerganov. The sidecar kinds live in one table in common/download, mapping each name to its download option and plan file; the server validates, builds its options and lists cached sidecars through it instead of its own lists. The unfinished downloads move into server_unfinished_downloads, a record with its own mutex and a struct per entry, and server_models exposes them through unfinished_downloads_get and unfinished_downloads_remove. * common: one sidecar table, and removals that keep shared files The resolution, the cached listing and the removal follow a single table of sidecar kinds, so a kind is declared once with its file keyword, whether it is a draft, its option and its plan file. The server reads the same table for the kinds it accepts, the default sidecars and the child flags. Removing a tag keeps the files another cached tag still loads, such as an mmproj named after the removed quant. A sidecar removed on its own goes for every tag that resolves it. An unfinished download records the exact paths of its partials when it starts. A removal deletes those without asking the hub, and keeps a partial another recorded download also writes. The record is taken once the child is spawned, and the files go under the models lock, so a download of the same tag cannot start while they are removed. * server: drop stale partials and refuse a tag that resolves nothing Recording a download again deletes the partials of the earlier record that the new plan no longer writes, such as those of an older hub revision or of a sidecar the new request leaves out. A posted tag that resolves no file of the repo is refused before the download starts. Removing a model follows a single path, whether the reload erased its entry during the wait or not. * server: fix the CI race between the router download tests The pytest workers run in parallel on one shared cache, so a download test could find the file another test had just fetched and see no progress, or start from the unfinished downloads record another test wrote. Each download test now runs on an empty cache of its own. * server: refuse a DELETE of an unknown name and decide downloads in load() A DELETE of a name that is neither a model nor a recorded download went to common_download_remove, and without a tag it wiped the whole repo cache, including the files of a loaded quant or of a download on its way. Such a name is now not found, as it was before. A POST that downloads over an existing entry checked its state outside the lock, so a load of the same model in between got its instance killed. The checks now run in load(), under the lock, right before the spawn. The fake load time of the tests applies to model loads only, since a download worker loads none. * server: add incomplete_sidecars to the /models API A sidecar added to a cached model and then paused kept its record, but the listing showed only the complete model, so a fresh UI lost the pause. The model entry now carries incomplete_sidecars, the kinds its unfinished download has not written yet; a POST with the kind resumes one and a DELETE with it drops its partial. * common : drop a kept partial when a resume gets 416 Now that partials survive between attempts, an old or complete partial makes every retry fail with the same 416. Delete it so the next attempt can start over from zero. Signed-off-by: Adrien Gallouët <angt@huggingface.co> * common: drop a kept partial when a resume gets 200 A server that ignores the range answers 200 to a resume, and the kept partial then fails every attempt the same way a 416 does. Drop it as well, so the next attempt starts over without a range. * server: refuse a download that shares a partial with one on its way Two downloads of quants that share a file, the mmproj or an mtp picked by the closest bits, appended to the same partial and renamed a file with interleaved bytes. A download whose plan writes a partial that a running download also writes is now refused until that one is done. * common : check the saved etag before resuming a partial download Before resuming, compare the etag saved with the partial to the one the server now serves. If it is missing or different, start over, so a resumed download never mixes bytes from two versions of the file. Signed-off-by: Adrien Gallouët <angt@huggingface.co> --------- Signed-off-by: Adrien Gallouët <angt@huggingface.co> Co-authored-by: Pascal <admin@serveurperso.com> Co-authored-by: Adrien Gallouët <angt@huggingface.co> |
||
|
Pascal
|
10b8b75534 |
tests: give the random model generator a per-tensor filler table (#30674)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* tests: give the random model generator a per-tensor filler table Muse Spark makes test-llama-context fail on state_load, cp_h and cp_d on about six percent of the random seeds, with an NMSE of 1.0e-05 to 1.4e-05 against a threshold of 1.0e-05. The sliding window is what exposes it: a restored state keeps the window alone, so the attention of an SWA layer reduces over a different cell layout and the continuation is not bit exact. Every SWA model carries that, at 1e-08. Muse Spark carried it at 1e-05. The generator is what turns 1e-08 into 1e-05. A norm preserving residual gate holds alpha^2 + beta^2 == 1 and the conversion resolves both weights from one raw weight, while the generator draws every tensor on its own. The pair then scaled the stream and the branch independently, the stream lost an arbitrary factor at each layer, and the model became an amplifier. The weights the test ran on were not a Muse Spark. Per-tensor rules lived inside the write path, where a flag picked the values of a tensor inside each of the two branches that store them, so a new rule meant a new flag threaded through both. The file asks to be rid of that in a TODO of its own. Give each rule a filler, pick one from a table keyed on a substring of the tensor name, and keep a single write path behind it. Mamba moves into the table and Muse Spark joins it: the gate filler draws the raw weight once per block and resolves alpha and beta the way the conversion does. Over a thousand seeds on a CPU build the worst case drops from 1.36e-05 to 2.48e-06 and the median from 4.10e-06 to 5.48e-07, with no seed above the threshold. The other 131 models come out identical to the bit, since no other architecture carries these tensors. ctest is green on a CPU build and on a CUDA build. * tests: draw every tensor filler from fill_normal The fillers repeated the generator and the distribution each time. The normal draw now takes its key and the other fillers transform its values, so the generator lives in one place. The generated models are identical to the bit. |
||
|
Sigbjørn Skjæret
|
f2f667b02d |
cuda : add grouped experts top-k fusion (#29181)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* fuse grouped experts * add tests * hip fix * add to graph_optimize * apply suggestion from am17an * hip fix * fix hip fix * fix fix fix * gate on hip version instead |
||
|
Xue Zhenliang
|
62318cd57d |
model : add Kolibri-1 (Aleph-Alpha) support (#30673)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* model : add Kolibri-1 (Aleph-Alpha) support Kolibri-1 is a bilingual German/English MoE reasoning model with 78B parameters and 3.46B active per token: 50 layers, 384 routed experts plus one shared expert. Attention interleaves four sliding layers at a 513 window with one full layer. The sliding layers rotate with NeoX RoPE, the full layers are NoPE. GQA runs 48 query heads against 4 KV heads at head_dim 128 with a QK-RMSNorm, and every block carries four RMSNorms, so the attention and the MLP output are normalized before they rejoin the residual stream. Routing picks the experts from logits + expert_bias and weights them with the sigmoid of the unbiased logits, without renorm. The converter reuses AfmoeModel for the per-expert checkpoint layout and the expert selection bias, and writes the shared expert and the per layer sliding window pattern. The released FP8 checkpoint needs no special code, it goes through the generic weight_scale_inv path. Three tensor names are added to the mapping (post_attn_norm, post_ffn_norm, moe.router.expert_bias) and the 128k vocabulary registers the qwen2 pre-tokenizer by hash. The arch is registered in test-llama-archs, with GQA and the sliding window pattern. Assisted-by: DeepSeek Harness (DeepSeek V4.1 Flash) * Update src/models/kolibri1.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Add fusion records for kolibri1 * Update src/models/kolibri1.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> |
||
|
Daniel Han
|
01fc3aae84 |
model: add Beam (Reflection) (#30670)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
|
||
|
Georgi Gerganov
|
88aaf325a7 |
tests : catch vanished rows in the fusion baseline (#30618)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* tests : update the Metal fusion baseline Re-record the MTL fusion baseline with the current build: 276 rows over 131 dummy models. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * tests : fail test-fusion when a baseline row stops matching The report was built from the counters the run produced, and the baseline was only ever used as a lookup table, so a fusion that stopped matching produced no row and was never compared. qwen4exp and mellum lost their fusions this way while the Fusion / metal job stayed green. Report a baseline row that the run does not reproduce as a regression. Scope the pass to the arches that were actually run so --model keeps working. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * pi : never write inline comments unless the user asks Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * tests : keep the mellum dummy model softmax-gated mellum reads expert_gating_func from the GGUF since #30578, while the fixture writes sigmoid for every MoE arch, so the Mellum dummy model stopped building the softmax MoE routing fusion. Write softmax for mellum, matching the arch default and the GGUFs that conversion/mellum.py produces. Re-record the MTL baseline: the routing row is back, 277 ok / 0 failed. Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * pi : use emoji in the AI stats disclosure Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * pi : list the disclosure model after the hardware Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * pi : use monochrome symbols in the AI stats disclosure Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) * tests : add muse-spark to the Metal fusion baseline Assisted-by: pi:llama.cpp/ggml-org/Qwen3.8-Flash-Next-GGUF (MacBook, M5 Max, 128GB) |
||
|
Karthik Vijayan
|
f12aaa344c |
Fix failing RocM tests (#30611)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
Co-authored-by: Karthik Vijayan <kavijay@microsoft.com> |
||
|
Max Krasnyansky
|
b797b9ca81 |
hexagon: tensor repacking rework and support for tiled and flat quantized and fp weights (#30635)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* hex-repack: Remove need_repack hack and streamline the rules -- breaks GET_ROWS tests for K-Quants, will fix in the next round Add support for GET_ROWS with tiled Q5_K Add support for raw k-quant tensors * hex-get-rows: vectorize raw k-quant ukernels * hex-binary: refactor dma-loop thread functions (no need for macros, etc) * hex-quant: move/cleanup flat and tiled dequant ukernels * hex-repack: add support for repacking/tiling F16/F32/BF16 2D weights * hex-get-rows: remove redundant cast * hex-repack: proper filtering for BF16 that were leaking through during the tests * hex-repack: streamling tiling rules (broken MM for now) * hex-repack: fix mmid after tiling updates * hex-repack: rename repacked to tiled in all flags/funcs/kernels/vars * hex-mm: fix valid_rows calc in various kernels * hex-mm: remove unused hvx_mm_nx_2d_flat * hex-mm: cleanup and optimize hvx-tiled vec_dot_f16 * hex-mm: another pass at hvx-tiled f16 matmul * hex-get-rows: optimize small src1 * hex-get-rows: one more pass at recovering get-rows perf regression * hex-repack: export GGML_HEXAGON_RT_REPACK via run.py * hex-mm-tiled: fix accuracy regression with hmx-tiled-f16 matmul * hex-mm: fold tiled_f16 checks into the general flow * hex-get-rows: remove dead checks for buffer_is_hexagon and tiling during supports_op * hex-quant: proper tail handling in new dequant ukernels * hex-get-rows: remove unused code and cleanup support checks * hex-mm: cleanup ne1 padding/tiling checks * hex-mm: cleanup fusion checks for tiled tensors * hex-repack: add a little can_repack helper to reduce duplication * hex-repack: never repack host buffers * hex-repack: factor out repack functions into separate .h and .cpp * hex-tensor: cleanup tensor ifaces for tiled/padded sizes * hex-repack: update k-quant tiling comments and cleanup #defines * hex-dcache: factor out l2/dcache dirty tracking and flushing * hex-tensor: update docs and remove #defs duplication * hex-tensor: remove unused/incorrect get_data interface * hex-get-rows: cleanup vtcm layout build * tests: disable huge MM tests * hex-dcache: add missing header |
||
|
Said Taghadouini
|
2f729cfa4c |
mtmd: lightonocr does not require longest edge to always be 1540, native resolution is the default (#30600)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
Assisted-by: Codex |
||
|
Pascal
|
ba3c05f71d |
model: add Muse Spark Open (text, vision and audio) (#30616)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* model: add Muse Spark Open (text, vision and audio) Muse Spark Open is a latent space MoE: the residual stream stays at 8192 while the 256 experts live in 4096, with a projection down and up around them, top-8 sigmoid routing with a correction bias. Attention interleaves three sliding layers at a 2048 window with rope for one global layer without rope, with a sigmoid output gate and a weightless QK-norm. Residuals are norm preserving, their gate is folded at conversion time. Vision reuses the Muse Glimmer tower. Audio is a causal convolutional pyramid with no attention and no spectrogram, four stages folding 16, 6, 5 and 4 samples into the channel dim, one embedding per 80 ms of 24 kHz. The quantizer keeps the latent projections and the attention gate at Q8_0, a little over 1% of the weights, since the whole residual stream passes through them and the low-bit types turned the output into a repeating loop. The model saver writes the post norm epsilon, and the arch is registered in test-llama-archs. Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: Pedro Cuenca <pedro@huggingface.co> * Muse Spark: drop the redundant forward expand * Muse Spark: drop the redundant forward expand and tighten the comments * Update src/llama-quant.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Update src/llama-graph.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Update conversion/muse_spark.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Update conversion/muse_spark.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Update conversion/muse_spark.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Update conversion/muse_spark.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: Pedro Cuenca <pedro@huggingface.co> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> |
||
|
Karthik Vijayan
|
3e4dfa8f39 |
models : add maion-coder (#29778)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* Add Support to maion-coder arch * iAdd Support to support Draft models * Address comments * Remove maion context shift function --------- Co-authored-by: Karthik Vijayan <kavijay@microsoft.com> |
||
|
Yuri Khrustalev
|
82e1e2fd84 |
jinja : write nan and inf in tojson as json.dumps does (#30483)
tojson wrote nan, inf and -inf for these floats, as values and as dict keys; json.dumps writes NaN, Infinity and -Infinity. The string and safe filters of a float keep nan and inf, as Python's str() does, so they no longer use the tojson function. float_to_json is renamed float_repr, as it now gives str(); float_json gives the json.dumps text. Also say in a comment why a last digit 9 is not bumped at a power of 2. |
||
|
losewayy
|
47f5a64ae1 |
vulkan : fix concurrent first device init (#29766)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* vulkan : fix concurrent first device init * tests : load models in parallel in test-thread-safety Loading them serially never exercised concurrent first backend init, so the crash reported in #29764 was invisible to this test. * vulkan : create fence before retaining device in buffer context |
||
|
William Tan
|
2130ca8476 |
jinja : fix loop body variable scoping (#30533)
Assisted-by: Codex |
||
|
Simone Iacomino
|
2ce20d22fd |
unicode : reject overlong UTF-8 and out-of-range codepoints (#30485)
* unicode : reject overlong UTF-8 and out-of-range codepoints * Update tests/test-unicode.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * tests: document byte-wise UTF-8 error recovery * break out of first test on failure --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> |
||
|
Nathan Wilson
|
733f57a999 |
vulkan : copy strided f16 KV once on AMD so reads spread across memory channels (#27703)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* vulkan : copy strided f16 KV once so reads spread across memory channels Assisted-by: Claude (Opus 4.8 / Opus 5) * tests : cover the f16 KV copy to scratch Assisted-by: Claude (Opus 4.8 / Opus 5) * vulkan : gate the f16 KV scratch copy to AMD Assisted-by: Claude (Opus 4.8 / Opus 5) * vulkan : reuse the generic copy shader for the f16 KV scratch fill The f16 clause needs only a strided copy, not a dequant, so drop the dedicated shader and dispatch the existing copy pipeline instead. The copy is described in source memory order so the strided access lands on the writes, which stream, instead of the reads, which would concentrate on a few memory channels. Assisted-by: Claude (Opus 4.8 / Opus 5 / Fable 5) * vulkan : extend the strided KV scratch copy to bf16 Assisted-by: Claude Opus 5 * vulkan : clean up comments Assisted-by: Claude Opus 5 * consolidate copy shaders * disable on RDNA4 --------- Co-authored-by: Ruben Ortlam <rortlam@redhat.com> |
||
|
cqderek
|
5a638aedb7 |
cpu: add F16 SWIGLU_OAI reference implementation and validation (#29200)
* cpu: add F16 SWIGLU_OAI reference implementation and F16 activation op test coverage
Adds ggml_compute_forward_swiglu_oai_f16() to the CPU backend, mirroring
the existing F32 path, and extends test-backend-ops with F16 test cases
for SWIGLU_OAI so downstream backends(e.g. Hexagon) can be CPU-diffed
for correctness.
* ggml-openvino: fix f16 SWIGLU_OAI type mismatch in alpha/one constants
translate_glu_swiglu_oai creates alpha_const and one as f32 constants
unconditionally, so f16 inputs hit a type mismatch in Swish/Add on GPU
("Swish inputs must have the same type but they are: f16 and f32").
translate_glu_swiglu_clamp already upcasts to f32 and converts back at
the end; apply the same pattern here.
---------
Co-authored-by: ravi9 <ravi.panchumarthy@intel.com>
|
||
|
Ethan Guo
|
6e66cb5d4e |
metal : add BF16 variant of the tensor API flash attention kernel (#30491)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* metal : add BF16 variant of the tensor API flash attention kernel * cont : move tensor FA kernels to kernels/fa_tensor.metal |
||
|
Pascal
|
e467e5b5e5 |
kv-cache: honor PARTIAL_ONLY in the plain KV cache (#30514)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* llama: honor PARTIAL_ONLY in the plain KV cache A non-SWA llama_kv_cache rolls back with seq_rm, so it holds no partial state, yet state_write ignored the flag and serialized every cell. A draft context made of a plain KV cache (MTP head, dense -md draft) therefore copied its whole KV into each server checkpoint, growing it by the draft KV size per token. The cache now writes and reads nothing for PARTIAL_ONLY unless it is SWA, and the iSWA wrapper no longer gates its base cache since it filters itself. * Update src/llama-kv-cache.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * llama: delegate PARTIAL_ONLY to the caches of every memory wrapper The hybrid, hybrid-idx and dsa-iswa wrappers skipped their attention caches on PARTIAL_ONLY, which also dropped an SWA window that is partial state. Each cache now applies the flag itself, so the wrappers only delegate. The indexer is marked stale only when a restore rewrites cells. test-llama-context gains a case checking that the partial state is not larger than the full state and, without SWA, does not grow with the tokens. --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> |
||
|
Aman Gupta
|
62df1815bc |
cuda, hip: GDN chunked kernel (#29353)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* cuda, hip: GDN chunked kernel Ported-from: https://github.com/tetherto/qvac-fabric-llm.cpp/pull/257 Source-commit: 53f0027d8070715fe9ad2131f3201bb0001d4eed Co-authored-by: Aman Gupta <aman.gupta@tether.io> * fix older builds * review fixes * cuda: enable GDN MMA on tested Ampere and Ada GPUs Enable SM86 and SM89 after RTX 3090 and RTX 4090 correctness and prefill benchmarks. Keep other untested architectures on the existing fallback. Assisted-by: Codex * cuda : use refactored swizzle API in GDN MMA * ggml : GDN only guarantees the final state when n_tokens > K * cuda : add short GDN MMA phase comments --------- Co-authored-by: gianni-cor <gianfranco.cordella@tether.io> Co-authored-by: Aman Gupta <aman.gupta@tether.io> |
||
|
Pascal
|
6868a52cb3 |
ggml: fix conv_1d result layout for batched input (#28738)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* ggml: fix conv_1d result layout for batched input The GEMM behind conv_1d folds the batch into its rows, so the product sits in memory as [OL, N, OC] while the reshape claims [OL, OC, N]. The shape is right and the contents are transposed, so nothing fires: no assert, no abort, the graph just runs and the channels come out interleaved with the batches. With N == 1 the two readings describe the same buffer, which is why the three vocoders in the tree never hit it, they all squeeze to 2d right after the call. Reshape to the real layout and permute OC and N back. The N == 1 path keeps its direct reshape, so the current callers build the same graph and pay for no extra copy. tests/test-conv-1d.cpp compares one batched convolution against N single batch convolutions built from the same weight and the same input slices. Five of the seven geometries fail before this change and all seven pass after. N == 1 and OC == 1 pass either way, since swapping two dimensions when one of them is 1 changes nothing. The batched path matters to downstream audio projects where several synthesis lanes share a single graph. Nothing in mtmd batches a vocoder today, each one streams one flow and carries its left context between calls, but the op is correct now if one of them grows a lane dimension. * ggml: build the same conv_1d graph for every batch size Address review from ggerganov: drop the N == 1 branch of ggml_conv_1d. The GEMM result is always reshaped to [OC, N, OL] and permuted back, as ggml_conv_2d does, so every batch size builds the same graph. For N == 1 this adds a cont on an already ordered tensor. |
||
|
Georgi Gerganov
|
3fb47677ea |
common : add --spec-draft-n-ubatch (#30459)
* common : add --spec-draft-n-ubatch Allow configuring the physical batch size of the draft context. The drafters feed it at most the drafted tokens per sequence and chunk larger batches by its own n_ubatch, so it does not need the ubatch of the target. A smaller value shrinks the draft compute buffer and leaves more VRAM for the target KV cache (the draft is fitted together with the target). The default 0 keeps the current behavior of following --n-ubatch. Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next * cont : remove comment [no ci] |
||
|
Georgi Gerganov
|
8c2134a875 | ggml : add note about mul_mat_id + disable mul_mat_id_dup tests (#30501) | ||
|
Yuri Khrustalev
|
77390baa18 |
jinja : write floats in tojson as Python does (#30362)
tojson printed floats through a C++ stream (6 significant digits, no ".0"), so 48.858844 became 48.8588 and 1.0 became 1 in tool definitions, tool-call arguments and other template output. Write the shortest digits that read back as the same value, as json.dumps does. The string and safe filters of a float use the same code. |
||
|
Sandro Hanea
|
10da8c463f |
unicode : add opt-in NFC normalization for Unigram tokenizers (#30390)
* unicode : add opt-in NFC normalization for Unigram tokenizers Note: used help from Codex, Sol 6.1 Medium * Fixing initial bot comments + linter error (using existing out instead of print) * removed test_nfc_tokenizer |
||
|
Yuri Khrustalev
|
04f7226a2c |
server : escape the special tokens without std::regex (#30243)
The d1-omni and kev prompts escape "<|name|>" in the request text with std::regex_replace. libstdc++ recurses for each char of the name, so a state of "<|" and 30000 letters overflows the stack of the HTTP thread and llama-server dies with SIGSEGV. server_decision_escape_special_tokens() writes the same text with a loop. test-server-component compares it with the regex on every string of up to 6 chars of "<|>a-". |
||
|
Xuan-Son Nguyen
|
ff026f8c0a |
chat: replace output string with parts (#30389)
* chat: replace output string with parts * reserve first |
||
|
Jeff Bolz
|
77c4540ce4 |
vulkan: Optimize pool_2d when the pooling window is large (#30119)
When the window is larger than a subgroup, use a whole subgroup to compute each output. |
||
|
Georgi Gerganov
|
a17e94d229 |
tests : consolidate context state tests into test-llama-context (#30023)
* tests : consolidate the context state tests into test-llama-context test-save-load-state, test-recurrent-state-rollback and test-state-restore-fragmented check the same object from three places: the KV cache and the recurrent state of a llama_context, saved, loaded, copied, rolled back. Each had its own main, its own loop over the models, its own copy of the token and the comparison helpers, and a run of all three loaded the same models several times. They are one binary now, and the case is the unit of the test: tests/llama-context/test-case.h the interface a case implements tests/llama-context/cases-kv.cpp the cases of the KV cache tests/llama-context/cases-rs.cpp the cases of the recurrent state tests/llama-context/test-util.cpp tokens, generation, comparison tests/llama-context/test-runner.cpp the models of a case, on threads tests/llama-context/test-log.cpp the log file of a model A run is a table: a row per model, a column per case, a cell holds the verdict of that pair. The models load on several threads (-j, --threads, min(cores, 4) by default), the table is printed when it is complete and the log of a model that failed is printed after it, in the order of the rows. A case returns a verdict and the reason for it, and the runner is the only one who reports - the test does not use tests/testing.h, a run over 130 models is a matrix of verdicts and not a tree of subtests. Staging no longer travels through the environment: llama_model_set_attn_rot replaces the LLAMA_ATTN_ROT_DISABLE variable, which a model loaded in one thread used to change for the models the other threads were loading, and llama_memory_get_buffers is added for the cases that copy the buffers of a memory. common_log_set_stdo keeps the console quiet while the table is being built. ggml_aligned_malloc allocates with posix_memalign when the thread sanitizer is on. On macOS it allocates with the Mach VM, which the sanitizer does not intercept, so it never sees a page range change hands and reports the first write into recycled memory as a race - 30 of them in this test, and every one of them false. Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next * cont : run single-threaded by default * cont : try to build the tests on all platforms * cont : trim down some body fat * cont : fix * cont : fix the fix |