master
1427 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 tc-mb
|
a08ab75ae7 |
mtmd : add MiniCPM-OCR 1.0 support (#30979)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
Signed-off-by: tc-mb <tianchi_cai@icloud.com> |
||
|
Tim Wang
|
a5d22276be |
server : support typed content input for RERANK endpoint (#29625)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* server : support typed content input for rerank endpoint Assisted-by: Opencode/Qwen3.8 27B * server : address rerank multimodal review findings - reject multimodal rerank input when the model has no rerank chat template (images were silently dropped) - translate handle_media download/base64 failures to 400 instead of 500 - reject negative top_n with 400 instead of letting resize throw - fix comment typos, redundant comments, indentation and trailing whitespace - README: document that TEI input accepts the same shape as Jina; drop stray blank line Assisted-by: opencode * Trimmed README --------- Co-authored-by: timothywang21 <timothywang21@users.noreply.github.com> |
||
|
Georgi Gerganov
|
1f4c2eeaa5 | ci : be able to reconfigure HTTP port for server pytest workers (#30972) | ||
|
Aleksander Grygier
|
84746c5c94 |
ui: Add "Show tool call in progress" setting (#30681)
* feat(ui): add Show tool call in progress display setting
Add the showToolCallInProgress key and its checkbox to Settings > Display,
right next to its "always show" counterpart. Upstream dropped a setting of
this exact name in
|
||
|
Aleksander Grygier
|
a6b83d426f |
ui : key draft sidecar badges by model instead of repo (#30919)
Two quants of one repo pulled each other's draft: the badge read every --model-draft arg of the repo, so each row showed both drafts. Assisted-by: DeepSeek-V4.1-Flash-1017 |
||
|
Maphist0
|
7775ffb7e4 |
model: Add support for SenseNova U1 model (#28919)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* model : add SenseNova U1 understanding support Add support for the dense SenseNova U1 understanding branch, including the U1.5-8B-MoT model. - Add HF-to-GGUF conversion for NEOChatModel, excluding MoT generation tensors and concatenating temporal and spatial Q/K normalization weights. - Register the sensenova_u1 GGUF architecture and custom spatial RoPE metadata. - Implement temporal/spatial Q/K normalization with 4-axis RoPE. - Keep U1-specific text position handling in the model implementation. - Add synthetic architecture coverage for sensenova_u1. Only the dense understanding branch is supported; generation branches are intentionally rejected during conversion. * conversion : treat one-expert SenseNova U1 configs as dense * mtmd : add SenseNova U1 vision support Add conversion support for the SenseNova U1 vision encoder and implement its CPU mtmd projector graph. The graph applies patch embedding, GELU, adjacent-pair 2D RoPE, spatial downsampling, and dense projection for dynamic-resolution image inputs. * mtmd : fix SenseNova U1 vision token layout * model : clean up SenseNova U1 RoPE metadata Replace the hard-coded spatial RoPE key with shared GGUF and LLM_KV definitions used by conversion, model loading, and architecture tests. Clarify why SenseNova U1 retains its custom text-position input: U1 uses [t, 0, 0, 0], while the generic M-RoPE input uses [t, t, t, 0]. * mtmd : simplify SenseNova U1 projector integration Map the SenseNova U1 dense projection to the standard mm.0 weight and bias tensors and reuse the shared projector fields in the loader and graph. Read vision parameters from the checkpoint configuration and remove the U1-specific tensor names and model fields while preserving its strided projection, bias, and row-major token layout. * mtmd : fix SenseNova U1 image positions and attention * fix spacing * mtmd : restore generic test input chunks * mtmd : restore SenseNova U1 integration after upstream merge * Update conversion/sensenova_u1.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Update src/models/sensenova-u1.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * model : remove SenseNova U1 BF16 activation casts Use standard backend precision handling for QKV, FFN, and output projections instead of casting inputs based on the output tensor type. Assisted-by: Codex * Update src/models/sensenova-u1.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> |
||
|
Xuan-Son Nguyen
|
0a1ef309cd |
mtmd: add state object for mtmd_gen (#30902)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
|
||
|
David Garcia
|
261700b896 |
server : forward llama.cpp sampling params on /v1/messages (#30856)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* server : forward llama.cpp sampling params on /v1/messages * server : pass the request through on /v1/messages instead of copying listed keys * NIT: Change comment from multiline to single line * server : accept llama.cpp params under extra_params on /v1/messages * server : remove test case for trivial functionality * server : show extra_params in the /v1/messages README example |
||
|
Unbug Lee
|
9a3b91c680 |
server : fix usage.completion_tokens undercount with n > 1 (#29486)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
Assisted-by: Copilot |
||
|
Xuan-Son Nguyen
|
5c0b49a23b |
server : add POST /tts endpoint for TTS models (#26603)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* mtmd: add audio out stream api * add /tts endpoint * add docs * wire up mtmd_helper_model_can_chat * upload speaker_ref via form-data * expose seed to api * fix header for pcm * wip * fix penalty_last_n * fix server_output_limits * fix pocket-tts stream * wip * add server_schema for tts * add server routing * server: coerce seed and stream from multipart form fields Form fields arrive as strings, so -F seed=42 and -F stream=true were rejected by the schema. seed joins the integer fields, parsed as a long long so a full uint32 seed fits, and stream is read as a boolean when it comes as a string. --------- Co-authored-by: Pascal <admin@serveurperso.com> |
||
|
Xuan-Son Nguyen
|
1242be66f6 |
server: add realtime transcription API (#30636)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* server: add realtime transcription API * drop transcription_sessions * proxy ws * add docs * server: let the API key middleware guard the websocket httplib runs the pre-routing handler before the websocket upgrade, so the API key middleware answers 401 at handshake like any other route. This covers the router proxy, whose children run keyless, and accepts the same key forms as HTTP (Bearer, raw key, X-Api-Key). * server: keep realtime slots out of the prompt loop A realtime slot waiting for audio is GENERATING with nothing to decode, so it fell through the prompt loop and took slot_batched. A request on a higher slot then failed can_batch_with on the task type and starved for the whole session. Realtime slots have no prompt: the loop sets them up once and always leaves them. * server: start the realtime session once its slot is live The audio stream was created in the prompt loop, which skips a slot that cannot batch with the one generating, and session.created went out before the task was posted: early appends failed and the reader quit, leaving the socket open and the slot held until the HTTP timeout. The slot now sets up its stream at launch and signals it with the begin result, session.created waits for it, and a reader that stops closes the socket so the session ends with it. * server: name the committed item in input_audio_buffer.committed Per the Realtime spec, item_id is the item made from the committed audio and previous_item_id the one committed before it, null for the first. committed now comes before the transcript that completes that item, and both go out with the item switch under one lock, so a delta cannot land between them. * server: use the GA transcription session shape The session object follows the GA Realtime schema: type transcription, with the input format, transcription model, turn detection and noise reduction under audio.input. The rate stays the model sample rate, which is what the client must send for now. --------- Co-authored-by: Pascal <admin@serveurperso.com> |
||
|
Takamichi Ohara
|
52ffa02745 |
model : support Nemotron-3(.5)-Embed (#30877)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
|
||
|
Nigel Bosch
|
ead9f32058 |
server: Support logprobs with speculative decoding (#27196)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* return logprobs in spec path * server: add regression test for logprobs with speculative decoding Asserts accepted draft tokens report the target model's logprobs, comparing a draft-enabled run against a no-draft baseline (#24271). Fails on master (top_logprobs empty, logprob 0.0), passes with the fix from #27196. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KARBWpPcijcBXzJEvttuwk --------- Co-authored-by: Jeonghyun Lee <sig.jhl@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Yuri Khrustalev
|
4a45de4f2c |
mtmd : lfm2 tiles and layout match transformers (#30261)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* mtmd : lfm2 tiles stretch the image to the grid Lfm2VlImageProcessor.crop_image_to_patches resizes the image straight to (512 * rows, 512 * cols) before it cuts the tiles. llama.cpp kept the aspect ratio and added black bars for LFM2 (only d1-omni stretched), so the tiles of d1-3B and every LFM2-VL / LFM2.5-VL model differed from transformers unless the image had the aspect ratio of the grid. With PAD_NONE the tiles are the same as in transformers. * mtmd : lfm2 layout with Python rounding in double precision Lfm2VlImageProcessor.smart_resize rounds with Python's round (half to even) and computes beta and the tile grid aspect ratio in double. llama.cpp used std::round and float, so 31926 of the 16.7M sizes in [1,4096]^2 got another thumbnail size or grid than transformers (and d1-omni vision.py), e.g. 400x600 -> 416x608 instead of 384x608 and 720x1600 -> 2x5 tiles instead of 2x4. The lfm2 smart resize and grid choice now follow the Python code and match it on every size of that range. * fix --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> |
||
|
PSEUDOTENSOR / Jonathan McKinney
|
7e3eeee048 |
model : add H2O-Lightning-4B decision model (#30361)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* model : add H2O-Lightning-4B decision model Adds the h2o-lightning decision type for h2oai/h2o-lightning-4b (Qwen3.5-4B with a merged decision LoRA). None of the existing types match this model. It answers with " A" rather than "A", uses up to 255 labels, and was trained on Python json.dumps text, so floats like 12.0 must render as 12.0. The converter writes the decision metadata and the systemone template from the model's serve_config.json and refuses a config that doesn't match. Tests skip unless the tiny test model is available locally. Assisted-by: Claude Opus 5.5 * rm redundant code * put labels to gguf --------- Co-authored-by: Jonathan C. McKinney <jon.mckinney@h2o.ai> Co-authored-by: Xuan Son Nguyen <son@huggingface.co> |
||
|
Xuan-Son Nguyen
|
5ba3b9fd3d |
mtmd: fix gemma 4 ffn act (#30882)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
|
||
|
TANGBUDU
|
5a9a34d888 |
server : avoid repeated mmproj memory margin addition on sleep/resume (#28233)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
Co-authored-by: TANGBUDU <TANGBUDU@users.noreply.github.com> |
||
|
Me42019
|
4e201388c3 |
server: Router wasn't propagating the client disconnects fully to the child, the response wait could cause a delayed cancellation (#29707)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* Router wasn't propagating the client disconnects fully to the child, the response wait could cause a delayed cancellation * tests: satisfy router disconnect type check * server : remove timing-sensitive router cancellation test Assisted-by: Codex |
||
|
Renzis
|
1ba568c5ad |
server: fix missing output_index in Responses API SSE events for compatibility (#30863)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* server: include output_index in Responses API SSE events * server: reuse Anthropic streaming code for fix output_index * server: remove unnecessary generated_tool_call_ids and oai_resp_tool_calls_started |
||
|
Tarek Dakhran
|
36140dbee2 |
mtmd : read LFM2 tiling params from GGUF metadata (#30840)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* mtmd : read LFM2 tiling params from GGUF metadata Write min_tiles, max_tiles, tile_size and max_pixels_tolerance from processor_config.json when converting LFM2-VL mmproj, read them in clip.cpp with the previous constants as defaults. do_image_splitting=false maps to min_tiles=max_tiles=1 (no tiling). Rebased version of #25524, original implementation by Michael DM (@michael-dm). Co-authored-by: Michael DM <usable_siren180@simplelogin.com> * fix test --------- Co-authored-by: Michael DM <usable_siren180@simplelogin.com> |
||
|
Peter van Velzen
|
d80619397c |
server : pass id_slot through the Anthropic /v1/messages conversion (#28554)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* server : pass id_slot through the Anthropic /v1/messages conversion server_chat_convert_anthropic_to_oai copies a small whitelist of common params from the Anthropic request into the internal OAI-style body and left id_slot out. Every other completion endpoint honours id_slot; a proxy that pins a session to the slot holding its KV cache needs the same on /v1/messages, or it silently falls back to automatic slot selection and, on hybrid/recurrent models, pays a full re-prefill. Adds a test asserting a request with id_slot set is served by that slot, observed via GET /slots. * server : shorten comments * rm test --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> |
||
|
aiqubits
|
1d88cb05f1 |
ui: hide the native clear button in SearchInput (#30719)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
SearchInput already renders its own clear button. Hide the browser's native search cancel button to avoid showing both controls. Fixes #28627 Signed-off-by: aiqubits <aiqubits@hotmail.com> |
||
|
bosh
|
f711c8f303 |
mtmd : match the transformers Qwen-VL resize (#30736)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
|
||
|
Xuan-Son Nguyen
|
7adf160c41 |
server: add --connect, allow p2p access to server outside LAN (#28277)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* init * wip * wire it up * make it clear that it's experimental * add props.connect_code * nits * fix windows build * server: do not advertise a connect code without the tunnel --connect-code on its own had no effect other than being served in /props, so the Web UI showed a remote access code that no tunnel was serving. Clear it when --connect is off, and say so once at startup. The teardown comment claimed that terminate() closes the child stdout. It does not: the log thread leaves fgets() only once every writer of that pipe is gone, so a descendant of llama-connect would block the join forever. Describe the real contract instead. Also restore the --reasoning-preserve default in the generated docs. * Update tools/connect/CMakeLists.txt Co-authored-by: Adrien Gallouët <angt@huggingface.co> * server: dial llama-connect on the bound port and a reachable host Pass ctx_http.port so --port 0 and the GCP override work, pick the dial host from --host (wildcards map to loopback, IPv6 is bracketed) and refuse --connect when only unix sockets are bound. --------- Co-authored-by: Pascal <admin@serveurperso.com> Co-authored-by: Adrien Gallouët <angt@huggingface.co> |
||
|
A-Raphie
|
ab7ed0d4b5 |
ui : fix attachment text preview clipped when a line is longer than the container (#30806)
In the attachment preview, SyntaxHighlightedCode's wrapper (overflow-auto) was not the element that ended up owning horizontal overflow: the inner code box kept its content clipped at the wrapper's width, so a single line longer than the preview pane was cut off with no scrollbar (scrollWidth == clientWidth while the code's own content measured tens of thousands of px). Give the pre/code boxes their intrinsic width (width: max-content, min-width: 100%) so long-line overflow propagates to the scrolling wrapper. Verified in the running WebUI: with a 6000-char line, wrapper scrollWidth goes from 990 (clipped, unscrollable) to 50449 with a working horizontal scrollbar; short content still fills the container exactly as before (min-width: 100%). Fixes #30796 |
||
|
Tarek Dakhran
|
f174553e16 |
server : fix swa checkpoint invalidation logic (#30666)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* server : fix swa checkpoint invalidation logic Assisted-by: Claude * add testcase that fails with > n_past, but works with > n_prefix |
||
|
Aleksander Grygier
|
2b57148b08 |
ui: Fix markdown tables and lists being cut off on phones (#30687)
* ui : keep markdown tables inside the chat column on phones The phone rule made the table wrapper full-bleed with -1rem margins, so the first and the last column sat under the screen edge. Keep the wrapper inside the chat column and cap it at its width, so a wide table scrolls in place instead. Assisted-by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL) * test : show a table-only model answer in the chat message story Adds a story with an answer that is mostly a table, so the table layout can be checked at a phone width. Assisted-by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL) * ui : indent markdown lists by 2rem 1.5rem put the marker too close to the item text. Assisted-by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL) |
||
|
ppAlexa
|
8f97d82da3 |
tools : reject truncated GGUF tensor data in split and export tools (#30497)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* The problem here in the repo is gguf-split can silently produce corrupted tensor output when the input GGUF file is truncated. The fix is to add tensor-data bounds checks and short-read checks in gguf-split.cpp and export-lora.cpp, and a regression test that checks truncated input is rejected and valid tensor data is preserved.# * Bug : GGUF-split can silently produce corrupted tensor output when the input GGUF file is truncated. Fix : add short-read checks in gguf-split.cpp (in copy_file_to_file in split_strategy) and export-lora.cpp (in read_tensor_data of file_input) so that truncated input is rejected. |
||
|
Aleksander Grygier
|
c0fd786264 |
ui : Discover & Download Models from HF Hub (#29584)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* common : resolve <quant>-<sidecar> download tags and list cached sidecars A Q4_0-mtp style tag now resolves the sidecar file when no model file matches it, so a solo draft or mmproj download actually pulls the file. Cached sidecar files list as their own entries so the state survives a restart, and removing such a tag deletes only the sidecar. Assisted-by: pi:zai-org/GLM-5.3 * ui : optional sanitized raw HTML in markdown Add an allowHtml prop to MarkdownContent: raw HTML found in the markdown is rendered after DOMPurify sanitization instead of being escaped as literal text. Default stays escaped. Assisted-by: pi:GLM-5.3-Flash * ui : add the discover models components The Discover surface: the curated-catalog list with its search and skeletons, the details pane with its readme, metadata, Hub stats and download options, and the standalone download progress bar that replaces the plain one. The quant download button sizes its select with a new xs trigger, so the select gains that size and the muted box look. The store stays incomplete when a repo fails, so the next mount retries it. Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash * ui : put the discover view in the models manager The manager dialog gains its second view: the Discover Models call to action fades the table out, slides the title with its compass mark, and fades the discover surface in; the arrow button returns to the table. Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash # Conflicts: # tools/ui/src/lib/components/ui/button/button.svelte * ui : add the Enable Discover Models setting The flag gates browsing and downloading HuggingFace GGUF models, so it is declared on the discover layer; the repo-tree lookup in the manager layer above reads it from here. Assisted-by: pi:llama.cpp/DeepSeek-V4.1-Flash * ui : resume model downloads from the pause point A stopped download keeps its partial file instead of deleting it, so re-posting the tag resumes from disk. The first progress record now carries the resume offset, the paused snapshot survives into the resumed entry, and a pause settles as soon as it is requested. Assisted-by: pi * ui : lead the discover list with downloads The discover sidebar lists running downloads above the suggested models, and steps aside while a query is active. A paused download stays in the manager table, where it is resumed or dropped. Assisted-by: pi * ui : make resuming a paused download work Re-posting the tag of a paused download hit the router's "already exists" check, because a paused download parks its entry as DOWNLOADED until the next reload: a download entry that is not running is now replaced and resumed. The UI keeps the paused row when the request still fails, instead of dropping it and looking finished. Assisted-by: pi * ui : remember unfinished model downloads A partial file is named after its content hash, so nothing on disk says which model it belongs to once the router restarts. The router now keeps the tags of the downloads it started, clears a tag when the download finishes or is deleted, and lists what is left as still downloading without starting anything. The UI adopts those tags as paused on every list fetch, so a refresh or a fresh open shows an incomplete download and can resume it from the files already on disk. Assisted-by: pi * ui : show a paused download's state on its chip A paused quant chip reads as paused at rest, the way a running one shows its spinner, and offers the resume action on hover. The selector's download row gets a View in Discover action beside its cancel. Assisted-by: pi * ui : collapse the download options by default The copyable download command starts collapsed so the option rows lead; expanding it stays one click away. Assisted-by: pi * ui : keep decision models out of the hub lists Decision models answer with calibrated probabilities instead of text, so llama-ui cannot run them: the hub data layer drops them from the catalog and every search, keyed by the decision-model tag or a text-classification pipeline. Assisted-by: pi * ui : read download tags as quant and sidecar A `<repo>:<quant>-<sidecar>` tag now parses into its quant and sidecar instead of one opaque quant token, and ModelsService.draftSidecarsByRepo collects the router listing's sidecar-only entries per repo, replacing the inline copy in the models store. Assisted-by: pi * ui : cluster draft downloads under their main model Tracked downloads group by repo: the main weights lead and the draft sidecars ride inside their item as compact sub-lines, on the selector, the discover list and the manager table. The main model's controls pause, resume and cancel the whole cluster, and the manager's download rows select into the configuration pane. Assisted-by: pi * ui : act on tracked downloads from the table rows The table passes its clusters' draft sub-lines to the model rows. A row's actions cascade pause and resume, an in-flight row offers Cancel download while a paused one keeps Delete from disk, and the confirm dialog's copy follows the state. Assisted-by: pi * ui : give downloads a configuration pane of their own Selecting a tracked download opens the pane in a download mode: live progress with pause, resume and cancel, and See in Discover, instead of metadata and load actions that a model without files cannot offer. Assisted-by: pi * ui : make the discover details self-fetching and responsive ModelsDiscoverDetailsContainer owns the details, tree and readme fetches so hosts outside the discover view can render the pane, and the details layout adapts to narrow panes with container queries instead of viewport breakpoints. Assisted-by: pi * ui : download a matching draft with the main model A main quant download now pulls the quant-matched draft of the most preferred kind the repo ships (DFlash before MTP), fire-and-forget, so speculative decoding is ready when the main lands. Manual draft chips stay available; the options row also collapses to one column in narrow panes. Assisted-by: pi * ui : keep discover off by default and couple it with the hub setting Discover Models starts unchecked. Enabling it turns the Hugging Face Hub setting on and disabling it turns it back off, so the checkbox never lies; with discover off the hub setting stays available on its own for the display data. Stored and reset state are corrected the same way on load. Assisted-by: pi * ui : open the manage models dialog on the manager view The dialog now opens on the manager instead of returning to the last view, discover entries still land on the discover view while the setting allows it, and the discover surfaces follow the setting's state. Assisted-by: pi * ui : give the discover search room for its focus ring The sticky search bar insets from the panel edges so the focus ring stays inside the scroll container, and rows scroll behind an opaque bar instead of through it. Assisted-by: pi * common: settle sidecar names on the listing and drop unfinished partials An infix or trailing sidecar token is ambiguous: draft heads use it (Model-MTP-BF16.gguf), so do full models with embedded heads (Model-Q8D-MTP.gguf). Such a name is a sidecar only next to token-free main weights of the same family, while mmproj or imatrix anywhere in the name, or a leading token, decides alone (mmproj_f16.gguf). A quant tag always carries its bit count, so a sidecar named after no quant (mtp-Model-it.gguf) gets the bare tag: the listing, the plan and remove all use it. An imatrix tag resolves into a plan slot of its own, downloaded but never wired into the params. remove resolves the tag the way a download does and deletes the partial files of every planned blob, so cancelling a running download or dropping one left unfinished by an earlier run leaves nothing on disk. The router passes its token and releases its lock around the call, and records the unfinished downloads with an atomic write. llama download succeeds on a sidecar-only plan and prints its paths. * ui : move dialog views and draft states into enums and constants The models dialog's views become the ModelsDialogView enum, the draft subline's states join ModelRowDownloadState, the view fade and the copy-feedback reset move to ui constants, and the dialog's compass marks come from DISCOVER_ICON. Assisted-by: pi * ui : drop the unused discover surface The dead openDiscoverModels twin, the commented-out command divider, the details pane's unused onClose and its header's close button, and a redundant type cast leave; the command's copy feedback joins the shared reset constant. Assisted-by: pi * ui : trim the download option comments and highlight the chat template The comment verbosity the options panel and its chips carried shrinks to the non-obvious why, and the chat template dialog renders through SyntaxHighlightedCode as jinja instead of a plain pre block. Assisted-by: pi * common: only a .gguf file is a sidecar, and mmproj/imatrix as whole words A non-GGUF file named like a sidecar (quantization-mmproj-Q8_0.json) is picked as the mmproj, and a sidecar word inside a longer word of a model name (Phi-2-OmniMatrix.i1-Q4_K_M.gguf) drops the model. * ui: dedupe draft sidecar badges per quant The draft sidecar badges deduped by kind and repo without the quant, so with a draft cached at two quants only one badge survived, and a delete of the other main quant found no badge to remove its sidecar with. Also restores the getDownloadEntries doc lost when the clusters method landed above it. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: adopt the incomplete download status and drop the paused-ids registry The store inferred an unfinished download from a downloading entry it held no local state for, which misread a download running in another tab as paused and offered a resume the server rejects. The listing carries the "incomplete" status now, so the adoption keys off it directly. The paused ids no longer need to survive in localStorage: the server remembers them across restarts, and the progress snapshot was stale after a reload anyway. The stale localStorage key is removed. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: quant-match the pause and cancel cascades Pause, resume and cancel cascaded to every draft sidecar of the repo while delete only took the quant-matched ones, so canceling a Q4_0 download also killed a Q8_0 draft that belongs to another quant's main. Every cascade acts on the quant-matched drafts now, matching what a delete takes with it. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: collapse the duplicated draft subline markup The downloaded and in-flight branches repeated the identical connector, label and badge markup; one block with a conditional tail keeps the two states in sync. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: scrub positioned styles out of sanitized model card html The style attribute stays allowed for the KaTeX output, which let a third party card draw a fixed overlay over the app. The sanitize hook drops position:fixed and position:absolute values and leaves the KaTeX layout styles alone. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: run the discover mount fetch once and reuse the shared repoOf helper fetch() toggles its loading flag synchronously inside the mount effect, which re-ran the fetch on every toggle and retried failed catalog repos endlessly; the effect now untracks the call. The local repoOf copy and the duplicate services import give way to the shared helpers. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: narrow the spec-type map to the draft sidecars SPEC_TYPE carried empty entries for the aux sidecars it is never indexed with; the map and the command component now type the badge as a draft sidecar through a draftBadgeOf helper instead of a cast. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: name the decision-model tags and markdown cache keys The decision-model hub tag and pipeline tag move into the huggingface constants next to the other pipeline tags, and the markdown transform cache prefixes get names instead of bare one-letter tokens. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: read the discover setting through one truthiness check Three places read the setting with !== false and one with Boolean(); all four read it the same way now. Assisted-by: pi (zai-org/GLM-5.3-Flash) * ui: compare the models dialog view against its enum The template compared the view state against the 'manage' literal in three spots while the script used the enum. Assisted-by: pi (zai-org/GLM-5.3-Flash) * nit: Remove `+` before draft sidecar badge * server: list unfinished downloads as incomplete A download that never finished was reported as "downloading", the same status a live download carries, so a client could not tell a running download from a dead one. The tags remembered in unfinished-downloads now list under a dedicated "incomplete" status, without an instance. Assisted-by: pi (zai-org/GLM-5.3-Flash) * server: download a model with its sidecars as -hf does Resolution goes back to master. POST /models takes the sidecar kinds to pull with the model (mmproj by default) and the download child gets them as -hf flags, so the server picks the files. /models lists real models only, each with the sidecars it has on disk, and incomplete downloads keep their sidecars across restarts. A stopped download keeps its partial files for the next attempt, and DELETE purges them through the same plan the download follows. * ui: download sidecars as parameters of their model The download options keep their chips, but a chip no longer posts a sidecar tag: the store maps it to its model and the sidecar kinds to pull, the quant chip takes its companion draft in the same download, and draft badges come from the sidecars each cached model lists. The server adds the sidecars a cached model lacks when it is posted again, and DELETE removes a single sidecar of a cached model. * ui : mirror the server's sidecar lookup in draft badges The server resolves a sidecar by the case-sensitive `<kind>-` token in the same directory as the weights. The UI matched the token loosely, so a badge could offer a draft the server never pulls. Assisted-by: DeepSeek-V4.1-Flash * ui : post the model's sidecar kinds with its download The store keeps one download per model: the chip's kinds ride along with the model's own sidecars, and a download row no longer expands into one row per sidecar. Draft badges carry no quant of their own yet, so a sidecar of a cached model is never labelled with the model's quant. Assisted-by: DeepSeek-V4.1-Flash * ui : join a quant's draft into its download chip One chip per quant, its draft badged inside it and its size covering both files. The draft is the file the server would pick: the same directory, the exact quant, else the closest bit depth, so a 4-bit model can pair with a 16-bit head. Assisted-by: DeepSeek-V4.1-Flash * ui : drop the draft download rows from the manager and selector A download is one row per model again: the drafts ride with it, and the rows read the sidecars on disk off the listing instead of pairing sidecar entries with their model. Pause, resume and cancel act on the model's own download. Assisted-by: DeepSeek-V4.1-Flash * ui : keep the discover download list one row per model The list leads with one row per downloading model, its sidecars as sub-lines. A row badges only the draft kinds the server can resolve, so a badge never outlives the file its lookup would pick. Assisted-by: DeepSeek-V4.1-Flash * ui : keep the discover chips readable in dark mode The metadata chips, the chat template button and the View on Hugging Face link carried a faint border and no surface over the dialog background, so they were near invisible. They take the tertiary button surface now, and their icons size with size-* so the variant's base rule leaves them alone. Assisted-by: DeepSeek-V4.1-Flash * ui : carry the draft through the download options One chip per main quant, its draft badged on it and the chip's downloaded state covering both files. The drafts sit behind an opt-in checkbox with an inline type pick, and each row opens its own command: it pins the row's quant and only names the draft type, so -hfd is gone. Assisted-by: DeepSeek-V4.1-Flash * ui : line the details skeleton up with the loaded pane Same root padding and container context as the loaded view, so the container queries resolve alike. The download options card pads its blocks instead of itself, the readme gets its card, and a description line stands in for the catalog paragraph. Assisted-by: DeepSeek-V4.1-Flash * common : drop the partials of a removed sidecar Removing a sidecar alone left the .downloadInProgress files of the sidecars a download had planned in blobs/. The removal purges them through the same plan the download follows, and the test covers a finished draft beside an unfinished one, whose partial goes while the model keeps its files. Assisted-by: DeepSeek-V4.1-Flash * server : name why a download is refused A model that is already downloading, or that is running, answered "already exists". It now says which: "is already downloading", or "is running, unload it first", both before the resumable check. The router test covers the two. Assisted-by: DeepSeek-V4.1-Flash * ui : post a download once per click Two quick clicks on the same chip posted twice, and the second answer was "already exists". The store tracks the model as pending, returns early while it is pending or in flight, and reads it as in flight before the server answers, so the chip shows the download at once. The new client test pins the flows: one POST per double click, the chip reading downloaded once the list reports the model, a resume keeping its sidecars, a cancel dropping the whole download, and a reload restoring the paused state. Assisted-by: DeepSeek-V4.1-Flash * ui : keep a plain download row the height of the others The sub-lines wrapper of a download row was rendered even with nothing in it, so its padding made the row taller than a plain one and the overlaid progress bar sat in that empty strip. It renders only with sub-lines now, in the selector and in the discover list, and the selector row takes the phone padding its neighbours have. Assisted-by: DeepSeek-V4.1-Flash * ui : badge a repo's draft once The args of a loaded model and the listing of a cached one report the same draft in different words, one with the file's quant and one with the kind alone, so the merge kept both and a repo row showed the same kind twice. The badges merge per kind now, the args one winning as it knows the quant, and the test pins both sources of a real repo. Assisted-by: DeepSeek-V4.1-Flash * server : report the sidecars of a preset or a download on its way A model the preset also names is registered as a preset, so its drafts were never reported and no row could badge them. A download on its way reported none either, though the record of the kinds it pulls is right there: those win over the plan now, the plan resolves through the preset's hf-repo, and a models dir stays out of it. Assisted-by: DeepSeek-V4.1-Flash * ui : badge a download's drafts and open its model from the row A draft still coming down badges the model id now, the way a landed one does, while an on-disk draft keeps its checked sub-line. The row itself opens the model, the status slot beside it pauses or resumes, and the manager resolves a focus on a download row so its panel opens. Assisted-by: DeepSeek-V4.1-Flash * common : report a removed sidecar that only had a partial Removing a sidecar alone dropped the result of the partial purge, so a sidecar that only ever landed as a .downloadInProgress blob answered false and the UI had no reason to refetch. The purge result is folded into the return now, and the test covers a partial-only removal. Assisted-by: DeepSeek-V4.1-Flash * ui : ignore a pause while the download POST is still on its way The chip reads as downloading before the server answers, so a quick second click unloaded a model that had not started and left the store neither in flight nor paused. A pause returns early while the POST is pending, before it touches the feed or the paused marker. The client test holds the POST open and pins it. Assisted-by: DeepSeek-V4.1-Flash * ui : offer a missing draft as its own chip A model already on disk without its draft had no way to add it: the quant chip carried the draft, so it read as a download of both. The quant chip keeps to its own state now, its joined draft shows only while it rides along or is on disk, and a downloaded quant missing its draft gets a chip of its own - badge, size, download - that posts the model with that kind, so the server keeps the files already there and pulls the draft. Assisted-by: DeepSeek-V4.1-Flash * server : list a paused sidecar download by the files on disk A model on disk whose sidecar download stops keeps its unfinished record, and the listing reported the kinds that record plans as if they were on disk: the UI checked a draft that never landed and offered no way to add it. The planned kinds now stand in only while the download runs. * chore: Clean up comments * server : list sidecars only for a model that resolves from a HF repo A preset on a local file has no HF repo, yet the listing resolved its name as one: the plan threw on the invalid repo format and GET /models failed for every model. The sidecars now come from the repo a preset names or a cached model is listed under, and a local file lists without them. * common : fix download cancel regression Signed-off-by: Adrien Gallouët <angt@huggingface.co> * common : preserve model partials when removing sidecars Signed-off-by: Adrien Gallouët <angt@huggingface.co> * server : one sidecar table and an unfinished downloads class Address review from @ggerganov. The sidecar kinds live in one table in common/download, mapping each name to its download option and plan file; the server validates, builds its options and lists cached sidecars through it instead of its own lists. The unfinished downloads move into server_unfinished_downloads, a record with its own mutex and a struct per entry, and server_models exposes them through unfinished_downloads_get and unfinished_downloads_remove. * common: one sidecar table, and removals that keep shared files The resolution, the cached listing and the removal follow a single table of sidecar kinds, so a kind is declared once with its file keyword, whether it is a draft, its option and its plan file. The server reads the same table for the kinds it accepts, the default sidecars and the child flags. Removing a tag keeps the files another cached tag still loads, such as an mmproj named after the removed quant. A sidecar removed on its own goes for every tag that resolves it. An unfinished download records the exact paths of its partials when it starts. A removal deletes those without asking the hub, and keeps a partial another recorded download also writes. The record is taken once the child is spawned, and the files go under the models lock, so a download of the same tag cannot start while they are removed. * server: drop stale partials and refuse a tag that resolves nothing Recording a download again deletes the partials of the earlier record that the new plan no longer writes, such as those of an older hub revision or of a sidecar the new request leaves out. A posted tag that resolves no file of the repo is refused before the download starts. Removing a model follows a single path, whether the reload erased its entry during the wait or not. * server: fix the CI race between the router download tests The pytest workers run in parallel on one shared cache, so a download test could find the file another test had just fetched and see no progress, or start from the unfinished downloads record another test wrote. Each download test now runs on an empty cache of its own. * server: refuse a DELETE of an unknown name and decide downloads in load() A DELETE of a name that is neither a model nor a recorded download went to common_download_remove, and without a tag it wiped the whole repo cache, including the files of a loaded quant or of a download on its way. Such a name is now not found, as it was before. A POST that downloads over an existing entry checked its state outside the lock, so a load of the same model in between got its instance killed. The checks now run in load(), under the lock, right before the spawn. The fake load time of the tests applies to model loads only, since a download worker loads none. * server: add incomplete_sidecars to the /models API A sidecar added to a cached model and then paused kept its record, but the listing showed only the complete model, so a fresh UI lost the pause. The model entry now carries incomplete_sidecars, the kinds its unfinished download has not written yet; a POST with the kind resumes one and a DELETE with it drops its partial. * common : drop a kept partial when a resume gets 416 Now that partials survive between attempts, an old or complete partial makes every retry fail with the same 416. Delete it so the next attempt can start over from zero. Signed-off-by: Adrien Gallouët <angt@huggingface.co> * common: drop a kept partial when a resume gets 200 A server that ignores the range answers 200 to a resume, and the kept partial then fails every attempt the same way a 416 does. Drop it as well, so the next attempt starts over without a range. * server: refuse a download that shares a partial with one on its way Two downloads of quants that share a file, the mmproj or an mtp picked by the closest bits, appended to the same partial and renamed a file with interleaved bytes. A download whose plan writes a partial that a running download also writes is now refused until that one is done. * common : check the saved etag before resuming a partial download Before resuming, compare the etag saved with the partial to the one the server now serves. If it is missing or different, start over, so a resumed download never mixes bytes from two versions of the file. Signed-off-by: Adrien Gallouët <angt@huggingface.co> --------- Signed-off-by: Adrien Gallouët <angt@huggingface.co> Co-authored-by: Pascal <admin@serveurperso.com> Co-authored-by: Adrien Gallouët <angt@huggingface.co> |
||
|
Gabe Goodhart
|
1268edac57 |
model : Granite switch decision 256 options (#30693)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* feat(convert): Attempt to come up with 256 single-token, semantic option markers Each switch model needs to identify a collection of single-token option markers that can be mapped to their respective logits for the output probability distribution. The simple upper/lower/numbers set only yields 66 unique markers, but the model can support up to 255. The challenge is that which combinations of characters result in single-token markers is model- specific, so we do a discovery exercise at conversion time by looking through the specific model's vocab for single-token sequences of 1+ plain letter strings (eg `A`, `AA`, `BB`, etc). Branch: GraniteSwitchDecisionFullOptions AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * docs: Note granite-switch with 256 option markers Branch: GraniteSwitchDecisionFullOptions AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * style: Remove extra newline Branch: GraniteSwitchDecisionFullOptions AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(convert): Use the computed labels in the template so c++ side is source of truth Branch: GraniteSwitchDecisionFullOptions AI-usage: full (Claude + Sonnet 5.5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat: Use LEV case for GRANITE_SWITCH w/ computed labels Branch: GraniteSwitchDecisionFullOptions AI-usage: full (Claude + Sonnet 5.5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * docs: Consolidate g-s into 255 bucket Branch: GraniteSwitchDecisionFullOptions AI-usage: full (Claude + Sonnet 5.5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> |
||
|
Gabe Goodhart
|
23f7aabc16 |
model : Granite Switch Decision (#30676)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* feat(convert): Add decision.chat to support decision models that can also chat Branch: GraniteSwitchDecision AI-usage: full (Claude + Sonnet 5.5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(convert): Add conversion for Granite Switch as decision model The template follows from the usage sample: https://huggingface.co/ibm-granite/granite-switch-4.1-8b#guardian-and-decision-side-by-side Branch: GraniteSwitchDecision AI-usage: draft (Claude + Sonnet 5.5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(common): Add support for chat+decision models in common Branch: GraniteSwitchDecision AI-usage: full (Claude + Sonnet 5.5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(common): Add granite switch decision model enum Branch: GraniteSwitchDecision AI-usage: full (Claude + Sonnet 5.5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(server): Plumb through decision_chat bool to support decision+chat models Branch: GraniteSwitchDecision AI-usage: full (Claude + Sonnet 5.5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * feat(server): Add Granite Switch decision support Branch: GraniteSwitchDecision AI-usage: draft (Claude + Sonnet 5.5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * docs: Update server README for granite switch decisions / chat+decisions Branch: GraniteSwitchDecision AI-usage: draft (Claude + Sonnet 5.5) Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * Revert "feat(convert): Add decision.chat to support decision models that can also chat" This reverts commit 952ef567647e53ac9beda7a1f79d6927240d068d. * Revert "feat(common): Add support for chat+decision models in common" This reverts commit 93ed782e3a8b2fd5968732c6525c7ad29ad4bea2. * Revert "feat(server): Plumb through decision_chat bool to support decision+chat models" This reverts commit 97e9af60ddaccbd82f1eb0a488adff43b16ee095. * fix: If Granite Switch supports decisions, it always supports text too Branch: GraniteSwitchDecision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> * docs: Remove reference to decision.chat in server README Branch: GraniteSwitchDecision AI-usage: none Signed-off-by: Gabe Goodhart <ghart@us.ibm.com> |
||
|
texasich
|
7182ef71e6 |
common: honor HTTP_PROXY/HTTPS_PROXY env vars in http client (#21752)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* common: honor HTTP_PROXY/HTTPS_PROXY env vars in http client common_http_client() was ignoring standard proxy env vars, causing downloads via -hf and similar flags to fail silently in environments behind an HTTP proxy (corporate networks, air-gapped infra, etc.). Checks HTTPS_PROXY/https_proxy for https:// URLs and HTTP_PROXY/http_proxy for http:// URLs, in that order. If a proxy URL is set and parseable, passes it to httplib's set_proxy() and set_proxy_basic_auth() as appropriate. Malformed proxy URLs are silently ignored so existing callers aren't affected. Fixes #21694 * common: drop misleading comment about generic proxy fallback The comment referenced a 'generic one' that doesn't exist in the code — both HTTP_PROXY and HTTPS_PROXY are scheme-specific. Reviewer noted the mismatch. Addresses review feedback from @ServeurpersoCom. * common : warn when the proxy URL from the environment is malformed * common : honor NO_PROXY when selecting the http proxy * ci : retrigger checks (test-thread-safety flake) * common : bypass the proxy for loopback destinations llama cli talks to its spawned server on 127.0.0.1 through common_http_client, so with HTTP_PROXY set and no NO_PROXY the health polling went to the corporate proxy, which rejects loopback targets, and the wait loop never exits ("Loading model..." hangs forever). A remote proxy can never reach the client machine's own loopback, so these destinations are now always contacted directly, before NO_PROXY is consulted: localhost, *.localhost, 127.0.0.0/8, ::1 and 0.0.0.0. Similar to Go's net/http. Proxy handling for downloads and remote --server-base endpoints is unchanged. Signed-off-by: texasich <101962694+texasich@users.noreply.github.com> * docs: move proxy tip from README to server docs per review: the top level README stays lean; the proxy howto now lives with the server docs under Usage Signed-off-by: texasich <101962694+texasich@users.noreply.github.com> --------- Signed-off-by: texasich <101962694+texasich@users.noreply.github.com> Co-authored-by: texasich <texasich@users.noreply.github.com> Co-authored-by: texasich <texasich@gmail.com> Co-authored-by: marecabo <23156476+marecabo@users.noreply.github.com> |
||
|
Xuan-Son Nguyen
|
2851b3bcbf |
mtmd: add mtmd_audio_stream + support voxtral-realtime (#30629)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* mtmd: add mtmd_audio_stream + support voxtral-realtime * address review |
||
|
Said Taghadouini
|
2f729cfa4c |
mtmd: lightonocr does not require longest edge to always be 1540, native resolution is the default (#30600)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
Assisted-by: Codex |
||
|
Pascal
|
ba3c05f71d |
model: add Muse Spark Open (text, vision and audio) (#30616)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* model: add Muse Spark Open (text, vision and audio) Muse Spark Open is a latent space MoE: the residual stream stays at 8192 while the 256 experts live in 4096, with a projection down and up around them, top-8 sigmoid routing with a correction bias. Attention interleaves three sliding layers at a 2048 window with rope for one global layer without rope, with a sigmoid output gate and a weightless QK-norm. Residuals are norm preserving, their gate is folded at conversion time. Vision reuses the Muse Glimmer tower. Audio is a causal convolutional pyramid with no attention and no spectrogram, four stages folding 16, 6, 5 and 4 samples into the channel dim, one embedding per 80 ms of 24 kHz. The quantizer keeps the latent projections and the attention gate at Q8_0, a little over 1% of the weights, since the whole residual stream passes through them and the low-bit types turned the output into a repeating loop. The model saver writes the post norm epsilon, and the arch is registered in test-llama-archs. Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: Pedro Cuenca <pedro@huggingface.co> * Muse Spark: drop the redundant forward expand * Muse Spark: drop the redundant forward expand and tighten the comments * Update src/llama-quant.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Update src/llama-graph.cpp Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Update conversion/muse_spark.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Update conversion/muse_spark.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Update conversion/muse_spark.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> * Update conversion/muse_spark.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: Pedro Cuenca <pedro@huggingface.co> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> |
||
|
Kartik Gulia
|
cd0a5b6a18 |
models : add Nemotron Super 3.5 VL and Nemotron Bolt 74B A9B support (#30520)
* add support * remove unneeded bos fix * add video flag * fix flake error * remove video-specific budgeting * convert : fix lilicorr-dflash bid clash * spec : enforce lilicorr n_tokens condition * spec : fix lilicorr walk [no release] |
||
|
Pascal
|
61ad80bf93 |
server: fix the off-by-two in the SWA reuse threshold (#30580)
The first evaluated token attends to the positions above pos - n_swa, so a SWA state starting at pos - n_swa + 1 holds the whole window, but the reuse threshold asked for two more cells. A saved SWA state keeps only the cells inside its window, so a state read back from a slot file or from the RAM prompt cache was always rejected and the slot rolled back to a checkpoint. The SWA threshold now matches the window, while a state without SWA keeps the exact bound. The test harness gains override_kv, used to give the SWA test model a window smaller than the test context. [no release] |
||
|
Pascal
|
ea736c1d5c |
server: save the draft state in the slot file (#30574)
The slot file held the target context and the context checkpoints, but never the draft context, so a restored slot drafted with an empty or stale draft KV until the cache refilled. The file now ends with the live draft state, the full draft context and the speculative state, in the same appendix style as the checkpoints. A draft state that does not fit the draft context is dropped and the draft starts empty, while older files and older servers keep working. [no release] |
||
|
Pascal
|
b0d239164b |
server: prefer empty slots and keep states overwritten by child tasks (#30567)
Slot selection takes free slots by id or by last use only, so child tasks often land on a slot holding a parent state while empty slots are idle, and overwrite it without saving it to the prompt cache. Free slots are now ordered empty first, then least recently used, for both the LRU fallback and the child slots, and a child task saves the state it overwrites. With 4 grouped decision requests in flight and 12 states in rotation, prompt cache reads go from 5 to 36. [no release] |
||
|
Aleksander Grygier
|
fd55cb98fe |
fix(ui): take the agentic turns default from DEFAULT_AGENTIC_CONFIG (#30582)
The registry hard-coded 10 for AGENTIC_MAX_TURNS while DEFAULT_AGENTIC_CONFIG said 100. The registry seeds SETTING_CONFIG_DEFAULT, so the loop's `|| DEFAULT_AGENTIC_CONFIG.maxTurns` fallback never fires and users got 10 turns. Reference the shared constant, as the adjacent MCP timeout does. Assisted by: llama-ui (ggml-org/Qwen3.8-Flash-Next:IQ4_NL) |
||
|
Pascal
|
2639a3b4e0 |
server: save only restorable slot states to the prompt cache (#30498)
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* server: save only restorable slot states to the prompt cache The idle-slot save loop stores every idle slot whenever a task starts, while only completion tasks restore from the prompt cache. Rerank, embedding, decision and infill states fill the cache up to its limit without ever being read back, and slots already cached are offered again at every task start. Each slot now tracks with cache_pending whether its state is restorable and missing from the cache: a completion release or an explicit slot restore sets it, prompt_save consumes it, prompt_clear resets it. Both save paths share this rule, and the LRU path restores only for completion tasks. * server: test that decision slots skip the prompt cache Decision tasks never mark their slot as cacheable, so starting new tasks must not save their idle slots to the RAM prompt cache, with or without a unified KV cache. * server: restore any reusing task from the prompt cache Embedding and rerank tasks never reuse their state, so their slots are cleared on release like child slots and the cache only holds states that a later task can restore. Infill and decision tasks now restore from the cache too, which replaces the cache_pending flag. Decision models disable the prompt cache unless --cache-ram is given, since their states seldom repeat. |
||
|
Georgi Gerganov
|
3fb47677ea |
common : add --spec-draft-n-ubatch (#30459)
* common : add --spec-draft-n-ubatch Allow configuring the physical batch size of the draft context. The drafters feed it at most the drafted tokens per sequence and chunk larger batches by its own n_ubatch, so it does not need the ubatch of the target. A smaller value shrinks the draft compute buffer and leaves more VRAM for the target KV cache (the draft is fitted together with the target). The default 0 keeps the current behavior of following --n-ubatch. Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next * cont : remove comment [no ci] |
||
|
Yuri Khrustalev
|
04f7226a2c |
server : escape the special tokens without std::regex (#30243)
The d1-omni and kev prompts escape "<|name|>" in the request text with std::regex_replace. libstdc++ recurses for each char of the name, so a state of "<|" and 30000 letters overflows the stack of the HTTP thread and llama-server dies with SIGSEGV. server_decision_escape_special_tokens() writes the same text with a loop. test-server-component compares it with the regex on every string of up to 6 chars of "<|>a-". |
||
|
Xuan-Son Nguyen
|
ff026f8c0a |
chat: replace output string with parts (#30389)
* chat: replace output string with parts * reserve first |
||
|
Xuan-Son Nguyen
|
efe3626b11 |
model: support vllm d3 models (#30383)
* models: support pplx-decider * vllm d2 * update * clarify naming rule * stricter validation * d3 |
||
|
Xuan-Son Nguyen
|
097cf33d94 |
model: support vllm decision 2 model (#30378)
* models: support pplx-decider * vllm d2 * update * clarify naming rule * stricter validation * fix ci * rm redundant filter_tensors |
||
|
Georgi Gerganov
|
972b5fcc19 |
llama : report compute graph reallocations during inference (#30329)
Add an optional ggml_backend_sched_realloc_callback that user code registers with ggml_backend_sched_set_realloc_callback to be notified when the buffers reserved for a graph are not sufficient and have to be reserved again. The callback reports whether the backend assignments and the graph size changed since the previous scheduling. llama_context registers the callback when creating the scheduler (wrapped into the new llama_context::sched_new helper) and treats the reallocations that follow a change of the backend assignments as expected, counting the rest by graph size - unchanged or changed from the previously scheduled graph. Both counters are reported on one line in the perf stats. Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next |
||
|
shamnadxyz
|
2f160ba59a |
ui: fix text attachment preview overflow (#29898)
* ui: fix text attachment preview overflow Set the max height to 80dvh and the max width to full on screens below medium and 80vw from medium onward. * style: run formatter |
||
|
Georgi Gerganov
|
5c8854fd88 |
rpc : turn GGML_RPC_DEBUG into a verbosity level and add logs (#29544)
* rpc : turn GGML_RPC_DEBUG into a verbosity level and add logs GGML_RPC_DEBUG is now parsed as a number: 0/unset disables debug logs, 1-3 emit increasingly detailed output (events, per-command trace, transport detail). Non-numeric values fall back to 1. The duplicated env/macro blocks in ggml-rpc.cpp and transport.cpp are replaced by a shared log.h, which becomes the single choke point for all logging of the RPC backend: LOG_ERROR/LOG_WARN/LOG_INFO for unconditional severity logs and LOG_DBG/LOG_DBG2/LOG_DBG3 for the verbosity-gated ones. The transport files no longer need ggml-impl.h, and the server banner now also goes through the ggml logger (stderr). Missing logs are added on both the client (handshake, buffer ops, tensor transfers, graph computes, cache decisions) and the server (per-command dispatch, graph nodes), including the negotiated transport via the new socket_t::transport_name(). Assisted-by: pi:llama.cpp/MiMo-V2.6-Flash-RL * rpc : align log levels with the documented verbosity semantics The first pass introduced GGML_RPC_DEBUG levels, but many call sites did not follow the documented semantics and some logs bypassed the macros entirely: - Route the remaining raw GGML_LOG_* call sites through the local LOG_* macros so that log.h stays the single choke point for RPC logging - Emit per-tensor transfers and per-command handler traces at level 2, and one-shot lifecycle events (backend and buffer type creation, buffer allocation, graph compute, graph cache eviction) at level 1 - Drop the duplicated enqueue trace in rpc_dispatcher::send, work() already logs every dispatched command together with its round-trip timing - Promote degraded-operation events to LOG_WARN: remote allocation failure, RDMA falling back to TCP, unexpected peer disconnect - Log graph cache eviction on both sides, since client and server must stay in sync for the incremental graph update to be valid - Distinguish Unknown command (opcode out of range) from Unhandled command (valid opcode with no handler) - Fix format specifiers (%zu for size_t, %u for device ids, 0x for hex pointers) and a missing newline in an error log - Document that llama.cpp applications drop ggml debug records unless the verbosity threshold is raised with -lv 5 Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next |
||
|
Pascal
|
23b0202a18 |
server: leave a busy slot untouched when a request pins it (#30295)
A request asking for a busy id_slot still ran the prompt cache update on that slot before being deferred. When the RAM cache held a better match, it was loaded into the slot while another request was still generating there, and that generation continued on the wrong context. The busy slot is now returned as is and the request waits for it. |