When every device reports zero free memory the split points are
normalized by a zero sum, the NaN split sends layers to a device index
past the end and loading aborts with vector::_M_range_check.
- add rpc_log_format_size() and print byte counts in KiB/MiB/GiB across the
RPC client, server and transport traces
- add rpc_log_format_tensor() and include tensor name, op, type and shape in
the set/get/memset/copy/init tensor logs
- fix rpc_cmd_name() returning UNKNOWN for SET_TENSOR_2D, GET_TENSOR_2D,
COMM_INIT, COMM_ALLREDUCE and COMM_FREE
- move the log helpers to ggml-rpc/log.{h,cpp} under a common rpc_log_ prefix
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
tojson printed floats through a C++ stream (6 significant digits, no
".0"), so 48.858844 became 48.8588 and 1.0 became 1 in tool definitions,
tool-call arguments and other template output. Write the shortest digits
that read back as the same value, as json.dumps does. The string and safe
filters of a float use the same code.
These graphs only produce embeddings, and build_pooling() already
expands its output for every pooling type, so the expand at the end of
the model graph is redundant.
The RDNA3.5 config table selects a 64-row / 128-thread tile up to J=32 and a
128-row / 256-thread tile above that. On gfx115x the smaller tile leaves enough
LDS free for a second workgroup per CU, while the larger one does not.
Extend the 64-row tile to the (type, J) pairs where it measured faster on
gfx1151, and leave every other entry untouched. J=96 is deliberately excluded
because it did not benefit.
test-backend-ops perf on gfx1151, 100 MUL_MAT shapes over the affected types
(N in 48..512, M both padded and unpadded), 3 rounds: median +2.1%, best
+26.0%, no shape outside run-to-run noise. 5 MUL_MAT_ID shapes: median +8.0%.
Assisted-by: Claude Opus 5
* llama : build the k-pool selection in the causal mask layout
- the glm5-next indexer folded the causal mask in through a [n_kv, n_tokens]
view: one reshape of the mask ahead of the add, and one of the result ahead
of every layer reading the selection, including the shared-indexer layers
- build the selection directly in the mask layout, as qwen4exp already does,
and drop the now-degenerate reshape in its attention as well
- assert the selection width matches the input it is taken from
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* cont : remove comment
* unicode : add opt-in NFC normalization for Unigram tokenizers
Note: used help from Codex, Sol 6.1 Medium
* Fixing initial bot comments + linter error (using existing out instead of print)
* removed test_nfc_tokenizer
The d1-omni and kev prompts escape "<|name|>" in the request text with
std::regex_replace. libstdc++ recurses for each char of the name, so a
state of "<|" and 30000 letters overflows the stack of the HTTP thread
and llama-server dies with SIGSEGV.
server_decision_escape_special_tokens() writes the same text with a loop.
test-server-component compares it with the regex on every string of up to
6 chars of "<|>a-".
* hexagon: add configurable HTP DCVS/HMX power mode
* hexagon: address PR review feedback (drop CLI flag, align QAIRT power modes, comment/log cleanup)
* hexagon: add --hex-power-mode flag to run.py
Hooks GGML_HEXAGON_POWER_MODE into the unified run.py runner, following
the existing --hex-nhvx pattern. The env var must be set before the
process touches the Hexagon backend registry (see comment in
ggml_hexagon_init), so this is exposed as a runner flag rather than a
common/arg.cpp CLI option.
* hexagon: keep HTP awake for DCVS power profiles
* Apply suggestion from @max-krasnyansky
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
The no-cache attention pads K, V and the mask to 256, but T5 also adds
a relative position bias of [n_tokens, n_tokens], so the add with KQ
fails unless n_tokens divides the padded width. Pad the bias with
zeros up to the KV dimension of the mask, like K and V.
* llama : do not map foreign ranges into a backend buffer
A buffer created from a host pointer pins everything that it wraps. When
several contexts share the same file (lazy tensors, MoE tensors kept on the
CPU with -cmoe), the single buffer that spans the first and the last tensor
of a context also wraps the ranges owned by the other contexts, pinning data
that the backend never reads.
For models with per-layer token embeddings this pins the whole lazy tensor
into the GPU buffer (27 GiB for Qwen3.8-Flash-Next IQ4_NL), defeating the
point of reading it lazily.
Instead, compute the file ranges of each context and subtract the ranges
owned by the other contexts, then create one buffer per range. Weights that
are mapped in more than one context (e.g. token_embd.weight) keep being part
of the range of every context that uses them.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* llama : enable prefetch on macOS
The prefetch hint of the tensors marked as prefetchable is a no-op on macOS,
so their pages are faulted in one by one while the graph runs. madvise
(MADV_WILLNEED) behaves as expected here, so add the platform to the guard.
With the lazy per-layer token embeddings of Qwen3.8-Flash-Next IQ4_NL read
from a cold file cache, prompt processing goes from 356 / 375 / 475 t/s to
735 / 749 / 788 t/s for pp512 / pp2048 / pp4096.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* llama : use a plain buffer list per context
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* graph: skip the LM head for pooled embeddings
A context that extracts pooled embeddings outputs one vector per
sequence, but every model still expanded its LM head, so the graph
computed n_vocab logits per token and the context kept a host buffer
for them.
The models now only declare t_logits, and set_outputs() expands it
unless the context extracts embeddings with a pooling type other than
NONE. Since the toggle changes the graph shape for a pooled context,
set_embeddings() schedules a re-reserve in that case.
* graph: drop the redundant t_embd expand
Address review from @CISC: with embeddings on, build_pooling() already
expands its output for every pooling type, and a decoder graph without
embeddings does not read t_embd, so set_outputs() only expands
t_logits.
* graph: add build_logits() for the LM head output
Address review from @ggerganov: the LM head is expanded by a dedicated
build_logits(), called from llama_model::build_graph() like
build_pooling(), and set_outputs() only marks the outputs again.
* graph: call build_logits() before build_sampling()
Address review from @ggerganov: build_sampling() reads t_logits, so the
LM head is built first. A pooled context then skips the sampling layers
too, since it has no logits to sample from.
* nit
* tests : consolidate the context state tests into test-llama-context
test-save-load-state, test-recurrent-state-rollback and
test-state-restore-fragmented check the same object from three places:
the KV cache and the recurrent state of a llama_context, saved, loaded,
copied, rolled back. Each had its own main, its own loop over the
models, its own copy of the token and the comparison helpers, and a run
of all three loaded the same models several times.
They are one binary now, and the case is the unit of the test:
tests/llama-context/test-case.h the interface a case implements
tests/llama-context/cases-kv.cpp the cases of the KV cache
tests/llama-context/cases-rs.cpp the cases of the recurrent state
tests/llama-context/test-util.cpp tokens, generation, comparison
tests/llama-context/test-runner.cpp the models of a case, on threads
tests/llama-context/test-log.cpp the log file of a model
A run is a table: a row per model, a column per case, a cell holds the
verdict of that pair. The models load on several threads (-j, --threads,
min(cores, 4) by default), the table is printed when it is complete and
the log of a model that failed is printed after it, in the order of the
rows. A case returns a verdict and the reason for it, and the runner is
the only one who reports - the test does not use tests/testing.h, a run
over 130 models is a matrix of verdicts and not a tree of subtests.
Staging no longer travels through the environment: llama_model_set_attn_rot
replaces the LLAMA_ATTN_ROT_DISABLE variable, which a model loaded in one
thread used to change for the models the other threads were loading, and
llama_memory_get_buffers is added for the cases that copy the buffers of
a memory. common_log_set_stdo keeps the console quiet while the table is
being built.
ggml_aligned_malloc allocates with posix_memalign when the thread
sanitizer is on. On macOS it allocates with the Mach VM, which the
sanitizer does not intercept, so it never sees a page range change hands
and reports the first write into recycled memory as a race - 30 of them
in this test, and every one of them false.
Assisted-by: pi:llama.cpp/Qwen3.8-Flash-Next
* cont : run single-threaded by default
* cont : try to build the tests on all platforms
* cont : trim down some body fat
* cont : fix
* cont : fix the fix
* ggml-cuda: gate CUDA graph debug logs behind GGML_CUDA_GRAPH_DEBUG
* ggml-cuda: cache env var lookups in static const instead of re-reading per call
* ggml-cuda: read env vars exactly once via ggml_cuda_env_flag/ggml_cuda_env_int
* ggml-cuda: consolidate env flag reads into single static accessors
* ggml-cuda : drop env helper comments
Assisted-by: pi:llama.cpp/Qwen3.8-27B
* cont : remove NDEBUG gates for CUDA graph logs
* vulkan : split matmul dispatch when it would exceed maxComputeWorkGroupCount
* vulkan : fix narrowing in chunked matmul dispatch
* vulkan : zero fill the A side of mul_mmq for k steps past the end
The A side skipped the shared memory write for the k steps past end_k, while the B side was already told the bound and zero filled. The A registers are read for all BK_STEP steps, so the stale scale went into the dot product and produced NaN at k = 32.
Pass the bound to block_a_to_shmem and write zeros when it is false, for every DATA_A type, mirroring the B side.
Also assert that the B and D subbuffer sizes are not VK_WHOLE_SIZE before they are adjusted per chunk, and add two batched cases of the chunked shape.
Signed-off-by: losewayy <104015127+losewayy@users.noreply.github.com>
* vulkan : remove redundant matmul tile adjustment
---------
Signed-off-by: losewayy <104015127+losewayy@users.noreply.github.com>
* tests: cover weighted scalar broadcasts
Assisted-by: OpenCode
* ggml : support extended scalar DMA binary ops
Assisted-by: OpenCode
* hex-binary: remove all non-dma paths
* hex-binary: add support for non-contig strides and cleanup dma preable a bit
* hex-binary: cleanup the gating and simplify dma retry
* hex-geglu-erf: add a gate for F32 only, we dont support F16 for this op
* tests: cover weighted repeat broadcast
Assisted-by: OpenCode
* hex-binary: update REPEAT and COMPLEX kernels to use dma as well (missed them last time)
* hex-add-id: add dma support for src2 (idx)
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
Georgi Gerganov