CI (3rd-party) / ubuntu-24-llguidance (push) Has been cancelled
CI (snapdragon) / android-ndk-snapdragon (push) Has been cancelled
CI (snapdragon) / linux-iot-snapdragon (push) Has been cancelled
CI (android) / default (push) Has been cancelled
CI (android) / ndk (push) Has been cancelled
CI (android) / arm64 (push) Has been cancelled
CI (apple) / macos-latest-arm64 (push) Has been cancelled
CI (apple) / macos-latest-x64 (push) Has been cancelled
CI (apple) / macos-latest-ios-xcode (push) Has been cancelled
CI (apple) / macos-latest-tvos (push) Has been cancelled
CI (apple) / macos-latest-visionos (push) Has been cancelled
CI (cpu) / build-cmake-pkg (push) Has been cancelled
CI (cpu) / ubuntu (arm64, ubuntu-24.04-arm) (push) Has been cancelled
CI (cpu) / ubuntu (x64, ubuntu-22.04) (push) Has been cancelled
CI (cpu) / windows / arm64 (push) Has been cancelled
CI (cpu) / windows / x64-cpu-static (push) Has been cancelled
CI (cpu) / windows / x64-openblas (push) Has been cancelled
CI (cross) / debian-13-loongarch64-cpu-cross (push) Has been cancelled
CI (cross) / debian-13-loongarch64-vulkan-cross (push) Has been cancelled
CI (cross) / ubuntu-24-riscv64-cpu-spacemit-ime-cross (push) Has been cancelled
CI (CUDA, ubuntu) / cuda (push) Has been cancelled
CI (CUDA, ubuntu) / hip (push) Has been cancelled
CI (CUDA, ubuntu) / musa (push) Has been cancelled
CI (ibm) / ubuntu-24-s390x (VXE OFF) (push) Has been cancelled
CI (ibm) / ubuntu-24-s390x (VXE ON) (push) Has been cancelled
CI (ibm) / ubuntu-26-zdnn-s390x (push) Has been cancelled
CI (ibm) / ubuntu-24-ppc64le (push) Has been cancelled
CI (opencl) / windows-2025-opencl-adreno (push) Has been cancelled
CI (openvino) / ubuntu-24-openvino (push) Has been cancelled
CI (openvino) / openvino-windows-2022 (push) Has been cancelled
CI (riscv) / ubuntu-cpu-riscv64-native (push) Has been cancelled
CI (sanitize) / ctest ([self-hosted X64 Linux], UNDEFINED) (push) Has been cancelled
CI (sanitize) / ctest (ubuntu-24.04, ADDRESS) (push) Has been cancelled
CI (sanitize) / ctest (ubuntu-24.04, THREAD) (push) Has been cancelled
CI (sycl) / ubuntu-24-sycl (fp16, ON) (push) Has been cancelled
CI (sycl) / ubuntu-24-sycl (fp32, OFF) (push) Has been cancelled
CI (sycl) / windows-latest-sycl (push) Has been cancelled
CI (virtgpu) / ubuntu-24-virtgpu (push) Has been cancelled
CI (vulkan) / ubuntu-arm64 (push) Has been cancelled
CI (vulkan) / ubuntu-llvmpipe (push) Has been cancelled
CI (vulkan) / windows / arm64 (push) Has been cancelled
CI (vulkan) / windows / x64 (push) Has been cancelled
CI (wasi) / ubuntu-webgpu (push) Has been cancelled
CI (wasm) / ubuntu-webgpu (push) Has been cancelled
CI (webgpu) / format (push) Has been cancelled
Check vendor / check-vendor (push) Has been cancelled
CI (self-hosted CPU backend) / cpu-x64-high-perf (push) Has been cancelled
CI (self-hosted CPU backend) / cpu-arm64-high-perf-graviton4 (push) Has been cancelled
CI (self-hosted CUDA backend) / gpu-cuda (push) Has been cancelled
CI (self-hosted CUDA backend) / gpu-rocm (push) Has been cancelled
CI (self-hosted KleidiAI backend) / cpu-arm64-graviton4-kleidiai (push) Has been cancelled
CI (self-hosted Metal backend) / gpu-metal (push) Has been cancelled
CI (self-hosted OpenVINO backend) / gpu-openvino-low-perf (push) Has been cancelled
CI (self-hosted Vulkan backend) / gpu-vulkan-nvidia-cm (push) Has been cancelled
CI (self-hosted Vulkan backend) / gpu-vulkan-nvidia-cm2 (push) Has been cancelled
CI (self-hosted Vulkan backend) / gpu-vulkan-apple (push) Has been cancelled
CI (self-hosted Vulkan backend) / gpu-vulkan-intel-linux (push) Has been cancelled
CI (self-hosted Vulkan backend) / gpu-vulkan-intel-windows (push) Has been cancelled
CI (self-hosted WebGPU backend) / gpu-webgpu-nvidia (push) Has been cancelled
CI (self-hosted WebGPU backend) / gpu-webgpu-apple (push) Has been cancelled
Code Style Checker / model-naming (push) Has been cancelled
EditorConfig Checker / editorconfig (push) Has been cancelled
HIP quality check / ubuntu-22-hip-quality-check (push) Has been cancelled
Models Backend Check / cuda (push) Has been cancelled
Models Backend Check / metal (push) Has been cancelled
Models Backend Check / rocm (push) Has been cancelled
Models Backend Check / vulkan-nvidia (push) Has been cancelled
Models Backend Check / vulkan-amd (push) Has been cancelled
Models Backend Check / webgpu-nvidia (push) Has been cancelled
flake8 Lint / Lint (push) Has been cancelled
Release / check-release (push) Has been cancelled
Server (sanitize) / server (RelWithDebInfo, ADDRESS) (push) Has been cancelled
Server (sanitize) / server (RelWithDebInfo, UNDEFINED) (push) Has been cancelled
Server (self-hosted) / server-metal (push) Has been cancelled
Server (self-hosted) / server-cuda (push) Has been cancelled
Server (self-hosted) / server-kleidiai (push) Has been cancelled
Server / ubuntu (push) Has been cancelled
Server / windows (push) Has been cancelled
UI (self-hosted) / Build static output (push) Has been cancelled
UI / Build static output (push) Has been cancelled
CI (snapdragon) / Test on QDC Device (QCS9075M) (push) Has been cancelled
CI (snapdragon) / Test on QDC Device (SM8850) (push) Has been cancelled
CI (apple) / macos-latest-swift (generic/platform=iOS) (push) Has been cancelled
CI (apple) / macos-latest-swift (generic/platform=macOS) (push) Has been cancelled
CI (apple) / macos-latest-swift (generic/platform=tvOS) (push) Has been cancelled
Release / macos-cpu (arm64, arm64, -DGGML_METAL_EMBED_LIBRARY=ON -DCMAKE_OSX_DEPLOYMENT_TARGET=13.3, macos-26) (push) Has been cancelled
Release / macos-cpu (x64, x64, -DGGML_METAL=OFF -DCMAKE_OSX_DEPLOYMENT_TARGET=13.3, macos-15-intel) (push) Has been cancelled
Release / ubuntu-cpu (arm64, ubuntu-24.04-arm) (push) Has been cancelled
Release / ubuntu-cpu (s390x, ubuntu-24.04-s390x) (push) Has been cancelled
Release / ubuntu-cpu (x64, ubuntu-22.04) (push) Has been cancelled
Release / ubuntu-vulkan (arm64, ubuntu-24.04-arm) (push) Has been cancelled
Release / ubuntu-vulkan (x64, ubuntu-22.04) (push) Has been cancelled
Release / ubuntu-cuda (13.4, arm64) (push) Has been cancelled
Release / ubuntu-cuda (12.8, x64) (push) Has been cancelled
Release / ubuntu-cuda (13.4, x64) (push) Has been cancelled
Release / android-arm64 (push) Has been cancelled
Release / android-arm64-snapdragon (push) Has been cancelled
Release / linux-arm64-snapdragon (push) Has been cancelled
Release / ubuntu-24-openvino (push) Has been cancelled
Release / windows-openvino (push) Has been cancelled
Release / windows-cpu / arm64 (push) Has been cancelled
Release / windows-cpu / x64 (push) Has been cancelled
Release / windows-rocm (10.0.0, x64, gfx1010;gfx1011;gfx1012;gfx1030;gfx1031;gfx1032;gfx1033;gfx1034;gfx1035;gfx1036;gfx1100;gfx1101;gfx1102;gfx1103;gfx1150;gfx1151;gfx1152;gfx1153;gfx1200;gfx1201) (push) Has been cancelled
Release / windows (arm64, opencl-adreno, -G "Ninja Multi-Config" -D CMAKE_TOOLCHAIN_FILE=cmake/arm64-windows-llvm.cmake -DCMAKE_PREFIX_PATH="$env:RUNNER_TEMP/opencl-arm64-release" -DGGML_OPENCL=ON -DGGML_OPENCL_USE_ADRENO_KERNELS=ON, ggml-opencl) (push) Has been cancelled
Release / windows (arm64, vulkan, -G "Ninja Multi-Config" -DGGML_VULKAN=ON -D CMAKE_TOOLCHAIN_FILE=cmake/arm64-windows-llvm.cmake, ggml-vulkan) (push) Has been cancelled
Release / windows (x64, vulkan, -DGGML_VULKAN=ON, ggml-vulkan) (push) Has been cancelled
Release / windows-cuda (13.4, arm64) (push) Has been cancelled
Release / windows-cuda (12.4, x64) (push) Has been cancelled
Release / windows-cuda (13.4, x64) (push) Has been cancelled
Release / windows-sycl (push) Has been cancelled
Release / ubuntu-24-sycl (fp16, ON) (push) Has been cancelled
Release / ubuntu-24-sycl (fp32, OFF) (push) Has been cancelled
Release / ubuntu-24-rocm (10.0.0, x64, gfx908;gfx90a;gfx942;gfx950;gfx1010;gfx1011;gfx1012;gfx1030;gfx1031;gfx1032;gfx1033;gfx1034;gfx1035;gfx1036;gfx1100;gfx1101;gfx1102;gfx1103;gfx1150;gfx1151;gfx1152;gfx1200;gfx1201) (push) Has been cancelled
Release / ios-xcode (push) Has been cancelled
Release / ui-build (push) Has been cancelled
UI (self-hosted) / Checks (push) Has been cancelled
UI (self-hosted) / E2E Tests (push) Has been cancelled
UI / Checks (push) Has been cancelled
UI / E2E Tests (push) Has been cancelled
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* opencl: add Adreno depthwise conv2d
Assisted-by: OpenCode
* opencl: use local program for depthwise conv2d
Assisted-by: OpenCode
* opencl: use singed type for args
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* Multi-column on Intel XMX matrix engines refactored
Multi-column (2 to 80 activation columns) mul_mat_vec_q for reordered K-quant and Q8_0 weights on the Intel XMX matrix engines (int8 DPAS, ESIMD), for the SYCL backend. It targets speculative decoding verification (2 to 8 tokens per step) and short prompt batches (9 to 80 tokens). Three types also use it for a single column (Q2_K, Q3_K, Q8_0).
* Fixed regression bug
real bug found: the first version produced NaN whenever the row count wasn't a multiple of 4
* Removed GGML_TYPE_Q8_0 activation
* rewritten xmx_mul_mat
64-byte alignment and 16-byte pitch check on the activations
* Implemented fused gate/up kernel
One launch computes up * act(gate) for both FFN weights, reading the activations once and applying SWIGLU or GEGLU in the epilogue. It covers up to 16 columns and works for all six XMX types.
* Removed the temporary XMX_NOGLU environment switch
* Q2_K min-sum optimization
Q2_K weight is d·scl·q − dmin·mc, where scl (the 4-bit scale) and mc (the 4-bit min code) apply to each half of a 32-weight group
* Added dispatch check and GRF sweep
tg128 on Q4_K_M goes from 26.67 to 28.23 tok/s on the B70 (+5.9%).
Q6_K is unchanged.
MUL_MAT passes 1727/1727 and MUL_MAT_VEC_FUSION 1361/1361.
Perplexity at batch 1 is 5.2064 with XMX and 5.2058 without.
* Renamed variables
* Added 2 environment variables
GGML_SYCL_XMX_MAX_COLS (default 80, any value 0 or higher) and GGML_SYCL_XMX_GLU_MAX_COLS (default 16, range 0 to 16): the largest batch for the fused gate/up kernel. It is clamped to 16 because that is the most the fused kernel is compiled for, so the variable can only lower it.
* Update esimd.hpp
1. All five quant structs were repeating the same boilerplate, so I moved it into shared pieces.
2. A new helper xmx_term does one DPAS, converts it to float and scales it per row. The four branches are now one or two lines each, and the shared Cb += z * d8r is written once.
The refactor passes all three op tests. MUL_MAT_VEC_FUSION is 1369/1369, MUL_MAT is 2281/2281 (the count includes your uncommitted f32 tests) and MUL_MAT_ID is 937/937.
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* sycl: fix CONV_2D wrong results for non-contiguous input
The kernel indexes the `input` as a contiguous [W, H, C, N] tensor, but
supports_op claimed the op for any layout. A non-contiguous (e.g. `CWHN`
permuted) input was read with the wrong element offsets, silently
producing wrong results.
Gate the CONV_2D op on `input` and `kernel` contiguity, matching the
CUDA backend contract, and assert input contiguity in the op.
`test-backend-ops -b SYCL0 -o CONV_2D`: goes from 16 failures to 0,
none.
```text
2006/2006 tests passed
Backend SYCL0: OK
Backend 2/2: CPU
Skipping
2/2 backends passed
OK
```
Assisted-by: pi (unsloth/Qwen3.8-27B-GGUF:Q4_K_M via llama.cpp)
* sycl : CONV_2D cwhn=1 support
Make the SYCL backend compute `GGML_OP_CONV_2D` correctly for `cwhn=1` (CWHN
channel-order) inputs and kernels, so the previously failing test-
backend-ops CONV_2D cases pass on SYCL instead of being skipped or
miscomputed.
- Implement CWHN layout support in `ggml/src/ggml-sycl/conv2d.cpp`
(`ggml_sycl_op_conv2d`).
- Update SYCL `supports_op` for `GGML_OP_CONV_2D` in
`ggml/src/ggml-sycl/ggml-sycl.cpp` to accept both layouts.
Assisted-by: pi (Greatjedi/Qwen3.8-27B-Intel-Arc-Tuned-GGUF via llama.cpp)
* sycl docs : update ops for SYCL
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* opencl: add F32-to-I32 ARGMAX
Assisted-by: OpenCode
* opencl: init to 0 to be consistent with cpu
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* llama: create new fused op for QSA scoring
* enroll glm5.3-next too
* tests : initialize the remaining tensors in test_kpool_mask
The harness sentinels were left uninitialized, so leftover memory could read as NaN and fail the NaN check.
* ggml: generalize the k-pool mask op into ggml_top_k_mask
The k-pool mask op only covered the pooled QSA selection of qwen4exp
and glm5-next, while the DSA models and the DeepSeek V4 CSA build the
same top-k mask by hand with fill, set_rows and add.
ggml_top_k_mask(mask, top_k, score, blocks, extra) keeps the mask
cells of the live picks and the extra cells, and sets the rest to
-inf. Every row tensor is 4D in the layout of top_k, and the mask is
broadcast to those rows, so a selection can also be made per head.
score, blocks and extra are optional: without score every pick is
live, without blocks a pick is a cell, and extra holds the cells a
row keeps without a pick.
llm_graph_context::build_fused_top_k_mask views the row tensors in
the layout of kq_mask and registers the fused node. The k-pool
models, the DSA attention of build_attn and the DeepSeek V4 CSA mask
use it when the probe enables the op, and keep their current path on
backends without it.
---------
Co-authored-by: Pascal <admin@serveurperso.com>
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* ggml-opencl : add IM2COL_3D support
Assisted-by: OpenCode
* opencl: use same type for kernel and host
* opencl: fix for older adreno
---------
Co-authored-by: Li He <lih@qti.qualcomm.com>
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* ggml-cpu : compute f32 gelu_quick and geglu_quick without the fp16 table
* ggml-cpu : keep the GGML_GELU_QUICK_FP16 table path
* ggml-cpu : add the VXE gelu_quick path
* tests : add test-gelu-quick for the f32 CPU kernels
* tests : drop inf/nan inputs from test-gelu-quick
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* llama : scale moe down projection per expert token
Replace the global amax rescaling for the MoE down projection with a per-expert-token factor computed by a POOL_1D max reduction along the intermediate dimension. Undo the scale on the output with the same exact power-of-two factor. Handle POOL_1D in the Meta split-state logic so a split along axis 0 becomes a replicated per-device partial result, enabling tensor parallelism.
Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* cont : simplify comments
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* llama : refactor amax src1 scaling result
Return the rescaled activation and the optional scale tensor through a small struct instead of an output pointer.
Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* cont : add TODO + clean-up
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* metal : optimize argmax and pool_1d
Add a two-pass partial reduction for large-row argmax and use cooperative threadgroup reductions for pool_1d. Allocate scratch space for the argmax partials through the Metal buffer allocator.
Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* tests : add pool_1d perf cases
Add representative POOL_1D max cases for the MoE down-projection amax shapes, reducing the whole intermediate dim.
Assisted-by: pi:llama.cpp/ggml-org/DeepSeek-V4-Flash-Vision-Exp-GGUF (Mac Studio, M2 Ultra, 192GB)
* cont : minor fix
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* opencl: fix OOB read in `mul_mv_mxfp4_f32_flat`
* opencl: remove large-N guard that forces CPU weight placement
* this condition claims that large batch produces incorrect results on
adreno and cannot be reproduced
* model loader uses N512 to probe op support, resulting in cpu fallback
* thus this condition does not seem necessary
* opencl: cap num of subgroups by workgroup size for q4_0 mc3
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: add small batch variant for q4_k dp4a GEMM
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: add a narrow variant of mul_mm_f16_f32_l4_lm
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
* opencl: add a one subgroup variant for gemv_noshuffle_q1_0_f32 for big M
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
---------
Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com>
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
The __ARM_NEON branch defines both GGML_CPU_COMPUTE_* macros but selects
only FP16_TO_FP32, so FP32_TO_FP16 falls through to the portable
software conversion while neon_compute_fp32_to_fp16, a single fcvt,
stays defined and referenced by nothing. __POWER9_VECTOR__ and
__riscv_zfhmin select both.
Add an #error pair that refuses a branch defining a COMPUTE macro
without selecting it. That is how this stayed unnoticed: the fallback
silently supplies a correct software implementation, so the only
symptom is a hardware instruction nothing ever reaches. The
FP16_TO_FP32 half carries a !defined(__F16C__) carve-out, because PR
30343 keeps that direction on the lookup table on purpose.
Replace the comment above the fallback: it singled out NEON and POWER9
and pointed at a selection that now happens in the branch itself.
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* hex-build: bump toolchain, enable v85
* hex-build: bump toolchain to v0.8 in the CI and build scripts
* hex-build: add a little python script for swiv needed on some iot devices
* hex-build: fix flake8 warnings
* hex-arch: update version cap to v85
* hex-inspect: flexible TOOLS_ROOT
* hex-build: update to snake_case naming
* hex-build: update leftover old toolchain refs
* hex-target: update build and run scripts to support ubuntu with adb transport
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
The fused moe_reduce kernel computes dst = sum(experts[e] * weights[e]) in a
single dispatch, reading experts and weights while writing dst. The graph
allocator computes tensor liveness on the unfused graph, where the inputs are
fully consumed by earlier nodes of the fused group, so it can legally recycle
the input memory as dst. In the fused kernel this becomes a
threadgroup-vs-threadgroup race that produces rare non-deterministic wrong
results (observed with test-llama-archs -a kolibri on the meta backend).
Reject the fusion when dst overlaps either input, same guard as the
mul_mat_add fusion. The unfused MUL + ADD path is always correct.
Assisted-by: pi:llama.cpp/ggml-org/GLM-5.3-Flash-GGUF (2x DGX Spark (RDMA, TP=2))
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* cuda: cap Stream-K blocks at the number of k-blocks
Stream-K normally launches one CUDA block per SM. For small matrices,
this can exceed the total number of output-tile/k-block pairs, leaving
blocks with no work and adding launch and fixup overhead. Cap the grid
at that count to remove these excess blocks without changing the useful
rounded K ranges or numerical results.
Assisted-by: Codex
* cuda: cap Stream-K blocks at the number of k iterations
Cap the grid at the number of K_vram iterations across all output tiles, rounding up partial iterations. This removes extra CUDA blocks left by the k-block cap.
Partial iterations can change FP32 summation grouping. MUL_MAT and MUL_MAT_ID backend tests pass on RTX 5090.
* Update ggml/src/ggml-cuda/mmq.cuh
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* CUDA: fix KQ write-after-read race in flash_attn_ext_vec
Assisted-by: ChatGPT
* CUDA: apply review suggestions for KQ sync comments
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
---------
Co-authored-by: Johannes Gäßler <johannesg@5d6.de>
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* hex-fa: support for masked block skipping to reduce DMA fetching
* hex-fa: finetuning the hmx->hvx crossover thresholds (boosting gemma-4)
* hex-fa: update hmx gate to account for mdev.count
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
Port of the Vulkan HC_POST_GATE fusion (#29520). hc_combine builds the
hc_post weights as 2*sigmoid(inject/hc); dsv4_hc_post_f32 now takes a
gated template flag and computes scale_out*sigmoid(scale_in*p) from the
raw weights, so the chain runs as one kernel instead of four.
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* ggml: support non-contiguous SUM (CPU, CUDA)
* tests: add non-contiguous SUM cases
* tests: SUM over non-contiguous views
* vulkan: support non-contiguous SUM
* webgpu: support non-contiguous SUM
Copilot Setup Steps / copilot-setup-steps (push) Has been cancelled
Check Pre-Tokenizer Hashes / pre-tokenizer-hashes (push) Has been cancelled
Python check requirements.txt / check-requirements (push) Has been cancelled
Python Type-Check / python type-check (push) Has been cancelled
Update Operations Documentation / update-ops-docs (push) Has been cancelled
* hex-hadamard: add support for hadamard ops
* hex-mm: reduce pkt counts in the q8_1 quantizer and vec_dot_q1
* hex-q1/q2: add support for flat and tiled GET_ROWS for Q1_0 and Q2_0
---------
Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com>
kurquhar