corpus.blog
56,966 blogs · 4,417,066 posts Claim your blog
Record · corpus.blog/posts/01a0877b-d96e-71dc-b69d-ce8b5b2ea162

My first Multi-GPU kernel: Writing All-to-all for AMD MI300X

gau-nernst.github.io · published 2 November 2025

A record is what was published and who pointed at it. The text of the post is not held here: read it at the source, or at its Wayback capture.

Read the post at gau-nernst.github.io Wayback capture
0blogs citing
First cited
—
Most recent
—
Rank this month
not ranked
Links from this post
49
Words captured
8,378

Cited by

No blog we hold has cited this post yet.

Links from this post

Link
Host
https://www.gpumode.com/v2/home
gpumode.com
https://github.com/gpu-mode/reference-kernels/blob/0080199b/problems/amd_distributed/all2all
github.com
https://github.com/gau-nernst/gpu-mode-kernels/tree/5ab701b2/amd-distributed/all2all
github.com
https://huggingface.co/deepseek-ai/DeepSeek-V3
huggingface.co
https://github.com/vllm-project/vllm/blob/v0.11.0/csrc/moe/moe_align_sum_kernels.cu
github.com
https://github.com/gpu-mode/reference-kernels/blob/0080199b/problems/amd_distributed/all2all/reference.py
github.com
https://docs.pytorch.org/docs/stable/distributed.html
docs.pytorch.org
https://github.com/gau-nernst/gpu-mode-kernels/blob/0080199b/amd-distributed/all2all/submission_v2.py
github.com
https://docs.pytorch.org/docs/stable/notes/multiprocessing.html
docs.pytorch.org
https://github.com/ROCm/iris/blob/0dfc460e/iris/hip.py
github.com
https://rocm.docs.amd.com/projects/HIP/en/docs-7.0.2/how-to/hip_runtime_api/memory_management/coherence_control.html
rocm.docs.amd.com
https://docs.open-mpi.org/en/main/man-openshmem/man3/OpenSHMEM.3.html
docs.open-mpi.org
https://github.com/ROCm/iris/blob/0dfc460e/iris/iris.py
github.com
https://github.com/perplexityai/pplx-kernels/blob/2bd6e30f/csrc/all_to_all/intranode_dispatch.cu
github.com
https://github.com/ByteDance-Seed/Triton-distributed/blob/12c23890/python/triton_dist/kernels/nvidia/gemm_allreduce.py
github.com
https://davekilian.com/acquire-release.html
davekilian.com
https://triton-lang.org/main/python-api/generated/triton.language.atomic_add.html
triton-lang.org
https://github.com/ROCm/rocSHMEM/blob/rocm-7.0.2/src/atomic.hpp
github.com
https://rocm.docs.amd.com/projects/llvm-project/en/latest/LLVM/llvm/html/AMDGPUUsage.html
rocm.docs.amd.com
https://github.com/gpu-mode/reference-kernels/blob/0080199b/problems/amd_distributed/eval.py
github.com
https://www.amd.com/content/dam/amd/en/documents/instinct-tech-docs/data-sheets/amd-instinct-mi300x-platform-data-sheet.pdf
amd.com
https://github.com/deepseek-ai/DeepEP
github.com
https://github.com/perplexityai/pplx-kernels
github.com
https://www.perplexity.ai/hub/blog/efficient-and-portable-mixture-of-experts-communication
perplexity.ai
https://github.com/gau-nernst/gpu-mode-kernels/blob/5ab701b2/amd-distributed/all2all/submission_v4.py
github.com
https://rocm.docs.amd.com/projects/HIP/en/docs-7.0.2/reference/hip_runtime_api/modules/cooperative_groups_reference.html
rocm.docs.amd.com
https://github.com/gau-nernst/gpu-mode-kernels/blob/5ab701b2/amd-distributed/all2all/submission_v5.py
github.com
https://docs.pytorch.org/docs/stable/profiler.html
docs.pytorch.org
https://github.com/gau-nernst/gpu-mode-kernels/blob/5ab701b2/amd-distributed/all2all/submission_v6.py
github.com
https://github.com/gau-nernst/gpu-mode-kernels/blob/5ab701b2/amd-distributed/all2all/submission_v7.py
github.com
https://github.com/gau-nernst/gpu-mode-kernels/blob/5ab701b2/amd-distributed/all2all/submission_v7b.py
github.com
https://developer.nvidia.com/nsight-compute
developer.nvidia.com
https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/edit?usp=sharing
docs.google.com
https://github.com/ROCm/iris/blob/0dfc460e/examples/common/utils.py
github.com
https://github.com/gau-nernst/gpu-mode-kernels/blob/5ab701b2/amd-distributed/all2all/trace_v8.json.gz
github.com
https://github.com/Dao-AILab/flash-attention/blob/v2.8.3/flash_attn/flash_attn_interface.py
github.com
https://github.com/gau-nernst/gpu-mode-kernels/blob/5ab701b2/amd-distributed/all2all/submission_v9.py
github.com
https://github.com/gau-nernst/gpu-mode-kernels/blob/5ab701b2/amd-distributed/all2all/trace_v9.json.gz
github.com
https://arxiv.org/abs/2502.19811
arxiv.org
https://github.com/gau-nernst/gpu-mode-kernels/blob/5ab701b2/amd-distributed/all2all/reference.py
github.com
https://github.com/gau-nernst/gpu-mode-kernels/blob/5ab701b2/amd-distributed/all2all/submission_v2.py
github.com
https://github.com/gpu-mode/reference-kernels/blob/0080199b/problems/amd_distributed/gemm-rs
github.com
https://github.com/gpu-mode/reference-kernels/blob/0080199b/problems/amd_distributed/ag-gemm
github.com
https://github.com/benenzhu/gpu-mode-kernels/tree/main/amd-distributed
github.com
https://github.com/benenzhu
github.com
https://github.com/ROCm/iris
github.com
https://www.youtube.com/watch?v=H2bzSn5ZPks
youtube.com
https://www.yottalabs.ai/
yottalabs.ai
https://ui.perfetto.dev/
ui.perfetto.dev