JBJFLab.
← 返回列表

GpuMemTool 技术解析：在 Windows 上按进程统计显存占用

2026-10-29 · 技术 · 源码仓库

需求很简单：列出每个进程占用了多少显存，能一键结束进程释放显存，再配上显存 / GPU 占用率的实时曲线。真正做起来，难点全在「数据从哪来」和「结束进程是不是真的结束了」这两件事上。

1. nvidia-smi 靠不住的那一半

在 Windows（WDDM 模式）下，nvidia-smi --query-compute-apps=pid,used_gpu_memory 的 used_gpu_memory 通常返回 [N/A]——它只对 TCC 模式或纯 CUDA 上下文有效。也就是说，nvidia-smi 能告诉你显卡用了多少显存，但不能告诉你每个进程用了多少。

要回答「谁占了我的显存」，只能换一个数据源：Windows 性能计数器。

2. 双数据源合并

数据来源
显存总量 / 已用 / 空闲、GPU 利用率、显存带宽利用率、温度、功耗nvidia-smi --query-gpu=...
逐进程显存占用性能计数器 Win32_PerfFormattedData_GPUPerformanceCounters_GPUProcessMemory 的 Local Usage
逐进程 GPU 占用 %同计数器集 GPUEngine 的 UtilizationPercentage（按 PID 汇总）
CUDA 进程显存nvidia-smi --query-compute-apps，与计数器值取较大者，来源列标注 CUDA / WDDM / CUDA+WDDM

实测校验：逐进程合计 2377 MB，nvidia-smi 报告的已用量 2491 MB，差值约 114 MB 属于驱动 / 系统保留，界面把它单独显示成「未归属(驱动/系统)」，避免用户困惑为什么加起来对不上。

另一个细节：进程级数据只能拿到全部显卡的合计——Windows 性能计数器无法把进程映射到某一块具体显卡。多卡机器上 GPU 级指标取第一块卡。

3. Stop-Process 会「假成功」

这是开发过程中最意外的坑。要结束的进程卡死时（只剩一个线程、工作集接近 0，却仍占着 1.6 GB 显存），Stop-Process 调用不报错也不退出。旧版本相信了返回值，于是界面显示「已结束」，进程却纹丝不动，看起来就是「按钮不生效」。

现在的处理流程是「三步 + 结果校验」：

  1. 先尝试正常关闭窗口，3 秒内没退出就 Stop-Process；
  2. 轮询确认进程是否真的消失——不看返回值，只看事实；
  3. 还在就自动回退到 WMI Win32_Process.Terminate（由 SYSTEM 权限的 WMI 服务执行，能结束一部分 Stop-Process 杀不掉的进程）；
  4. 仍然杀不掉就如实报告，并建议按 Win + Ctrl + Shift + B 重启显卡驱动，或重启系统。

配套的保护机制：蓝屏级进程（System、csrss、lsass 等）直接禁止结束；保护工具自身和父进程；explorer / dwm 这类外壳进程二次确认；真正结束前重新读取该 PID 的进程名，若名字对不上说明 PID 已被系统复用，跳过，避免误杀。

4. v2.0 把采集速度提了约 5 倍

v1.1 单轮采集要 2433 ms，而界面刷新间隔最短 1 秒——采集比刷新还慢，就会变成背靠背采样、CPU 长期跑满。v2.0 逐个拆掉热点：

  • nvidia-smi 的两次调用改为并行（用 .NET Process 而不是 PowerShell Job）；
  • GPUEngine 计数器的 556 个实例、单次约 1.1 秒，改为按需查询：GPU 忙才查，空闲直接按 0 处理；
  • 进程元数据从 Win32_Process（715 ms）换成 Get-Process（17 ms）；
  • 命令行信息原本每轮全量读取，改为「查看详细信息」时按需读取，这是原来最大的一笔开销；
  • 修复采集耗时超过刷新间隔时变成背靠背采样的问题。

结果：单轮采集从 2433 ms 降到约 420 ms（GPU 空闲）/ 970 ms（GPU 忙），快约 5 倍。

5. 零依赖的交付形态

整个工具只用系统自带的 Windows PowerShell 5.1 + WinForms，不装 Python、.NET SDK 或任何第三方库。交付物是单个 GpuMemTool.exe（约 180 KB），脚本内嵌在 exe 里，双击即用，启动时自己请求 UAC 提权；每次启动会把内嵌脚本释放到 %LOCALAPPDATA%\GpuMemTool\ 并校验内容，一致就不重写。

界面部分：6 个指标卡（显存、GPU 利用率、带宽利用率、温度、功耗、进程合计），可点击任意表头排序的进程列表，实时双轴曲线（左轴显存 MB、右轴 GPU %），右键结束进程 / 强制结束 / 复制 PID / 打开文件位置，以及 CSV 导出。也支持命令行无界面模式：

GpuMemTool.exe -SelfTest
GpuMemTool.exe -KillPid 1234,5678 -Force

6. 小结

这个项目最有价值的部分不是界面，而是两个「反直觉」的发现：nvidia-smi 在 WDDM 下查不到逐进程显存，以及 Stop-Process 可能假成功。把这两点处理妥当，工具才算真正可用。

仓库地址：git.jbjflab.top/admin/gpu-mem-tool（含稳定版 v1.1 与性能优化版 v2.0）。

The requirement is simple: list how much VRAM each process uses, let the user kill a process to free it, and add real-time VRAM / GPU utilisation curves. In practice, the hard parts are all about where the data comes from and whether killing a process actually kills it.

1. The half of nvidia-smi you can't rely on

On Windows (WDDM mode), used_gpu_memory from nvidia-smi --query-compute-apps=pid,used_gpu_memory usually returns [N/A] — it is only valid in TCC mode or pure CUDA contexts. In other words, nvidia-smi can tell you how much VRAM the GPU is using, but not how much each process is using.

To answer “who took my VRAM”, you need a different data source: Windows performance counters.

2. Merging two data sources

DataSource
Total / used / free VRAM, GPU utilisation, memory-bandwidth utilisation, temperature, powernvidia-smi --query-gpu=...
Per-process VRAM usagePerformance counter Win32_PerfFormattedData_GPUPerformanceCounters_GPUProcessMemory, Local Usage
Per-process GPU utilisation %UtilizationPercentage from the same counter set's GPUEngine (summed by PID)
CUDA process VRAMnvidia-smi --query-compute-apps, compared with the counter value (larger wins), labelled CUDA / WDDM / CUDA+WDDM

Measured check: the per-process sum is 2377 MB against nvidia-smi's 2491 MB used; the ~114 MB difference is driver / system reserved, displayed separately in the UI as “unattributed (driver/system)” so users aren't puzzled why the numbers don't add up.

Another detail: process-level data is aggregated across all GPUs — Windows performance counters cannot map a process to a specific graphics card. On multi-GPU machines, GPU-level metrics come from the first GPU.

3. Stop-Process can “succeed” silently

This was the most surprising trap during development. When the target process is stuck (one thread left, working set near zero, yet still holding 1.6 GB of VRAM), Stop-Process returns without error and without exiting. The old version trusted the return value, so the UI said “terminated” while the process didn't move — looking exactly like “the button does nothing”.

The current flow is “three steps + result verification”:

  1. First try closing the window normally; if it doesn't exit within 3 seconds, call Stop-Process;
  2. Poll to confirm whether the process actually disappeared — don't trust the return value, trust the facts;
  3. If it's still there, fall back to WMI Win32_Process.Terminate (executed by the SYSTEM WMI service; it can kill some processes that Stop-Process cannot);
  4. If it still can't be killed, report it honestly and suggest pressing Win + Ctrl + Shift + B to restart the graphics driver, or rebooting.

Guard rails: blue-screen-level processes (System, csrss, lsass, …) are blocked outright; the tool protects itself and its parent process; shell processes such as explorer / dwm require double confirmation; before actually killing, it re-reads the process name for that PID — if the name doesn't match, the PID has been reused by the system, so it skips to avoid killing the wrong process.

4. v2.0 made collection about 5x faster

v1.1 took 2433 ms per collection round, while the shortest refresh interval is 1 second — when collection is slower than refresh, sampling becomes back-to-back and the CPU stays pegged. v2.0 removes the hot spots one by one:

  • The two nvidia-smi calls run in parallel (using .NET Process instead of PowerShell Jobs);
  • The GPUEngine counter — 556 instances, about 1.1 s each — is now queried on demand: only when the GPU is busy; idle counts as 0;
  • Process metadata switched from Win32_Process (715 ms) to Get-Process (17 ms);
  • Command-line info used to be read in full every round; now it's read on demand when opening the details view — previously the biggest cost;
  • Fixed back-to-back sampling when collection took longer than the refresh interval.

Result: a collection round dropped from 2433 ms to about 420 ms (GPU idle) / 970 ms (GPU busy) — roughly 5x faster.

5. A zero-dependency delivery format

The whole tool only uses the built-in Windows PowerShell 5.1 + WinForms; no Python, .NET SDK or third-party libraries required. The deliverable is a single GpuMemTool.exe (~180 KB) with the script embedded, double-click to run, requesting UAC elevation itself. On each start it releases the embedded script to %LOCALAPPDATA%\GpuMemTool\ and verifies the content, rewriting only if it differs.

UI: six metric cards (VRAM, GPU utilisation, memory-bandwidth utilisation, temperature, power, process total), a process list sortable by any column, real-time dual-axis curves (left axis VRAM MB, right axis GPU %), right-click to end process / force kill / copy PID / open file location, and CSV export. It also supports a headless command-line mode:

GpuMemTool.exe -SelfTest
GpuMemTool.exe -KillPid 1234,5678 -Force

6. Summary

The most valuable part of this project isn't the UI but two counter-intuitive findings: nvidia-smi can't report per-process VRAM under WDDM, and Stop-Process may silently fail. Handle those two properly and the tool becomes genuinely usable.

Repository: git.jbjflab.top/admin/gpu-mem-tool (stable v1.1 and heavily optimised v2.0).

本文由 JBJFLab 写于 2026 年 10 月。