sysinfo: retrieving process GPU usage information

A few months ago, I wrote sysinfo: Getting GPUs which described how I implemented the retrieval of GPU information in the sysinfo crate. And some years ago, I wrote sysinfo: how to extract systems' information. Although you don't need to read them beforehand to read this blog post, I might go quickly over details that I described in these blog posts, so don't hesitate to give them a read if something is unclear as the explanation is likely in one of these two.

So now, what is this blog post about? After adding support to retrieve GPU information at the system's level (how much a GPU is used), I then added support for how much each process was using GPU in this pull request. This blog post will describe how this information is retrieved.

Retrieving process GPU usage information

I'll start with the system where this feature was the simplest to implement:

Windows

As surprising as it sounds, it was very straight forward to retrieve this information on Windows. This system provides a query system. In this case, we need two queries (per process):

GPU Engine(pid_{PID}_*)\Utilization Percentage
GPU Process Memory(pid_{PID}_*)\Local Usage

Where you replace {PID} with the process' PID. Only tweak: once the query handle is created, we don't immediately query the information, we wait for the next time to have useable information.

Interestingly enough, the "GPU Engine Utilization Percentage" is per GPU whereas the "GPU Process Memory" is for all GPUs. So for the first one, the query returns a list of percentage you need to sum. The second one directly returns the value we want.

That's it.

Linux

A bit more tricky. In each process folder in /proc/{PID}, there are two folders: fd and fdinfo. fd contains the list of all files currently opened by {PID} and fdinfo contains the information for each opened fd. So a file in fd has its equivalent one in fdinfo:

$ ls /proc/158668/fd
0  1 10  12  14  16  18  2
$ ls /proc/158668/fdinfo
0  1 10  12  14  16  18  2

And the content of an fdinfo file looks like this:

$ cat /proc/158668/fdinfo/1
pos:    0
flags:  02
mnt_id: 12
ino:    59559
scm_fds: 0

So now, why should we care about these files? Because "everything is a file in Linux", including your GPU "connection" (the driver). So now, we need to find the information we need to extract. Luckily, it's standardized and you can find this information here. And now, we need to parse it and make sense of it.

First thing to note, you can have multiple "connections" to a same GPU but they report the same numbers. So to prevent reparsing the same information more than once, we need an ID to be used as a key. drm-client-id and drm-pdev (which is the GPU's PCI) can be used for that.

Then to compute the GPU usage, we need to know how much time the process used. This information is provided by drm-engine-*. Why *? Because you can have different kind of GPU usage (video processing, 3D, etc). So we need to sum them all as sysinfo (at least for now) doesn't differentiate between each kind of GPU usage. The value is in ns. Then the computation is as follows:

GPU usage time = current GPU usage time - previous GPU usage time
delta time = now - previous update time

GPU percentage usage = 100 * GPU usage time / delta time

As for the GPU memory, the key is drm-total-* (* for the same reason as previously) and the unit needs to be parsed and converted into bytes.

Implementing this introduced a major performance regression, I'll come back to this in a follow-up blog post.

macOS

And now comes the worst one out of the three supported systems for this feature. The reason: no public API to query this information. And to make things perfect: the information changes between versions of macOS and depending on the hardware. Fun!

In macOS, you have IO registries, which contains "planes" (yeah, no clue, doesn't matter). You can picture it as a tree of variable depth with variable number of nodes with a lot of possible different keys (some of which might be trees). Anyway. What we're looking for is the accumulatedGPUtime key and its associated PID (provided by the IOUserClientCreator key which value looks like pid 1, program name). So going through the whole thing recursively should provide this information.

On some macOS versions, both keys are in the same node. In some others, accumulatedGPUtime is stored in a child node of AppUsage. So to resume, either we have:

{
  "lastSubmittedTime" = 2729276743468257
  "IOUserClientCreator" = "pid 71361, Firefox GPU Help"
  "accumulatedGPUTime" = 45970
}

or:

{
  "AppUsage" = (
    {
      "API"="Metal",
      "lastSubmittedTime"=606937906041250,
      "accumulatedGPUTime"=10749838353000
    },
    {
      "API"="Metal",
      "lastSubmittedTime"=606509081566166,
      "accumulatedGPUTime"=664361375
    },
    {
      "API"="Metal",
      "lastSubmittedTime"=530914597456791,
      "accumulatedGPUTime"=1015230000
    }
  )
  "IOUserClientCreator" = "pid 403, WindowServer"
  "CommandQueueCount" = 3
  "API" = "Metal"
  "CommandQueueCountMax" = 5
}

... and potentially more I don't know about yet.

So for a given PID, you sum all accumulatedGPUTime and then the computation is the same as for Linux. What about GPU memory you ask? macOS doesn't provide the information. However, interesting to be noted: on most macs, there is no GPU dedicated VRAM, it directly uses the RAM.

Words of the end

Cat was too asleep to provide some luck.