[Question] Pinning PCM to specific cores (PCM is not honoring isolcpus)

[hoax-killer]

Hi,

I did not see a mailing list hence putting up the question here.

I have 48 cores in my machine, and isolated 46 cores by adding isolcpus=2-47 as kernel param.
When I run PCM, it still runs on all the 48 cores. Is there a way for me to pin PCM to specific cores or simply honor the cores that have been configured to be isolated from kernel scheduling?

Thanks.

[opcm]

There is a discussion tab in the repo. Transferring the question there.

I have little experience with isolcpus. Could you please post PCM output with isolcpus here?

[hoax-killer]

Sure. Providing both -- the kernel config. as well as the PCM output.

Kernel parameters

umh:~/pcm$ cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-4.15.0-122-generic ro iommu=pt intel_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=256 isolcpus=2-47 intel_idle.max_cstate=0 processor.max_cstate=0 rcu_nocb_poll=2-47 audit=0 nohz_full=2-47 nosoftlockup

---

PCM output

umh:~/pcm$ sudo ./pcm.x 0.1 -csv=./test2.log

 Processor Counter Monitor  ($Format:%ci ID=%h$)


Linux arch_perfmon flag  : yes
IBRS and IBPB supported  : yes
STIBP supported          : yes
Spec arch caps supported : no
Number of physical cores: 48
Number of logical cores: 48
Number of online logical cores: 48
Threads (logical cores) per physical core: 1
Num sockets: 2
Physical cores per socket: 24
Last level cache slices per socket: 24
Core PMU (perfmon) version: 4
Number of core PMU generic (programmable) counters: 3
Width of generic (programmable) counters: 48 bits
Number of core PMU fixed counters: 3
Width of fixed counters: 48 bits
Nominal core frequency: 2700000000 Hz
IBRS enabled in the kernel   : no
STIBP enabled in the kernel  : no
Package thermal spec power: 205 Watt; Package minimum power: 90 Watt; Package maximum power: 413 Watt;
INFO: Linux perf interface to program uncore PMUs is present
Socket 0: 2 memory controllers detected with total number of 6 channels. 3 QPI ports detected. 2 M2M (mesh to memory) blocks detected. 0 Home Agents detected. 3 M3UPI blocks detected.
Socket 1: 2 memory controllers detected with total number of 6 channels. 3 QPI ports detected. 2 M2M (mesh to memory) blocks detected. 0 Home Agents detected. 3 M3UPI blocks detected.
Delay: 0.1
Trying to use Linux perf events...
Successfully programmed on-core PMU using Linux perf
Socket 0
Max QPI link 0 speed: 23.3 GBytes/second (10.4 GT/second)
Max QPI link 1 speed: 23.3 GBytes/second (10.4 GT/second)
Max QPI link 2 speed: 21.5 GBytes/second (9.6 GT/second)
Socket 1
Max QPI link 0 speed: 23.3 GBytes/second (10.4 GT/second)
Max QPI link 1 speed: 23.3 GBytes/second (10.4 GT/second)
Max QPI link 2 speed: 21.5 GBytes/second (9.6 GT/second)

Detected Intel(R) Xeon(R) Platinum 8168 CPU @ 2.70GHz "Intel(r) microarchitecture codename Skylake-SP" stepping 4 microcode level 0x2006906
^CDEBUG: caught signal to interrupt (Interrupt).
Cleaning up
 Zeroed uncore PMU registers
 Freeing up all RMIDs

Is there a quick way for me to only make it spawn threads only on a specific number of core ids?

[opcm]

I think this should work: PCM_NO_PERF=1 taskset -c 0-1 ./pcm.x -r
pcm will only use cores 0-1 in user space. The kernel still can use all cores (and PCM gets the perfmon register values from the kernel driver).

[hoax-killer]

Thanks. I tried to stress test by decreasing the time interval. As soon as I start PCM, I can still see the overheads on other cores besides 0 and 1. Here is the command I used to test.

sudo PCM_NO_PERF=1 taskset -c 0-1 ./pcm.x 0.01 -r -csv=./test2.log

Is this expected with higher sampling frequencies?

[opcm]

with higher sampling frequencies PCM queries the kernel driver more often. This is the kernel overhead time on these cores, not pcm user-space time. You can check the cores allowed to be used by pcm user space process with this command:

cat /proc/`pgrep pcm`/status | grep Cpus_allowed_list

[hoax-killer]

Hmm I see, that makes sense, thanks @opcm
Yes, I tried the command it lists cores 0 and 1 as in the allowed list for this process.

Nonetheless, the command you provided here solves my query.

Thanks!