Parse CALI_CONFIG for Configuration Parameters

[michaelmckinsey1]

Summary

• This PR is a bugfix, preventing an issue where two caliper files are generated from a single run of RAJAPerf.

• It does the following:

  • Modifies the Caliper implementation in src/common/KernelBase.cpp to parse the CALI_CONFIG, removing any provided filename, and passing the other variables correctly into the config manager.
  • Unsets the CALI_CONFIG environment variable, disabling the default config manger.

Explanation

The CALI_CONFIG environment variable enables activating one (or more) of Caliper's built-in performance profiling configurations. For the Caliper implementation in RAJAPerf, we manage the configuration separately for each variant-tuning combination. This means we generate a separate Caliper file for each variant-tuning combination. Allowing CALI_CONFIG to be set, means allowing usage such as CALI_CONFIG="spot(output=my_file.cali)" raja-perf --variants Base_CUDA --tunings block_128 block 256, resulting in 3 Caliper files Base_CUDA-block_128.cali, Base_CUDA-block_256.cali, and my_file.cali, which does not make sense.

We allow modification of the Caliper configuration through the --add-to-spot-config and --add-to-cali-config arguments, while this PR essentially passes the parameters in the CALI_CONFIG through the same channels as these arguments.

Example

Example 1 - many parameters

When there are configuration parameters provided in both CALI_CONFIG and through the command line arguments, --add-to-cali-config and --add-to-spot-config, the parameters are merged into one config Profile: spot(output=/g/g20/mckinsey/tmp/nvhpc_rajaperf/RAJAPerf/scripts/bin/lassen_scripts/1/131072/data/RAJA_CUDA-block_256.cali,profile.cuda,time.exclusive,mem.highwatermark),profile.mpi

export CALI_CONFIG_MODE="${CALI_CONFIG_MODE},time.exclusive,mem.highwatermark";
export CALI_CONFIG="spot(output=my_file.cali${CALI_CONFIG_MODE})";
srun raja-perf
    --variants "RAJA_CUDA" \
    --tunings "block_256" \
    --add-to-cali-config "profile.mpi" \
    --add-to-spot-config "profile.cuda"

results in:

Configuration options in CALI_CONFIG will be parsed and added to the internal RAJAPerf Caliper config manager.
Setting up suite based on input...
CALI_CONFIG: spot(output=my_file.cali,time.exclusive,mem.highwatermark)
WARNING: Removing requested output name from config: 'output=my_file.cali'. Output cali file name will be automatically generated.
Caliper ran Spot config check
Profile: spot(output=/g/g20/mckinsey/tmp/nvhpc_rajaperf/RAJAPerf/scripts/bin/lassen_scripts/1/131072/data/RAJA_CUDA-block_256.cali,profile.cuda,time.exclusive,mem.highwatermark),profile.mpi

Example 2 - No command line arguments

If only output=my_file.cali is provided, the CALI_CONFIG will be ignored.

export CALI_CONFIG="spot(output=my_file.cali)";
srun raja-perf
    --variants "RAJA_CUDA" \
    --tunings "block_256"

Configuration options in CALI_CONFIG will be parsed and added to the internal RAJAPerf Caliper config manager.
Setting up suite based on input...
CALI_CONFIG: spot(output=my_file.cali,)
WARNING: Removing requested output name from config: 'output=my_file.cali'. Output cali file name will be automatically generated.
Profile: spot(output=/g/g20/mckinsey/tmp/nvhpc_rajaperf/RAJAPerf/scripts/bin/lassen_scripts/1/131072/data/RAJA_CUDA-block_256.cali)

Example 3 - multiple tunings in benchpark

Running RAJAPerf without the --variant and --tuning arguments automatically runs all available variant/tuning combinations, which results in a cali file per combination. We can see that providing a single file name does not make sense in this case.

ramble on

CALI_CONFIG: spot(output=/usr/WS1/mckinsey/benchpark/exp/rajaperf-fix-lassen/rajaperf-fix/LlnlSierra-0b88187/workspace/experiments/raja-perf/suite/raja-perf_suite_strong_scaling_cuda_caliper_time_1/raja-perf_suite_strong_scaling_cuda_caliper_time_1.cali,time.exclusive)
WARNING: Removing requested output name from config: 'output=/usr/WS1/mckinsey/benchpark/exp/rajaperf-fix-lassen/rajaperf-fix/LlnlSierra-0b88187/workspace/experiments/raja-perf/suite/raja-perf_suite_strong_scaling_cuda_caliper_time_1/raja-perf_suite_strong_scaling_cuda_caliper_time_1.cali'. Output cali file name will be automatically generated.
Profile: spot(output=./RAJA_CUDA-blkdev_direct_256.cali,time.exclusive)
CALI_CONFIG: spot(output=/usr/WS1/mckinsey/benchpark/exp/rajaperf-fix-lassen/rajaperf-fix/LlnlSierra-0b88187/workspace/experiments/raja-perf/suite/raja-perf_suite_strong_scaling_cuda_caliper_time_1/raja-perf_suite_strong_scaling_cuda_caliper_time_1.cali,time.exclusive)
WARNING: Removing requested output name from config: 'output=/usr/WS1/mckinsey/benchpark/exp/rajaperf-fix-lassen/rajaperf-fix/LlnlSierra-0b88187/workspace/experiments/raja-perf/suite/raja-perf_suite_strong_scaling_cuda_caliper_time_1/raja-perf_suite_strong_scaling_cuda_caliper_time_1.cali'. Output cali file name will be automatically generated.
Profile: spot(output=./RAJA_CUDA-atomic_direct_256.cali,time.exclusive)
CALI_CONFIG: spot(output=/usr/WS1/mckinsey/benchpark/exp/rajaperf-fix-lassen/rajaperf-fix/LlnlSierra-0b88187/workspace/experiments/raja-perf/suite/raja-perf_suite_strong_scaling_cuda_caliper_time_1/raja-perf_suite_strong_scaling_cuda_caliper_time_1.cali,time.exclusive)
WARNING: Removing requested output name from config: 'output=/usr/WS1/mckinsey/benchpark/exp/rajaperf-fix-lassen/rajaperf-fix/LlnlSierra-0b88187/workspace/experiments/raja-perf/suite/raja-perf_suite_strong_scaling_cuda_caliper_time_1/raja-perf_suite_strong_scaling_cuda_caliper_time_1.cali'. Output cali file name will be automatically generated.
Profile: spot(output=./RAJA_CUDA-block_64.cali,time.exclusive)
...

[pearce8]

@MrBurmark we have a matching PR on the benchpark side (), and I want to make sure we get all of the cases working, and have @daboehme's approval for both as well. So, hang tight, we want to get this right. If you are aware of pending release dates, please let us know what we want to aim for.

[michaelmckinsey1]

LLNL/benchpark#618 Enables unsetting the CALI_CONFIG environment variable when running RAJAperf in Benchpark, avoiding the Warning about the filename being ignored.