Skip to content

Capacity Expansion

Jump to bottom
Honggyu Kim edited this page Jul 19, 2024 · 18 revisions

In a heterogeneous memory system, each type of memory has its own performance characteristics. Improper data placement in such systems can lead to significant performance degradation. HMSDK provides 2-tier memory management to optimize system performance through the proper placement of data . This is supported via DAMON framework in linux kernel and its DAMON-based Operation Schemes, called DAMOS.

HMSDK supports two new DAMOS actions; demotion from fast tiers and promotion from slow tiers. This prevents hot pages from being stuck on slow tiers, which makes performance degradation and cold pages can be proactively demoted to slow tiers so that the system can increase the chance to allocate more hot pages to fast tiers.

The damo user space tool helps enabling DAMON and applying those new DAMOS actions based on the given yaml config file generated by tools/gen_config.py. This can be especially useful when the 2-tier memory system has high memory pressure on its first tier DRAM NUMA nodes by efficiently utilizing second tier CXL NUMA nodes.

User Guide

1. Installation

1.1. HMSDK Components

This section describes the HMSDK components.

linux: linux kernel for HMSDK support

damo: userspace tool for 2-tier memory management

  • user space tool to enable migration across fast and slow memory tiers.

tools: HMSDK tools contains

  • gen-config.py to generate a damo config for 2-tier memory management scheme.

1.2. Installation

You can download HMSDK repository from GitHub. Make sure you attach --recursive since HMSDK includes additional repositories as submodules.

$ git clone --recursive https://github.com/skhynix/hmsdk.git
$ cd hmsdk

This includes downloading the entire linux git history, so git cloning with --shallow-submodules will significantly reduce the download time.

Building kernel

Please read this link for the general linux kernel build.

Since HMSDK takes advantages of DAMON in linux kernel, additional DAMON related build configurations have to be enabled as follows.

$ cd hmsdk/linux
$ cp /boot/config-$(uname -r) .config
$ make menuconfig
$ echo 'CONFIG_DAMON=y' >> /.config
$ echo 'CONFIG_DAMON_VADDR=y' >> /.config
$ echo 'CONFIG_DAMON_PADDR=y' >> /.config
$ echo 'CONFIG_DAMON_SYSFS=y' >> /.config
$ echo 'CONFIG_MEMCG=y' >> /.config
$ make -j$(nproc)
$ sudo make INSTALL_MOD_STRIP=1 modules_install
$ sudo make install

You can run uname -r to verify if the kernel has been installed correctly.

$ uname -r
6.6.0-hmsdk2.0+

2. How to Use

HMSDK offers support for hot/cold memory tiering through the use of DAMON and its userspace tool, damo. To begin using HMSDK, ensure that you have installed the HMSDK kernel along with DAMON. To check if DMAON is properly installed, enter the following command:

$ if grep CONFIG_DAMON /boot/config-$(uname -r); then echo "installed"; fi

Starting HMSDK

To start HMSDK, you can use the damo tool. Before starting, ensure that you have a proper configuration file.

# The -d/--demote and -p/--promote options can be used multiple times.
$ sudo ./tools/gen_config.py -d SRC DEST -p SRC DEST -o hmsdk.yaml

# Enable demotion to slow tier. This prevents from swapping out from fast tier.
$ echo true | sudo tee /sys/kernel/mm/numa/demotion_enabled

# make sure cgroup2 is mounted under /sys/fs/cgroup, then create "hmsdk" directory below. 
$ sudo mount -t cgroup2 none /sys/fs/cgroup
$ echo '+memory' | sudo tee /sys/fs/cgroup/cgroup.subtree_control
$ sudo mkdir -p /sys/fs/cgroup/hmsdk

# Start HMSDK based on hmsdk.yaml.
$ sudo ./damo/damo start hmsdk.yaml

Please refer to the detail usage of gen_config.py at Tools section below.

Stopping HMSDK

$ sudo ./damo/damo stop

3. Tools (damo and gen_config.py)

damo

Regarding the damo usage, please refer to damo usage.

gen_config.py

gen_config.py is a python script that can generate a proper damo config file for HMSDK 2-tier memory management scheme.

For example, if the system contains a CXL memory recognized as NUMA node 2 as the hardware topology shown in the following example.

  $ numactl --hardware
    available: 3 nodes (0-2)
    node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
    node 0 size: 31956 MB
    node 0 free: 29028 MB
    node 1 cpus: 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55
    node 1 size: 32242 MB
    node 1 free: 31032 MB
    node 2 cpus:
    node 2 size: 96761 MB
    node 2 free: 96659 MB
    node distances:
    node   0   1   2
      0:  10  21  14
      1:  21  10  24
      2:  14  24  10

Then a config file can be generated providing node 0, 1 for dram nodes and node 2 for CXL node.

$ sudo ./tools/gen_config.py --demote 0 2 --demote 1 2 --promote 2 0 --promote 2 1 -o hmsdk.yaml

Please note that the generated hmsdk.yaml contains cgroup filter under the cgroup name as "hmsdk" by default.

If you want to apply HMSDK globally, use -g/--global option when running gen_config.py as follows.

$ sudo ./tools/gen_config.py -d 0 2 -d 1 2 -p 2 0 -p 2 1 -g -o hmsdk.yaml

Please note that -d and -p are short options of --demote and --promote.

Performance Results

TBD

Clone this wiki locally