On Tue, Feb 23, 2021 at 11:01:51AM +0800, Yu Chuan wrote:

Signed-off-by: Yu Chuan <13186087857@163.com> --- doc/job/os_mount.md | 263 ++++++++++++++++++++++++++++++-------------- 1 file changed, 179 insertions(+), 84 deletions(-)

diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index 226251a8bace..0d0afd6469a8 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -1,94 +1,189 @@ -# Summary -========= +Summary +=======

-`os_mount` defines the mount method of testbox's root partition. +`os_mount` defines the rootfs type.

-It has the following optional values: - - nfs - - cifs - - initramfs - - container - - local +The below types are supported: + - nfs + - cifs + - initramfs + - container + - local

Usage example: - - submit iperf.yaml testbox=vm-2p8g os_mount=nfs - - submit iperf.yaml testbox=vm-2p8g os_mount=cifs - - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs - - submit iperf.yaml testbox=dc-8g os_mount=container - - submit iperf.yaml testbox=vm-2p8g os_mount=local - -# Optional Values -================= - -## Work flow when os_mount=local - -1. user submit job with os_mount: local - - optional {kernel_append_root}: - - save_root_partition: give an iconic string, then the root partition data used by this job will be retained. - and you can use the root partition data of this job in the futher job by {use_root_partition}. - - use_root_partition : give an iconic string, then the root partition data generated by the - pervious job which has the {save_root_partition} field will be used. - - root_partition_size: specify the size of root partition, *default is 10G*. - - - tips: - - value of {save_root_partition} and {use_root_partition} need to be iconic enough, uuid is recommended. - - value of {save_root_partition} and {use_root_partition} will be the suffix of logical volume. + ``` + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=nfs + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=cifs + submit -m -c borrow-1h.yaml testbox=vm-2p16g os_mount=initramfs + submit -m -c borrow-1h.yaml testbox=dc-8g os_mount=container + submit -m -c borrow-1h.yaml testbox=taishan200-2280-2s48p-256g os_mount=local + ``` + +os_mount=local +============== + +`nfs`, `cifs`, `initramfs`, `container`, rootfs of them all are in memory. +So if you want to run your job in disk file system, you need to specify +`os_mount=local` in your job.yaml. + +BTW, the job's rootfs will be placed on a logical volume. + +## description + +### temporary rootfs data *Defaultly* + +If you only add `os_mount: local` in your job.yaml, the logical volume of rootfs +will be deleted and re-created in initrd stage before every job executes. + +The brief flow is as follow:

as follow => as follows

+ ``` + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - will never boot form this lv, just use it as a source data, and take a

form => from

+ snapshot of it to get the real boot logical volume. + - srv_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp}

srv_lv => src_lv Thanks, Xijian

+ ${boot_lv} -- boot logical volume: + - will boot from this. + - boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: create it, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} exists: delete it + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. + ``` + +### persistent rootfs data + +When you need to persist the rootfs data of a job, and use it in the subsequent job(s), +two fields in `kernel_append_root` will help you: `save_root_partition`, `use_root_partition`. + +The brief flow is as follow:

as follow => as follows

+ ``` + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - if have ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${use_root_partition} + - if no ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + ${boot_lv} -- boot logical volume: + - if have ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${save_root_partition} + - if no ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: + - if have ${use_root_partition}, exit 1. + - if no ${use_root_partition}, create ${src_lv}, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} != ${src_lv}: + - if ${boot_lv} exists: delete it + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. + ``` + +Demo usage: + ``` + - in 20210218, you submit a job-20210218.yaml, and you want to persist the rootfs data of + job-20210218.yaml so that it can be used by the subsequent jobs. + Then you need add the follow field in your job-20210218.yaml: + kernel_append_root: save_root_partition=zhangsan_local_for_iperf_20210218 + + - in 20210219, you submit a job-20210219.yaml, and you want to use the rootfs data of + job-20210218.yaml. + Then you need add the follow field in your job-20210219.yaml: + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 + ``` + +Notes: + - obviously, ${save_root_partition} and ${use_root_partition} must be the same, and unique. + - you must 'save_root_partition' firstly, then you can 'use_root_partition'. otherwise, your job will fail. + +## work flow + +1. user submit job + job fields: + ``` + os: openeuler + os_arch: aarch64 + os_version: 20.03 + os_mount: local + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 + ```

2. scheduler return the ipxe_str to testbox - ``` - dhcp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz - imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} - boot - ``` + ``` + dhcp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/initrd.lkp + kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src=${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} initrd=initrd.lkp ${kernel_append_root} + boot + ```

3. dracut step of boot - ``` - if have {use_root_partition}; then - base_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 - - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - - if {save_root_partition} == {use_root_partition}; then - boot from /dev/mappaer/os-{boot_lv_name} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - else - boot_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - boot from /dev/mappaer/os-{boot_lv_name} - fi - - else: - base_lv_name=openeuler_aarch64_20.03_{timestamp} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { - # create logical volume - lvcreate -L {root_partition_size} -n {base_lv_name} os || exit 1 - - # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} - mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt - mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 - cp -a /mnt/. /mnt1/ - umount /mnt /mnt1 - - # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly - lvchange -p r /dev/mapper/os-{base_lv_name} - } - - boot_lv_name=openeuler_aarch64_20.03 - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - fi - ``` + ``` + analysis_kernel_cmdline_params() + { + for i in $(cat /proc/cmdline) + do + [ "$i" =~ "rootfs_src=" ] && export ROOTFS_SRC=${i#rootfs_src=} + done + + [ -z "${ROOTFS_SRC}" ] && die "cannot find var in kernel cmdline params: rootfs_src" + + export TIMESTAMP=$(basename ${ROOTFS_SRC}) + } + + rsync_src_lv() + { + local src_lv=$1 + + lvdisplay ${src_lv} > /dev/null && return + + # create logical volume + lvcreate --size 10G --name $(basename ${src_lv}) os || exit + + # rsync nfsroot to ${src_lv} + mount -t nfs ${ROOTFS_SRC} /mnt + mkdir /mnt1 && mount ${src_lv} /mnt1 + cp -a /mnt/. /mnt1/ + umount /mnt /mnt1 + + # change permission of ${src_lv} to readonly + lvchange --permission r ${src_lv} + } + + snapshot_boot_lv() + { + local src_lv=$1 + local boot_lv=$2 + + [ "$src_lv" == "$boot_lv" ] && return + + lvremove --force ${boot_lv} + lvcreate --size 10G --name $(basename ${boot_lv}) --snapshot ${src_lv} || exit + } + + main() + { + analysis_kernel_cmdline_params + + if [ -z "${use_root_partition}" ]; then + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${TIMESTAMP}" + rsync_src_lv ${src_lv} + else + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + lvdisplay ${src_lv} > /dev/null || exit + fi + + if [ -z "${save_root_partition}" ]; then + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03" + else + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + fi + + snapshot_boot_lv ${src_lv} ${boot_lv} + boot_from_boot_lv + } + + main() + ```

4. execute the job -- 2.23.0