- Compass-ci - mailweb.openeuler.org

[PATCH v4 compass-ci] doc: optimize os_mount.md
by Yu Chuan 24 Feb '21

24 Feb '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 265 ++++++++++++++++++++++++++++++-------------- 1 file changed, 181 insertions(+), 84 deletions(-) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index 226251a8bace..1e6b349a136d 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -1,94 +1,191 @@ -# Summary -========= +Summary +======= -`os_mount` defines the mount method of testbox's root partition. +`os_mount` defines the rootfs type. -It has the following optional values: - - nfs - - cifs - - initramfs - - container - - local +The below types are supported: + - nfs + - cifs + - initramfs + - container + - local Usage example: - - submit iperf.yaml testbox=vm-2p8g os_mount=nfs - - submit iperf.yaml testbox=vm-2p8g os_mount=cifs - - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs - - submit iperf.yaml testbox=dc-8g os_mount=container - - submit iperf.yaml testbox=vm-2p8g os_mount=local - -# Optional Values -================= - -## Work flow when os_mount=local - -1. user submit job with os_mount: local - - optional {kernel_append_root}: - - save_root_partition: give an iconic string, then the root partition data used by this job will be retained. - and you can use the root partition data of this job in the futher job by {use_root_partition}. - - use_root_partition : give an iconic string, then the root partition data generated by the - pervious job which has the {save_root_partition} field will be used. - - root_partition_size: specify the size of root partition, *default is 10G*. - - - tips: - - value of {save_root_partition} and {use_root_partition} need to be iconic enough, uuid is recommended. - - value of {save_root_partition} and {use_root_partition} will be the suffix of logical volume. + ``` + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=nfs + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=cifs + submit -m -c borrow-1h.yaml testbox=vm-2p16g os_mount=initramfs + submit -m -c borrow-1h.yaml testbox=dc-8g os_mount=container + submit -m -c borrow-1h.yaml testbox=taishan200-2280-2s48p-256g os_mount=local + ``` + +os_mount=local +============== + +`nfs`, `cifs`, `initramfs`, `container`, rootfs of them all are in memory. +So if you want to run your job in disk file system, you need to specify +`os_mount=local` in your job.yaml. + +BTW, the job's rootfs will be placed on a logical volume. + +## description + +### temporary rootfs data *Defaultly* + +If you only add `os_mount: local` in your job.yaml, the logical volume of rootfs +will be deleted and re-created in initrd stage before every job executes. + +The brief flow is as follows: + ``` + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - will never boot from this lv, just use it as a source data, and take a + snapshot of it to get the real boot logical volume. + - src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + - ${timestamp} is just a version number, which is the latest rootfs version + on the nfs server in cluster. + ${boot_lv} -- boot logical volume: + - will boot from this lv. + - boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: create it, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} exists: delete it + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. + ``` + +### persistent rootfs data + +When you need to persist the rootfs data of a job, and use it in the subsequent job(s), +two fields in `kernel_append_root` will help you: `save_root_partition`, `use_root_partition`. + +The brief flow is as follows: + ``` + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - if have ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${use_root_partition} + - if no ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + ${boot_lv} -- boot logical volume: + - if have ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${save_root_partition} + - if no ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: + - if have ${use_root_partition}, exit 1. + - if no ${use_root_partition}, create ${src_lv}, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} != ${src_lv}: + - if ${boot_lv} exists: delete it + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. + ``` + +Demo usage: + ``` + - in 20210218, you submit a job-20210218.yaml, and you want to persist the rootfs data of + job-20210218.yaml so that it can be used by the subsequent jobs. + Then you need add the follow field in your job-20210218.yaml: + kernel_append_root: save_root_partition=zhangsan_local_for_iperf_20210218 + + - in 20210219, you submit a job-20210219.yaml, and you want to use the rootfs data of + job-20210218.yaml. + Then you need add the follow field in your job-20210219.yaml: + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 + ``` + +Notes: + - obviously, ${save_root_partition} and ${use_root_partition} must be the same, and unique. + - you must 'save_root_partition' firstly, then you can 'use_root_partition'. otherwise, your job will fail. + +## work flow + +1. user submit job + job fields: + ``` + os: openeuler + os_arch: aarch64 + os_version: 20.03 + os_mount: local + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 + ``` 2. scheduler return the ipxe_str to testbox - ``` - dhcp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz - imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} - boot - ``` + ``` + dhcp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/initrd.lkp + kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src=${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} initrd=initrd.lkp ${kernel_append_root} + boot + ``` 3. dracut step of boot - ``` - if have {use_root_partition}; then - base_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 - - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - - if {save_root_partition} == {use_root_partition}; then - boot from /dev/mappaer/os-{boot_lv_name} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - else - boot_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - boot from /dev/mappaer/os-{boot_lv_name} - fi - - else: - base_lv_name=openeuler_aarch64_20.03_{timestamp} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { - # create logical volume - lvcreate -L {root_partition_size} -n {base_lv_name} os || exit 1 - - # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} - mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt - mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 - cp -a /mnt/. /mnt1/ - umount /mnt /mnt1 - - # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly - lvchange -p r /dev/mapper/os-{base_lv_name} - } - - boot_lv_name=openeuler_aarch64_20.03 - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - fi - ``` + ``` + analysis_kernel_cmdline_params() + { + for i in $(cat /proc/cmdline) + do + [ "$i" =~ "rootfs_src=" ] && export ROOTFS_SRC=${i#rootfs_src=} + done + + [ -z "${ROOTFS_SRC}" ] && die "cannot find var in kernel cmdline params: rootfs_src" + + export TIMESTAMP=$(basename ${ROOTFS_SRC}) + } + + rsync_src_lv() + { + local src_lv=$1 + + lvdisplay ${src_lv} > /dev/null && return + + # create logical volume + lvcreate --size 10G --name $(basename ${src_lv}) os || exit + + # rsync nfsroot to ${src_lv} + mount -t nfs ${ROOTFS_SRC} /mnt + mkdir /mnt1 && mount ${src_lv} /mnt1 + cp -a /mnt/. /mnt1/ + umount /mnt /mnt1 + + # change permission of ${src_lv} to readonly + lvchange --permission r ${src_lv} + } + + snapshot_boot_lv() + { + local src_lv=$1 + local boot_lv=$2 + + [ "$src_lv" == "$boot_lv" ] && return + + lvremove --force ${boot_lv} + lvcreate --size 10G --name $(basename ${boot_lv}) --snapshot ${src_lv} || exit + } + + main() + { + analysis_kernel_cmdline_params + + if [ -z "${use_root_partition}" ]; then + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${TIMESTAMP}" + rsync_src_lv ${src_lv} + else + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + lvdisplay ${src_lv} > /dev/null || exit + fi + + if [ -z "${save_root_partition}" ]; then + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03" + else + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + fi + + snapshot_boot_lv ${src_lv} ${boot_lv} + boot_from_boot_lv + } + + main() + ``` 4. execute the job -- 2.23.0

2 2

[PATCH lkp-tests] jobs/: add os-multi-dimension-arch.yaml
by Li Ping 23 Feb '21

23 Feb '21

2 2

[PATCH lkp-tests] lib: export secrets.yaml in job_init
by Cao Xueliang 23 Feb '21

23 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/job-init.sh | 2 ++ lib/job.sh | 18 +++++++++++++++--- 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/lib/job-init.sh b/lib/job-init.sh index 1fb898872..27d3c6d0b 100755 --- a/lib/job-init.sh +++ b/lib/job-init.sh @@ -1,4 +1,5 @@ . $LKP_SRC/lib/upload.sh +. $LKP_SRC/lib/job.sh mount_cgroup() { @@ -270,4 +271,5 @@ job_init() job_redirect_stdout_stderr job_env + read_secret_vars } diff --git a/lib/job.sh b/lib/job.sh index dc65f68d8..87ca7ee67 100755 --- a/lib/job.sh +++ b/lib/job.sh @@ -1,8 +1,20 @@ #!/bin/sh +read_secret_vars() +{ + read_yaml_vars "$(dirname $job_file)/secrets.yaml" secrets_ +} + read_env_vars() { - [ -f "$TMP/env.yaml" ] || return 0 + read_yaml_vars "$TMP/env.yaml" +} + +read_yaml_vars() +{ + local file=$1 + local prefix=$2 + [ -f "$file" ] || return 0 local key local val @@ -11,8 +23,8 @@ read_env_vars() do [ "${key%[a-zA-Z0-9_]:}" != "$key" ] || continue key=${key%:} - export "$key=$val" - done < $TMP/env.yaml + export "$prefix$key=$val" + done < $file return 0 } -- 2.23.0

1 0

find latest github git pushes
by Wu Fengguang 23 Feb '21

23 Feb '21

Hi Yuanchao, I researched github and have 2 finding 1) a registered user can watch a number of repos and get notifications. however the notification type does not include "git push/new commit". There is only "new pull request", which looks much less useful for us. 2) we can still take advantage of its search API to find latest updated repos, like this https://github.com/search?l=&o=desc&q=pushed%3A%3E2021-02-21+filename%3AREA… 7,543 repository results clean-code-craft-tcq-1/modular-c-AnPra7 modular-c-AnPra7 created by GitHub Classroom C MIT license Updated 3 minutes ago trpo2021/chessviz-Jamesss1337 chessviz-Jamesss1337 created by GitHub Classroom C Updated 3 minutes ago mubu43/ExploringC C Updated 3 minutes ago BlankyShi/gitTest gitTest C Updated 4 minutes ago aabelque/malloc C Updated 4 minutes ago ythan88/r-card C Updated 4 minutes ago koipkoi/pokeemerald-kr A korean translation project for Pokémon Emerald. C Updated 4 minutes ago OpenMPT/openmpt Official read-only git mirror of the OpenMPT and libopenmpt subversion repository at https://source.openmpt.org/. Thi… libopenmpt tracker audio music demoscene music-creation protracker impulsetracker screamtracker fasttracker 223 C Updated 4 minutes ago silabs-DenverL/ADCxSyncCapture ADC0 and ADC1 Synchronization Capture C MIT license Updated 4 minutes ago gopro/gopro-lib-node.gl OpenGL engine for building and rendering graph-based scenes 32 C Apache-2.0 license Updated 4 minutes ago Thanks, Fengguang

1 2

[PATCH compass-ci] 2-network/iptables: fix iptables holding the xtables lock
by Cui Yili 23 Feb '21

23 Feb '21

[why] error: Another app is currently holding the xtables lock. Perhaps you want to use the -w option? [how] add lock wait time. iptables -w 10... Signed-off-by: Cui Yili <2268260388(a)qq.com> --- sparrow/2-network/iptables | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/sparrow/2-network/iptables b/sparrow/2-network/iptables index 48794f5..73f9749 100755 --- a/sparrow/2-network/iptables +++ b/sparrow/2-network/iptables @@ -8,9 +8,9 @@ BR0_IFACE=br0 BR0_SUBNET=172.18.0.0/16 # iptables -t nat -F -iptables -I FORWARD 1 -j ACCEPT -iptables -t nat -A POSTROUTING -o "$PUB_IFACE" -s $BR0_SUBNET -j MASQUERADE -iptables -t nat -A POSTROUTING -o $BR0_IFACE -d $BR0_SUBNET -j MASQUERADE +iptables -w 10 -I FORWARD 1 -j ACCEPT +iptables -w 10 -t nat -A POSTROUTING -o "$PUB_IFACE" -s $BR0_SUBNET -j MASQUERADE +iptables -w 10 -t nat -A POSTROUTING -o $BR0_IFACE -d $BR0_SUBNET -j MASQUERADE command -v firewall-cmd > /dev/null || { exit 0 -- 2.23.0

1 0

[PATCH compass-ci] support-os-os_arch-os_version.md: delete the cifs method of archlinux
by Wang Chenglong 23 Feb '21

23 Feb '21

[Why] When start archlinux with CIFS, the system will get mangled names error. So only show the NFS method for user. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/support-os-os_arch-os_version.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/support-os-os_arch-os_version.md b/doc/support-os-os_arch-os_version.md index 5d30486..93fb47b 100644 --- a/doc/support-os-os_arch-os_version.md +++ b/doc/support-os-os_arch-os_version.md @@ -18,5 +18,5 @@ centos aarch64 7 container, 7.8.2003 cifs/nfs, debian aarch64 sid cifs/nfs, ,initramfs x86 sid cifs/nfs, -archlinux aarch64 2020-11-12 cifs/nfs, +archlinux aarch64 2020-11-12 nfs, ``` -- 2.23.0

1 0

[PATCH v3 compass-ci] doc: optimize os_mount.md
by Yu Chuan 23 Feb '21

23 Feb '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 263 ++++++++++++++++++++++++++++++-------------- 1 file changed, 179 insertions(+), 84 deletions(-) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index 226251a8bace..0d0afd6469a8 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -1,94 +1,189 @@ -# Summary -========= +Summary +======= -`os_mount` defines the mount method of testbox's root partition. +`os_mount` defines the rootfs type. -It has the following optional values: - - nfs - - cifs - - initramfs - - container - - local +The below types are supported: + - nfs + - cifs + - initramfs + - container + - local Usage example: - - submit iperf.yaml testbox=vm-2p8g os_mount=nfs - - submit iperf.yaml testbox=vm-2p8g os_mount=cifs - - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs - - submit iperf.yaml testbox=dc-8g os_mount=container - - submit iperf.yaml testbox=vm-2p8g os_mount=local - -# Optional Values -================= - -## Work flow when os_mount=local - -1. user submit job with os_mount: local - - optional {kernel_append_root}: - - save_root_partition: give an iconic string, then the root partition data used by this job will be retained. - and you can use the root partition data of this job in the futher job by {use_root_partition}. - - use_root_partition : give an iconic string, then the root partition data generated by the - pervious job which has the {save_root_partition} field will be used. - - root_partition_size: specify the size of root partition, *default is 10G*. - - - tips: - - value of {save_root_partition} and {use_root_partition} need to be iconic enough, uuid is recommended. - - value of {save_root_partition} and {use_root_partition} will be the suffix of logical volume. + ``` + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=nfs + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=cifs + submit -m -c borrow-1h.yaml testbox=vm-2p16g os_mount=initramfs + submit -m -c borrow-1h.yaml testbox=dc-8g os_mount=container + submit -m -c borrow-1h.yaml testbox=taishan200-2280-2s48p-256g os_mount=local + ``` + +os_mount=local +============== + +`nfs`, `cifs`, `initramfs`, `container`, rootfs of them all are in memory. +So if you want to run your job in disk file system, you need to specify +`os_mount=local` in your job.yaml. + +BTW, the job's rootfs will be placed on a logical volume. + +## description + +### temporary rootfs data *Defaultly* + +If you only add `os_mount: local` in your job.yaml, the logical volume of rootfs +will be deleted and re-created in initrd stage before every job executes. + +The brief flow is as follow: + ``` + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - will never boot form this lv, just use it as a source data, and take a + snapshot of it to get the real boot logical volume. + - srv_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + ${boot_lv} -- boot logical volume: + - will boot from this. + - boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: create it, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} exists: delete it + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. + ``` + +### persistent rootfs data + +When you need to persist the rootfs data of a job, and use it in the subsequent job(s), +two fields in `kernel_append_root` will help you: `save_root_partition`, `use_root_partition`. + +The brief flow is as follow: + ``` + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - if have ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${use_root_partition} + - if no ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + ${boot_lv} -- boot logical volume: + - if have ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${save_root_partition} + - if no ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: + - if have ${use_root_partition}, exit 1. + - if no ${use_root_partition}, create ${src_lv}, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} != ${src_lv}: + - if ${boot_lv} exists: delete it + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. + ``` + +Demo usage: + ``` + - in 20210218, you submit a job-20210218.yaml, and you want to persist the rootfs data of + job-20210218.yaml so that it can be used by the subsequent jobs. + Then you need add the follow field in your job-20210218.yaml: + kernel_append_root: save_root_partition=zhangsan_local_for_iperf_20210218 + + - in 20210219, you submit a job-20210219.yaml, and you want to use the rootfs data of + job-20210218.yaml. + Then you need add the follow field in your job-20210219.yaml: + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 + ``` + +Notes: + - obviously, ${save_root_partition} and ${use_root_partition} must be the same, and unique. + - you must 'save_root_partition' firstly, then you can 'use_root_partition'. otherwise, your job will fail. + +## work flow + +1. user submit job + job fields: + ``` + os: openeuler + os_arch: aarch64 + os_version: 20.03 + os_mount: local + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 + ``` 2. scheduler return the ipxe_str to testbox - ``` - dhcp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz - imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} - boot - ``` + ``` + dhcp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/initrd.lkp + kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src=${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} initrd=initrd.lkp ${kernel_append_root} + boot + ``` 3. dracut step of boot - ``` - if have {use_root_partition}; then - base_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 - - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - - if {save_root_partition} == {use_root_partition}; then - boot from /dev/mappaer/os-{boot_lv_name} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - else - boot_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - boot from /dev/mappaer/os-{boot_lv_name} - fi - - else: - base_lv_name=openeuler_aarch64_20.03_{timestamp} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { - # create logical volume - lvcreate -L {root_partition_size} -n {base_lv_name} os || exit 1 - - # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} - mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt - mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 - cp -a /mnt/. /mnt1/ - umount /mnt /mnt1 - - # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly - lvchange -p r /dev/mapper/os-{base_lv_name} - } - - boot_lv_name=openeuler_aarch64_20.03 - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - fi - ``` + ``` + analysis_kernel_cmdline_params() + { + for i in $(cat /proc/cmdline) + do + [ "$i" =~ "rootfs_src=" ] && export ROOTFS_SRC=${i#rootfs_src=} + done + + [ -z "${ROOTFS_SRC}" ] && die "cannot find var in kernel cmdline params: rootfs_src" + + export TIMESTAMP=$(basename ${ROOTFS_SRC}) + } + + rsync_src_lv() + { + local src_lv=$1 + + lvdisplay ${src_lv} > /dev/null && return + + # create logical volume + lvcreate --size 10G --name $(basename ${src_lv}) os || exit + + # rsync nfsroot to ${src_lv} + mount -t nfs ${ROOTFS_SRC} /mnt + mkdir /mnt1 && mount ${src_lv} /mnt1 + cp -a /mnt/. /mnt1/ + umount /mnt /mnt1 + + # change permission of ${src_lv} to readonly + lvchange --permission r ${src_lv} + } + + snapshot_boot_lv() + { + local src_lv=$1 + local boot_lv=$2 + + [ "$src_lv" == "$boot_lv" ] && return + + lvremove --force ${boot_lv} + lvcreate --size 10G --name $(basename ${boot_lv}) --snapshot ${src_lv} || exit + } + + main() + { + analysis_kernel_cmdline_params + + if [ -z "${use_root_partition}" ]; then + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${TIMESTAMP}" + rsync_src_lv ${src_lv} + else + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + lvdisplay ${src_lv} > /dev/null || exit + fi + + if [ -z "${save_root_partition}" ]; then + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03" + else + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + fi + + snapshot_boot_lv ${src_lv} ${boot_lv} + boot_from_boot_lv + } + + main() + ``` 4. execute the job -- 2.23.0

2 2

[PATCH v4 lkp-tests] stats/ansible_test: stats result for ansible migrating
by Wang Chenglong 23 Feb '21

23 Feb '21

[How] ansible_stats_openeuler < ${job_output} Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 135 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 124 insertions(+), 11 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 4c7d07b3e..bd60a3ebc 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,15 +1,128 @@ #!/usr/bin/env ruby -while (line = STDIN.gets) - case line.chomp! - when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ - puts "ansible_test.total.nr_ok: #{$1}" - puts "ansible_test.total.nr_changed: #{$2}" - puts "ansible_test.total.nr_failed: #{$4}" - puts "ansible_test.total.nr_skipped: #{$5}" - puts "ansible_test.total.nr_rescued: #{$6}" - puts "ansible_test.total.nr_ignored: #{$7}" - when /playbook_run_on_fail/ - puts line +require 'json' + +def parse_msg_1(ansible_failed_json) + case ansible_failed_json.chomp + + # When running some plug-in, the file was not found + when /No file was found when using (.+)\./ + puts "files.error.no-file-was-found-when-using.#{$1}: 1" + puts "files.error.no-file-was-found-when-using.#{$1}.message: #{@ansible_failed_info}" + + # {"changed": false, "cmd": "csf -v", "msg": "[Errno 2] No such file or directory: b'csf': b'csf'", "rc": 2} + when /No such file or directory: b (.+) :/ + puts "error.no-such-file-or-directory.#{$1}: 1" + puts "error.no-such-file-or-directory.#{$1}.message: #{@ansible_failed_info}" + + # The task includes an option with an undefined variable + when /: (.+) is undefined/ + puts "syntax.error.an-undefined-variable.#{$1}: 1" + puts "syntax.error.an-undefined-variable.#{$1}.message: #{@ansible_failed_info}" + + # {"changed": true, "cmd": ["packaging/installer/install-required-packages.sh", "--non-interactive", "netdata"], + # "msg": "non-zero return code", "rc": 1} + when /non-zero return code/ + puts "error.non-zero-return-code.#{ansible_failed_json}: 1" + puts "error.non-zero-return-code.#{ansible_failed_json}.message: #{@ansible_failed_info}" + + # Some service could not find the requested host + when /Could not find the requested service (.+): host/ + puts "request.error.could-not-find-service-host.#{$1}: 1" + puts "request.error.could-not-find-service-host.#{$1}.messgae: #{@ansible_failed_info}" + + # Unable to start some services + when /Unable to start service (.+):/ + puts "service.error.unable-to-start-service.#{$1}: 1" + puts "service.error.unable-to-start-service.#{$1}.message: #{@ansible_failed_info}" + end +end + +def parse_msg_2(ansible_failed_json) + case ansible_failed_json.chomp + + # {"changed": false, "msg": "Failed to download packages: Cannot download 7.10.2/filebeat-7.10.2-aarch64.rpm: + # All mirrors were tried", "results": []} + when /Failed to download packages: Cannot download (.+):/ + puts "download.error.failed-to-download-package.#{$1}: 1" + puts "download.error.failed-to-download-package.#{$1}.message: #{@ansible_failed_info}" + + # {"changed": false, "msg": "Failure downloading https://rpms.remirepo.net/enterprise/remi-release-20.rpm, + # HTTP Error 404: Not Found"} + when /Failure downloading (.+), / + puts "download.error.Failure-downloading.[#{$1}]: 1" + puts "download.error.Failure-downloading.[#{$1}]: #{@ansible_failed_info}" + + # Failed to download the yum repodata + # {"changed": false, "msg": "Failed to download metadata for repo 'docker-ce-stable': Cannot download repomd.xml", + # "rc": 1, "results": []} + when /Failed to download metadata for repo (.+) :/ + puts "download.error.failed-to-download-metadata.#{$1}: 1" + puts "download.error.failed-to-download-metadata.#{$1}.message: #{@ansible_failed_info}" + + # Some role in not sipported this system + # {"changed": false, "msg": "Distribution openEuler is not supported by this role!"} + when /(.+) is not supported/ + puts "system.error.not-supported.#{$1.gsub(/ /, '-')}: 1" + puts "system.error.not-supported.message: #{@ansible_failed_info}" + + # failed to create temporary content file: The read operation timed out. + when /The read operation timed out/ + puts 'connection.error.read-timed-out: 1' + puts "connection.error.read-timed-out.message: #{@ansible_failed_info}" end end + +def parse_failures(ansible_failed_json) + ansible_failed_json.each do |items| + case items.chomp + # Failed to install some of the specified packages. + # {"changed": false, "failures": ["No package erlang available."], "msg": + # "Failed to install some of the specified packages", "rc": 1, "results": []} + when /No package (.+) available/ + puts "yum.error.no-package-available.#{$1}: 1" + puts "yum.error.no-package-available.#{$1}.message: #{@ansible_failed_info}" + end + end +end + +def parse_message(ansible_failed_json) + case ansible_failed_json.chomp + + # Could not find or access ansible yaml + # {"ansible_facts": {}, "ansible_included_var_files": [], "changed": false, "message": + # "Could not find or access 'openEuler.yml' Searched in: + # /root/.ansible/roles/ansible-unzip/vars/openEuler.yml /root/.ansible/roles/ansible-unzip/openEuler.yml i + # /root/.ansible/roles/ansible-unzip/tasks/vars/openEuler.yml /root/.ansible/roles/ansible-unzip/tasks/openEuler.yml + # /root/.ansible/vars/openEuler.yml /root/.ansible/openEuler.yml on the Ansible Controller. + # If you are using a module and expect the file to exist on the remote, see the remote_src option"} + when /Could not find or access (.+) / + puts "yaml.error.not-find-or-access-yaml.#{$1}: 1" + puts "yaml.error.not-find-or-access-yaml.#{$1}.message: #{@ansible_failed_info}" + end +end + +def common_error_id(line) + line = line.chomp + line.gsub(/'/, ' ') + line.gsub(/{%/, ' ') + line.gsub(/%}/, ' ') + line.gsub(/{},/, '[],') + line.gsub(/\\n\\t/, ' ') + line +end + +while (line = STDIN.gets) + line = common_error_id(line) + next unless line =~ /(FAILED!|failed:).*({.*})/ + + @ansible_failed_info = $2 + next if @ansible_failed_info.empty? + + ansible_failed_json = JSON.load(@ansible_failed_info) + + parse_msg_1 ansible_failed_json['msg'] unless ansible_failed_json['msg'].nil? + parse_msg_2 ansible_failed_json['msg'] unless ansible_failed_json['msg'].nil? + parse_failures ansible_failed_json['failures'] unless ansible_failed_json['failures'].nil? + parse_message ansible_failed_json['message'] unless ansible_failed_json['message'].nil? +end -- 2.23.0

4 6

[PATCH lkp-tests] lib/job2sh.rb: delete secrets field when job2sh
by Cao Xueliang 23 Feb '21

23 Feb '21

We don't show the secrets field in job.sh file, so delete secrets field when job2sh. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/job2sh.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 7f9cf562f..82ec7d249 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -318,6 +318,7 @@ class Job2sh < Job unless job job = (@jobx || @job).clone # a shallow copy so that delete_if won't impact @job job.delete_if { |key, val| parse_one([], key, val, :PASS_EXPORT_ENV) } + job.delete_if { |key, val| key == "secrets" } end @cur_func = :run_job -- 2.23.0

1 0

[PATCH compass-ci] sched: add the secrets field processing logic
by Cao Xueliang 23 Feb '21

23 Feb '21

You can add the secrets field which you don't want to make public but the SUT need it in job.yaml. secrets: my_token:xxx The scheduler will save the secrets value to the redis id2secrets table, then delete the field from job.yaml. When consume the job, the scheduler will read the value from redis, then create a secrets.yaml in job.cgz. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sbin/create-job-cpio.sh | 1 + src/scheduler/create_job_cpio.cr | 13 +++++++++++++ src/scheduler/submit_job.cr | 7 +++++++ 3 files changed, 21 insertions(+) diff --git a/sbin/create-job-cpio.sh b/sbin/create-job-cpio.sh index e18bc0d..afb316c 100755 --- a/sbin/create-job-cpio.sh +++ b/sbin/create-job-cpio.sh @@ -9,6 +9,7 @@ cd "$1" || exit install -m775 -D -t lkp/scheduled job.sh install -m664 -D -t lkp/scheduled job.yaml +[ -f secrets.yaml ] && install -m664 -D -t lkp/scheduled secrets.yaml find lkp | cpio --quiet -o -H newc | gzip > job.cgz diff --git a/src/scheduler/create_job_cpio.cr b/src/scheduler/create_job_cpio.cr index 51d79f7..5634f67 100644 --- a/src/scheduler/create_job_cpio.cr +++ b/src/scheduler/create_job_cpio.cr @@ -74,8 +74,21 @@ class Sched script_lines = JSON.parse(script_lines) end + def create_secrets_yaml(job_id, base_dir) + secrets = @redis.hash_get("id2secrets", job_id) + return nil unless secrets + + secrets_yaml = base_dir + "/#{job_id}/secrets.yaml" + prepare_dir(secrets_yaml) + + File.open(secrets_yaml, "w") do |file| + YAML.dump(JSON.parse(secrets), file) + end + end + def create_job_cpio(job_content : JSON::Any, base_dir : String) job_content = job_content.as_h + create_secrets_yaml(job_content["id"], base_dir) # put job2sh in an array if job_content.has_key?("job2sh") diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 1837cb1..fe9efa1 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -144,8 +144,15 @@ class Sched JSON.parse(response[1].to_json)["id"].to_s if response[0] == 200 end + def save_secrets(job, job_id) + return nil unless job["secrets"]? + + @redis.hash_set("id2secrets", job_id, job["secrets"]?.to_json) + job.delete("secrets") + end # add job content to es and return a response def add_job(job, job_id) + save_secrets(job, job_id) job.update_id(job_id) @es.set_job_content(job) end -- 2.23.0

1 0

2025

2024

2023

2022

2021

2020

Compass-ci