February 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] doc/manual/submit-job.md: fix link error
by Wei Jihui 24 Feb '21

24 Feb '21

the links file name has changed to english. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- doc/manual/submit-job.en.md | 2 +- doc/manual/submit-job.zh.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/manual/submit-job.en.md b/doc/manual/submit-job.en.md index 2748fbe..de73f8f 100644 --- a/doc/manual/submit-job.en.md +++ b/doc/manual/submit-job.en.md @@ -2,7 +2,7 @@ ### Prerequisites -The Compass-CI client has been installed. For details, see [Installing the Local Compass-CI Client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/��…. +The Compass-CI client has been installed. For details, see [Installing the Local Compass-CI Client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/in…. ### Purpose diff --git a/doc/manual/submit-job.zh.md b/doc/manual/submit-job.zh.md index 92faa5c..77e9566 100644 --- a/doc/manual/submit-job.zh.md +++ b/doc/manual/submit-job.zh.md @@ -2,7 +2,7 @@ ### �� -��[��compass-ci��.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/��compass-ci��.md )�� +��[��compass-ci��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/install-cci-client.md )�� ### �� -- 2.23.0

1 0

[PATCH compass-ci] doc/stats-rule.md
by Wang Chenglong 24 Feb '21

24 Feb '21

[Why] The rules to follow when writing the stats script. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/stats-rule.md | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100644 doc/stats-rule.md diff --git a/doc/stats-rule.md b/doc/stats-rule.md new file mode 100644 index 0000000..94affa7 --- /dev/null +++ b/doc/stats-rule.md @@ -0,0 +1,37 @@ +${LKP_SERVER}/stats/${script} stat�� +========================================== + +## ��es�� + +1.��error_id�� + - original: {"changed": false, "failures": ["No package yum-plugin-priorities available."], "msg": "Failed to + install some of the specified packages", "rc": 1, "results": []} + + - error_id: error.No-package-yum-plugin-priorities-available: 1 + +2.�� + - catch all kind errors + ��error_id�� + ��jobs��'��'��es��error_id.��stats/${script}�� + +3.��error message�� + # original error message + # {"changed": false, "msg": "Failed to download packages: Cannot download 7.10.2/filebeat-7.10.2-aarch64.rpm: + # All mirrors were tried", "results": []} + when /(Failed to download packages: Cannot download .+):/ + output_error $1 + end + +4.��error_id�� + - �� + def common_error_id(line) + line = line.chomp + line.gsub!(/ /, '-') + line.gsub!(/:/, '') # error.:-'__mysql_packages'-is-undefined: 1 + line.gsub!(/,/, '') + line.gsub!(/'/, '') # Failed-to-download-metadata-for-repo-'dockerrepo: 1 + line.gsub!(/!/, '') # Distribution-openEuler-is-not-supported-by-this-role!: 1 + line.gsub!(/{%/, ' ') + line.gsub!(/%}/, ' ') + line + end -- 2.23.0

2 2

[PATCH lab-z9] hosts/*: add default hdd_partitions
by Cui Yili 24 Feb '21

24 Feb '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- hosts/vm-1p1g | 1 + hosts/vm-2p16g | 1 + hosts/vm-2p1g | 1 + hosts/vm-2p4g | 1 + hosts/vm-2p8g | 1 + 5 files changed, 5 insertions(+) diff --git a/hosts/vm-1p1g b/hosts/vm-1p1g index 3af3646..b278e80 100644 --- a/hosts/vm-1p1g +++ b/hosts/vm-1p1g @@ -4,3 +4,4 @@ template: kvm nr_node: 1 nr_cpu: 1 memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p16g b/hosts/vm-2p16g index 10847a7..cd49afc 100644 --- a/hosts/vm-2p16g +++ b/hosts/vm-2p16g @@ -4,3 +4,4 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 16G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p1g b/hosts/vm-2p1g index 0293ce9..4fffd2a 100644 --- a/hosts/vm-2p1g +++ b/hosts/vm-2p1g @@ -4,3 +4,4 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p4g b/hosts/vm-2p4g index 219c4c9..41d0a3b 100644 --- a/hosts/vm-2p4g +++ b/hosts/vm-2p4g @@ -4,3 +4,4 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 4G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p8g b/hosts/vm-2p8g index 6249322..8bc5cad 100644 --- a/hosts/vm-2p8g +++ b/hosts/vm-2p8g @@ -4,3 +4,4 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: /dev/vda -- 2.23.0

2 1

[PATCH compass-ci] doc/manual/submit-job.md: fix link error
by Wei Jihui 24 Feb '21

24 Feb '21

the links file name has changed to english. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- doc/manual/submit-job.en.md | 2 +- doc/manual/submit-job.zh.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/manual/submit-job.en.md b/doc/manual/submit-job.en.md index 2748fbe..de73f8f 100644 --- a/doc/manual/submit-job.en.md +++ b/doc/manual/submit-job.en.md @@ -2,7 +2,7 @@ ### Prerequisites -The Compass-CI client has been installed. For details, see [Installing the Local Compass-CI Client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/��…. +The Compass-CI client has been installed. For details, see [Installing the Local Compass-CI Client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/in…. ### Purpose diff --git a/doc/manual/submit-job.zh.md b/doc/manual/submit-job.zh.md index 92faa5c..77e9566 100644 --- a/doc/manual/submit-job.zh.md +++ b/doc/manual/submit-job.zh.md @@ -2,7 +2,7 @@ ### �� -��[��compass-ci��.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/��compass-ci��.md )�� +��[��compass-ci��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/install-cci-client.md )�� ### �� -- 2.23.0

1 0

[PATCH v4 compass-ci] doc: optimize os_mount.md
by Yu Chuan 24 Feb '21

24 Feb '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 265 ++++++++++++++++++++++++++++++-------------- 1 file changed, 181 insertions(+), 84 deletions(-) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index 226251a8bace..1e6b349a136d 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -1,94 +1,191 @@ -# Summary -========= +Summary +======= -`os_mount` defines the mount method of testbox's root partition. +`os_mount` defines the rootfs type. -It has the following optional values: - - nfs - - cifs - - initramfs - - container - - local +The below types are supported: + - nfs + - cifs + - initramfs + - container + - local Usage example: - - submit iperf.yaml testbox=vm-2p8g os_mount=nfs - - submit iperf.yaml testbox=vm-2p8g os_mount=cifs - - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs - - submit iperf.yaml testbox=dc-8g os_mount=container - - submit iperf.yaml testbox=vm-2p8g os_mount=local - -# Optional Values -================= - -## Work flow when os_mount=local - -1. user submit job with os_mount: local - - optional {kernel_append_root}: - - save_root_partition: give an iconic string, then the root partition data used by this job will be retained. - and you can use the root partition data of this job in the futher job by {use_root_partition}. - - use_root_partition : give an iconic string, then the root partition data generated by the - pervious job which has the {save_root_partition} field will be used. - - root_partition_size: specify the size of root partition, *default is 10G*. - - - tips: - - value of {save_root_partition} and {use_root_partition} need to be iconic enough, uuid is recommended. - - value of {save_root_partition} and {use_root_partition} will be the suffix of logical volume. + ``` + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=nfs + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=cifs + submit -m -c borrow-1h.yaml testbox=vm-2p16g os_mount=initramfs + submit -m -c borrow-1h.yaml testbox=dc-8g os_mount=container + submit -m -c borrow-1h.yaml testbox=taishan200-2280-2s48p-256g os_mount=local + ``` + +os_mount=local +============== + +`nfs`, `cifs`, `initramfs`, `container`, rootfs of them all are in memory. +So if you want to run your job in disk file system, you need to specify +`os_mount=local` in your job.yaml. + +BTW, the job's rootfs will be placed on a logical volume. + +## description + +### temporary rootfs data *Defaultly* + +If you only add `os_mount: local` in your job.yaml, the logical volume of rootfs +will be deleted and re-created in initrd stage before every job executes. + +The brief flow is as follows: + ``` + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - will never boot from this lv, just use it as a source data, and take a + snapshot of it to get the real boot logical volume. + - src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + - ${timestamp} is just a version number, which is the latest rootfs version + on the nfs server in cluster. + ${boot_lv} -- boot logical volume: + - will boot from this lv. + - boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: create it, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} exists: delete it + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. + ``` + +### persistent rootfs data + +When you need to persist the rootfs data of a job, and use it in the subsequent job(s), +two fields in `kernel_append_root` will help you: `save_root_partition`, `use_root_partition`. + +The brief flow is as follows: + ``` + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - if have ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${use_root_partition} + - if no ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + ${boot_lv} -- boot logical volume: + - if have ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${save_root_partition} + - if no ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: + - if have ${use_root_partition}, exit 1. + - if no ${use_root_partition}, create ${src_lv}, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} != ${src_lv}: + - if ${boot_lv} exists: delete it + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. + ``` + +Demo usage: + ``` + - in 20210218, you submit a job-20210218.yaml, and you want to persist the rootfs data of + job-20210218.yaml so that it can be used by the subsequent jobs. + Then you need add the follow field in your job-20210218.yaml: + kernel_append_root: save_root_partition=zhangsan_local_for_iperf_20210218 + + - in 20210219, you submit a job-20210219.yaml, and you want to use the rootfs data of + job-20210218.yaml. + Then you need add the follow field in your job-20210219.yaml: + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 + ``` + +Notes: + - obviously, ${save_root_partition} and ${use_root_partition} must be the same, and unique. + - you must 'save_root_partition' firstly, then you can 'use_root_partition'. otherwise, your job will fail. + +## work flow + +1. user submit job + job fields: + ``` + os: openeuler + os_arch: aarch64 + os_version: 20.03 + os_mount: local + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 + ``` 2. scheduler return the ipxe_str to testbox - ``` - dhcp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz - imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} - boot - ``` + ``` + dhcp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/initrd.lkp + kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src=${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} initrd=initrd.lkp ${kernel_append_root} + boot + ``` 3. dracut step of boot - ``` - if have {use_root_partition}; then - base_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 - - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - - if {save_root_partition} == {use_root_partition}; then - boot from /dev/mappaer/os-{boot_lv_name} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - else - boot_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - boot from /dev/mappaer/os-{boot_lv_name} - fi - - else: - base_lv_name=openeuler_aarch64_20.03_{timestamp} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { - # create logical volume - lvcreate -L {root_partition_size} -n {base_lv_name} os || exit 1 - - # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} - mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt - mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 - cp -a /mnt/. /mnt1/ - umount /mnt /mnt1 - - # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly - lvchange -p r /dev/mapper/os-{base_lv_name} - } - - boot_lv_name=openeuler_aarch64_20.03 - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - fi - ``` + ``` + analysis_kernel_cmdline_params() + { + for i in $(cat /proc/cmdline) + do + [ "$i" =~ "rootfs_src=" ] && export ROOTFS_SRC=${i#rootfs_src=} + done + + [ -z "${ROOTFS_SRC}" ] && die "cannot find var in kernel cmdline params: rootfs_src" + + export TIMESTAMP=$(basename ${ROOTFS_SRC}) + } + + rsync_src_lv() + { + local src_lv=$1 + + lvdisplay ${src_lv} > /dev/null && return + + # create logical volume + lvcreate --size 10G --name $(basename ${src_lv}) os || exit + + # rsync nfsroot to ${src_lv} + mount -t nfs ${ROOTFS_SRC} /mnt + mkdir /mnt1 && mount ${src_lv} /mnt1 + cp -a /mnt/. /mnt1/ + umount /mnt /mnt1 + + # change permission of ${src_lv} to readonly + lvchange --permission r ${src_lv} + } + + snapshot_boot_lv() + { + local src_lv=$1 + local boot_lv=$2 + + [ "$src_lv" == "$boot_lv" ] && return + + lvremove --force ${boot_lv} + lvcreate --size 10G --name $(basename ${boot_lv}) --snapshot ${src_lv} || exit + } + + main() + { + analysis_kernel_cmdline_params + + if [ -z "${use_root_partition}" ]; then + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${TIMESTAMP}" + rsync_src_lv ${src_lv} + else + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + lvdisplay ${src_lv} > /dev/null || exit + fi + + if [ -z "${save_root_partition}" ]; then + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03" + else + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + fi + + snapshot_boot_lv ${src_lv} ${boot_lv} + boot_from_boot_lv + } + + main() + ``` 4. execute the job -- 2.23.0

2 2

[PATCH lkp-tests] jobs/: add os-multi-dimension-arch.yaml
by Li Ping 23 Feb '21

23 Feb '21

2 2

[PATCH lkp-tests] lib: export secrets.yaml in job_init
by Cao Xueliang 23 Feb '21

23 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/job-init.sh | 2 ++ lib/job.sh | 18 +++++++++++++++--- 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/lib/job-init.sh b/lib/job-init.sh index 1fb898872..27d3c6d0b 100755 --- a/lib/job-init.sh +++ b/lib/job-init.sh @@ -1,4 +1,5 @@ . $LKP_SRC/lib/upload.sh +. $LKP_SRC/lib/job.sh mount_cgroup() { @@ -270,4 +271,5 @@ job_init() job_redirect_stdout_stderr job_env + read_secret_vars } diff --git a/lib/job.sh b/lib/job.sh index dc65f68d8..87ca7ee67 100755 --- a/lib/job.sh +++ b/lib/job.sh @@ -1,8 +1,20 @@ #!/bin/sh +read_secret_vars() +{ + read_yaml_vars "$(dirname $job_file)/secrets.yaml" secrets_ +} + read_env_vars() { - [ -f "$TMP/env.yaml" ] || return 0 + read_yaml_vars "$TMP/env.yaml" +} + +read_yaml_vars() +{ + local file=$1 + local prefix=$2 + [ -f "$file" ] || return 0 local key local val @@ -11,8 +23,8 @@ read_env_vars() do [ "${key%[a-zA-Z0-9_]:}" != "$key" ] || continue key=${key%:} - export "$key=$val" - done < $TMP/env.yaml + export "$prefix$key=$val" + done < $file return 0 } -- 2.23.0

1 0

find latest github git pushes
by Wu Fengguang 23 Feb '21

23 Feb '21

Hi Yuanchao, I researched github and have 2 finding 1) a registered user can watch a number of repos and get notifications. however the notification type does not include "git push/new commit". There is only "new pull request", which looks much less useful for us. 2) we can still take advantage of its search API to find latest updated repos, like this https://github.com/search?l=&o=desc&q=pushed%3A%3E2021-02-21+filename%3AREA… 7,543 repository results clean-code-craft-tcq-1/modular-c-AnPra7 modular-c-AnPra7 created by GitHub Classroom C MIT license Updated 3 minutes ago trpo2021/chessviz-Jamesss1337 chessviz-Jamesss1337 created by GitHub Classroom C Updated 3 minutes ago mubu43/ExploringC C Updated 3 minutes ago BlankyShi/gitTest gitTest C Updated 4 minutes ago aabelque/malloc C Updated 4 minutes ago ythan88/r-card C Updated 4 minutes ago koipkoi/pokeemerald-kr A korean translation project for Pokémon Emerald. C Updated 4 minutes ago OpenMPT/openmpt Official read-only git mirror of the OpenMPT and libopenmpt subversion repository at https://source.openmpt.org/. Thi… libopenmpt tracker audio music demoscene music-creation protracker impulsetracker screamtracker fasttracker 223 C Updated 4 minutes ago silabs-DenverL/ADCxSyncCapture ADC0 and ADC1 Synchronization Capture C MIT license Updated 4 minutes ago gopro/gopro-lib-node.gl OpenGL engine for building and rendering graph-based scenes 32 C Apache-2.0 license Updated 4 minutes ago Thanks, Fengguang

1 2

[PATCH compass-ci] 2-network/iptables: fix iptables holding the xtables lock
by Cui Yili 23 Feb '21

23 Feb '21

[why] error: Another app is currently holding the xtables lock. Perhaps you want to use the -w option? [how] add lock wait time. iptables -w 10... Signed-off-by: Cui Yili <2268260388(a)qq.com> --- sparrow/2-network/iptables | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/sparrow/2-network/iptables b/sparrow/2-network/iptables index 48794f5..73f9749 100755 --- a/sparrow/2-network/iptables +++ b/sparrow/2-network/iptables @@ -8,9 +8,9 @@ BR0_IFACE=br0 BR0_SUBNET=172.18.0.0/16 # iptables -t nat -F -iptables -I FORWARD 1 -j ACCEPT -iptables -t nat -A POSTROUTING -o "$PUB_IFACE" -s $BR0_SUBNET -j MASQUERADE -iptables -t nat -A POSTROUTING -o $BR0_IFACE -d $BR0_SUBNET -j MASQUERADE +iptables -w 10 -I FORWARD 1 -j ACCEPT +iptables -w 10 -t nat -A POSTROUTING -o "$PUB_IFACE" -s $BR0_SUBNET -j MASQUERADE +iptables -w 10 -t nat -A POSTROUTING -o $BR0_IFACE -d $BR0_SUBNET -j MASQUERADE command -v firewall-cmd > /dev/null || { exit 0 -- 2.23.0

1 0

[PATCH compass-ci] support-os-os_arch-os_version.md: delete the cifs method of archlinux
by Wang Chenglong 23 Feb '21

23 Feb '21

[Why] When start archlinux with CIFS, the system will get mangled names error. So only show the NFS method for user. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/support-os-os_arch-os_version.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/support-os-os_arch-os_version.md b/doc/support-os-os_arch-os_version.md index 5d30486..93fb47b 100644 --- a/doc/support-os-os_arch-os_version.md +++ b/doc/support-os-os_arch-os_version.md @@ -18,5 +18,5 @@ centos aarch64 7 container, 7.8.2003 cifs/nfs, debian aarch64 sid cifs/nfs, ,initramfs x86 sid cifs/nfs, -archlinux aarch64 2020-11-12 cifs/nfs, +archlinux aarch64 2020-11-12 nfs, ``` -- 2.23.0

1 0

[PATCH v3 compass-ci] doc: optimize os_mount.md
by Yu Chuan 23 Feb '21

23 Feb '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 263 ++++++++++++++++++++++++++++++-------------- 1 file changed, 179 insertions(+), 84 deletions(-) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index 226251a8bace..0d0afd6469a8 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -1,94 +1,189 @@ -# Summary -========= +Summary +======= -`os_mount` defines the mount method of testbox's root partition. +`os_mount` defines the rootfs type. -It has the following optional values: - - nfs - - cifs - - initramfs - - container - - local +The below types are supported: + - nfs + - cifs + - initramfs + - container + - local Usage example: - - submit iperf.yaml testbox=vm-2p8g os_mount=nfs - - submit iperf.yaml testbox=vm-2p8g os_mount=cifs - - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs - - submit iperf.yaml testbox=dc-8g os_mount=container - - submit iperf.yaml testbox=vm-2p8g os_mount=local - -# Optional Values -================= - -## Work flow when os_mount=local - -1. user submit job with os_mount: local - - optional {kernel_append_root}: - - save_root_partition: give an iconic string, then the root partition data used by this job will be retained. - and you can use the root partition data of this job in the futher job by {use_root_partition}. - - use_root_partition : give an iconic string, then the root partition data generated by the - pervious job which has the {save_root_partition} field will be used. - - root_partition_size: specify the size of root partition, *default is 10G*. - - - tips: - - value of {save_root_partition} and {use_root_partition} need to be iconic enough, uuid is recommended. - - value of {save_root_partition} and {use_root_partition} will be the suffix of logical volume. + ``` + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=nfs + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=cifs + submit -m -c borrow-1h.yaml testbox=vm-2p16g os_mount=initramfs + submit -m -c borrow-1h.yaml testbox=dc-8g os_mount=container + submit -m -c borrow-1h.yaml testbox=taishan200-2280-2s48p-256g os_mount=local + ``` + +os_mount=local +============== + +`nfs`, `cifs`, `initramfs`, `container`, rootfs of them all are in memory. +So if you want to run your job in disk file system, you need to specify +`os_mount=local` in your job.yaml. + +BTW, the job's rootfs will be placed on a logical volume. + +## description + +### temporary rootfs data *Defaultly* + +If you only add `os_mount: local` in your job.yaml, the logical volume of rootfs +will be deleted and re-created in initrd stage before every job executes. + +The brief flow is as follow: + ``` + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - will never boot form this lv, just use it as a source data, and take a + snapshot of it to get the real boot logical volume. + - srv_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + ${boot_lv} -- boot logical volume: + - will boot from this. + - boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: create it, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} exists: delete it + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. + ``` + +### persistent rootfs data + +When you need to persist the rootfs data of a job, and use it in the subsequent job(s), +two fields in `kernel_append_root` will help you: `save_root_partition`, `use_root_partition`. + +The brief flow is as follow: + ``` + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - if have ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${use_root_partition} + - if no ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + ${boot_lv} -- boot logical volume: + - if have ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${save_root_partition} + - if no ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: + - if have ${use_root_partition}, exit 1. + - if no ${use_root_partition}, create ${src_lv}, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} != ${src_lv}: + - if ${boot_lv} exists: delete it + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. + ``` + +Demo usage: + ``` + - in 20210218, you submit a job-20210218.yaml, and you want to persist the rootfs data of + job-20210218.yaml so that it can be used by the subsequent jobs. + Then you need add the follow field in your job-20210218.yaml: + kernel_append_root: save_root_partition=zhangsan_local_for_iperf_20210218 + + - in 20210219, you submit a job-20210219.yaml, and you want to use the rootfs data of + job-20210218.yaml. + Then you need add the follow field in your job-20210219.yaml: + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 + ``` + +Notes: + - obviously, ${save_root_partition} and ${use_root_partition} must be the same, and unique. + - you must 'save_root_partition' firstly, then you can 'use_root_partition'. otherwise, your job will fail. + +## work flow + +1. user submit job + job fields: + ``` + os: openeuler + os_arch: aarch64 + os_version: 20.03 + os_mount: local + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 + ``` 2. scheduler return the ipxe_str to testbox - ``` - dhcp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz - imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} - boot - ``` + ``` + dhcp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/initrd.lkp + kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src=${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} initrd=initrd.lkp ${kernel_append_root} + boot + ``` 3. dracut step of boot - ``` - if have {use_root_partition}; then - base_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 - - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - - if {save_root_partition} == {use_root_partition}; then - boot from /dev/mappaer/os-{boot_lv_name} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - else - boot_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - boot from /dev/mappaer/os-{boot_lv_name} - fi - - else: - base_lv_name=openeuler_aarch64_20.03_{timestamp} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { - # create logical volume - lvcreate -L {root_partition_size} -n {base_lv_name} os || exit 1 - - # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} - mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt - mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 - cp -a /mnt/. /mnt1/ - umount /mnt /mnt1 - - # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly - lvchange -p r /dev/mapper/os-{base_lv_name} - } - - boot_lv_name=openeuler_aarch64_20.03 - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - fi - ``` + ``` + analysis_kernel_cmdline_params() + { + for i in $(cat /proc/cmdline) + do + [ "$i" =~ "rootfs_src=" ] && export ROOTFS_SRC=${i#rootfs_src=} + done + + [ -z "${ROOTFS_SRC}" ] && die "cannot find var in kernel cmdline params: rootfs_src" + + export TIMESTAMP=$(basename ${ROOTFS_SRC}) + } + + rsync_src_lv() + { + local src_lv=$1 + + lvdisplay ${src_lv} > /dev/null && return + + # create logical volume + lvcreate --size 10G --name $(basename ${src_lv}) os || exit + + # rsync nfsroot to ${src_lv} + mount -t nfs ${ROOTFS_SRC} /mnt + mkdir /mnt1 && mount ${src_lv} /mnt1 + cp -a /mnt/. /mnt1/ + umount /mnt /mnt1 + + # change permission of ${src_lv} to readonly + lvchange --permission r ${src_lv} + } + + snapshot_boot_lv() + { + local src_lv=$1 + local boot_lv=$2 + + [ "$src_lv" == "$boot_lv" ] && return + + lvremove --force ${boot_lv} + lvcreate --size 10G --name $(basename ${boot_lv}) --snapshot ${src_lv} || exit + } + + main() + { + analysis_kernel_cmdline_params + + if [ -z "${use_root_partition}" ]; then + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${TIMESTAMP}" + rsync_src_lv ${src_lv} + else + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + lvdisplay ${src_lv} > /dev/null || exit + fi + + if [ -z "${save_root_partition}" ]; then + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03" + else + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + fi + + snapshot_boot_lv ${src_lv} ${boot_lv} + boot_from_boot_lv + } + + main() + ``` 4. execute the job -- 2.23.0

2 2

[PATCH v4 lkp-tests] stats/ansible_test: stats result for ansible migrating
by Wang Chenglong 23 Feb '21

23 Feb '21

[How] ansible_stats_openeuler < ${job_output} Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 135 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 124 insertions(+), 11 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 4c7d07b3e..bd60a3ebc 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,15 +1,128 @@ #!/usr/bin/env ruby -while (line = STDIN.gets) - case line.chomp! - when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ - puts "ansible_test.total.nr_ok: #{$1}" - puts "ansible_test.total.nr_changed: #{$2}" - puts "ansible_test.total.nr_failed: #{$4}" - puts "ansible_test.total.nr_skipped: #{$5}" - puts "ansible_test.total.nr_rescued: #{$6}" - puts "ansible_test.total.nr_ignored: #{$7}" - when /playbook_run_on_fail/ - puts line +require 'json' + +def parse_msg_1(ansible_failed_json) + case ansible_failed_json.chomp + + # When running some plug-in, the file was not found + when /No file was found when using (.+)\./ + puts "files.error.no-file-was-found-when-using.#{$1}: 1" + puts "files.error.no-file-was-found-when-using.#{$1}.message: #{@ansible_failed_info}" + + # {"changed": false, "cmd": "csf -v", "msg": "[Errno 2] No such file or directory: b'csf': b'csf'", "rc": 2} + when /No such file or directory: b (.+) :/ + puts "error.no-such-file-or-directory.#{$1}: 1" + puts "error.no-such-file-or-directory.#{$1}.message: #{@ansible_failed_info}" + + # The task includes an option with an undefined variable + when /: (.+) is undefined/ + puts "syntax.error.an-undefined-variable.#{$1}: 1" + puts "syntax.error.an-undefined-variable.#{$1}.message: #{@ansible_failed_info}" + + # {"changed": true, "cmd": ["packaging/installer/install-required-packages.sh", "--non-interactive", "netdata"], + # "msg": "non-zero return code", "rc": 1} + when /non-zero return code/ + puts "error.non-zero-return-code.#{ansible_failed_json}: 1" + puts "error.non-zero-return-code.#{ansible_failed_json}.message: #{@ansible_failed_info}" + + # Some service could not find the requested host + when /Could not find the requested service (.+): host/ + puts "request.error.could-not-find-service-host.#{$1}: 1" + puts "request.error.could-not-find-service-host.#{$1}.messgae: #{@ansible_failed_info}" + + # Unable to start some services + when /Unable to start service (.+):/ + puts "service.error.unable-to-start-service.#{$1}: 1" + puts "service.error.unable-to-start-service.#{$1}.message: #{@ansible_failed_info}" + end +end + +def parse_msg_2(ansible_failed_json) + case ansible_failed_json.chomp + + # {"changed": false, "msg": "Failed to download packages: Cannot download 7.10.2/filebeat-7.10.2-aarch64.rpm: + # All mirrors were tried", "results": []} + when /Failed to download packages: Cannot download (.+):/ + puts "download.error.failed-to-download-package.#{$1}: 1" + puts "download.error.failed-to-download-package.#{$1}.message: #{@ansible_failed_info}" + + # {"changed": false, "msg": "Failure downloading https://rpms.remirepo.net/enterprise/remi-release-20.rpm, + # HTTP Error 404: Not Found"} + when /Failure downloading (.+), / + puts "download.error.Failure-downloading.[#{$1}]: 1" + puts "download.error.Failure-downloading.[#{$1}]: #{@ansible_failed_info}" + + # Failed to download the yum repodata + # {"changed": false, "msg": "Failed to download metadata for repo 'docker-ce-stable': Cannot download repomd.xml", + # "rc": 1, "results": []} + when /Failed to download metadata for repo (.+) :/ + puts "download.error.failed-to-download-metadata.#{$1}: 1" + puts "download.error.failed-to-download-metadata.#{$1}.message: #{@ansible_failed_info}" + + # Some role in not sipported this system + # {"changed": false, "msg": "Distribution openEuler is not supported by this role!"} + when /(.+) is not supported/ + puts "system.error.not-supported.#{$1.gsub(/ /, '-')}: 1" + puts "system.error.not-supported.message: #{@ansible_failed_info}" + + # failed to create temporary content file: The read operation timed out. + when /The read operation timed out/ + puts 'connection.error.read-timed-out: 1' + puts "connection.error.read-timed-out.message: #{@ansible_failed_info}" end end + +def parse_failures(ansible_failed_json) + ansible_failed_json.each do |items| + case items.chomp + # Failed to install some of the specified packages. + # {"changed": false, "failures": ["No package erlang available."], "msg": + # "Failed to install some of the specified packages", "rc": 1, "results": []} + when /No package (.+) available/ + puts "yum.error.no-package-available.#{$1}: 1" + puts "yum.error.no-package-available.#{$1}.message: #{@ansible_failed_info}" + end + end +end + +def parse_message(ansible_failed_json) + case ansible_failed_json.chomp + + # Could not find or access ansible yaml + # {"ansible_facts": {}, "ansible_included_var_files": [], "changed": false, "message": + # "Could not find or access 'openEuler.yml' Searched in: + # /root/.ansible/roles/ansible-unzip/vars/openEuler.yml /root/.ansible/roles/ansible-unzip/openEuler.yml i + # /root/.ansible/roles/ansible-unzip/tasks/vars/openEuler.yml /root/.ansible/roles/ansible-unzip/tasks/openEuler.yml + # /root/.ansible/vars/openEuler.yml /root/.ansible/openEuler.yml on the Ansible Controller. + # If you are using a module and expect the file to exist on the remote, see the remote_src option"} + when /Could not find or access (.+) / + puts "yaml.error.not-find-or-access-yaml.#{$1}: 1" + puts "yaml.error.not-find-or-access-yaml.#{$1}.message: #{@ansible_failed_info}" + end +end + +def common_error_id(line) + line = line.chomp + line.gsub(/'/, ' ') + line.gsub(/{%/, ' ') + line.gsub(/%}/, ' ') + line.gsub(/{},/, '[],') + line.gsub(/\\n\\t/, ' ') + line +end + +while (line = STDIN.gets) + line = common_error_id(line) + next unless line =~ /(FAILED!|failed:).*({.*})/ + + @ansible_failed_info = $2 + next if @ansible_failed_info.empty? + + ansible_failed_json = JSON.load(@ansible_failed_info) + + parse_msg_1 ansible_failed_json['msg'] unless ansible_failed_json['msg'].nil? + parse_msg_2 ansible_failed_json['msg'] unless ansible_failed_json['msg'].nil? + parse_failures ansible_failed_json['failures'] unless ansible_failed_json['failures'].nil? + parse_message ansible_failed_json['message'] unless ansible_failed_json['message'].nil? +end -- 2.23.0

4 6

[PATCH lkp-tests] lib/job2sh.rb: delete secrets field when job2sh
by Cao Xueliang 23 Feb '21

23 Feb '21

We don't show the secrets field in job.sh file, so delete secrets field when job2sh. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/job2sh.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 7f9cf562f..82ec7d249 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -318,6 +318,7 @@ class Job2sh < Job unless job job = (@jobx || @job).clone # a shallow copy so that delete_if won't impact @job job.delete_if { |key, val| parse_one([], key, val, :PASS_EXPORT_ENV) } + job.delete_if { |key, val| key == "secrets" } end @cur_func = :run_job -- 2.23.0

1 0

[PATCH compass-ci] sched: add the secrets field processing logic
by Cao Xueliang 23 Feb '21

23 Feb '21

You can add the secrets field which you don't want to make public but the SUT need it in job.yaml. secrets: my_token:xxx The scheduler will save the secrets value to the redis id2secrets table, then delete the field from job.yaml. When consume the job, the scheduler will read the value from redis, then create a secrets.yaml in job.cgz. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sbin/create-job-cpio.sh | 1 + src/scheduler/create_job_cpio.cr | 13 +++++++++++++ src/scheduler/submit_job.cr | 7 +++++++ 3 files changed, 21 insertions(+) diff --git a/sbin/create-job-cpio.sh b/sbin/create-job-cpio.sh index e18bc0d..afb316c 100755 --- a/sbin/create-job-cpio.sh +++ b/sbin/create-job-cpio.sh @@ -9,6 +9,7 @@ cd "$1" || exit install -m775 -D -t lkp/scheduled job.sh install -m664 -D -t lkp/scheduled job.yaml +[ -f secrets.yaml ] && install -m664 -D -t lkp/scheduled secrets.yaml find lkp | cpio --quiet -o -H newc | gzip > job.cgz diff --git a/src/scheduler/create_job_cpio.cr b/src/scheduler/create_job_cpio.cr index 51d79f7..5634f67 100644 --- a/src/scheduler/create_job_cpio.cr +++ b/src/scheduler/create_job_cpio.cr @@ -74,8 +74,21 @@ class Sched script_lines = JSON.parse(script_lines) end + def create_secrets_yaml(job_id, base_dir) + secrets = @redis.hash_get("id2secrets", job_id) + return nil unless secrets + + secrets_yaml = base_dir + "/#{job_id}/secrets.yaml" + prepare_dir(secrets_yaml) + + File.open(secrets_yaml, "w") do |file| + YAML.dump(JSON.parse(secrets), file) + end + end + def create_job_cpio(job_content : JSON::Any, base_dir : String) job_content = job_content.as_h + create_secrets_yaml(job_content["id"], base_dir) # put job2sh in an array if job_content.has_key?("job2sh") diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 1837cb1..fe9efa1 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -144,8 +144,15 @@ class Sched JSON.parse(response[1].to_json)["id"].to_s if response[0] == 200 end + def save_secrets(job, job_id) + return nil unless job["secrets"]? + + @redis.hash_set("id2secrets", job_id, job["secrets"]?.to_json) + job.delete("secrets") + end # add job content to es and return a response def add_job(job, job_id) + save_secrets(job, job_id) job.update_id(job_id) @es.set_job_content(job) end -- 2.23.0

1 0

[PATCH lab-z9] add muti-qemu hosts and alter the multi-qemu.yaml
by Bai Jing 23 Feb '21

23 Feb '21

Signed-off-by: Bai Jing <799286817(a)qq.com> --- allot/idle/taishan200-2280-2s48p-256g--a63 | 1 + allot/idle/taishan200-2280-2s48p-256g--a64 | 1 + allot/idle/taishan200-2280-2s48p-256g--a65 | 1 + allot/idle/taishan200-2280-2s48p-256g--a66 | 1 + allot/idle/taishan200-2280-2s48p-256g--a67 | 1 + allot/idle/taishan200-2280-2s48p-256g--a68 | 1 + allot/idle/taishan200-2280-2s48p-256g--a69 | 1 + allot/idle/taishan200-2280-2s48p-256g--a70 | 1 + .../multi-qemu.yaml | 15 ++++++++++++--- 9 files changed, 20 insertions(+), 3 deletions(-) create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a63 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a64 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a65 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a66 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a67 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a68 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a69 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a70 diff --git a/allot/idle/taishan200-2280-2s48p-256g--a63 b/allot/idle/taishan200-2280-2s48p-256g--a63 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a63 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a64 b/allot/idle/taishan200-2280-2s48p-256g--a64 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a64 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a65 b/allot/idle/taishan200-2280-2s48p-256g--a65 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a65 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a66 b/allot/idle/taishan200-2280-2s48p-256g--a66 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a66 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a67 b/allot/idle/taishan200-2280-2s48p-256g--a67 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a67 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a68 b/allot/idle/taishan200-2280-2s48p-256g--a68 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a68 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a69 b/allot/idle/taishan200-2280-2s48p-256g--a69 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a69 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a70 b/allot/idle/taishan200-2280-2s48p-256g--a70 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a70 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml b/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml index ed55c08..87f684e 100644 --- a/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml +++ b/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml @@ -12,14 +12,23 @@ fs: simplify-ci: multi-qemu-0: - nr_vm: 20 - tbox_group: vm-2p8g + nr_vm: 2 + tbox_group: vm-1p1g multi-qemu-1: nr_vm: 2 tbox_group: vm-2p4g multi-qemu-2: + nr_vm: 12 + tbox_group: vm-2p8g +multi-qemu-3: + nr_vm: 6 + tbox_group: vm-2p16g +multi-qemu-4: nr_vm: 2 - tbox_group: vm-1p1g + tbox_group: vm-2p32g +multi-qemu-5: + nr_vm: 2 + tbox_group: vm-4p64g sleep: 100d os: openeuler -- 2.23.0

1 0

[PATCH lkp-tests] setup/simplify-ci: git failed to exit directly
by Cui Yili 23 Feb '21

23 Feb '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- setup/simplify-ci | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index c8f8a5a53..5aede9cdb 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -14,7 +14,7 @@ git_ci() EOF fi - git clone https://gitee.com/wu_fengguang/compass-ci.git /c/compass-ci + git clone https://gitee.com/wu_fengguang/compass-ci.git /c/compass-ci || return 1 } dev_env() @@ -68,7 +68,7 @@ run_service() main() { - git_ci + git_ci || return 1 install_env run_service } -- 2.23.0

1 0

[PATCH compass-ci] sbin/auto_submit: Adapting to new config style of DEFAULTS files
by Li Yuanchao 23 Feb '21

23 Feb '21

As the config style of DEFAULTS files in upstream-repos changes, the auto_submit need change to adapt to it. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 53 ++++++++++++++++-------------------------------- 1 file changed, 18 insertions(+), 35 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index b408158..857af29 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -57,28 +57,13 @@ class AutoSubmit system(real_argvs.join(' ')) end - def get_branch_commands(submit_commands) - submit_commands.each do |element| - return element['branch_commands'] if element['branch_commands'] - end - return nil - end + def submit_specific_branch(submit_argv, newrefs_info, branches) + branches.each do |branch_name| + branch = "refs/heads/#{branch_name}" + next unless newrefs_info['new_refs']['heads'][branch] - def submit_specific_branch(submit_argv, newrefs_info, branch_commands) - return newrefs_info['new_refs']['heads'] if branch_commands.nil? - - non_specific_commits = {} - newrefs_info['new_refs']['heads'].each do |branch, commit_id| - branch_name = branch.delete_prefix('refs/heads/') - if branch_commands[branch_name].nil? - non_specific_commits[branch] = commit_id - next - end - argvs = Array.new(submit_argv) - argvs.push(branch_commands[branch_name]) - submit_one_job(argvs, newrefs_info['git_repo'], branch_name, commit_id) + submit_one_job(submit_argv, newrefs_info['git_repo'], branch_name, newrefs_info['new_refs']['heads'][branch]) end - return non_specific_commits end def submit_non_specific(non_specific_commits, git_repo, submit_argv) @@ -90,13 +75,13 @@ class AutoSubmit # Add support of different branch different job # newrefs_info['submit'] is like: - # ["submit command 1", - # "submit command 2", + # [ + # { "command" => "submit command 1", + # "branches" => [ "master", "next", ... ] + # }, # ..., - # { "branch_commands" => { "branch_1" => "submit command 1", - # ..., - # "branch_n" => "submit command n" - # } + # { "command" => "submit command n", + # "branches" => [ "branch_name_a", "branch_name_b", ... ] # } # ] def submit_job(newrefs_info) @@ -105,16 +90,14 @@ class AutoSubmit return unless newrefs_info['submit'] - branch_commands = get_branch_commands(newrefs_info['submit']) - non_specific_commits = submit_specific_branch(submit_argv, newrefs_info, branch_commands) - return if non_specific_commits.empty? - - newrefs_info['submit'].each do |argv_config| - next if argv_config.is_a? Hash - + newrefs_info['submit'].each do |element| argvs = Array.new(submit_argv) - argvs.push(argv_config) - submit_non_specific(non_specific_commits, newrefs_info['git_repo'], argvs) + argvs.push(element['command']) + if element['branches'] + submit_specific_branch(argvs, newrefs_info, element['branches']) + else + submit_non_specific(newrefs_info['new_refs']['heads'], newrefs_info['git_repo'], argvs) + end end end -- 2.23.0

1 0

[PATCH lkp-tests] tests/rpmbuild-pkg: redirect the sync_dest directory
by Li Ping 23 Feb '21

23 Feb '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- tests/rpmbuild-pkg | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 9f455f0d0..38de16e66 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -6,6 +6,7 @@ [ -n "$rpm_repo" ] || die "rpm_repo is empty" package_name=${rpm_repo##*/} +[[ "$os_mount" = "cifs" ]] && os_mount="nfs" sync_dest="/initrd/rpmbuild-pkg/${os_mount}/${os}/${os_arch}/${os_version}/${package_name}" init_workspace() -- 2.23.0

1 0

[PATCH v2 lkp-tests] jobs/: add os-arch-version-mount.yaml for multi-dimension test
by Li Ping 23 Feb '21

23 Feb '21

1 1

[PATCH compass-ci] container/scheduler/start: fix read config
by Wei Jihui 23 Feb '21

23 Feb '21

scheduler service need read config from /etc/compass-ci/[service|scheduler], not only /etc/compass-ci/default. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- container/scheduler/start | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/container/scheduler/start b/container/scheduler/start index c32a7a8..4aaa71d 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -46,7 +46,6 @@ DEFAULT_CCI = "#{CCI_REPOS}/compass-ci" LAB = defaults['lab'] || 'z9' SCHED_PORT = defaults['SCHED_PORT'] SCHED_HOST = defaults['SCHED_HOST'] -DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' docker_rm "s001-alpine-#{SCHED_PORT}" @@ -68,9 +67,9 @@ cmd = %W[ -e CCI_SRC=#{DEFAULT_CCI} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} - -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} -v /srv/cci/scheduler/alpine:/srv/cci/scheduler + -v /etc/compass-ci:/etc/compass-ci:ro -v /etc/localtime:/etc/localtime:ro -v /srv/result:/srv/result -v /srv/initrd:/srv/initrd:ro -- 2.23.0

1 0

[PATCH compass-ci] container/assign-account: optimize account login permission
by Luan Shengde 23 Feb '21

23 Feb '21

[why] the mail robot sets no keyword 'enable_login' when assigning account. the applied account via sending 'apply account' email was set no permission to login. before: set enable login: has key: enable_login key value: true set disable login has key: enable_login key value: false or has no key: enable_login now: set enable login: has key: enable_login key value: true set disable login: has key: enable_login key value: false keep the default: has no key: enable_login Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 7e0a5a7..228d468 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -136,7 +136,7 @@ class AccountStorage def permit_login_config(login_name) if @data.key?('enable_login') && @data['enable_login'] %x(usermod -s /usr/bin/zsh #{login_name}) - else + elsif @data.key?('enable_login') && ! @data['enable_login'] %x(usermod -s /sbin/nologin #{login_name}) end end -- 2.23.0

1 0

[PATCH lab-z9] idle/*/multi-qemu.yaml: raid the added disk
by Cui Yili 23 Feb '21

23 Feb '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- .../multi-qemu.yaml | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml b/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml index 8182a97..ed55c08 100644 --- a/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml +++ b/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml @@ -1,8 +1,14 @@ suite: multi-qemu category: functional -disk: 1HDD -fs: ext4 +nr_hdd: {{nr_hdd_partitions}} +disk: + +md: + - RAID0 + +fs: + - ext4 simplify-ci: multi-qemu-0: -- 2.23.0

2 1

[PATCH compass-ci] sched: delete job.cgz from result root dir
by Cao Xueliang 22 Feb '21

22 Feb '21

We has the job.yaml and job.sh in result root dir, no need to save the job.cgz. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/create_job_cpio.cr | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/scheduler/create_job_cpio.cr b/src/scheduler/create_job_cpio.cr index 1101a0f..51d79f7 100644 --- a/src/scheduler/create_job_cpio.cr +++ b/src/scheduler/create_job_cpio.cr @@ -134,8 +134,7 @@ class Sched # the job.yaml is not final version files = ["#{src_dir}/job.sh", - "#{src_dir}/job.yaml", - "#{src_dir}/job.cgz"] + "#{src_dir}/job.yaml"] FileUtils.cp(files, dst_dir) end end -- 2.23.0

1 0

[PATCH lkp-tests 2/2] spec/submit/link_jobs: replace latency_stats to perf-sched
by Wei Jihui 22 Feb '21

22 Feb '21

commit 4b1eaa97a0d7dd2455929c0734ed88631b1aa263: include/category/benchmark: replace latency_stats to perf-ddsched Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- ...4-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml index 5457a131..0c3eb69d 100644 --- a/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml +++ b/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml @@ -51,7 +51,7 @@ meminfo: slabinfo: interrupts: lock_stat: -latency_stats: +perf-sched: softirqs: bdi_dev_mapping: diskstats: @@ -103,7 +103,7 @@ monitors: slabinfo: interrupts: lock_stat: - latency_stats: + perf-sched: softirqs: bdi_dev_mapping: diskstats: -- 2.23.0

1 0

[PATCH lkp-tests 1/2] spec/submit: update submit spec result
by Wei Jihui 22 Feb '21

22 Feb '21

all vm testboxs add hdd_partitions Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit/job_on_fail/job-3000.yaml | 1 + ...1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml | 1 + spec/submit/link_jobs2/job-1000-2-100000ss-add.yaml | 1 + spec/submit/link_jobs2/job-10000-2-100000ss-add.yaml | 1 + spec/submit/link_jobs2/job-20-2-6000ss-cache.yaml | 1 + spec/submit/matrix/job-1-300-whetstone-double.yaml | 1 + spec/submit/matrix/job-3-300-shell1.yaml | 1 + spec/submit/merge_yaml/job-defaults.yaml | 1 + spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml | 1 + spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml | 1 + spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml | 1 + 11 files changed, 11 insertions(+) diff --git a/spec/submit/job_on_fail/job-3000.yaml b/spec/submit/job_on_fail/job-3000.yaml index 11d89b2b..5d766ba0 100644 --- a/spec/submit/job_on_fail/job-3000.yaml +++ b/spec/submit/job_on_fail/job-3000.yaml @@ -11,6 +11,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml index 466a2054..5457a131 100644 --- a/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml +++ b/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml @@ -34,6 +34,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 kmsg: boot-time: diff --git a/spec/submit/link_jobs2/job-1000-2-100000ss-add.yaml b/spec/submit/link_jobs2/job-1000-2-100000ss-add.yaml index 9a613c00..d20eed7e 100644 --- a/spec/submit/link_jobs2/job-1000-2-100000ss-add.yaml +++ b/spec/submit/link_jobs2/job-1000-2-100000ss-add.yaml @@ -15,6 +15,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/link_jobs2/job-10000-2-100000ss-add.yaml b/spec/submit/link_jobs2/job-10000-2-100000ss-add.yaml index 8bf52d0f..8df242aa 100644 --- a/spec/submit/link_jobs2/job-10000-2-100000ss-add.yaml +++ b/spec/submit/link_jobs2/job-10000-2-100000ss-add.yaml @@ -15,6 +15,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/link_jobs2/job-20-2-6000ss-cache.yaml b/spec/submit/link_jobs2/job-20-2-6000ss-cache.yaml index 8cb78287..cf3a162b 100644 --- a/spec/submit/link_jobs2/job-20-2-6000ss-cache.yaml +++ b/spec/submit/link_jobs2/job-20-2-6000ss-cache.yaml @@ -15,6 +15,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/matrix/job-1-300-whetstone-double.yaml b/spec/submit/matrix/job-1-300-whetstone-double.yaml index 78f79fa7..4ba40ba5 100644 --- a/spec/submit/matrix/job-1-300-whetstone-double.yaml +++ b/spec/submit/matrix/job-1-300-whetstone-double.yaml @@ -12,6 +12,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/matrix/job-3-300-shell1.yaml b/spec/submit/matrix/job-3-300-shell1.yaml index afe5a953..294cbdd3 100644 --- a/spec/submit/matrix/job-3-300-shell1.yaml +++ b/spec/submit/matrix/job-3-300-shell1.yaml @@ -12,6 +12,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/merge_yaml/job-defaults.yaml b/spec/submit/merge_yaml/job-defaults.yaml index f72fd6ac..ae2f3c39 100644 --- a/spec/submit/merge_yaml/job-defaults.yaml +++ b/spec/submit/merge_yaml/job-defaults.yaml @@ -9,6 +9,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml index a2355b50..b7b0ef0e 100644 --- a/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml +++ b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml @@ -13,6 +13,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml index 8a6317cb..144fba5a 100644 --- a/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml +++ b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml @@ -13,6 +13,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml index 15701ad0..74c4e380 100644 --- a/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml +++ b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml @@ -13,6 +13,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: -- 2.23.0

1 0