- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] scheduler/find_job_boot.cr: docker add job_id
by Wu Zhende 12 Mar '21

12 Mar '21

When the startup parameter is docker, the value of job_id is returned. Set job_id as docker name, so we can use "docker rm -f #{job_id}" to kill the docker. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/find_job_boot.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ce1a2f8..a45a6d9 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -144,6 +144,7 @@ class Sched private def get_boot_container(job : Job) response = Hash(String, String).new + response["job_id"] = job.id.to_s response["docker_image"] = "#{job.docker_image}" response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code/dev-env: fix failed to submit job
by Liu Yinsi 12 Mar '21

12 Mar '21

[error] root@taishan200-2280-2s64p-256g--a123 /c/compass-ci/container# submit host-info.yaml submit_id=9bca6036-4602-44ab-92df-729058ac1818 Traceback (most recent call last): 5: from /c/lkp-tests/sbin/submit:136:in `<main>' 4: from /c/lkp-tests/sbin/submit:136:in `each' 3: from /c/lkp-tests/sbin/submit:150:in `block in <main>' 2: from /c/lkp-tests/lib/job.rb:744:in `each_jobs' 1: from /c/lkp-tests/lib/job.rb:736:in `each_job' /c/lkp-tests/sbin/submit:151:in `block (2 levels) in <main>': Please configure SCHED_HOST (RuntimeError) [why] for user locally deploy compass-ci, env $SCHED_HOST not exists in /etc/compass-ci/defaults/$server_name.yaml, cause failed to submit, so add it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 403ccb1..fdf7894 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -36,6 +36,8 @@ server_name=$(hostname | cut -f1 -d.) mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF +SCHED_HOST: $sched_host +SCHED_PORT: $sched_port lab: nolab EOF -- 2.23.0

1 0

[PATCH lkp-tests] tests/rpmbuild-pkg: change rpmbuild-pkg upload_dir
by Li Ping 12 Mar '21

12 Mar '21

--- tests/rpmbuild-pkg | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index b641e97d1..a8c8eb6b1 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -6,7 +6,7 @@ [ -n "$rpm_repo" ] || die "rpm_repo is empty" package_name=${rpm_repo##*/} -sync_dest="/initrd/rpmbuild-pkg/${os}/${os_arch}/${os_version}/${package_name}" +sync_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${os_arch}/${package_name}" init_workspace() { @@ -24,7 +24,7 @@ download_rpm_repo() { local filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') - for pkgfile in "${filelist[@]}" + for pkgfile in ${filelist[@]} do local dir="SOURCES" echo $pkgfile | egrep "\.spec$" && dir="SPECS" -- 2.23.0

1 0

[PATCH v2 compass-ci 3/3] doc/job/os_mount.md: change the fields of job.yaml
by Xu Xijian 11 Mar '21

11 Mar '21

The fields that users need to give is the suffix of partitions, so give them appropriate names. use_root_partition => src_lv_suffix save_root_partition => boot_lv_suffix Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/os_mount.md | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index 6c1637b..eab95ed 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -56,23 +56,25 @@ The brief flow is as follows: ## persistent rootfs data -When you need to persist the rootfs data of a job, and use it in the subsequent job(s), two fields in `kernel_custom_params` will help you: `save_root_partition`, `use_root_partition`. +When you need to persist the rootfs data of a job, and use it in the subsequent job(s), two fields in `kernel_custom_params` will help you: 'src_lv_suffix', 'boot_lv_suffix'. The brief flow is as follows: ``` 1. boot and request scheduler for job. 2. initrd stage: + use_root_partition seems like /dev/mapper/os-${os}_${os_arch}_${os_version}_${src_lv_suffix} + save_root_partition seems like /dev/mapper/os-${os}_${os_arch}_${os_version}_${boot_lv_suffix} - firstly, we need two logical volume: ${src_lv} -- src logical volume: - - if have ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${use_root_partition} + - if have ${use_root_partition}, src_lv=${use_root_partition} - if no ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} ${boot_lv} -- boot logical volume: - - if have ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${save_root_partition} + - if have ${save_root_partition}, boot_lv=${save_root_partition} - if no ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} - if ${src_lv} not exists: - - if have ${use_root_partition}, exit 1. - - if no ${use_root_partition}, create ${src_lv}, and rsync the rootfs from cluster nfs server. + - if have ${use_root_partition}, reboot. + - if no ${use_root_partition}, create ${src_lv}, and sync the rootfs from cluster nfs server. - if ${boot_lv} != ${src_lv}: - if ${boot_lv} exists: delete it - create ${boot_lv} as the snapshot of ${src_lv}. @@ -87,12 +89,12 @@ Demo usage: rootfs data of job-20210218.yaml so that it can be used by the subsequent jobs. Then you need add the follow field in your job-20210218.yaml: - kernel_custom_params: save_root_partition=zhangsan_local_for_iperf_20210218 + kernel_custom_params: boot_lv_suffix=zhangsan_local_for_iperf_20210218 - in 20210219, you submit a job-20210219.yaml, and you want to use the rootfs data of job-20210218.yaml. Then you need add the follow field in your job-20210219.yaml: - kernel_custom_params: use_root_partition=zhangsan_local_for_iperf_20210218 + kernel_custom_params: src_lv_suffix=zhangsan_local_for_iperf_20210218 ``` Notes: @@ -109,16 +111,19 @@ Notes: os_arch: aarch64 os_version: 20.03 os_mount: local - kernel_custom_params: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 + kernel_custom_params: src_lv_suffix=zhangsan_local_for_iperf_20210218 boot_lv_suffix=zhangsan_local_for_iperf_20210219 ``` 2. scheduler return the custom_ipxe to testbox + os_info = "#{os}_#{os_arch}_#{os_version}" + use_root_partition = "/dev/mapper/os-#{os_info}_#{src_lv_suffix}" + save_root_partition = "/dev/mapper/os-#{os_info}_#{boot_lv_suffix}" ``` #!ipxe dhcp initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/initrd.lkp - kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src=${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} initrd=initrd.lkp ${kernel_custom_params} + kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 initrd=initrd.lkp local use_root_partition=#{use_root_partition} save_root_partition=#{save_root_partition} rw ... boot ``` -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] container/dracut-initrd: change someof the handle logic according to scheme doc
by Xu Xijian 11 Mar '21

11 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- .../dracut-initrd/bin/set-local-sysroot.sh | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh index dc9000b..240551a 100644 --- a/container/dracut-initrd/bin/set-local-sysroot.sh +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -5,9 +5,9 @@ analyse_kernel_cmdline_params() { # example: $nfs_server_ip:/os/${os}/${os_arch}/${os_version}-snapshots/20210310005959 rootfs_src=$(echo $"$rootfs" | sed 's/\///') timestamp="$(basename $root)" - os="$(echo $root_src | awk -F '/|-' '{print $2}')" - os_arch="$(echo $root_src | awk -F '/|-' '{print $3}')" - os_version="$(echo $root_src | awk -F '/|-' '{print $4}')" + os="$(echo $rootfs_src | awk -F '/|-' '{print $2}')" + os_arch="$(echo $rootfs_src | awk -F '/|-' '{print $3}')" + os_version="$(echo $rootfs_src | awk -F '/|-' '{print $4}')" os_info="${os}_${os_arch}_${os_version}" export rootfs_src timestamp os_info } @@ -58,22 +58,22 @@ analyse_kernel_cmdline_params sed -i "s/^locking_type = .*/locking_type = 1/" /etc/lvm/lvm.conf use_root_partition="$(getarg use_root_partition=)" -if [ -n "$use_root_partition" ]; then - src_lv="$use_root_partition" # /dev/mapper/os-${os_info}_${timestamp} +if [ -z "$use_root_partition" ]; then + src_lv="/dev/mapper/os-${os_info}_$timestamp" sync_src_lv "$src_lv" else - src_lv="/dev/mapper/os-${os_info}_$timestamp" - [ -e "src_lv" ] || { + src_lv="$use_root_partition" + [ -e "$src_lv" ] || { echo "warn dracut: FATAL: no src_lv with local mount, reboot" reboot } fi save_root_partition="$(getarg save_root_partition=)" -if [ -n "$save_root_partition" ]; then - boot_lv="$save_root_partition" -else +if [ -z "$save_root_partition" ]; then boot_lv="/dev/mapper/os-${os_info}" +else + boot_lv="$save_root_partition" fi snapshot_boot_lv "$src_lv" "$boot_lv" -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] container/dracut-initrd: add basename into openeuler's 21xx+ kernel
by Xu Xijian 11 Mar '21

11 Mar '21

[why] It seems openeuler 21xx+ kernel not support cmd basename, our overlay need this cmd, so add basename into dracut module to enable initrd. [errlog] //lib/dracut/hooks/pre-pivot/10-overlay-mount.sh: line 50: basename: command not found //lib/dracut/hooks/pre-pivot/10-overlay-mount.sh: line 65: basename: command not found Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 45e895c..494588a 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -17,7 +17,7 @@ cp -a /usr/local/bin/cifs-lib.sh /usr/lib/dracut/modules.d/95cifs/ cat overlay-lkp.sh >> /usr/lib/dracut/modules.d/90overlay-root/overlay-mount.sh sed -i "/install() {/a\ inst /usr/bin/awk" /usr/lib/dracut/modules.d/40network/module-setup.sh -sed -i "/install() {/a\ inst /sbin/mke2fs" /usr/lib/dracut/modules.d/98dracut-systemd/module-setup.sh +sed -i "/install() {/a\ inst /sbin/mke2fs /usr/bin/basename" /usr/lib/dracut/modules.d/98dracut-systemd/module-setup.sh pre_mount_file="/usr/lib/dracut/modules.d/98dracut-systemd/dracut-pre-mount.sh" [ "$(sed -n '$p' $pre_mount_file)" = "exit 0" ] && sed -i '$d' "$pre_mount_file" cat set-local-sysroot.sh >> "$pre_mount_file" -- 2.23.0

1 0

[PATCH lkp-tests] add multi-qemu-x86.yaml
by Zhang Dewan 11 Mar '21

11 Mar '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/multi-qemu-x86.yaml | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 jobs/multi-qemu-x86.yaml diff --git a/jobs/multi-qemu-x86.yaml b/jobs/multi-qemu-x86.yaml new file mode 100644 index 00000000..5539c1ad --- /dev/null +++ b/jobs/multi-qemu-x86.yaml @@ -0,0 +1,23 @@ +suite: multi-qemu +category: functional +simplify-ci: +runtime: 100d +multi-qemu-0: + nr_vm: 5 + tbox_group: vm-2p1g-x86 +multi-qemu-1: + nr_vm: 5 + tbox_group: vm-2p2g-x86 +multi-qemu-2: + nr_vm: 5 + tbox_group: vm-2p4g-x86 +multi-qemu-3: + nr_vm: 10 + tbox_group: vm-2p8g-x86 +multi-qemu-4: + nr_vm: 5 + tbox_group: vm-2p16g-x86 +multi-qemu-5: + nr_vm: 1 + tbox_group: vm-2p32g-x86 +sleep: -- 2.23.0

1 0

[PATCH v2 lab-z9 2/2] add x86 vm host file
by Zhang Dewan 11 Mar '21

11 Mar '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/vm-1p1g-x86 | 3 ++- hosts/vm-2p16g-x86 | 3 ++- hosts/vm-2p1g-x86 | 3 ++- hosts/vm-2p32g-x86 | 3 ++- hosts/vm-2p4g-x86 | 3 ++- hosts/vm-2p8g-x86 | 3 ++- 6 files changed, 12 insertions(+), 6 deletions(-) diff --git a/hosts/vm-1p1g-x86 b/hosts/vm-1p1g-x86 index 5c0e081..b278e80 100644 --- a/hosts/vm-1p1g-x86 +++ b/hosts/vm-1p1g-x86 @@ -1,5 +1,6 @@ +arch: aarch64 provider: qemu -template: pxe +template: kvm nr_node: 1 nr_cpu: 1 memory: 1G diff --git a/hosts/vm-2p16g-x86 b/hosts/vm-2p16g-x86 index 8abda03..cd49afc 100644 --- a/hosts/vm-2p16g-x86 +++ b/hosts/vm-2p16g-x86 @@ -1,5 +1,6 @@ +arch: aarch64 provider: qemu -template: pxe +template: kvm nr_node: 1 nr_cpu: 2 memory: 16G diff --git a/hosts/vm-2p1g-x86 b/hosts/vm-2p1g-x86 index 60658d3..4fffd2a 100644 --- a/hosts/vm-2p1g-x86 +++ b/hosts/vm-2p1g-x86 @@ -1,5 +1,6 @@ +arch: aarch64 provider: qemu -template: pxe +template: kvm nr_node: 1 nr_cpu: 2 memory: 1G diff --git a/hosts/vm-2p32g-x86 b/hosts/vm-2p32g-x86 index 2d7bb13..c7cc583 100644 --- a/hosts/vm-2p32g-x86 +++ b/hosts/vm-2p32g-x86 @@ -1,5 +1,6 @@ +arch: aarch64 provider: qemu -template: pxe +template: kvm nr_node: 1 nr_cpu: 2 memory: 32G diff --git a/hosts/vm-2p4g-x86 b/hosts/vm-2p4g-x86 index 638f803..41d0a3b 100644 --- a/hosts/vm-2p4g-x86 +++ b/hosts/vm-2p4g-x86 @@ -1,5 +1,6 @@ +arch: aarch64 provider: qemu -template: pxe +template: kvm nr_node: 1 nr_cpu: 2 memory: 4G diff --git a/hosts/vm-2p8g-x86 b/hosts/vm-2p8g-x86 index caa6a88..8bc5cad 100644 --- a/hosts/vm-2p8g-x86 +++ b/hosts/vm-2p8g-x86 @@ -1,5 +1,6 @@ +arch: aarch64 provider: qemu -template: pxe +template: kvm nr_node: 1 nr_cpu: 2 memory: 8G -- 2.23.0

1 0

[PATCH v2 lab-z9 1/2] add x86 vm host file
by Zhang Dewan 11 Mar '21

11 Mar '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/vm-1p1g-x86 | 6 ++++++ hosts/vm-2p16g-x86 | 6 ++++++ hosts/vm-2p1g-x86 | 6 ++++++ hosts/vm-2p32g-x86 | 6 ++++++ hosts/vm-2p4g-x86 | 6 ++++++ hosts/vm-2p8g-x86 | 6 ++++++ 6 files changed, 36 insertions(+) create mode 100644 hosts/vm-1p1g-x86 create mode 100644 hosts/vm-2p16g-x86 create mode 100644 hosts/vm-2p1g-x86 create mode 100644 hosts/vm-2p32g-x86 create mode 100644 hosts/vm-2p4g-x86 create mode 100644 hosts/vm-2p8g-x86 diff --git a/hosts/vm-1p1g-x86 b/hosts/vm-1p1g-x86 new file mode 100644 index 0000000..5c0e081 --- /dev/null +++ b/hosts/vm-1p1g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 1 +memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p16g-x86 b/hosts/vm-2p16g-x86 new file mode 100644 index 0000000..8abda03 --- /dev/null +++ b/hosts/vm-2p16g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 16G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p1g-x86 b/hosts/vm-2p1g-x86 new file mode 100644 index 0000000..60658d3 --- /dev/null +++ b/hosts/vm-2p1g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p32g-x86 b/hosts/vm-2p32g-x86 new file mode 100644 index 0000000..2d7bb13 --- /dev/null +++ b/hosts/vm-2p32g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 32G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p4g-x86 b/hosts/vm-2p4g-x86 new file mode 100644 index 0000000..638f803 --- /dev/null +++ b/hosts/vm-2p4g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 4G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p8g-x86 b/hosts/vm-2p8g-x86 new file mode 100644 index 0000000..caa6a88 --- /dev/null +++ b/hosts/vm-2p8g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 8G +hdd_partitions: /dev/vda -- 2.23.0

1 0

[PATCH v1 lab-z9] add x86 vm host file
by Zhang Dewan 11 Mar '21

11 Mar '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/vm-1p1g-x86 | 6 ++++++ hosts/vm-2p16g-x86 | 6 ++++++ hosts/vm-2p1g-x86 | 6 ++++++ hosts/vm-2p32g-x86 | 6 ++++++ hosts/vm-2p4g-x86 | 6 ++++++ hosts/vm-2p8g-x86 | 6 ++++++ 6 files changed, 36 insertions(+) create mode 100644 hosts/vm-1p1g-x86 create mode 100644 hosts/vm-2p16g-x86 create mode 100644 hosts/vm-2p1g-x86 create mode 100644 hosts/vm-2p32g-x86 create mode 100644 hosts/vm-2p4g-x86 create mode 100644 hosts/vm-2p8g-x86 diff --git a/hosts/vm-1p1g-x86 b/hosts/vm-1p1g-x86 new file mode 100644 index 0000000..5c0e081 --- /dev/null +++ b/hosts/vm-1p1g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 1 +memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p16g-x86 b/hosts/vm-2p16g-x86 new file mode 100644 index 0000000..8abda03 --- /dev/null +++ b/hosts/vm-2p16g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 16G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p1g-x86 b/hosts/vm-2p1g-x86 new file mode 100644 index 0000000..60658d3 --- /dev/null +++ b/hosts/vm-2p1g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p32g-x86 b/hosts/vm-2p32g-x86 new file mode 100644 index 0000000..2d7bb13 --- /dev/null +++ b/hosts/vm-2p32g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 32G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p4g-x86 b/hosts/vm-2p4g-x86 new file mode 100644 index 0000000..638f803 --- /dev/null +++ b/hosts/vm-2p4g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 4G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p8g-x86 b/hosts/vm-2p8g-x86 new file mode 100644 index 0000000..caa6a88 --- /dev/null +++ b/hosts/vm-2p8g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 8G +hdd_partitions: /dev/vda -- 2.23.0

1 0

[PATCH lkp-tests 2/2] add x86 vm host file
by Zhang Dewan 11 Mar '21

11 Mar '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/vm-1p1g-x86 | 6 ++++++ hosts/vm-2p16g-x86 | 6 ++++++ hosts/vm-2p1g-x86 | 6 ++++++ hosts/vm-2p32g-x86 | 6 ++++++ hosts/vm-2p4g-x86 | 6 ++++++ hosts/vm-2p8g-x86 | 6 ++++++ 6 files changed, 36 insertions(+) create mode 100644 hosts/vm-1p1g-x86 create mode 100644 hosts/vm-2p16g-x86 create mode 100644 hosts/vm-2p1g-x86 create mode 100644 hosts/vm-2p32g-x86 create mode 100644 hosts/vm-2p4g-x86 create mode 100644 hosts/vm-2p8g-x86 diff --git a/hosts/vm-1p1g-x86 b/hosts/vm-1p1g-x86 new file mode 100644 index 00000000..5c0e081c --- /dev/null +++ b/hosts/vm-1p1g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 1 +memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p16g-x86 b/hosts/vm-2p16g-x86 new file mode 100644 index 00000000..8abda03f --- /dev/null +++ b/hosts/vm-2p16g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 16G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p1g-x86 b/hosts/vm-2p1g-x86 new file mode 100644 index 00000000..60658d38 --- /dev/null +++ b/hosts/vm-2p1g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p32g-x86 b/hosts/vm-2p32g-x86 new file mode 100644 index 00000000..2d7bb132 --- /dev/null +++ b/hosts/vm-2p32g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 32G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p4g-x86 b/hosts/vm-2p4g-x86 new file mode 100644 index 00000000..638f803f --- /dev/null +++ b/hosts/vm-2p4g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 4G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p8g-x86 b/hosts/vm-2p8g-x86 new file mode 100644 index 00000000..caa6a88f --- /dev/null +++ b/hosts/vm-2p8g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 8G +hdd_partitions: /dev/vda -- 2.23.0

1 0

[PATCH lkp-tests 1/2] add a host group in lkp-tests/hosts
by Zhang Dewan 11 Mar '21

11 Mar '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s48p-256g | 1 + 1 file changed, 1 insertion(+) create mode 100644 hosts/taishan200-2280-2s48p-256g diff --git a/hosts/taishan200-2280-2s48p-256g b/hosts/taishan200-2280-2s48p-256g new file mode 100644 index 00000000..6399b995 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g @@ -0,0 +1 @@ +arch: aarch64 -- 2.23.0

1 0

help star
by Wu Fengguang 11 Mar '21

11 Mar '21

Xueliang is our help star, thanks Xueliang! ~/todo/bin% ./help-star 19.6 Wu Fengguang 18.6 Cao Xueliang 15.9 Wei Jihui 15.2 Yu Chuan 10.8 Wu Zhende 8.8 Luan Shengde 7.8 Liu Yinsi 6.8 Wang Chenglong 6.7 Lin Jiaxin 5.6 Bai Jing 5.4 Wang Yong 5.3 Cui Yili 5.0 Xu Xijian 4.8 Li Yuanchao 4.6 Liu Shaofei 3.3 Lu Weitao 3.1 Zhang Dewan 1.8 Du Kaitian 0.9 Li Ping 0.0 Zhang Yale

1 0

review star
by Wu Fengguang 11 Mar '21

11 Mar '21

Jihui is our review star again, thanks Jihui! ~/todo/bin% ./review-star Review email records in the last 7 days total review paragraph point people 30 17 26 24 Xu Xijian 80 18 27 22 Wei Jihui 87 5 14 19 Yu Chuan 51 4 5 9 Cao Xueliang 32 13 32 7 Luan Shengde 16 3 3 3 Wu Zhende 5 2 1 2 Cui Yili 15 5 7 2 Wu Fengguang 5 2 2 1 Lin Jiaxin 9 0 0 0 Zhang Dewan 11 3 8 0 Liu Shaofei 3 0 0 0 Wang Yong 21 0 0 0 Liu Yinsi 13 0 0 0 Li Yuanchao 13 0 0 0 Bai Jing 26 1 2 0 Lu Weitao 12 1 1 0 Wang Chenglong 28 0 0 0 Li Ping

1 0

git count
by Wu Fengguang 11 Mar '21

11 Mar '21

Yu Chuan is our code star, thanks Yu Chuan! ~/todo/bin% ./git-count 7 18 Yu Chuan 10 Xu Xijian 10 Wei Jihui 8 Wu Zhende 7 Liu Yinsi 5 Li Yuanchao 5 Luan Shengde 4 Bai Jing 4 Lu Weitao 4 Li Ping 3 Cao Xueliang 2 Liu Shaofei 2 Wang Yong 2 Lin Jiaxin 1 Zhang Dewan 0 Du Kaitian 0 Wu Fengguang 0 Zhang Yale 0 Cui Yili 0 Wang Chenglong

1 0

[PATCH lkp-tests] tests/ansible_test: ansible test case
by Wang Chenglong 11 Mar '21

11 Mar '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- tests/ansible_test | 84 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 84 insertions(+) create mode 100755 tests/ansible_test diff --git a/tests/ansible_test b/tests/ansible_test new file mode 100755 index 000000000..36164bcae --- /dev/null +++ b/tests/ansible_test @@ -0,0 +1,84 @@ +#!/bin/bash +# - ansible_repo + +. $LKP_SRC/lib/debug.sh + +check_vars() +{ + ansible_host=localhost + echo $ansible_host + + [ -n "$ansible_repo" ] || die "ansible_repo is empty" + benchmark=${ansible_repo##*/} + + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible +} + +install_ansible() +{ + if [ "$os" == "openeuler" ]; then + yum -y reinstall git + yum -y install python3 + pip3 install ansible + elif [ "$os" == "centos" ]; then + yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm + yum -y reinstall git + yum -y install ansible + elif [ "$os" == "debian" ]; then + apt-get install -y ansible + fi + + command -v ansible > /dev/null && { + ansible localhost -m ping + [ $? -eq 0 ] && { + echo "Ansible is successfully deployed!" + } + } +} + +download_ansible_role() +{ + + ansible_roles_path=$ansible_path/roles + [ -d $ansible_roles_path ] || mkdir -p $ansible_roles_path + cd $ansible_roles_path + git clone git://github.com/$ansible_repo.git || exit 1 + # git clone git://$GIT_SERVER/$ansible_repo.git || exit + # ansible-galaxy install $username.$rolename + if [ $? -eq 0 ]; then + echo "$benchmark is successfully installed" + else + echo "Please check whether your ansible_repo is correct!" + fi +} + +ansible_playbook_run() +{ + cat <<-EOF > "$ansible_path"/site.yml + - hosts: $ansible_host + roles: + - role: $benchmark + EOF + [ $? -eq 0 ] && { + echo "Playbook has been generated!" + } + + ansible-playbook "$ansible_path"/site.yml + + if [ $? -eq 0 ]; then + echo "Your playbook is running successfully!" + else + echo "Your playbook failed to run!" && exit 1 + fi +} + +main() +{ + check_vars + install_ansible + download_ansible_role + ansible_playbook_run +} + +main -- 2.23.0

3 4

[PATCH v3 lkp-tests] add new type error
by Bai Jing 11 Mar '21

11 Mar '21

[input] Error: Unknown repo: 'powertools' [output] unknown-repo.powertools: 1 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/openeuler_docker | 8 +++++++- tests/openeuler_docker | 43 +++++++++++++++++++++++------------------- 2 files changed, 31 insertions(+), 20 deletions(-) diff --git a/stats/openeuler_docker b/stats/openeuler_docker index f85c9ca63..4c65c6bb3 100755 --- a/stats/openeuler_docker +++ b/stats/openeuler_docker @@ -6,12 +6,18 @@ sys_set = Set[] while (line = STDIN.gets) case line.chomp! - # Error: Unable to find a match: docker-registry + # Error: Unable to find a match: docker-registry mock xx when /Error: Unable to find a match: (.+)/ $1.split.each do |repo| repo_set << repo end + # Error: Unknown repo: 'powertools' + when /Error: Unknown repo: (.+)/ + repo = $1.delete!("'") + puts "unknown-repo.#{repo}: 1" + puts "unknown-repo.#{repo}.message: #{line}" + # /bin/sh: passwd: command not found when /\/bin\/sh: (.+): command not found/ puts "sh.command-not-found.#{$1}: 1" diff --git a/tests/openeuler_docker b/tests/openeuler_docker index b77b890a9..5270bde6e 100755 --- a/tests/openeuler_docker +++ b/tests/openeuler_docker @@ -13,7 +13,7 @@ cd $BENCHMARK_ROOT -centos_one_ok=false +nr_openeuler_fail=0 fit_repo() { @@ -21,7 +21,7 @@ fit_repo() git clone "$dockerfile_url" || exit 0 file_dir=$(find ./ -type f -name "*Dockerfile*" | xargs grep -HE '^FROM centos:(centos|)[78]') [ -n "$file_dir" ] || { - echo "current-repo not support centos" + echo "current-repo-not-support-centos: 1" exit 0 } } @@ -42,21 +42,23 @@ centos_docker() [ -n "$image" ] || image=${file#*/}_$i if [ -f "$file_dir/build.sh" ] && [ -x "$file_dir/build.sh" ]; then "$file_dir/build.sh" > /dev/null 2> /dev/null && { - centos_one_ok=true handle_dockerfile } else docker_build > /dev/null 2> /dev/null && { - centos_one_ok=true handle_dockerfile } fi let i++ done - [ "$centos_one_ok" == "false" ] && { - echo "centos all failed" + if [ "$nr_openeuler_fail" != 0 ]; then + echo "openeuler failed times: $nr_openeuler_fail" + echo "openeuler_valid_url: $dockerfile_url" + exit 1 + else + echo "current-repo-not-support-centos: 1" exit 0 - } + fi } handle_FROM() @@ -81,19 +83,18 @@ handle_epelrpm() } } -handle_groupadd() +add_base_commands() { - grep -qw "groupadd*" "$file" && { - sed -i '/groupadd*/i\ yum -y install shadow && \\' "$file" - } + # grep -qw "groupadd*" "$file" && { + # fix missing useradd, groupadd, chpasswd, etc. commands + sed -i '/FROM /a\RUN yum -y install shadow tar' "$file" + # } } handle_epel_repo() { - grep -qwE '.* wget .*(epel|CentOS-Base)\.repo' "$file" && { - sed 's|^|#|g' "$file" - } + sed -i '/.* wget .*(epel|CentOS-Base)\.repo/ s|^|#|g' "$file" } handle_dockerfile() @@ -101,15 +102,18 @@ handle_dockerfile() flag=1 handle_FROM handle_epel - handle_groupadd - docker_build "-openeuler" - [ $? -ne 0 ] || docker_run "-openeuler" + add_base_commands + if docker_build "-openeuler"; then + docker_run "-openeuler" + else + let nr_openeuler_fail++ + fi } docker_build() { docker build -f "$file" -t "${image}$1" . - [ $? -ne 0 ] && { + [ $? != 0 ] && { [ "$flag" == "1" ] && echo "${image}$1.build.fail: 1" return 1 } @@ -120,7 +124,8 @@ docker_build() docker_run() { docker run -d --name "${image}" "${image}$1" - docker_stats=$(docker inspect ${image} | grep Running | awk -F ' ' '{print $2}' | tr -d ',') + wait + docker_stats=$(docker inspect -f '{{.State.Running}}' ${image}) [ $docker_stats == false ] && { echo "${image}.run.fail: 1" return 1 -- 2.23.0

2 2

[PATCH v4 lkp-tests] add new type error
by Bai Jing 11 Mar '21

11 Mar '21

[input] Error: Unknown repo: 'powertools' [output] unknown-repo.powertools: 1 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/openeuler_docker | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/stats/openeuler_docker b/stats/openeuler_docker index f85c9ca63..4c65c6bb3 100755 --- a/stats/openeuler_docker +++ b/stats/openeuler_docker @@ -6,12 +6,18 @@ sys_set = Set[] while (line = STDIN.gets) case line.chomp! - # Error: Unable to find a match: docker-registry + # Error: Unable to find a match: docker-registry mock xx when /Error: Unable to find a match: (.+)/ $1.split.each do |repo| repo_set << repo end + # Error: Unknown repo: 'powertools' + when /Error: Unknown repo: (.+)/ + repo = $1.delete!("'") + puts "unknown-repo.#{repo}: 1" + puts "unknown-repo.#{repo}.message: #{line}" + # /bin/sh: passwd: command not found when /\/bin\/sh: (.+): command not found/ puts "sh.command-not-found.#{$1}: 1" -- 2.23.0

1 0

[PATCH compass-ci 1/2] container/web-backend: add API /get_table_error
by Lu Weitao 11 Mar '21

11 Mar '21

[Example] curl -X GET 'localhost:30000/get_job_error?group_id=wcl_ansible-openeuler-03-10' return: { "filter":{"group_id":"wcl_ansible-openeuler-03-10"}, "table_fields":["job_id","error_id","error_message","link to result"], "data_table":[ [ "crystal.1354921", "ansible_test.error.Unable-to-start-service-httpd-Job-for-httpdservice-xxx.message", "{\"changed\": false, \"msg\": \"Unable to start service httpd: Job for httpd.service failed because a timeout was exceeded.}", "http://172.17.0.1:11300/result/ansible_test/2021-03-10/vm-2p16g--wcl1/opene… ate/crystal.1354921" ], [ "crystal.1354929", ... ] ... ] } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/web-backend | 19 +++++++++ src/lib/web_backend.rb | 67 +++++++++++++++++++++++++++++++ 2 files changed, 86 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 74eda85..fa2a53d 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -146,3 +146,22 @@ end get '/get_repo_statistics' do new_refs_statistics(params) end + +# GET /get_job_error?suite=virttest&tbox_group=vm-2p8g +# Response +# { +# "filter":{"group_id":"wcl_ansible-openeuler-03-10"}, +# "table_fields":["job_id","error_id","error_message","link to result"], +# "data_table":[ +# [ +# "crystal.1354921", +# "ansible_test.error.Unable-to-start-service-httpd-Job-for-httpdservice-failed.message", +# "{\"changed\": false, \"msg\": \"Unable to start service httpd: Job for httpd.service failed because a timeout was exceeded.} +# "http://172.17.0.1:11300/result/ansible_test/2021-03-10/vm-2p16g--wcl1/opene…" +# ], +# ... +# ] +# } +get '/get_job_error' do + get_job_error(params) +end diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 89c0f3c..f09fc3d 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -12,6 +12,7 @@ require "#{CCI_SRC}/lib/compare.rb" require "#{CCI_SRC}/lib/constants.rb" require "#{CCI_SRC}/lib/es_query.rb" require "#{CCI_SRC}/lib/matrix2.rb" +require "#{CCI_SRC}/lib/params_group.rb" require "#{CCI_SRC}/lib/compare_data_format.rb" UPSTREAM_REPOS_PATH = ENV['UPSTREAM_REPOS_PATH'] || '/c/upstream-repos' @@ -548,3 +549,69 @@ def new_refs_statistics(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + + +# ------------------------------------------------------------------------------------------- +# job error table like: +# job_id error_id error_message link to result +# ------------------------------------------------------------------------------------- +# crystal.630608 "stderr.xxx" "messag:xxxx" https://$host:$port/$result_root +# ... +# ------------------------------------------------------------------------------------------- + +def get_job_error(params) + begin + body = job_error_body(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get error table error'] + end + + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end + +def job_error_body(params) + error_table = get_error_table(params) + { + filter: params, + table_fields: ["job_id", "error_id", "error_message", "link to result"], + data_table: error_table, + }.to_json +end + +def get_error_table(filter_items) + error_table = [] + + job_list = get_job_list(filter_items) + job_list.each do |job| + error_table += get_error_from_job(job) + end + + error_table +end + +def get_job_list(items) + es = ESQuery.new(ES_HOST, ES_PORT) + query_results = es.multi_field_query(items) + + extract_jobs_list(query_results['hits']['hits']) +end + +# get all error_id from one job +def get_error_from_job(job) + job_error = [] + job['stats'].each do |metric, value| + next unless metric.end_with?('.message', 'log') + + result_host = ENV['SRV_HTTP_RESULT_HOST'] || SRV_HTTP_RESULT_HOST + result_port = ENV['SRV_HTTP_RESULT_PORT'] || SRV_HTTP_RESULT_PORT + job_error << [ + job['id'], + metric, + value, + "http://#{result_host}:#{result_port}#{job['result_root']}" + ] + end + + job_error +end -- 2.23.0

2 2

[PATCH v2 compass-ci 3/3] src/scheduler: add local mount of kernel_common_params
by Xu Xijian 11 Mar '21

11 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/kernel_params.cr | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index bb5983f..8ad5029 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -4,8 +4,12 @@ class Job private def kernel_common_params common_params = "user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job ip=dhcp" - return "#{common_params} rw" if "#{self.os_mount}" == "local" - return "#{common_params} rootovl ro" + return "#{common_params} rootovl ro" unless "#{self.os_mount}" == "local" + + os_info = "#{os}_#{os_arch}_#{os_version}" + use_root_partition = "/dev/mapper/os-#{os_info}_#{src_lv_suffix}" if @hash["src_lv_suffix"]? != nil + save_root_partition = "/dev/mapper/os-#{os_info}_#{boot_lv_suffix}" if @hash["boot_lv_suffix"]? != nil + return "#{common_params} local use_root_partition=#{use_root_partition} save_root_partition=#{save_root_partition} rw" end private def kernel_custom_params @@ -20,7 +24,7 @@ class Job "cifs" => "root=cifs://#{OS_HTTP_HOST}#{os_real_path}" + ",guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino", "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0", - "local" => "root=/dev/mapper/os-#{self.os}_#{self.os_arch}_#{self.os_version}", + "local" => "root=#{OS_HTTP_HOST}:#{os_real_path}", # root is just used to temporarily mount a root in initqueue stage when lvm is not ready "container" => "", } -- 2.23.0

2 1

[PATCH v4 compass-ci 2/2] lib/dump-stat.rb: refactor lkp-tests/sbin/dump-stat
by Lu Weitao 11 Mar '21

11 Mar '21

[Why] support handle hash from lkp-tests/stats/$script.rb Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/dump_stat.rb | 194 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 194 insertions(+) create mode 100644 lib/dump_stat.rb diff --git a/lib/dump_stat.rb b/lib/dump_stat.rb new file mode 100644 index 0000000..1c0695e --- /dev/null +++ b/lib/dump_stat.rb @@ -0,0 +1,194 @@ +# SPDX-License-Identifier: GPL-2.0-only + +LKP_SRC ||= ENV['LKP_SRC'] || '/c/lkp-tests' + +require "#{LKP_SRC}/lib/statistics" +require "#{LKP_SRC}/lib/bounds" +require "#{LKP_SRC}/lib/yaml" +require "#{LKP_SRC}/lib/job" +require "#{LKP_SRC}/lib/string_ext" +require "#{LKP_SRC}/lib/log" +require 'set' + +UNSTRUCTURED_MONITORS = %w[ftrace].to_set + +def warn_stat(msg, monitor) + log_warn msg + log_warn "check #{RESULT_ROOT}/#{monitor}" +end + +# dump stat which input by lkp-tests/stats/$script return +# input: +# eg-1: +# { +# "pgfree" => [275506, 280018], +# ... +# } +# eg-2: +# { +# "iperf.tcp.sender.bps" => 34804801216.197174, +# "iperf.tcp.receiver.bps" => "34804762215.18231" +# } +module DumpStat + def self.dump_stat(monitor, stat_result) + @result = {} + @invalid_records = [] + @record_index = 0 + @monitor = monitor + + stat_result.each do |key, value| + key = key.resolve_invalid_bytes + next if key[0] == '#' + next if value.empty? || value == 0 + next if monitor =~ /^(dmesg|kmsg)$/ && key =~ /^(message|pattern):/ + + if key =~ /[ \t]/ + @invalid_records.push @record_index + warn_stat "whitespace in stats name: #{key}", @monitor + return nil # for exit current stats/script dump-stat + end + next if assign_log_message(key, value) + + k = @monitor + '.' + key + @result[k] ||= [] + fill_zero(k) + if value.is_a?(String) + value = check_string_value(k, value, @monitor) + next unless value + return nil unless number?(value, @invalid_records) + + value = value.index('.') ? value.to_f : value.to_i + elsif value.is_a?(Array) + (0..value.size - 1).each do |i| + next unless value[i].is_a?(String) + + value[i] = check_string_value(k, value[i], @monitor) + next unless value[i] + return nil unless number?(value[i], @invalid_records) + + value[i] = value[i].index('.') ? value[i].to_f : value[i].to_i + valid_stats_verification(k, value[i]) + end + @result[k] = value + next + end + valid_stats_verification(k, value) + @result[k].push value + end + return nil if @result.empty? + + remove_zero_stats + delete_invalid_number(@result, @invalid_records, @monitor) + cols_verifation + return nil unless useful_result?(@result) + + save_json(@result, "#{RESULT_ROOT}/#{(a)monitor}.json", @result.size * @min_cols > 1000) + end + + # keep message | log line which key end with .message|.log + def self.assign_log_message(key, value) + if key.end_with?('.message', '.log') + k = @monitor + '.' + key + @result[k] = value + return true + end + + false + end + + def self.fill_zero(key) + size = @result[key].size + if @record_index < size + @record_index = size + elsif (@record_index - size).positive? + # fill 0 for missing values + @result[key].concat([0] * (@record_index - size)) + end + end + + def self.valid_stats_verification(key, value) + return nil if valid_stats_range? key, value + + @invalid_records.push @record_index + puts "outside valid range: #{value} in #{key} #{RESULT_ROOT}" + end + + def self.remove_zero_stats + @max_cols = 0 + @min_cols = Float::INFINITY + @min_cols_stat = '' + @max_cols_stat = '' + zero_stats = [] + @result.each do |key, val| + next if key.end_with?('.message', '.log') + + if @max_cols < val.size + @max_cols = val.size + @max_cols_stat = key + end + if @min_cols > val.size + @min_cols = val.size + @min_cols_stat = key + end + next if val[0] != 0 + next if val[-1] != 0 + next if val.sum != 0 + + zero_stats << key + end + zero_stats.each { |x| @result.delete x } + end + + def self.cols_verifation + return nil unless @min_cols < @max_cols && !UNSTRUCTURED_MONITORS.include?(@monitor) + + if @min_cols == @max_cols - 1 + @result.each { |_k, y| y.pop if y.size == @max_cols } + puts "Last record seems incomplete. Truncated #{RESULT_ROOT}/#{(a)monitor}.json" + else + warn_stat "Not a matrix: value size is different - #{@min_cols_stat}: #{@min_cols} != #{@max_cols_stat}: #{@max_cols}: #{RESULT_ROOT}/#{(a)monitor}.json", @monitor + end + end +end + +def check_string_value(key, value, monitor) + # value terminator is expected. If not, throw out an error warning. + warn_stat "no line terminator in stats value: #{value}", monitor if value.chomp!.nil? + + value.strip! + if value.empty? + warn_stat "empty stat value of #{key}", monitor + return nil + end + + return value +end + +# only number is valid +def number?(value, invalid_records) + unless value.numeric? + invalid_records.push record_index + warn_stat "invalid stats value: #{value}", monitor + return nil + end + + true +end + +def delete_invalid_number(result, invalid_records, monitor) + return nil if monitor == 'ftrace' + + invalid_records.reverse_each do |index| + result.each do |_k, value| + value.delete_at index + end + end +end + +def useful_result?(result) + return nil if result.empty? + return nil if result.values[0].size.zero? + return nil if result.values[-1].size.zero? + + true +end -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] sbin/result2stats: support call function from stats/script
by Lu Weitao 11 Mar '21

11 Mar '21

[Why] old logical: lkp-tests/stats/script < script_log > tmpfile lkp-tests/sbin/dump-stat < tmpfile > $script.json this flow will escape special characters [How] use return of function to keep original characters lkp-tests/stats/script.rb func(read(script_log)) --> hash compass-ci/lib/dump-stats.rb hash --> dump_stats() --> $script.json [Dependence] lkp-tests/stats/$script.rb, for example: file_name: lkp-tests/stats/proc-vmstats.rb function name: proc_vmstats() input: Array(String) ["time: 231231234", "nr_free_page: 89721", ...] output: Hash(String, Array(Number|String) | Number | String) { "time" => [231231234, 231231235], "nr_free_page" => [89721,88564], ... } or { "time" => 231231234, "nr_free_page" => 89721, ... } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/stats.rb | 21 ++++++++++++++++----- lib/stats_wrapper.rb | 44 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+), 5 deletions(-) diff --git a/lib/stats.rb b/lib/stats.rb index c01d38d..a02f074 100644 --- a/lib/stats.rb +++ b/lib/stats.rb @@ -14,11 +14,18 @@ end def available_stats vailable_stats = [] + available_stats_func = [] Dir.open(LKP_SRC + '/stats').each do |file| - vailable_stats << file unless file.start_with?('.') + next if file.start_with?('.') + + if file.end_with?('.rb') + available_stats_func << file.chomp('.rb') + else + vailable_stats << file + end end - vailable_stats + [vailable_stats, available_stats_func] end def assign_default_stats(stats_list) @@ -32,13 +39,17 @@ class Stats def initialize(result_root) @result_root = result_root @job = load_yaml(File.join(result_root, 'job.yaml')) - @available_stats = available_stats + @available_stats, @available_stats_func = available_stats end def extract_stats stats_list = assign_stats_list stats_list.each do |stat| - StatsWrapper.wrapper(*stat) + if @available_stats_func.include?(stat) + StatsWrapper.wrapper_func(*stat) + else + StatsWrapper.wrapper(*stat) + end end end @@ -48,7 +59,7 @@ class Stats stats_list << ['time', @job['suite'] + '.time'] stats_list << 'stderr' @job.each_key do |k| - if @available_stats.include?(k) + if @available_stats.include?(k) || @available_stats_func.include?(k) stats_list << k end end diff --git a/lib/stats_wrapper.rb b/lib/stats_wrapper.rb index 8a6d6a9..faa81df 100644 --- a/lib/stats_wrapper.rb +++ b/lib/stats_wrapper.rb @@ -6,6 +6,11 @@ LKP_SRC ||= ENV['LKP_SRC'] || '/c/lkp-tests' require "#{LKP_SRC}/lib/log.rb" require 'tempfile' require 'English' +require_relative './dump_stat' + +Dir[File.expand_path("#{LKP_SRC}/stats/*.rb")].uniq.each do |file| + require file +end PROGRAM_DIR = "#{LKP_SRC}/stats" @@ -35,6 +40,27 @@ module StatsWrapper delete_log_package end + def self.wrapper_func(program, program_time = nil) + @program = program + @stats_group = program_time || program + @log = "#{RESULT_ROOT}/#{@stats_group}" + + return unless File.exist?("#{(a)log}.yaml") || pretreatment + + if File.exist?("#{(a)log}.yaml") + log_lines = read_log("#{(a)log}.yaml") + stat_result = parse_simple_log_yaml(log_lines) + else + log_lines = read_log(@log) + call_func_cmd = "#{(a)program.gsub('-', '_')}(log_lines)" # eg: proc_vmstats(log_lines) + stat_result = eval(call_func_cmd) + end + return unless DumpStat.dump_stat(@stats_group, stat_result) + + check_empty_json + delete_log_package + end + def self.pretreatment return unless available_program? return unless unzip_log @@ -261,3 +287,21 @@ module StatsWrapper File.delete(@tmpfile) if File.exist?(@tmpfile) end end + +# read line of log +# return Array(String) +def read_log(log_path) + return nil unless File.exist?(log_path) + + File.readlines(log_path) +end + +def parse_simple_log_yaml(log_lines) + result = Hash.new { |hash, key| hash[key] = [] } + log_lines.each do |line| + key, value = line.split(/:?\s+/) + result[key] << value + end + + result +end -- 2.23.0

1 0

[PATCH compass-ci 2/2] container/web-backend/start: add SRV_HTTP_RESULT/PORT to env
by Lu Weitao 11 Mar '21

11 Mar '21

[Why] the API /get_job_error need the host/port Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/start | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/container/web-backend/start b/container/web-backend/start index 941fb11..6731d27 100755 --- a/container/web-backend/start +++ b/container/web-backend/start @@ -9,6 +9,10 @@ . $CCI_SRC/container/defconfig.sh docker_rm web-backend +load_cci_defaults +[[ $SRV_HTTP_RESULT_HOST ]] || SRV_HTTP_RESULT_HOST=$(curl ip.sb) +[[ $SRV_HTTP_RESULT_PORT ]] || SRV_HTTP_RESULT_PORT=11300 + cmd=( docker run --name web-backend @@ -17,6 +21,8 @@ cmd=( -p 32767:32767 -e ES_PORT=9200 + -e SRV_HTTP_RESULT_HOST=$SRV_HTTP_RESULT_HOST + -e SRV_HTTP_RESULT_PORT=$SRV_HTTP_RESULT_PORT -e CCI_SRC=/c/compass-ci -e LKP_SRC=/c/lkp-tests -e UPSTREAM_REPOS_PATH=/c/upstream-repos -- 2.23.0

1 0

[PATCH compass-ci] container: add start depends
by Cao Xueliang 11 Mar '21

11 Mar '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/auto-submit/start-depends | 2 +- container/delimiter/start-depends | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/container/auto-submit/start-depends b/container/auto-submit/start-depends index 38fc696..b5a1e0c 100755 --- a/container/auto-submit/start-depends +++ b/container/auto-submit/start-depends @@ -1,2 +1,2 @@ rabbitmq -es +register-accounts diff --git a/container/delimiter/start-depends b/container/delimiter/start-depends index 95df1a1..880f5b1 100755 --- a/container/delimiter/start-depends +++ b/container/delimiter/start-depends @@ -1,2 +1,3 @@ taskqueue send-mail +register-accounts -- 2.23.0

2 2

[PATCH v2 compass-ci 2/3] container/dracut-initrd: set sysroot in local mount situation
by Xu Xijian 11 Mar '21

11 Mar '21

When mount local filesystem in os starting, we will create or use a local existed logical volume to replace remote filesystem. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- .../dracut-initrd/bin/set-local-sysroot.sh | 83 +++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100644 container/dracut-initrd/bin/set-local-sysroot.sh diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh new file mode 100644 index 0000000..dc9000b --- /dev/null +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -0,0 +1,83 @@ +#!/bin/sh + +analyse_kernel_cmdline_params() { + rootfs="$(getarg root=)" + # example: $nfs_server_ip:/os/${os}/${os_arch}/${os_version}-snapshots/20210310005959 + rootfs_src=$(echo $"$rootfs" | sed 's/\///') + timestamp="$(basename $root)" + os="$(echo $root_src | awk -F '/|-' '{print $2}')" + os_arch="$(echo $root_src | awk -F '/|-' '{print $3}')" + os_version="$(echo $root_src | awk -F '/|-' '{print $4}')" + os_info="${os}_${os_arch}_${os_version}" + export rootfs_src timestamp os_info +} + +sync_src_lv() { + local src_lv="$1" + + [ -e "$src_lv" ] && return + + # create logical volume + src_lv_devname="$(basename $src_lv)" + lvm lvcreate -y -L 10G --name "${src_lv_devname#os-}" os + mke2fs -t ext4 -F "$src_lv" + + # sync nfsroot to $src_lv + mkdir -p /mnt1 && mount -t nfs "$rootfs_src" /mnt1 + mkdir -p /mnt2 && mount "$src_lv" /mnt2 + cp -a /mnt1/. /mnt2/ + umount /mnt1 /mnt2 + + # change permission of "$src_lv" to readonly + lvm lvchange --permission r "$src_lv" +} + +snapshot_boot_lv() { + local src_lv="$1" + local boot_lv="$2" + + [ "$src_lv" == "$boot_lv" ] && return + + lvm lvremove --force "$boot_lv" + boot_lv_devname="$(basename $boot_lv)" + lvm lvcreate --size 10G --name ${boot_lv_devname#os-} --snapshot "$src_lv" +} + +set_sysroot() { + boot_lv="$1" + umount "$NEWROOT" + mount "$boot_lv" "$NEWROOT" +} + +if ! getargbool 0 local; then + exit 0 +fi + +analyse_kernel_cmdline_params + +sed -i "s/^locking_type = .*/locking_type = 1/" /etc/lvm/lvm.conf + +use_root_partition="$(getarg use_root_partition=)" +if [ -n "$use_root_partition" ]; then + src_lv="$use_root_partition" # /dev/mapper/os-${os_info}_${timestamp} + sync_src_lv "$src_lv" +else + src_lv="/dev/mapper/os-${os_info}_$timestamp" + [ -e "src_lv" ] || { + echo "warn dracut: FATAL: no src_lv with local mount, reboot" + reboot + } +fi + +save_root_partition="$(getarg save_root_partition=)" +if [ -n "$save_root_partition" ]; then + boot_lv="$save_root_partition" +else + boot_lv="/dev/mapper/os-${os_info}" +fi + +snapshot_boot_lv "$src_lv" "$boot_lv" + +set_sysroot "$boot_lv" + +exit 0 -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] container/dracut-initrd: add handling of pre-mount stage into setup-dracut.sh
by Xu Xijian 11 Mar '21

11 Mar '21

Add the content of set-local-sysroot.sh into the tail of dracut/modules.d/98dracut-systemd/dracut-pre-mount.sh. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 849480d..45e895c 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -18,3 +18,6 @@ cp -a /usr/local/bin/cifs-lib.sh /usr/lib/dracut/modules.d/95cifs/ cat overlay-lkp.sh >> /usr/lib/dracut/modules.d/90overlay-root/overlay-mount.sh sed -i "/install() {/a\ inst /usr/bin/awk" /usr/lib/dracut/modules.d/40network/module-setup.sh sed -i "/install() {/a\ inst /sbin/mke2fs" /usr/lib/dracut/modules.d/98dracut-systemd/module-setup.sh +pre_mount_file="/usr/lib/dracut/modules.d/98dracut-systemd/dracut-pre-mount.sh" +[ "$(sed -n '$p' $pre_mount_file)" = "exit 0" ] && sed -i '$d' "$pre_mount_file" +cat set-local-sysroot.sh >> "$pre_mount_file" -- 2.23.0

1 0

[PATCH v2 compass-ci 3/3] lib/git_mirror.rb: github download workaround
by Li Yuanchao 10 Mar '21

10 Mar '21

There is a problem to download from github by http or https. fatal: unable to access 'https://github.com/liyc-github/AvxToNeon/': Failed to connect to github.com port 443: Connection timed out When it occurs, change url to "git://xxx". Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 8f23af3..50a6c75 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -39,19 +39,27 @@ class GitMirror return url end + def stderr_443?(stderr_info) + return true if stderr_info.include?('Failed to connect to github.com port 443') + + return false + end + def git_clone(url, mirror_dir) url = get_url(Array(url)[0]) 10.times do - %x(git clone --mirror --depth 1 #{url} #{mirror_dir} 2>&1) + stderr = %x(git clone --mirror --depth 1 #{url} #{mirror_dir} 2>&1) return true if File.directory?(mirror_dir) && File.exist?("#{mirror_dir}/config") + + url = "git://#{url.split('://')[1]}" if stderr_443?(stderr) end return false end def git_fetch(mirror_dir) if File.exist?("#{mirror_dir}/shallow") + FileUtils.rm("#{mirror_dir}/shallow.lock") if File.exist?("#{mirror_dir}/shallow.lock") %x(git -C #{mirror_dir} fetch --unshallow 2>&1) - return false end fetch_info = %x(git -C #{mirror_dir} fetch 2>&1) @@ -59,6 +67,11 @@ class GitMirror if fetch_info.include?(ERR_MESSAGE) && Dir.empty?(mirror_dir) FileUtils.rmdir(mirror_dir) end + + if stderr_443?(fetch_info) + url = %x(git -C #{mirror_dir} ls-remote --get-url origin).chomp + %x(git -C #{mirror_dir} remote set-url origin git://#{url.split('://')[1]}) + end return fetch_info.include? '->' end @@ -66,7 +79,7 @@ class GitMirror url = get_url(Array(url)[0]) git_url = %x(git -C #{mirror_dir} ls-remote --get-url origin).chomp - return true if url == git_url + return true if git_url == url || git_url.include?(url.split('://')[1]) return false end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] sbin/auto_submit: change the path to do git command
by Li Yuanchao 10 Mar '21

10 Mar '21

As the path of repo mirrors under /srv/git changes, the path to execute git command should change too. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 857af29..faebb2f 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -44,14 +44,16 @@ class AutoSubmit submit_argv end - def submit_one_job(submit_argv, git_repo, branch, commit_id) - inactive_time = %x(git -C /srv/git/#{git_repo}.git log --pretty=format:"%cr" -1 #{commit_id}) + def submit_one_job(submit_argv, git_repo, belong, branch, commit_id) + mirror_dir = "/srv/git/#{belong}/#{git_repo}.git" + inactive_time = %x(git -C #{mirror_dir} log --pretty=format:"%cr" -1 #{commit_id}) return if inactive_time =~ /(month|year)/ real_argvs = Array.new(submit_argv) real_argvs.push("upstream_branch=#{branch}") real_argvs.push("upstream_commit=#{commit_id}") - tag = %x(git -C /srv/git/#{git_repo}.git tag --points-at #{commit_id}) + real_argvs.push("upstream_dir=#{belong}") + tag = %x(git -C #{mirror_dir} tag --points-at #{commit_id}) real_argvs.push("upstream_tag=#{tag}") unless tag.empty? system(real_argvs.join(' ')) @@ -60,16 +62,17 @@ class AutoSubmit def submit_specific_branch(submit_argv, newrefs_info, branches) branches.each do |branch_name| branch = "refs/heads/#{branch_name}" - next unless newrefs_info['new_refs']['heads'][branch] + commit_id = newrefs_info['new_refs']['heads'][branch] + next unless commit_id - submit_one_job(submit_argv, newrefs_info['git_repo'], branch_name, newrefs_info['new_refs']['heads'][branch]) + submit_one_job(submit_argv, newrefs_info['git_repo'], newrefs_info['belong'], branch_name, commit_id) end end - def submit_non_specific(non_specific_commits, git_repo, submit_argv) - non_specific_commits.each do |branch, commit_id| + def submit_non_specific(newrefs_info, submit_argv) + newrefs_info['new_refs']['heads'].each do |branch, commit_id| branch = branch.delete_prefix('refs/heads/') - submit_one_job(submit_argv, git_repo, branch, commit_id) + submit_one_job(submit_argv, newrefs_info['git_repo'], newrefs_info['belong'], branch, commit_id) end end @@ -96,7 +99,7 @@ class AutoSubmit if element['branches'] submit_specific_branch(argvs, newrefs_info, element['branches']) else - submit_non_specific(newrefs_info['new_refs']['heads'], newrefs_info['git_repo'], argvs) + submit_non_specific(newrefs_info, argvs) end end end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] lib/git_mirror.rb: don't count new ref if it is not new
by Li Yuanchao 10 Mar '21

10 Mar '21

When repos are added, new_refs_count will increase. But many repos are not updated that day, that cause new_refs_count increase much more than real new refs number. Now only count those really new. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 53c9e59..8f23af3 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -194,7 +194,7 @@ class MirrorMain feedback_info[:new_refs] = new_refs send_message(feedback_info) - new_refs_log(git_repo, new_refs[:heads].length) + new_refs_log(git_repo, new_refs[:heads].length) if last_commit_new?(git_repo) end def do_push(fork_key) @@ -327,7 +327,7 @@ class MirrorMain def update_fork_stat(git_repo, possible_new_refs) update_stat_fetch(git_repo) - update_stat_new_refs(git_repo) if possible_new_refs + update_stat_new_refs(git_repo) if possible_new_refs && last_commit_new?(git_repo) es_repo_update(git_repo) end end @@ -521,4 +521,11 @@ class MirrorMain nr_new_branch: nr_new_branch }.to_json) end + + def last_commit_new?(git_repo) + inactive_time = %x(git -C /srv/git/#{@git_info[git_repo]['belong']}/#{git_repo}.git log --pretty=format:"%cr" -1) + return false if inactive_time =~ /(day|week|month|year)/ + + return true + end end -- 2.23.0

1 0

[PATCH v2 lkp-tests 11/11] install-iso-pre: add step: submit install-iso-sync-rootfs.yaml
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index 830c6bfd8a6a..ef4a0659fe02 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -236,7 +236,22 @@ submit_install_iso_firstboot_yaml() submit_install_iso_sync_rootfs_yaml() { - : + log_info "start submit install-iso-sync-rootfs.yaml" + + local sync_rootfs_yaml="${LKP_SRC}/jobs/install-iso-sync-rootfs.yaml" + [ -f "${sync_rootfs_yaml}" ] || die "cannot find ${sync_rootfs_yaml}" + + ${LKP_SRC}/sbin/submit \ + ${sync_rootfs_yaml} \ + suite=install-iso-sync-rootfs-${ROOTFS_SUFFIX} \ + iso_os=${iso_os} \ + iso_arch=${iso_arch} \ + iso_version=${iso_version} \ + NFS_SERVER_HOST=${NFS_SERVER_HOST} \ + NFS_ROOT_DIR=${NFS_ROOT_DIR} \ + ROOTFS_SUFFIX=${ROOTFS_SUFFIX} \ + VG_NAME=${VG_NAME} \ + testbox=$testbox || die "submit ${sync_rootfs_yaml} failed" } main() -- 2.23.0

1 0

[PATCH v2 lkp-tests 10/11] install-iso-pre: add step: submit install-iso-firstboot.yaml
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index cc939ba20ce8..830c6bfd8a6a 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -218,7 +218,20 @@ submit_install_iso_yaml() submit_install_iso_firstboot_yaml() { - : + log_info "start submit install-iso-firstboot.yaml" + + local firstboot_yaml="${LKP_SRC}/jobs/install-iso-firstboot.yaml" + [ -f "${firstboot_yaml}" ] || die "cannot find ${firstboot_yaml}" + + ${LKP_SRC}/sbin/submit \ + ${firstboot_yaml} \ + suite=install-iso-firstboot-${ROOTFS_SUFFIX} \ + os=${iso_os} \ + os_arch=${iso_arch} \ + os_version=${iso_version} \ + os_mount=local \ + kernel_custom_params="src_lv_suffix=${ROOTFS_SUFFIX} boot_lv_suffix=${ROOTFS_SUFFIX}" \ + testbox=$testbox || die "submit ${firstboot_yaml} failed" } submit_install_iso_sync_rootfs_yaml() -- 2.23.0

1 0

[PATCH v2 lkp-tests 09/11] install-iso-pre: add step: submit install-iso.yaml
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index 000c1fd6811b..cc939ba20ce8 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -185,9 +185,35 @@ prepare_submit() EOF } +prepare_install_iso_yaml() +{ + log_info "start prepare install-iso.yaml" + + local tftp_root="tftp://${TFTP_SERVER_HOST}/EFI/BOOT/${iso_os}/${iso_arch}/${iso_version}" + local custom_ipxe + + custom_ipxe="#!ipxe\\\\n" + custom_ipxe="${custom_ipxe}\\\\ndhcp" + custom_ipxe="${custom_ipxe}\\\\ninitrd ${tftp_root}/initrd.img" + custom_ipxe="${custom_ipxe}\\\\nkernel ${tftp_root}/vmlinux ro inst.geoloc=0 console=ttyS0 smmu.bypassdev=0x1000:0x17 smmu.bypassde1000:0x15 viff video=VGA-1:640x480-32@60me initrd=initrd.img ip=dhcp inst.ks=http://${HTTP_SERVER_HOST}:${HTTP_SERVER_PORT}/os/kickstart/ks/${iso_os}_${iso_arch}_${iso_version}.cfg" + custom_ipxe="${custom_ipxe}\\\\nboot\\\\n" + + log_info "custom_ipxe: $custom_ipxe" + sed -i "/custom_ipxe:/ccustom_ipxe: \"${custom_ipxe}\"" ${install_iso_yaml} +} + submit_install_iso_yaml() { - : + log_info "start submit install-iso.yaml" + + local install_iso_yaml="${LKP_SRC}/jobs/install-iso.yaml" + [ -f "${install_iso_yaml}" ] || die "cannot find ${install_iso_yaml}" + + prepare_install_iso_yaml + + ${LKP_SRC}/sbin/submit \ + ${install_iso_yaml} \ + testbox=$testbox || die "submit ${install_iso_yaml} failed" } submit_install_iso_firstboot_yaml() -- 2.23.0

1 0

[PATCH v2 lkp-tests 08/11] install-iso-pre: add step: prepare_submit
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- distro/depends/install-iso-pre | 2 ++ tests/install-iso-pre | 18 +++++++++++++++++- 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/distro/depends/install-iso-pre b/distro/depends/install-iso-pre index 6f88eb919896..4569e1e3cdc0 100644 --- a/distro/depends/install-iso-pre +++ b/distro/depends/install-iso-pre @@ -1 +1,3 @@ lvm2 +ruby +git diff --git a/tests/install-iso-pre b/tests/install-iso-pre index 7b9e6fd2107c..000c1fd6811b 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -166,7 +166,23 @@ prepare_ks() prepare_submit() { - : + ${LKP_SRC}/sbin/install-dependencies.sh + + export CCI_REPOS="/c" + + mkdir -p $CCI_REPOS && cd $CCI_REPOS + git clone $LAB_Z9_URL || die "clone lab-z9 git repo failed: $LAB_Z9_URL" + + local config_yaml="/etc/compass-ci/defaults/sparrow.yaml" + mkdir -p $(dirname $config_yaml) + cat >> $config_yaml <<-EOF + SCHED_HOST: ${SCHED_HOST} + SCHED_PORT: ${SCHED_PORT} + my_name: ${my_name} + my_email: ${my_email} + my_token: ${MY_TOKEN} + lab: z9 + EOF } submit_install_iso_yaml() -- 2.23.0

1 0

[PATCH v2 lkp-tests 07/11] install-iso-pre: add step: prepare_ks
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 72 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 71 insertions(+), 1 deletion(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index d800704f648c..7b9e6fd2107c 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -89,9 +89,79 @@ prepare_lv() create_lv } +generate_ks() +{ + log_info "start generate $ks_file" + + local nic=$(ip route get ${LKP_SERVER} | awk '{print $3; exit}') + + cat > ${ks_file} <<-EOF + # Use network installation + url --url="http://${HTTP_SERVER_HOST}:${HTTP_SERVER_PORT}/${HTTP_ROOT_DIR}/${iso_os}/${iso_arch}/${iso_version}/" + + # Network information + network --bootproto=dhcp --device=${nic} --ipv6=auto --activate + network --hostname=localhost.localdomain + + # Specifies a list of disks for the installation program to use. All other disks are ignored. + ignoredisk --only-use=${rootfs_disk#/dev/} + + # if installation target is a logical volume, bootloader should be none. + bootloader --location=none + logvol / --vgname=${VG_NAME} --name=${LV_NAME} --useexisting + + # Root password + rootpw --iscrypted ${ROOTFS_ROOT_PASSWD} + + # Reboot after installation + reboot --eject + + # Use text mode install + text + + # Keyboard layouts + keyboard --vckeymap=us --xlayouts='us' + + # System language + lang en_US.UTF-8 + + # Do not configure the X Window System + skipx + + # System services + services --enabled="chronyd" + + # System timezone + timezone Asia/Shanghai --isUtc + + %packages + @^minimal-environment + + %end + + %anaconda + pwpolicy root --minlen=8 --minquality=1 --notstrict --nochanges --notempty + pwpolicy user --minlen=8 --minquality=1 --notstrict --nochanges --emptyok + pwpolicy luks --minlen=8 --minquality=1 --notstrict --nochanges --notempty + %end + EOF +} + +upload_ks() +{ + log_info "start upload ks to rsync-server: ${RSYNC_SERVER_HOST}:${RSYNC_SERVER_PORT}" + + rsync -vrtopg --port ${RSYNC_SERVER_PORT} ${ks_file} ${RSYNC_SERVER_HOST}::ks/ || die "upload ks to rsync-server failed, please check." +} + prepare_ks() { - : + log_info "start prepare kickstart file" + + local ks_file=/tmp/${iso_os}_${iso_arch}_${iso_version}.ks + + generate_ks + upload_ks } prepare_submit() -- 2.23.0

1 0

[PATCH v2 lkp-tests 06/11] install-iso-pre: add step: prepare_lv
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- distro/depends/install-iso-pre | 1 + tests/install-iso-pre | 46 +++++++++++++++++++++++++++++++++- 2 files changed, 46 insertions(+), 1 deletion(-) create mode 100644 distro/depends/install-iso-pre diff --git a/distro/depends/install-iso-pre b/distro/depends/install-iso-pre new file mode 100644 index 000000000000..6f88eb919896 --- /dev/null +++ b/distro/depends/install-iso-pre @@ -0,0 +1 @@ +lvm2 diff --git a/tests/install-iso-pre b/tests/install-iso-pre index 4817f6d8eb37..d800704f648c 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -40,9 +40,53 @@ check_params() done } +create_vg() +{ + log_info "start create vg:$VG_NAME" + + local disk + + for disk in "${rootfs_disk[@]}" + do + log_info "start handle disk: $disk" + pvdisplay ${disk} > /dev/null || pvcreate ${disk} || die "create pv failed: $disk" + + if vgdisplay ${VG_NAME} > /dev/null; then + vgextend ${VG_NAME} ${disk} + else + vgcreate ${VG_NAME} ${disk} + fi + done + +} + +create_lv() +{ + log_info "start create lv:$LV_NAME" + + lvcreate -y -L 10G -n ${LV_NAME} ${VG_NAME} || die "create logical volume failed." +} + prepare_lv() { - : + log_info "start prepare logical volume" + + # activate lvm + vgchange -ay + + # VG_NAME defines in job.yaml + export VG_NAME + export LV_NAME="${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX}" + + # lv existed + lvdisplay /dev/mapper/${VG_NAME}-${LV_NAME} > /dev/null && { + log_info "iso already install to this testbox, so rootfs maybe already generated ok. please check it." + exit 0 + } + + # lv not existed + vgdisplay ${VG_NAME} > /dev/null || create_vg + create_lv } prepare_ks() -- 2.23.0

1 0

[PATCH v2 lkp-tests 05/11] feat: step 4 of iso2rootfs: add entry job
by Yu Chuan 10 Mar '21

10 Mar '21

Framework of install-iso-pre. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/install-iso-pre.yaml | 52 +++++++++++++++++++++++ tests/install-iso-pre | 86 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 138 insertions(+) create mode 100644 jobs/install-iso-pre.yaml create mode 100755 tests/install-iso-pre diff --git a/jobs/install-iso-pre.yaml b/jobs/install-iso-pre.yaml new file mode 100644 index 000000000000..a9dbe46b4bf3 --- /dev/null +++ b/jobs/install-iso-pre.yaml @@ -0,0 +1,52 @@ +suite: install-iso-pre + +install-iso-pre: + ################# + # iso info + ################# + iso_os: + iso_arch: + iso_version: + + ################# + # http server info, which stored the installation source + ################# + HTTP_SERVER_HOST: + HTTP_SERVER_PORT: + HTTP_ROOT_DIR: + + ################# + # rsync-server info, used for upload ks + ################# + RSYNC_SERVER_HOST: + RSYNC_SERVER_PORT: + + ################# + # tftp server info, used for kickstart auto install + ################# + TFTP_SERVER_HOST: + + ################# + # nfs server info, used for store rootfs + ################# + NFS_SERVER_HOST: + NFS_ROOT_DIR: + + ################# + # result rootfs suffix, format: $(date "+%Y%m%d%H%M%S") + ################# + ROOTFS_SUFFIX: + + ################# + # name of volume group which stores the logical volume of rootfs + ################# + VG_NAME: + + ################# + # lab_z9 git repo clone url, used for find host file when submit + ################# + LAB_Z9_URL: + +secrets: + ROOTFS_ROOT_PASSWD: + MY_TOKEN: diff --git a/tests/install-iso-pre b/tests/install-iso-pre new file mode 100755 index 000000000000..4817f6d8eb37 --- /dev/null +++ b/tests/install-iso-pre @@ -0,0 +1,86 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +set -e + +. $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/log.sh + +check_params() +{ + log_info "start check params" + + local required_vars=( + "iso_os" + "iso_arch" + "iso_version" + "ROOTFS_SUFFIX" + "VG_NAME" + + "HTTP_SERVER_HOST" + "HTTP_SERVER_PORT" + "HTTP_ROOT_DIR" + "ROOTFS_ROOT_PASSWD" + "rootfs_disk" + + "RSYNC_SERVER_HOST" + "RSYNC_SERVER_PORT" + "TFTP_SERVER_HOST" + "NFS_SERVER_HOST" + "NFS_ROOT_DIR" + + "LAB_Z9_URL" + "MY_TOKEN" + ) + + for i in "${required_vars[@]}" + do + [ -n "$(eval echo \$$i)" ] || die "cannot get value of var: $i" + done +} + +prepare_lv() +{ + : +} + +prepare_ks() +{ + : +} + +prepare_submit() +{ + : +} + +submit_install_iso_yaml() +{ + : +} + +submit_install_iso_firstboot_yaml() +{ + : +} + +submit_install_iso_sync_rootfs_yaml() +{ + : +} + +main() +{ + check_params + + prepare_lv + prepare_ks + prepare_submit + + submit_install_iso_yaml + submit_install_iso_firstboot_yaml + submit_install_iso_sync_rootfs_yaml +} + +main -- 2.23.0

1 0

[PATCH v2 lkp-tests 04/11] install-iso-sync-rootfs: add step: sync_rootfs
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- distro/depends/install-iso-sync-rootfs | 2 + tests/install-iso-sync-rootfs | 53 +++++++++++++++++++++++++- 2 files changed, 54 insertions(+), 1 deletion(-) create mode 100644 distro/depends/install-iso-sync-rootfs diff --git a/distro/depends/install-iso-sync-rootfs b/distro/depends/install-iso-sync-rootfs new file mode 100644 index 000000000000..2abb6b833291 --- /dev/null +++ b/distro/depends/install-iso-sync-rootfs @@ -0,0 +1,2 @@ +nfs-common +lvm2 diff --git a/tests/install-iso-sync-rootfs b/tests/install-iso-sync-rootfs index e5de4d72240e..34096229ccfb 100755 --- a/tests/install-iso-sync-rootfs +++ b/tests/install-iso-sync-rootfs @@ -7,6 +7,20 @@ set -e . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/log.sh +SKIP_SYNC_DIRS=( + "srv" + "opt" + "mnt" + "media" + "home" + "lost+found" + "dev" + "proc" + "run" + "sys" + "var/log" +) + check_params() { log_info "start check params" @@ -28,9 +42,46 @@ check_params() done } +prepare_exclude_file() +{ + local exclude_file=$1 + + rm -rf $exclude_file + for i in "${SKIP_SYNC_DIRS[@]}" + do + echo $i >> $exclude_file + done +} + sync_rootfs_to_nfs_server() { - : + # mount nfs rootfs + local nfs_rootfs_mount_point=/tmp/nfs_rootfs_${ROOTFS_SUFFIX} + + mkdir -p ${nfs_rootfs_mount_point} + mount -t nfs ${NFS_SERVER_HOST}:${NFS_ROOT_DIR}/${os}/${os_arch}/${os_version}-iso-${ROOTFS_SUFFIX} ${nfs_rootfs_mount_point} || die "mount nfs rootfs failed" + + # activate the lvm + vgchange -ay + + # mount local logical volume + local lv_rootfs_mount_point=/tmp/lv_rootfs_${ROOTFS_SUFFIX} + + mkdir -p ${lv_rootfs_mount_point} + mount /dev/mapper/${VG_NAME}-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX} ${lv_rootfs_mount_point} || die "mount lv rootfs failed" + + # sync rootfs + local exclude_file=/tmp/exclude_file_${ROOTFS_SUFFIX} + + prepare_exclude_file "$exclude_file" + rsync -az ${lv_rootfs_mount_point}/. ${nfs_rootfs_mount_point} --exclude-from=$exclude_file || die "copy out rootfs failed" + cd ${nfs_rootfs_mount_point} && mkdir -p "${SKIP_SYNC_DIRS[@]}" + + # umount nfs rootfs + umount $nfs_rootfs_mount_point || log_warn "umount failed: nfs_rootfs_mount_point" + + # umount local logical volume + umount $lv_rootfs_mount_point || log_warn "umount failed: $lv_rootfs_mount_point" } main() -- 2.23.0

1 0

[PATCH v2 lkp-tests 03/11] feat: step 3 of iso2rootfs: copy out rootfs in second boot stage
by Yu Chuan 10 Mar '21

10 Mar '21

Framework of install-iso-sync-rootfs. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/install-iso-sync-rootfs.yaml | 22 +++++++++++++++ tests/install-iso-sync-rootfs | 46 +++++++++++++++++++++++++++++++ 2 files changed, 68 insertions(+) create mode 100644 jobs/install-iso-sync-rootfs.yaml create mode 100755 tests/install-iso-sync-rootfs diff --git a/jobs/install-iso-sync-rootfs.yaml b/jobs/install-iso-sync-rootfs.yaml new file mode 100644 index 000000000000..0235b10c718b --- /dev/null +++ b/jobs/install-iso-sync-rootfs.yaml @@ -0,0 +1,22 @@ +suite: install-iso-sync-rootfs +category: benchmark +install-iso-sync-rootfs: + ################# + # nfs target path: ${NFS_SERVER_HOST}:${NFS_ROOT_DIR}/${iso_os}/${iso_arch}/${iso_version} + ################# + iso_os: + iso_arch: + iso_version: + NFS_SERVER_HOST: + NFS_ROOT_DIR: + + ################# + # result rootfs suffix, format: $(date "+%Y%m%d%H%M%S") + ################# + ROOTFS_SUFFIX: + + ################# + # name of volume group which stores the logical volume of rootfs + ################# + VG_NAME: + diff --git a/tests/install-iso-sync-rootfs b/tests/install-iso-sync-rootfs new file mode 100755 index 000000000000..e5de4d72240e --- /dev/null +++ b/tests/install-iso-sync-rootfs @@ -0,0 +1,46 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +set -e + +. $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/log.sh + +check_params() +{ + log_info "start check params" + + local required_vars=( + "iso_os" + "iso_arch" + "iso_version" + "NFS_SERVER_HOST" + "NFS_ROOT_DIR" + + "ROOTFS_SUFFIX" + "VG_NAME" + ) + + for i in "${required_vars[@]}" + do + [ -n "$(eval echo \$$i)" ] || die "cannot get value of var: $i" + done +} + +sync_rootfs_to_nfs_server() +{ + : +} + +main() +{ + check_params + + sync_rootfs_to_nfs_server + + lvchange --permission r /dev/mapper/${VG_NAME}-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX} || + die "change logical volume permission to readonly failed:/dev/mapper/${VG_NAME}-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX}" +} + +main -- 2.23.0

1 0

[PATCH v2 lkp-tests 02/11] feat: step 2 of iso2rootfs: reboot directly in firstboot stage
by Yu Chuan 10 Mar '21

10 Mar '21

[Why] Firstboot stage: the first boot after iso installation is complete. At this stage, the system will run some initial configurations. So we need copy out the rootfs in second boot stage, and do nothing in firstboot stage. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/install-iso-firstboot.yaml | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 jobs/install-iso-firstboot.yaml diff --git a/jobs/install-iso-firstboot.yaml b/jobs/install-iso-firstboot.yaml new file mode 100644 index 000000000000..7be9eaaab184 --- /dev/null +++ b/jobs/install-iso-firstboot.yaml @@ -0,0 +1,10 @@ +suite: install-iso-firstboot + +sleep: 1 + +# os_mount must be equal to 'local' +os_mount: local + +# kernel_custom_params must have 'boot_lv_suffix' and 'src_lv_suffix', +# and they must be the same. so can boot from the specify logical volume +kernel_custom_params: -- 2.23.0

1 0

[PATCH v2 lkp-tests 01/11] feat: step 1 of iso2rootfs: install iso to testbox and reboot
by Yu Chuan 10 Mar '21

10 Mar '21

[Why] When testbox request job in pxe -> ipxe step, scheduler returns a response composed of ipxe cmd. So we add 'custom_ipxe' field of job.yaml, then we can customize the response what testbox get, then testbox can auto install iso by executing the specified ipxe cmd. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/install-iso.yaml | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 jobs/install-iso.yaml diff --git a/jobs/install-iso.yaml b/jobs/install-iso.yaml new file mode 100644 index 000000000000..bcccc6b76f85 --- /dev/null +++ b/jobs/install-iso.yaml @@ -0,0 +1,3 @@ +suite: install-iso + +custom_ipxe: -- 2.23.0

1 0

[PATCH lkp-tests 11/11] install-iso-pre: add step: submit install-iso-release.yaml
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index 17a821cf3c14..4d6bae205f19 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -248,7 +248,22 @@ submit_install_iso_firstboot_yaml() submit_install_iso_release_yaml() { - : + log_info "start submit install-iso-release.yaml" + + local release_yaml="${LKP_SRC}/jobs/install-iso-release.yaml" + [ -f "${release_yaml}" ] || die "cannot find ${release_yaml}" + + ${LKP_SRC}/sbin/submit \ + ${release_yaml} \ + suite=install-iso-release-${ROOTFS_SUFFIX} \ + iso_os=${iso_os} \ + iso_arch=${iso_arch} \ + iso_version=${iso_version} \ + NFS_SERVER_HOST=${NFS_SERVER_HOST} \ + NFS_ROOT_DIR=${NFS_ROOT_DIR} \ + ROOTFS_SUFFIX=${ROOTFS_SUFFIX} \ + VG_NAME=${VG_NAME} \ + testbox=$testbox || die "submit ${release_yaml} failed" } main() -- 2.23.0

2 2

[PATCH v2 lkp-tests] add test script for dockerfile os migrating
by Bai Jing 10 Mar '21

10 Mar '21

Signed-off-by: Bai Jing <799286817(a)qq.com> --- tests/openeuler_docker | 142 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 142 insertions(+) create mode 100755 tests/openeuler_docker diff --git a/tests/openeuler_docker b/tests/openeuler_docker new file mode 100755 index 000000000..d713924bf --- /dev/null +++ b/tests/openeuler_docker @@ -0,0 +1,142 @@ +#!/bin/sh +# - dockerfile_url + +# exit 1 if openEuler failed +# exit 0 if +# - no centos support +# - centos failed + +. $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/log.sh + +[ -n "$dockerfile_url" ] || die "dockerfile_url is empty" + +cd $BENCHMARK_ROOT + +nr_openeuler_fail=0 + +fit_repo() +{ + yum reinstall -y git + git clone "$dockerfile_url" || exit 0 + file_dir=$(find ./ -type f -name "*Dockerfile*" | xargs grep -HE '^FROM centos:(centos|)[78]') + [ -n "$file_dir" ] || { + echo "current-repo-not-support-centos: 1" + exit 0 + } +} + +openeuler_ready() +{ + docker load < $BENCHMARK_ROOT/openeuler_docker/openeuler-20.03-lts-v1.tar +} + +centos_docker() +{ + flag=0 + i=1 + for file in $(find ./ -type f -name "*Dockerfile*" | xargs grep -HE '^FROM centos:(centos|)[78]' | awk -F: '{print $1}') + do + file_dir=${file%/*} + image=$(echo "$file" | awk -F '/' '{print $3}' | tr 'A-Z' 'a-z') + [ -n "$image" ] || image=${file#*/}_$i + if [ -f "$file_dir/build.sh" ] && [ -x "$file_dir/build.sh" ]; then + "$file_dir/build.sh" > /dev/null 2> /dev/null && { + handle_dockerfile + } + else + docker_build > /dev/null 2> /dev/null && { + handle_dockerfile + } + fi + let i++ + done + if [ "$nr_openeuler_fail" != 0 ]; then + echo "openeuler failed times: $nr_openeuler_fail" + echo "openeuler_valid_url: $dockerfile_url" + exit 1 + else + echo "current-repo-not-support-centos: 1" + exit 0 + fi +} + +handle_FROM() +{ + sed -i "s/^FROM .*/FROM openeuler-20.03-lts:v1/g" "$file" + +} + +handle_epel() +{ + grep -qw "epel-release" "$file" && { + sed -i "s/epel-release/bash/g" "$file" + } +} + +handle_epelrpm() +{ + + grep -qw "https://.*epel-release.*rpm" "$file" && { + sed -i "s|https://.*epel-release.*rpm|bash|g" "$file" + + } +} + +add_base_commands() +{ + # grep -qw "groupadd*" "$file" && { + # fix missing useradd, groupadd, chpasswd, etc. commands + sed -i '/FROM /a\RUN yum -y install shadow tar' "$file" +} + +handle_epel_repo() +{ + sed -i '/.* wget .*(epel|CentOS-Base)\.repo/ s|^|#|g' "$file" +} + +handle_dockerfile() +{ + flag=1 + handle_FROM + handle_epel + add_base_commands + if docker_build "-openeuler"; then + docker_run "-openeuler" + else + let nr_openeuler_fail++ + fi +} + +docker_build() +{ + docker build -f "$file" -t "${image}$1" . + [ $? != 0 ] && { + [ "$flag" == "1" ] && echo "${image}$1.build.fail: 1" + return 1 + } + [ "$flag" == "1" ] && echo "${image}$1.build.success: 1" + flag=0 +} + +docker_run() +{ + docker run -d --name "${image}" "${image}$1" + wait + docker_stats=$(docker inspect -f '{{.State.Running}}' ${image}) + [ $docker_stats == false ] && { + echo "${image}.run.fail: 1" + return 1 + } + + echo "${image}.run.success: 1" +} + +docker_test() +{ + fit_repo + openeuler_ready + centos_docker +} + +docker_test -- 2.23.0

2 1

[PATCH lkp-tests 10/11] install-iso-pre: add step: submit install-iso-firstboot.yaml
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index 45567feddd9c..17a821cf3c14 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -218,7 +218,32 @@ submit_install_iso_yaml() submit_install_iso_firstboot_yaml() { - : + log_info "start submit install-iso-firstboot.yaml" + + local firstboot_yaml="${LKP_SRC}/jobs/install-iso-firstboot.yaml" + [ -f "${firstboot_yaml}" ] || die "cannot find ${firstboot_yaml}" + + log_info " + ${LKP_SRC}/sbin/submit + ${firstboot_yaml} + suite=install-iso-firstboot-${ROOTFS_SUFFIX} + os=${iso_os} + os_arch=${iso_arch} + os_version=${iso_version} + os_mount=local + kernel_custom_params=\"use_root_partition=${ROOTFS_SUFFIX} save_root_partition=${ROOTFS_SUFFIX}\" + testbox=$testbox + " + + ${LKP_SRC}/sbin/submit \ + ${firstboot_yaml} \ + suite=install-iso-firstboot-${ROOTFS_SUFFIX} \ + os=${iso_os} \ + os_arch=${iso_arch} \ + os_version=${iso_version} \ + os_mount=local \ + kernel_custom_params="use_root_partition=${ROOTFS_SUFFIX} save_root_partition=${ROOTFS_SUFFIX}" \ + testbox=$testbox || die "submit ${firstboot_yaml} failed" } submit_install_iso_release_yaml() -- 2.23.0

2 4

[PATCH lkp-tests 09/11] install-iso-pre: add step: submit install-iso.yaml
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index d867f1441003..45567feddd9c 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -182,9 +182,38 @@ prepare_submit() EOF } +prepare_install_iso_yaml() +{ + log_info "start prepare install-iso.yaml" + + custom_ipxe="#!ipxe\\\\n" + custom_ipxe="${custom_ipxe}\\\\ndhcp" + custom_ipxe="${custom_ipxe}\\\\ninitrd tftp://${TFTP_SERVER_HOST}/EFI/BOOT/${iso_os}/${iso_arch}/${iso_version}/initrd.img" + custom_ipxe="${custom_ipxe}\\\\nkernel tftp://${TFTP_SERVER_HOST}/EFI/BOOT/${iso_os}/${iso_arch}/${iso_version}/vmlinux ro inst.geoloc=0 console=ttyS0 smmu.bypassdev=0x1000:0x17 smmu.bypassde1000:0x15 viff video=VGA-1:640x480-32@60me initrd=initrd.img ip=dhcp inst.ks=http://${HTTP_SERVER_HOST}:${HTTP_SERVER_PORT}/os/kickstart/ks/${iso_os}_${iso_arch}_${iso_version}.cfg" + custom_ipxe="${custom_ipxe}\\\\nboot\\\\n" + + log_info "custom_ipxe: $custom_ipxe" + sed -i "/custom_ipxe:/ccustom_ipxe: \"${custom_ipxe}\"" ${install_iso_yaml} +} + submit_install_iso_yaml() { - : + log_info "start submit install-iso.yaml" + + local install_iso_yaml="${LKP_SRC}/jobs/install-iso.yaml" + [ -f "${install_iso_yaml}" ] || die "cannot find ${install_iso_yaml}" + + prepare_install_iso_yaml + + log_info " + ${LKP_SRC}/sbin/submit + ${install_iso_yaml} + testbox=$testbox + " + + ${LKP_SRC}/sbin/submit \ + ${install_iso_yaml} \ + testbox=$testbox || die "submit ${install_iso_yaml} failed" } submit_install_iso_firstboot_yaml() -- 2.23.0

2 2

[PATCH lkp-tests 07/11] install-iso-pre: add step: prepare_ks
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 72 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 71 insertions(+), 1 deletion(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index 85e2a5fd42ec..e27c9ea80232 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -86,9 +86,79 @@ prepare_lv() create_lv } +generate_ks() +{ + log_info "start generate $ks_file" + + local nic=$(ip route get ${LKP_SERVER} | awk '{print $3; exit}') + + cat > ${ks_file} <<-EOF + # Use network installation + url --url="http://${HTTP_SERVER_HOST}:${HTTP_SERVER_PORT}/${HTTP_ROOT_DIR}/${iso_os}/${iso_arch}/${iso_version}/" + + # Network information + network --bootproto=dhcp --device=${nic} --ipv6=auto --activate + network --hostname=localhost.localdomain + + # Specifies a list of disks for the installation program to use. All other disks are ignored. + ignoredisk --only-use=${rootfs_disk#/dev/} + + # if install to special lvm, bootloader should be none. + bootloader --location=none + logvol / --vgname=${VG_NAME} --name=${LV_NAME} --useexisting + + # Root password + rootpw --iscrypted ${ROOTFS_ROOT_PASSWD} + + # Reboot after installation + reboot --eject + + # Use text mode install + text + + # Keyboard layouts + keyboard --vckeymap=us --xlayouts='us' + + # System language + lang en_US.UTF-8 + + # Do not configure the X Window System + skipx + + # System services + services --enabled="chronyd" + + # System timezone + timezone Asia/Shanghai --isUtc + + %packages + @^minimal-environment + + %end + + %anaconda + pwpolicy root --minlen=8 --minquality=1 --notstrict --nochanges --notempty + pwpolicy user --minlen=8 --minquality=1 --notstrict --nochanges --emptyok + pwpolicy luks --minlen=8 --minquality=1 --notstrict --nochanges --notempty + %end + EOF +} + +upload_ks() +{ + log_info "start upload ks to rsync-server: ${RSYNC_SERVER_HOST}:${RSYNC_SERVER_PORT}" + + rsync -vrtopg --port ${RSYNC_SERVER_PORT} ${ks_file} ${RSYNC_SERVER_HOST}::ks/ || die "upload ks to rsync-server failed, please check." +} + prepare_ks() { - : + log_info "start prepare kickstart file" + + local ks_file=/tmp/${iso_os}_${iso_arch}_${iso_version}.ks + + generate_ks + upload_ks } prepare_submit() -- 2.23.0

2 2

[PATCH lkp-tests 06/11] install-iso-pre: add step: prepare_lv
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- distro/depends/install-iso-pre | 1 + tests/install-iso-pre | 43 +++++++++++++++++++++++++++++++++- 2 files changed, 43 insertions(+), 1 deletion(-) create mode 100644 distro/depends/install-iso-pre diff --git a/distro/depends/install-iso-pre b/distro/depends/install-iso-pre new file mode 100644 index 000000000000..6f88eb919896 --- /dev/null +++ b/distro/depends/install-iso-pre @@ -0,0 +1 @@ +lvm2 diff --git a/tests/install-iso-pre b/tests/install-iso-pre index 4e8cbe22fcd4..85e2a5fd42ec 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -40,9 +40,50 @@ check_params() done } +create_vg() +{ + log_info "start create vg:$VG_NAME" + + local disk + + for disk in "${rootfs_disk[@]}" + do + log_info "start handle disk: $disk" + pvdisplay ${disk} > /dev/null || pvcreate ${disk} || die "create pv failed: $disk" + + if vgdisplay ${VG_NAME} > /dev/null; then + vgextend ${VG_NAME} ${disk} + else + vgcreate ${VG_NAME} ${disk} + fi + done + +} + +create_lv() +{ + log_info "start create lv:$LV_NAME" + + lvcreate -y -L 10G -n ${LV_NAME} ${VG_NAME} || die "create logical volume failed." +} + prepare_lv() { - : + log_info "start prepare logical volume" + + # VG_NAME defines in job.yaml + export VG_NAME + export LV_NAME="${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX}" + + # lv existed + lvdisplay /dev/mapper/${VG_NAME}-${LV_NAME} > /dev/null && { + log_info "iso already install to this testbox, so rootfs maybe already generated ok. please check it." + exit 0 + } + + # lv not existed + vgdisplay ${VG_NAME} > /dev/null || create_vg + create_lv } prepare_ks() -- 2.23.0

2 2

[PATCH lkp-tests 04/11] install-iso-release: add step: release_rootfs
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- distro/depends/install-iso-release | 2 ++ tests/install-iso-release | 53 +++++++++++++++++++++++++++++- 2 files changed, 54 insertions(+), 1 deletion(-) create mode 100644 distro/depends/install-iso-release diff --git a/distro/depends/install-iso-release b/distro/depends/install-iso-release new file mode 100644 index 000000000000..2abb6b833291 --- /dev/null +++ b/distro/depends/install-iso-release @@ -0,0 +1,2 @@ +nfs-common +lvm2 diff --git a/tests/install-iso-release b/tests/install-iso-release index 23e8fbff8e4f..8c925ee2bcd8 100755 --- a/tests/install-iso-release +++ b/tests/install-iso-release @@ -7,6 +7,20 @@ set -e . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/log.sh +SKIP_RELEASE_DIRS=( + "srv" + "opt" + "mnt" + "media" + "home" + "lost+found" + "dev" + "proc" + "run" + "sys" + "var/log" +) + check_params() { log_info "start check params" @@ -29,9 +43,46 @@ check_params() done } +prepare_exclude_file() +{ + local exclude_file=$1 + + rm -rf $exclude_file + for i in "${SKIP_RELEASE_DIRS[@]}" + do + echo $i >> $exclude_file + done +} + release_rootfs() { - : + # mount nfs rootfs + local nfs_rootfs_mount_point=/tmp/nfs_rootfs_${ROOTFS_SUFFIX} + + mkdir -p ${nfs_rootfs_mount_point} + mount -t nfs ${NFS_SERVER_HOST}:${NFS_ROOT_DIR}/${os}/${os_arch}/${os_version}-iso-${ROOTFS_SUFFIX} ${nfs_rootfs_mount_point} || die "mount nfs rootfs failed" + + # activate the lvm + vgchange -ay + + # mount local logical volume + local lv_rootfs_mount_point=/tmp/lv_rootfs_${ROOTFS_SUFFIX} + + mkdir -p ${lv_rootfs_mount_point} + mount /dev/mapper/${VG_NAME}-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX} ${lv_rootfs_mount_point} || die "mount lv rootfs failed" + + # release rootfs + local exclude_file=/tmp/exclude_file_${ROOTFS_SUFFIX} + + prepare_exclude_file "$exclude_file" + rsync -az ${lv_rootfs_mount_point}/. ${nfs_rootfs_mount_point} --exclude-from=$exclude_file || die "copy out rootfs failed" + cd ${nfs_rootfs_mount_point} && mkdir -p "${SKIP_RELEASE_DIRS[@]}" + + # umount nfs rootfs + umount $nfs_rootfs_mount_point || log_warn "umount failed: nfs_rootfs_mount_point" + + # umount local logical volume + umount $lv_rootfs_mount_point || log_warn "umount failed: $lv_rootfs_mount_point" } main() -- 2.23.0

2 2

[PATCH compass-ci] lib/lifecycle.cr: optimize the restart queue
by Wu Zhende 10 Mar '21

10 Mar '21

1.delete machine's history don't need this info. 2.add testbox this field is used to find the corresponding host when a phssical machine is restarted. 3.modify mq queue testbox: vm-2p8g.wuzhende-1 before queue: wuzhende now queue: vm-2p8g.wuzhende Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index 3dab6e8..1eb6264 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -110,7 +110,7 @@ class Lifecycle def mq_event_loop puts "deal job events" - q = @mq.ch.queue("job_mq") + q = @mq.ch.queue("job_mq", durable: false) q.subscribe(no_ack: false) do |msg| event = JSON.parse(msg.body_io.to_s) job_state = event["job_state"]? @@ -311,6 +311,8 @@ class Lifecycle return if Time.local < deadline mq_queue = get_machine_reboot_queue(testbox) + machine.as_h.delete("history") + machine.as_h["testbox"] = JSON::Any.new(testbox) @mq.pushlish_confirm(mq_queue, machine.to_json, durable: true) machine["state"] = "rebooting_queue" @@ -320,7 +322,7 @@ class Lifecycle def get_machine_reboot_queue(testbox) if testbox.includes?(".") - testbox =~ /.*\.(.*)-\d+$/ + testbox =~ /(.*)-\d+$/ else testbox =~ /(.*)--.*/ end -- 2.23.0

1 0

[PATCH lkp-tests 03/11] feat: step 3 of iso2rootfs: copy out rootfs in second boot stage
by Yu Chuan 10 Mar '21

10 Mar '21

Framework of install-iso-release. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/install-iso-release.yaml | 26 +++++++++++++++++++ tests/install-iso-release | 47 +++++++++++++++++++++++++++++++++++ 2 files changed, 73 insertions(+) create mode 100644 jobs/install-iso-release.yaml create mode 100755 tests/install-iso-release diff --git a/jobs/install-iso-release.yaml b/jobs/install-iso-release.yaml new file mode 100644 index 000000000000..fabb2cd50902 --- /dev/null +++ b/jobs/install-iso-release.yaml @@ -0,0 +1,26 @@ +suite: install-iso-release +category: benchmark +install-iso-release: + ################# + # iso info + ################# + iso_os: + iso_arch: + iso_version: + + ################# + # nfs server info, used for store rootfs + ################# + NFS_SERVER_HOST: + NFS_ROOT_DIR: + + ################# + # result rootfs suffix, format: $(date "+%Y%m%d%H%M%S") + ################# + ROOTFS_SUFFIX: + + ################# + # name of volume group which stores the logical volume of rootfs + ################# + VG_NAME: + diff --git a/tests/install-iso-release b/tests/install-iso-release new file mode 100755 index 000000000000..23e8fbff8e4f --- /dev/null +++ b/tests/install-iso-release @@ -0,0 +1,47 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +set -e + +. $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/log.sh + +check_params() +{ + log_info "start check params" + + local required_vars=( + "iso_os" + "iso_arch" + "iso_version" + + "NFS_SERVER_HOST" + "NFS_ROOT_DIR" + + "ROOTFS_SUFFIX" + "VG_NAME" + ) + + for i in "${required_vars[@]}" + do + [ -n "$(eval echo \$$i)" ] || die "cannot get value of var: $i" + done +} + +release_rootfs() +{ + : +} + +main() +{ + check_params + + release_rootfs + + lvchange --permission r /dev/mapper/${VG_NAME}-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX} || + die "change logical volume permission to readonly failed:/dev/mapper/${VG_NAME}-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX}" +} + +main -- 2.23.0

2 2

[PATCH compass-ci] docker/run.sh: change hostname
by Wang Yong 10 Mar '21

10 Mar '21

[before] Build Host : dc-8g [after] Build Host : dc-8g.compass-ci.net this will change Build Host in RPM build Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- providers/docker/run.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 2810641..2dfe36d 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -23,7 +23,7 @@ busybox_path=$(command -v busybox) cmd=( docker run --rm - --hostname $host + --hostname $host.compass-ci.net -m $memory --tmpfs /tmp:rw,exec,nosuid,nodev -e CCI_SRC=/c/compass-ci -- 2.23.0

1 0