- Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci 1/2] scheduler/find_job_boot.cr: docker add job_id
by Wu Zhende 12 Mar '21

12 Mar '21

When the startup parameter is docker, the value of job_id is returned. Set job_id as docker name, so we can use "docker rm -f #{job_id}" to kill the docker. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/find_job_boot.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ce1a2f8..a45a6d9 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -144,6 +144,7 @@ class Sched private def get_boot_container(job : Job) response = Hash(String, String).new + response["job_id"] = job.id.to_s response["docker_image"] = "#{job.docker_image}" response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") -- 2.23.0

1 0

[PATCH v2 compass-ci] scheduler/find_job_boot.cr: docker add job_id
by Wu Zhende 12 Mar '21

12 Mar '21

When the startup parameter is docker, the value of job_id is returned. Set job_id as docker name, so we can use "docker rm -f #{job_id}" to kill the docker. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/find_job_boot.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ce1a2f8..a45a6d9 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -144,6 +144,7 @@ class Sched private def get_boot_container(job : Job) response = Hash(String, String).new + response["job_id"] = job.id.to_s response["docker_image"] = "#{job.docker_image}" response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") -- 2.23.0

1 0

[PATCH compass-ci] extract-stats: wait_for refresh doc to ES DB
by Lu Weitao 12 Mar '21

12 Mar '21

[Why] We need update doc(job) to ES completed, then send delimiter task to task-queue Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index dbb3fe2..44c327d 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -81,6 +81,7 @@ class StatsWorker { :index => "jobs", :type => "_doc", :id => job_id, + :refresh => "wait_for", :body => {:doc => update_content}, } ) -- 2.23.0

1 0

[PATCH compass-ci] scheduler/find_job_boot.cr: docker add job_id
by Wu Zhende 12 Mar '21

12 Mar '21

When the startup parameter is docker, the value of job_id is returned. Set job_id as docker name, so we can use "docker rm -f #{job_id}" to kill the docker. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/find_job_boot.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ce1a2f8..a45a6d9 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -144,6 +144,7 @@ class Sched private def get_boot_container(job : Job) response = Hash(String, String).new + response["job_id"] = job.id.to_s response["docker_image"] = "#{job.docker_image}" response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code/dev-env: fix failed to submit job
by Liu Yinsi 12 Mar '21

12 Mar '21

[error] root@taishan200-2280-2s64p-256g--a123 /c/compass-ci/container# submit host-info.yaml submit_id=9bca6036-4602-44ab-92df-729058ac1818 Traceback (most recent call last): 5: from /c/lkp-tests/sbin/submit:136:in `<main>' 4: from /c/lkp-tests/sbin/submit:136:in `each' 3: from /c/lkp-tests/sbin/submit:150:in `block in <main>' 2: from /c/lkp-tests/lib/job.rb:744:in `each_jobs' 1: from /c/lkp-tests/lib/job.rb:736:in `each_job' /c/lkp-tests/sbin/submit:151:in `block (2 levels) in <main>': Please configure SCHED_HOST (RuntimeError) [why] for user locally deploy compass-ci, env $SCHED_HOST not exists in /etc/compass-ci/defaults/$server_name.yaml, cause failed to submit, so add it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 403ccb1..fdf7894 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -36,6 +36,8 @@ server_name=$(hostname | cut -f1 -d.) mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF +SCHED_HOST: $sched_host +SCHED_PORT: $sched_port lab: nolab EOF -- 2.23.0

1 0

[PATCH lkp-tests] tests/rpmbuild-pkg: change rpmbuild-pkg upload_dir
by Li Ping 12 Mar '21

12 Mar '21

--- tests/rpmbuild-pkg | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index b641e97d1..a8c8eb6b1 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -6,7 +6,7 @@ [ -n "$rpm_repo" ] || die "rpm_repo is empty" package_name=${rpm_repo##*/} -sync_dest="/initrd/rpmbuild-pkg/${os}/${os_arch}/${os_version}/${package_name}" +sync_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${os_arch}/${package_name}" init_workspace() { @@ -24,7 +24,7 @@ download_rpm_repo() { local filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') - for pkgfile in "${filelist[@]}" + for pkgfile in ${filelist[@]} do local dir="SOURCES" echo $pkgfile | egrep "\.spec$" && dir="SPECS" -- 2.23.0

1 0

[PATCH v2 compass-ci 3/3] doc/job/os_mount.md: change the fields of job.yaml
by Xu Xijian 11 Mar '21

11 Mar '21

The fields that users need to give is the suffix of partitions, so give them appropriate names. use_root_partition => src_lv_suffix save_root_partition => boot_lv_suffix Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/os_mount.md | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index 6c1637b..eab95ed 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -56,23 +56,25 @@ The brief flow is as follows: ## persistent rootfs data -When you need to persist the rootfs data of a job, and use it in the subsequent job(s), two fields in `kernel_custom_params` will help you: `save_root_partition`, `use_root_partition`. +When you need to persist the rootfs data of a job, and use it in the subsequent job(s), two fields in `kernel_custom_params` will help you: 'src_lv_suffix', 'boot_lv_suffix'. The brief flow is as follows: ``` 1. boot and request scheduler for job. 2. initrd stage: + use_root_partition seems like /dev/mapper/os-${os}_${os_arch}_${os_version}_${src_lv_suffix} + save_root_partition seems like /dev/mapper/os-${os}_${os_arch}_${os_version}_${boot_lv_suffix} - firstly, we need two logical volume: ${src_lv} -- src logical volume: - - if have ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${use_root_partition} + - if have ${use_root_partition}, src_lv=${use_root_partition} - if no ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} ${boot_lv} -- boot logical volume: - - if have ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${save_root_partition} + - if have ${save_root_partition}, boot_lv=${save_root_partition} - if no ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} - if ${src_lv} not exists: - - if have ${use_root_partition}, exit 1. - - if no ${use_root_partition}, create ${src_lv}, and rsync the rootfs from cluster nfs server. + - if have ${use_root_partition}, reboot. + - if no ${use_root_partition}, create ${src_lv}, and sync the rootfs from cluster nfs server. - if ${boot_lv} != ${src_lv}: - if ${boot_lv} exists: delete it - create ${boot_lv} as the snapshot of ${src_lv}. @@ -87,12 +89,12 @@ Demo usage: rootfs data of job-20210218.yaml so that it can be used by the subsequent jobs. Then you need add the follow field in your job-20210218.yaml: - kernel_custom_params: save_root_partition=zhangsan_local_for_iperf_20210218 + kernel_custom_params: boot_lv_suffix=zhangsan_local_for_iperf_20210218 - in 20210219, you submit a job-20210219.yaml, and you want to use the rootfs data of job-20210218.yaml. Then you need add the follow field in your job-20210219.yaml: - kernel_custom_params: use_root_partition=zhangsan_local_for_iperf_20210218 + kernel_custom_params: src_lv_suffix=zhangsan_local_for_iperf_20210218 ``` Notes: @@ -109,16 +111,19 @@ Notes: os_arch: aarch64 os_version: 20.03 os_mount: local - kernel_custom_params: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 + kernel_custom_params: src_lv_suffix=zhangsan_local_for_iperf_20210218 boot_lv_suffix=zhangsan_local_for_iperf_20210219 ``` 2. scheduler return the custom_ipxe to testbox + os_info = "#{os}_#{os_arch}_#{os_version}" + use_root_partition = "/dev/mapper/os-#{os_info}_#{src_lv_suffix}" + save_root_partition = "/dev/mapper/os-#{os_info}_#{boot_lv_suffix}" ``` #!ipxe dhcp initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/initrd.lkp - kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src=${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} initrd=initrd.lkp ${kernel_custom_params} + kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 initrd=initrd.lkp local use_root_partition=#{use_root_partition} save_root_partition=#{save_root_partition} rw ... boot ``` -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] container/dracut-initrd: change someof the handle logic according to scheme doc
by Xu Xijian 11 Mar '21

11 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- .../dracut-initrd/bin/set-local-sysroot.sh | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh index dc9000b..240551a 100644 --- a/container/dracut-initrd/bin/set-local-sysroot.sh +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -5,9 +5,9 @@ analyse_kernel_cmdline_params() { # example: $nfs_server_ip:/os/${os}/${os_arch}/${os_version}-snapshots/20210310005959 rootfs_src=$(echo $"$rootfs" | sed 's/\///') timestamp="$(basename $root)" - os="$(echo $root_src | awk -F '/|-' '{print $2}')" - os_arch="$(echo $root_src | awk -F '/|-' '{print $3}')" - os_version="$(echo $root_src | awk -F '/|-' '{print $4}')" + os="$(echo $rootfs_src | awk -F '/|-' '{print $2}')" + os_arch="$(echo $rootfs_src | awk -F '/|-' '{print $3}')" + os_version="$(echo $rootfs_src | awk -F '/|-' '{print $4}')" os_info="${os}_${os_arch}_${os_version}" export rootfs_src timestamp os_info } @@ -58,22 +58,22 @@ analyse_kernel_cmdline_params sed -i "s/^locking_type = .*/locking_type = 1/" /etc/lvm/lvm.conf use_root_partition="$(getarg use_root_partition=)" -if [ -n "$use_root_partition" ]; then - src_lv="$use_root_partition" # /dev/mapper/os-${os_info}_${timestamp} +if [ -z "$use_root_partition" ]; then + src_lv="/dev/mapper/os-${os_info}_$timestamp" sync_src_lv "$src_lv" else - src_lv="/dev/mapper/os-${os_info}_$timestamp" - [ -e "src_lv" ] || { + src_lv="$use_root_partition" + [ -e "$src_lv" ] || { echo "warn dracut: FATAL: no src_lv with local mount, reboot" reboot } fi save_root_partition="$(getarg save_root_partition=)" -if [ -n "$save_root_partition" ]; then - boot_lv="$save_root_partition" -else +if [ -z "$save_root_partition" ]; then boot_lv="/dev/mapper/os-${os_info}" +else + boot_lv="$save_root_partition" fi snapshot_boot_lv "$src_lv" "$boot_lv" -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] container/dracut-initrd: add basename into openeuler's 21xx+ kernel
by Xu Xijian 11 Mar '21

11 Mar '21

[why] It seems openeuler 21xx+ kernel not support cmd basename, our overlay need this cmd, so add basename into dracut module to enable initrd. [errlog] //lib/dracut/hooks/pre-pivot/10-overlay-mount.sh: line 50: basename: command not found //lib/dracut/hooks/pre-pivot/10-overlay-mount.sh: line 65: basename: command not found Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 45e895c..494588a 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -17,7 +17,7 @@ cp -a /usr/local/bin/cifs-lib.sh /usr/lib/dracut/modules.d/95cifs/ cat overlay-lkp.sh >> /usr/lib/dracut/modules.d/90overlay-root/overlay-mount.sh sed -i "/install() {/a\ inst /usr/bin/awk" /usr/lib/dracut/modules.d/40network/module-setup.sh -sed -i "/install() {/a\ inst /sbin/mke2fs" /usr/lib/dracut/modules.d/98dracut-systemd/module-setup.sh +sed -i "/install() {/a\ inst /sbin/mke2fs /usr/bin/basename" /usr/lib/dracut/modules.d/98dracut-systemd/module-setup.sh pre_mount_file="/usr/lib/dracut/modules.d/98dracut-systemd/dracut-pre-mount.sh" [ "$(sed -n '$p' $pre_mount_file)" = "exit 0" ] && sed -i '$d' "$pre_mount_file" cat set-local-sysroot.sh >> "$pre_mount_file" -- 2.23.0

1 0

[PATCH lkp-tests] add multi-qemu-x86.yaml
by Zhang Dewan 11 Mar '21

11 Mar '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/multi-qemu-x86.yaml | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 jobs/multi-qemu-x86.yaml diff --git a/jobs/multi-qemu-x86.yaml b/jobs/multi-qemu-x86.yaml new file mode 100644 index 00000000..5539c1ad --- /dev/null +++ b/jobs/multi-qemu-x86.yaml @@ -0,0 +1,23 @@ +suite: multi-qemu +category: functional +simplify-ci: +runtime: 100d +multi-qemu-0: + nr_vm: 5 + tbox_group: vm-2p1g-x86 +multi-qemu-1: + nr_vm: 5 + tbox_group: vm-2p2g-x86 +multi-qemu-2: + nr_vm: 5 + tbox_group: vm-2p4g-x86 +multi-qemu-3: + nr_vm: 10 + tbox_group: vm-2p8g-x86 +multi-qemu-4: + nr_vm: 5 + tbox_group: vm-2p16g-x86 +multi-qemu-5: + nr_vm: 1 + tbox_group: vm-2p32g-x86 +sleep: -- 2.23.0

1 0