January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH v4 compass-ci 1/4] scheduler: divide field 'kernel_params' into two fields
by Ren Wen 11 Jan '21

11 Jan '21

Original: 'kernel_params' is composed of common kernel params and kernel params for boot.ipxe. Now divide this field into 'kernel_params' and 'ipxe_kernel_params'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 11 +++-------- src/scheduler/kernel_params.cr | 9 ++++----- 2 files changed, 7 insertions(+), 13 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 218e2e1..260b1ea 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -100,6 +100,7 @@ class Job user_lkp_src kernel_uri kernel_params + ipxe_kernel_params docker_image kernel_version linux_vmlinuz_path @@ -488,14 +489,8 @@ class Job return temp_initrds end - private def initrds_basename - basenames = "" - - get_initrds().each do |initrd| - basenames += "initrd=#{File.basename(initrd)} " - end - - return basenames + private def initrds_basename : Array(String) + return os_mount == "container" ? [] of String : get_initrds.map { |initrd| "initrd=#{File.basename(initrd)}" } end private def set_initrds_uri diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index 76606aa..dea9215 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -2,7 +2,6 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Job - private def kernel_common_params return "user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" end @@ -15,10 +14,10 @@ class Job os_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}") fs2root = { - "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path} #{initrds_basename()}", + "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path}", "cifs" => "root=cifs://#{OS_HTTP_HOST}#{os_real_path}" + - ",guest,ro,hard,vers=1.0,noacl,nouser_xattr #{initrds_basename()}", - "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0 #{initrds_basename()}", + ",guest,ro,hard,vers=1.0,noacl,nouser_xattr", + "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0", "container" => "", } @@ -33,6 +32,6 @@ class Job kernel_params_values = "#{kernel_common_params()} #{kernel_custom_params()} #{self.kernel_append_root} #{kernel_console()}" kernel_params_values = kernel_params_values.split(" ").map(&.strip()).reject!(&.empty?) @hash["kernel_params"] = JSON.parse(kernel_params_values.to_json) + @hash["ipxe_kernel_params"] = JSON.parse(initrds_basename.to_json) end - end -- 2.23.0

1 1

[PATCH v4 compass-ci 4/4] scheduler: a new boot type - libvirt
by Ren Wen 11 Jan '21

11 Jan '21

Requested by: curl 'http://<SCHED_HOST>:<SCHED_PORT>/boot.libvirt/mac/<mac>' Response some job fields in json format: - job_id - kernel_uri - kernel_params - initrds_uri - result_root - LKP_SERVER - templates (provided by libvirt) - RESULT_WEBDAV_PORT (for file upload/download) Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 5e6044a..625c74c 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -7,7 +7,7 @@ class Sched boot_type = @env.params.url["boot_type"] case boot_type - when "ipxe" + when "ipxe", "libvirt" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) when "grub" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) @@ -170,6 +170,22 @@ class Sched return response end + private def get_boot_libvirt(job : Job) + _kernel_params = job["kernel_params"]? + _kernel_params = _kernel_params.as_a.map(&.to_s).join(" ") if _kernel_params + + return { + "job_id" => job.id, + "kernel_uri" => job.kernel_uri, + "initrds_uri" => job["initrds_uri"]?, + "kernel_params" => _kernel_params, + "result_root" => job.result_root, + "LKP_SERVER" => job["LKP_SERVER"]?, + "templates" => job["templates"]?, + "RESULT_WEBDAV_PORT" => job["RESULT_WEBDAV_PORT"]? || "3080", + }.to_json + end + def set_id2upload_dirs(job) @redis.hash_set("sched/id2upload_dirs", job.id, job.upload_dirs) end @@ -184,6 +200,8 @@ class Sched return job ? get_boot_grub(job) : grub_msg("No job now") when "container" return job ? get_boot_container(job) : Hash(String, String).new.to_json + when "libvirt" + return job ? get_boot_libvirt(job) : {"job_id" => ""}.to_json else raise "Not defined boot type #{boot_type}" end -- 2.23.0

1 0

[PATCH v3 compass-ci] scheduler: a new boot type - libvirt
by Ren Wen 11 Jan '21

11 Jan '21

Requested by: curl 'http://<SCHED_HOST>:<SCHED_PORT>/boot.libvirt/mac/<mac>' Response some job fields in json format: - job_id - kernel_uri - kernel_params - initrds_uri - result_root - LKP_SERVER - templates (provided by libvirt) Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 5e6044a..625c74c 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -7,7 +7,7 @@ class Sched boot_type = @env.params.url["boot_type"] case boot_type - when "ipxe" + when "ipxe", "libvirt" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) when "grub" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) @@ -170,6 +170,22 @@ class Sched return response end + private def get_boot_libvirt(job : Job) + _kernel_params = job["kernel_params"]? + _kernel_params = _kernel_params.as_a.map(&.to_s).join(" ") if _kernel_params + + return { + "job_id" => job.id, + "kernel_uri" => job.kernel_uri, + "initrds_uri" => job["initrds_uri"]?, + "kernel_params" => _kernel_params, + "result_root" => job.result_root, + "LKP_SERVER" => job["LKP_SERVER"]?, + "templates" => job["templates"]?, + "RESULT_WEBDAV_PORT" => job["RESULT_WEBDAV_PORT"]? || "3080", + }.to_json + end + def set_id2upload_dirs(job) @redis.hash_set("sched/id2upload_dirs", job.id, job.upload_dirs) end @@ -184,6 +200,8 @@ class Sched return job ? get_boot_grub(job) : grub_msg("No job now") when "container" return job ? get_boot_container(job) : Hash(String, String).new.to_json + when "libvirt" + return job ? get_boot_libvirt(job) : {"job_id" => ""}.to_json else raise "Not defined boot type #{boot_type}" end -- 2.23.0

1 1

[PATCH v2 compass-ci] scheduler: a new boot type - libvirt
by Ren Wen 11 Jan '21

11 Jan '21

Requested by: curl 'http://<SCHED_HOST>:<SCHED_PORT>/boot.libvirt/mac/<mac>' Response some job fields in json format: - job_id - kernel_uri - kernel_params - initrds_uri - result_root - LKP_SERVER - templates (provided by libvirt) Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 5e6044a..6130246 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -7,7 +7,7 @@ class Sched boot_type = @env.params.url["boot_type"] case boot_type - when "ipxe" + when "ipxe", "libvirt" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) when "grub" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) @@ -170,6 +170,21 @@ class Sched return response end + private def get_boot_libvirt(job : Job) + _kernel_params = job["kernel_params"]? + _kernel_params = _kernel_params.as_a.map(&.to_s).join(" ") if _kernel_params + + return { + "job_id" => job.id, + "kernel_uri" => job.kernel_uri, + "initrds_uri" => job["initrds_uri"]?, + "kernel_params" => _kernel_params, + "result_root" => job.result_root, + "LKP_SERVER" => job["LKP_SERVER"]?, + "templates" => job["templates"]?, + }.to_json + end + def set_id2upload_dirs(job) @redis.hash_set("sched/id2upload_dirs", job.id, job.upload_dirs) end @@ -184,6 +199,8 @@ class Sched return job ? get_boot_grub(job) : grub_msg("No job now") when "container" return job ? get_boot_container(job) : Hash(String, String).new.to_json + when "libvirt" + return job ? get_boot_libvirt(job) : {"job_id" => ""}.to_json else raise "Not defined boot type #{boot_type}" end -- 2.23.0

1 1

[PATCH compass-ci] fix: change tool's name to docker2rootfs
by Wang Chenglong 11 Jan '21

11 Jan '21

[Why] Change tool's name to docker2rootfs, like iso2rootfs. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/{docker-rootfs => docker2rootfs}/README.md | 0 container/{docker-rootfs => docker2rootfs}/common | 0 container/{docker-rootfs => docker2rootfs}/packages-to-install | 0 container/{docker-rootfs => docker2rootfs}/run | 0 container/{docker-rootfs => docker2rootfs}/setup-image | 0 5 files changed, 0 insertions(+), 0 deletions(-) rename container/{docker-rootfs => docker2rootfs}/README.md (100%) rename container/{docker-rootfs => docker2rootfs}/common (100%) rename container/{docker-rootfs => docker2rootfs}/packages-to-install (100%) rename container/{docker-rootfs => docker2rootfs}/run (100%) rename container/{docker-rootfs => docker2rootfs}/setup-image (100%) diff --git a/container/docker-rootfs/README.md b/container/docker2rootfs/README.md similarity index 100% rename from container/docker-rootfs/README.md rename to container/docker2rootfs/README.md diff --git a/container/docker-rootfs/common b/container/docker2rootfs/common similarity index 100% rename from container/docker-rootfs/common rename to container/docker2rootfs/common diff --git a/container/docker-rootfs/packages-to-install b/container/docker2rootfs/packages-to-install similarity index 100% rename from container/docker-rootfs/packages-to-install rename to container/docker2rootfs/packages-to-install diff --git a/container/docker-rootfs/run b/container/docker2rootfs/run similarity index 100% rename from container/docker-rootfs/run rename to container/docker2rootfs/run diff --git a/container/docker-rootfs/setup-image b/container/docker2rootfs/setup-image similarity index 100% rename from container/docker-rootfs/setup-image rename to container/docker2rootfs/setup-image -- 2.23.0

1 0

[PATCH lkp-tests] tests/*: reduce parameters
by Wang Yong 11 Jan '21

11 Jan '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/cci-depends | 5 ++++- tests/cci-makepkg | 5 ++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/tests/cci-depends b/tests/cci-depends index 8baf023f3..8b1c73194 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -1,9 +1,12 @@ #!/bin/bash # - benchmark -# - os_mount + +# We'll use the below global env vars, but not treat them as parameter to avoid +# duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ # - os # - os_arch # - os_version +# - os_mount . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/misc-base.sh diff --git a/tests/cci-makepkg b/tests/cci-makepkg index e737955b9..5af04b203 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -1,9 +1,12 @@ #!/bin/bash # - benchmark -# - os_mount + +# We'll use the below global env vars, but not treat them as parameter to avoid +# duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ # - os # - os_arch # - os_version +# - os_mount ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, -- 2.23.0

1 0

[PATCH compass-ci 2/2] sbin/auto_submit.yaml: update repos for build-pkg jobs
by Liu Shaofei 11 Jan '21

11 Jan '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit.yaml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index 22172eb..3ad132e 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -19,5 +19,7 @@ l/linux/archlinux-kernel: - testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml - testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml archlinux: -- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-pkg.yaml +- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=cifs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=dc-8g os=openeuler os_version=20.03-pre os_mount=container os_arch=aarch64 docker_image=openeuler:20.03-pre queue=dc-build-pkg build-pkg.yaml +- testbox=dc-8g os=archlinux os_version=2021-01-06-pre os_mount=container os_arch=aarch64 docker_image=archlinux:2021-01-06-pre queue=dc-build-pkg build-pkg.yaml -- 2.23.0

1 0

[PATCH compass-ci 1/2] sbin/auto_submit: only submit jobs of master branch for repos
by Liu Shaofei 11 Jan '21

11 Jan '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 51ecc6a..02140d7 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -31,7 +31,10 @@ class AutoSubmit def submit(newrefs_info, submit_argv) newrefs_info['new_refs']['heads'].each do |branch, commit_id| real_argvs = Array.new(submit_argv) - real_argvs.push("upstream_branch=#{branch.delete_prefix('refs/heads/')}") + branch_master = branch.delete_prefix('refs/heads/') + next unless branch_master == 'master' + + real_argvs.push("upstream_branch=#{branch_master}") real_argvs.push("upstream_commit=#{commit_id}") tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{commit_id}) real_argvs.push("upstream_tag=#{tag}") unless tag.empty? -- 2.23.0

1 0

[PATCH compass-ci] container/defconfig: optimize
by Luan Shengde 11 Jan '21

11 Jan '21

[why] when submitting jobs, the job will read date from /etc/compass-ci/defaults/*.yaml. the read date is open for user, but there are some sensitive data in the yaml files, like email address, email auth code, my_token .etc. [how] move the sensitive to yaml file in another directory: /etc/compass-ci/account/*.yaml Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/defconfig.rb | 1 + container/defconfig.sh | 7 ++++++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/container/defconfig.rb b/container/defconfig.rb index d1e2af0..fd13312 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -8,6 +8,7 @@ require 'yaml' def cci_defaults hash = {} Dir.glob(['/etc/compass-ci/defaults/*.yaml', + '/etc/compass-ci/accounts/*.yaml', "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| hash.update YAML.load_file(file) || {} end diff --git a/container/defconfig.sh b/container/defconfig.sh index 410ed75..cec6b2b 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -8,7 +8,12 @@ load_cci_defaults() { shopt -s nullglob - for i in /etc/compass-ci/defaults/*.yaml $HOME/.config/compass-ci/defaults/*.yaml + yaml_file=( + /etc/compass-ci/defaults/*.yaml + /etc/compass-ci/account/*.yaml + "$HOME"/.config/compass-ci/defaults/*.yaml + ) + for i in "${yaml_file[@]}" do create_yaml_variables "$i" done -- 2.23.0

1 0

[PATCH lkp-tests] tests/host-info: computing vm memory
by Wei Jihui 11 Jan '21

11 Jan '21

[why] it can not compute vm memory when vm memory less than 16G. [how] keep the round-up feature when memory bigger then 16G, and get actual memory when memory less then 16G. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/host-info | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/tests/host-info b/tests/host-info index fe55d1ff..86b49cf9 100755 --- a/tests/host-info +++ b/tests/host-info @@ -5,7 +5,12 @@ get_memory() { memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024/16}') - memory=$(echo `expr $(echo $memory_new | awk '{print int($0)+1}') \* 16`)G + if [ `expr $memory_new \> 16` -eq 1 ]; then + memory=$(echo `expr $(echo $memory_new | awk '{print int($0)+1}') \* 16`)G + else + memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') + memory=$(echo $memory_new | awk '{print int($0)+1}')G + fi echo "memory: $memory" } -- 2.23.0

1 0