- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 1/4] assign-account: update Dockerfile
by Luan Shengde 26 Dec '20

26 Dec '20

[why] no need to install command 'curl' in the container Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/assign-account/Dockerfile b/container/assign-account/Dockerfile index 12faafa..1d9c36c 100644 --- a/container/assign-account/Dockerfile +++ b/container/assign-account/Dockerfile @@ -8,7 +8,7 @@ ENV DEBIAN_FRONTEND noninteractive COPY conf/sources.list* /etc/apt/ RUN apt-get update && \ - apt-get install -y ssh ruby-sinatra ruby-json curl && \ + apt-get install -y ssh ruby-sinatra ruby-json && \ ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ echo 'Asia/Shanghai' > /etc/timezone COPY assign-account.rb /usr/local/bin/ -- 2.23.0

1 0

[PATCH v3 lkp-tests] sbin/run_job: Automatic task execution
by Hu Xuejiao 26 Dec '20

26 Dec '20

[purpose] Verify that a local job can be consumed properly Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100755 sbin/run_job diff --git a/sbin/run_job b/sbin/run_job new file mode 100755 index 00000000..c6cf9e85 --- /dev/null +++ b/sbin/run_job @@ -0,0 +1,73 @@ +#!/bin/bash + +cci_makepkg_file="$LKP_SRC/jobs/cci-makepkg.yaml" +grep "atomic" $cci_makepkg_file || sed -i 's|benchmark:|& atomic|g' $cci_makepkg_file + +submit_one_job() +{ + cmd=( + submit -m $yaml + os=${single_arg[0]} + os_arch=${single_arg[1]} + os_version=${single_arg[2]} + os_mount=${single_arg[3]} + testbox=$testbox + queue="${tbox_group}.${HOSTNAME}" + ) + output=$("${cmd[@]}") + echo "$output" | grep -o -E 'job_state=finished[^\"]*' +} + +submit_job() +{ + yaml_args=("$cci_makepkg_file" "host-info.yaml") + for yaml in "${yaml_args[@]}" + do + submit_one_job & + done +} + +run_qemu() +{ + local nr_vm=4 + local testbox="vm-2p8g" + local tbox_group="vm-2p8g" + local hostname="${tbox_group}.${HOSTNAME}" + local queues="${tbox_group}.${HOSTNAME}" + + $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" + + qemu_args=( + "openeuler aarch64 20.03 cifs" + "centos aarch64 7.6.1810 initramfs" + ) + for qemu_arg in "${qemu_args[@]}" + do + local single_arg=($(echo $qemu_arg)) + submit_job + done +} + +run_docker() +{ + local nr_container=2 + local tbox_group="dc-8g" + local testbox="dc-8g" + local hostname="${tbox_group}.${HOSTNAME}" + local queues="${tbox_group}.${HOSTNAME}" + + $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues + + docker_args=( + "centos aarch64 7.6.1810 container" + ) + for docker_arg in "${docker_args[@]}" + do + local single_arg=($(echo $docker_arg)) + submit_job + done +} + +run_qemu +run_docker +wait -- 2.23.0

1 0

[PATCH v3 compass-ci] lib: add build_my_info_client
by Luan Shengde 26 Dec '20

26 Dec '20

purpose: build my_info for users when they installing compass-ci as private environment if not, will face account verification problems when they submitting jobs - store my_info to ES - write my_info to default/lab yaml file Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/build_my_info_client.rb | 79 +++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100755 lib/build_my_info_client.rb diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb new file mode 100755 index 0000000..2a664b4 --- /dev/null +++ b/lib/build_my_info_client.rb @@ -0,0 +1,79 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'fileutils' +require 'io/console' +require_relative '../lib/es_client' + +# purpose: +# build my_info +# store my_info to ES +# write my_info to default/lab yaml file +# for user install compass-ci at their local server +# user meet verification problems when submit jobs +# add calling in installing script to meet the verification function +# usage: +# require_relative 'build_my_info_client' +# +# build_my_info = BuildMyInfo.new(email, name, lab) +# build_my_info.config_my_info +class BuildMyInfo + def initialize(my_email, my_name, lab) + @lab = lab || 'z9' + @my_info = { + 'my_email' => my_email, + 'my_name' => my_name, + 'my_token' => %x(uuidgen).chomp + } + end + + def config_default_yaml + default_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/defaults" + FileUtils.mkdir_p default_yaml_dir unless File.directory? default_yaml_dir + default_yaml_file = "#{default_yaml_dir}/account.yaml" + FileUtils.touch(default_yaml_file) unless File.exist? default_yaml_file + + default_yaml_info = YAML.load_file(default_yaml_file) || {} + default_yaml_info['my_email'] = @my_info['my_email'] + default_yaml_info['my_name'] = @my_info['my_name'] + + File.open(default_yaml_file, 'w') do |f| + f.puts default_yaml_info.to_yaml + end + end + + def config_lab_yaml + lab_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/include/lab" + FileUtils.mkdir_p lab_yaml_dir unless File.directory? lab_yaml_dir + lab_yaml_file = "#{lab_yaml_dir}/#{(a)lab}.yaml" + FileUtils.touch(lab_yaml_file) unless File.exist? lab_yaml_file + + lab_yaml_info = YAML.load_file(lab_yaml_file) || {} + lab_yaml_info['my_token'] = @my_info['my_token'] + + File.open(lab_yaml_file, 'w') do |f| + f.puts lab_yaml_info.to_yaml + end + end + + def complete_my_info + @my_info['my_login_name'] = nil + @my_info['my_commit_url'] = nil + @my_info['my_ssh_pubkey'] = [] + end + + def store_account_info + complete_my_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def config_my_info + config_default_yaml + config_lab_yaml + store_account_info + end +end -- 2.23.0

2 5

[PATCH compass-ci] providers: qemu/kvm.sh check kernel file exists
by Xiao Shenwei 26 Dec '20

26 Dec '20

[why] if os_http not working, the kernel file will can't be download or the size is 0, then the qemu-kvm will boot by pxe, the vm will request the next job again. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/qemu/kvm.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index b9f5716..fe2345f 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -51,6 +51,11 @@ do #done < /tftpboot/boot.ipxe-centos done < $ipxe_script +[ -s "$kernel" ] || { + echo "can't get kernel or kernel size is 0" + exit +} + [ -n "$initrds" ] || { exit } -- 2.23.0

1 0

[PATCH compass-ci] [RFC] providers/qemu: adapt kvm.sh for x86_64/riscv64
by Liu Yinsi 26 Dec '20

26 Dec '20

run qemu in different arch have different options, put same options in $kvm, different options in options_$arch. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 41 +++++++++++++++++++++++++++++++++++------ 1 file changed, 35 insertions(+), 6 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index b9f5716..56cf8c0 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -16,7 +16,7 @@ if [ ! -f "$log_file" ]; then fi qemu=qemu-system-aarch64 -command -v $qemu >/dev/null || qemu=qemu-kvm +command -v $qemu >/dev/null || qemu=qemu-kvm || qemu=qemu-system-riscv64 echo $SCHED_PORT ipxe_script=ipxe_script @@ -67,15 +67,10 @@ sleep 5 kvm=( $qemu - -machine virt-4.0,accel=kvm,gic-version=3 -kernel $kernel -initrd $initrd -smp $nr_cpu -m $memory - -cpu Kunpeng-920 - -device virtio-gpu-pci - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -no-reboot -nographic @@ -83,4 +78,38 @@ kvm=( -monitor null ) +options_aarch64=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -device virtio-gpu-pci + -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} +) + +options_x86_64=( + -device virtio-gpu-pci + -bios /usr/share/ovmf/OVMF.fd + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} +) + +options_riscv64=( + -machine virt + -device virtio-net-device,netdev=net1,mac=${mac} + -netdev bridge,br=br0,id=net1,helper=/usr/lib/qemu/qemu-bridge-helper + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} +) + +case $qemu in + qemu-system-aarch64) + kvm=(${kvm[@]} ${args_aarch64[@]}) + ;; + qemu-kvm) + [ $(arch) == "x86_64" ] && kvm=(${kvm[@]} ${args_x86_64[@]}) + [ $(arch) == "aarch64" ] && kvm=(${kvm[@]} ${args_aarch64[@]}) + ;; + qemu-system-riscv) + kvm=(${kvm[@]} ${args_riscv64[@]}) + ;; +esac + "${kvm[@]}" --append "${append}" -- 2.23.0

1 1

[PATCH v2 compass-ci] result-webdav/nginx.conf: solve the problem that the symbolic link cannot be established
by Cui Yili 25 Dec '20

25 Dec '20

[why] Solve the problem that the symbolic link cannot be established when the directory is not created [how] Create a directory first. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 1feddd4..cc21926 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -54,7 +54,7 @@ http { end link_name = link_name..".cgz" root_path = "/srv/"..path - io.popen("cd "..root_path.."&& ln -sf "..upload_file.." "..link_name) + io.popen("mkdir -p "..root_path.." && cd "..root_path.."&& ln -sf "..upload_file.." "..link_name) end redis_host = os.getenv("REDIS_HOST") -- 2.23.0

1 0

[PATCH compass-ci] result-webdav/nginx.conf: solve the problem that the symbolic link cannot be established
by Cui Yili 25 Dec '20

25 Dec '20

[why] Solve the problem that the symbolic link cannot be established when the directory is not created [how] Create a directory first. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 1 + 1 file changed, 1 insertion(+) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 1feddd4..00abc30 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -54,6 +54,7 @@ http { end link_name = link_name..".cgz" root_path = "/srv/"..path + io.popen("mkdir -p "..root_path) io.popen("cd "..root_path.."&& ln -sf "..upload_file.." "..link_name) end -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] lib/matrix2.rb: fix incorrect average and stddev
by Lu Weitao 25 Dec '20

25 Dec '20

[Why] We just skip useless job when make job_list --> matrix, had not did: job_list.size - 1, the average: sum($job['stats']['metric'])/job_list.size will be incorrect. standard deviation is the same reason [How] Remove useless job before job_list --> matrix, and do minor refactoring Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 11 ++--------- lib/params_group.rb | 16 ++++++++++++++-- 2 files changed, 16 insertions(+), 11 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 14e9558..3c4035e 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -83,7 +83,7 @@ def create_matrix(job_list) suites = [] job_list.each do |job| stats = job['stats'] - next unless stats && assign_suites(suites, job, stats) + suites << job['suite'] if job['suite'] stats.each do |key, value| next if key.include?('timestamp') @@ -97,17 +97,10 @@ def create_matrix(job_list) return matrix, suites end -def assign_suites(suites, job, stats) - return unless job['suite'] && stats.keys.any? { |stat| stat.start_with?(job['suite']) } - - suites << job['suite'] -end - # input: query results from es_query # return: matrix def combine_query_data(query_data) - job_list = query_data['hits']['hits'] - job_list.map! { |job| job['_source'] } + job_list = extract_jobs_list(query_data['hits']['hits']) create_matrix(job_list) end diff --git a/lib/params_group.rb b/lib/params_group.rb index 57cd5c1..7bb7f5b 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -38,9 +38,21 @@ def auto_group(jobs_list, dimensions) end def extract_jobs_list(jobs_list) - jobs_list.map do |job| - job['_source'] + jobs_list.map! do |job| + job['_source'] if job_is_useful?(job) end + + jobs_list.compact +end + +def job_is_useful?(job) + stats = job['_source']['stats'] + return unless stats + + suite = job['_source']['suite'] + return unless suite && stats.keys.any? { |stat| stat.start_with?(suite) } + + true end def group(jobs_list, dimensions) -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] lib/matrix2.rb: remove duplicate code
by Lu Weitao 25 Dec '20

25 Dec '20

[Why] Had handle empty group in lib/params_group.rb def filter_groups(groups) groups.each do |group_key, value| value.each_key do |dim_key| value.delete(dim_key) if value[dim_key].empty? end groups.delete(group_key) if groups[group_key].empty? end end Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 4 ---- 1 file changed, 4 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index d9d86e8..14e9558 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -124,10 +124,6 @@ def combine_group_query_data(job_list, dims) suites_hash = {} groups = auto_group(job_list, dims) groups.each do |group_key, value| - if value.empty? - groups.delete(group_key) - next - end suite_list = [] value.each do |dimension_key, jobs| groups[group_key][dimension_key], suites = create_matrix(jobs) -- 2.23.0

1 0

[PATCH v4 compass-ci] doc/tutorial.md: fix abnormal display in Markdown
by Lu Kaiyi 25 Dec '20

25 Dec '20

the middle part in * and * is italics for markdown document, if want show '*' in this condition, add ` in both sides of path. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- doc/tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..bf72630 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -258,7 +258,7 @@ job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAM 4) 备注目前提供 archlinux 下已有 PKGBUILD 项目构建测试，暂不支持自主导入 PKGBUILD 文件。 - PKGBUILD 存放路径： /srv/git/archlinux/*/*/PKGBUILD。 + PKGBUILD 存放路径： `/srv/git/archlinux/*/*/PKGBUILD`。 ### depends -- 2.23.0

1 0

[PATCH compass-ci] lib/es_query: add select keyword from index
by Lu Weitao 25 Dec '20

25 Dec '20

[Why] We need only get specific keyword instead of entire document. it's like SQL: SELECT DISTINCT column FROM table Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/es_query.rb | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 3bea624..0e34bfc 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -61,6 +61,31 @@ class ESQuery return result[0]['_source'] end + + # select doc_field from index + # input: + # eg: suite (@index: jobs) + # output: + # [ + # {"key"=>"build-pkg", "doc_count"=>90841}, + # {"key"=>"cci-depends", "doc_count"=>4636}, + # {"key"=>"cci-makepkg", "doc_count"=>3647}, + # ... + # ] + def query_specific_fields(field) + query = { + aggs: { + "all_#{field}" => { + terms: { field: field, size: 1000 } + } + }, + size: 0 + } + result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field}"]['buckets'] + return nil if result.empty? + + result + end end # Range Query Example: -- 2.23.0

1 0

[PATCH compass-ci] docker-rootfs: use array save suffix
by Wang Chenglong 25 Dec '20

25 Dec '20

[Why] the docker image usually use '.tar' and '.tar.xz' as the suffix. so use array save the suffix. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/common | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/container/docker-rootfs/common b/container/docker-rootfs/common index fe37da9..89fad87 100755 --- a/container/docker-rootfs/common +++ b/container/docker-rootfs/common @@ -29,7 +29,7 @@ usage() Usage: ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> - src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ .tar, tar.xz ]. dst_rootfs_new_abspath: destination absolute path to create for rootfs Example: @@ -43,14 +43,14 @@ check_docker_img_file() local docker_img_name local docker_img_suffix - allow_docker_img_suffix='tar.xz' + allow_docker_img_suffix=('tar' 'tar.xz') docker_img_name=$(basename "$1") docker_img_suffix=${docker_img_name##*.} [ "$docker_img_suffix" == 'tar.xz' ] || { - docker_img_suffix=$(echo "$docker_img_name" |awk -F '.' '{print $(NF-1)"."$NF}') - echo "${allow_docker_img_suffix}" |grep -wq "$docker_img_suffix" || { - echo "[ERROR] Only support: .tar.xz file!" + docker_img_suffix=$(echo "$docker_img_name" | awk -F '.' '{print $(NF-1)"."$NF}') + echo "${allow_docker_img_suffix[@]}" |grep -wq "$docker_img_suffix" || { + echo "[ERROR] Only support: .tar .tar.xz file!" exit 2 } } -- 2.23.0

2 1

[PATCH compass-ci 3/3] scheduler: idle-job: submit idle job when finding no job
by Ren Wen 25 Dec '20

25 Dec '20

There're 2 types of queues to find job in order: 1. user queue: 'sched/<taishan-a12>/<user-email>' 2. idle queue: 'sched/<taishan-a12>/idle' Auto submit idle jobs when finding no job. This feature is for physical machines. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 29 +++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 5f71b32..8483b25 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -53,31 +53,41 @@ class Sched end def get_queues(host) - queues = [] of String + default_queues = [] of String + sub_queues = [] of String queues_str = @redis.hash_get("sched/host2queues", host) - return queues unless queues_str + return default_queues, sub_queues unless queues_str queues_str.split(',', remove_empty: true) do |item| - queues << item.strip + default_queues << item.strip end - sub_queues = [] of String - queues.each do |queue| - matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") + default_queues.each do |queue| + matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") matched_queues.each do |mq| match_data = "#{mq}".match(%r(^#{QUEUE_NAME_BASE}/sched/(#{queue}/.+)/ready$)) sub_queues << match_data[1] if match_data end end - return rand_queues(sub_queues) + return default_queues, sub_queues end def get_job_from_queues(queues, testbox) + default_queues, sub_queues = queues + rand_sub_queues = rand_queues(sub_queues) + idle_queues = [] of String + + default_queues.each do |queue| + idle_queues << "#{queue}/idle" + end + + all_queues = rand_sub_queues + idle_queues + job = nil - queues.each do |queue| + all_queues.each do |queue| job = prepare_job("sched/#{queue}", testbox) return job if job end @@ -91,6 +101,9 @@ class Sched if job create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) + else + # for physical machines now + spawn { auto_submit_idle_job(host) } end return boot_content(job, boot_type) -- 2.23.0

1 0

[PATCH compass-ci 2/3] scheduler: idle-job: hand over idle job
by Ren Wen 25 Dec '20

25 Dec '20

When submitting an idle job, the job will be submitted to "queues/sched/<queue>/idle"; When consuming an idle job, the job will be moved to "queues/sched/<queue>/in_process"; It will hand over job from "queues/sched/<queue>/idle/in_process" before. It's wrong for idle job, so fix it. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/close_job.cr | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index 9b4da6a..bce6b33 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -14,8 +14,11 @@ class Sched raise "es set job content fail!" end + subqueue = job.subqueue + queue = (subqueue == "idle" ? job.queue : "#{job.queue}/#{subqueue}") + response = @task_queue.hand_over_task( - "sched/#{job.queue}/#{job.subqueue}", "extract_stats", job_id + "sched/#{queue}", "extract_stats", job_id ) if response[0] != 201 raise "#{response}" -- 2.23.0

1 0

[PATCH compass-ci 1/3] scheduler: idle-job: fix queue when submitting job
by Ren Wen 25 Dec '20

25 Dec '20

[Before]: 1) job has a field 'idle_job' when submitting idle job. 2) field 'uuid' is used with remote user to do a rate limit. [Now]: 1) idle job has a subqueue 'idle'. 2) use email instead of 'uuid'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 7 ------- 1 file changed, 7 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index c322d0f..c09d145 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -114,13 +114,6 @@ class Sched def submit_single_job(job) queue = "#{job.queue}/#{job.subqueue}" - # only single job will has "idle job" and "execute rate limiter" - if job["idle_job"].empty? - queue += "#{job.get_uuid_tag}" - else - queue = "#{queue}/idle" - end - job_id = add_task(queue, job.lab) return [{ "job_id" => "0", -- 2.23.0

1 0

[PATCH lkp-tests] lib/bootstrap.sh: judge whether has tail cmd before use it
by Wei Jihui 25 Dec '20

25 Dec '20

if is_docker is true, it will use tail cmd, so judge it before use it Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/bootstrap.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index c9a9f89f..ae902207 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -314,6 +314,8 @@ announce_bootup() redirect_stdout_stderr() { + has_cmd tail || return + if is_docker; then [ -e /tmp/stdout ] || touch /tmp/stdout [ -e /tmp/stderr ] || touch /tmp/stderr @@ -328,7 +330,6 @@ redirect_stdout_stderr() fi [ -c /dev/kmsg ] || return - has_cmd tail || return exec > /tmp/stdout exec 2> /tmp/stderr -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] lib/matrix2.rb: fix incorrect average and stddev
by Lu Weitao 25 Dec '20

25 Dec '20

[Why] We just skip useless job when make job_list --> matrix, had not did: job_list.size - 1, the average: sum($job['stats']['metric'])/job_list.size will be incorrect. standard deviation is the same reason [How] Remove useless job before job_list --> matrix, and do minor refactoring Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 11 ++--------- lib/params_group.rb | 16 ++++++++++++++-- 2 files changed, 16 insertions(+), 11 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 14e9558..ce64da1 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -83,7 +83,7 @@ def create_matrix(job_list) suites = [] job_list.each do |job| stats = job['stats'] - next unless stats && assign_suites(suites, job, stats) + suites << job['suite'] stats.each do |key, value| next if key.include?('timestamp') @@ -97,17 +97,10 @@ def create_matrix(job_list) return matrix, suites end -def assign_suites(suites, job, stats) - return unless job['suite'] && stats.keys.any? { |stat| stat.start_with?(job['suite']) } - - suites << job['suite'] -end - # input: query results from es_query # return: matrix def combine_query_data(query_data) - job_list = query_data['hits']['hits'] - job_list.map! { |job| job['_source'] } + job_list = extract_jobs_list(query_data['hits']['hits']) create_matrix(job_list) end diff --git a/lib/params_group.rb b/lib/params_group.rb index 57cd5c1..7bb7f5b 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -38,9 +38,21 @@ def auto_group(jobs_list, dimensions) end def extract_jobs_list(jobs_list) - jobs_list.map do |job| - job['_source'] + jobs_list.map! do |job| + job['_source'] if job_is_useful?(job) end + + jobs_list.compact +end + +def job_is_useful?(job) + stats = job['_source']['stats'] + return unless stats + + suite = job['_source']['suite'] + return unless suite && stats.keys.any? { |stat| stat.start_with?(suite) } + + true end def group(jobs_list, dimensions) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/matrix2.rb: remove duplicate code
by Lu Weitao 25 Dec '20

25 Dec '20

[Why] Had handle empty group in lib/params_group.rb def filter_groups(groups) groups.each do |group_key, value| value.each_key do |dim_key| value.delete(dim_key) if value[dim_key].empty? end groups.delete(group_key) if groups[group_key].empty? end end Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 4 ---- 1 file changed, 4 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index d9d86e8..14e9558 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -124,10 +124,6 @@ def combine_group_query_data(job_list, dims) suites_hash = {} groups = auto_group(job_list, dims) groups.each do |group_key, value| - if value.empty? - groups.delete(group_key) - next - end suite_list = [] value.each do |dimension_key, jobs| groups[group_key][dimension_key], suites = create_matrix(jobs) -- 2.23.0

1 0

[PATCH lkp-tests] sbin/run_job: Automatic task execution
by Hu Xuejiao 25 Dec '20

25 Dec '20

[purpose] Verify that a local job can be consumed properly Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100755 sbin/run_job diff --git a/sbin/run_job b/sbin/run_job new file mode 100755 index 00000000..14e02b2f --- /dev/null +++ b/sbin/run_job @@ -0,0 +1,73 @@ +#!/bin/bash + +cci_makepkg_file="$LKP_SRC/jobs/cci-makepkg.yaml" +grep "atomic" $cci_makepkg_file || sed -i 's|benchmark:|& atomic|g' $cci_makepkg_file + +submit_one_job() +{ + cmd=( + submit -m $yaml + os=${single_arg[0]} + os_arch=${single_arg[1]} + os_version=${single_arg[2]} + os_mount=${single_arg[3]} + testbox=$testbox + queue="$tbox_group.$HOSTNAME" + ) + output=$("${cmd[@]}") + echo "$output" | grep -o -E 'job_state=finished[^\"]*' +} + +submit_job() +{ + yaml_args=("$cci_makepkg_file" "host-info.yaml") + for yaml in "${yaml_args[@]}" + do + submit_one_job & + done +} + +run_qemu() +{ + local nr_vm=4 + local testbox="vm-2p8g" + local tbox_group="vm-2p8g" + local hostname=$tbox_group.$HOSTNAME + local queues="$tbox_group.$HOSTNAME" + + $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" + + qemu_args=( + "openeuler aarch64 20.03 cifs" + "centos aarch64 7.6.1810 initramfs" + ) + for qemu_arg in "${qemu_args[@]}" + do + local single_arg=($(echo $qemu_arg)) + submit_job + done +} + +run_docker() +{ + local nr_container=2 + local tbox_group="dc-8g" + local testbox="dc-8g" + local hostname=$tbox_group.$HOSTNAME + local queues="$tbox_group.$HOSTNAME" + + $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues + + docker_args=( + "centos aarch64 7.6.1810 container" + ) + for docker_arg in "${docker_args[@]}" + do + local single_arg=($(echo $docker_arg)) + submit_job + done +} + +run_qemu +run_docker +wait -- 2.23.0

2 1

[PATCH v7 compass-ci] tutorial.md: update initramfs tutorial.md
by Wang Chenglong 25 Dec '20

25 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/tutorial.md | 71 ++++++++++++------------------------------------- 1 file changed, 17 insertions(+), 54 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..2409c86 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -150,61 +150,24 @@ job yaml ��[YAML](http://yaml.org/YAM #### �� - 1. �� os �� rootfs�� openEuler �� - - �� docker �� rootfs - - 1) �� openEuler �� docker �� - - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) �� docker �� - ```bash - docker load -i openEuler-docker.aarch64 - ``` - - 3) �� openEuler �� - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) �� docker �� rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - - �� qemu.img(qcow2��)�� rootfs (openEuler��) - - - 1) �� openEule r�� qcow2 �� - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… - ``` - 2) ��{compass-ci}/container/qcow2rootfs ��rootfs - ```bash - cd {compass-ci}/container/qcow2rootfs - ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs - ``` - 2. ��rootfs - 1. ��chroot�� rootfs (�� root ��) - ```bash - chroot openEuler-rootfs - ``` - 2. �� - - a. �� root �� - b. �� ssh �� - c. �� - d. �� docker �� osimage �� - > 1. �� - > 2. �� centos ��rpm�� - > 3. �� yum �� - > 4. �� docker �� - - 3. �� rootfs�� - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` + 1. �� openEuler �� + ```bash + cd {compass-ci}/rootfs/initramfs/openeuler/aarch64/20.03-LTS/ + ./run + ``` + + 2. ��/�� + �� {compass-ci} ��rootfs�� rootfs ��/�� + ```bash + mkdir rootfs + cd rootfs + zcat ../$os_name.cgz | cpio -idmv + find ./lib/modules/* | cpio -o -Hnewc | gzip -9 > $HOME/modules-$os_name.cgz + find ./usr/src/kernels/* | cpio -o -Hnewc | gzip -9 > $HOME/headers-$os_name.cgz + ``` + #### FAQ + 1. �� Unable to mount root fs on unknown-block�� - �� ```bash -- 2.23.0

1 0

[PATCH v5 lkp-tests] tests/build-pkg: add config file support
by Wang Yong 25 Dec '20

25 Dec '20

[Why] config file in some software need storage and recall now we can add config value in yaml and cp ${config} to ${cci_dir}/${pkgname}/ in PKGBUILD so that config file can be uploaded correctly $ cat test.yaml ... config: ${config} ... Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index addff9f7b..44bd1c10d 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -4,6 +4,7 @@ # - upstream_commit # - git_tag # - pkg_ver +# - config # upstream_url # We'll use the below global env vars, but not treat them as parameter to avoid @@ -20,6 +21,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/distro/common +. $LKP_SRC/lib/http.sh check_vars() { @@ -31,6 +33,24 @@ check_vars() [ -n "$os_mount" ] || die "os_mount is empty" } +get_config() +{ + local cci_http_host=${INITRD_HTTP_HOST} + local cci_http_port=${INITRD_HTTP_PORT:-8800} + export cci_dir=/cci/build-config + local remote_dir="http://${cci_http_host}:${cci_http_port}${cci_dir}/${pkgname}" + + mkdir -p ${cci_dir} + + [ -n "$config" ] || return + http_get_file "${remote_dir}/${config}" "${cci_dir}/${pkgname}/${config}" +} + +upload_config() +{ + upload_to_target_dir "${cci_dir}/${pkgname}" +} + mount_dest() { # the same image is mounted to cifs and nfs, the generated cgz files @@ -46,7 +66,7 @@ get_pkgfile() { curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "${2##*/}" - } +} request_pkg() { @@ -125,13 +145,17 @@ build_source_pkg() fi } replace_source PKGBUILD.src - create_cgz_name + get_config PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf - [ "$?" == 0 ] || exit 1 + local build_state="$?" + + upload_config + + [ "${build_state}" == "0" ] || exit 1 create_softlink upload_to_target_dir ${PKG_MNT}/${pack_to}/${pkgname} } -- 2.23.0

1 0

[PATCH v4 lkp-tests] tests/build-pkg: add config file support
by Wang Yong 25 Dec '20

25 Dec '20

[Why] config file in some software need storage and recall now we can add config value in yaml and cp ${config} to ${cci_dir}/${pkgname}/ in PKGBUILD so that config file can be uploaded correctly $ cat test.yaml ... config: ${config} ... Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index addff9f7b..06a58d498 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -4,6 +4,7 @@ # - upstream_commit # - git_tag # - pkg_ver +# - config # upstream_url # We'll use the below global env vars, but not treat them as parameter to avoid @@ -20,6 +21,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/distro/common +. $LKP_SRC/lib/http.sh check_vars() { @@ -31,6 +33,29 @@ check_vars() [ -n "$os_mount" ] || die "os_mount is empty" } +check_config_var() +{ + [ -n "$config" ] || return +} + +get_config() +{ + local cci_http_host=${INITRD_HTTP_HOST} + local cci_http_port=${INITRD_HTTP_PORT:-8800} + export cci_dir=/cci/build-config + local remote_dir="http://${cci_http_host}:${cci_http_port}${cci_dir}/${pkgname}" + + mkdir -p ${cci_dir} + + check_config_var + http_get_file "${remote_dir}/${config}" "${cci_dir}/${pkgname}/${config}" +} + +upload_config() +{ + upload_to_target_dir "${cci_dir}/${pkgname}" +} + mount_dest() { # the same image is mounted to cifs and nfs, the generated cgz files @@ -46,7 +71,7 @@ get_pkgfile() { curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "${2##*/}" - } +} request_pkg() { @@ -125,13 +150,17 @@ build_source_pkg() fi } replace_source PKGBUILD.src - create_cgz_name + get_config PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf - [ "$?" == 0 ] || exit 1 + local build_state="$?" + + upload_config + + [ "${build_state}" == "0" ] || exit 1 create_softlink upload_to_target_dir ${PKG_MNT}/${pack_to}/${pkgname} } -- 2.23.0

2 2

[PATCH v2 compass-ci] doc: update README.md
by Liu Yinsi 24 Dec '20

24 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 113 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 87 insertions(+), 26 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index ccb1d15..a9a4bb0 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -1,26 +1,87 @@ -Deploy a mini instance in one single machine. - -Prepare: - 1. hardware - Server: at least prepare a server - ProductType: ThaiShan200-2280 - Arch: aarch64 - Memory: 8G - CPU: 64 nuclear - DiskSpace: 500G - - 2. software - OS: openEuler-aarch64-20.03 LTS - git: suggest 2.23.0 - - 3. network - Internet is available - - 4. /os for store rootfs - >= 300G - -Steps: - umask 002 - git clone https://gitee.com/wu_fengguang/compass-ci.git - cd compass-ci/sparrow - ./install-tiny +# 本地搭建 compass-ci 服务器节点 + +在 openEuler 系统一键部署 compass-ci 环境，当前已支持 openEuler-aarch64-20.03-LTS 系统环境，以下配置仅供参考。 + +## 准备工作 +- 硬件 + 服务器类型：ThaiShan200-2280 (建议) + 架构：aarch64 + 内存：>= 32GB + CPU：64 nuclear (建议) + 硬盘：>= 500G (建议划分独立分区) + + >**说明：** + >划分较小独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… + >划分较大独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… + +- 软件 + OS：openEuler-aarch64-20.03 LTS + git：2.23.0版本 (建议) + 预留空间：>= 300G + 网络：可以访问互联网 + + >**说明：** + >openEuler 系统安装详细操作请参考[添加测试用例](https://openeuler.org/zh/docs/20.03_LTS/docs/Installati… + +### 操作指导 +1. 登录 openEuler 系统 + +2. 创建工作目录并设置文件权限 + ```bash + mkdir demo && cd demo && umask 002 + ``` +3. 克隆 compass-ci 项目代码到 demo 目录 + ```bash + git clone https://gitee.com/wu_fengguang/compass-ci.git + ``` +4. 执行一键部署脚本 install-tiny + ```bash + cd compass-ci/sparrow && ./install-tiny + ``` +5. 生成lkp-aarch64.cgz压缩包 + ```bash + cd /c/compass-ci/container/lkp-initrd && ./run + ``` +6. 验证账号 + ```bash + cd /c/compass-ci/sbin && ./build-my-info.rb + 按照提示输入账号名和邮箱即可 + +7. 测试环境是否可以提交job测试 + ```bash + submit iperf.yaml testbox=vm-2p8g + ``` + + 执行上述命令正常情况下会提示信息如下： + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + compass-ci搭建完毕，下面就可以开始进行测试了。 + +8. 下载rootfs（以下载openeuler/aarch64/20.03为例，需要在哪个系统上测试就去对应的/srv/os/目录下使用wget命令下载cgz文件包） + ```bash + mkdir -p /srv/os/openeuler/aarch64/ + cd /srv/os/openeuler/aarch64 + wget http://124.90.34.227:11300/os/test/openeuler/aarch64/20.03.cgz + ``` + +9. 解压cgz包 + ```bash + gzip -dc 20.03.cgz | cpio -idv + ``` + +10. [使用 compass-ci 平台测试开源项目](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… + +11. [编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testca… + +12. 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… + 注意：os_mount 必须指定为cifs + +13. 运行测试任务 + ```bash + cd /c/compass-ci/providers/ && ./my-qemu.sh + ``` + +14. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… id查看output文件 + ```bash + cd /srv/result/iperf/$job_id/output + ``` -- 2.23.0

2 1

[PATCH lkp-tests] bootstrap.sh: need a blank line after single return
by Xu Xijian 24 Dec '20

24 Dec '20

A blank line is needed after single return, more than 1 return cmd together doesn't need. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- lib/bootstrap.sh | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index ad1c4a2f2..bd168a62a 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -235,7 +235,6 @@ run_ntpdate() [ -z "$NO_NETWORK" ] || return [ -n "$LKP_SERVER" ] || return is_clearlinux && clearlinux_timesync && return - [ -x '/usr/sbin/ntpdate' ] || return local hour="$(date +%H)" @@ -371,6 +370,7 @@ install_deb() # solved the dependent relationship, so here we ignore the dependency errors echo "install debs round one: dpkg -i --force-confdef --force-depends $files" dpkg -i --force-confdef --force-depends $files 2>/tmp/dpkg_error && return + grep -v "$filter_info" /tmp/dpkg_error # round two, install all debs one by one accroding to keep-deb which is in sequence @@ -397,6 +397,7 @@ install_deb() install_rpms() { [ -d /opt/rpms ] || return + echo "install $(ls /opt/rpms/*.rpm)" rpm -ivh --force --ignoresize /opt/rpms/*.rpm } @@ -458,6 +459,7 @@ fixup_packages() mount_debugfs() { is_docker && return + check_mount debug /sys/kernel/debug -t debugfs } @@ -467,6 +469,7 @@ cleanup_pkg_cache() local pkg_cache=$1 local cleanup_stamp=$pkg_cache/cleanup_stamp/$(date +%U) [ -d "$cleanup_stamp" ] && return + mkdir $cleanup_stamp -p find "$pkg_cache" $ -type f -mtime +7 -delete $ -or $ -type d -ctime +7 -empty -delete $ @@ -497,6 +500,7 @@ mount_rootfs() set_job_state 'load_disk_fail' return 1 } + ROOTFS_DIR=/opt/rootfs mkdir -p $ROOTFS_DIR mount $rootfs_partition $ROOTFS_DIR || { @@ -536,7 +540,6 @@ netconsole_init() { # don't init netconsole at local run [ "$LKP_LOCAL_RUN" = "1" ] && return - [ -z "$netconsole_port" ] && return [ -n "$NO_NETWORK" ] && return @@ -545,6 +548,7 @@ netconsole_init() # Select the interface that can access netconsole server netconsole_interface=$(show_default_interface) [ -n "$netconsole_server" ] || return + # eth0 is default interface if netconsole_interface is null. modprobe netconsole netconsole=@/$netconsole_interface,$netconsole_port@$netconsole_server/ } @@ -755,6 +759,7 @@ boot_init() install_pkgs() { [ -d /opt/pkgs ] || return + echo "install $(ls /opt/pkgs/*.pkg.tar*)" pacman -U --needed --noconfirm /opt/pkgs/*.pkg.tar* } -- 2.23.0

1 0

[PATCH compass-ci 3/3] sbin/auto_submit.yaml: deploy-cci is not completely adapted in centos
by Hu Xuejiao 24 Dec '20

24 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/auto_submit.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index e1eb39b..fce1dc0 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -2,7 +2,6 @@ a/AvxToNeon/AvxToNeon: - testbox=vm-2p8g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 api-avx2neon.yaml c/compass-ci/compass-ci: - testbox=taishan200-2280-2s64p-256g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml -- testbox=taishan200-2280-2s64p-256g os=centos os_version=7.6.1810 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml - testbox=taishan200-2280-2s64p-256g os=debian os_version=sid os_mount=initramfs os_arch=aarch64 deploy-cci.yaml archlinux: - testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- 2.23.0

2 1

[PATCH v2 lkp-tests] bin/run_job: Automatic task execution
by Hu Xuejiao 24 Dec '20

24 Dec '20

[purpose] Verify that a local job can be consumed properly Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100755 sbin/run_job diff --git a/sbin/run_job b/sbin/run_job new file mode 100755 index 00000000..14e02b2f --- /dev/null +++ b/sbin/run_job @@ -0,0 +1,73 @@ +#!/bin/bash + +cci_makepkg_file="$LKP_SRC/jobs/cci-makepkg.yaml" +grep "atomic" $cci_makepkg_file || sed -i 's|benchmark:|& atomic|g' $cci_makepkg_file + +submit_one_job() +{ + cmd=( + submit -m $yaml + os=${single_arg[0]} + os_arch=${single_arg[1]} + os_version=${single_arg[2]} + os_mount=${single_arg[3]} + testbox=$testbox + queue="$tbox_group.$HOSTNAME" + ) + output=$("${cmd[@]}") + echo "$output" | grep -o -E 'job_state=finished[^\"]*' +} + +submit_job() +{ + yaml_args=("$cci_makepkg_file" "host-info.yaml") + for yaml in "${yaml_args[@]}" + do + submit_one_job & + done +} + +run_qemu() +{ + local nr_vm=4 + local testbox="vm-2p8g" + local tbox_group="vm-2p8g" + local hostname=$tbox_group.$HOSTNAME + local queues="$tbox_group.$HOSTNAME" + + $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" + + qemu_args=( + "openeuler aarch64 20.03 cifs" + "centos aarch64 7.6.1810 initramfs" + ) + for qemu_arg in "${qemu_args[@]}" + do + local single_arg=($(echo $qemu_arg)) + submit_job + done +} + +run_docker() +{ + local nr_container=2 + local tbox_group="dc-8g" + local testbox="dc-8g" + local hostname=$tbox_group.$HOSTNAME + local queues="$tbox_group.$HOSTNAME" + + $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues + + docker_args=( + "centos aarch64 7.6.1810 container" + ) + for docker_arg in "${docker_args[@]}" + do + local single_arg=($(echo $docker_arg)) + submit_job + done +} + +run_qemu +run_docker +wait -- 2.23.0

1 0

[PATCH v2 lkp-tests] bin/run_job: Automatic task execution
by Hu Xuejiao 24 Dec '20

24 Dec '20

[purpose] Verify that a local job can be consumed properly Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100755 sbin/run_job diff --git a/sbin/run_job b/sbin/run_job new file mode 100755 index 00000000..d45571b3 --- /dev/null +++ b/sbin/run_job @@ -0,0 +1,74 @@ +#!/bin/bash + +cci_makepkg_file="$LKP_SRC/jobs/cci-makepkg.yaml" +grep "atomic" $cci_makepkg_file || sed -i 's|benchmark:|& atomic|g' $cci_makepkg_file + +submit_one_job() +{ + cmd=( + submit -m $yaml + os=${single_arg[0]} + os_arch=${single_arg[1]} + os_version=${single_arg[2]} + os_mount=${single_arg[3]} + testbox=$testbox + queue="$tbox_group.$HOSTNAME" + ) + echo ${cmd[@]} + output=$("${cmd[@]}") + echo "$output" | grep -o -E 'job_state=finished[^\"]*' +} + +submit_job() +{ + yaml_args=("$cci_makepkg_file" "host-info.yaml") + for yaml in "${yaml_args[@]}" + do + submit_one_job & + done +} + +run_qemu() +{ + local nr_vm=4 + local testbox="vm-2p8g" + local tbox_group="vm-2p8g" + local hostname=$tbox_group.$HOSTNAME + local queues="$tbox_group.$HOSTNAME" + + $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" + + qemu_args=( + "openeuler aarch64 20.03 cifs" + "centos aarch64 7.6.1810 initramfs" + ) + for qemu_arg in "${qemu_args[@]}" + do + local single_arg=($(echo $qemu_arg)) + submit_job + done +} + +run_docker() +{ + local nr_container=2 + local tbox_group="dc-8g" + local testbox="dc-8g" + local hostname=$tbox_group.$HOSTNAME + local queues="$tbox_group.$HOSTNAME" + + $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues + + docker_args=( + "centos aarch64 7.6.1810 container" + ) + for docker_arg in "${docker_args[@]}" + do + local single_arg=($(echo $docker_arg)) + submit_job + done +} + +run_qemu +run_docker +wait -- 2.23.0

2 1

[PATCH compass-ci 2/3] 3-code/git: add repository version pointing
by Hu Xuejiao 24 Dec '20

24 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 494107e..2159ddc 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -13,11 +13,18 @@ cd /c || exit # modify and manual run for now: [ -d "compass-ci/.git" ] || { git clone https://gitee.com/wu_fengguang/compass-ci.git - ln -s compass-ci cci + + cd /c/compass-ci || exit + [ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" + + ln -s compass-ci /c/cci } [ -d "lkp-tests/.git" ] || { git clone https://gitee.com/wu_fengguang/lkp-tests.git + + cd /c/lkp-tests || exit + [ -n "$lkp_commitid" ] && git reset --hard "$lkp_commitid" } [ -d "upstream-repos/.git" ] || { -- 2.23.0

1 0

[PATCH compass-ci 1/3] doc/job: add SCHED_HOST, SCHED_PORT description
by Hu Xuejiao 24 Dec '20

24 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- doc/job/SCHED_HOST-SCHED_PORT.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 doc/job/SCHED_HOST-SCHED_PORT.md diff --git a/doc/job/SCHED_HOST-SCHED_PORT.md b/doc/job/SCHED_HOST-SCHED_PORT.md new file mode 100644 index 0000000..74c628a --- /dev/null +++ b/doc/job/SCHED_HOST-SCHED_PORT.md @@ -0,0 +1,15 @@ +# SCHED_HOST, SCHED_PORT +Meaning: +- The job task depends on scheduler service, which will distribute tasks to test + machine. All tasks will run in test machine. +- `Scheduler` is one of our service, `SCHED` is the abbreviation of `Scheduler` + these variables mean: + SCHED_HOST: the scheduler service ip. + SCHED_PORT: the scheduler service port. + +Usage example: + +```$LKP_SRC/include/lab/z9 +SCHED_HOST: api.compass-ci.openeuler.org +SCHED_PORT: 11306 +``` -- 2.23.0

1 0

[PATCH lkp-tests] lib/bootstrap.sh: fix redirect_stdout_stderr for docker
by Xiao Shenwei 24 Dec '20

24 Dec '20

[why] exec command will redirect stdout stderr to /tmp/stdout, /tmp/stderr but docker log based on /dev/stdout and /dev/stderr, it cause the task status cannot be viewed from the console, which is difficult to debug. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- lib/bootstrap.sh | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index ad1c4a2f2..c9a9f89f9 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -315,6 +315,13 @@ announce_bootup() redirect_stdout_stderr() { if is_docker; then + [ -e /tmp/stdout ] || touch /tmp/stdout + [ -e /tmp/stderr ] || touch /tmp/stderr + tail -f /tmp/stdout > /dev/stdout & + echo $! >> /tmp/pid-tail-global + tail -f /tmp/stderr > /dev/stderr & + echo $! >> /tmp/pid-tail-global + exec > /tmp/stdout exec 2> /tmp/stderr return -- 2.23.0

1 0

[PATCH v2 lkp-tests] adaptation/archlinux: add archlinux mapping
by Wang Yong 24 Dec '20

24 Dec '20

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/adaptation/archlinux | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/distro/adaptation/archlinux b/distro/adaptation/archlinux index 89163cf41..c52212e08 100644 --- a/distro/adaptation/archlinux +++ b/distro/adaptation/archlinux @@ -72,3 +72,11 @@ libtirpc: libtirpc1 initramfs-tools: openssh-server: openssh openssh-client: openssh +autopoint: gettext +g++: gcc +libarchive13: libarchive +libbz2-1.0: bzip2 +libssl: openssl +libtool-bin: libtool +linux-perf: perf +ruby-dev: ruby -- 2.23.0

1 0

[PATCH compass-ci] lib/env.cr: rename to web_env.cr
by Wu Zhende 24 Dec '20

24 Dec '20

[Why] env.cr is the same as the file name provided by the crystal. [Error] /c/cci/scheduler $ crystal a.cr Showing last frame. Use --error-trace for full trace. In /usr/lib/crystal/core/json/from_json.cr:140:5 140 | def Set.new(pull : JSON::PullParser) Error: undefined constant Set Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 2 +- src/lib/{env.cr => web_env.cr} | 0 src/scheduler/scheduler.cr | 2 +- 3 files changed, 2 insertions(+), 2 deletions(-) rename src/lib/{env.cr => web_env.cr} (100%) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 6caa5b9..2d9dba6 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -5,7 +5,7 @@ require "kemal" require "yaml" require "./job" -require "./env" +require "./web_env" require "./block_helper" require "./taskqueue_api" require "./remote_git_client" diff --git a/src/lib/env.cr b/src/lib/web_env.cr similarity index 100% rename from src/lib/env.cr rename to src/lib/web_env.cr diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index 2103f7c..6b032ee 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -3,7 +3,7 @@ require "kemal" -require "../lib/env" +require "../lib/web_env" require "../lib/sched" require "../lib/json_logger" -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sparrow/4-docker/buildall: build dist image when it doesn't exist
by Xu Xijian 24 Dec '20

24 Dec '20

If the dist docker image doesn't exist for some reason, run build script to create it. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- sparrow/4-docker/buildall | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 32801d1..9e66ac9 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -32,7 +32,8 @@ do_one() ( cd $container [ "$container_name" == 'ssh-r' ] && exit - if [ "$action" != "reboot" ]; then + docker images | grep -w -q "$container_name" + if [ "$?" == 0 ] && [ "$action" != "reboot" ]; then [ -x build ] && ./build [ -x install ] && ./install fi -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] code-spec.md: adjust markdown document format
by Xu Xijian 24 Dec '20

24 Dec '20

In markdown documents, text for bash should be contained in block as belows: ```SHELL content SHELL Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/code-spec.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/code-spec.md b/doc/code-spec.md index d44c6fe..a717f64 100644 --- a/doc/code-spec.md +++ b/doc/code-spec.md @@ -21,10 +21,12 @@ code注意事项 - puts/printf字符串应当容易搜索(grep)。特别注意包含足够长的固定内容substring. 例如： +```SHELL die "test yaml$${test_yaml}$ not exist in ${LKP_SRC}/jobs !!!" ==> die "cannot find test yaml ($test_yaml) in $LKP_SRC/jobs" ^^^^^^^^^^^^^^^^^^^^^ grep-friendly fixed substring +``` 重构友好编码 ============ -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_client: add send_mail_text function
by Luan Shengde 24 Dec '20

24 Dec '20

for user use mail_client to send mail with 'mail content' data type Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail_client.rb | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/mail_client.rb b/lib/mail_client.rb index b13492f..f63e65d 100644 --- a/lib/mail_client.rb +++ b/lib/mail_client.rb @@ -24,4 +24,9 @@ class MailClient resource = RestClient::Resource.new("http://#{@host}:#{@port}/send_mail_encode") resource.post(Base64.encode64(mail_data)) end + + def send_mail_text(mail_text) + resource = RestClient::Resource.new("http://#{@host}:#{@port}/send_mail_text") + resource.post(mail_text) + end end -- 2.23.0

1 0

[PATCH lkp-tests] tests/cci-depends: update depends for cci-depends
by Wang Yong 24 Dec '20

24 Dec '20

[Why] cci-depends need base packages to generate cgz file, and mount operation is deprecated Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/depends/cci-depends | 3 ++- tests/cci-depends | 2 ++ 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/distro/depends/cci-depends b/distro/depends/cci-depends index 1f7f73ac3..0dc9d1501 100644 --- a/distro/depends/cci-depends +++ b/distro/depends/cci-depends @@ -1,2 +1,3 @@ -cifs-utils cpio +curl +gzip diff --git a/tests/cci-depends b/tests/cci-depends index 42d5ef676..8baf023f3 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -44,6 +44,8 @@ check_shared_package "$benchmark" && { share_debian_package +distro_install_depends cci-depends + fixup_distro_mirror update -- 2.23.0

1 0

[PATCH v3 compass-ci] doc: add runtime field in borrow-machine.en.md and borrow-machine.zh.md
by Zhang Yale 24 Dec '20

24 Dec '20

User can specify the borrow time in "runtime" field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/manual/borrow-machine.en.md | 9 +++++---- doc/manual/borrow-machine.zh.md | 9 +++++---- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index bed9fcc..b14ea0a 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -203,6 +203,7 @@ Ensure that you have performed the following operations according to the [apply- suite: borrow testcase: borrow + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -212,10 +213,10 @@ Ensure that you have performed the following operations according to the [apply- %> sshd: # sleep at the bottom - sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # Use the VIM editor to change the value of the sleep field. + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml + runtime: 1h + # Use the VIM editor to change the value of the runtime field. hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # After changing the value, submit the request again. hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 3f500e2..80c15d1 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -207,6 +207,7 @@ suite: borrow testcase: borrow + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -216,10 +217,10 @@ %> sshd: # sleep at the bottom - sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # ��vim��sleep�� + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml + runtime: 1h + ��vim��runtime�� hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # ��submit�� hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

1 0

[PATCH v2 compass-ci] lib: add build_my_info_client
by Luan Shengde 24 Dec '20

24 Dec '20

purpose: for user build their infos when installing compassci - store my_info to ES - write my_info to default/lab yaml file if not, job will fail for verification problems for user private environment Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/build_my_info_client.rb | 79 +++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100755 lib/build_my_info_client.rb diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb new file mode 100755 index 0000000..e4a4335 --- /dev/null +++ b/lib/build_my_info_client.rb @@ -0,0 +1,79 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'fileutils' +require 'io/console' +require_relative '../lib/es_client' + +# purpose: +# build my_info +# store my_info to ES +# write my_info to default/lab yaml file +# for user install compass-ci at their local server +# user meet verification problems when submit jobs +# add calling in installing script to meet the verification function +# usage: +# require_relative 'build_my_info_client' +# +# build_my_info = BuildMyInfo.new(email, name, lab) +# build_my_info.config_my_info +class BuildMyInfo + def initialize(my_email, my_name, lab) + @lab = lab || 'z9' + @my_info = { + 'my_email' => my_email, + 'my_name' => my_name, + 'my_token' => %x(uuidgen).chomp + } + end + + def config_default_yaml + default_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/defaults" + FileUtils.mkdir_p default_yaml_dir unless File.directory? default_yaml_dir + default_yaml_file = "#{default_yaml_dir}/account.yaml" + FileUtils.touch(default_yaml_file) unless File.exist? default_yaml_file + + default_yaml_info = YAML.load_file(default_yaml_file) || {} + default_yaml_info['my_email'] = @my_info['my_email'] + default_yaml_info['my_name'] = @my_info['my_name'] + + File.open(default_yaml_file, 'w') do |f| + f.puts default_yaml_info.to_yaml + end + end + + def config_lab_yaml + lab_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/include/lab" + FileUtils.mkdir_p lab_yaml_dir unless File.directory? lab_yaml_dir + lab_yaml_file = "#{lab_yaml_dir}/#{(a)lab}.yaml" + FileUtils.touch(lab_yaml_file) unless File.exist? lab_yaml_file + + lab_yaml_info = YAML.load_file(lab_yaml_file) || {} + lab_yaml_info['my_token'] = @my_info['my_token'] + + File.open(lab_yaml_file, 'w') do |f| + f.puts lab_yaml_info.to_yaml + end + end + + def complete_my_info + @my_info['my_login_name'] = '' + @my_info['my_commit_url'] = '' + @my_info['my_ssh_pubkey'] = [] + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def config_my_info + config_default_yaml + config_lab_yaml + complete_my_info + store_account_info + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci] doc: add runtime field in borrow-machine.en.md and borrow-machine.zh.md
by Zhang Yale 24 Dec '20

24 Dec '20

User can specify the sleep time in "runtime" field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/manual/borrow-machine.en.md | 9 +++++---- doc/manual/borrow-machine.zh.md | 9 +++++---- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index b9c2bc4..cfb67aa 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -203,6 +203,7 @@ Ensure that you have performed the following operations according to the [apply- suite: borrow testcase: borrow + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -212,10 +213,10 @@ Ensure that you have performed the following operations according to the [apply- %> sshd: # sleep at the bottom - sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # Use the VIM editor to change the value of the sleep field. + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml + runtime: 1h + # Use the VIM editor to change the value of the runtime field. hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # After changing the value, submit the request again. hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 06d4db6..66f6876 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -207,6 +207,7 @@ suite: borrow testcase: borrow + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -216,10 +217,10 @@ %> sshd: # sleep at the bottom - sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # ��vim��sleep�� + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml + runtime: 1h + ��vim��runtime�� hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # ��submit�� hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

2 2

[PATCH v7 compass-ci] docker-rootfs: add different install methods
by Wang Chenglong 24 Dec '20

24 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image index 787a42c..62e1c05 100755 --- a/container/docker-rootfs/setup-image +++ b/container/docker-rootfs/setup-image @@ -2,10 +2,19 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. $LKP_SRC/lib/env.sh + install_packages() { - yum install -y --skip-broken $(</tmp/packages-to-install) - yum clean all + if [ has_cmd yum ]; then + yum install -y --skip-broken $(</tmp/packages-to-install) + yum clean all + elif [ has_cmd apt-get ]; then + apt update + apt-get install -y --fix-missing $(</tmp/packages-to-install) + elif [ has_cmd pacman ]; then + pacman -Syu --noconfirm $(</tmp/packages-to-install) + fi } remove_file() -- 2.23.0

1 1

[PATCH v2 compass-ci] update document
by Luan Shengde 24 Dec '20

24 Dec '20

update documents with new email address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- README.en.md | 2 +- README.zh.md | 2 +- doc/mail-robot.md | 2 +- doc/manual/apply-account.md | 2 +- doc/manual/borrow-machine.en.md | 2 +- doc/manual/borrow-machine.zh.md | 2 +- ...50\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" | 2 +- 7 files changed, 7 insertions(+), 7 deletions(-) diff --git a/README.en.md b/README.en.md index c082167..632a480 100644 --- a/README.en.md +++ b/README.en.md @@ -44,7 +44,7 @@ If a new error ID is generated, the bisect is automatically triggered to locate **Logging in to the Test Environment** -1. Send an email to the compass-ci(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. +1. Send an email to the compass-ci-reply(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. 2. [Complete the environment configuration](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma… as instructed by the email. 3. Add the **sshd** field to the test task, submit the task, and [log in to the test environment](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu…. diff --git a/README.zh.md b/README.zh.md index c2140b0..1339ef0 100644 --- a/README.zh.md +++ b/README.zh.md @@ -47,7 +47,7 @@ Compass-CI 监控很多开源软件 git repos，一旦检测到代码更新， **登录测试环境** -1. 向 compass-ci(a)qq.com 发送邮件[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual… +1. 向 compass-ci-reply(a)qq.com 发送邮件[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual… 2. 根据邮件反馈内容[完成环境配置](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/… 3. 在测试任务中添加 sshd 字段，提交相应的任务，[登录测试环境](https://gitee.com/wu_fengguang/compass-ci/blob/master/d… diff --git a/doc/mail-robot.md b/doc/mail-robot.md index d3778e7..e97912c 100644 --- a/doc/mail-robot.md +++ b/doc/mail-robot.md @@ -11,7 +11,7 @@ apply account/uuid for user ## usage build email as the following section: allowed email format -send mail to: compass-ci(a)qq.com +send mail to: compass-ci-reply(a)qq.com ## allowed email format: diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index 7e1a6a6..08e7618 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -6,7 +6,7 @@ Email template: --- From: {{ David Rientjes <rientjes(a)google.com> }} - To: compass-ci(a)qq.com + To: compass-ci-reply(a)qq.com Subject: apply account my oss commit: {{ https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… }} diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index b9c2bc4..bed9fcc 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -3,7 +3,7 @@ Ensure that you have performed the following operations according to the [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…: - Send an email to apply for an account. -- Receive an email from compass-ci(a)qq.com. +- Receive an email from compass-ci-reply(a)qq.com. - Configure the local environment. # Applying for a Test Machine (VM) diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 06d4db6..3f500e2 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -2,7 +2,7 @@ 请确认您已按照 [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… - send apply account email. -- receive email from compass-ci(a)qq.com. +- receive email from compass-ci-reply(a)qq.com. - local environment configuration. diff --git "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" index dd115f7..6f0736b 100644 --- "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" +++ "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" @@ -6,7 +6,7 @@ Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客 :exclamation: 前提条件：按照 [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… 完成如下动作： - send apply account email -- receive email from compass-ci(a)qq.com +- receive email from compass-ci-reply(a)qq.com - get reply email and follow its instructions to - setup default config -- 2.23.0

1 0

[PATCH compass-ci] update document
by Luan Shengde 24 Dec '20

24 Dec '20

update documents with new email address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- README.en.md | 2 +- README.zh.md | 2 +- doc/mail-robot.md | 2 +- doc/manual/apply-account.md | 2 +- doc/manual/borrow-machine.en.md | 2 +- doc/manual/borrow-machine.zh.md | 2 +- ...50\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" | 2 +- 7 files changed, 7 insertions(+), 7 deletions(-) diff --git a/README.en.md b/README.en.md index c082167..632a480 100644 --- a/README.en.md +++ b/README.en.md @@ -44,7 +44,7 @@ If a new error ID is generated, the bisect is automatically triggered to locate **Logging in to the Test Environment** -1. Send an email to the compass-ci(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. +1. Send an email to the compass-ci-reply(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. 2. [Complete the environment configuration](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma… as instructed by the email. 3. Add the **sshd** field to the test task, submit the task, and [log in to the test environment](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu…. diff --git a/README.zh.md b/README.zh.md index c2140b0..1339ef0 100644 --- a/README.zh.md +++ b/README.zh.md @@ -47,7 +47,7 @@ Compass-CI �� git repos�� **��** -1. �� compass-ci(a)qq.com ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md)�� +1. �� compass-ci-reply(a)qq.com ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md)�� 2. ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E6%9C%AC%E5%9C%B0%E5%AE%89%E8%A3%85compass-ci%E5%AE%A2%E6%88%B7%E7%AB%AF.md)�� 3. �� sshd ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E5%A6%82%E4%BD%95%E7%99%BB%E5%BD%95%E6%B5%8B%E8%AF%95%E6%9C%BA%E8%B0%83%E6%B5%8B%E4%BB%BB%E5%8A%A1.md)�� diff --git a/doc/mail-robot.md b/doc/mail-robot.md index d3778e7..e97912c 100644 --- a/doc/mail-robot.md +++ b/doc/mail-robot.md @@ -11,7 +11,7 @@ apply account/uuid for user ## usage build email as the following section: allowed email format -send mail to: compass-ci(a)qq.com +send mail to: compass-ci-reply(a)qq.com ## allowed email format: diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index 7e1a6a6..08e7618 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -6,7 +6,7 @@ Email template: --- From: {{ David Rientjes <rientjes(a)google.com> }} - To: compass-ci(a)qq.com + To: compass-ci-reply(a)qq.com Subject: apply account my oss commit: {{ https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… }} diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index b9c2bc4..bed9fcc 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -3,7 +3,7 @@ Ensure that you have performed the following operations according to the [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…: - Send an email to apply for an account. -- Receive an email from compass-ci(a)qq.com. +- Receive an email from compass-ci-reply(a)qq.com. - Configure the local environment. # Applying for a Test Machine (VM) diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 06d4db6..3f500e2 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -2,7 +2,7 @@ �� [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… - send apply account email. -- receive email from compass-ci(a)qq.com. +- receive email from compass-ci-reply(a)qq.com. - local environment configuration. diff --git "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" index dd115f7..6f0736b 100644 --- "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" +++ "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" @@ -6,7 +6,7 @@ Compass-CI �� [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) �� :exclamation: �� [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… �� - send apply account email -- receive email from compass-ci(a)qq.com +- receive email from compass-ci-reply(a)qq.com - get reply email and follow its instructions to - setup default config -- 2.23.0

1 0

[PATCH compass-ci] doc: and runtime field in borrow-machine.en.md and borrow-machine.zh.md
by Zhang Yale 24 Dec '20

24 Dec '20

User can specify the sleep time in "runtime" field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/manual/borrow-machine.en.md | 9 +++++---- doc/manual/borrow-machine.zh.md | 9 +++++---- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index b9c2bc4..cfb67aa 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -203,6 +203,7 @@ Ensure that you have performed the following operations according to the [apply- suite: borrow testcase: borrow + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -212,10 +213,10 @@ Ensure that you have performed the following operations according to the [apply- %> sshd: # sleep at the bottom - sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # Use the VIM editor to change the value of the sleep field. + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml + runtime: 1h + # Use the VIM editor to change the value of the runtime field. hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # After changing the value, submit the request again. hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 06d4db6..66f6876 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -207,6 +207,7 @@ suite: borrow testcase: borrow + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -216,10 +217,10 @@ %> sshd: # sleep at the bottom - sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # ��vim��sleep�� + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml + runtime: 1h + ��vim��runtime�� hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # ��submit�� hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

2 2

[PATCH v6 compass-ci] docker-rootfs: add different install methods
by Wang Chenglong 24 Dec '20

24 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image index 787a42c..ffe4937 100755 --- a/container/docker-rootfs/setup-image +++ b/container/docker-rootfs/setup-image @@ -4,8 +4,15 @@ install_packages() { - yum install -y --skip-broken $(</tmp/packages-to-install) - yum clean all + if [ $(command -v yum) >/dev/null ]; then + yum install -y --skip-broken $(</tmp/packages-to-install) + yum clean all + elif [ $(command -v apt-get) >/dev/null ]; then + apt update + apt-get install -y --fix-missing $(</tmp/packages-to-install) + elif [ $(command -v pacman) >/dev/null ]; then + pacman -Syu --noconfirm $(</tmp/packages-to-install) + fi } remove_file() -- 2.23.0

2 2

[PATCH v4 compass-ci] fix: qcow2rootfs/bin/common: unable to understand vmlinux
by Wang Chenglong 24 Dec '20

24 Dec '20

[error] [WARNNING] Restoring the owner of qcow2 file Finding vmlinuz under /tmp/a/boot ... /c/compass-ci/container/qcow2rootfs/bin/common: line 135: ROOTFS_VMLINUZ_FILE: unbound variable [why] some kernels are called vmlinux instead of vmlinuz. --- container/qcow2rootfs/bin/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/qcow2rootfs/bin/common b/container/qcow2rootfs/bin/common index da39f52..baf936c 100755 --- a/container/qcow2rootfs/bin/common +++ b/container/qcow2rootfs/bin/common @@ -127,7 +127,7 @@ get_rootfs_kernel() { exit 5 } local vmlinuz_file vmlinuz kernel - vmlinuz_file=$(find ./boot -name "vmlinuz-*" |grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz_file=$(find ./boot -name "vmlinu[z|x]-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel -- 2.23.0

2 2

[PATCH v2 compass-ci] modify assign-account-email
by Li Ping 24 Dec '20

24 Dec '20

[why]: source /root/.bashrc && /root/.bash_profile follow this step can not success. Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/mail-robot/lib/assign-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index 77ed4e8..466586d 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -37,7 +37,7 @@ def build_apply_account_email(my_info) git clone https://gitee.com/wu_fengguang/lkp-tests.git cd lkp-tests make install - source ${HOME}/.bashrc && source ${HOME}/.bash_profile + source \\${HOME}/.\\${SHELL##*/}rc 3) submit job reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md -- 2.23.0

4 10

[PATCH compass-ci] doc/manual: update apply-account.md
by Luan Shengde 24 Dec '20

24 Dec '20

used new email address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/apply-account.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index 7e1a6a6..08e7618 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -6,7 +6,7 @@ Email template: --- From: {{ David Rientjes <rientjes(a)google.com> }} - To: compass-ci(a)qq.com + To: compass-ci-reply(a)qq.com Subject: apply account my oss commit: {{ https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… }} -- 2.23.0

1 0

[PATCH v2 lkp-tests] lib/job.rb: fix job symbol key check
by Wei Jihui 24 Dec '20

24 Dec '20

[why] 1. load job from result's job yaml has key which is start with "#! ", like '#! hosts/vm-2p16g': this key should be deleted. 2. load job from a yaml maybe has symbol key, like # key ---> :'# key': nil but symbol key does not has start_with function. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 7553312e..35e0ceee 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -254,7 +254,6 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end - hash.delete_if { |key, _| key.start_with?('#!') } hash.merge!(@overrides) @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end @@ -290,6 +289,12 @@ class Job jobs_array = [] hash.each { |key, value| next unless key =~ /^\w.*\|.*\w$/ + next unless key.is_a?(String) + + if key.start_with?('#! ') + hash.delete(key) + next + end key_array = key.split('|').map(&:strip) [value].flatten.each do |v| -- 2.23.0

2 1

[PATCH compass-ci] sbin: optimize build-my-info.rb
by Luan Shengde 24 Dec '20

24 Dec '20

mv config part to lib/build_my_info_client.rb lib/build_my_info_client will enable been called to build my_info Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/build-my-info.rb | 42 +++++------------------------------------- 1 file changed, 5 insertions(+), 37 deletions(-) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index d75f767..40193af 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -3,48 +3,16 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -require 'json' -require 'fileutils' require 'io/console' require_relative '../lib/es_client' +require_relative '../lib/build_my_info_client' print 'email: ' my_email = $stdin.echo = gets.chomp print 'name: ' my_name = $stdin.echo = gets.chomp -my_token = %x(uuidgen).chomp +print 'lab: ' +lab = $stdin.echo = gets.chomp -my_info = { - 'my_name' => my_name, - 'my_email' => my_email, - 'my_token' => my_token -} - -def store_account_info(my_info) - es = ESClient.new(index: 'accounts') - es.put_source_by_id(my_info['my_email'], my_info) -end - -def config_default_yaml(my_info) - yaml_dir = "#{ENV['HOME']}/.config/compass-ci/defaults" - FileUtils.mkdir_p yaml_dir unless File.directory? yaml_dir - yaml_file = "#{yaml_dir}/account.yaml" - FileUtils.touch(yaml_file) unless File.exist? yaml_file - - yaml_info = YAML.load_file(yaml_file) || {} - yaml_info.update my_info - - File.open(yaml_file, 'w') do |f| - f.puts yaml_info.to_yaml - end -end - -def complete_my_info(my_info) - my_info['my_login_name'] = '' - my_info['my_commit_url'] = '' - my_info['my_ssh_pubkey'] = [] -end - -config_default_yaml(my_info) -complete_my_info(my_info) -store_account_info(my_info) +build_my_info = BuildMyInfo.new(my_email, my_name, lab) +build_my_info.config_my_info -- 2.23.0

1 0

[PATCH lkp-tests] fix: extra empty line
by Ren Wen 24 Dec '20

24 Dec '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- tests/host-info | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/host-info b/tests/host-info index 0ab8e6c97..e0c866257 100755 --- a/tests/host-info +++ b/tests/host-info @@ -106,7 +106,6 @@ get_ipmi_ip() is_docker && return local ip=$(ipmitool lan print | awk 'NR==3{print $4}') [ -n "$ip" ] && echo "ipmi_ip: $ip" - } main() -- 2.23.0

2 2