March 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 1/4] sparrow/install-client: add deploy compass-ci client script
by Liu Yinsi 03 Mar '21

03 Mar '21

deploy compass-ci client, provides function: submit job to compass-ci server and boot testbox. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-client | 71 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100755 sparrow/install-client diff --git a/sparrow/install-client b/sparrow/install-client new file mode 100755 index 0000000..8d69267 --- /dev/null +++ b/sparrow/install-client @@ -0,0 +1,71 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +[[ $CCI_SRC ]] || export CCI_SRC=$(cd $(dirname $(realpath $0)); git rev-parse --show-toplevel) + +. $CCI_SRC/lib/log.sh + +check_server_ip() +{ + [ -z "$server_ip" ] && . 0-package/read-config + [ -z "$server_ip" ] && { + log_error "env \$server_ip is not exists, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 + } +} + +install_env() +{ + cd /c/compass-ci/sparrow || return + check_server_ip + 0-package/install + 1-storage/tiny + 5-build/ipxe & + 1-storage/permission + 2-network/br0 + 2-network/iptables + 3-code/git + 3-code/dev-env + . /etc/profile.d/compass.sh + 5-build/register-account +} + +boot_ipxe() +{ + SCHED_HOST=$server_ip + sed -i "s%172.17.0.1%${SCHED_HOST}%g" /tftpboot/boot.ipxe + sed -i "s%3000%${SCHED_PORT:-3000}%g" /tftpboot/boot.ipxe +} + +run_service() +{ + ( + cd $CCI_SRC/container/dnsmasq || return + ./build + ./start + boot_ipxe + )& + ( + cd $CCI_SRC/container/qemu-efi || return + ./build + ./install + )& + ( + cd $CCI_SRC/container/fluentd-base || return + ./build + + cd $CCI_SRC/container/sub-fluentd || return + ./build + ./start + )& +} + +main() +{ + install_env + run_service +} + +main +wait -- 2.23.0

2 2

[PATCH compass-ci] service/scheduler: set lifecycle info when boot
by Wu Zhende 03 Mar '21

03 Mar '21

When a machine calls boot API, no matter whether the job is retrieved or not, it indicates that the machine is alive. Therefore, we need to update the ES storage information. If the job is obtained, set the deadline. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 8 ++++++++ src/lib/sched.cr | 37 +++++++++++++++++++--------------- src/scheduler/find_job_boot.cr | 3 ++- 3 files changed, 31 insertions(+), 17 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 63ec8fb..f4ca5de 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -242,6 +242,14 @@ class Job self["rootfs"] = "#{os}-#{os_version}-#{os_arch}" end + def get_deadline + runtime = (self["timeout"]? || self["runtime"]?).to_s + runtime = 1800 if runtime.empty? + + # reserve 300 seconds for system startup, hw machine will need such long time + (Time.local + (runtime.to_i32 * 2 + 300).second).to_s("%Y-%m-%dT%H:%M:%S+0800") + end + def set_result_root update_tbox_group_from_testbox # id must exists, need update tbox_group self["result_root"] = File.join("/result/#{suite}/#{submit_date}/#{tbox_group}/#{rootfs}", "#{pp_params}", "#{id}") diff --git a/src/lib/sched.cr b/src/lib/sched.cr index dfec365..5f28de9 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -103,12 +103,15 @@ class Sched @log.warn(e) end + def get_time + Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800") + end + def update_tbox_wtmp testbox = "" hash = Hash(String, String | Nil).new - time = Time.local.to_s("%Y-%m-%d %H:%M:%S") - hash["time"] = time + hash["time"] = get_time %w(mac ip job_id tbox_name tbox_state).each do |parameter| if (value = @env.params.query[parameter]?) @@ -117,7 +120,6 @@ class Sched testbox = value when "tbox_state" hash["state"] = value - hash["deadline"] = nil if value == "rebooting" when "mac" hash["mac"] = normalize_mac(value) else @@ -136,23 +138,26 @@ class Sched @log.warn(e) end - def set_tbox_boot_wtmp(job : Job) - time = Time.local - booting_time = time.to_s("%Y-%m-%dT%H:%M:%S") - - runtime = (job["timeout"]? || job["runtime"]?).to_s - runtime = 1800 if runtime.empty? + def set_lifecycle(job, testbox) + if job + deadline = job.get_deadline + job["deadline"] = deadline + job["job_state"] = "boot" + state = "booting" + job_id = job["id"] + else + deadline = nil + job_id = "" + state = "requesting" + end - # reserve 300 seconds for system startup, hw machine will need such long time - deadline = (time + (runtime.to_i32 * 2 + 300).second).to_s("%Y-%m-%dT%H:%M:%S") hash = { - "job_id" => job["id"], - "state" => "booting", - "booting_time" => booting_time, + "job_id" => job_id, + "state" => state, + "time" => get_time, "deadline" => deadline } - - @es.update_tbox(job["testbox"], hash) + @es.update_tbox(testbox.to_s, hash) end def report_ssh_port diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 3323d4d..b7fe0e7 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -100,10 +100,11 @@ class Sched def get_job_boot(host, boot_type) queues = get_queues(host) job = get_job_from_queues(queues, host) + set_lifecycle(job, host) if job + @es.set_job_content(job) create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) - set_tbox_boot_wtmp(job) else # for physical machines spawn { auto_submit_idle_job(host) } -- 2.23.0

1 0

[PATCH compass-ci] lib: parse mail list from yaml by type as to and bcc mail list
by Cao Xueliang 03 Mar '21

03 Mar '21

Yaml file define: delimiter: to: mai1, mail3 bcc: mail2, mail3 The mail_bisect_result calls parse_mail_list('delimiter') to get the delimiter mail list. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 14 +++++++++++--- lib/mail_list.yaml | 3 +++ lib/parse_mail_list.rb | 13 +++++++++++++ 3 files changed, 27 insertions(+), 3 deletions(-) create mode 100644 lib/mail_list.yaml create mode 100644 lib/parse_mail_list.rb diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 104f098..23a8ade 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -6,6 +6,7 @@ require_relative 'git' require_relative 'es_query' require_relative 'constants' require_relative 'mail_client' +require_relative 'parse_mail_list' require_relative 'assign_account_client' # compose and send email for bisect result @@ -22,10 +23,17 @@ class MailBisectResult @git_commit = GitCommit.new(@work_dir, @commit_id) @to = @git_commit.author_email # now send mail to review - @bcc = 'caoxl(a)crystal.ci, caoxl78320(a)163.com, huming15(a)163.com, wfg(a)mail.ustc.edu.cn' + @bcc = 'caoxl78320(a)163.com, huming15(a)163.com, wfg(a)mail.ustc.edu.cn' + end + + def parse_mail_info + mail_hash = parse_mail_list('delimiter') + @to = mail_hash['to'] if mail_hash.key?('to') + @bcc = mail_hash['bcc'] if mail_hash.key?('bcc') end def create_send_email + parse_mail_info send_report_mail(compose_mail) send_account_mail rm_work_dir @@ -35,7 +43,7 @@ class MailBisectResult subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" prefix_srv = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_PORT}" bisect_job_url = ENV['result_root'] ? "bisect job result directory:\n#{prefix_srv}#{ENV['result_root']}\n" : '' - bisect_report_doc = "bisect email doc:\nhttps://gitee.com/wu_fengguang/compass-ci/blob/master/doc/bisect_emai…" + report_doc = "bisect email doc:\nhttps://gitee.com/wu_fengguang/compass-ci/blob/master/doc/bisect_emai…" pkgbuild_repo_url = "PKGBUILD:\n#{prefix_srv}/git/#{@pkgbuild_repo}\n" first_bad_commit_job_url = "first bad commit job result directory:\n#{prefix_srv}#{@first_bad_commit_result_root}\n" @@ -58,7 +66,7 @@ class MailBisectResult #{pkgbuild_repo_url} #{first_bad_commit_job_url} #{bisect_job_url} - #{bisect_report_doc} + #{report_doc} Regards, Compass CI team BODY diff --git a/lib/mail_list.yaml b/lib/mail_list.yaml new file mode 100644 index 0000000..d982c93 --- /dev/null +++ b/lib/mail_list.yaml @@ -0,0 +1,3 @@ +delimiter: + to: caoxl78320(a)163.com + bcc: caoxl78320(a)163.com, wfg(a)mail.ustc.edu.cn, ljx.joe(a)qq.com, huming15(a)163.com diff --git a/lib/parse_mail_list.rb b/lib/parse_mail_list.rb new file mode 100644 index 0000000..49b5c88 --- /dev/null +++ b/lib/parse_mail_list.rb @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# frozen_string_literal: true + +require 'yaml' + +def parse_mail_list(type) + content = {} + mail_list_yaml = './mail_list.yaml' + content = YAML.safe_load(File.open(mail_list_yaml)) if FileTest.exists?(mail_list_yaml) + return {} unless content[type] + + return content[type] +end -- 2.23.0

2 5

[PATCH compass-ci] src/stats_worker:delete filter error_id and add log
by Cao Xueliang 02 Mar '21

02 Mar '21

We add the filter error_id in commit: 1c154d888fab1fce4623663b4a88b9a1630171c2, we don't need it now, so delete the filter error_id, add the new_error_id log for statistical. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/extract-stats/stats_worker.cr | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 01bae21..836d1f8 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -82,25 +82,20 @@ class StatsWorker } ) - error_id = select_error_id(check_new_error_ids(error_ids, job_id)) - if error_id + new_error_ids = check_new_error_ids(error_ids, job_id) + unless new_error_ids.empty? + sample_error_id = new_error_ids.sample STDOUT.puts "send a delimiter task: job_id is #{job_id}" - @tq.add_task(DELIMITER_TASK_QUEUE, JSON.parse({"error_id" => error_id, + @tq.add_task(DELIMITER_TASK_QUEUE, JSON.parse({"error_id" => sample_error_id, "job_id" => job_id, "lab" => LAB}.to_json)) + msg = %({"job_id": "#{job_id}", "new_error_id": "#{sample_error_id}"}) + system "echo '#{msg}'" end msg = %({"job_id": "#{job_id}", "job_state": "extract_finished"}) system "echo '#{msg}'" end - def select_error_id(new_error_ids : Array) - new_error_ids.each do |error_id| - return error_id if /(cpp|c|h):(warning|error)/i =~ error_id - end - - return nil - end - def check_new_error_ids(error_ids : Array, job_id : String) new_error_ids = [] of String error_ids.each do |error_id| -- 2.23.0

1 0

[PATCH v3 compass-ci 4/4] container/dracut-initrd: add 90lkp module into Dockerfile
by Xu Xijian 02 Mar '21

02 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/Dockerfile | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/dracut-initrd/Dockerfile b/container/dracut-initrd/Dockerfile index 7f8448f..b8c551f 100644 --- a/container/dracut-initrd/Dockerfile +++ b/container/dracut-initrd/Dockerfile @@ -6,6 +6,8 @@ FROM debian MAINTAINER Wu Fenguang <wfg(a)mail.ustc.edu.cn> ADD bin /usr/local/bin +ADD modules.d /usr/lib/modules.d +COPY modules.d/90lkp /usr/lib/dracut/modules.d/90lkp COPY conf/sources.list* /etc/apt/ COPY conf/add-modules.conf /etc/dracut.conf.d/ WORKDIR /usr/local/bin -- 2.23.0

1 0

[PATCH v3 compass-ci 2/4] container/dracut-initrd: README of module 90lkp
by Xu Xijian 02 Mar '21

02 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/modules.d/90lkp/README | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 container/dracut-initrd/modules.d/90lkp/README diff --git a/container/dracut-initrd/modules.d/90lkp/README b/container/dracut-initrd/modules.d/90lkp/README new file mode 100644 index 0000000..7d7a205 --- /dev/null +++ b/container/dracut-initrd/modules.d/90lkp/README @@ -0,0 +1,5 @@ +dracut lkp module + +This module is to deploy lkp module into rootfs. +Module lkp(Linux Kernel Performance) is a test framework, which is used +to make self-defined test for os kernel. -- 2.23.0

1 0

[PATCH v3 compass-ci 1/4] container/dracut-initrd: add module 90lkp/module-setup.sh
by Xu Xijian 02 Mar '21

02 Mar '21

90lkp module a module to deploy lkp module into rootfs. module-setup.sh is a base setup script for every kernel module. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- .../modules.d/90lkp/module-setup.sh | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 container/dracut-initrd/modules.d/90lkp/module-setup.sh diff --git a/container/dracut-initrd/modules.d/90lkp/module-setup.sh b/container/dracut-initrd/modules.d/90lkp/module-setup.sh new file mode 100755 index 0000000..c188600 --- /dev/null +++ b/container/dracut-initrd/modules.d/90lkp/module-setup.sh @@ -0,0 +1,19 @@ +#!/bin/bash + +check() { + return 0 +} + +depends() { + # We do not depend on any modules - just some root + return 0 +} + +# called by dracut +installkernel() { + return 0 +} + +install() { + inst_hook pre-pivot 10 "$moddir/lkp-deploy.sh" +} -- 2.23.0

1 0

[PATCH compass-ci 2/2] test2
by Xu Xijian 02 Mar '21

02 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/test2 | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 container/dracut-initrd/test2 diff --git a/container/dracut-initrd/test2 b/container/dracut-initrd/test2 new file mode 100644 index 0000000..e69de29 -- 2.23.0

1 1

[PATCH compass-ci 1/2] test1
by Xu Xijian 02 Mar '21

02 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/test1 | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 container/dracut-initrd/test1 diff --git a/container/dracut-initrd/test1 b/container/dracut-initrd/test1 new file mode 100644 index 0000000..e69de29 -- 2.23.0

1 1

[PATCH v2 compass-ci 2/2] container/dracut-initrd: add module 90lkp/lkp-deploy.sh
by Xu Xijian 02 Mar '21

02 Mar '21

lkp-deploy.sh does the concrete work of 90lkp, it adds the lkp scripts into testbox's rootfs to run test job. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- .../modules.d/90lkp/lkp-deploy.sh | 48 +++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100755 container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh diff --git a/container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh b/container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh new file mode 100755 index 0000000..e88959b --- /dev/null +++ b/container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh @@ -0,0 +1,48 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +. /lib/dracut-lib.sh + +# transfer LKP dirs +[ -d /lkp ] || return 0 + +cp -a /lkp "$NEWROOT"/ + +for i in /lkp/lkp/src/rootfs/addon/* /usr/local/* /opt/* +do + dir=$(basename "$i") + + [ "$i" != "${i%/\*}" ] && continue # skip: i='dir/*' + + [ -d "$NEWROOT/$dir" ] || mkdir -p "$NEWROOT/$dir" + + for j in "$i"/* "$i"/.??* + do + [ "$j" != "${j%/\*}" ] && continue # skip: j='dir/*' + + [ -f "$j" ] && { + cp -a "$j" "$NEWROOT/$dir"/ + continue + } + + subdir=$(basename "$j") + + [ -d "$NEWROOT/$dir/$subdir" ] || mkdir -p "$NEWROOT/$dir/$subdir" + + for k in "$j"/* + do + [ "$k" != "${k%/\*}" ] && continue # skip: k='dir/*' + + cp -a "$j"/* "$NEWROOT/$dir/$subdir"/ + done + done +done + +[ -d /usr/src ] && + cp -a /usr/src "$NEWROOT"/usr/ + +kmdir=/lib/modules/$(uname -r) +if test -d "$kmdir" && ! test -d "$NEWROOT/$kmdir"; then + cp -an "$kmdir" "$NEWROOT"/lib/modules/ + cp -an /lib/firmware "$NEWROOT"/lib/ +fi -- 2.23.0

2 2