- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] lib/matrix2.rb: refactor performance optimization
by Lu Kaiyi 28 Nov '20

28 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 82cf033..2507b77 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -61,9 +61,7 @@ def create_stats(result_root) end def samples_fill_missing_zeros(value, size) - samples = value || [0] * size - samples << 0 while samples.size < size - samples + value.concat([0] * (size - value.size)) end # input: job_list -- 2.23.0

1 0

[PATCH compass-ci] scheduler: fix condition when setting default queue
by Ren Wen 28 Nov '20

28 Nov '20

Hash.*has_key?* returns true although there is no value of the key. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index e836d1f..55bac7f 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -247,7 +247,7 @@ class Job end private def set_queue - return if @hash.has_key?("queue") + return unless self["queue"].empty? # set default value self["queue"] = tbox_group -- 2.23.0

3 3

[PATCH compass-ci] answerback-email: fix bugs
by Luan Shengde 28 Nov '20

28 Nov '20

1. add default option for tool [why] the tool will show help message if no option specified [how] if no option specified, it will add '-h' as default option 2. fix undefined method 'empty' [why] the original data is defined nil, but not empty for my_email when execute answerback-email with no -e or -f, it will throw error as: raceback (most recent call last): 2: from ./answerback-email.rb:213:in `<main>' 1: from ./answerback-email.rb:184:in `send_account' ./answerback-email.rb:155:in `check_my_email': undefined method `empty?' for nil:NilClass (NoMethodError) [how] use '.nil?' instead of '.empty?' Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index c6bfdd3..1aaad75 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -136,7 +136,10 @@ options = OptionParser.new do |opts| end end -options.parse!(ARGV) +if ARGV.empty? + ARGV << '-h' + options.parse!(ARGV) +end def apply_account(my_info, conf_info) apply_info = {} @@ -152,7 +155,7 @@ def check_my_email(my_info) message += "use -e to add an email address for applying account\n" message += 'or use -f to add an email file' - raise message if my_info['my_email'].empty? + raise message if my_info['my_email'].nil? end def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) -- 2.23.0

1 0

[PATCH v2 compass-ci] scheduler: fix the paths to find cluster config file
by Ren Wen 27 Nov '20

27 Nov '20

To find cluster config file at *lab-z9/cluster/* first, *lkp-tests/cluster/* otherwise. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/constants.cr | 3 +++ src/scheduler/submit_job.cr | 14 +++++++++----- 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 1b37333..1afdda0 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -24,6 +24,9 @@ OS_HTTP_PORT = (ENV.has_key?("OS_HTTP_PORT") ? ENV["OS_HTTP_PORT"] : 8000).to_i3 SRV_OS = "/srv/os" SRV_INITRD = "/srv/initrd" +CCI_REPOS = "/c" +LAB_REPO = "lab-z9" + INITRD_HTTP_PREFIX = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" OS_HTTP_PREFIX = "http://#{OS_HTTP_HOST}:#{OS_HTTP_PORT}" SCHED_HTTP_PREFIX = "http://#{SCHED_HOST}:#{SCHED_PORT}" diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 2836275..c60b99d 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -15,7 +15,7 @@ class Sched else cluster_config = get_cluster_config(cluster_file, job.lkp_initrd_user, - job.os_arch) + job.os_arch).not_nil! return submit_cluster_job(job, cluster_config) end rescue ex @@ -152,12 +152,16 @@ class Sched @es.set_job_content(job) end - # get cluster config using own lkp_src cluster file, - # a hash type will be returned def get_cluster_config(cluster_file, lkp_initrd_user, os_arch) lkp_src = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) - cluster_file_path = Path.new(lkp_src, "cluster", cluster_file) - return YAML.parse(File.read(cluster_file_path)) + + cluster_file_paths = [ + Path.new(CCI_REPOS, LAB_REPO, "cluster", cluster_file), + Path.new(lkp_src, "cluster", cluster_file) + ] + cluster_file_paths.each do |f| + return YAML.parse(File.read(f)) if File.file?(f) + end end def get_commit_date(job) -- 2.23.0

1 0

[PATCH v2 compass-ci 4/4] user-client/maintain: send report mail
by Liu Yinsi 27 Nov '20

27 Nov '20

call $CCI_SRC/lib/mail_client.rb to mail. 'LOG' is a file => walk-test.report, job result will write in it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- user-client/maintain/walk-os-test/walk-os-iperf-test | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index 17edb60..923d1ed 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -36,6 +36,17 @@ OS_MOUNT = %w[cifs initramfs].freeze EMAIL_ADDRESS = (ENV['USER']).to_s.freeze +def mail_report + message = File.read(LOG).chomp + data = " + subject: os rootfs test report + to: #{EMAIL_ADDRESS} + body: #{message}" + + email = MailClient.new() + email.send_mail(data) +end + def write_report(report_all) File.open(LOG, 'a') do |f| log = Logger.new(f, 'weekly') -- 2.23.0

1 0

[PATCH v2 compass-ci 3/4] user-client/maintain: write job result into report
by Liu Yinsi 27 Nov '20

27 Nov '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- user-client/maintain/walk-os-test/walk-os-iperf-test | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index 202908e..17edb60 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -36,6 +36,16 @@ OS_MOUNT = %w[cifs initramfs].freeze EMAIL_ADDRESS = (ENV['USER']).to_s.freeze +def write_report(report_all) + File.open(LOG, 'a') do |f| + log = Logger.new(f, 'weekly') + log.formatter = proc { |_severity, datetime, _progname, msg| + "[#{datetime}]: #{msg}\n" + } + log.info(report_all) + end +end + def monitor(query, actions, timeout) monitor = Monitor.new monitor.overrides = query -- 2.23.0

1 0

[PATCH v2 compass-ci 2/4] user-client/maintain: get job result by monitor and es query
by Liu Yinsi 27 Nov '20

27 Nov '20

1. monitor job, start time => job_state is 'boot', end time => job_state is 'extract_finished'; 2. query es by field 'job_state' and 'result_root'. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- .../maintain/walk-os-test/walk-os-iperf-test | 38 +++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index ba142d0..202908e 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -36,6 +36,44 @@ OS_MOUNT = %w[cifs initramfs].freeze EMAIL_ADDRESS = (ENV['USER']).to_s.freeze +def monitor(query, actions, timeout) + monitor = Monitor.new + monitor.overrides = query + monitor.action = actions + monitor.run(timeout) + monitor +end + +def monitor_state(job_id) + time_list = [] + + %w[boot extract_finished].each do |state| + result_array = monitor({ 'job_id' => job_id, 'job_state' => state }, { 'stop' => true }, '1800').result + time_list.push(Time.parse(result_dict[0]['time'])) unless result_array.empty? + end + time_list +end + +def es_search(job_id) + es_list = [] + + es = ESQuery.new + result_dict = es.query_by_id(job_id) + %w[job_state result_root].each do |field| + es_list.push(result_dict[field]) unless result_dict.nil? + end + es_list +end + +def find_by_id(job_id) + start_time, end_time = monitor_state(job_id) + cost_time = end_time - start_time unless end_time.nil? + + job_state, result_root = es_search(job_id) + + return start_time, end_time, cost_time.to_i.floor, job_state, result_root +end + def run_qemu Process.fork do %x(#{ENV['CCI_SRC']}/providers/my-qemu.sh >/dev/null 2>&1) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/4] refactor: user-client/maintain: walk-os-iperf-test
by Liu Yinsi 27 Nov '20

27 Nov '20

1. submit job; 2. get job result by monitor and es query; 3. write job result into report; 4. send report mail; 5. cron task, test once an hours, mail once a week. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- .../maintain/walk-os-test/walk-os-iperf-test | 148 +++++++++++------- 1 file changed, 88 insertions(+), 60 deletions(-) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index 21e1428..ba142d0 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -1,62 +1,90 @@ -#!/bin/bash -e +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -# os test rounds -test_times=1 - -test_logs=walk-test.report -test_yaml=iperf-walk-os.yaml -test_os=( - 'openeuler aarch64 20' - 'openeuler aarch64 20.03' - 'centos aarch64 7.6' - 'centos aarch64 7.8' - 'centos aarch64 8.1.1911' - 'debian aarch64 sid' -) - -trap 'echo User Cancelled! && exit' SIGINT - -sec2date() { - date -d@$1 +"%Y%m%d_%H:%M:%S" -} - -logging() { - if [ 'new' == "$1" ] ; then - printf "%-5s %-8s %-25s %-20s %-20s %-8s %-10s\n" ${@:2} > $test_logs - else - printf "%-5s %-8s %-25s %-20s %-20s %-8s %-10s\n" $@ >> $test_logs - fi -} - -create_test_yaml() { - local hwarch os os_arch os_ver cur_os - hwarch=$(arch) - cur_os="$@" - os=$(echo $cur_os |cut -d' ' -f1) - os_arch=$(echo $cur_os |cut -d' ' -f2) - os_ver=$(echo $cur_os |cut -d' ' -f3) - sed "s/USER/$USER/" $test_yaml > test.yaml - sed -i "s/OS_ARCH/$os_arch/" test.yaml - sed -i "s/OS_VER/$os_ver/" test.yaml - sed -i "s/OS/$os/" test.yaml - echo "$hwarch $os/$os_arch/$os_ver" -} - -job_test() { - local job_id bef aft - job_id=$(submit test.yaml |awk '{print $NF}') - bef=$(date +%s) - ./my-qemu.sh - aft=$(date +%s) - logging "$1" "$2" "$3" "$(sec2date $bef)" "$(sec2date $aft)" "$((aft - bef))" "$job_id" -} - -logging new "Index" "hwArch" "OS" "Begin" "End" "Cost/sec" "Job_ID" -for round in $(seq $test_times) -do - for idx in "${!test_os[@]}" - do - job_test $round $(create_test_yaml "${test_os[$idx]}") - done -done +# rootfs test via job, including cifs, initramfs files +# boot way kvm. +# input: job yaml, os, os_version, os_arch, os_mount, testbox. +# output: a test report, including log print time, job ID, os, runtime, job state and url to check logs, and send mail. +# Eg: test report format: +# [2020-11-26 19:22:22 +0800]: ["crystal.143338", "centos", "aarch64", "7.6.1810", "cifs", 2020-11-26 19:21:12 +0800, +# 2020-11-26 19:22:21 +0800, 69, "finished", "/result/iperf-walk-os-test/vm-2p8g/2020-11-26/crystal.143338"] + +require "#{ENV['CCI_SRC']}/lib/es_query" +require "#{ENV['LKP_SRC']}/lib/monitor" +require "#{ENV['CCI_SRC']}/lib/mail_client" + +require 'time' +require 'logger' +require 'rufus-scheduler' + +TEST_OS = ['openeuler aarch64 20.03', + 'centos aarch64 7.6.1810', + 'centos aarch64 7.8.2003', + 'centos aarch64 8.1.1911', + 'debian aarch64 sid', + 'archlinux aarch64 2020-11-12'].freeze + +TEST_YAML = 'iperf-walk-os.yaml' + +TESTBOX = 'vm-2p8g' + +LOG = 'walk-test.report' + +OS_MOUNT = %w[cifs initramfs].freeze + +EMAIL_ADDRESS = (ENV['USER']).to_s.freeze + +def run_qemu + Process.fork do + %x(#{ENV['CCI_SRC']}/providers/my-qemu.sh >/dev/null 2>&1) + end +end + +def submit_job(os_str, os_mount) + `submit --my-queue 'os os_arch os_version=#{os_str}' 'testbox=#{TESTBOX}' 'os_mount=#{os_mount}' #{TEST_YAML}` +end + +def traversal_os_mount + OS_MOUNT.each do |os_mount| + traversal_test_os(os_mount) + end +end + +def traversal_test_os(os_mount) + TEST_OS.each do |os_str| + test_rootfs(os_str, os_mount) + end +end + +def test_rootfs(os_str, os_mount) + message = submit_job(os_str, os_mount).chomp + job_id = message.split('=')[1] + run_qemu + + job_result = if message.include? 'error' + [] + else + find_by_id(job_id) + end + + report_all = job_id.split(', ') + os_str.split + os_mount.split + job_result + write_report(report_all) +end + +def cron_job + test = Rufus::Scheduler.new + test.cron '0 8-22 * * *' do + traversal_os_mount + end + + mail = Rufus::Scheduler.new + mail.cron '0 8 * * 1' do + mail_report + end + + [test, mail].each(&:join) +end + +cron_job -- 2.23.0

1 0

[PATCH v5 compass-ci 3/3] openeuler: build openeuler initramfs image with one command
by Wang Chenglong 27 Nov '20

27 Nov '20

[Why] Use the tool to create a new openeuler-$version initramfs image with one command, and we also can customize the pre-installed software by configuring ./packages-to-install. [How] We create a link button for the user to use. $CCI_SRC/rootfs/initramfs/openeuler/aarch64/${os_version}/build --- container/osimage/openeuler/run | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100755 container/osimage/openeuler/run diff --git a/container/osimage/openeuler/run b/container/osimage/openeuler/run new file mode 100755 index 0000000..5d7da72 --- /dev/null +++ b/container/osimage/openeuler/run @@ -0,0 +1,28 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ${CCI_SRC}/container/osimage/openeuler/lib +. ${CCI_SRC}/container/defconfig.sh + +export_root_passwd +DIR=$(dirname $(realpath $0)) + +cmd=( + docker run + --name initramfs_${image} + -v $DIR/create-image:/root/bin/create-image + -v $DIR/packages-to-install:/tmp/packages-to-install + -v $DIR/files-to-exclude:/tmp/files-to-exclude + -v $DIR/${image}.repo:/etc/yum.repos.d/openEuler.repo + -e ROOT_PASSWD=$ROOT_PASSWD + -e IMAGE_NAME=$IMAGE_NAME + $image + /root/bin/create-image +) + +load_docker_image +"${cmd[@]}" +cp_image_to_host +docker_rm init_docker &> /dev/null +echo "build finished" -- 2.23.0

1 0

[PATCH v5 compass-ci 2/3] openeuler: generate the cgz file in container
by Wang Chenglong 27 Nov '20

27 Nov '20

Install packages, set rootfs, pack images package and other actions in docker. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/create-image | 37 ++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/osimage/openeuler/create-image diff --git a/container/osimage/openeuler/create-image b/container/osimage/openeuler/create-image new file mode 100755 index 0000000..631bb14 --- /dev/null +++ b/container/osimage/openeuler/create-image @@ -0,0 +1,37 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +install_packages() +{ + yum -y install --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +setup_rootfs() +{ + ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime + rm -f /etc/machine-id /var/lib/dbus/machine-id + systemd-machine-id-setup +} + +setup_login() { + [ -n "$ROOT_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +pack_cgz() +{ + echo "Packing package. Please wait." + find / ! -path "/tmp/${IMAGE_NAME}" | grep -vf /tmp/files-to-exclude | cpio -o -Hnewc | gzip -9 > /tmp/${IMAGE_NAME} + chmod 644 /tmp/${IMAGE_NAME} +} + +install_packages +setup_rootfs +setup_login +pack_cgz -- 2.23.0

1 0