- Compass-ci - mailweb.openeuler.org

[PATCH v4 compass-ci 2/4] container/submit: build file
by Luan Shengde 02 Feb '21

02 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/submit/build diff --git a/container/submit/build b/container/submit/build new file mode 100755 index 0000000..fba4281 --- /dev/null +++ b/container/submit/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t submit:latest . -- 2.23.0

1 0

[PATCH v4 compass-ci 1/4] container/submit: Dockerfile
by Luan Shengde 02 Feb '21

02 Feb '21

Install lkp-tests and its deps in the container instead of install them at localhost. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/Dockerfile | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 container/submit/Dockerfile diff --git a/container/submit/Dockerfile b/container/submit/Dockerfile new file mode 100644 index 0000000..5ea0fcc --- /dev/null +++ b/container/submit/Dockerfile @@ -0,0 +1,36 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +# it may take a long time, even break up if you use default source address to install pkgs. +# you can delete the following step if you do not want to use it. +RUN echo 'deb [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\n \ +\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\n' \ +> /etc/apt/sources.list + +RUN apt-get update && \ + apt-get install -y git ruby ruby-dev make libssl-dev gcc g++ sudo uuid-runtime + +# it may take a long time, even break up if you use default source address for the gem. +# delete the following step if you do not want to use the default one. +RUN gem sources --add https://gems.ruby-china.com/ && \ + gem sources --remove https://rubygems.org/ + +RUN gem install git faye-websocket activesupport rest-client + +WORKDIR /root + +RUN git clone https://gitee.com/wu_fengguang/lkp-tests.git +RUN cd lkp-tests && make install -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests/cci-depends: fix generate wrong symlink file
by Wang Yong 02 Feb '21

02 Feb '21

[before] $ ll $LKP_SRC/distro/depends/sysbench-cpu sysbench-cpu -> sysbench $ tree /srv/initrd/deps/$os/$os_arch/$os_version/ . ├── sysbench_20200903.cgz ├── sysbench.cgz -> sysbench_20200903.cgz └── sysbench-cpu.cgz -> sysbench.cgz all cgz file under one directory, so we can generate symlink when depends file is a symlink [after] $ tree /srv/initrd/deps/$os/$os_arch/$os_version/sysbench-cpu . ├── sysbench-cpu_20210128.cgz └── sysbench-cpu.cgz -> sysbench-cpu_20210128.cgz now we upload cgz file through curl, it's not appropriate to generate symlink directly, now generate original cgz file and rename it Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/common | 13 +++++++++++++ tests/cci-depends | 9 ++------- 2 files changed, 15 insertions(+), 7 deletions(-) diff --git a/distro/common b/distro/common index 1fd7df4e8..da2a57a79 100755 --- a/distro/common +++ b/distro/common @@ -94,3 +94,16 @@ upload_to_target_dir() upload_one_curl "${dir}/${file}" "${dir}" >/dev/null } + +rename_symlink_name() +{ + [ "${BM_NAME}" == "${benchmark}" ] && return + + cd "$pack_to" + [ -f "${BM_NAME}".cgz ] || return + rm "${BM_NAME}".cgz + date=$(ls -F | grep -v "@" | grep -Eo "[0-9]{8}") + mv "${BM_NAME}"_"$date".cgz "${benchmark}"_"$date".cgz + + ln -s "${benchmark}"_"$date".cgz "${benchmark}".cgz +} diff --git a/tests/cci-depends b/tests/cci-depends index 8ac1d7e58..0684a1025 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -38,13 +38,6 @@ umask 002 chown .lkp "$pack_to" } -check_shared_package "$benchmark" && { - ln -sf $BM_NAME.cgz $pack_to/$benchmark.cgz - echo "create symlink for shared package: $benchmark -> $BM_NAME" - - exit 0 -} - share_debian_package distro_install_depends cci-depends @@ -59,4 +52,6 @@ fixup_arch_install $pack_arch pack_benchmark_deps +rename_symlink_name + upload_to_target_dir ${pack_to} -- 2.23.0

1 1

[PATCH v2 compass-ci 3/3] lib/stats_wrapper.rb support handle $result_root/$script.yaml
by Lu Weitao 02 Feb '21

02 Feb '21

[Example] iperf.yaml: { tcp.sender.bps: 34804801216.197174 tcp.receiver.bps: 34804762215.18231 } iperf.json: { "iperf.tcp.sender.bps": [ 34804801216.197174 ], "iperf.tcp.receiver.bps": [ 34804762215.18231 ] } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/stats_wrapper.rb | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/lib/stats_wrapper.rb b/lib/stats_wrapper.rb index debe53b..8a6d6a9 100644 --- a/lib/stats_wrapper.rb +++ b/lib/stats_wrapper.rb @@ -24,7 +24,7 @@ module StatsWrapper @stats_group = program_time || program @log = "#{RESULT_ROOT}/#{@stats_group}" - return unless pretreatment + return unless File.exist?("#{(a)log}.yaml") || pretreatment return unless create_tmpfile check_tmpfile @@ -180,6 +180,15 @@ module StatsWrapper system "xzcat #{(a)dmesg_log}.xz > #{@dmesg_log}" if File.exist?("#{(a)dmesg_log}.xz") end + def self.create_yaml_tmpfile(file) + File.readlines(file).each do |line| + next if line =~ /{|}/ + + line.lstrip! + File.write(@tmpfile, line, mode: 'a') + end + end + def self.create_tmpfile tmp = Tempfile.new('lkp-stats.', '/tmp') @tmpfile = tmp.path @@ -191,6 +200,8 @@ module StatsWrapper check #{@tmpfile}") create_status = false end + elsif File.exist?("#{(a)log}.yaml") + create_yaml_tmpfile("#{(a)log}.yaml") else %x(#{PROGRAM_DIR}/#{@program} < /dev/null > #{@tmpfile}) unless $CHILD_STATUS.exitstatus.zero? -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] translate lkp-tests/stats/wrapper to compass-ci/lib/stats_wrapper.rb
by Lu Weitao 02 Feb '21

02 Feb '21

translate lkp-tests/stats/wrapper(shell) to compass-ci/lib/stats_wrapper.rb(ruby) [Why] we need reduce calls between processes to improve performance Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/stats_wrapper.rb | 252 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 252 insertions(+) create mode 100644 lib/stats_wrapper.rb diff --git a/lib/stats_wrapper.rb b/lib/stats_wrapper.rb new file mode 100644 index 0000000..debe53b --- /dev/null +++ b/lib/stats_wrapper.rb @@ -0,0 +1,252 @@ +# SPDX-License-Identifier: GPL-2.0-only + +# frozen_string_literal: true + +LKP_SRC ||= ENV['LKP_SRC'] || '/c/lkp-tests' +require "#{LKP_SRC}/lib/log.rb" +require 'tempfile' +require 'English' + +PROGRAM_DIR = "#{LKP_SRC}/stats" + +# exit processing the stats if the program is not in the program_list. +# program_list is a file that records all the programs(setups, monitors, tests, +# daemons) being executed during the test. +# this is to solve the problem for cluster test jobs running on the server node +# where no program's log file is generated for running server daemons but the +# stats processing for the program running on client node will also be handled +# on server node due to the current Job2sh algorithm. + +# default stats are not in the program_list +module StatsWrapper + def self.wrapper(program, program_time = nil) + @program = program + @stats_group = program_time || program + @log = "#{RESULT_ROOT}/#{@stats_group}" + + return unless pretreatment + return unless create_tmpfile + + check_tmpfile + warn_empty_stats + return unless create_stat + + check_empty_json + delete_log_package + end + + def self.pretreatment + return unless available_program? + return unless unzip_log + + extract_kmsg_dmesg + check_incomplete_run(@log) + check_soft_timeout + return if check_empty_output + return if check_binary_output + + true + end + + def self.available_program? + yaml_data = File.read("#{LKP_SRC}/etc/default_stats.yaml") + return true if yaml_data =~ /^#{@program}:/ + + return unless File.exist?("#{RESULT_ROOT}/program_list") + + pro_list = File.read("#{RESULT_ROOT}/program_list") + pro_list =~ /#{@program}/ + end + + def self.check_incomplete_run(file) + return if File.size?(file) + return unless File.exist?("#{LKP_SRC}/tests/#{@program}") + + data = "# missing #{@program} #{file}\nis_incomplete_run: 1" + File.write("#{RESULT_ROOT}/last_state", data, mode: 'a') + end + + def self.check_soft_timeout + return unless File.exist?("#{RESULT_ROOT}/soft_timeout") + + last_state = File.readlines("#{RESULT_ROOT}/last_state") + last_state.map!(&:chomp!) + data = 'soft_timeout: 1' + return if last_state.indlude?(data) + + File.write("#{RESULT_ROOT}/last_state", data, mode: 'a') + end + + def self.delete_log + File.delete(@log) if File.exist?(@log) + File.delete("#{(a)log}.gz") if File.exist?("#{(a)log}.gz") + end + + def self.check_empty_output + return if File.size?(@log) + return if %w[tcrypt kernel-size perf-profile].include?(@program) + return if @program == 'dmesg' && File.size?("#{RESULT_ROOT}/kmsg") + return if @program == 'kmsg' && File.size?("#{RESULT_ROOT}/dmesg") + + delete_log + true + end + + def self.check_binary_output + return if %w[dmesg kmsg mpstat iostat].include?(@program) + + # kmsg may actually read the dmesg file + # refer to the exception cases in check_empty_output() + return unless File.exist?(@log) + return unless File.read(@log) =~ /\x0/ + + log_warn "skip binary file #{@log}" + + true + end + + def self.warn_empty_stats + return if File.size?(@tmpfile) + return if %w[dmesg ftrace turbostat perf-profile].include?(@program) + return unless File.size?("#{RESULT_ROOT}/time") + return if File.size?("#{RESULT_ROOT}/last_state") + return if File.read("#{LKP_SRC}/etc/failure") =~ /^#{(a)program}\./ + return if File.read("#{LKP_SRC}/etc/pass") =~ /^#{(a)program}\./ + + log_warn "empty stats for #{@log}" + end + + def self.check_exist_json + return if File.exist?("#{RESULT_ROOT}/#{(a)stats_group}.json") || File.exist?("#{RESULT_ROOT}/#{(a)stats_group}.json.gz") + return if File.exist?("#{RESULT_ROOT}/last_stat") && File.read("#{RESULT_ROOT}/last_state") =~ /is_incomplete_run/ + + log_warn "no generate json file for #{@stats_group}, check #{RESULT_ROOT}" + data = "# no json file for #{@stats_group}\nis_incomplete_run: 1" + File.write("#{RESULT_ROOT}/last_state", data, mode: 'a') + end + + def self.check_empty_json + testcase = RESULT_ROOT.split('/')[2] + testcase[4, testcase.length] if testcase =~ /^kvm:/ + + return if %w[borrow boot].include?(testcase) + return unless File.exist?("#{RESULT_ROOT}/job.sh") + + # testcase maybe different with run_case + # take fio-basic-1hdd-write.yaml job as example: + # - testcase will be fio-basc + # - run_case will be fio + + run_case = nil + job_data = File.readlines("#{RESULT_ROOT}/job.sh") + job_data.each do |line| + next unless line.chomp! =~ %r{LKP_SRC/tests/wrapper} + + parse = line.split(' ')[-1] + run_case = parse + break + end + + return unless run_case == @stats_group + + check_exist_json + end + + def self.unzip_log + if File.exist?("#{(a)log}.gz") + return unless File.size?("#{(a)log}.gz") + + system "zcat #{(a)log}.gz > #{@log}" + elsif File.exist?("#{(a)log}.xz") + return unless File.size?("#{(a)log}.xz") + + system "xzcat #{(a)log}.xz > #{@log}" + end + + true + end + + def self.extract_kmsg_dmesg + @kmsg_log = "#{RESULT_ROOT}/kmsg" + # extract kmsg for kmsg related stats + if @log =~ %r{^#{RESULT_ROOT}/(boot-memory|boot-time|tcrypt|dmesg)$} + system "xzcat #{(a)kmsg_log}.xz" if File.exist?("#{(a)kmsg_log}.xz") + end + + @dmesg_log = "#{RESULT_ROOT}/dmesg" + # extract dmesg for dmesg related stats + return unless @log =~ %r{^#{RESULT_ROOT}/kmsg$} + + system "xzcat #{(a)dmesg_log}.xz > #{@dmesg_log}" if File.exist?("#{(a)dmesg_log}.xz") + end + + def self.create_tmpfile + tmp = Tempfile.new('lkp-stats.', '/tmp') + @tmpfile = tmp.path + create_status = true + if File.exist?(@log) + %x(#{PROGRAM_DIR}/#{@program} #{@log} < #{@log} > #{@tmpfile}) + unless $CHILD_STATUS.exitstatus.zero? + log_error("#{PROGRAM_DIR}/#{@program} #{@log} < #{@log} exit code #{$CHILD_STATUS.exitstatus}, + check #{@tmpfile}") + create_status = false + end + else + %x(#{PROGRAM_DIR}/#{@program} < /dev/null > #{@tmpfile}) + unless $CHILD_STATUS.exitstatus.zero? + log_error("#{PROGRAM_DIR}/#{@program} < /dev/null exit code #{$CHILD_STATUS.exitstatus}, check #{@tmpfile}") + create_status = false + end + end + + create_status + end + + def self.check_tmpfile + str = File.read("#{LKP_SRC}/etc/failure") + str += File.read("#{LKP_SRC}/etc/pass") + + str =~ /^#{(a)program}\./ || check_incomplete_run(@tmpfile) + end + + def self.dump_stat(stats_group, file) + %x(#{LKP_SRC}/sbin/dump-stat #{stats_group} < #{file}) + unless $CHILD_STATUS.exitstatus.zero? + log_error("#{LKP_SRC}/sbin/dump-stat #{@program} exit code #{$CHILD_STATUS.exitstatus}, check #{file}") + return nil + end + + File.delete(file) + + true + end + + def self.create_stat + if @program == 'ftrace' + Dir.foreach(RESULT_ROOT) do |file| + if file =~ /^#{@program}\..*\.yaml/ + stats_group = File.basename(file, '.yaml') + dump_stat(stats_group, file) + end + end + else + dump_stat(@stats_group, @tmpfile) + end + end + + def self.delete_log_package + if File.exist?("#{(a)log}.gz") + File.delete(@log) + elsif File.exist?("#{(a)log}.xz") + File.delete(@log) + end + + # delete temporarily extracted kmsg above + File.delete(@kmsg_log) if File.exist?("#{(a)kmsg_log}.xz") + + # delete temporarily extracted dmesg above + File.delete(@dmesg_log) if File.exist?("#{(a)dmesg_log}.xz") + + File.delete(@tmpfile) if File.exist?(@tmpfile) + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] sbin/result2stats: get stats_program_list from job.yaml
by Lu Weitao 02 Feb '21

02 Feb '21

[Why] Get rid of dependence on job.sh [How] stats_program_list include: - stat-programs from job[key] which matched with lkp-tests/stats* - job[$suite] - time job[$suite].time - stderr - default stats: lkp-tests/etc/default_stats.yaml Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 2 +- lib/stats.rb | 59 +++++++++++++++++++++++++++++++++++++++++++++++ sbin/result2stats | 16 +++++-------- 3 files changed, 66 insertions(+), 11 deletions(-) create mode 100644 lib/stats.rb diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2195ed4..ab92986 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -2,7 +2,7 @@ # frozen_string_literal: true -LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +LKP_SRC ||= ENV['LKP_SRC'] || '/c/lkp-tests' require "#{LKP_SRC}/lib/stats" require "#{LKP_SRC}/lib/yaml" diff --git a/lib/stats.rb b/lib/stats.rb new file mode 100644 index 0000000..acd807a --- /dev/null +++ b/lib/stats.rb @@ -0,0 +1,59 @@ +# SPDX-License-Identifier: GPL-2.0-only + +# frozen_string_literal: true + +require_relative './stats_wrapper' + +def load_yaml(path) + unless File.file?(path) + warn "#{path} does not exist" + exit + end + YAML.load_file(path) +end + +def available_stats + vailable_stats = [] + Dir.open(LKP_SRC + '/stats').each do |file| + vailable_stats << file unless file.start_with?('.') + end + + vailable_stats +end + +def assign_default_stats(stats_list) + default_stats = YAML.load_file(LKP_SRC + '/etc/default_stats.yaml').keys + default_stats.each { |stat| stats_list << stat } +end + +# extract each result from the script output +# format each result as $script.json +class Stats + def initialize(result_root) + @result_root = result_root + @job = load_yaml(File.join(result_root, 'job.yaml')) + @available_stats = available_stats + end + + def extract_stats + stats_list = assign_stats_list + stats_list.each do |stat| + StatsWrapper.wrapper(*stat) + end + end + + def assign_stats_list + stats_list = [] + stats_list << @job['suite'] + stats_list << ['time', @job['suite'] + '.time'] + stats_list << 'stderr' + @job.each_key do |k| + if @available_stats.include?(k) + stats_list << k + end + end + assign_default_stats(stats_list) + + stats_list + end +end diff --git a/sbin/result2stats b/sbin/result2stats index 6229b3c..6a09075 100755 --- a/sbin/result2stats +++ b/sbin/result2stats @@ -7,16 +7,10 @@ CCI_SRC = File.dirname(__dir__) require 'optparse' require "#{CCI_SRC}/lib/matrix2" +require_relative '../lib/stats.rb' File.umask 0o002 -def extract_stats(result_root) - ENV['RESULT_ROOT'] = result_root - job_script = result_root + '/job.sh' - - system job_script, 'extract_stats' -end - opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: result2stats result_root' opts.separator 'eg: result2stats /srv/result/iperf/vm-1p1g/2020-09-30/crystal.91198' @@ -36,7 +30,9 @@ argv = if ARGV == [] end opt_parser.parse!(argv) +RESULT_ROOT = argv[0] || ENV['RESULT_ROOT'] +ENV['RESULT_ROOT'] = RESULT_ROOT -result_root = argv[0] || ENV['RESULT_ROOT'] -extract_stats result_root -create_stats result_root +stats = Stats.new(RESULT_ROOT) +stats.extract_stats +create_stats RESULT_ROOT -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/git_mirror.rb: fix not use latest git remote url
by Lin Jiaxin 02 Feb '21

02 Feb '21

[why] When upstream-repos' url has changed https://git.archlinux.org/linux ==> https://mirrors.tuna.tsinghua.edu.cn/git/linux.git The bare git repo's remote url has not changed together. /srv/git/l/linux/linux.git% git ls-remote --get-url origin https://git.archlinux.org/linux Because the original logic only git fetch when git repo is existed. Now add check for git url. If git url has changed, rm the original git repo and clone again. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/git_mirror.rb | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 13cb7c0..f1c66b3 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -63,16 +63,29 @@ class GitMirror return fetch_info.include? '->' end + def url_changed?(url, mirror_dir) + url = get_url(Array(url)[0]) + git_url = %x(git -C #{mirror_dir} ls-remote --get-url origin).chomp + + return true if url == git_url + + return false + end + + def git_repo_download(url, mirror_dir) + return git_clone(url, mirror_dir) unless File.directory?(mirror_dir) + + return git_fetch(mirror_dir) if url_changed?(url, mirror_dir) + + FileUtils.rm_r(mirror_dir) + return git_clone(url, mirror_dir) + end + def mirror_sync fork_info = @queue.pop mirror_dir = "/srv/git/#{fork_info['git_repo']}" mirror_dir = "#{mirror_dir}.git" unless fork_info['is_submodule'] - possible_new_refs = false - if File.directory?(mirror_dir) - possible_new_refs = git_fetch(mirror_dir) - else - possible_new_refs = git_clone(fork_info['url'], mirror_dir) - end + possible_new_refs = git_repo_download(fork_info['url'], mirror_dir) feedback(fork_info['git_repo'], possible_new_refs) end -- 2.23.0

1 0

[PATCH compass-ci] doc/manual: change doc descriptions for "submit -o"
by Liu Shaofei 02 Feb '21

02 Feb '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- doc/manual/submit-job.en.md | 2 +- doc/manual/submit-job.zh.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/manual/submit-job.en.md b/doc/manual/submit-job.en.md index 2748fbe..a98145f 100644 --- a/doc/manual/submit-job.en.md +++ b/doc/manual/submit-job.en.md @@ -114,7 +114,7 @@ options: You can run the **-o DIR** command to save the generated YAML file to the specified directory **DIR**. An example is shown in the following figure: ``` - submit -o ~/iperf.yaml + submit -o ~/ testbox=vm-2p8g iperf.yaml ``` After the command is executed, the YAML file that has been processed by the **submit** command is generated in the specified directory. diff --git a/doc/manual/submit-job.zh.md b/doc/manual/submit-job.zh.md index 92faa5c..44fbaa8 100644 --- a/doc/manual/submit-job.zh.md +++ b/doc/manual/submit-job.zh.md @@ -102,13 +102,13 @@ options: * **-o��** - ��-o DIR ��yaml�� DIR �� + �� -o ��yaml�� ``` - submit -o ~/iperf.yaml testbox=vm-2p8g + submit -o ~/ testbox=vm-2p8g iperf.yaml ``` - �� submit �� yaml �� + �� submit �� yaml �� * **-a��** -- 2.23.0

2 2

[PATCH v3 compass-ci 3/3] mail-robot: delete useless file
by Luan Shengde 02 Feb '21

02 Feb '21

[why] the file part has been merged to parse-apply-account-email Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/gitee-commit-url-check.rb | 45 ------------------- 1 file changed, 45 deletions(-) delete mode 100755 container/mail-robot/lib/gitee-commit-url-check.rb diff --git a/container/mail-robot/lib/gitee-commit-url-check.rb b/container/mail-robot/lib/gitee-commit-url-check.rb deleted file mode 100755 index af31e17..0000000 --- a/container/mail-robot/lib/gitee-commit-url-check.rb +++ /dev/null @@ -1,45 +0,0 @@ -#!/usr/bin/env ruby -# SPDX-License-Identifier: MulanPSL-2.0+ -# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -# frozen_string_literal: true - -require 'json' -require 'mail' - -# used to check commit available for hub gitee.com -# it is called when: -# - ParseApplyAccountEmail execute commit_url_availability -# - hub is gitee.com -# gitee_commit_check -# clone the repo -# check commit available -class GiteeCommitUrlCheck - def initialize(my_info, url, base_url) - @my_info = my_info - @url = url - @base_url = base_url - end - - def gitee_commit_check - repo_url = [@base_url, 'git'].join('.') - repo_dir = repo_url.split('/')[-1] - commit_id = @url.split('/')[-1] - - Dir.chdir '/tmp' - %x(/usr/bin/git clone --bare #{repo_url} #{repo_dir}) - - email_index = %x(/usr/bin/git -C #{repo_dir} show #{commit_id}).index @my_info['my_email'] - - FileUtils.rm_rf repo_dir - - gitee_commit_exist(email_index) - end - - def gitee_commit_exist(email_index) - return if email_index - - error_message = 'COMMIT_URL_NOT_AVAILABLE' - - raise error_message - end -end -- 2.23.0

1 0

[PATCH v3 compass-ci] providers/libvirt: split domain.xml to some common templates
by Xiao Shenwei 02 Feb '21

02 Feb '21

[why] libvirt can boot a vm based on a xml file named domain.xml. it contains many elements like os, device, cpu. to meet a requirements of a single module or a combination of multiple modules. split the domain.xml may be better. here are some scenarios: case 1: the user provides a complete domain.xml file case 2: the user only provides a cpu.xml template case 3: the user provides cpu.xml and disk.xml or other templates Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/templates/active.xml | 3 +++ providers/libvirt/templates/clock.xml | 5 +++++ providers/libvirt/templates/cpu.xml | 3 +++ providers/libvirt/templates/devices.xml | 3 +++ providers/libvirt/templates/disk.xml | 7 +++++++ providers/libvirt/templates/domain.xml | 10 ++++++++++ providers/libvirt/templates/emulator.xml | 1 + providers/libvirt/templates/interface.xml | 6 ++++++ providers/libvirt/templates/memory.xml | 1 + providers/libvirt/templates/name.xml | 1 + providers/libvirt/templates/options.yaml | 23 +++++++++++++++++++++++ providers/libvirt/templates/os.xml | 8 ++++++++ providers/libvirt/templates/seclabel.xml | 4 ++++ providers/libvirt/templates/serial.xml | 4 ++++ 14 files changed, 79 insertions(+) create mode 100644 providers/libvirt/templates/active.xml create mode 100644 providers/libvirt/templates/clock.xml create mode 100644 providers/libvirt/templates/cpu.xml create mode 100644 providers/libvirt/templates/devices.xml create mode 100644 providers/libvirt/templates/disk.xml create mode 100644 providers/libvirt/templates/domain.xml create mode 100644 providers/libvirt/templates/emulator.xml create mode 100644 providers/libvirt/templates/interface.xml create mode 100644 providers/libvirt/templates/memory.xml create mode 100644 providers/libvirt/templates/name.xml create mode 100644 providers/libvirt/templates/options.yaml create mode 100644 providers/libvirt/templates/os.xml create mode 100644 providers/libvirt/templates/seclabel.xml create mode 100644 providers/libvirt/templates/serial.xml diff --git a/providers/libvirt/templates/active.xml b/providers/libvirt/templates/active.xml new file mode 100644 index 0000000..8ed7ee4 --- /dev/null +++ b/providers/libvirt/templates/active.xml @@ -0,0 +1,3 @@ +<on_poweroff>destroy</on_poweroff> +<on_reboot>destroy</on_reboot> +<on_crash>destroy</on_crash> diff --git a/providers/libvirt/templates/clock.xml b/providers/libvirt/templates/clock.xml new file mode 100644 index 0000000..12d0ef8 --- /dev/null +++ b/providers/libvirt/templates/clock.xml @@ -0,0 +1,5 @@ +<clock offset='utc'> + <timer name='rtc' tickpolicy='catchup' track='guest'/> + <timer name='hpet' present='no'/> + <timer name='pit' tickpolicy='delay'/> +</clock> diff --git a/providers/libvirt/templates/cpu.xml b/providers/libvirt/templates/cpu.xml new file mode 100644 index 0000000..1f95dc6 --- /dev/null +++ b/providers/libvirt/templates/cpu.xml @@ -0,0 +1,3 @@ +<vcpu placement='static'><%= nr_cpu %></vcpu> +<cpu mode='host-passthrough' check='none'> +</cpu> diff --git a/providers/libvirt/templates/devices.xml b/providers/libvirt/templates/devices.xml new file mode 100644 index 0000000..83cd847 --- /dev/null +++ b/providers/libvirt/templates/devices.xml @@ -0,0 +1,3 @@ +<devices> + <graphics type='vnc' autoport='yes' listen='0.0.0.0'><graphics> +</devices> diff --git a/providers/libvirt/templates/disk.xml b/providers/libvirt/templates/disk.xml new file mode 100644 index 0000000..7aecef0 --- /dev/null +++ b/providers/libvirt/templates/disk.xml @@ -0,0 +1,7 @@ +<% if disk %> +<disk type='block' device='disk'> + <driver name='qemu' type='raw'/> + <source dev='<%= disk %>'/> + <target dev='vda' bus='virtio'/> +</disk> +<% end %> diff --git a/providers/libvirt/templates/domain.xml b/providers/libvirt/templates/domain.xml new file mode 100644 index 0000000..a3fdb63 --- /dev/null +++ b/providers/libvirt/templates/domain.xml @@ -0,0 +1,10 @@ +<?xml version="1.0" character="utf-8"?> +<domain type='kvm'> + <resource> + <partition>/machine</partition> + </resource> + <features> + <acpi/> + <gic version='3'/> + </features> +</domain> diff --git a/providers/libvirt/templates/emulator.xml b/providers/libvirt/templates/emulator.xml new file mode 100644 index 0000000..e915215 --- /dev/null +++ b/providers/libvirt/templates/emulator.xml @@ -0,0 +1 @@ +<emulator><%=qemu_path%></emulator> diff --git a/providers/libvirt/templates/interface.xml b/providers/libvirt/templates/interface.xml new file mode 100644 index 0000000..5d4c04f --- /dev/null +++ b/providers/libvirt/templates/interface.xml @@ -0,0 +1,6 @@ +<interface type='bridge'> + <source bridge='br0'/> + <mac address='<%= mac %>'/> + <target dev='vnet0'/> + <model type='virtio'/> +</interface> diff --git a/providers/libvirt/templates/memory.xml b/providers/libvirt/templates/memory.xml new file mode 100644 index 0000000..34aec49 --- /dev/null +++ b/providers/libvirt/templates/memory.xml @@ -0,0 +1 @@ +<memory unit='GiB'><%= memory.to_i %></memory> diff --git a/providers/libvirt/templates/name.xml b/providers/libvirt/templates/name.xml new file mode 100644 index 0000000..5dc98f4 --- /dev/null +++ b/providers/libvirt/templates/name.xml @@ -0,0 +1 @@ +<name><%= job_id %></name> diff --git a/providers/libvirt/templates/options.yaml b/providers/libvirt/templates/options.yaml new file mode 100644 index 0000000..c41ee29 --- /dev/null +++ b/providers/libvirt/templates/options.yaml @@ -0,0 +1,23 @@ +# Domain option: +# use templates/domain.xml if user not specified. +domain: domain + +# Default options: +# these elements must be replaced for each domain.xml. +default: + - name + - os + - memory + - serial + - interface + - active + - emulator + +# Replaceable options: +# these elements can be replaced by user configurations. +replaceable: + - devices + - cpu + - seclabel + - disk + - clock diff --git a/providers/libvirt/templates/os.xml b/providers/libvirt/templates/os.xml new file mode 100644 index 0000000..3ac880d --- /dev/null +++ b/providers/libvirt/templates/os.xml @@ -0,0 +1,8 @@ +<os> + <type arch='<%= arch %>' machine='virt'>hvm</type> + <loader readonly='yes' type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader> + <kernel><%= kernel %></kernel> + <initrd><%= initrd %></initrd> + <cmdline><%= cmdline %></cmdline> + <nvram>/var/lib/libvirt/qemu/nvram/<%= job_id %>.fd</nvram> +</os> diff --git a/providers/libvirt/templates/seclabel.xml b/providers/libvirt/templates/seclabel.xml new file mode 100644 index 0000000..5c10a77 --- /dev/null +++ b/providers/libvirt/templates/seclabel.xml @@ -0,0 +1,4 @@ +<seclabel type='dynamic' model='dac' relabel='yes'> + <label>+0:+0</label> + <imagelabel>+0:+0</imagelabel> +</seclabel> diff --git a/providers/libvirt/templates/serial.xml b/providers/libvirt/templates/serial.xml new file mode 100644 index 0000000..2edc882 --- /dev/null +++ b/providers/libvirt/templates/serial.xml @@ -0,0 +1,4 @@ +<serial type='file'> + <source path="<%= log_file %>"/> + <target port="0"/> +</serial> -- 2.23.0

1 0

[PATCH compass-ci] create-image: fix error path
by Wang Chenglong 02 Feb '21

02 Feb '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/create-image | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/osimage/openeuler/create-image b/container/osimage/openeuler/create-image index 7f3d982..ed0bc14 100755 --- a/container/osimage/openeuler/create-image +++ b/container/osimage/openeuler/create-image @@ -37,7 +37,7 @@ setup_login() { pack_cgz() { echo "Packing package. Please wait." - find / ! -path "/tmp/${IMAGE_NAME}" | grep -vf /tmp/files-to-exclude | cpio -o -Hnewc | gzip -9 > /tmp/${IMAGE_NAME} + find ./ ! -path "./tmp/${IMAGE_NAME}" | grep -vf ./tmp/files-to-exclude | cpio -o -Hnewc | gzip -9 > ./tmp/${IMAGE_NAME} chmod 644 /tmp/${IMAGE_NAME} } -- 2.23.0

1 0

[PATCH lkp-tests] makepkg: avoid some unnecessary stderr
by Lin Jiaxin 02 Feb '21

02 Feb '21

unnecessary stder like this: ==> /tmp/stderr <== % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed ^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0^M10 0 129 0 0 137 0 --:--:-- --:--:-- --:--:-- 137 ^M 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0^M100 255k 0 255k 0 0 110k 0 --:--:-- 0:00:02 --:--:-- 313k ==> WARNING: Skipping verification of source file PGP signatures. ==> /tmp/stderr <== fcgi-2.4.2.tar.gz ... Passed ==> /tmp/stderr <== Cloning into bare repository '//shadowsocks-libev-git'... ==> /tmp/stderr <== shadowsocks-libev-git ... Skipped shadowsocks-libev@.service ... Passed shadowsocks-libev-server@.service ... Passed shadowsocks-libev-redir@.service ... Passed shadowsocks-libev-tunnel@.service ... Passed Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- etc/makepkg.conf | 12 ++++++------ sbin/makepkg | 18 +++++++++--------- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/etc/makepkg.conf b/etc/makepkg.conf index 5d73399d..b2beaedb 100644 --- a/etc/makepkg.conf +++ b/etc/makepkg.conf @@ -8,12 +8,12 @@ # #-- The download utilities that makepkg should use to acquire sources # Format: 'protocol::agent' -DLAGENTS=('file::/usr/bin/my_curl -gqC - -o %o %u' - 'ftp::/usr/bin/my_curl -gqfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o %u' - 'http::/usr/bin/my_curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' - 'https::/usr/bin/my_curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' - 'rsync::/usr/bin/rsync --no-motd -z %u %o' - 'scp::/usr/bin/scp -C %u %o') +DLAGENTS=('file::/usr/bin/my_curl -sS -gqC - -o %o %u' + 'ftp::/usr/bin/my_curl -sS -gqfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o %u' + 'http::/usr/bin/my_curl -sS -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' + 'https::/usr/bin/my_curl -sS -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' + 'rsync::/usr/bin/rsync -q --no-motd -z %u %o' + 'scp::/usr/bin/scp -q -C %u %o') # Other common tools: # /usr/bin/snarf diff --git a/sbin/makepkg b/sbin/makepkg index 498b4872..7a92ea71 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -566,7 +566,7 @@ download_git() { if [[ ! -d "$dir" ]] || dir_is_empty "$dir" ; then msg2 "$(gettext "Cloning %s %s repo...")" "${repo}" "git" - if ! git clone --mirror "$url" "$dir"; then + if ! git clone -q --mirror "$url" "$dir"; then error "$(gettext "Failure while downloading %s %s repo")" "${repo}" "git" plain "$(gettext "Aborting...")" exit 1 @@ -580,7 +580,7 @@ download_git() { exit 1 fi msg2 "$(gettext "Updating %s %s repo...")" "${repo}" "git" - if ! git fetch --all -p; then + if ! git fetch -q --all -p; then # only warn on failure to allow offline builds warning "$(gettext "Failure while updating %s %s repo")" "${repo}" "git" fi @@ -609,13 +609,13 @@ extract_git() { if [[ -d "${dir##*/}" ]]; then updating=1 cd_safe "${dir##*/}" - if ! git fetch; then + if ! git fetch -q; then error "$(gettext "Failure while updating working copy of %s %s repo")" "${repo}" "git" plain "$(gettext "Aborting...")" exit 1 fi cd_safe "$srcdir" - elif ! git clone "$dir"; then + elif ! git clone -q "$dir"; then error "$(gettext "Failure while creating working copy of %s %s repo")" "${repo}" "git" plain "$(gettext "Aborting...")" exit 1 @@ -645,7 +645,7 @@ extract_git() { fi if [[ $ref != "origin/HEAD" ]] || (( updating )) ; then - if ! git checkout --force --no-track -B makepkg $ref; then + if ! git checkout -q --force --no-track -B makepkg $ref; then error "$(gettext "Failure while creating working copy of %s %s repo")" "${repo}" "git" plain "$(gettext "Aborting...")" exit 1 @@ -1270,10 +1270,10 @@ verify_integrity_one() { local source_name=$1 integ=$2 expectedsum=$3 local file="$(get_filename "$source_name")" - printf ' %s ... ' "$file" >&2 + printf ' %s ... ' "$file" if [[ $expectedsum = 'SKIP' ]]; then - printf '%s\n' "$(gettext "Skipped")" >&2 + printf '%s\n' "$(gettext "Skipped")" return fi @@ -1285,7 +1285,7 @@ verify_integrity_one() { local realsum="$(openssl dgst -${integ} "$file")" realsum="${realsum##* }" if [[ ${expectedsum,,} = "$realsum" ]]; then - printf '%s\n' "$(gettext "Passed")" >&2 + printf '%s\n' "$(gettext "Passed")" else printf '%s\n' "$(gettext "FAILED")" >&2 return 1 @@ -1553,7 +1553,7 @@ check_source_integrity() { warning "$(gettext "Skipping verification of source file checksums.")" check_pgpsigs "$@" elif (( SKIPPGPCHECK )); then - warning "$(gettext "Skipping verification of source file PGP signatures.")" + # warning "$(gettext "Skipping verification of source file PGP signatures.")" check_checksums "$@" else check_checksums "$@" -- 2.23.0

2 1

[PATCH compass-ci] lib/git_mirror.rb: fix not use latest git remote url
by Lin Jiaxin 01 Feb '21

01 Feb '21

[why] When upstream-repos' url has changed https://git.archlinux.org/linux ==> https://mirrors.tuna.tsinghua.edu.cn/git/linux.git The bare git repo's remote url has not changed together. /srv/git/l/linux/linux.git% git ls-remote --get-url origin https://git.archlinux.org/linux Because the original logic only git fetch when git repo is existed. Now add check for git url. If git url has changed, rm the original git repo and clone again. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/git_mirror.rb | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 13cb7c0..84a908b 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -63,16 +63,29 @@ class GitMirror return fetch_info.include? '->' end + def git_url_check(url, mirror_dir) + url = get_url(Array(url)[0]) + git_url = %x(git -C #{mirror_dir} ls-remote --get-url origin).chomp + + return true if url == git_url + + return false + end + + def git_repo_download(url, mirror_dir) + return git_clone(url, mirror_dir) unless File.directory?(mirror_dir) + + return git_fetch(mirror_dir) if git_url_check(url, mirror_dir) + + FileUtils.rm_r(mirror_dir) + return git_clone(url, mirror_dir) + end + def mirror_sync fork_info = @queue.pop mirror_dir = "/srv/git/#{fork_info['git_repo']}" mirror_dir = "#{mirror_dir}.git" unless fork_info['is_submodule'] - possible_new_refs = false - if File.directory?(mirror_dir) - possible_new_refs = git_fetch(mirror_dir) - else - possible_new_refs = git_clone(fork_info['url'], mirror_dir) - end + possible_new_refs = git_repo_download(fork_info['url'], mirror_dir) feedback(fork_info['git_repo'], possible_new_refs) end -- 2.23.0

2 1

[PATCH v2 compass-ci] container/web-backend: add api to query new_refs count as echart
by Li Yuanchao 01 Feb '21

01 Feb '21

Query the new_refs count data from es, and format it as echart needs, with 'x_params' and 'new_ref_times'. Return it to webapi. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 13 +++++++++++ src/lib/web_backend.rb | 38 +++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index ff8fcff..74eda85 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -133,3 +133,16 @@ end get '/get_tbox_state' do get_tbox_state(params) end + +# GET /get_repo_statistics?git_repo=$git_repo +# Response: +# - { +# "title": "new refs statistics", +# "unit": "times", +# "x_name": "date", +# "source": [["x_params", $date1, ..., $date_today], +# ["new_ref_times", $times_1, ..., $times_today]] +# } +get '/get_repo_statistics' do + new_refs_statistics(params) +end diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 61e3334..291fd9f 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -510,3 +510,41 @@ def get_tbox_state(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def get_echart(statistics) + echart = { + 'title' => 'new refs statistics', + 'unit' => 'times', + 'x_name' => 'date', + 'source' => [['x_params'], ['new_ref_times']] + } + dev = 1 + # The day will be 2021-01-01 + day = Date.new(2021, 1, 1) + today = Date.today + while day <= today + day_s = day.to_s + echart['source'][0][dev] = day_s + echart['source'][1][dev] = statistics[day_s] || 0 + day += 1 + dev += 1 + end + echart +end + +def query_repo_statistics(params) + query = { "query": { "match": { "_id": params[:git_repo] } } } + result = ES_CLIENT.search(index: 'repo', body: query)['hits'] + statistics = result['total'].positive? ? result['hits'][0]['_source']['new_refs_count'] : {} + get_echart(statistics) +end + +def new_refs_statistics(params) + begin + body = query_repo_statistics(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'new refs statistics error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: add a new os_mount named local
by Wu Zhende 01 Feb '21

01 Feb '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 2 +- src/scheduler/kernel_params.cr | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 964c520..c013aa7 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -344,7 +344,7 @@ class Job end # defaults to the 1st value - VALID_OS_MOUNTS = ["initramfs", "nfs", "cifs", "container"] + VALID_OS_MOUNTS = ["initramfs", "nfs", "cifs", "container", "local"] private def set_os_mount if is_docker_job? diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index dea9215..ed64bea 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -18,6 +18,7 @@ class Job "cifs" => "root=cifs://#{OS_HTTP_HOST}#{os_real_path}" + ",guest,ro,hard,vers=1.0,noacl,nouser_xattr", "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0", + "local" => "root=/dev/mapper/os-#{self.os}_#{self.os_arch}_#{self.os_version}", "container" => "", } -- 2.23.0

1 0

[PATCH compass-ci] container/web-backend: add api to query new_refs count as echart
by Li Yuanchao 01 Feb '21

01 Feb '21

Query the new_refs count data from es, and format it as echart needs, with 'x_params' and 'y_params'. Return it to webapi. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 13 +++++++++++ src/lib/web_backend.rb | 39 +++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index ff8fcff..6f85f35 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -133,3 +133,16 @@ end get '/get_tbox_state' do get_tbox_state(params) end + +# GET /get_repo_statistics?git_repo=$git_repo +# Response: +# - { +# "title": "new refs statistics", +# "unit": "times", +# "x_name": "date", +# "x_params": [$date1, ..., $date_today], +# "y_params": [$times_1, ..., $times_today] +# } +get '/get_repo_statistics' do + new_refs_statistics(params) +end diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 61e3334..1c29c88 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -510,3 +510,42 @@ def get_tbox_state(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def get_echart(statistics) + echart = { + 'title' => 'new refs statistics', + 'unit' => 'times', + 'x_name' => 'date', + 'x_params' => [], + 'y_params' => [] + } + dev = 0 + # The day will be 2021-01-01 + day = Date.new(2021, 1, 1) + today = Date.today + while day <= today + day_s = day.to_s + echart['x_params'][dev] = day_s + echart['y_params'][dev] = statistics[day_s] || 0 + day += 1 + dev += 1 + end + echart +end + +def query_repo_statistics(params) + query = { "query": { "match": { "_id": params[:git_repo] } } } + result = ES_CLIENT.search(index: 'repo', body: query)['hits'] + statistics = result['total'].positive? ? result['hits'][0]['_source']['new_refs_count'] : {} + get_echart(statistics) +end + +def new_refs_statistics(params) + begin + body = query_repo_statistics(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'new refs statistics error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

2 2

[PATCH compass-ci 2/2] service/lifecycle: to process the machine that times out
by Wu Zhende 01 Feb '21

01 Feb '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 51 ++++++++++++++++++++++++++++++++++++++ src/lifecycle/lifecycle.cr | 8 ++++++ 2 files changed, 59 insertions(+) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index af5cd07..75a18ec 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -4,6 +4,7 @@ require "kemal" require "yaml" +require "./scheduler_api" require "./web_env" require "../scheduler/elasticsearch_client" @@ -14,6 +15,7 @@ class Lifecycle @es = Elasticsearch::Client.new @env = env @log = env.log.as(JSONLogger) + @scheduler_api = SchedulerAPI.new end def alive(version) @@ -36,4 +38,53 @@ class Lifecycle } @es.search("testbox", query) end + + def get_timeout_testbox(size) + # why use "now+8h" + # The time zone in the ES database is 0 + # add 8 hours to the current time and convert it to time zone 0 + query = { + "size" => size, + "query" =>{ + "range" => { + "deadline" => { + "lt" => "now+8h" + } + } + } + } + + results = @es.search("testbox", query) + results["hits"]["hits"] + end + + def process_timeout_testbox + size = @env.params.query["size"]? || 20 + results = get_timeout_testbox(size) + return results unless results.is_a?(JSON::Any) + + results.as_a.each do |result| + hostname = result["_id"].to_s + hash = result["_source"] + close_lifecycle(hostname, hash, "timeout") + end + end + + # 1.reboot testbox, do in the fature + # 2.update testbox info in es + # 3.close job + def close_lifecycle(hostname, hash, close_reason) + state = hash["state"]? == "rebooting" ? hash["state"] : "forcibly_rebooting" + testbox_update_info = { + "deadline" => nil, + "state" => state + } + @es.update_tbox(hostname, testbox_update_info) + + # If the value of state is rebooting, + # the job has been closed and don't need to close it again + return if state == "rebooting" + + @scheduler_api.close_job(hash["job_id"], close_reason).status_message + end end diff --git a/src/lifecycle/lifecycle.cr b/src/lifecycle/lifecycle.cr index 4154a27..13d36b7 100644 --- a/src/lifecycle/lifecycle.cr +++ b/src/lifecycle/lifecycle.cr @@ -29,4 +29,12 @@ module Cycle get "/get_running_testbox" do |env| env.lifecycle.get_running_testbox.to_json end + + # curl http://localhost:11311/process_timeout_testbox?size=10 + # size indicates the number of records to be processed at a time + get "/process_timeout_testbox" do |env| + env.lifecycle.process_timeout_testbox + + "done" + end end -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/scheduler_api.cr: added parameter "job_state"
by Wu Zhende 01 Feb '21

01 Feb '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/scheduler_api.cr | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/lib/scheduler_api.cr b/src/lib/scheduler_api.cr index b8dd514..0fdcd12 100644 --- a/src/lib/scheduler_api.cr +++ b/src/lib/scheduler_api.cr @@ -10,10 +10,14 @@ class SchedulerAPI @host = ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1" end - def close_job(job_id) + def close_job(job_id, job_state = nil) + url = "/~lkp/cgi-bin/lkp-post-run?job_id=#{job_id}" + url += "&job_state=#{job_state}" if job_state + client = HTTP::Client.new(@host, port: @port) - response = client.get("/~lkp/cgi-bin/lkp-post-run?job_id=#{job_id}") + response = client.get(url) client.close() + return response end end -- 2.23.0

1 0

[PATCH compass-ci] lib/params_group.rb: change group_key value parse format
by Wang Yong 01 Feb '21

01 Feb '21

[before] os=openeuler/os_arch=aarch64/os_version=20.03/pp.sysbench-cpu.cpu_max_prime=20000/pp.sysbench-cpu.nr_threads=96/pp.sysbench-cpu.runtime=300/tbox_group=taishan200-2280-2s48p-256g [after] os=openeuler os_arch=aarch64 os_version=20.03 pp.sysbench-cpu.cpu_max_prime=20000 pp.sysbench-cpu.nr_threads=96 pp.sysbench-cpu.runtime=300 tbox_group=taishan200-2280-2s48p-256g it's easy to paste value and pass these to es-find Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- lib/params_group.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 9be2cbe..2082c5d 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -124,7 +124,7 @@ end def get_group_key(group_params) group_str = group_params.each.map do |k, v| "#{k}=#{v}" - end.sort!.join('/') + end.sort!.join(' ') return group_str end -- 2.23.0

2 1

[PATCH v3 compass-ci 4/4] container/submit: README file
by Luan Shengde 01 Feb '21

01 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/README.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 container/submit/README.md diff --git a/container/submit/README.md b/container/submit/README.md new file mode 100644 index 0000000..6772961 --- /dev/null +++ b/container/submit/README.md @@ -0,0 +1,26 @@ +# Install lkp-tests and its depends in container. +# With docker container, will suit most of the linux OSes. +# Avoid much repetitive testing work to adapt various OSes. + +Prepare work: + apply account + config default yaml + generate ssh keys + +Installation: + git clone https://gitee.com/wu_fengguang/compass-ci.git + + cd compass-ci/container/submit + ./build + + Add soft link for executable file 'submit' to $PATH: + + ln -s $PWD/submit /usr/bin/submit + +Usage: + You can directly use the command 'submit' to submit jobs. + It's the same as you install the lkp-tests at your local host. + + Example: + + submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

1 0

[PATCH v3 compass-ci 3/4] container/submit: submit file
by Luan Shengde 01 Feb '21

01 Feb '21

executable file for container: submit it can be run as the real command submit in the container. when you run it, it will calls the command 'submit' in the container. add ln for it to PATH, you can run it at anywhere. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/submit | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 container/submit/submit diff --git a/container/submit/submit b/container/submit/submit new file mode 100755 index 0000000..20fdf61 --- /dev/null +++ b/container/submit/submit @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +suffix=$(date +%m%d-%H%M) +cmd=( + docker run + --rm + --name=submit-$USER-$suffix + -it + -v /etc/compass-ci:/etc/compass-ci:ro + -v $HOME/.config:/root/.config:ro + -v $HOME/.ssh:/root/.ssh:rw + submit + /root/lkp-tests/sbin/submit "$@" +) + +"${cmd[@]}" + -- 2.23.0

1 0

[PATCH v3 compass-ci 2/4] container/submit: build file
by Luan Shengde 01 Feb '21

01 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/submit/build diff --git a/container/submit/build b/container/submit/build new file mode 100755 index 0000000..fba4281 --- /dev/null +++ b/container/submit/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t submit:latest . -- 2.23.0

1 0

[PATCH v3 compass-ci 1/4] container/submit: Dockerfile
by Luan Shengde 01 Feb '21

01 Feb '21

Install lkp-tests and its deps in the container instead of install them at localhost. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/Dockerfile | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 container/submit/Dockerfile diff --git a/container/submit/Dockerfile b/container/submit/Dockerfile new file mode 100644 index 0000000..f957af8 --- /dev/null +++ b/container/submit/Dockerfile @@ -0,0 +1,36 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +# it may take a long time if use default source address if in zh. +# you can delete the following step if you do not want to use it. +RUN echo 'deb [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\n \ +\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\n' \ +> /etc/apt/sources.list + +RUN apt-get update && \ + apt-get install -y git ruby ruby-dev make libssl-dev gcc g++ sudo uuid-runtime + +# it may take a long time, even break up if you use default source address for the gem. +# delete the following step if you do not want to use the default one. +RUN gem sources --add https://gems.ruby-china.com/ && \ + gem sources --remove https://rubygems.org/ + +RUN gem install git faye-websocket activesupport rest-client + +WORKDIR /root + +RUN git clone https://gitee.com/wu_fengguang/lkp-tests.git +RUN cd lkp-tests && make install -- 2.23.0

1 0

[PATCH compass-ci] git-mirror: fix build error
by Yu Chuan 01 Feb '21

01 Feb '21

[Why] After `apt add ruby-dev` is executed, the `gem` command can be available. [Error Msg] Step 6/7 : RUN apk update && gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && apk upgrade && apk add--no-cache git && apk add ruby-dev make gcc g++ openssh ---> Running in 4c1f8708c072 fetch http://mirrors.huaweicloud.com/alpine/v3.8/main/aarch64/APKINDEX.tar.gz fetch http://mirrors.huaweicloud.com/alpine/v3.8/community/aarch64/APKINDEX.tar.gz v3.8.5-65-g097f0358a9 [http://mirrors.huaweicloud.com/alpine/v3.8/main] v3.8.5-37-gf06ffe835a [http://mirrors.huaweicloud.com/alpine/v3.8/community] OK: 9385 distinct packages available /bin/sh: gem: not found The command '/bin/sh -c apk update && gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && apk upgrade && apk add --no-cache git && apk add ruby-dev make gcc g++ openssh' returned a non-zero code: 127 Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/git-mirror/Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/git-mirror/Dockerfile b/container/git-mirror/Dockerfile index 2eac95c54ca1..18613eddf30e 100644 --- a/container/git-mirror/Dockerfile +++ b/container/git-mirror/Dockerfile @@ -10,10 +10,10 @@ RUN adduser -u 1090 -D lkp RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk update && \ - gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ apk upgrade && \ apk add --no-cache git && \ - apk add ruby-dev make gcc g++ openssh + apk add ruby-dev make gcc g++ openssh && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ RUN umask 002 && \ gem install bunny json PriorityQueue elasticsearch --no-rdoc --no-ri -- 2.23.0

2 3

[PATCH v2 compass-ci] git-mirror: fix build error
by Yu Chuan 01 Feb '21

01 Feb '21

[Why] After `apt add ruby-dev` is executed, the `gem` command can be available. And 'change gem source' should also be before executing 'gem install xxx'. [Error Msg] Step 6/7 : RUN apk update && gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && apk upgrade && apk add--no-cache git && apk add ruby-dev make gcc g++ openssh ---> Running in 4c1f8708c072 fetch http://mirrors.huaweicloud.com/alpine/v3.8/main/aarch64/APKINDEX.tar.gz fetch http://mirrors.huaweicloud.com/alpine/v3.8/community/aarch64/APKINDEX.tar.gz v3.8.5-65-g097f0358a9 [http://mirrors.huaweicloud.com/alpine/v3.8/main] v3.8.5-37-gf06ffe835a [http://mirrors.huaweicloud.com/alpine/v3.8/community] OK: 9385 distinct packages available /bin/sh: gem: not found The command '/bin/sh -c apk update && gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && apk upgrade && apk add --no-cache git && apk add ruby-dev make gcc g++ openssh' returned a non-zero code: 127 Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/git-mirror/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/git-mirror/Dockerfile b/container/git-mirror/Dockerfile index 2eac95c54ca1..127d0109d929 100644 --- a/container/git-mirror/Dockerfile +++ b/container/git-mirror/Dockerfile @@ -10,11 +10,11 @@ RUN adduser -u 1090 -D lkp RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk update && \ - gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ apk upgrade && \ apk add --no-cache git && \ apk add ruby-dev make gcc g++ openssh RUN umask 002 && \ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ gem install bunny json PriorityQueue elasticsearch --no-rdoc --no-ri -- 2.23.0

1 0

[PATCH compass-ci 1/4] container/submit: Dockerfile
by Luan Shengde 01 Feb '21

01 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/Dockerfile | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 container/submit/Dockerfile diff --git a/container/submit/Dockerfile b/container/submit/Dockerfile new file mode 100644 index 0000000..e4c2cbd --- /dev/null +++ b/container/submit/Dockerfile @@ -0,0 +1,31 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN echo 'deb [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib\n \ +deb [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\n \ +\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib\n \ +deb-src [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\n' \ +> /etc/apt/sources.list + +RUN apt-get update && \ + apt-get install -y git ruby ruby-dev make libssl-dev gcc g++ sudo uuid-runtime + +RUN gem sources --add https://gems.ruby-china.com/ +RUN gem sources --remove https://rubygems.org/ +RUN gem install git faye-websocket activesupport rest-client + +WORKDIR /root + +RUN git clone https://gitee.com/wu_fengguang/lkp-tests.git +RUN cd lkp-tests && make install -- 2.23.0

2 4

[PATCH compass-ci 2/2] lib/build_my_info_client: fix always generate new my_token
by Luan Shengde 01 Feb '21

01 Feb '21

[why] everytime invoking the script, it will generate a new my_token for an existing user [how] for old user, only if it has added a new token for the class, the patch will update the my_token, else it will use the old one in the ES. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/build_my_info_client.rb | 41 ++++++++++++++++++++++++++----------- 1 file changed, 29 insertions(+), 12 deletions(-) mode change 100644 => 100755 lib/build_my_info_client.rb diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb old mode 100644 new mode 100755 index c946bec..9d2aaea --- a/lib/build_my_info_client.rb +++ b/lib/build_my_info_client.rb @@ -21,17 +21,16 @@ require_relative '../lib/es_client' # build_my_info = BuildMyInfo.new(email, name, lab) # build_my_info.config_my_info class BuildMyInfo - def initialize(my_email, my_name, lab, my_token = nil) + def initialize(my_email, my_name, lab, my_token) @lab = lab || 'nolab' - @my_token = my_token || %x(uuidgen).chomp @my_info = { 'my_email' => my_email, - 'my_name' => my_name, - 'my_token' => @my_token, - 'my_login_name' => nil, - 'my_commit_url' => nil, - 'my_ssh_pubkey' => [] + 'my_name' => my_name } + unless my_token.nil? + @my_info['my_token'] = my_token + end + @es = ESClient.new(index: 'accounts') end def config_default_yaml @@ -50,6 +49,26 @@ class BuildMyInfo end end + def search_my_info + @es.query_by_id(@my_info['my_email']) || {} + end + + def update_from_es(my_info_es) + my_info_es.update @my_info + @my_info.update my_info_es + end + + def complete_my_info + my_info_es = search_my_info + update_from_es(my_info_es) unless my_info_es.empty? + + @my_info['my_token'] = %x(uuidgen).chomp if @my_info['my_token'].nil? + end + + def store_account_info + @es.put_source_by_id(@my_info['my_email'], @my_info) + end + def config_lab_yaml lab_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/include/lab" FileUtils.mkdir_p lab_yaml_dir unless File.directory? lab_yaml_dir @@ -64,14 +83,12 @@ class BuildMyInfo end end - def store_account_info - es = ESClient.new(index: 'accounts') - es.put_source_by_id(@my_info['my_email'], @my_info) - end - def config_my_info + complete_my_info + config_default_yaml config_lab_yaml + store_account_info end end -- 2.23.0

1 0

[PATCH compass-ci 1/2] sbin/build-my-info: optimize options
by Luan Shengde 01 Feb '21

01 Feb '21

optimize options: add option -t: generate new my_token add option -s: search my_info for the email add option -h: output help message enable user search my_info for the email enable reset my_token for my_info Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/build-my-info.rb | 56 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 48 insertions(+), 8 deletions(-) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index fd75f51..1b99408 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -8,26 +8,66 @@ require_relative '../lib/build_my_info_client' require 'optparse' option = { - my_name: `git config --global user.name`.chomp, - my_email: `git config --global user.email`.chomp, - lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp + 'my_email' => `git config --global user.email`.chomp, + 'my_name' => `git config --global user.name`.chomp, + 'lab' => `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp } +es_search = false + options = OptionParser.new do |opts| + opts.banner = "Usage: build-my-info [-e email] [-n name] [-l lab] [-t]\n" + opts.banner += " build-my-info [-e email] -s\n" + + opts.separator '' + opts.separator 'options:' + opts.on('-e email', 'my_email') do |email| - option[:my_email] = email + option['my_email'] = email end opts.on('-n name', 'my_name') do |name| - option[:my_name] = name + option['my_name'] = name end opts.on('-l lab', 'lab') do |lab| - option[:lab] = lab + option['lab'] = lab + end + + opts.on('-t', 'my_token') do + option['my_token'] = %x(uuidgen).chomp + end + + opts.on('-s', 'search info for the email address') do + es_search = true + end + + opts.on_tail('-h', 'show this message') do + puts opts + exit end end +ARGV << '-h' if ARGV.empty? + options.parse! -build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -build_my_info.config_my_info +build_my_info = BuildMyInfo.new(option['my_email'], option['my_name'], option['lab'], option['my_token']) + +if es_search + my_info = build_my_info.search_my_info + if my_info.empty? + puts "No info for #{option['my_email']} yet.\n" + else + formatter = "%-15s %-15s\n" + my_info.each do |k, v| + printf formatter, "#{k}:", v + end + end +else + begin + build_my_info.config_my_info + rescue StandardError => e + puts e.backtrace + end +end -- 2.23.0

1 0

help star
by Wu Fengguang 01 Feb '21

01 Feb '21

Yu Chuan is our help star, thanks Yu Chuan! 21.1 Wu Fengguang 17.1 Yu Chuan 11.3 Wei Jihui 11.1 Cao Xueliang 9.4 Liu Shaofei 8.3 Zhang Dewan 7.5 Luan Shengde 7.3 Liu Yinsi 6.9 Xu Xijian 6.6 Xiao Shenwei 5.4 Wu Zhende 5.1 Li Yuanchao 4.6 Lu Weitao 4.5 Lin Jiaxin 4.2 Bai Jing 4.0 Wang Yong 3.6 Du Kaitian 3.4 Li Ping 2.7 Ren Wen 2.1 Wang Chenglong 1.6 Cui Yili 1.1 Lu Kaiyi 0.9 Zhang Yale 0.0 Tong Qunfeng 0.0 Hu Xuejiao 0.0 Wang Shuai 0.0 Cui Xucui

1 0

[PATCH compass-ci] sparrow/install-server: add script to deploy compass-ci container services
by Liu Yinsi 01 Feb '21

01 Feb '21

[why] For user locally deploy compass-ci, some users' hosts can not access internet. [how] deploy all container services in the host machine, other functions: submit job and run qemu test, will deploy in other machine which can access internet by $LKP_SRC/setup/simply-ci. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-server | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100755 sparrow/install-server diff --git a/sparrow/install-server b/sparrow/install-server new file mode 100755 index 0000000..c532956 --- /dev/null +++ b/sparrow/install-server @@ -0,0 +1,20 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# For your reference only. +# It's better to run the below scripts step by step. + +[[ $CCI_SRC ]] || export CCI_SRC=$(dirname $(dirname $(realpath $0))) +cd $CCI_SRC/sparrow || exit + +0-package/common +1-storage/tiny +1-storage/permission +2-network/br0 +2-network/iptables +2-network/nfs +2-network/cifs +3-code/dev-env +. /etc/profile.d/compass.sh +4-docker/buildall run-only +7-systemd/systemd-setup -- 2.23.0

1 0

[PATCH v2 compass-ci] providers/libvirt: split domain.xml to some common templates
by Xiao Shenwei 01 Feb '21

01 Feb '21

[why] libvirt can boot a vm based on a xml file named domain.xml. it contains many elements like os, device, cpu. to meet a requirements of a single module or a combination of multiple modules. split the domain.xml may be better. here are some scenarios: case 1: the user provides a complete domain.xml file case 2: the user only provides a cpu.xml template case 3: the user provides cpu.xml and disk.xml or other templates Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/templates/active.xml | 3 +++ providers/libvirt/templates/clock.xml | 5 +++++ providers/libvirt/templates/cpu.xml | 3 +++ providers/libvirt/templates/devices.xml | 3 +++ providers/libvirt/templates/disk.xml | 7 +++++++ providers/libvirt/templates/domain.xml | 10 ++++++++++ providers/libvirt/templates/emulator.xml | 1 + providers/libvirt/templates/interface.xml | 6 ++++++ providers/libvirt/templates/memory.xml | 1 + providers/libvirt/templates/name.xml | 1 + providers/libvirt/templates/options.yaml | 23 +++++++++++++++++++++++ providers/libvirt/templates/os.xml | 8 ++++++++ providers/libvirt/templates/seclabel.xml | 4 ++++ providers/libvirt/templates/serial.xml | 4 ++++ 14 files changed, 79 insertions(+) create mode 100644 providers/libvirt/templates/active.xml create mode 100644 providers/libvirt/templates/clock.xml create mode 100644 providers/libvirt/templates/cpu.xml create mode 100644 providers/libvirt/templates/devices.xml create mode 100644 providers/libvirt/templates/disk.xml create mode 100644 providers/libvirt/templates/domain.xml create mode 100644 providers/libvirt/templates/emulator.xml create mode 100644 providers/libvirt/templates/interface.xml create mode 100644 providers/libvirt/templates/memory.xml create mode 100644 providers/libvirt/templates/name.xml create mode 100644 providers/libvirt/templates/options.yaml create mode 100644 providers/libvirt/templates/os.xml create mode 100644 providers/libvirt/templates/seclabel.xml create mode 100644 providers/libvirt/templates/serial.xml diff --git a/providers/libvirt/templates/active.xml b/providers/libvirt/templates/active.xml new file mode 100644 index 0000000..8ed7ee4 --- /dev/null +++ b/providers/libvirt/templates/active.xml @@ -0,0 +1,3 @@ +<on_poweroff>destroy</on_poweroff> +<on_reboot>destroy</on_reboot> +<on_crash>destroy</on_crash> diff --git a/providers/libvirt/templates/clock.xml b/providers/libvirt/templates/clock.xml new file mode 100644 index 0000000..12d0ef8 --- /dev/null +++ b/providers/libvirt/templates/clock.xml @@ -0,0 +1,5 @@ +<clock offset='utc'> + <timer name='rtc' tickpolicy='catchup' track='guest'/> + <timer name='hpet' present='no'/> + <timer name='pit' tickpolicy='delay'/> +</clock> diff --git a/providers/libvirt/templates/cpu.xml b/providers/libvirt/templates/cpu.xml new file mode 100644 index 0000000..1f95dc6 --- /dev/null +++ b/providers/libvirt/templates/cpu.xml @@ -0,0 +1,3 @@ +<vcpu placement='static'><%= nr_cpu %></vcpu> +<cpu mode='host-passthrough' check='none'> +</cpu> diff --git a/providers/libvirt/templates/devices.xml b/providers/libvirt/templates/devices.xml new file mode 100644 index 0000000..83cd847 --- /dev/null +++ b/providers/libvirt/templates/devices.xml @@ -0,0 +1,3 @@ +<devices> + <graphics type='vnc' autoport='yes' listen='0.0.0.0'><graphics> +</devices> diff --git a/providers/libvirt/templates/disk.xml b/providers/libvirt/templates/disk.xml new file mode 100644 index 0000000..7aecef0 --- /dev/null +++ b/providers/libvirt/templates/disk.xml @@ -0,0 +1,7 @@ +<% if disk %> +<disk type='block' device='disk'> + <driver name='qemu' type='raw'/> + <source dev='<%= disk %>'/> + <target dev='vda' bus='virtio'/> +</disk> +<% end %> diff --git a/providers/libvirt/templates/domain.xml b/providers/libvirt/templates/domain.xml new file mode 100644 index 0000000..a3fdb63 --- /dev/null +++ b/providers/libvirt/templates/domain.xml @@ -0,0 +1,10 @@ +<?xml version="1.0" character="utf-8"?> +<domain type='kvm'> + <resource> + <partition>/machine</partition> + </resource> + <features> + <acpi/> + <gic version='3'/> + </features> +</domain> diff --git a/providers/libvirt/templates/emulator.xml b/providers/libvirt/templates/emulator.xml new file mode 100644 index 0000000..e915215 --- /dev/null +++ b/providers/libvirt/templates/emulator.xml @@ -0,0 +1 @@ +<emulator><%=qemu_path%></emulator> diff --git a/providers/libvirt/templates/interface.xml b/providers/libvirt/templates/interface.xml new file mode 100644 index 0000000..5d4c04f --- /dev/null +++ b/providers/libvirt/templates/interface.xml @@ -0,0 +1,6 @@ +<interface type='bridge'> + <source bridge='br0'/> + <mac address='<%= mac %>'/> + <target dev='vnet0'/> + <model type='virtio'/> +</interface> diff --git a/providers/libvirt/templates/memory.xml b/providers/libvirt/templates/memory.xml new file mode 100644 index 0000000..34aec49 --- /dev/null +++ b/providers/libvirt/templates/memory.xml @@ -0,0 +1 @@ +<memory unit='GiB'><%= memory.to_i %></memory> diff --git a/providers/libvirt/templates/name.xml b/providers/libvirt/templates/name.xml new file mode 100644 index 0000000..5dc98f4 --- /dev/null +++ b/providers/libvirt/templates/name.xml @@ -0,0 +1 @@ +<name><%= job_id %></name> diff --git a/providers/libvirt/templates/options.yaml b/providers/libvirt/templates/options.yaml new file mode 100644 index 0000000..c41ee29 --- /dev/null +++ b/providers/libvirt/templates/options.yaml @@ -0,0 +1,23 @@ +# Domain option: +# use templates/domain.xml if user not specified. +domain: domain + +# Default options: +# these elements must be replaced for each domain.xml. +default: + - name + - os + - memory + - serial + - interface + - active + - emulator + +# Replaceable options: +# these elements can be replaced by user configurations. +replaceable: + - devices + - cpu + - seclabel + - disk + - clock diff --git a/providers/libvirt/templates/os.xml b/providers/libvirt/templates/os.xml new file mode 100644 index 0000000..3ac880d --- /dev/null +++ b/providers/libvirt/templates/os.xml @@ -0,0 +1,8 @@ +<os> + <type arch='<%= arch %>' machine='virt'>hvm</type> + <loader readonly='yes' type='pflash'>/usr/share/edk2/aarch64/QEMU_EFI-pflash.raw</loader> + <kernel><%= kernel %></kernel> + <initrd><%= initrd %></initrd> + <cmdline><%= cmdline %></cmdline> + <nvram>/var/lib/libvirt/qemu/nvram/<%= job_id %>.fd</nvram> +</os> diff --git a/providers/libvirt/templates/seclabel.xml b/providers/libvirt/templates/seclabel.xml new file mode 100644 index 0000000..5c10a77 --- /dev/null +++ b/providers/libvirt/templates/seclabel.xml @@ -0,0 +1,4 @@ +<seclabel type='dynamic' model='dac' relabel='yes'> + <label>+0:+0</label> + <imagelabel>+0:+0</imagelabel> +</seclabel> diff --git a/providers/libvirt/templates/serial.xml b/providers/libvirt/templates/serial.xml new file mode 100644 index 0000000..2edc882 --- /dev/null +++ b/providers/libvirt/templates/serial.xml @@ -0,0 +1,4 @@ +<serial type='file'> + <source path="<%= log_file %>"/> + <target port="0"/> +</serial> -- 2.23.0

1 0

review star
by Wu Fengguang 01 Feb '21

01 Feb '21

Xueliang is our review star again, thanks Xueliang! Review email records in the last 7 days total review paragraph point people 53 23 77 41 Wu Fengguang 89 32 38 32 Cao Xueliang 80 25 32 19 Wei Jihui 66 18 34 7 Xiao Shenwei 25 10 18 6 Li Yuanchao 24 4 12 6 Yu Chuan 49 7 15 6 Luan Shengde 21 7 6 5 Cui Yili 29 10 13 5 Liu Shaofei 22 5 9 3 Lin Jiaxin 34 5 7 3 Wang Yong 12 2 2 2 Bai Jing 22 2 3 1 Lu Weitao 4 1 1 1 Xu Xijian 14 1 3 1 Li Ping 96 4 10 1 Liu Yinsi 46 3 4 1 Wu Zhende 1 0 0 0 Zhang Yale 3 0 0 0 Ren Wen 19 1 2 0 Zhang Dewan 18 2 3 0 Wang Chenglong

1 0

git count
by Wu Fengguang 01 Feb '21

01 Feb '21

Wu Zhende is our code star, thanks Wu Zhende! 12 Wu Zhende 10 Liu Yinsi 9 Xiao Shenwei 7 Cao Xueliang 6 Wei Jihui 6 Yu Chuan 5 Li Yuanchao 4 Lu Weitao 4 Wu Fengguang 3 Bai Jing 3 Wang Chenglong 3 Lin Jiaxin 3 Zhang Dewan 2 Xu Xijian 2 Liu Shaofei 2 Li Ping 1 Wang Yong 1 Luan Shengde 0 Tong Qunfeng 0 Hu Xuejiao 0 Du Kaitian 0 Sun Yukui 0 Zhang Yale 0 Zhang Yu 0 Cui Yili 0 Ren Wen 0 Lu Kaiyi 0 Shi Zhichao

1 0

[PATCH compass-ci] git_mirror.rb: fix that upstream-repos reload failed
by Li Yuanchao 01 Feb '21

01 Feb '21

As the file list get by git command will have a '\n' at the end of each line, the method can't find the file with the wrong name. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 13cb7c0..cc5c7f0 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -268,6 +268,7 @@ class MirrorMain file_list.each_line do |file| next if File.basename(file) == '.ignore' + file = file.chomp repo_dir = "#{REPO_DIR}/#{file}" load_repo_file(repo_dir, File.dirname(file), File.basename(file)) if File.file?(repo_dir) end -- 2.23.0

1 0

[PATCH lab-z9] add host-file of a35 a37 a39
by Zhang Dewan 01 Feb '21

01 Feb '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s48p-256g--a35 | 19 +++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a37 | 27 +++++++++++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a39 | 27 +++++++++++++++++++++++++++ 3 files changed, 73 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a35 create mode 100644 hosts/taishan200-2280-2s64p-256g--a37 create mode 100644 hosts/taishan200-2280-2s64p-256g--a39 diff --git a/hosts/taishan200-2280-2s48p-256g--a35 b/hosts/taishan200-2280-2s48p-256g--a35 new file mode 100644 index 0000000..5d9d500 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a35 @@ -0,0 +1,19 @@ +memory: 256G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886a71a5 + - /dev/disk/by-id/scsi-362c97b1c6eedb00026e527a2b8b998f2 +mac_addr: + - 44:67:47:c9:ec:ec + - 44:67:47:c9:ec:ed + - 44:67:47:c9:ec:ee + - 44:67:47:c9:ec:ef + - 44:67:47:c9:ec:f0 + - 44:67:47:c9:ec:f1 + - 44:67:47:c9:ec:f2 + - 44:67:47:c9:ec:f3 +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +model_name: Kunpeng-920 +ipmi_ip: 9.3.9.12 diff --git a/hosts/taishan200-2280-2s64p-256g--a37 b/hosts/taishan200-2280-2s64p-256g--a37 new file mode 100644 index 0000000..657707a --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a37 @@ -0,0 +1,27 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399c802ef2d +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43322-part1 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43322-part2 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43322-part3 +mac_addr: + - 84:46:fe:89:b9:66 + - 84:46:fe:89:b9:67 + - 84:46:fe:89:b9:68 + - 84:46:fe:89:b9:69 + - ac:b3:b5:1e:d5:eb + - ac:b3:b5:1e:d5:ec + - ac:b3:b5:1e:d5:ed + - ac:b3:b5:1e:d5:ee + - ac:b3:b5:1e:d6:3b + - ac:b3:b5:1e:d6:3c + - ac:b3:b5:1e:d6:3d + - ac:b3:b5:1e:d6:3e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.2.2 diff --git a/hosts/taishan200-2280-2s64p-256g--a39 b/hosts/taishan200-2280-2s64p-256g--a39 new file mode 100644 index 0000000..97d99d9 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a39 @@ -0,0 +1,27 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399981347d5 +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part1 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part2 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part3 +mac_addr: + - 84:46:fe:89:b9:98 + - 84:46:fe:89:b9:99 + - 84:46:fe:89:b9:9a + - 84:46:fe:89:b9:9b + - ac:b3:b5:1e:db:2b + - ac:b3:b5:1e:db:2c + - ac:b3:b5:1e:db:2d + - ac:b3:b5:1e:db:2e + - ac:b3:b5:1e:d8:3b + - ac:b3:b5:1e:d8:3c + - ac:b3:b5:1e:d8:3d + - ac:b3:b5:1e:d8:3e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.2.4 -- 2.23.0

2 4

[PATCH v2 lkp-tests 2/2] sbin/install-dependencies: replace gem sources
by Liu Yinsi 01 Feb '21

01 Feb '21

Encapsulating functions install_gem_pkg for easy gem sources replacement Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/install-dependencies.sh | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index c16385e0f..012c45499 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -13,7 +13,6 @@ linux_dep() ubuntu|debian) export DEBIAN_FRONTEND=noninteractive sudo apt-get install -yqm ruby-dev libssl-dev gcc g++ - sudo gem install -f faye-websocket git activesupport rest-client ;; openEuler|fedora|rhel|centos) if type dnf > /dev/null 2>&1; then @@ -21,7 +20,6 @@ linux_dep() else sudo yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp fi - sudo gem install -f git activesupport rest-client faye-websocket ;; *) echo "$ID not support! please install dependencies manually." && exit 1 @@ -34,7 +32,12 @@ mac_dep() { brew install ruby write_shell_profile "export PATH=/usr/local/opt/ruby/bin:$PATH" - sudo gem install -f git activesupport rest-client +} + +install_gem_pkg() +{ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ + sudo gem install -f git activesupport rest-client faye-websocket } run() @@ -46,6 +49,8 @@ run() else echo "$DISTRO not supported!" && exit 1 fi + + install_gem_pkg } set_env() -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] sbin/install-dependencies: exit if conditions are not met
by Liu Yinsi 01 Feb '21

01 Feb '21

[why] auto install dependencies must meet conditions, if not, exit script. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/install-dependencies.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index f2a9155f6..c16385e0f 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -24,7 +24,7 @@ linux_dep() sudo gem install -f git activesupport rest-client faye-websocket ;; *) - echo "$ID not support! please install dependencies manually." + echo "$ID not support! please install dependencies manually." && exit 1 ;; esac } @@ -44,7 +44,7 @@ run() elif is_system "Darwin"; then mac_dep else - echo "$DISTRO not supported!" + echo "$DISTRO not supported!" && exit 1 fi } -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: delete fields that need to be reloaded for reproduction task
by Li Ping 01 Feb '21

01 Feb '21

[why] 1. load job from result's job yaml has key "my_email", like my_email: compass-ci-robot(a)qq.com my_name: compass-ci-robot lab: z9 2. load job from result's job yaml has key "SCHED_HOST" which from /etc/compass-ci/defaults/*.yaml, like SCHED_HOST: 172.168.131.2 SCHED_PORT: 3000 each of these fields needs to be reset when submit job.yaml for reprodution task. however, these fields already exist in yaml file, can not to be reload, so delete these fields when submit reproduction job. Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/job.rb | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 640b25a5f..867c3d193 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -258,6 +258,13 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end + if hash.has_key?('id') + hash.delete_if { |key, _| key == "SCHED_HOST" } + hash.delete_if { |key, _| key == "SCHED_PORT" } + hash.delete_if { |key, _| key == "my_name" } + hash.delete_if { |key, _| key == "my_email" } + hash.delete_if { |key, _| key == "lab" } + end revise_hash(hash, load_include_yamls(@default_yamls), false) unless @default_yamls.empty? revise_hash(hash, load_include_yamls(@override_yamls), true) unless @override_yamls.empty? -- 2.23.0

3 4

[PATCH v2 compass-ci] providers/lib: introduce resource.rb
by Xiao Shenwei 01 Feb '21

01 Feb '21

function: 1. check whether the host machine has the condition for executing the task 2. download the network files Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/lib/resource.rb | 93 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) create mode 100644 providers/lib/resource.rb diff --git a/providers/lib/resource.rb b/providers/lib/resource.rb new file mode 100644 index 0000000..bcce612 --- /dev/null +++ b/providers/lib/resource.rb @@ -0,0 +1,93 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# initializing QEMU parameters +# qemu libvirt +# - qemu-kvm <emulator></emulator> +# - nr_cpu <vcpu></vcpu> +# - memory <memory></memory> +# - arch <type arch=''></type> +# - log_file <serial></serial> +# - kernel <kernel></kernel> +# - initrd <initrd></initrd> +# - append <cmdline></cmdline> +class Resource + attr_reader :info + + def initialize(hostname, logger) + @hostname = hostname + @logger = logger + @info = {} + qemu_path + arch + log_file + parse_host_config + end + + def parse_response(response) + @response = Hashugar.new(response) + kernel + initrd + cmdline + end + + private + + def qemu_path + @info['qemu_path'] = %x(command -v qemu-kvm).chomp + raise 'can not find available qemu command' if @info['qemu_path'].empty? + end + + def arch + @info['arch'] = %x(arch).chomp + end + + def log_file + @info['log_file'] = "/srv/cci/serial/logs/#{@hostname}" + end + + def parse_host_config + host_file = "#{ENV['LKP_SRC']}/hosts/#{(a)hostname.split('.')[0]}" + @info.merge!(YAML.safe_load(File.read(host_file))) + end + + def load_file(url) + system "wget --timestamping --progress=bar:force #{url}" + basename = File.basename(url) + file_size = %x(ls -s --block-size=M "#{basename}").chomp + @logger.info("Load file size: #{file_size}") + File.realpath(basename) + end + + def kernel + @info['kernel'] = load_file((a)response.kernel_uri) + @logger.info("Kernel path: #{@info['kernel']}") + end + + def merge_initrd_files(file_list, target_name) + return if file_list.size.zero? + + initrds = file_list.join(' ') + system "cat #{initrds} > #{target_name}" + end + + def initrd + initrds_uri = @response.initrds_uri + initrds_path = [] + initrds_uri.each do |url| + initrds_path << load_file(url) + end + merge_initrd_files(initrds_path, 'initrd') + @info['initrd'] = File.realpath('initrd') + @logger.info("Initrd path: #{@info['initrd']}") + end + + def cmdline + @info['cmdline'] = @response.kernel_params + @logger.info("Cmdline: #{@info['cmdline']}") + end +end -- 2.23.0

2 2

[PATCH lkp-tests] setup/simplify-ci: delete default value
by Liu Yinsi 01 Feb '21

01 Feb '21

[why] when call setup/simplify-ci to locally deploy compass-ci client, default value SCHED_HOST 172.17.0.1 is not enough, because compass-ci server deploy in other server, so delete default value, add params $server_ip. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index df18272f3..8af51339e 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -2,8 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${SCHED_HOST:=172.17.0.1} -: ${SCHED_PORT:=3000} +server_ip=$1 git_ci() { @@ -21,7 +20,8 @@ dev_env() { export sched_host=$SCHED_HOST export sched_port=$SCHED_PORT - 3-code/dev-env + [ -n "server_ip" ] || 3-code/dev-env + 3-code/dev-env $server_ip } install_env() -- 2.23.0

3 4

[PATCH compass-ci 4/4] sparrow/5-build: pack lkp-tests
by Liu Yinsi 01 Feb '21

01 Feb '21

[why] for user locally deploy compass-ci, submit job will check lkp-$arch.cgz exist, so call container lkp-initrd/run in sparrow script, it is convenient and does not need users to operate. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/5-build/sources | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/5-build/sources b/sparrow/5-build/sources index 48a740b..cea3b00 100755 --- a/sparrow/5-build/sources +++ b/sparrow/5-build/sources @@ -3,3 +3,5 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. $CCI_SRC/sbin/download-rootfs + +$CCI_SRC/container/lkp-initrd/run -- 2.23.0

2 2

[PATCH compass-ci] container/srv-http: bind mount volumn
by Liu Yinsi 01 Feb '21

01 Feb '21

[why] for locally deploy compass-ci, if their host can't access internet, we provider service for them to download docker images cgz file, then they no need to build images by themselves. usage: wget http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT}/initrd/dockerimage/ima… Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/srv-http/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/srv-http/start b/container/srv-http/start index 945cde0..fa56fca 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -14,6 +14,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result:/srv/result:ro -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro + -v /srv/initrd/dockerimage:/srv/initrd/dockerimage:ro -v /srv/os:/srv/os:ro -v /srv/git/archlinux:/srv/git/archlinux:ro -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro -- 2.23.0

2 1

[PATCH v2 compass-ci] lib/scheduler_api.cr: provide scheduler api
by Wu Zhende 01 Feb '21

01 Feb '21

Enable other crystal services to invoke the scheduler api. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/scheduler_api.cr | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 src/lib/scheduler_api.cr diff --git a/src/lib/scheduler_api.cr b/src/lib/scheduler_api.cr new file mode 100644 index 0000000..b8dd514 --- /dev/null +++ b/src/lib/scheduler_api.cr @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "http/client" + +# scheduler API class +class SchedulerAPI + def initialize + @port = ENV.has_key?("SCHED_PORT") ? ENV["SCHED_PORT"].to_i32 : 3000 + @host = ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1" + end + + def close_job(job_id) + client = HTTP::Client.new(@host, port: @port) + response = client.get("/~lkp/cgi-bin/lkp-post-run?job_id=#{job_id}") + client.close() + return response + end +end -- 2.23.0

2 1

[PATCH v2 compass-ci] scheduler/close_job.cr: job_state can be specified
by Wu Zhende 01 Feb '21

01 Feb '21

The job may be closed due to timeout or other reasons Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/close_job.cr | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index bce6b33..b3e6716 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -8,6 +8,10 @@ class Sched job = @redis.get_job(job_id) + # update job_state + job_state = @env.params.query["job_state"]? + job["job_state"] = job_state if job_state + response = @es.set_job_content(job) if response["_id"] == nil # es update fail, raise exception @@ -26,7 +30,8 @@ class Sched @redis.remove_finished_job(job_id) - @log.info(%({"job_id": "#{job_id}", "job_state": "complete"})) + job_state ||= "complete" + @log.info(%({"job_id": "#{job_id}", "job_state": "#{job_state}"})) rescue e @log.warn(e) end -- 2.23.0

2 1

[PATCH v1 compass-ci] providers/lib: introduce resource.rb
by Xiao Shenwei 01 Feb '21

01 Feb '21

function: 1. check whether the host machine has the condition for executing the task 2. download the network files Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/lib/resource.rb | 93 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) create mode 100644 providers/lib/resource.rb diff --git a/providers/lib/resource.rb b/providers/lib/resource.rb new file mode 100644 index 0000000..93bc9ce --- /dev/null +++ b/providers/lib/resource.rb @@ -0,0 +1,93 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# initializing QEMU parameters +# qemu libvirt +# - qemu-kvm <emulator></emulator> +# - nr_cpu <vcpu></vcpu> +# - memory <memory></memory> +# - arch <type arch=''></type> +# - log_file <serial></serial> +# - kernel <kernel></kernel> +# - initrd <initrd></initrd> +# - append <cmdline></cmdline> +class Resource + attr_reader :info + + def initialize(hostname, logger) + @hostname = hostname + @logger = logger + @info = {} + qemu_path + arch + log_file + parse_host_config + end + + def parse_response(response) + @response = Hashugar.new(response) + kernel + initrd + cmdline + end + + private + + def qemu_path + @info['qemu_path'] = %x(command -v qemu-kvm).chomp + raise 'can not find available qemu command' if @info['qemu_path'].empty? + end + + def arch + @info['arch'] = %x(arch).chomp + end + + def log_file + @info['log_file'] = "/srv/cci/serial/logs/#{@hostname}" + end + + def parse_host_config + host_file = "#{ENV['LKP_SRC']}/hosts/#{(a)hostname.split('.')[0]}" + @info.merge!(YAML.safe_load(File.read(host_file))) + end + + def load_file(url) + system "wget --timestamping --progress=bar:force #{url}" + basename = File.basename(url) + file_size = %x(ls -s --block-size=M "#{basename}").chomp + @logger.info("Load file size: #{file_size}") + File.realpath(basename) + end + + def kernel + @info['kernel'] = load_file((a)response.kernel_uri) + @logger.info("Kernel path: #{@info['kernel']}") + end + + def merge_initrd_files(file_list, target_name) + return if file_list.size.zero? + + initrds = file_list.join(' ') + system "cat #{initrds} > #{target_name}" + end + + def initrd + initrds_uri = @response.initrds_uri + initrds_path = [] + initrds_uri.each do |url| + initrds_path << load_file(url) + end + merge_initrd_files(initrds_path, 'initrd') + @info['initrd'] = File.realpath('initrd') + @logger.info("Initrd path: #{@info['initrd']}") + end + + def cmdline + @info['cmdline'] = @response.kernel_params + @logger.info("Cmdline : #{@info['cmdline']}") + end +end -- 2.23.0

2 2

[PATCH compass-ci] src/lib/job.cr: check host files in $lab repos
by Li Ping 01 Feb '21

01 Feb '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 964c520..12f1e05 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -143,6 +143,7 @@ class Job check_required_keys() check_account_info() + get_lab_hosts_file() set_defaults() end @@ -382,6 +383,22 @@ class Job end end + private def get_lab_hosts_file + if !@hash["host_file"]? + lab_hosts_file_name = @hash["testbox"] + error_msg = "hosts file not exist: #{lab_hosts_file_name}" + lab_hosts_file = "#{ENV["CCI_REPOS"]}/lab-#{@hash["lab"]}/hosts/#{lab_hosts_file_name}" + if File.file?(lab_hosts_file) + host_hash = YAML.parse(File.read(lab_hosts_file)) + @hash.merge!(JSON.parse(host_hash.to_json).as_h) + else + raise error_msg + end + else + @hash.delete("host_file") + end + end + private def check_account_info error_msg = "Failed to verify the account.\n" error_msg += "Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco…" -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: delete get_lab_host_file
by Li Ping 01 Feb '21

01 Feb '21

[why] lab_host_files are on the server side and should be read on the server side. Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/job.rb | 23 ++++++----------------- 1 file changed, 6 insertions(+), 17 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 867c3d193..087ca226e 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -354,6 +354,7 @@ class Job if hosts_file hwconfig = load_yaml(hosts_file, {}) @job[source_file_symkey(hosts_file)] = nil + @job["host_file"] = "#{hosts_file}" @job.merge!(hwconfig) { |_k, a, _b| a } # job's key/value has priority over hwconfig @load_hosts_done = true end @@ -369,26 +370,14 @@ class Job return @job.include?('tbox_group') end - def get_lab_hosts_file - lab_hosts_file_name = @job['testbox'] - lab_hosts_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{lab_hosts_file_name}" - - if File.file?(lab_hosts_file) - lab_hosts_file - else - nil - end - end - def get_hosts_file - lab_hosts_file = get_lab_hosts_file - return lab_hosts_file if lab_hosts_file - hosts_file_name = @job['tbox_group'].split(/\.|--/)[0] hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" - return hosts_file if File.file?(hosts_file) - - raise ArgumentError, "hosts file not exist: #{hosts_file_name}, maybe need check testbox field" + if File.file?(hosts_file) + hosts_file + else + nil + end end def include_files -- 2.23.0

1 0

[PATCH v1 compass-ci] providers/libvirt: check host resource and download file
by Xiao Shenwei 01 Feb '21

01 Feb '21

if the host machine does not have the capability of executing tasks, raise error message. if not, download dependent file based on response Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/run.rb | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/providers/libvirt/run.rb b/providers/libvirt/run.rb index b4f8b4c..a500c31 100755 --- a/providers/libvirt/run.rb +++ b/providers/libvirt/run.rb @@ -8,6 +8,7 @@ require 'digest/md5' require_relative "#{ENV['CCI_SRC']}/lib/log" require_relative "#{ENV['CCI_SRC']}/lib/sched_client" require_relative "#{ENV['CCI_SRC']}/providers/lib/context" +require_relative "#{ENV['CCI_SRC']}/providers/lib/resource" def create_logger(hostname) filename = "#{hostname}.log" @@ -51,10 +52,13 @@ def main(hostname, queues) logger = create_logger(hostname) mac = compute_mac hostname context = Context.new(mac, hostname, queues) + resource = Resource.new(hostname, logger) sched_client = SchedClient.new response = request_job(context, sched_client, logger) return if response.nil? + resource.parse_response(response) + context.merge!(resource.info) sched_client.delete_mac2host(mac) sched_client.delete_host2queues(hostname) end -- 2.23.0

1 0

[PATCH compass-ci 4/4] container/submit: README file
by Luan Shengde 01 Feb '21

01 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/README.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 container/submit/README.md diff --git a/container/submit/README.md b/container/submit/README.md new file mode 100644 index 0000000..0c66118 --- /dev/null +++ b/container/submit/README.md @@ -0,0 +1,23 @@ +# install lkp-tests and its depends in container +# with docker container, will suit almost all the linux OS. +# avoid much repetitive testing work to adapt. + +prepare work: + apply account + config default yaml + generate ssh keys + +installation: + git clone https://gitee.com/wu_fengguang/compass-ci.git + cd compass-ci/container/submit + ./build + edit executable file: run + add map: + $HOME/.config:/root/.config:ro + $HOME/.ssh:/root/.ssh:ro + /etc/compass-ci:/etc/compass-ci:ro + add soft link for executable file 'run' to $PATH: + ln -s $PWD/run /usr/bin/submit + +then you can directly use the command 'submit' to submit jobs +it's the same as you install the lkp-tests at your localhost -- 2.23.0

2 2

[PATCH v5 compass-ci 2/2] providers/qemu: add name parameter for qemu-kvm
by Xiao Shenwei 01 Feb '21

01 Feb '21

[why] we need reboot a vm based on hostname or job_id. so, add hostname and job_id to vm process information. [how] -name guest=$hostname,process=$job_id if you need reboot the vm: pkill -f $job_id Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/qemu/kvm.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 66099ee..dd657cb 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -39,6 +39,9 @@ parse_ipxe_script() ;; initrd) file=$(basename "$b") + [ $file == "job.cgz" ] && { + job_id=$(basename $(dirname "$b")) + } wget --timestamping -a ${log_file} --progress=bar:force $b initrds+="$file " ;; @@ -144,6 +147,7 @@ public_option() { kvm=( $qemu + -name guest=$hostname,process=$job_id -kernel $kernel -initrd $initrd -smp $nr_cpu -- 2.23.0

1 0