- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] lib/job.cr: temporarily cancel account verification
by Wu Zhende 22 Dec '20

22 Dec '20

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index e6f7e6a..2554db1 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -347,13 +347,14 @@ class Job error_msg = "Failed to verify the account.\n" error_msg += "Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco…" account_info = @es.get_account(self["my_email"]) - raise account_info unless account_info.is_a?(JSON::Any) + # raise account_info unless account_info.is_a?(JSON::Any) - @account_info = account_info.as_h + # @account_info = account_info.as_h + @account_info = account_info.as_h if account_info.is_a?(JSON::Any) - raise error_msg if @account_info["found"]? == false - raise error_msg unless self["my_name"] == @account_info["my_name"].to_s - raise error_msg unless self["my_token"] == @account_info["my_token"] + # raise error_msg if @account_info["found"]? == false + # raise error_msg unless self["my_name"] == @account_info["my_name"].to_s + # raise error_msg unless self["my_token"] == @account_info["my_token"] @hash.delete("my_uuid") @hash.delete("my_token") -- 2.23.0

1 0

[PATCH lkp-tests] adaptation/archlinux: add archlinux mapping
by Wang Yong 22 Dec '20

22 Dec '20

add mapping in order to run cci-depends job for build-pkg Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/adaptation/archlinux | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/distro/adaptation/archlinux b/distro/adaptation/archlinux index 89163cf41..c52212e08 100644 --- a/distro/adaptation/archlinux +++ b/distro/adaptation/archlinux @@ -72,3 +72,11 @@ libtirpc: libtirpc1 initramfs-tools: openssh-server: openssh openssh-client: openssh +autopoint: gettext +g++: gcc +libarchive13: libarchive +libbz2-1.0: bzip2 +libssl: openssl +libtool-bin: libtool +linux-perf: perf +ruby-dev: ruby -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests: fix return value
by Zhang Yu 22 Dec '20

22 Dec '20

[why] if /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0 both not exit, the value of return should be 0 not 1 Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/host-info b/tests/host-info index 3984727b4..fe55d1ff2 100755 --- a/tests/host-info +++ b/tests/host-info @@ -102,7 +102,7 @@ get_ipmi_ip() # 802.1q VLAN ID : Disabled # RMCP+ Cipher Suites : 0,1,2,3,17 # Cipher Suite Priv Max : XuuaXXXXXXXXXXX - [ -e /dev/ipmi0 ] || [ -e /dev/ipmi/0 ] || [ -e /dev/ipmidev/0 ] || return + [ -e /dev/ipmi0 ] || [ -e /dev/ipmi/0 ] || [ -e /dev/ipmidev/0 ] || return 0 local ip=$(ipmitool lan print | awk 'NR==3{print $4}') [ -n "$ip" ] && echo "ipmi_ip: $ip" } -- 2.23.0

1 0

[PATCH v4 compass-ci 1/6] fetch-mail: config file for fetchmail
by Luan Shengde 22 Dec '20

22 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/fetchmailrc.template | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 container/fetch-mail/fetchmailrc.template diff --git a/container/fetch-mail/fetchmailrc.template b/container/fetch-mail/fetchmailrc.template new file mode 100644 index 0000000..8c7e82b --- /dev/null +++ b/container/fetch-mail/fetchmailrc.template @@ -0,0 +1,5 @@ +poll imap.qq.com with proto IMAP +username "FETCH_MAIL_ADDRESS" +password "FETCH_MAIL_AUTH_CODE" +mda "/usr/bin/procmail -f %F -d %T" +nokeep -- 2.23.0

2 2

[PATCH v4 compass-ci 2/6] fetch-mail: config file for procmail
by Luan Shengde 22 Dec '20

22 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/procmailrc.template | 25 ++++++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 container/fetch-mail/procmailrc.template diff --git a/container/fetch-mail/procmailrc.template b/container/fetch-mail/procmailrc.template new file mode 100644 index 0000000..9cea4bc --- /dev/null +++ b/container/fetch-mail/procmailrc.template @@ -0,0 +1,25 @@ +DROPPRIVS=yes +PATH=$HOME/bin:/usr/local/bin:/usr/bin:/bin +MAILDIR=FETCH_MAIL_DIR +MAILLIST=${MAILDIR} +DEFAULT=${MAILDIR}/FETCH_MAIL_BOX/ +ORGMAIL=${DEFAULT}/FETCH_MAIL_BOX/ +LOGFILE=${MAILDIR}/log/procmail # recommended + +UMASK=002 + +DISPLAY=:0 +XAUTHORITY=$HOME/.Xauthority + +# One often receives duplicate messages for cross-postings to mailing +# lists, or explicit replies to a post from you in addition to the +# mailing list message. +# +# The following recipe eliminates all duplicate messages. It tells +# formail(1) to keep a cache file of the Message-IDs of the most recent +# messages you've received. Since Message-IDs are guaranteed unique for +# distinct messages, they are ideally suited to weed out duplicates. + +# LOCKFILE=$MAILDIR/.msgid.lock +:0 Wh: $MAILDIR/.msgid.lock +| formail -D 655360 $MAILDIR/log/msgid -- 2.23.0

2 2

[PATCH v2 compass-ci] lib/matrix2.rb: fix rubocop warning
by Lu Weitao 22 Dec '20

22 Dec '20

[Why] lib/matrix2.rb:81:1: C: Metrics/AbcSize: Assignment Branch Condition size for create_matrix is too high. [<5, 16, 5> 17.49/15] def create_matrix(job_list) ... ^^^^^^^^^^^^^^^^^^^^^^^^^^^ lib/matrix2.rb:81:1: C: Metrics/CyclomaticComplexity: Cyclomatic complexity for create_matrix is too high. [9/7] def create_matrix(job_list) ... ^^^^^^^^^^^^^^^^^^^^^^^^^^^ lib/matrix2.rb:81:1: C: Metrics/MethodLength: Method has too many lines. [18/15] def create_matrix(job_list) ... ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index a7b83c6..19eb8f1 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -83,13 +83,7 @@ def create_matrix(job_list) suites = [] job_list.each do |job| stats = job['stats'] - next unless stats - - if job['suite'] - next unless stats.keys.any? { |stat| stat.start_with?(job['suite']) } - - suites << job['suite'] - end + next unless stats && assign_suites(suites, job, stats) stats.each do |key, value| next if key.include?('timestamp') @@ -103,6 +97,12 @@ def create_matrix(job_list) return matrix, suites end +def assign_suites(suites, job, stats) + return unless job['suite'] && stats.keys.any? { |stat| stat.start_with?(job['suite']) } + + suites << job['suite'] +end + # input: query results from es_query # return: matrix def combine_query_data(query_data) -- 2.23.0

1 0

[PATCH compass-ci] lib/matrix2.rb: fix rubocop offenses
by Lu Kaiyi 22 Dec '20

22 Dec '20

[error] Inspecting 1 file C Offenses: matrix2.rb:160:35: C: Style/TrailingUnderscoreVariable: Do not use trailing _s in parallel assignment. Prefer groups[group_key][dim_key], = create_matrix(jobs). groups[group_key][dim_key], _ = create_matrix(jobs) ^^ Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index dfc606e..5f538bd 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -157,7 +157,7 @@ def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) groups.each do |group_key, dims| dims.each do |dim_key, jobs| - groups[group_key][dim_key], _ = create_matrix(jobs) + groups[group_key][dim_key], = create_matrix(jobs) end end -- 2.23.0

1 0

[PATCH compass-ci] lib/matrix2.rb: fix rubocop warning
by Lu Weitao 22 Dec '20

22 Dec '20

[Why] lib/matrix2.rb:81:1: C: Metrics/AbcSize: Assignment Branch Condition size for create_matrix is too high. [<5, 16, 5> 17.49/15] def create_matrix(job_list) ... ^^^^^^^^^^^^^^^^^^^^^^^^^^^ lib/matrix2.rb:81:1: C: Metrics/CyclomaticComplexity: Cyclomatic complexity for create_matrix is too high. [9/7] def create_matrix(job_list) ... ^^^^^^^^^^^^^^^^^^^^^^^^^^^ lib/matrix2.rb:81:1: C: Metrics/MethodLength: Method has too many lines. [18/15] def create_matrix(job_list) ... ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index a7b83c6..fc805d8 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -85,12 +85,7 @@ def create_matrix(job_list) stats = job['stats'] next unless stats - if job['suite'] - next unless stats.keys.any? { |stat| stat.start_with?(job['suite']) } - - suites << job['suite'] - end - + assign_suites(suites, job, stats) stats.each do |key, value| next if key.include?('timestamp') @@ -103,6 +98,12 @@ def create_matrix(job_list) return matrix, suites end +def assign_suites(suites, job, stats) + return unless job['suite'] && stats.keys.any? { |stat| stat.start_with?(job['suite']) } + + suites << job['suite'] +end + # input: query results from es_query # return: matrix def combine_query_data(query_data) -- 2.23.0

1 1

[PATCH v2 compass-ci] lib/matrix2.rb remove the *timestamp* from matrix
by Lu Weitao 22 Dec '20

22 Dec '20

[Why] In compare results: the value of *timestamp*(metric) doesn't mean the number of occurrences, it may confuse user like bellow: openeuler centos archLinux metric -------------------- ------------------------------ ------------------------------ ------------------------------ fails:runs change fails:runs change fails:runs | | | | | 3585:13 +6913.7% 344:1 -2.6e+4% 11:1 dmesg.timestamp:last 2350:13 +1.6e+4% 344:1 -1.7e+4% 10:1 kmsg.timestamp:last 2:13 -15.4% 0:1 +84.6% 1:1 last_state.exit_fail [How] remove the metric that include "timestamp" from matrix Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index c712e32..2769f7c 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -87,6 +87,8 @@ def create_matrix(job_list) next unless stats stats.each do |key, value| + next if key.include?('timestamp') + matrix[key] = [] unless matrix[key] matrix[key] << value end -- 2.23.0

3 3

[PATCH compass-ci] lib/matrix2.rb: remove duplicate code
by Lu Weitao 22 Dec '20

22 Dec '20

[Why] Had handle empty group in lib/params_group.rb def filter_groups(groups) groups.each do |group_key, value| value.each_key do |dim_key| value.delete(dim_key) if value[dim_key].empty? end groups.delete(group_key) if groups[group_key].empty? end end Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 4 ---- 1 file changed, 4 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index dfc606e..a7b83c6 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -124,10 +124,6 @@ def combine_group_query_data(job_list, dims) suites_hash = {} groups = auto_group(job_list, dims) groups.each do |group_key, value| - if value.empty? - groups.delete(group_key) - next - end suite_list = [] value.each do |dimension_key, jobs| groups[group_key][dimension_key], suites = create_matrix(jobs) -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests/build-pkg: add config file support
by Wang Yong 22 Dec '20

22 Dec '20

[Why] config file in some software need storage and recall now we can add config value in yaml and cp ${config} to ${cci_dir}/${pkgname}/ in PKGBUILD so that config file can be uploaded correctly $ cat test.yaml ... config: ${config} ... Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index addff9f7b..06a58d498 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -4,6 +4,7 @@ # - upstream_commit # - git_tag # - pkg_ver +# - config # upstream_url # We'll use the below global env vars, but not treat them as parameter to avoid @@ -20,6 +21,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/distro/common +. $LKP_SRC/lib/http.sh check_vars() { @@ -31,6 +33,29 @@ check_vars() [ -n "$os_mount" ] || die "os_mount is empty" } +check_config_var() +{ + [ -n "$config" ] || return +} + +get_config() +{ + local cci_http_host=${INITRD_HTTP_HOST} + local cci_http_port=${INITRD_HTTP_PORT:-8800} + export cci_dir=/cci/build-config + local remote_dir="http://${cci_http_host}:${cci_http_port}${cci_dir}/${pkgname}" + + mkdir -p ${cci_dir} + + check_config_var + http_get_file "${remote_dir}/${config}" "${cci_dir}/${pkgname}/${config}" +} + +upload_config() +{ + upload_to_target_dir "${cci_dir}/${pkgname}" +} + mount_dest() { # the same image is mounted to cifs and nfs, the generated cgz files @@ -46,7 +71,7 @@ get_pkgfile() { curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "${2##*/}" - } +} request_pkg() { @@ -125,13 +150,17 @@ build_source_pkg() fi } replace_source PKGBUILD.src - create_cgz_name + get_config PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf - [ "$?" == 0 ] || exit 1 + local build_state="$?" + + upload_config + + [ "${build_state}" == "0" ] || exit 1 create_softlink upload_to_target_dir ${PKG_MNT}/${pack_to}/${pkgname} } -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/job.cr: check account info
by Wu Zhende 22 Dec '20

22 Dec '20

If submit yaml in /srv/result, will not check account info. Fix it Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4d0b5df..e6f7e6a 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -51,17 +51,23 @@ class Job def initialize(job_content : JSON::Any, id) @hash = job_content.as_h + @es = Elasticsearch::Client.new + @account_info = Hash(String, JSON::Any).new # init job with "-1", or use the original job_content["id"] id = "-1" if "#{id}" == "" if initialized? - return if @hash["id"] == "#{id}" + if @hash["id"] == "#{id}" + return unless @hash.has_key?("my_uuid") || @hash.has_key?("my_token") + + check_account_info() + return + end end - @es = Elasticsearch::Client.new @hash["id"] = JSON::Any.new("#{id}") - @account_info = Hash(String, JSON::Any).new + check_required_keys() check_account_info() set_defaults() -- 2.23.0

2 1

[PATCH v2 lkp-tests] daemon: fix ssh login testbox for docker
by Xiao Shenwei 22 Dec '20

22 Dec '20

[why] systemctl is not supported when use docker [how] use the sshd command Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- daemon/sshd | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/daemon/sshd b/daemon/sshd index 41fa99adf..6bfda467e 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -6,19 +6,31 @@ # sshr_port_base . $LKP_SRC/lib/http.sh +. $LKP_SRC/lib/env.sh sshr_ip=$LKP_SERVER [ -n "$sshr_port" ] || sshr_port=5050 [ -n "$sshr_port_len" ] || sshr_port_len=2000 [ -n "$sshr_port_base" ] || sshr_port_base=50000 +generate_key_file() +{ + echo -e "y\n" | ssh-keygen -t rsa -b 2048 -N '' -f /etc/ssh/ssh_host_rsa_key +} + run_ssh() { [ -n "$my_ssh_pubkey" ] || return umask 0077 mkdir -p /root/.ssh echo "$my_ssh_pubkey" > /root/.ssh/authorized_keys - systemctl start sshd + if is_docker; then + generate_key_file + sshd_path=$(cmd_path sshd) + $sshd_path -q + else + systemctl start sshd + fi } data_success() -- 2.23.0

4 5

Re: [PATCH lkp-tests] fix lkp make install error
by Wu Fengguang 22 Dec '20

22 Dec '20

TO ALL Best practice is to use realpath. >-. $LKP_SRC/lib/env.sh >-. $LKP_SRC/sbin/set-env.sh >+. lib/env.sh >+. sbin/set-env.sh SCRIPT_DIR=$(dirname $(realpath $0)) PROJECT_DIR=$(dirname $SCRIPT_DIR) $PROJECT_DIR/lib/env.sh $PROJECT_DIR/sbin/set-env.sh Thanks, Fengguang

2 1

[PATCH lkp-tests] daemon/multi-qemu: add $CACHE_DIR judgment
by Hu Xuejiao 22 Dec '20

22 Dec '20

$CACHE_DIR is not created in crystal Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- daemon/multi-qemu | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/daemon/multi-qemu b/daemon/multi-qemu index 90197dae..87b40758 100755 --- a/daemon/multi-qemu +++ b/daemon/multi-qemu @@ -13,7 +13,7 @@ multi_qemu() export CCI_SRC=${CCI_SRC:-"/c/compass-ci"} local hostname=$tbox_group.$HOSTNAME - cd "$CACHE_DIR" + [ -n "$CACHE_DIR" ] && cd "$CACHE_DIR" $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" } -- 2.23.0

3 2

[PATCH compass-ci] lib: add assign_account_client
by Luan Shengde 22 Dec '20

22 Dec '20

add assign_account_client for code calling to assign account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign_account_client.rb | 83 ++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100755 lib/assign_account_client.rb diff --git a/lib/assign_account_client.rb b/lib/assign_account_client.rb new file mode 100755 index 0000000..d7e0623 --- /dev/null +++ b/lib/assign_account_client.rb @@ -0,0 +1,83 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require 'set' +require 'optparse' +require_relative '../container/defconfig' +require_relative 'es_client' +require_relative '../container/mail-robot/lib/assign-account-email' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_HOST + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) +JUMPER_HOST = defaults['JUMPER_HOST'] +JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 +LAB = ENV['lab'] + +# used for other codes calling to assign account for user +class AutoAssignAccount + def initialize(user_info) + @my_info = user_info + + @my_info_es = {} + end + + def update_from_es + account_infos = ESQuery.new(index: 'accounts') + my_account_info = account_infos.query_by_id(@my_info['my_email']) || {} + + my_account_info.update @my_info + @my_info.update my_account_info + @my_info['my_ssh_pubkey'] = [] unless my_account_info['my_ssh_pubkey'] + + @my_info['my_token'] = %x(uuidgen).chomp if @my_info['my_login_name'].nil? + end + + def apply_account + apply_info = {} + + apply_info.update @my_info + apply_info['enable_login'] = true + apply_info['is_update_account'] = true unless apply_info['my_login_name'].nil? + apply_info['lab'] = LAB + + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') + JSON.parse account_info_str + end + + def update_my_info_from_account_info + account_info = apply_account + + @my_info['my_login_name'] = account_info['my_login_name'] + @my_info['my_ssh_pubkey'] << account_info['my_jumper_pubkey'] unless account_info['my_jumper_pubkey'].nil? + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def send_mail + message = build_apply_account_email(@my_info) + %x(curl -XPOST "#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text" -d "#{message}") + end + + def send_account + update_from_es + update_my_info_from_account_info + + store_account_info + send_mail + end +end -- 2.23.0

2 2

[PATCH v2 lkp-tests 1/4] bin/install-dependencies.sh: refactor file
by Wei Jihui 22 Dec '20

22 Dec '20

all dependent functions is move to lib/env.sh, sbin/set-env.sh in not required. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/install-dependencies.sh | 40 +++++++++++++++++------------------- 1 file changed, 19 insertions(+), 21 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 7605a29f..b4d3ad22 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -1,30 +1,28 @@ #!/bin/bash . $LKP_SRC/lib/env.sh -. $LKP_SRC/sbin/set-env.sh -# choose install function base on -# DISTRIBUTION +# choose install function base on DISTRIBUTION linux_dep() { -source /etc/os-release -case $ID in -ubuntu|debian) - export DEBIAN_FRONTEND=noninteractive - apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client - ;; -openEuler|fedora|rhel|centos) - if type dnf > /dev/null 2>&1; then - dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - else - yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - fi - sudo gem install -f git activesupport rest-client - ;; -*) - echo "$ID not support! please install dependencies manually." - ;; -esac + source /etc/os-release + case $ID in + ubuntu|debian) + export DEBIAN_FRONTEND=noninteractive + apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client + ;; + openEuler|fedora|rhel|centos) + if type dnf > /dev/null 2>&1; then + dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + else + yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + fi + sudo gem install -f git activesupport rest-client + ;; + *) + echo "$ID not support! please install dependencies manually." + ;; + esac } -- 2.23.0

3 3

[PATCH v2 lkp-tests] tests/build-pkg: add config file support
by Wang Yong 21 Dec '20

21 Dec '20

[Why] config file in some software need storage and recall now we can add config value in yaml and cp ${config} to ${cci_dir}/${benchmark}/ in PKGBUILD so that config file can be uploaded correctly $ cat test.yaml ... config: ${config} ... Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index addff9f7b..e539be923 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -4,6 +4,7 @@ # - upstream_commit # - git_tag # - pkg_ver +# - config # upstream_url # We'll use the below global env vars, but not treat them as parameter to avoid @@ -20,6 +21,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/distro/common +. $LKP_SRC/lib/http.sh check_vars() { @@ -31,6 +33,29 @@ check_vars() [ -n "$os_mount" ] || die "os_mount is empty" } +check_config_var() +{ + [ -n "$config" ] || return +} + +get_config() +{ + local cci_http_host=${INITRD_HTTP_HOST} + local cci_http_port=${INITRD_HTTP_PORT:-8800} + export cci_dir=/cci/config + local remote_dir="http://${cci_http_host}:${cci_http_port}${cci_dir}/${pkgname}" + + mkdir -p ${cci_dir} + + check_config_var + http_get_file "${remote_dir}/${config}" "${cci_dir}/${pkgname}/${config}" +} + +upload_config() +{ + upload_to_target_dir "${cci_dir}/${pkgname}" +} + mount_dest() { # the same image is mounted to cifs and nfs, the generated cgz files @@ -46,7 +71,7 @@ get_pkgfile() { curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "${2##*/}" - } +} request_pkg() { @@ -125,13 +150,17 @@ build_source_pkg() fi } replace_source PKGBUILD.src - create_cgz_name + get_config PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf - [ "$?" == 0 ] || exit 1 + local build_state="$?" + + upload_config + + [ "${build_state}" == "0" ] || exit 1 create_softlink upload_to_target_dir ${PKG_MNT}/${pack_to}/${pkgname} } -- 2.23.0

1 0

[PATCH lkp-tests] daemon/multi-docker: add default value of queues
by Hu Xuejiao 21 Dec '20

21 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- daemon/multi-docker | 2 ++ 1 file changed, 2 insertions(+) diff --git a/daemon/multi-docker b/daemon/multi-docker index 257645b8..7759fdf3 100755 --- a/daemon/multi-docker +++ b/daemon/multi-docker @@ -6,6 +6,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +: ${queues:="$tbox_group.$HOSTNAME,$tbox_group.$(arch)"} + multi_docker() { export CCI_SRC=${CCI_SRC:-"/c/compass-ci"} -- 2.23.0

1 0

[PATCH v4 compass-ci 3/3] lib: use suites_hash instead of suites_list to keep consistent
by Lu Kaiyi 21 Dec '20

21 Dec '20

[why] suite_list maybe not consistent with group key if sort group matrixes. [how] use suites_hash instead of suites_list to keep correct congruent relationship with same group_key. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 4 ++-- lib/compare_matrixes.rb | 10 +++++----- lib/matrix2.rb | 6 +++--- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index da761ef..9090c3b 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -62,8 +62,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices, suites_list = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, suites_list, options) + groups_matrices, suites_hash = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_hash, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 0727797..6e4c2aa 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -387,25 +387,25 @@ end # input: group matrices # output: pre compare result of each group # the result with more comparison objects first -def compare_group_matrices(group_matrices, suites_list, options) +def compare_group_matrices(group_matrices, suites_hash, options) result_str = '' group_matrices_array = sort_by_matrix_size(group_matrices) group_matrices_array.each do |matrice_kv| - result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_list, options) + result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_hash[matrice_kv[0]], options) end result_str end -def get_matrix_str(matrice_key, matrice_value, suites_list, options) +def get_matrix_str(matrice_key, matrice_value, suite_list, options) m_list = [] m_titles = [] matrice_value.each do |dim, matrix| m_titles << dim m_list << matrix end - return compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) if options[:no_print] + return compare_matrixes(m_list, suite_list, m_titles, matrice_key, options: options) if options[:no_print] - print compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) + print compare_matrixes(m_list, suite_list, m_titles, matrice_key, options: options) return '' end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 4393a65..dfc606e 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -121,7 +121,7 @@ end # ... # } def combine_group_query_data(job_list, dims) - suites_list = [] + suites_hash = {} groups = auto_group(job_list, dims) groups.each do |group_key, value| if value.empty? @@ -133,10 +133,10 @@ def combine_group_query_data(job_list, dims) groups[group_key][dimension_key], suites = create_matrix(jobs) suite_list.concat(suites) end - suites_list << suite_list + suites_hash[group_key] = suite_list end - return groups, suites_list + return groups, suites_hash end # input: -- 2.23.0

1 0

[PATCH v4 compass-ci 2/3] lib/compare_matrixes.rb: don't remove the field that start with $suite
by Lu Kaiyi 21 Dec '20

21 Dec '20

if the metric field start with $suite, even if the field is unchanged, still keep it in compare result. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_matrixes.rb | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index b7ea5a4..0727797 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -176,11 +176,15 @@ def get_matrixes_values(matrixes_list, options) matrixes_values end -def remove_unchanged_field(matrixes_values) +def remove_unchanged_field(matrixes_values, suite_list) # remove unchanged field from matrixes values and remove :changed key # matrixes_values.each_key do |success| matrixes_values[success].delete_if do |field| + if suite_list.any? { |suite| field.start_with?(suite) } + matrixes_values[success][field][:changed] = true + end + !matrixes_values[success][field].delete(:changed) end end @@ -203,7 +207,7 @@ def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key options = { 'perf-profile': 5, theme: :none, no_print: false }.merge(options) matrixes_values = get_matrixes_values(matrixes_list, options) - remove_unchanged_field(matrixes_values) if matrixes_list.length > 1 + remove_unchanged_field(matrixes_values, suite_list) if matrixes_list.length > 1 no_print = options[:no_print] result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( -- 2.23.0

1 0

[PATCH v4 compass-ci 1/3] lib/matrix2.rb: discard the no $suite related job
by Lu Kaiyi 21 Dec '20

21 Dec '20

if job has suite, but no $suite related stats metric, then discard this job when create matrix. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index bff4eee..4393a65 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -82,10 +82,15 @@ def create_matrix(job_list) matrix = {} suites = [] job_list.each do |job| - suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats + if job['suite'] + next unless stats.keys.any? { |stat| stat.start_with?(job['suite']) } + + suites << job['suite'] + end + stats.each do |key, value| next if key.include?('timestamp') -- 2.23.0

1 0

[PATCH compass-ci] service/scheduler: move the service logic to the Sched class
by Wu Zhende 21 Dec '20

21 Dec '20

Keep scheduler.cr neat and do not perform operations related to env. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 40 +++++++++++++++++++++++++++------- src/scheduler/find_job_boot.cr | 2 +- src/scheduler/scheduler.cr | 32 ++++----------------------- 3 files changed, 37 insertions(+), 37 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a1ef28e..f7220f9 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -51,20 +51,44 @@ class Sched mac.gsub(":", "-") end - def set_host_mac(mac : String, hostname : String) - @redis.hash_set("sched/mac2host", normalize_mac(mac), hostname) + def set_host_mac + if (hostname = @env.params.query["hostname"]?) && (mac = @env.params.query["mac"]?) + @redis.hash_set("sched/mac2host", normalize_mac(mac), hostname) + + "Done" + else + "No yet!" + end end - def del_host_mac(mac : String) - @redis.hash_del("sched/mac2host", normalize_mac(mac)) + def del_host_mac + if mac = @env.params.query["mac"]? + @redis.hash_del("sched/mac2host", normalize_mac(mac)) + + "Done" + else + "No yet!" + end end - def set_host2queues(hostname : String, queues : String) - @redis.hash_set("sched/host2queues", hostname, queues) + def set_host2queues + if (queues = @env.params.query["queues"]?) && (hostname = @env.params.query["host"]?) + @redis.hash_set("sched/host2queues", hostname, queues) + + "Done" + else + "No yet!" + end end - def del_host2queues(hostname : String) - @redis.hash_del("sched/host2queues", hostname) + def del_host2queues + if hostname = @env.params.query["host"]? + @redis.hash_del("sched/host2queues", hostname) + + "Done" + else + "No yet!" + end end def update_tbox_wtmp diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 7565375..9b3db88 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -29,7 +29,7 @@ class Sched # then hostname will be "sut-00-01-02-03-04" !!! def submit_host_info_job(mac) host = "sut-#{mac}" - set_host_mac(mac, host) + @redis.hash_set("sched/mac2host", normalize_mac(mac), host) Jobfile::Operate.auto_submit_job( "#{ENV["LKP_SRC"]}/jobs/host-info.yaml", diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index 7ec149c..2103f7c 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -79,46 +79,22 @@ module Scheduler # # curl -X PUT "http://localhost:3000/set_host_mac?hostname=wfg&mac=00-01-02-03-04-05" put "/set_host_mac" do |env| - if (client_hostname = env.params.query["hostname"]?) && (client_mac = env.params.query["mac"]?) - env.sched.set_host_mac(client_mac, client_hostname) - - "Done" - else - "No yet!" - end + env.sched.set_host_mac end # curl -X PUT "http://localhost:3000/set_host2queues?queues=vm-2p8g.aarch64&host=vm-2p8g.a…" put "/set_host2queues" do |env| - if (client_queues = env.params.query["queues"]?) && (client_host = env.params.query["host"]?) - env.sched.set_host2queues(client_host, client_queues) - - "Done" - else - "No yet" - end + env.sched.set_host2queues end # curl -X PUT "http://localhost:3000/del_host_mac?mac=00-01-02-03-04-05" put "/del_host_mac" do |env| - if client_mac = env.params.query["mac"]? - env.sched.del_host_mac(client_mac) - - "Done" - else - "No yet!" - end + env.sched.del_host_mac end # curl -X PUT "http://localhost:3000/del_host2queues?host=vm-2p8g.aarch64" put "/del_host2queues" do |env| - if client_host = env.params.query["host"]? - env.sched.del_host2queues(client_host) - - "Done" - else - "No yet!" - end + env.sched.del_host2queues end # client(runner) report job's status -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: check account info
by Wu Zhende 21 Dec '20

21 Dec '20

If submit yaml in /srv/result, will not check account info. Fix it Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4d0b5df..e6f7e6a 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -51,17 +51,23 @@ class Job def initialize(job_content : JSON::Any, id) @hash = job_content.as_h + @es = Elasticsearch::Client.new + @account_info = Hash(String, JSON::Any).new # init job with "-1", or use the original job_content["id"] id = "-1" if "#{id}" == "" if initialized? - return if @hash["id"] == "#{id}" + if @hash["id"] == "#{id}" + return unless @hash.has_key?("my_uuid") || @hash.has_key?("my_token") + + check_account_info() + return + end end - @es = Elasticsearch::Client.new @hash["id"] = JSON::Any.new("#{id}") - @account_info = Hash(String, JSON::Any).new + check_required_keys() check_account_info() set_defaults() -- 2.23.0

1 0

[PATCH compass-ci 4/4] web_backend.rb: optimize the way of query data for compare
by Li Yuanchao 21 Dec '20

21 Dec '20

Before: When do compare with dimension, first query 100(default size) jobs by filters, then group these jobs by dimension. But if one kind of job is done too many, other kinds of jobs may can not be got. After: Fisrt query how many kinds of dimensions in es, take dimension = 'os' as an example, the result can be ["openeuler", "archlinux", "centos", "debian", "kylin", "archLinux"] Then query 100 jobs by each os, then group all these jobs. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/compare.rb | 2 +- lib/matrix2.rb | 3 +- src/lib/web_backend.rb | 64 ++++++++++++++++++++++++++++-------------- 3 files changed, 45 insertions(+), 24 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index db264f8..da761ef 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -69,7 +69,7 @@ end def create_groups_matrices_list(conditions, dims) es = ESQuery.new(ES_HOST, ES_PORT) query_results = es.multi_field_query(conditions) - combine_group_query_data(query_results, dims) + combine_group_query_data(query_results['hits']['hits'], dims) end # ------------------------------------------------------------------------------------------- diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2507b77..b6cfa79 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -109,9 +109,8 @@ end # group2_key => {...} # ... # } -def combine_group_query_data(query_data, dims) +def combine_group_query_data(job_list, dims) suites_list = [] - job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| if value.empty? diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index c4182a3..a680b8a 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -34,7 +34,7 @@ ALL_FIELDS = FIELDS + NOT_SHOW_FIELDS NOT_NEED_EXIST_FIELDS = %w[error_ids upstream_repo].freeze PREFIX_SEARCH_FIELDS = ['tbox_group'].freeze ES_CLIENT = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") -COMPARE_RECORDS_NUMBER = 50 +COMPARE_RECORDS_NUMBER = 100 def es_query(query) ES_CLIENT.search index: 'jobs*', body: query @@ -148,30 +148,21 @@ def get_es_must(params) must end -def do_get_groups_matrices(query, dimension, total, size, from) - result = es_query(query) - matrices, suites_list = combine_group_query_data(result, dimension) - while matrices.empty? - from += size - break if from > total - - query[:from] = from - result = es_query(query) - matrices, suites_list = combine_group_query_data(result, dimension) - end - [matrices, suites_list] +def get_dimension_list(dimension) + query = { size: 0, aggs: { dims: { terms: { size: 10000, field: dimension[0] } } } } + buckets = es_query(query)['aggregations']['dims']['buckets'] + dimension_list = [] + buckets.each { |dims_agg| dimension_list << dims_agg['key'] } + return dimension_list end -def get_groups_matrices(conditions, dimension, must, size, from) - must += build_multi_field_subquery_body(conditions) - count_query = { query: { bool: { must: must } } } - total = es_count(count_query) - return {} if total < 1 - +def query_dimension(dim_field, dim_value, must, size, from) + must_dim = Array.new(must) + must_dim << { term: { dim_field => dim_value } } query = { query: { bool: { - must: must + must: must_dim } }, size: size, @@ -180,8 +171,39 @@ def get_groups_matrices(conditions, dimension, must, size, from) start_time: { order: 'desc' } }] } + es_query(query)['hits']['hits'] +end + +def get_dimension_job_list(dimension, must, size, from) + dimension_list = get_dimension_list(dimension) + job_list = [] + dimension_list.each do |dim| + job_list += query_dimension(dimension[0], dim, must, size, from) + end + job_list +end + +def do_get_groups_matrices(must, dimension, total, size, from) + job_list = get_dimension_job_list(dimension, must, size, from) + + matrices, suites_list = combine_group_query_data(job_list, dimension) + while matrices.empty? + from += size + break if from > total + + job_list = get_dimension_job_list(dimension, must, size, from) + matrices, suites_list = combine_group_query_data(job_list, dimension) + end + [matrices, suites_list] +end + +def get_groups_matrices(conditions, dimension, must, size, from) + must += build_multi_field_subquery_body(conditions) + count_query = { query: { bool: { must: must } } } + total = es_count(count_query) + return {} if total < 1 - do_get_groups_matrices(query, dimension, total, size, from) + do_get_groups_matrices(must, dimension, total, size, from) end def get_compare_body(params) -- 2.23.0

2 1

[PATCH v2 lkp-tests 2/4] lib/env.sh: move functions to env.sh
by Wei Jihui 21 Dec '20

21 Dec '20

move functions:write_shell_profile and write_host to lib/env.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/env.sh | 30 ++++++++++++++++++++++++++++++ sbin/set-env.sh | 39 --------------------------------------- 2 files changed, 30 insertions(+), 39 deletions(-) delete mode 100755 sbin/set-env.sh diff --git a/lib/env.sh b/lib/env.sh index 8ac42a83..9f37d8cc 100755 --- a/lib/env.sh +++ b/lib/env.sh @@ -125,3 +125,33 @@ shell_profile() [ -f "$profile" ] || touch $profile echo $profile } + +write_shell_profile() +{ + shell_profile_file=$(shell_profile) + if [ $# -gt 0 ]; then + echo "$@" >> $shell_profile_file + fi + + source $shell_profile_file +} + +write_host() +{ + if is_system "Linux"; then + nr_cpu=$(nproc) + memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') + else + nr_cpu=$(sysctl -n hw.logicalcpu) + memory_total=$(top -l 1 | grep MemRegions | awk '{print $2}') + fi + memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') + memory=$(echo $memory_new | awk '{print int($0)+1}')G + + cat > hosts/$(hostname) <<-EOF + nr_cpu: $nr_cpu + memory: $memory + hdd_partitions: + ssd_partitions: + EOF +} diff --git a/sbin/set-env.sh b/sbin/set-env.sh deleted file mode 100755 index cc7f9050..00000000 --- a/sbin/set-env.sh +++ /dev/null @@ -1,39 +0,0 @@ -#!/bin/sh - -. $LKP_SRC/lib/env.sh - -write_shell_profile() -{ - shell_profile_file=$(shell_profile) - if [ $# -gt 0 ]; then - echo $@ >> $shell_profile_file - else - echo "export LKP_SRC=$PWD" >> $shell_profile_file - echo "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" >> $shell_profile_file - fi - - source $shell_profile_file -} - -write_host() -{ - if is_system "Linux"; then - nr_cpu=$(nproc) - memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') - else - nr_cpu=$(sysctl -n hw.logicalcpu) - memory_total=$(top -l 1 | grep MemRegions | awk '{print $2}') - fi - memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') - memory=$(echo $memory_new | awk '{print int($0)+1}')G - - cat > hosts/$(hostname) <<-EOF - nr_cpu: $nr_cpu - memory: $memory - hdd_partitions: - ssd_partitions: - EOF -} - -write_shell_profile -write_host -- 2.23.0

2 1

[PATCH compass-ci] container/delimiter: install util-linux, terminal-table, mail
by Cao Xueliang 21 Dec '20

21 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/delimiter/Dockerfile b/container/delimiter/Dockerfile index 3db5890..93c52c6 100644 --- a/container/delimiter/Dockerfile +++ b/container/delimiter/Dockerfile @@ -6,8 +6,8 @@ FROM alpine:3.11 RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ apk update && \ apk add --no-cache 'ruby-dev' \ - 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' RUN umask 002 && \ echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ - gem install rest-client activesupport git json yaml threadpool elasticsearch faye-websocket + gem install rest-client activesupport git json yaml threadpool elasticsearch faye-websocket terminal-table mail -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] lib/matrix2.rb: discard the no $suite related job
by Lu Kaiyi 21 Dec '20

21 Dec '20

if job has suite, but no $suite related stats metric, then discard this job when create matrix. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2769f7c..ef5d935 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -82,10 +82,15 @@ def create_matrix(job_list) matrix = {} suites = [] job_list.each do |job| - suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats + if job['suite'] + next unless stats.keys.any? { |stat| stat.start_with?(job['suite']) } + + suites << job['suite'] + end + stats.each do |key, value| next if key.include?('timestamp') -- 2.23.0

2 1

[PATCH v3 compass-ci 2/3] lib/compare_matrixes.rb: don't remove the field that start with $suite
by Lu Kaiyi 21 Dec '20

21 Dec '20

if the metric field start with $suite, even if the field is unchanged, still keep it in compare result. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_matrixes.rb | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index b7ea5a4..0727797 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -176,11 +176,15 @@ def get_matrixes_values(matrixes_list, options) matrixes_values end -def remove_unchanged_field(matrixes_values) +def remove_unchanged_field(matrixes_values, suite_list) # remove unchanged field from matrixes values and remove :changed key # matrixes_values.each_key do |success| matrixes_values[success].delete_if do |field| + if suite_list.any? { |suite| field.start_with?(suite) } + matrixes_values[success][field][:changed] = true + end + !matrixes_values[success][field].delete(:changed) end end @@ -203,7 +207,7 @@ def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key options = { 'perf-profile': 5, theme: :none, no_print: false }.merge(options) matrixes_values = get_matrixes_values(matrixes_list, options) - remove_unchanged_field(matrixes_values) if matrixes_list.length > 1 + remove_unchanged_field(matrixes_values, suite_list) if matrixes_list.length > 1 no_print = options[:no_print] result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( -- 2.23.0

2 1

[PATCH v3 compass-ci 3/3] lib: use suites_hash instead of suites_list to keep consistent
by Lu Kaiyi 21 Dec '20

21 Dec '20

[why] suite_list maybe not consistent with group key if sort group matrixes. [how] use suites_hash instead of suites_list to keep correct congruent relationship with same group_key. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 4 ++-- lib/compare_matrixes.rb | 10 +++++----- lib/matrix2.rb | 6 +++--- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index db264f8..b5251fc 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -62,8 +62,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices, suites_list = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, suites_list, options) + groups_matrices, suites_hash = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_hash, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 0727797..6e4c2aa 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -387,25 +387,25 @@ end # input: group matrices # output: pre compare result of each group # the result with more comparison objects first -def compare_group_matrices(group_matrices, suites_list, options) +def compare_group_matrices(group_matrices, suites_hash, options) result_str = '' group_matrices_array = sort_by_matrix_size(group_matrices) group_matrices_array.each do |matrice_kv| - result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_list, options) + result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_hash[matrice_kv[0]], options) end result_str end -def get_matrix_str(matrice_key, matrice_value, suites_list, options) +def get_matrix_str(matrice_key, matrice_value, suite_list, options) m_list = [] m_titles = [] matrice_value.each do |dim, matrix| m_titles << dim m_list << matrix end - return compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) if options[:no_print] + return compare_matrixes(m_list, suite_list, m_titles, matrice_key, options: options) if options[:no_print] - print compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) + print compare_matrixes(m_list, suite_list, m_titles, matrice_key, options: options) return '' end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index ef5d935..891c3b0 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -121,7 +121,7 @@ end # ... # } def combine_group_query_data(query_data, dims) - suites_list = [] + suites_hash = {} job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| @@ -134,10 +134,10 @@ def combine_group_query_data(query_data, dims) groups[group_key][dimension_key], suites = create_matrix(jobs) suite_list.concat(suites) end - suites_list << suite_list + suites_hash[group_key] = suite_list end - return groups, suites_list + return groups, suites_hash end # input: -- 2.23.0

1 0

[PATCH compass-ci 3/4] src/lib/web_backend.rb: add testbox api implementation methods
by Li Yuanchao 21 Dec '20

21 Dec '20

Add implementation methods for web api GET /get_tbox_state?testbox=$testbox Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 07e2e9e..c4182a3 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -461,3 +461,29 @@ def query_testboxes end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def get_tbox_state_body(params) + query = { + "query": { + "match": { + "_id": params[:testbox] + } + } + } + body = ES_CLIENT.search(index: 'testbox', body: query)['hits']['hits'][0]['_source'] + + { + testbox: params[:testbox], + states: body + }.to_json +end + +def get_tbox_state(params) + begin + body = get_tbox_state_body(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get testbox state error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

1 0

[PATCH compass-ci 2/4] container/web-backend: add api for web search testbox history
by Li Yuanchao 21 Dec '20

21 Dec '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 11ba604..ff8fcff 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -110,3 +110,26 @@ end get '/get_testboxes' do query_testboxes end + +# GET /get_tbox_state?testbox=$testbox +# Response: +# - { +# testbox: $testbox, +# states: { +# "state": "$current_state", +# "job_id": "$current_job_id", +# "history": [ +# { +# "time": "$history_time", +# "mac": "$mac", +# "ip": "$ip", +# "job_id": "$history_job_id", +# "state": "$history_state" +# }, +# ... +# ] +# } +# } +get '/get_tbox_state' do + get_tbox_state(params) +end -- 2.23.0

1 0

[PATCH compass-ci 1/4] sched.cr: put testbox state to es
by Li Yuanchao 21 Dec '20

21 Dec '20

History of testbox would be kept, to be searched by web api. [sample] { "_index" : "testbox", "_type" : "_doc", "_id" : "vm-2p16g.liyuanchao-969257", "_score" : 1.0, "_source" : { "state" : "rebooting", "job_id" : "crystal.190886", "history" : [ { "time" : "2020-12-15 19:52:14", "mac" : "0a-f5-c0-1f-f0-dd", "ip" : "172.18.202.96", "job_id" : "crystal.190886", "state" : "running" }, { "time" : "2020-12-15 19:57:22", "mac" : "0a-f5-c0-1f-f0-dd", "ip" : "172.18.202.96", "job_id" : "crystal.190886", "state" : "rebooting" } ] } } Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/sched.cr | 1 + src/scheduler/elasticsearch_client.cr | 21 +++++++++++++++++++++ 2 files changed, 22 insertions(+) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a1ef28e..ed6e538 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -90,6 +90,7 @@ class Sched end @redis.update_wtmp(testbox, hash) + @es.update_tbox(testbox, hash) # json log hash["testbox"] = testbox diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index 7c92204..b17d7f8 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -87,6 +87,27 @@ class Elasticsearch::Client ) end + def update_tbox(testbox : String, wtmp_hash : Hash) + query = {:index => "testbox", :type => "_doc", :id => testbox} + result = @client.get_source(query) if @client.exists(query) + history = result["history"].as_a? if result.is_a?(JSON::Any) + history ||= [] of JSON::Any + history << JSON.parse(wtmp_hash.to_json) + + @client.create( + { + :index => "testbox", + :type => "_doc", + :id => testbox, + :body => { + :state => wtmp_hash["state"], + :job_id => wtmp_hash["job_id"], + :history => history + } + } + ) + end + private def create(job_content : JSON::Any, job_id : String) return @client.create( { -- 2.23.0

1 0

[PATCH v3 lkp-tests] lib/monitor: set monitor_url based on job
by Wu Zhende 21 Dec '20

21 Dec '20

If @job is exist and monitor_url is not configured, set monitor_url based on @job's SCHED_HOST Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index f65cfccf1..283cd788d 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -24,7 +24,6 @@ class Monitor # 1 means timeout exit @exit_status_code = 0 @defaults = {} - load_default @result = [] @stop_query = {} @reason = nil @@ -33,6 +32,10 @@ class Monitor def load_default return unless @monitor_url == '' + if host = @job["SCHED_HOST"] + return @monitor_url = "ws://#{host}:11310/filter" + end + Dir.glob(['/etc/compass-ci/monitor/*.yaml', "#{ENV['HOME']}/.config/compass-ci/monitor/*.yaml"]).each do |file| next unless File.exist? file @@ -98,6 +101,7 @@ class Monitor def run(timeout = nil) merge_overrides + load_default field_check @query.each do |k, v| -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] lib/matrix2.rb: remove the *timestamp* from matrix
by Lu Weitao 21 Dec '20

21 Dec '20

[Why] In compare results: the value of *timestamp*(metric) doesn't mean the number of occurrences, it may confuse user like bellow: openeuler centos archLinux metric -------------------- ------------------------------ ------------------------------ ------------------------------ fails:runs change fails:runs change fails:runs | | | | | 3585:13 +6913.7% 344:1 -2.6e+4% 11:1 dmesg.timestamp:last 2350:13 +1.6e+4% 344:1 -1.7e+4% 10:1 kmsg.timestamp:last 2:13 -15.4% 0:1 +84.6% 1:1 last_state.exit_fail [How] remove the metric that include "timestamp" from matrix Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index c712e32..2769f7c 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -87,6 +87,8 @@ def create_matrix(job_list) next unless stats stats.each do |key, value| + next if key.include?('timestamp') + matrix[key] = [] unless matrix[key] matrix[key] << value end -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] lib/matrix2.rb: fix rubocop warning
by Lu Weitao 21 Dec '20

21 Dec '20

[Why] lib/matrix2.rb:75:1: C: Metrics/MethodLength: Method has too many lines. [16/15] def create_matrix(job_list) ... Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2507b77..c712e32 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -64,6 +64,12 @@ def samples_fill_missing_zeros(value, size) value.concat([0] * (size - value.size)) end +def matrix_fill_miss_zeros(matrix, col_size) + matrix.each_value do |value| + samples_fill_missing_zeros(value, col_size) + end +end + # input: job_list # return: matrix of Hash(String, Array(Number)) # Eg: matrix: { @@ -86,9 +92,7 @@ def create_matrix(job_list) end end col_size = job_list.size - matrix.each_value do |value| - samples_fill_missing_zeros(value, col_size) - end + matrix_fill_miss_zeros(matrix, col_size) return matrix, suites end -- 2.23.0

1 0

[PATCH v2 lkp-tests 4/4] install.sh: grant execute permission
by Wei Jihui 21 Dec '20

21 Dec '20

cd lkp-test ./install.sh && source ~/.${SHELL##*/}rc Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- install.sh | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 install.sh diff --git a/install.sh b/install.sh old mode 100644 new mode 100755 -- 2.23.0

1 0

[PATCH v2 lkp-tests 3/4] install.sh: refactor install flow
by Wei Jihui 21 Dec '20

21 Dec '20

1. when exec install.sh, no $LKP_SRC 2. first enable $LKP_SRC env 3. mv sbin/install-dependencies.sh to install.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- Makefile | 2 -- install.sh | 10 +++++++--- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/Makefile b/Makefile index 0b2d09a2..ae062973 100644 --- a/Makefile +++ b/Makefile @@ -14,8 +14,6 @@ subsystem: install: mkdir -p $(TARGET_DIR_BIN) ln -sf $(shell pwd)/bin/lkp $(TARGET_DIR_BIN)/lkp - bash sbin/set-env.sh - bash sbin/install-dependencies.sh .PHONY: doc doc: diff --git a/install.sh b/install.sh index cea1654f..2bd17d54 100644 --- a/install.sh +++ b/install.sh @@ -1,7 +1,11 @@ #!/bin/bash -. $LKP_SRC/lib/env.sh - -make +. lib/env.sh +write_host +write_shell_profile "export LKP_SRC=$PWD" +write_shell_profile "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" source $(shell_profile) + +. sbin/install-dependencies.sh +make install -- 2.23.0

1 0

[PATCH compass-ci] doc: update READEME.md
by Liu Yinsi 21 Dec '20

21 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 111 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 86 insertions(+), 25 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index ccb1d15..7c20a53 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -1,26 +1,87 @@ -Deploy a mini instance in one single machine. - -Prepare: - 1. hardware - Server: at least prepare a server - ProductType: ThaiShan200-2280 - Arch: aarch64 - Memory: 8G - CPU: 64 nuclear - DiskSpace: 500G - - 2. software - OS: openEuler-aarch64-20.03 LTS - git: suggest 2.23.0 - - 3. network - Internet is available - - 4. /os for store rootfs - >= 300G - -Steps: - umask 002 +### 本地搭建 compass-ci 服务器节点 + +在 openEuler 系统一键部署 compass-ci 环境，当前已支持 openEuler-aarch64-20.03-LTS 系统环境，以下配置仅供参考。 + +#### 准备工作 +- 硬件 + 服务器类型：ThaiShan200-2280 (建议) + 架构：aarch64 + 内存：>= 32GB + CPU：64 nuclear (建议) + 硬盘：>= 500G (建议划分独立分区) + + >**说明：** + >划分较小独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… + >划分较大独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… + +- 软件 + OS：openEuler-aarch64-20.03 LTS + git：2.23.0版本 (建议) + 预留空间：>= 300G + 网络：可以访问互联网 + + >**说明：** + >openEuler 系统安装详细操作请参考[添加测试用例](https://openeuler.org/zh/docs/20.03_LTS/docs/Installati… + +#### 操作指导 +1. 登录 openEuler 系统 + +2. 创建工作目录并设置文件权限 + ```bash + mkdir demo && cd demo && umask 002 + ``` +3. 克隆 compass-ci 项目代码到 demo 目录 + ```bash git clone https://gitee.com/wu_fengguang/compass-ci.git - cd compass-ci/sparrow - ./install-tiny + ``` +4. 执行一键部署脚本 install-tiny + ```bash + cd compass-ci/sparrow && ./install-tiny + ``` +5. 生成lkp-aarch64.cgz压缩包 + ```bash + cd /c/compass-ci/container/lkp-initrd && ./run + ``` +6. 验证账号 + ```bash + cd /c/compass-ci/sbin && ./build-my-info.rb + 按照提示输入账号名和邮箱即可 + +7. 测试环境是否可以提交job测试 + ```bash + submit iperf.yaml testbox=vm-2p8g + ``` + + 执行上述命令正常情况下会提示信息如下： + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + compass-ci搭建完毕，下面就可以开始进行测试了。 + +8. 下载rootfs（以下载openeuler/aarch64/20.03为例，需要在哪个系统上测试就去对应的/srv/os/目录下使用wget命令下载cgz文件包） + ```bash + mkdir -p /srv/os/openeuler/aarch64/ + cd /srv/os/openeuler/aarch64 + wget http://124.90.34.227:11300/os/test/openeuler/aarch64/20.03.cgz + ``` + +9. 解压cgz包 + ```bash + gzip -dc 20.03.cgz | cpio -idv + ``` + +8. [使用 compass-ci 平台测试开源项目](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… + +9. [编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testca… + +10. 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… + 注意：os_mount 必须指定为cifs + +11. 运行测试任务 + ```bash + cd /c/compass-ci/providers/ && ./my-qemu.sh + ``` + +12. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… id查看output文件 + ```bash + cd /srv/result/iperf/$job_id/output + ``` -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] lib/compare_matrixes.rb: don't remove the field that start with $suite
by Lu Kaiyi 21 Dec '20

21 Dec '20

if the metric field start with $suite, even if the field is unchanged, still keep it in compare result. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_matrixes.rb | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index b7ea5a4..0727797 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -176,11 +176,15 @@ def get_matrixes_values(matrixes_list, options) matrixes_values end -def remove_unchanged_field(matrixes_values) +def remove_unchanged_field(matrixes_values, suite_list) # remove unchanged field from matrixes values and remove :changed key # matrixes_values.each_key do |success| matrixes_values[success].delete_if do |field| + if suite_list.any? { |suite| field.start_with?(suite) } + matrixes_values[success][field][:changed] = true + end + !matrixes_values[success][field].delete(:changed) end end @@ -203,7 +207,7 @@ def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key options = { 'perf-profile': 5, theme: :none, no_print: false }.merge(options) matrixes_values = get_matrixes_values(matrixes_list, options) - remove_unchanged_field(matrixes_values) if matrixes_list.length > 1 + remove_unchanged_field(matrixes_values, suite_list) if matrixes_list.length > 1 no_print = options[:no_print] result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/matrix2.rb: discard the no $suite related job
by Lu Kaiyi 21 Dec '20

21 Dec '20

if job has suite, but no $suite related stats metric, then discard this job when create matrix. --- lib/matrix2.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2507b77..f1b2c78 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,10 +76,15 @@ def create_matrix(job_list) matrix = {} suites = [] job_list.each do |job| - suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats + if job['suite'] + next unless stats.keys.any? { |stat| stat.start_with?(job['suite']) } + + suites << job['suite'] + end + stats.each do |key, value| matrix[key] = [] unless matrix[key] matrix[key] << value -- 2.23.0

1 0

[PATCH v2 lkp-tests] lib/monitor: get host from job
by Wu Zhende 21 Dec '20

21 Dec '20

If @job is exist and monitor_url is not configured, set monitor_url based on SCHED_HOST Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index f65cfccf1..283cd788d 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -24,7 +24,6 @@ class Monitor # 1 means timeout exit @exit_status_code = 0 @defaults = {} - load_default @result = [] @stop_query = {} @reason = nil @@ -33,6 +32,10 @@ class Monitor def load_default return unless @monitor_url == '' + if host = @job["SCHED_HOST"] + return @monitor_url = "ws://#{host}:11310/filter" + end + Dir.glob(['/etc/compass-ci/monitor/*.yaml', "#{ENV['HOME']}/.config/compass-ci/monitor/*.yaml"]).each do |file| next unless File.exist? file @@ -98,6 +101,7 @@ class Monitor def run(timeout = nil) merge_overrides + load_default field_check @query.each do |k, v| -- 2.23.0

2 1

Re: [PATCH v2 compass-ci] lib/matrix2.rb remove the *timestamp* from matrix
by Ren Wen 21 Dec '20

21 Dec '20

Why remove ':' after 'lib/matrix2.rb'? Thanks, RenWen Reply-To: In-Reply-To: <20201221022051.3690974-1-luweitaobe(a)163.com> On Mon, Dec 21, 2020 at 10:20:51AM +0800, Lu Weitao wrote: >[Why] >In compare results: >the value of *timestamp*(metric) doesn't mean the number of occurrences, >it may confuse user like bellow: > > openeuler centos archLinux metric >-------------------- ------------------------------ ------------------------------ ------------------------------ > fails:runs change fails:runs change fails:runs > | | | | | > 3585:13 +6913.7% 344:1 -2.6e+4% 11:1 dmesg.timestamp:last > 2350:13 +1.6e+4% 344:1 -1.7e+4% 10:1 kmsg.timestamp:last > 2:13 -15.4% 0:1 +84.6% 1:1 last_state.exit_fail > >[How] >remove the metric that include "timestamp" from matrix > >Signed-off-by: Lu Weitao <luweitaobe(a)163.com> >--- > lib/matrix2.rb | 2 ++ > 1 file changed, 2 insertions(+) > >diff --git a/lib/matrix2.rb b/lib/matrix2.rb >index c712e32..2769f7c 100644 >--- a/lib/matrix2.rb >+++ b/lib/matrix2.rb >@@ -87,6 +87,8 @@ def create_matrix(job_list) > next unless stats > > stats.each do |key, value| >+ next if key.include?('timestamp') >+ > matrix[key] = [] unless matrix[key] > matrix[key] << value > end >-- >2.23.0 >

2 1

[PATCH lkp-tests 2/4] lib/env.sh: move functions to env.sh
by Wei Jihui 21 Dec '20

21 Dec '20

move functions:write_shell_profile and write_host to lib/env.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/env.sh | 30 ++++++++++++++++++++++++++++++ sbin/set-env.sh | 39 --------------------------------------- 2 files changed, 30 insertions(+), 39 deletions(-) delete mode 100755 sbin/set-env.sh diff --git a/lib/env.sh b/lib/env.sh index 8ac42a83..d360f163 100755 --- a/lib/env.sh +++ b/lib/env.sh @@ -125,3 +125,33 @@ shell_profile() [ -f "$profile" ] || touch $profile echo $profile } + +write_shell_profile() +{ + shell_profile_file=$(shell_profile) + if [ $# -gt 0 ]; then + echo $@ >> $shell_profile_file + fi + + source $shell_profile_file +} + +write_host() +{ + if is_system "Linux"; then + nr_cpu=$(nproc) + memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') + else + nr_cpu=$(sysctl -n hw.logicalcpu) + memory_total=$(top -l 1 | grep MemRegions | awk '{print $2}') + fi + memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') + memory=$(echo $memory_new | awk '{print int($0)+1}')G + + cat > hosts/$(hostname) <<-EOF + nr_cpu: $nr_cpu + memory: $memory + hdd_partitions: + ssd_partitions: + EOF +} diff --git a/sbin/set-env.sh b/sbin/set-env.sh deleted file mode 100755 index cc7f9050..00000000 --- a/sbin/set-env.sh +++ /dev/null @@ -1,39 +0,0 @@ -#!/bin/sh - -. $LKP_SRC/lib/env.sh - -write_shell_profile() -{ - shell_profile_file=$(shell_profile) - if [ $# -gt 0 ]; then - echo $@ >> $shell_profile_file - else - echo "export LKP_SRC=$PWD" >> $shell_profile_file - echo "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" >> $shell_profile_file - fi - - source $shell_profile_file -} - -write_host() -{ - if is_system "Linux"; then - nr_cpu=$(nproc) - memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') - else - nr_cpu=$(sysctl -n hw.logicalcpu) - memory_total=$(top -l 1 | grep MemRegions | awk '{print $2}') - fi - memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') - memory=$(echo $memory_new | awk '{print int($0)+1}')G - - cat > hosts/$(hostname) <<-EOF - nr_cpu: $nr_cpu - memory: $memory - hdd_partitions: - ssd_partitions: - EOF -} - -write_shell_profile -write_host -- 2.23.0

2 1

Re: [PATCH compass-ci 2/2] lib/matrix2.rb remove the *timestamp* form matrix
by Ren Wen 21 Dec '20

21 Dec '20

spell: form -> from Thanks, RenWen Reply-To: In-Reply-To: <20201219133430.1143483-2-luweitaobe(a)163.com> On Sat, Dec 19, 2020 at 09:34:30PM +0800, Lu Weitao wrote: >[Why] >In compare results: >the value of *timestamp*(metric) doesn't mean the number of occurrences, >it may confuse user like bellow: > > openeuler centos archLinux metric >-------------------- ------------------------------ ------------------------------ ------------------------------ > fails:runs change fails:runs change fails:runs > | | | | | > 3585:13 +6913.7% 344:1 -2.6e+4% 11:1 dmesg.timestamp:last > 2350:13 +1.6e+4% 344:1 -1.7e+4% 10:1 kmsg.timestamp:last > 2:13 -15.4% 0:1 +84.6% 1:1 last_state.exit_fail > >[How] >remove the metric that include "timestamp" from matrix > >Signed-off-by: Lu Weitao <luweitaobe(a)163.com> >--- > lib/matrix2.rb | 2 ++ > 1 file changed, 2 insertions(+) > >diff --git a/lib/matrix2.rb b/lib/matrix2.rb >index c712e32..2769f7c 100644 >--- a/lib/matrix2.rb >+++ b/lib/matrix2.rb >@@ -87,6 +87,8 @@ def create_matrix(job_list) > next unless stats > > stats.each do |key, value| >+ next if key.include?('timestamp') >+ > matrix[key] = [] unless matrix[key] > matrix[key] << value > end >-- >2.23.0 >

2 1

[PATCH v2 compass-ci] providers/docker: set the private key of the testbox
by Xiao Shenwei 21 Dec '20

21 Dec '20

[why] login the docker testbox without password [how] Add ssh private key for docker, for automatically establish the ssh reverse tunnel between sut and port map container. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/bin/entrypoint.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/providers/docker/bin/entrypoint.sh b/providers/docker/bin/entrypoint.sh index 63cb605..feb5edf 100755 --- a/providers/docker/bin/entrypoint.sh +++ b/providers/docker/bin/entrypoint.sh @@ -2,4 +2,6 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +mkdir -p /root/.ssh +cp /lkp/lkp/src/rootfs/addon/root/.ssh/id_rsa /root/.ssh/ sh /lkp/lkp/src/rootfs/addon/etc/init.d/lkp-bootstrap -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/matrix2.rb: fix rubocop warning
by Lu Weitao 21 Dec '20

21 Dec '20

[Why] lib/matrix2.rb:75:1: C: Metrics/MethodLength: Method has too many lines. [16/15] def create_matrix(job_list) ... Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2507b77..c712e32 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -64,6 +64,12 @@ def samples_fill_missing_zeros(value, size) value.concat([0] * (size - value.size)) end +def matrix_fill_miss_zeros(matrix, col_size) + matrix.each_value do |value| + samples_fill_missing_zeros(value, col_size) + end +end + # input: job_list # return: matrix of Hash(String, Array(Number)) # Eg: matrix: { @@ -86,9 +92,7 @@ def create_matrix(job_list) end end col_size = job_list.size - matrix.each_value do |value| - samples_fill_missing_zeros(value, col_size) - end + matrix_fill_miss_zeros(matrix, col_size) return matrix, suites end -- 2.23.0

2 1

Re: [PATCH compass-ci 2/3] assign-account-email: remove .bash_profile usage
by Luan Shengde 21 Dec '20

21 Dec '20

On Sat, Dec 19, 2020 at 11:23:43AM +0800, Li Ping wrote: >On Sat, Dec 19, 2020 at 11:09:54AM +0800, Wu Fengguang wrote: >>Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> >>--- >> container/mail-robot/lib/assign-account-email.rb | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >>diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb >>index 77ed4e8..ee3b574 100755 >>--- a/container/mail-robot/lib/assign-account-email.rb >>+++ b/container/mail-robot/lib/assign-account-email.rb >>@@ -37,7 +37,7 @@ def build_apply_account_email(my_info) >> git clone https://gitee.com/wu_fengguang/lkp-tests.git >> cd lkp-tests >> make install >>- source ${HOME}/.bashrc && source ${HOME}/.bash_profile >>+ source ${HOME}/.${SHELL##*/}rc > > use ${HOME} will become root in email Use \\${HOME} will resolve this problem. Thanks, Luan Shengde > >> >> 3) submit job >> reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md >>-- >>2.23.0 >>

1 0

[PATCH lkp-tests] tests/build-pkg: add config file support
by Wang Yong 21 Dec '20

21 Dec '20

[Why] config file in some software need storage and recall now we can add config value in yaml and cp ${config} to ${cci_dir}/${benchmark}/ in PKGBUILD so that config file can be uploaded correctly $ cat test.yaml ... config: ${config} ... Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 39 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index addff9f7b..0a143813f 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -5,6 +5,7 @@ # - git_tag # - pkg_ver # upstream_url +# - config # We'll use the below global env vars, but not treat them as parameter to avoid # duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ @@ -20,6 +21,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/distro/common +. $LKP_SRC/lib/http.sh check_vars() { @@ -31,6 +33,29 @@ check_vars() [ -n "$os_mount" ] || die "os_mount is empty" } +check_config_var() +{ + [ -n "$config" ] || return +} + +get_config() +{ + local cci_http_host=${INITRD_HTTP_HOST} + local cci_http_port=${INITRD_HTTP_PORT:-8800} + export cci_dir=/cci/config + local remote_dir="http://${cci_http_host}:${cci_http_port}${cci_dir}/${benchmark}" + + mkdir -p ${cci_dir} + + check_config_var + http_get_file "${remote_dir}/${config}" "${cci_dir}/${benchmark}/${config}" +} + +upload_config() +{ + upload_to_target_dir "${cci_dir}/${benchmark}" +} + mount_dest() { # the same image is mounted to cifs and nfs, the generated cgz files @@ -71,6 +96,13 @@ create_cgz_name() [ -n "$git_tag" ] && { cgz_name="${cgz_path}/${git_tag}.cgz" + # download PKGBUILD-$tag + benchmark="${upstream_repo##*/}" + export benchmark="${benchmark%.git}" + [ -n "$PKGBUILD_TAG" ] && { + get_pkgfile "pkg/$benchmark" "PKGBUILD-$PKGBUILD_TAG" + [ -f "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG not exist" + [ -s "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG is empty" } [ -n "$pkg_ver" ] && { @@ -131,7 +163,11 @@ build_source_pkg() PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf - [ "$?" == 0 ] || exit 1 + local build_state="$?" + + upload_config + + [ "${build_state}" == "0" ] || exit 1 create_softlink upload_to_target_dir ${PKG_MNT}/${pack_to}/${pkgname} } @@ -139,4 +175,5 @@ build_source_pkg() check_vars mount_dest request_pkg +get_config build_source_pkg -- 2.23.0

2 2

[PATCH compass-ci] lib/matrix2.rb: discard the no $suite related job
by Lu Kaiyi 19 Dec '20

19 Dec '20

if job has suite, but no $suite related stats metric, then discard this job when create matrix. --- lib/matrix2.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2507b77..f1b2c78 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,10 +76,15 @@ def create_matrix(job_list) matrix = {} suites = [] job_list.each do |job| - suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats + if job['suite'] + next unless stats.keys.any? { |stat| stat.start_with?(job['suite']) } + + suites << job['suite'] + end + stats.each do |key, value| matrix[key] = [] unless matrix[key] matrix[key] << value -- 2.23.0

2 2

[PATCH compass-ci 2/2] lib/matrix2.rb remove the *timestamp* form matrix
by Lu Weitao 19 Dec '20

19 Dec '20

[Why] In compare results: the value of *timestamp*(metric) doesn't mean the number of occurrences, it may confuse user like bellow: openeuler centos archLinux metric -------------------- ------------------------------ ------------------------------ ------------------------------ fails:runs change fails:runs change fails:runs | | | | | 3585:13 +6913.7% 344:1 -2.6e+4% 11:1 dmesg.timestamp:last 2350:13 +1.6e+4% 344:1 -1.7e+4% 10:1 kmsg.timestamp:last 2:13 -15.4% 0:1 +84.6% 1:1 last_state.exit_fail [How] remove the metric that include "timestamp" from matrix Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index c712e32..2769f7c 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -87,6 +87,8 @@ def create_matrix(job_list) next unless stats stats.each do |key, value| + next if key.include?('timestamp') + matrix[key] = [] unless matrix[key] matrix[key] << value end -- 2.23.0

1 0

[PATCH compass-ci] lib/commit_status.rb: query latest job_id by commit_id
by Lu Kaiyi 19 Dec '20

19 Dec '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/commit_status.rb | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/lib/commit_status.rb b/lib/commit_status.rb index 81aeb29..f1cbfbc 100644 --- a/lib/commit_status.rb +++ b/lib/commit_status.rb @@ -25,6 +25,17 @@ def query_latest_good_commit(job, error_id) return latest_good_commit end +def query_latest_job_id(commit_id) + items = { + 'upstream_commit' => commit_id + } + jobs_list = query_jobs_from_es(items) + return nil if jobs_list.empty? + + jobs_list.sort_by! { |job| job['end_time'] } + jobs_list.last['id'] +end + def query_jobs_from_es(items) es = ESQuery.new(ES_HOST, ES_PORT) result = es.multi_field_query items -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: check load host config from job
by Wei Jihui 19 Dec '20

19 Dec '20

@defaults will merge to @job, and set @default empty. if load host config from @job, so do not need to load host in advance. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 42b008c7..2ed06edc 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -317,17 +317,6 @@ class Job spec_file_context.each_key { |k| @job.delete k } end - def check_load_hosts_config(defaults_config) - return if @load_hosts_done - - if defaults_config.include?('testbox') - @job['tbox_group'] = tbox_group(defaults_config['testbox']) - elsif defaults_config.include?('tbox_group') - @job['tbox_group'] = defaults_config['tbox_group'] - end - return @job.include?('tbox_group') - end - def load_hosts_config return if @job.include?(:no_defaults) return if @load_hosts_done @@ -354,7 +343,7 @@ class Job def get_lab_hosts_file lab_hosts_file_name = @job['testbox'] - lab_hosts_file = "#{ENV['CCI_REPOS']}/lab-#{@defaults['lab']}/hosts/#{lab_hosts_file_name}" + lab_hosts_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{lab_hosts_file_name}" if File.file?(lab_hosts_file) lab_hosts_file @@ -409,7 +398,6 @@ class Job @defaults[source_file_symkey(file)] = nil revise_hash(@defaults, defaults, true) @defaults.merge!(@overrides) - load_hosts_config if check_load_hosts_config(@defaults) end @file_loaded[file] = true true -- 2.23.0

1 0

[PATCH compass-ci] scheduler-https/start: fix missing env key
by Wu Zhende 19 Dec '20

19 Dec '20

Some key don't define in scheduler-https. Synchronize with scheduler. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler-https/start | 34 +++++++++++++++++++++++---------- 1 file changed, 24 insertions(+), 10 deletions(-) diff --git a/container/scheduler-https/start b/container/scheduler-https/start index 9898c1a..98940d7 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -28,17 +28,32 @@ names = Set.new %w[ INITRD_HTTP_PORT TASKQUEUE_PORT TASKQUEUE_HOST + GIT_SERVER + MASTER_FLUENTD_HOST + MASTER_FLUENTD_PORT + SSHR_PORT + SSHR_PORT_BASE + SSHR_PORT_LEN + lab ] defaults = relevant_defaults(names) +defaults['SSHR_PORT'] ||= 5050 +defaults['SSHR_PORT_BASE'] ||= 50000 +defaults['SSHR_PORT_LEN'] ||= 2000 +defaults['SCHED_PORT'] ||= '3000' +defaults['SCHED_HOST'] ||= '172.17.0.1' +defaults['MASTER_FLUENTD_PORT'] ||= '24224' +defaults['SCHED_HTTPS_PORT'] ||= 11306 +defaults['SCHED_HTTPS_HOST'] ||= 'api.compass-ci.openeuler.org' env = docker_env(defaults) -DEFAULT_LKP = '/c/lkp-tests' -DEFAULT_CCI = '/c/compass-ci' -SCHED_PORT = defaults['SCHED_PORT'] || '3000' -SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' -SCHED_HTTPS_PORT = defaults['SCHED_HTTPS_PORT'] || 11306 -SCHED_HTTPS_HOST = defaults['SCHED_HTTPS_HOST'] || 'api.compass-ci.openeuler.org' +CCI_REPOS = ENV['CCI_REPOS'] || '/c' +DEFAULT_LKP = "#{CCI_REPOS}/lkp-tests" +DEFAULT_CCI = "#{CCI_REPOS}/compass-ci" +LAB = defaults['lab'] || 'z9' +SCHED_HTTPS_PORT = defaults['SCHED_HTTPS_PORT'] +SCHED_HTTPS_HOST = defaults['SCHED_HTTPS_HOST'] docker_rm "s001-alpine-#{SCHED_HTTPS_PORT}" @@ -48,14 +63,13 @@ cmd = %W[ --name s001-alpine-#{SCHED_HTTPS_PORT} -d -u 1090:1090 - -p #{SCHED_HTTPS_PORT}:#{SCHED_PORT} + -p #{SCHED_HTTPS_PORT}:#{defaults['SCHED_PORT']} ] + env + %W[ -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} - -e SCHED_HOST=#{SCHED_HOST} - -e SCHED_PORT=#{SCHED_PORT} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} -v /srv/cci/scheduler/alpine:/srv/cci/scheduler -v /etc/localtime:/etc/localtime:ro -v /srv/result:/srv/result @@ -65,7 +79,7 @@ cmd = %W[ --log-opt mode=non-blocking --log-opt max-buffer-size=4m --log-opt tag=scheduler-#{SCHED_HTTPS_PORT} - sch-ruby-a:v0.00d-#{SCHED_PORT} + sch-ruby-a:v0.00d-#{defaults['SCHED_PORT']} ] cmd += ['sh', '-c', 'umask 002 && ./scheduler --ssl --ssl-key-file /ca/server.key --ssl-cert-file /ca/server.crt'] -- 2.23.0

1 0

[PATCH lkp-tests 4/4] install.sh: grant execute permission
by Wei Jihui 19 Dec '20

19 Dec '20

cd lkp-test ./install.sh && source ~/.${SHELL##*/}rc Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- install.sh | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 install.sh diff --git a/install.sh b/install.sh old mode 100644 new mode 100755 -- 2.23.0

1 0

[PATCH lkp-tests 3/4] install.sh: refactor install flow
by Wei Jihui 19 Dec '20

19 Dec '20

1. when exec install.sh, no $LKP_SRC 2. first enable $LKP_SRC env 3. mv sbin/install-dependencies.sh to install.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- Makefile | 2 -- install.sh | 10 +++++++--- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/Makefile b/Makefile index 0b2d09a2..ae062973 100644 --- a/Makefile +++ b/Makefile @@ -14,8 +14,6 @@ subsystem: install: mkdir -p $(TARGET_DIR_BIN) ln -sf $(shell pwd)/bin/lkp $(TARGET_DIR_BIN)/lkp - bash sbin/set-env.sh - bash sbin/install-dependencies.sh .PHONY: doc doc: diff --git a/install.sh b/install.sh index cea1654f..2bd17d54 100644 --- a/install.sh +++ b/install.sh @@ -1,7 +1,11 @@ #!/bin/bash -. $LKP_SRC/lib/env.sh - -make +. lib/env.sh +write_host +write_shell_profile "export LKP_SRC=$PWD" +write_shell_profile "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" source $(shell_profile) + +. sbin/install-dependencies.sh +make install -- 2.23.0

1 0

[PATCH lkp-tests 1/4] bin/install-dependencies.sh: refactor file
by Wei Jihui 19 Dec '20

19 Dec '20

all dependent functions is move to lib/env.sh, sbin/set-env.sh in not required. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/install-dependencies.sh | 40 +++++++++++++++++------------------- 1 file changed, 19 insertions(+), 21 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 7605a29f..b4d3ad22 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -1,30 +1,28 @@ #!/bin/bash . $LKP_SRC/lib/env.sh -. $LKP_SRC/sbin/set-env.sh -# choose install function base on -# DISTRIBUTION +# choose install function base on DISTRIBUTION linux_dep() { -source /etc/os-release -case $ID in -ubuntu|debian) - export DEBIAN_FRONTEND=noninteractive - apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client - ;; -openEuler|fedora|rhel|centos) - if type dnf > /dev/null 2>&1; then - dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - else - yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - fi - sudo gem install -f git activesupport rest-client - ;; -*) - echo "$ID not support! please install dependencies manually." - ;; -esac + source /etc/os-release + case $ID in + ubuntu|debian) + export DEBIAN_FRONTEND=noninteractive + apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client + ;; + openEuler|fedora|rhel|centos) + if type dnf > /dev/null 2>&1; then + dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + else + yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + fi + sudo gem install -f git activesupport rest-client + ;; + *) + echo "$ID not support! please install dependencies manually." + ;; + esac } -- 2.23.0

1 0

[PATCH compass-ci] result-webdav/nginx.conf: create soft connection after package upload
by Cui Yili 19 Dec '20

19 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 8b54314..f077fca 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -37,6 +37,25 @@ http { end) return resultstrlist end + function symbolic_link(path, upload_file) + level_dir = split(path, "/") + if level_dir[1] ~= "initrd" or upload_file == nil then + return + end + if level_dir[2] == "pkg" or level_dir[2] == "build-pkg" then + link_name = "latest" + else + tmp_name = string.match(upload_file, "(.*)_%d+%.cgz") + if not tmp_name then + link_name = tmp_name + else + return + end + end + link_name = link_name..".cgz" + root_path = "/srv/"..path + io.popen("cd "..root_path.."&& ln -sf "..upload_file.." "..link_name) + end redis_host = os.getenv("REDIS_HOST") redis_port = os.getenv("REDIS_PORT") @@ -65,11 +84,13 @@ http { dirs = split(res, ",") for i = 1, #dirs do if string.find(path, dirs[i], 1, true) then + symbolic_link(path, upload_file) return end end else if string.find(path, res, 1, true) then + symbolic_link(path, upload_file) return end end -- 2.23.0

1 0

[PATCH compass-ci] lib/error_messages.rb: fix error when file is not existed
by Lin Jiaxin 19 Dec '20

19 Dec '20

Traceback (most recent call last): 4: from compare_error_messages.rb:63:in `<main>' 3: from compare_error_messages.rb:20:in `get_compare_result' 2: from /home/linjiaxin/compass-ci/lib/error_messages.rb:33:in `obtain_error_messages' 1: from /home/linjiaxin/compass-ci/lib/error_messages.rb:33:in `open' /home/linjiaxin/compass-ci/lib/error_messages.rb:33:in `initialize': No such file or directory @ rb_sysopen - /srv/result/build-pkg/2020-12-16/vm-2p32g.linjiaxin/openeuler-20.03-aarch64/archlinux-community--fcgi-trunk-HEAD/crystal.193087/build-pkg (Errno::ENOENT) Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/lib/error_messages.rb b/lib/error_messages.rb index 5bbc329..61f8070 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -30,12 +30,14 @@ class ErrorMessages end def obtain_error_messages - File.open((a)file).each_line do |line| - next if extract_error_message(line) - next unless @in_stderr - next unless @error_message['error_line'] =~ /error|warning/i + if FileTest.exist?(@file) + File.open((a)file).each_line do |line| + next if extract_error_message(line) + next unless @in_stderr + next unless @error_message['error_line'] =~ /error|warning/i - @error_messages[@error_message['error_line']] << @error_message['error_line'] + @error_message['error_details'] + @error_messages[@error_message['error_line']] << @error_message['error_line'] + @error_message['error_details'] + end end return @error_messages end -- 2.23.0

1 0

[PATCH v2 compass-ci] lib: add compare_error_messages.rb
by Lin Jiaxin 19 Dec '20

19 Dec '20

Provide formatted error messages and new error number for git bisect. formatted error messages example: previous error: fcgiapp.c:1717:22: warning: comparison of integer expressions of different signedness: ‘int’ and ‘long unsigned int’ [-Wsign-compare] 1717 | if(headerLen < sizeof(header)) { | ^ new error: >> libtool: warning: relinking 'libfcgi++.la' Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 59 +++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 lib/compare_error_messages.rb diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb new file mode 100644 index 0000000..0b51fc3 --- /dev/null +++ b/lib/compare_error_messages.rb @@ -0,0 +1,59 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'terminal-table' +require_relative 'es_query' +require_relative 'error_messages' +require_relative "#{ENV['LKP_SRC']}/lib/common" + +def get_compare_result(previous_job_id, later_job_id) + es = ESQuery.new + + previous_error_ids = es.query_by_id(previous_job_id)['error_ids'] + later_es_result = es.query_by_id(later_job_id) + later_error_ids = later_es_result['error_ids'] + later_result_file = File.join('/srv', later_es_result['result_root'], 'build-pkg') + + new_error_ids = get_new_error_ids(later_error_ids, previous_error_ids) + error_messages = ErrorMessages.new(later_result_file).obtain_error_messages + new_error_number, formatted_error_messages = format_error_messages(error_messages, new_error_ids) + + # new error use ">>" as an identifier in the error header + return new_error_number, formatted_error_messages +end + +def get_new_error_ids(later_error_ids, previous_error_ids) + return nil_to_empty_array(later_error_ids) - nil_to_empty_array(previous_error_ids) +end + +def nil_to_empty_array(array) + array = [] if array.nil? + return array +end + +def format_error_messages(error_messages, new_error_ids) + formatted_error_messages = Terminal::Table.new + formatted_error_messages.style = { border_x: '', border_y: '', border_i: '', padding_left: 0 } + new_error_number = 0 + + error_messages.each do |k, v| + if new_error_ids.include?('build-pkg.' + build_pkg_error_id(k)) + new_error_number += 1 + formatted_error_messages = add_sign(formatted_error_messages, '>>', v) + else + formatted_error_messages = add_sign(formatted_error_messages, ' ', v) + end + end + + return new_error_number, formatted_error_messages.to_s +end + +def add_sign(formatted_error_messages, sign, set) + set.each do |value| + formatted_error_messages.add_row([sign, value]) + end + + return formatted_error_messages +end -- 2.23.0

1 0

[PATCH compass-ci] lib: add compare_error_messages.rb
by Lin Jiaxin 19 Dec '20

19 Dec '20

Provide formatted error messages and new error number for git bisect. formatted error messages example: previous error: fcgiapp.c:1717:22: warning: comparison of integer expressions of different signedness: ‘int’ and ‘long unsigned int’ [-Wsign-compare] 1717 | if(headerLen < sizeof(header)) { | ^ new error: >> libtool: warning: relinking 'libfcgi++.la' Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 50 +++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 lib/compare_error_messages.rb diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb new file mode 100644 index 0000000..c96edef --- /dev/null +++ b/lib/compare_error_messages.rb @@ -0,0 +1,50 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'terminal-table' +require_relative 'es_query' +require_relative 'error_messages' +require_relative "#{ENV['LKP_SRC']}/lib/common" + +def get_compare_result(previous_job_id, later_job_id) + es = ESQuery.new + + previous_error_ids = es.query_by_id(previous_job_id)['error_ids'] + later_es_result = es.query_by_id(later_job_id) + later_error_ids = later_es_result['error_ids'] + later_result_file = File.join('/srv', later_es_result['result_root'], 'build-pkg') + + new_error_ids = later_error_ids - previous_error_ids + error_messages = ErrorMessages.new(later_result_file).obtain_error_messages + new_error_number, formatted_error_messages = format_error_messages(error_messages, new_error_ids) + + # new error use ">>" as an identifier in the error header + return new_error_number, formatted_error_messages +end + +def format_error_messages(error_messages, new_error_ids) + formatted_error_messages = Terminal::Table.new + formatted_error_messages.style = { border_x: '', border_y: '', border_i: '', padding_left: 0 } + new_error_number = 0 + + error_messages.each do |k, v| + if new_error_ids.include?('build-pkg.' + build_pkg_error_id(k)) + new_error_number += 1 + formatted_error_messages = add_sign(formatted_error_messages, '>>', v) + else + formatted_error_messages = add_sign(formatted_error_messages, ' ', v) + end + end + + return new_error_number, formatted_error_messages.to_s +end + +def add_sign(formatted_error_messages, sign, set) + set.each do |value| + formatted_error_messages.add_row([sign, value]) + end + + return formatted_error_messages +end -- 2.23.0

1 0

[PATCH v5 compass-ci 2/2] container: http-initrd container support job read /srv/cci/build-config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/initrd-http/root/etc/nginx/conf.d/default.conf | 4 ++++ container/initrd-http/start | 1 + 2 files changed, 5 insertions(+) diff --git a/container/initrd-http/root/etc/nginx/conf.d/default.conf b/container/initrd-http/root/etc/nginx/conf.d/default.conf index f2834c7..18813a2 100644 --- a/container/initrd-http/root/etc/nginx/conf.d/default.conf +++ b/container/initrd-http/root/etc/nginx/conf.d/default.conf @@ -13,4 +13,8 @@ server { location /initrd { autoindex on; } + + location /cci { + autoindex on; + } } diff --git a/container/initrd-http/start b/container/initrd-http/start index ed5d443..8341050 100755 --- a/container/initrd-http/start +++ b/container/initrd-http/start @@ -14,6 +14,7 @@ cmd=( --name initrd-http -p ${INITRD_HTTP_PORT:-8800}:80 -v /srv/initrd:/usr/share/nginx/html/initrd:ro + -v /srv/cci:/usr/share/nginx/html/cci:ro -v /etc/localtime:/etc/localtime:ro -d initrd-http -- 2.23.0

1 0

[PATCH v5 compass-ci 1/2] sched: refactor set_upload_dirs and add config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

When build-pkg job has config in job.yaml need support cci/build-config dir write. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/result-webdav/start | 1 + src/lib/job.cr | 25 ++++++++++++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-) diff --git a/container/result-webdav/start b/container/result-webdav/start index bbbbb24..2852f24 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -21,6 +21,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ + -v /srv/cci/build-config/:/srv/cci/build-config/ result-webdav ) diff --git a/src/lib/job.cr b/src/lib/job.cr index a23475d..4d0b5df 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -233,18 +233,33 @@ class Job set_upload_dirs() end - def set_upload_dirs + def get_package_dir + if @hash["cci-makepkg"]? || @hash["cci-depends"]? || @hash["build-pkg"]? + mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup + common_dir = "#{mount_type}/#{os}/#{os_arch}/#{os_version}" + end + if @hash["cci-makepkg"]? - package_dir = ",/initrd/pkg" + package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" elsif @hash["cci-depends"]? - package_dir = ",/initrd/deps" + package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["build-pkg"]? - package_dir = ",/initrd/build-pkg" + if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] + else + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-1] + end + package_dir = ",/initrd/build-pkg/#{common_dir}/#{package_name}" + package_dir += ",/cci/build-config" if @hash["config"]? else package_dir = "" end - self["upload_dirs"] = "#{result_root}#{package_dir}" + return package_dir + end + + def set_upload_dirs + self["upload_dirs"] = "#{result_root}#{get_package_dir}" end private def set_result_service -- 2.23.0

1 0

[PATCH v4 compass-ci 2/2] container: http-initrd container support job write cci/build-config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/initrd-http/root/etc/nginx/conf.d/default.conf | 4 ++++ container/initrd-http/start | 1 + 2 files changed, 5 insertions(+) diff --git a/container/initrd-http/root/etc/nginx/conf.d/default.conf b/container/initrd-http/root/etc/nginx/conf.d/default.conf index f2834c7..546e824 100644 --- a/container/initrd-http/root/etc/nginx/conf.d/default.conf +++ b/container/initrd-http/root/etc/nginx/conf.d/default.conf @@ -13,4 +13,8 @@ server { location /initrd { autoindex on; } + + location /build-config { + autoindex on; + } } diff --git a/container/initrd-http/start b/container/initrd-http/start index ed5d443..ac8506e 100755 --- a/container/initrd-http/start +++ b/container/initrd-http/start @@ -14,6 +14,7 @@ cmd=( --name initrd-http -p ${INITRD_HTTP_PORT:-8800}:80 -v /srv/initrd:/usr/share/nginx/html/initrd:ro + -v /srv/cci/build-config:/usr/share/nginx/html/build-config:ro -v /etc/localtime:/etc/localtime:ro -d initrd-http -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] sched: refactor set_upload_dirs and add config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

When build-pkg job has config in job.yaml need support cci/build-config dir write. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/result-webdav/start | 1 + src/lib/job.cr | 25 ++++++++++++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-) diff --git a/container/result-webdav/start b/container/result-webdav/start index bbbbb24..2852f24 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -21,6 +21,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ + -v /srv/cci/build-config/:/srv/cci/build-config/ result-webdav ) diff --git a/src/lib/job.cr b/src/lib/job.cr index a23475d..4d0b5df 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -233,18 +233,33 @@ class Job set_upload_dirs() end - def set_upload_dirs + def get_package_dir + if @hash["cci-makepkg"]? || @hash["cci-depends"]? || @hash["build-pkg"]? + mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup + common_dir = "#{mount_type}/#{os}/#{os_arch}/#{os_version}" + end + if @hash["cci-makepkg"]? - package_dir = ",/initrd/pkg" + package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" elsif @hash["cci-depends"]? - package_dir = ",/initrd/deps" + package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["build-pkg"]? - package_dir = ",/initrd/build-pkg" + if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] + else + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-1] + end + package_dir = ",/initrd/build-pkg/#{common_dir}/#{package_name}" + package_dir += ",/cci/build-config" if @hash["config"]? else package_dir = "" end - self["upload_dirs"] = "#{result_root}#{package_dir}" + return package_dir + end + + def set_upload_dirs + self["upload_dirs"] = "#{result_root}#{get_package_dir}" end private def set_result_service -- 2.23.0

1 0

[PATCH compass-ci] doc: update documents
by Luan Shengde 19 Dec '20

19 Dec '20

update documents: my_uuid => my_token Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 6 +++--- doc/manual/apply-account.md | 2 +- ...43\205compass-ci\345\256\242\346\210\267\347\253\257.md" | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/mail-robot.md b/doc/mail-robot.md index 06e4e87..1e8d618 100644 --- a/doc/mail-robot.md +++ b/doc/mail-robot.md @@ -58,14 +58,14 @@ attahcment first attachment - apply_my_account - my_uuid - generate uuid + my_token + generate my_token apply_account invoke ApplyJumperAccount to apply new account with my_info and pub_key my_info: - my_email - my_name - - my_uuid + - my_token apply_jumper_account required data: pub_key complete my_info diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index d9a03c8..8ae6c6d 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -25,7 +25,7 @@ Which contains account information for you: my_email my_name - my_uuid + my_token ## step 3: one-time setup diff --git "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" index 5a02431..160fdd0 100644 --- "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" +++ "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" @@ -14,7 +14,7 @@ Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客 ```SHELL mkdir -p ~/.config/compass-ci/defaults/ cat >> ~/.config/compass-ci/defaults/${USER}.yaml <<-EOF - my_uuid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx + my_token: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx my_email: xxx my_name: xx EOF -- 2.23.0

2 2

[PATCH v3 compass-ci] sched: refactor set_upload_dirs and add config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

When build-pkg job has config in job.yaml need support cci/build-config dir write. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/result-webdav/start | 1 + src/lib/job.cr | 25 ++++++++++++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-) diff --git a/container/result-webdav/start b/container/result-webdav/start index bbbbb24..2852f24 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -21,6 +21,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ + -v /srv/cci/build-config/:/srv/cci/build-config/ result-webdav ) diff --git a/src/lib/job.cr b/src/lib/job.cr index a23475d..d5e8ad5 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -233,18 +233,33 @@ class Job set_upload_dirs() end - def set_upload_dirs + def get_package_dir + if @hash["cci-makepkg"]? || @hash["cci-depends"]? || @hash["build-pkg"]? + mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup + common_dir = "#{mount_type}/#{os}/#{os_arch}/#{os_version}" + end + if @hash["cci-makepkg"]? - package_dir = ",/initrd/pkg" + package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" elsif @hash["cci-depends"]? - package_dir = ",/initrd/deps" + package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["build-pkg"]? - package_dir = ",/initrd/build-pkg" + if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] + else + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-1] + end + package_dir = ",/initrd/build-pkg/#{common_dir}/#{package_name}" + package_dir += ",/cci/config" if @hash["config"]? else package_dir = "" end - self["upload_dirs"] = "#{result_root}#{package_dir}" + return package_dir + end + + def set_upload_dirs + self["upload_dirs"] = "#{result_root}#{get_package_dir}" end private def set_result_service -- 2.23.0

1 1

[PATCH v2 compass-ci] sched: refactor set_upload_dirs and add config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

When build-pkg job has config in job.yaml need support cci/build-config dir write. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/result-webdav/start | 1 + src/lib/job.cr | 25 ++++++++++++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-) diff --git a/container/result-webdav/start b/container/result-webdav/start index bbbbb24..a719806 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -21,6 +21,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ + -v /srv/cci/config/:/srv/cci/config/ result-webdav ) diff --git a/src/lib/job.cr b/src/lib/job.cr index a23475d..d5e8ad5 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -233,18 +233,33 @@ class Job set_upload_dirs() end - def set_upload_dirs + def get_package_dir + if @hash["cci-makepkg"]? || @hash["cci-depends"]? || @hash["build-pkg"]? + mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup + common_dir = "#{mount_type}/#{os}/#{os_arch}/#{os_version}" + end + if @hash["cci-makepkg"]? - package_dir = ",/initrd/pkg" + package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" elsif @hash["cci-depends"]? - package_dir = ",/initrd/deps" + package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["build-pkg"]? - package_dir = ",/initrd/build-pkg" + if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] + else + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-1] + end + package_dir = ",/initrd/build-pkg/#{common_dir}/#{package_name}" + package_dir += ",/cci/config" if @hash["config"]? else package_dir = "" end - self["upload_dirs"] = "#{result_root}#{package_dir}" + return package_dir + end + + def set_upload_dirs + self["upload_dirs"] = "#{result_root}#{get_package_dir}" end private def set_result_service -- 2.23.0

1 1

[PATCH compass-ci] modify options for auto submit api-avx2neon.yaml
by Li Ping 19 Dec '20

19 Dec '20

Signed-off-by: Li Ping <1477412247(a)qq.com> --- sbin/auto_submit.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index d0f4fbe..e1eb39b 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -1,5 +1,5 @@ a/AvxToNeon/AvxToNeon: -- testbox=vm-2p8g os=centos os_version=7.6.1810 os_mount=initramfs os_arch=aarch64 api-avx2neon.yaml +- testbox=vm-2p8g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 api-avx2neon.yaml c/compass-ci/compass-ci: - testbox=taishan200-2280-2s64p-256g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml - testbox=taishan200-2280-2s64p-256g os=centos os_version=7.6.1810 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml -- 2.23.0

1 0

[PATCH v1 compass-ci] sched: refactor set_upload_dirs and add config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

When build-pkg job has config in job.yaml need support cci/config dir read and write. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/result-webdav/start | 1 + src/lib/job.cr | 19 ++++++++++++++----- 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/container/result-webdav/start b/container/result-webdav/start index bbbbb24..a719806 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -21,6 +21,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ + -v /srv/cci/config/:/srv/cci/config/ result-webdav ) diff --git a/src/lib/job.cr b/src/lib/job.cr index a23475d..85159f5 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -233,18 +233,27 @@ class Job set_upload_dirs() end - def set_upload_dirs + def get_package_dir + if @hash["cci-makepkg"]? or @hash["cci-depends"]? or @hash["build-pkg"]? + common_dir = "#{os_mount}/#{os}/#{os_arch}/#{os_version}" + end + if @hash["cci-makepkg"]? - package_dir = ",/initrd/pkg" + package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" elsif @hash["cci-depends"]? - package_dir = ",/initrd/deps" + package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["build-pkg"]? - package_dir = ",/initrd/build-pkg" + package_dir = ",/initrd/build-pkg/#{common_dir}/#{@hash["pkgbuild_repo"].split("/")[-1]}" + package_dir += ",/cci/config" if @hash["config"]? else package_dir = "" end - self["upload_dirs"] = "#{result_root}#{package_dir}" + return package_dir + end + + def set_upload_dirs + self["upload_dirs"] = "#{result_root}#{get_package_dir}" end private def set_result_service -- 2.23.0

2 3

[PATCH lkp-tests] lib/monitor: get host from job
by Wu Zhende 19 Dec '20

19 Dec '20

If @job is exist and monitor_url is not configured, set monitor_url based on SCHED_HOST Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index f65cfccf1..283cd788d 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -24,7 +24,6 @@ class Monitor # 1 means timeout exit @exit_status_code = 0 @defaults = {} - load_default @result = [] @stop_query = {} @reason = nil @@ -33,6 +32,10 @@ class Monitor def load_default return unless @monitor_url == '' + if host = @job["SCHED_HOST"] + return @monitor_url = "ws://#{host}:11310/filter" + end + Dir.glob(['/etc/compass-ci/monitor/*.yaml', "#{ENV['HOME']}/.config/compass-ci/monitor/*.yaml"]).each do |file| next unless File.exist? file @@ -98,6 +101,7 @@ class Monitor def run(timeout = nil) merge_overrides + load_default field_check @query.each do |k, v| -- 2.23.0

1 0

[PATCH compass-ci] modify assign-account-email
by Li Ping 19 Dec '20

19 Dec '20

[why]: source /root/.bashrc && /root/.bash_profile follow this step can not success. Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/mail-robot/lib/assign-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index 77ed4e8..dccbd40 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -37,7 +37,7 @@ def build_apply_account_email(my_info) git clone https://gitee.com/wu_fengguang/lkp-tests.git cd lkp-tests make install - source ${HOME}/.bashrc && source ${HOME}/.bash_profile + source ~/.${SHELL##*/}rc 3) submit job reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md -- 2.23.0

2 1

[PATCH compass-ci] sched: refactor set_upload_dirs and add config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

When build-pkg job has config in job.yaml need support cci/config dir read and write. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/result-webdav/start | 1 + src/lib/job.cr | 17 ++++++++++++----- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/container/result-webdav/start b/container/result-webdav/start index bbbbb24..a719806 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -21,6 +21,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ + -v /srv/cci/config/:/srv/cci/config/ result-webdav ) diff --git a/src/lib/job.cr b/src/lib/job.cr index a23475d..5644ddf 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -233,18 +233,25 @@ class Job set_upload_dirs() end - def set_upload_dirs + def get_package_dir + if @hash["cci-makepkg"]? or @hash["cci-depends"]? or @hash["build-pkg"]? + common_dir = "#{os_mount}/#{os}/#{os_arch}/#{os_version}" + end + if @hash["cci-makepkg"]? - package_dir = ",/initrd/pkg" + package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" elsif @hash["cci-depends"]? - package_dir = ",/initrd/deps" + package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["build-pkg"]? - package_dir = ",/initrd/build-pkg" + package_dir = ",/initrd/build-pkg/#{common_dir}/#{@hash["pkgbuild_repo"].split("/")[-1]}" + package_dir += ",/cci/config" if @hash["config"]? else package_dir = "" end + end - self["upload_dirs"] = "#{result_root}#{package_dir}" + def set_upload_dirs + self["upload_dirs"] = "#{result_root}#{get_package_dir}" end private def set_result_service -- 2.23.0

1 1

[PATCH compass-ci 3/3] rename my_uuid to my_token
by Wu Fengguang 19 Dec '20

19 Dec '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/mail-robot.md | 6 +++--- doc/manual/apply-account.md | 4 ++-- sbin/build-my-info.rb | 6 +++--- sbin/es-accounts-mapping.sh | 2 +- src/delimiter/utils.rb | 4 ++-- 5 files changed, 11 insertions(+), 11 deletions(-) diff --git a/doc/mail-robot.md b/doc/mail-robot.md index 06e4e87..d3778e7 100644 --- a/doc/mail-robot.md +++ b/doc/mail-robot.md @@ -58,14 +58,14 @@ attahcment first attachment - apply_my_account - my_uuid + my_token generate uuid apply_account invoke ApplyJumperAccount to apply new account with my_info and pub_key my_info: - - my_email - my_name - - my_uuid + - my_email + - my_token apply_jumper_account required data: pub_key complete my_info diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index d9a03c8..7e1a6a6 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -23,9 +23,9 @@ Email template: Which contains account information for you: - my_email my_name - my_uuid + my_email + my_token ## step 3: one-time setup diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index 73f7390..d75f767 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -12,12 +12,12 @@ print 'email: ' my_email = $stdin.echo = gets.chomp print 'name: ' my_name = $stdin.echo = gets.chomp -my_uuid = %x(uuidgen).chomp +my_token = %x(uuidgen).chomp my_info = { - 'my_email' => my_email, 'my_name' => my_name, - 'my_uuid' => my_uuid + 'my_email' => my_email, + 'my_token' => my_token } def store_account_info(my_info) diff --git a/sbin/es-accounts-mapping.sh b/sbin/es-accounts-mapping.sh index d5566dc..d04b0f3 100755 --- a/sbin/es-accounts-mapping.sh +++ b/sbin/es-accounts-mapping.sh @@ -18,7 +18,7 @@ else "my_commit_url": { "type": "keyword" }, - "my_uuid": { + "my_token": { "type": "keyword" }, "my_email": { diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index e87bbbe..41803f3 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -101,9 +101,9 @@ module Utils raise "query #{DELIMITER_ACCONUT} account info failed!" unless account_info job['suite'] = 'bisect' - job['my_uuid'] = account_info['my_uuid'] job['my_name'] = account_info['my_name'] job['my_email'] = account_info['my_email'] + job['my_token'] = account_info['my_token'] job['bad_job_id'] = job_id job.delete('error_ids') @@ -127,7 +127,7 @@ module Utils def create_bisect_log(git_dir) FileUtils.mkdir_p TMP_RESULT_ROOT unless File.exist? TMP_RESULT_ROOT - log_file = File.join(TMP_RESULT_ROOT, "bisect.log") + log_file = File.join(TMP_RESULT_ROOT, 'bisect.log') log_content = parse_bisect_log(git_dir) File.open(log_file, 'w') do |f| log_content.each { |line| f.puts(line) } -- 2.23.0

1 0

[PATCH compass-ci 2/3] assign-account-email: remove .bash_profile usage
by Wu Fengguang 19 Dec '20

19 Dec '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- container/mail-robot/lib/assign-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index 77ed4e8..ee3b574 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -37,7 +37,7 @@ def build_apply_account_email(my_info) git clone https://gitee.com/wu_fengguang/lkp-tests.git cd lkp-tests make install - source ${HOME}/.bashrc && source ${HOME}/.bash_profile + source ${HOME}/.${SHELL##*/}rc 3) submit job reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md -- 2.23.0

1 0

[PATCH compass-ci 1/3] doc/manual: improve install client document
by Wu Fengguang 19 Dec '20

19 Dec '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- ...ci\345\256\242\346\210\267\347\253\257.md" | 27 +++++++------------ 1 file changed, 10 insertions(+), 17 deletions(-) diff --git "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" index 5a02431..dd115f7 100644 --- "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" +++ "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" @@ -2,32 +2,25 @@ Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客户端，通过本地安装 lkp-tests 可以手动提交测试任务 +## 申请帐号 + :exclamation: 前提条件：按照 [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… 完成如下动作： - send apply account email - receive email from compass-ci(a)qq.com +- get reply email and follow its instructions to + - setup default config -## Getting started +## 下载安装 lkp-tests -1. setup default config +Run the following command to install/setup lkp-test: - run the following command to add the below setup to default config file -```SHELL - mkdir -p ~/.config/compass-ci/defaults/ - cat >> ~/.config/compass-ci/defaults/${USER}.yaml <<-EOF - my_uuid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx - my_email: xxx - my_name: xx - EOF -``` -2. download lkp-tests and dependencies - - run the following command to install the lkp-test and effect the configuration ```SHELL git clone https://gitee.com/wu_fengguang/lkp-tests.git cd lkp-tests make install - source ~/.bashrc && source ~/.bash_profile + source ~/.${SHELL##*/}rc ``` -3. submit job - Now try [submitting a job to compass-ci](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… +## submit job + +Now try [submitting a job to compass-ci](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… -- 2.23.0

1 0

[PATCH compass-ci] fetch-mail: update Dockerfile
by Luan Shengde 19 Dec '20

19 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/Dockerfile | 1 + 1 file changed, 1 insertion(+) diff --git a/container/fetch-mail/Dockerfile b/container/fetch-mail/Dockerfile index 0434e09..99abf86 100644 --- a/container/fetch-mail/Dockerfile +++ b/container/fetch-mail/Dockerfile @@ -5,6 +5,7 @@ FROM centos:latest MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive +RUN groupadd -g 1147 mailer RUN useradd -u 1144 -g 1147 mailer RUN yum update -y RUN yum install -y procmail fetchmail -- 2.23.0

1 0

[PATCH lkp-tests] daemon: fix ssh login testbox for docker
by Xiao Shenwei 18 Dec '20

18 Dec '20

[why] systemctl is not supported when use docker [how] use the sshd command Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- daemon/sshd | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/daemon/sshd b/daemon/sshd index 41fa99adf..6bfda467e 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -6,19 +6,31 @@ # sshr_port_base . $LKP_SRC/lib/http.sh +. $LKP_SRC/lib/env.sh sshr_ip=$LKP_SERVER [ -n "$sshr_port" ] || sshr_port=5050 [ -n "$sshr_port_len" ] || sshr_port_len=2000 [ -n "$sshr_port_base" ] || sshr_port_base=50000 +generate_key_file() +{ + echo -e "y\n" | ssh-keygen -t rsa -b 2048 -N '' -f /etc/ssh/ssh_host_rsa_key +} + run_ssh() { [ -n "$my_ssh_pubkey" ] || return umask 0077 mkdir -p /root/.ssh echo "$my_ssh_pubkey" > /root/.ssh/authorized_keys - systemctl start sshd + if is_docker; then + generate_key_file + sshd_path=$(cmd_path sshd) + $sshd_path -q + else + systemctl start sshd + fi } data_success() -- 2.23.0

1 0

[PATCH compass-ci] providers/docker: set the private key of the testbox
by Xiao Shenwei 18 Dec '20

18 Dec '20

[why] ssh password-free login for testbox of docker [how] Add ssh private key for docker, for automatically establish the ssh reverse tunnel between sut and port map container. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/bin/entrypoint.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/providers/docker/bin/entrypoint.sh b/providers/docker/bin/entrypoint.sh index 63cb605..feb5edf 100755 --- a/providers/docker/bin/entrypoint.sh +++ b/providers/docker/bin/entrypoint.sh @@ -2,4 +2,6 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +mkdir -p /root/.ssh +cp /lkp/lkp/src/rootfs/addon/root/.ssh/id_rsa /root/.ssh/ sh /lkp/lkp/src/rootfs/addon/etc/init.d/lkp-bootstrap -- 2.23.0

1 0

help star
by Wu Fengguang 18 Dec '20

18 Dec '20

Wei Jihui is our help star, thanks Wei Jihui! 24.0 Wu Fengguang 22.8 Wei Jihui 21.4 Yu Chuan 18.7 Cao Xueliang 12.0 Wu Zhende 10.6 Xiao Shenwei 9.2 Lu Weitao 8.5 Hu Xuejiao 7.8 Wang Yong 7.4 Liu Shaofei 6.4 Luan Shengde 5.9 Xu Xijian 5.7 Cui Yili 5.3 Zhang Yu 5.3 Zhang Dewan 5.2 Zhang Yale 5.0 Bai Jing 4.8 Liu Yinsi 4.4 Lin Jiaxin 4.3 Li Ping 4.0 Wang Chenglong 2.7 Sun Yukui 2.6 Lu Kaiyi 2.4 Ren Wen 2.2 Du Kaitian 1.3 Li Yuanchao 0.0 Tong Qunfeng 0.0 Wang Shuai 0.0 Cui Xucui

1 0

git count
by Wu Fengguang 18 Dec '20

18 Dec '20

Shengde is our code star again, thanks Shengde! wfg@crystal ~/todo% bin/git-count 14 Luan Shengde 13 Cao Xueliang 12 Wu Fengguang 10 Liu Yinsi 10 Sun Yukui 10 Wu Zhende 8 Ren Wen 7 Zhang Yu 7 Li Yuanchao 6 Cui Yili 6 Wang Chenglong 6 Lu Kaiyi 5 Liu Shaofei 5 Lu Weitao 5 Wei Jihui 5 Yu Chuan 5 Wang Yong 4 Hu Xuejiao 3 Bai Jing 3 Xu Xijian 3 Zhang Yale 3 Lin Jiaxin 3 Li Ping 2 Xiao Shenwei 0 Tong Qunfeng 0 Du Kaitian 0 Zhang Dewan

1 0

review star
by Wu Fengguang 18 Dec '20

18 Dec '20

Xueliang is our review star, thanks Xueliang! wfg@crystal ~/todo% bin/review-star Review email records in the last 7 days total review paragraph point people 121 21 24 24 Cao Xueliang 48 30 57 20 Wu Fengguang 74 16 23 15 Wei Jihui 69 7 20 10 Luan Shengde 40 13 11 7 Liu Shaofei 42 14 20 7 Liu Yinsi 29 4 13 5 Yu Chuan 8 2 6 4 Xu Xijian 36 8 7 4 Lu Kaiyi 26 4 5 4 Xiao Shenwei 36 6 4 3 Zhang Yu 22 6 7 3 Lin Jiaxin 13 4 12 2 Sun Yukui 21 5 12 2 Ren Wen 14 4 2 2 Cui Yili 6 2 1 1 Bai Jing 18 3 2 1 Hu Xuejiao 30 4 4 1 Wang Chenglong 38 1 1 1 Wu Zhende 7 2 0 0 Zhang Yale 15 3 4 0 Li Ping 9 0 0 0 Li Yuanchao 6 2 2 0 Zhang Dewan 10 0 0 0 Wang Yong 25 2 4 0 Lu Weitao

1 0

[PATCH compass-ci] lib/es_query: add query filed from index
by Lu Weitao 18 Dec '20

18 Dec '20

[Why] We need only get specific field instead of entire document. it's like SQL: SELECT DISTINCT column FROM table Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/es_query.rb | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 3bea624..86c078f 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -61,6 +61,30 @@ class ESQuery return result[0]['_source'] end + + # select doc_field from index + # input: + # eg: suite (@index: jobs) + # output: + # [ + # {"key"=>"build-pkg", "doc_count"=>90841}, + # {"key"=>"cci-depends", "doc_count"=>4636}, + # {"key"=>"cci-makepkg", "doc_count"=>3647}, + # ... + # ] + def query_specific_fields(field) + query = { + aggs: { + "all_#{field}" => { + terms: { field: field, size: 1000 } + } + }, + size: 0 + } + result = @client.search index: @index + '*', body: query + + result ['aggregations']["all_#{field}"]['buckets'] + end end # Range Query Example: -- 2.23.0

1 1

[PATCH compass-ci] mail-robot/assign-account-email: add config for lab
by Wei Jihui 18 Dec '20

18 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- container/mail-robot/lib/assign-account-email.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index 834e96f..77ed4e8 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -24,6 +24,7 @@ def build_apply_account_email(my_info) cat >> ~/.config/compass-ci/defaults/account.yaml <<-EOF my_email: #{my_info['my_email']} my_name: #{my_info['my_name']} + lab: #{ENV['lab']} EOF mkdir -p ~/.config/compass-ci/include/lab cat >> ~/.config/compass-ci/include/lab/#{ENV['lab']}.yaml <<-EOF -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/2] distro/adaptation: add sshd mapping
by Wang Yong 18 Dec '20

18 Dec '20

add sshd mapping in different OS Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/adaptation/archlinux | 2 ++ distro/adaptation/centos | 1 + distro/adaptation/fedora | 1 + distro/adaptation/kylin | 2 ++ distro/adaptation/openeuler | 2 ++ distro/adaptation/opensuse | 1 + distro/adaptation/oracle | 1 + 7 files changed, 10 insertions(+) diff --git a/distro/adaptation/archlinux b/distro/adaptation/archlinux index 53474d6c4..89163cf41 100644 --- a/distro/adaptation/archlinux +++ b/distro/adaptation/archlinux @@ -70,3 +70,5 @@ xutils-dev: imake xz-utils: xz libtirpc: libtirpc1 initramfs-tools: +openssh-server: openssh +openssh-client: openssh diff --git a/distro/adaptation/centos b/distro/adaptation/centos index e7fabf36d..2cfe3877c 100644 --- a/distro/adaptation/centos +++ b/distro/adaptation/centos @@ -334,3 +334,4 @@ junit4: junit liboath-dev: liboath-devel lsb-release: redhat-lsb linux-base: +openssh-client: openssh-clients diff --git a/distro/adaptation/fedora b/distro/adaptation/fedora index 38b7e9dc4..6686bc8c2 100644 --- a/distro/adaptation/fedora +++ b/distro/adaptation/fedora @@ -162,3 +162,4 @@ network-manager: NetworkManager javahelper: junit4: junit linux-base: +openssh-client: openssh-clients diff --git a/distro/adaptation/kylin b/distro/adaptation/kylin index f143fde96..e9286effd 100644 --- a/distro/adaptation/kylin +++ b/distro/adaptation/kylin @@ -147,3 +147,5 @@ lsb-release: openjdk: java-1.8.0-openjdk ruby-full: ruby ruby-devel linux-base: +openssh-server: openssh +openssh-client: openssh diff --git a/distro/adaptation/openeuler b/distro/adaptation/openeuler index e72f34fd2..09120489b 100644 --- a/distro/adaptation/openeuler +++ b/distro/adaptation/openeuler @@ -293,3 +293,5 @@ lsb-release: openjdk: java-1.8.0-openjdk ruby-full: ruby ruby-devel rubygem-rdoc linux-base: +openssh-server: openssh +openssh-client: openssh diff --git a/distro/adaptation/opensuse b/distro/adaptation/opensuse index 868d06b0d..f061c0894 100644 --- a/distro/adaptation/opensuse +++ b/distro/adaptation/opensuse @@ -13,3 +13,4 @@ libtirpc: libtirpc3 # Go get it yourself: https://github.com/schacon/ruby-git # We have no package for it yet ruby-git: +openssh-client: openssh-clients diff --git a/distro/adaptation/oracle b/distro/adaptation/oracle index af8415ec3..522b3d869 100644 --- a/distro/adaptation/oracle +++ b/distro/adaptation/oracle @@ -135,3 +135,4 @@ initramfs-tools: init-system-helpers: javahelper: linux-base: +openssh-client: openssh-clients -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] distro/depends: add sshd depends file
by Wang Yong 18 Dec '20

18 Dec '20

add depends file for sshd Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/depends/sshd | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 distro/depends/sshd diff --git a/distro/depends/sshd b/distro/depends/sshd new file mode 100644 index 000000000..85ba955ae --- /dev/null +++ b/distro/depends/sshd @@ -0,0 +1,2 @@ +openssh-server +openssh-client -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: fix error for submit remote job
by Wu Zhende 18 Dec '20

18 Dec '20

[Error] submit iperf.yaml failed, got job_id=0, error: Missing uuid for remote job [How] Don't need to check uuid for remote job now. All jobs use my_token Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index e7b9faf..dc43cbf 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -185,18 +185,6 @@ class Job # handle by me, then keep connect to me self["LKP_SERVER"] = SCHED_HOST self["LKP_CGI_PORT"] = SCHED_PORT.to_s - - # need further uuid check (validate? exist? no need?) - if self["SCHED_HOST"] != SCHED_HOST # remote submited job - # ?further fix to 127.0.0.1 (from remote ssh port forwarding) - # ?even set self["SCHED_HOST"] and self["SCHED_PORT"] - - if self["uuid"] == "" - puts "Job's SCHED_HOST is #{self["SCHED_HOST"]}, " + - "current scheduler IP is: #{SCHED_HOST}" - raise "Missing uuid for remote job" - end - end end private def set_sshr_info -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: add reference url
by Wu Zhende 18 Dec '20

18 Dec '20

submit iperf.yaml testbox=vm-2p8g submit iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_token'. Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index e7b9faf..e046072 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -325,7 +325,11 @@ class Job private def check_required_keys REQUIRED_KEYS.each do |key| if !@hash[key]? - raise "Missing required job key: '#{key}'" + error_msg = "Missing required job key: '#{key}'." + if ["my_email", "my_name", "my_token"].includes?(key) + error_msg += "\nPlease refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco…" + end + raise error_msg end end end -- 2.23.0

1 0

[PATCH compass-ci] container/initrd-http: support "/srv/cci/" dir download
by Cao Xueliang 18 Dec '20

18 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/initrd-http/root/etc/nginx/conf.d/default.conf | 4 ++++ container/initrd-http/start | 1 + 2 files changed, 5 insertions(+) diff --git a/container/initrd-http/root/etc/nginx/conf.d/default.conf b/container/initrd-http/root/etc/nginx/conf.d/default.conf index f2834c7..18813a2 100644 --- a/container/initrd-http/root/etc/nginx/conf.d/default.conf +++ b/container/initrd-http/root/etc/nginx/conf.d/default.conf @@ -13,4 +13,8 @@ server { location /initrd { autoindex on; } + + location /cci { + autoindex on; + } } diff --git a/container/initrd-http/start b/container/initrd-http/start index ed5d443..8341050 100755 --- a/container/initrd-http/start +++ b/container/initrd-http/start @@ -14,6 +14,7 @@ cmd=( --name initrd-http -p ${INITRD_HTTP_PORT:-8800}:80 -v /srv/initrd:/usr/share/nginx/html/initrd:ro + -v /srv/cci:/usr/share/nginx/html/cci:ro -v /etc/localtime:/etc/localtime:ro -d initrd-http -- 2.23.0

1 0

[PATCH v2 compass-ci] doc: and runtime field in borrow-machine.en.md and borrow-machine.zh.md
by Zhang Yale 18 Dec '20

18 Dec '20

User can specify the sleep time in "runtime" field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/manual/borrow-machine.en.md | 11 ++++++----- doc/manual/borrow-machine.zh.md | 11 ++++++----- 2 files changed, 12 insertions(+), 10 deletions(-) diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index b9c2bc4..0277984 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -202,7 +202,8 @@ Ensure that you have performed the following operations according to the [apply- hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml suite: borrow testcase: borrow - + + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -212,10 +213,10 @@ Ensure that you have performed the following operations according to the [apply- %> sshd: # sleep at the bottom - sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # Use the VIM editor to change the value of the sleep field. + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml + runtime: 1h + # Use the VIM editor to change the value of the runtime field. hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # After changing the value, submit the request again. hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 06d4db6..5bc951d 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -206,7 +206,8 @@ hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml suite: borrow testcase: borrow - + + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -216,10 +217,10 @@ %> sshd: # sleep at the bottom - sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # ��vim��sleep�� + runtime: 1h + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml + runtime: 1h + # ��vim��runtime�� hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # ��submit�� hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/2] tests/build-pkg: disable 2>&1 when run makepkg
by Lin Jiaxin 18 Dec '20

18 Dec '20

Make error will redirect itself to stdout now. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 59625299..addff9f7 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -129,7 +129,7 @@ build_source_pkg() create_cgz_name PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ - $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf 2>&1 + $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf [ "$?" == 0 ] || exit 1 create_softlink -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] revoke commit: 2dad3102dc5c11d02e66eab9840bcdffb11e93f0
by Lin Jiaxin 18 Dec '20

18 Dec '20

[Why] We modify the make source code let it's stderr to stdout when compile error. So no need for this commit. before: all stderr => stdout => file: build-pkg => stats/build-pkg -- when run makepkg But stats/build-pkg can only analyze make error. So it ignores some other errors. after: unknown error => stderr => file: stderr => stats/stderr make stderr => stdout => file: build-pkg => stats/build-pkg Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/makepkg | 18 +++--------------- tests/build-pkg | 4 ++-- 2 files changed, 5 insertions(+), 17 deletions(-) diff --git a/sbin/makepkg b/sbin/makepkg index 131ae690..498b4872 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -1761,19 +1761,11 @@ run_build() { [[ -d /usr/lib/ccache/bin ]] && export PATH="/usr/lib/ccache/bin:$PATH" fi - if [[ $STDERR_REDIR == y ]];then - run_function_safe "build" 2>&1 - else - run_function_safe "build" - fi + run_function_safe "build" } run_check() { - if [[ $STDERR_REDIR == y ]];then - run_function_safe "check" 2>&1 - else - run_function_safe "check" - fi + run_function_safe "check" } run_package() { @@ -1784,11 +1776,7 @@ run_package() { pkgfunc="package_$1" fi - if [[ $STDERR_REDIR == y ]];then - run_function_safe "$pkgfunc" 2>&1 - else - run_function_safe "$pkgfunc" - fi + run_function_safe "$pkgfunc" } build_id() { diff --git a/tests/build-pkg b/tests/build-pkg index dd1ee120..59625299 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -128,8 +128,8 @@ build_source_pkg() create_cgz_name - PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" STDERR_REDIR=y\ - $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf + PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ + $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf 2>&1 [ "$?" == 0 ] || exit 1 create_softlink -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: rename my_uuid to my_token
by Wu Zhende 18 Dec '20

18 Dec '20

my_token is more expressive Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1d44559..fa8e2dc 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -319,7 +319,7 @@ class Job testbox my_email my_name - my_uuid + my_token ] private def check_required_keys @@ -340,9 +340,10 @@ class Job raise error_msg if @account_info["found"]? == false raise error_msg unless self["my_name"] == @account_info["my_name"].to_s - raise error_msg unless self["my_uuid"] == @account_info["my_uuid"] + raise error_msg unless self["my_token"] == @account_info["my_token"] @hash.delete("my_uuid") + @hash.delete("my_token") end private def get_initialized_keys @@ -366,7 +367,7 @@ class Job "SCHED_HOST", "SCHED_PORT"] - initialized_keys -= ["my_uuid", + initialized_keys -= ["my_token", "kernel_version", "kernel_uri", "kernel_params", -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: fix incorrect git_tag value
by Liu Shaofei 18 Dec '20

18 Dec '20

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index fd2697e25..dd1ee120f 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -97,7 +97,7 @@ replace_source() echo "$url" | egrep 'git\+|\.git$' && { [ -n "$repo_dir" ] && { url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}" - [ -n ${git_tag} ] && { + [ -n "$git_tag" ] && { url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#tag=${git_tag}" } } -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: fix incorrect git_tag value
by Liu Shaofei 18 Dec '20

18 Dec '20

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index fd2697e25..9cb51ff6f 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -17,7 +17,6 @@ ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, ## make customizations, generate meta-info, and package everything together. ## See: https://wiki.archlinux.org/index.php/Makepkg - . $LKP_SRC/lib/debug.sh . $LKP_SRC/distro/common @@ -97,7 +96,7 @@ replace_source() echo "$url" | egrep 'git\+|\.git$' && { [ -n "$repo_dir" ] && { url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}" - [ -n ${git_tag} ] && { + [ -n "$git_tag" ] && { url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#tag=${git_tag}" } } -- 2.23.0

1 1

[PATCH compass-ci] fix: idle job for subqueue: submit idle jobs when finding no job
by Ren Wen 17 Dec '20

17 Dec '20

[Processes] There're 2 types of queues to consume: 1. user queue : sched/<taishan-a12>/<my_email> 2. idle queue : sched/<taishan-a12>/idle Consume Queue 1 at first, next Queue2 if no job found in Queue1. If there's no job in all Queues, auto submit idle jobs to idle queue. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 7565375..567016e 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -51,31 +51,38 @@ class Sched end def get_queues(host) - queues = [] of String + default_queues = [] of String + sub_queues = [] of String queues_str = @redis.hash_get("sched/host2queues", host) - return queues unless queues_str + return default_queues, sub_queues unless queues_str queues_str.split(',', remove_empty: true) do |item| - queues << item.strip + default_queues << item.strip end - sub_queues = [] of String - queues.each do |queue| - matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") + default_queues.each do |queue| + matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") matched_queues.each do |mq| match_data = "#{mq}".match(%r(^#{QUEUE_NAME_BASE}/sched/(#{queue}/.+)/ready$)) sub_queues << match_data[1] if match_data end end - return rand_queues(sub_queues) + return default_queues, sub_queues end def get_job_from_queues(queues, testbox) + default_queues, sub_queues = queues + all_queues = rand_queues(sub_queues) + + default_queues.each do |queue| + all_queues << "#{queue}/idle" + end + job = nil - queues.each do |queue| + all_queues.each do |queue| job = prepare_job("sched/#{queue}", testbox) return job if job end @@ -89,6 +96,8 @@ class Sched if job create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) + else + spawn { auto_submit_idle_job(host) } end return boot_content(job, boot_type) -- 2.23.0

1 0

[PATCH compass-ci] src/lib/web_backend.rb remove non-existent os_version
by Lu Weitao 17 Dec '20

17 Dec '20

remove non-existent os_version of compare_candidates Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/lib/web_backend.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 07e2e9e..cacafde 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -97,7 +97,7 @@ def compare_candidates_body suite: all_suite, OS: [ { os: 'openeuler', os_version: ['1.0', '20.03'] }, - { os: 'centos', os_version: ['7.6', '7.8', '8.1', 'sid'] }, + { os: 'centos', os_version: ['7.6', '7.8', '8.1'] }, { os: 'debian', os_version: %w[10 sid] }, { os: 'archlinux', os_version: ['5.5.0-1'] } ], -- 2.23.0

1 0

[PATCH lab-z9] hosts: add taishan200-2280-2s64p-256g--a121
by Wei Jihui 17 Dec '20

17 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- hosts/taishan200-2280-2s64p-256g--a121 | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 hosts/taishan200-2280-2s64p-256g--a121 diff --git a/hosts/taishan200-2280-2s64p-256g--a121 b/hosts/taishan200-2280-2s64p-256g--a121 new file mode 100644 index 0000000..9c87315 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a121 @@ -0,0 +1,22 @@ +memory: 256G +nr_ssd_partitions: 1 +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43943 +mac_addr: + - 44:67:47:e9:76:95 + - 44:67:47:e9:76:96 + - 44:67:47:e9:76:97 + - 44:67:47:e9:76:98 + - ac:b3:b5:1e:d3:ab + - ac:b3:b5:1e:d3:ac + - ac:b3:b5:1e:d3:ad + - ac:b3:b5:1e:d3:ae + - ac:b3:b5:1e:d6:0b + - ac:b3:b5:1e:d6:0c + - ac:b3:b5:1e:d6:0d + - ac:b3:b5:1e:d6:0e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.12.2 -- 2.23.0

1 0

[PATCH compass-ci] container/web-backend: install terminal-table
by Li Yuanchao 17 Dec '20

17 Dec '20

As code changes, 'terminal-table' is needed in this container. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/web-backend/Dockerfile b/container/web-backend/Dockerfile index 4fb6610..2ac5148 100644 --- a/container/web-backend/Dockerfile +++ b/container/web-backend/Dockerfile @@ -11,7 +11,7 @@ COPY conf/sources.list /etc/apt/ RUN apt-get update && \ apt-get install -y git ruby-json ruby-rails ruby-sinatra && \ gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ - gem install elasticsearch redis git + gem install elasticsearch redis git terminal-table COPY web-backend /usr/local/bin/ -- 2.23.0

1 0

[PATCH compass-ci] fix: idle job for subqueue: also hand over job from idle queue
by Ren Wen 17 Dec '20

17 Dec '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/close_job.cr | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index d8f80d7..a389a5c 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -14,8 +14,11 @@ class Sched raise "es set job content fail!" end + subqueue = job.subqueue + queue = subqueue == "idle" ? job.queue : "#{job.queue}/#{job.subqueue}" + response = @task_queue.hand_over_task( - "sched/#{job.queue}/#{job.subqueue}", "extract_stats", job_id + "sched/#{queue}", "extract_stats", job_id ) if response[0] != 201 raise "#{response}" -- 2.23.0

1 0