- Compass-ci - mailweb.openeuler.org

[PATCH v3 lkp-tests] tests/build-pkg: add config file support
by Wang Yong 22 Dec '20

22 Dec '20

[Why] config file in some software need storage and recall now we can add config value in yaml and cp ${config} to ${cci_dir}/${pkgname}/ in PKGBUILD so that config file can be uploaded correctly $ cat test.yaml ... config: ${config} ... Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index addff9f7b..06a58d498 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -4,6 +4,7 @@ # - upstream_commit # - git_tag # - pkg_ver +# - config # upstream_url # We'll use the below global env vars, but not treat them as parameter to avoid @@ -20,6 +21,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/distro/common +. $LKP_SRC/lib/http.sh check_vars() { @@ -31,6 +33,29 @@ check_vars() [ -n "$os_mount" ] || die "os_mount is empty" } +check_config_var() +{ + [ -n "$config" ] || return +} + +get_config() +{ + local cci_http_host=${INITRD_HTTP_HOST} + local cci_http_port=${INITRD_HTTP_PORT:-8800} + export cci_dir=/cci/build-config + local remote_dir="http://${cci_http_host}:${cci_http_port}${cci_dir}/${pkgname}" + + mkdir -p ${cci_dir} + + check_config_var + http_get_file "${remote_dir}/${config}" "${cci_dir}/${pkgname}/${config}" +} + +upload_config() +{ + upload_to_target_dir "${cci_dir}/${pkgname}" +} + mount_dest() { # the same image is mounted to cifs and nfs, the generated cgz files @@ -46,7 +71,7 @@ get_pkgfile() { curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "${2##*/}" - } +} request_pkg() { @@ -125,13 +150,17 @@ build_source_pkg() fi } replace_source PKGBUILD.src - create_cgz_name + get_config PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf - [ "$?" == 0 ] || exit 1 + local build_state="$?" + + upload_config + + [ "${build_state}" == "0" ] || exit 1 create_softlink upload_to_target_dir ${PKG_MNT}/${pack_to}/${pkgname} } -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/job.cr: check account info
by Wu Zhende 22 Dec '20

22 Dec '20

If submit yaml in /srv/result, will not check account info. Fix it Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4d0b5df..e6f7e6a 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -51,17 +51,23 @@ class Job def initialize(job_content : JSON::Any, id) @hash = job_content.as_h + @es = Elasticsearch::Client.new + @account_info = Hash(String, JSON::Any).new # init job with "-1", or use the original job_content["id"] id = "-1" if "#{id}" == "" if initialized? - return if @hash["id"] == "#{id}" + if @hash["id"] == "#{id}" + return unless @hash.has_key?("my_uuid") || @hash.has_key?("my_token") + + check_account_info() + return + end end - @es = Elasticsearch::Client.new @hash["id"] = JSON::Any.new("#{id}") - @account_info = Hash(String, JSON::Any).new + check_required_keys() check_account_info() set_defaults() -- 2.23.0

2 1

[PATCH v2 lkp-tests] daemon: fix ssh login testbox for docker
by Xiao Shenwei 22 Dec '20

22 Dec '20

[why] systemctl is not supported when use docker [how] use the sshd command Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- daemon/sshd | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/daemon/sshd b/daemon/sshd index 41fa99adf..6bfda467e 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -6,19 +6,31 @@ # sshr_port_base . $LKP_SRC/lib/http.sh +. $LKP_SRC/lib/env.sh sshr_ip=$LKP_SERVER [ -n "$sshr_port" ] || sshr_port=5050 [ -n "$sshr_port_len" ] || sshr_port_len=2000 [ -n "$sshr_port_base" ] || sshr_port_base=50000 +generate_key_file() +{ + echo -e "y\n" | ssh-keygen -t rsa -b 2048 -N '' -f /etc/ssh/ssh_host_rsa_key +} + run_ssh() { [ -n "$my_ssh_pubkey" ] || return umask 0077 mkdir -p /root/.ssh echo "$my_ssh_pubkey" > /root/.ssh/authorized_keys - systemctl start sshd + if is_docker; then + generate_key_file + sshd_path=$(cmd_path sshd) + $sshd_path -q + else + systemctl start sshd + fi } data_success() -- 2.23.0

4 5

Re: [PATCH lkp-tests] fix lkp make install error
by Wu Fengguang 22 Dec '20

22 Dec '20

TO ALL Best practice is to use realpath. >-. $LKP_SRC/lib/env.sh >-. $LKP_SRC/sbin/set-env.sh >+. lib/env.sh >+. sbin/set-env.sh SCRIPT_DIR=$(dirname $(realpath $0)) PROJECT_DIR=$(dirname $SCRIPT_DIR) $PROJECT_DIR/lib/env.sh $PROJECT_DIR/sbin/set-env.sh Thanks, Fengguang

2 1

[PATCH lkp-tests] daemon/multi-qemu: add $CACHE_DIR judgment
by Hu Xuejiao 22 Dec '20

22 Dec '20

$CACHE_DIR is not created in crystal Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- daemon/multi-qemu | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/daemon/multi-qemu b/daemon/multi-qemu index 90197dae..87b40758 100755 --- a/daemon/multi-qemu +++ b/daemon/multi-qemu @@ -13,7 +13,7 @@ multi_qemu() export CCI_SRC=${CCI_SRC:-"/c/compass-ci"} local hostname=$tbox_group.$HOSTNAME - cd "$CACHE_DIR" + [ -n "$CACHE_DIR" ] && cd "$CACHE_DIR" $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" } -- 2.23.0

3 2

[PATCH compass-ci] lib: add assign_account_client
by Luan Shengde 22 Dec '20

22 Dec '20

add assign_account_client for code calling to assign account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign_account_client.rb | 83 ++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100755 lib/assign_account_client.rb diff --git a/lib/assign_account_client.rb b/lib/assign_account_client.rb new file mode 100755 index 0000000..d7e0623 --- /dev/null +++ b/lib/assign_account_client.rb @@ -0,0 +1,83 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require 'set' +require 'optparse' +require_relative '../container/defconfig' +require_relative 'es_client' +require_relative '../container/mail-robot/lib/assign-account-email' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_HOST + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) +JUMPER_HOST = defaults['JUMPER_HOST'] +JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 +LAB = ENV['lab'] + +# used for other codes calling to assign account for user +class AutoAssignAccount + def initialize(user_info) + @my_info = user_info + + @my_info_es = {} + end + + def update_from_es + account_infos = ESQuery.new(index: 'accounts') + my_account_info = account_infos.query_by_id(@my_info['my_email']) || {} + + my_account_info.update @my_info + @my_info.update my_account_info + @my_info['my_ssh_pubkey'] = [] unless my_account_info['my_ssh_pubkey'] + + @my_info['my_token'] = %x(uuidgen).chomp if @my_info['my_login_name'].nil? + end + + def apply_account + apply_info = {} + + apply_info.update @my_info + apply_info['enable_login'] = true + apply_info['is_update_account'] = true unless apply_info['my_login_name'].nil? + apply_info['lab'] = LAB + + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') + JSON.parse account_info_str + end + + def update_my_info_from_account_info + account_info = apply_account + + @my_info['my_login_name'] = account_info['my_login_name'] + @my_info['my_ssh_pubkey'] << account_info['my_jumper_pubkey'] unless account_info['my_jumper_pubkey'].nil? + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def send_mail + message = build_apply_account_email(@my_info) + %x(curl -XPOST "#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text" -d "#{message}") + end + + def send_account + update_from_es + update_my_info_from_account_info + + store_account_info + send_mail + end +end -- 2.23.0

2 2

[PATCH v2 lkp-tests 1/4] bin/install-dependencies.sh: refactor file
by Wei Jihui 22 Dec '20

22 Dec '20

all dependent functions is move to lib/env.sh, sbin/set-env.sh in not required. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/install-dependencies.sh | 40 +++++++++++++++++------------------- 1 file changed, 19 insertions(+), 21 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 7605a29f..b4d3ad22 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -1,30 +1,28 @@ #!/bin/bash . $LKP_SRC/lib/env.sh -. $LKP_SRC/sbin/set-env.sh -# choose install function base on -# DISTRIBUTION +# choose install function base on DISTRIBUTION linux_dep() { -source /etc/os-release -case $ID in -ubuntu|debian) - export DEBIAN_FRONTEND=noninteractive - apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client - ;; -openEuler|fedora|rhel|centos) - if type dnf > /dev/null 2>&1; then - dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - else - yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - fi - sudo gem install -f git activesupport rest-client - ;; -*) - echo "$ID not support! please install dependencies manually." - ;; -esac + source /etc/os-release + case $ID in + ubuntu|debian) + export DEBIAN_FRONTEND=noninteractive + apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client + ;; + openEuler|fedora|rhel|centos) + if type dnf > /dev/null 2>&1; then + dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + else + yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + fi + sudo gem install -f git activesupport rest-client + ;; + *) + echo "$ID not support! please install dependencies manually." + ;; + esac } -- 2.23.0

3 3

[PATCH v2 lkp-tests] tests/build-pkg: add config file support
by Wang Yong 21 Dec '20

21 Dec '20

[Why] config file in some software need storage and recall now we can add config value in yaml and cp ${config} to ${cci_dir}/${benchmark}/ in PKGBUILD so that config file can be uploaded correctly $ cat test.yaml ... config: ${config} ... Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index addff9f7b..e539be923 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -4,6 +4,7 @@ # - upstream_commit # - git_tag # - pkg_ver +# - config # upstream_url # We'll use the below global env vars, but not treat them as parameter to avoid @@ -20,6 +21,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/distro/common +. $LKP_SRC/lib/http.sh check_vars() { @@ -31,6 +33,29 @@ check_vars() [ -n "$os_mount" ] || die "os_mount is empty" } +check_config_var() +{ + [ -n "$config" ] || return +} + +get_config() +{ + local cci_http_host=${INITRD_HTTP_HOST} + local cci_http_port=${INITRD_HTTP_PORT:-8800} + export cci_dir=/cci/config + local remote_dir="http://${cci_http_host}:${cci_http_port}${cci_dir}/${pkgname}" + + mkdir -p ${cci_dir} + + check_config_var + http_get_file "${remote_dir}/${config}" "${cci_dir}/${pkgname}/${config}" +} + +upload_config() +{ + upload_to_target_dir "${cci_dir}/${pkgname}" +} + mount_dest() { # the same image is mounted to cifs and nfs, the generated cgz files @@ -46,7 +71,7 @@ get_pkgfile() { curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "${2##*/}" - } +} request_pkg() { @@ -125,13 +150,17 @@ build_source_pkg() fi } replace_source PKGBUILD.src - create_cgz_name + get_config PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf - [ "$?" == 0 ] || exit 1 + local build_state="$?" + + upload_config + + [ "${build_state}" == "0" ] || exit 1 create_softlink upload_to_target_dir ${PKG_MNT}/${pack_to}/${pkgname} } -- 2.23.0

1 0

[PATCH lkp-tests] daemon/multi-docker: add default value of queues
by Hu Xuejiao 21 Dec '20

21 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- daemon/multi-docker | 2 ++ 1 file changed, 2 insertions(+) diff --git a/daemon/multi-docker b/daemon/multi-docker index 257645b8..7759fdf3 100755 --- a/daemon/multi-docker +++ b/daemon/multi-docker @@ -6,6 +6,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +: ${queues:="$tbox_group.$HOSTNAME,$tbox_group.$(arch)"} + multi_docker() { export CCI_SRC=${CCI_SRC:-"/c/compass-ci"} -- 2.23.0

1 0

[PATCH v4 compass-ci 3/3] lib: use suites_hash instead of suites_list to keep consistent
by Lu Kaiyi 21 Dec '20

21 Dec '20

[why] suite_list maybe not consistent with group key if sort group matrixes. [how] use suites_hash instead of suites_list to keep correct congruent relationship with same group_key. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 4 ++-- lib/compare_matrixes.rb | 10 +++++----- lib/matrix2.rb | 6 +++--- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index da761ef..9090c3b 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -62,8 +62,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices, suites_list = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, suites_list, options) + groups_matrices, suites_hash = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_hash, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 0727797..6e4c2aa 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -387,25 +387,25 @@ end # input: group matrices # output: pre compare result of each group # the result with more comparison objects first -def compare_group_matrices(group_matrices, suites_list, options) +def compare_group_matrices(group_matrices, suites_hash, options) result_str = '' group_matrices_array = sort_by_matrix_size(group_matrices) group_matrices_array.each do |matrice_kv| - result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_list, options) + result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_hash[matrice_kv[0]], options) end result_str end -def get_matrix_str(matrice_key, matrice_value, suites_list, options) +def get_matrix_str(matrice_key, matrice_value, suite_list, options) m_list = [] m_titles = [] matrice_value.each do |dim, matrix| m_titles << dim m_list << matrix end - return compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) if options[:no_print] + return compare_matrixes(m_list, suite_list, m_titles, matrice_key, options: options) if options[:no_print] - print compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) + print compare_matrixes(m_list, suite_list, m_titles, matrice_key, options: options) return '' end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 4393a65..dfc606e 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -121,7 +121,7 @@ end # ... # } def combine_group_query_data(job_list, dims) - suites_list = [] + suites_hash = {} groups = auto_group(job_list, dims) groups.each do |group_key, value| if value.empty? @@ -133,10 +133,10 @@ def combine_group_query_data(job_list, dims) groups[group_key][dimension_key], suites = create_matrix(jobs) suite_list.concat(suites) end - suites_list << suite_list + suites_hash[group_key] = suite_list end - return groups, suites_list + return groups, suites_hash end # input: -- 2.23.0

1 0