- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] result-webdav: if judgment error and some file symlinks are not created
by Cui Yili 22 Dec '20

22 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index f077fca..1feddd4 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -46,7 +46,7 @@ http { link_name = "latest" else tmp_name = string.match(upload_file, "(.*)_%d+%.cgz") - if not tmp_name then + if tmp_name then link_name = tmp_name else return -- 2.23.0

2 1

[PATCH lkp-tests] Makefile: delete set-env in makefile
by Wei Jihui 22 Dec '20

22 Dec '20

set-env is move to install-dependencies.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- Makefile | 1 - 1 file changed, 1 deletion(-) diff --git a/Makefile b/Makefile index 0b2d09a2..236bea25 100644 --- a/Makefile +++ b/Makefile @@ -14,7 +14,6 @@ subsystem: install: mkdir -p $(TARGET_DIR_BIN) ln -sf $(shell pwd)/bin/lkp $(TARGET_DIR_BIN)/lkp - bash sbin/set-env.sh bash sbin/install-dependencies.sh .PHONY: doc -- 2.23.0

1 0

[PATCH lkp-tests] jobs: fix to report errors when submit jobs
by Liu Shaofei 22 Dec '20

22 Dec '20

[why] /home/liushaofei/lkp-tests/lib/job.rb:257:in `block (2 levels) in load': undefined method `start_with?' for :"# pkgbuild_repo examples":Symbol (NoMethodError) ERROR -- /home/liushaofei/lkp-tests/jobs/build-pkg.yaml: undefined method `start_with?' for :"# pkgbuild_repo examples":Symbol Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- jobs/build-pkg.yaml | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/jobs/build-pkg.yaml b/jobs/build-pkg.yaml index b17fd2ade..7a47bd724 100644 --- a/jobs/build-pkg.yaml +++ b/jobs/build-pkg.yaml @@ -1,9 +1,8 @@ -suite: build-pkg -category: functional - -# pkgbuild_repo examples +# the field of "pkgbuild_repo" refers to the path of PKGBUILD, the examples are as follow: # pkgbuild_repo: archlinux/aur-j/java8-openjdk # pkgbuild_repo: archlinux/community//keepalived/trunk # pkgbuild_repo: archlinux/packages//prometheus-node-exporter/trunk +suite: build-pkg +category: functional build-pkg: -- 2.23.0

1 0

[PATCH compass-ci] sparrow/4-docker/buildall: skip ssh-r
by Wu Zhende 22 Dec '20

22 Dec '20

Do not restart the ssh-r container randomly. Otherwise, the established connection will be disconnected, and the requested hosts can't be logged in. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sparrow/4-docker/buildall | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 98d3ce4..32801d1 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -31,6 +31,7 @@ do_one() mkdir $tmpdir/$container_name 2>/dev/null && ( cd $container + [ "$container_name" == 'ssh-r' ] && exit if [ "$action" != "reboot" ]; then [ -x build ] && ./build [ -x install ] && ./install -- 2.23.0

1 0

[PATCH compass-ci] lib/constants: MAIL_HOST=>SEND_MAIL_HOST, MAIL_PORT=>SEND_MAIL_PORT
by Cao Xueliang 22 Dec '20

22 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/constants.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/constants.rb b/lib/constants.rb index 3d1ef1d..f0938f2 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -8,8 +8,8 @@ config = cci_defaults ES_HOST = config['ES_HOST'] || '172.17.0.1' ES_PORT = config['ES_PORT'] || 9200 -MAIL_HOST = config['MAIL_HOST'] || '172.17.0.1' -MAIL_PORT = config['MAIL_PORT'] || 11311 +MAIL_HOST = config['SEND_MAIL_HOST'] || '172.17.0.1' +MAIL_PORT = config['SEND_MAIL_PORT'] || 49000 SRV_HTTP_HOST = config['SRV_HTTP_HOST'] || ENV['SRV_HTTP_HOST'] || '172.17.0.1' SRV_HTTP_PORT = config['SRV_HTTP_PORT'] || ENV['SRV_HTTP_PORT'] || 11300 -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib: add assign_account_client
by Luan Shengde 22 Dec '20

22 Dec '20

add assign_account_client for code calling to assign account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign_account_client.rb | 83 ++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100755 lib/assign_account_client.rb diff --git a/lib/assign_account_client.rb b/lib/assign_account_client.rb new file mode 100755 index 0000000..d7e0623 --- /dev/null +++ b/lib/assign_account_client.rb @@ -0,0 +1,83 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require 'set' +require 'optparse' +require_relative '../container/defconfig' +require_relative 'es_client' +require_relative '../container/mail-robot/lib/assign-account-email' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_HOST + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) +JUMPER_HOST = defaults['JUMPER_HOST'] +JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 +LAB = ENV['lab'] + +# used for other codes calling to assign account for user +class AutoAssignAccount + def initialize(user_info) + @my_info = user_info + + @my_info_es = {} + end + + def update_from_es + account_infos = ESQuery.new(index: 'accounts') + my_account_info = account_infos.query_by_id(@my_info['my_email']) || {} + + my_account_info.update @my_info + @my_info.update my_account_info + @my_info['my_ssh_pubkey'] = [] unless my_account_info['my_ssh_pubkey'] + + @my_info['my_token'] = %x(uuidgen).chomp if @my_info['my_login_name'].nil? + end + + def apply_account + apply_info = {} + + apply_info.update @my_info + apply_info['enable_login'] = true + apply_info['is_update_account'] = true unless apply_info['my_login_name'].nil? + apply_info['lab'] = LAB + + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') + JSON.parse account_info_str + end + + def update_my_info_from_account_info + account_info = apply_account + + @my_info['my_login_name'] = account_info['my_login_name'] + @my_info['my_ssh_pubkey'] << account_info['my_jumper_pubkey'] unless account_info['my_jumper_pubkey'].nil? + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def send_mail + message = build_apply_account_email(@my_info) + %x(curl -XPOST "#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text" -d "#{message}") + end + + def send_account + update_from_es + update_my_info_from_account_info + + store_account_info + send_mail + end +end -- 2.23.0

1 1

[PATCH compass-ci 2/2] lib/mail-post: add send_mail_encode function
by Luan Shengde 22 Dec '20

22 Dec '20

[why] the http protocol will parse in fail if the request data has special characters. or it's hard to analysis the data if encoded or not on the send mail service. add stand alone interface for it Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/lib/mail-post.rb b/lib/mail-post.rb index df4d255..ac455fa 100644 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -8,6 +8,7 @@ require 'json' require 'yaml' require 'open3' require 'mail' +require 'base64' set :bind, '0.0.0.0' set :port, ENV['SEND_MAIL_PORT'] @@ -37,6 +38,19 @@ post '/send_mail_text' do send_mail(mail_info) end +post '/send_mail_encode' do + data_decode = Base64.decode64(request.body.read) + data = Mail.read_from_string(data_decode) + + mail_info = { + 'subject' => data.subject, + 'to' => data.to, + 'body' => data.body.decoded + } + + send_mail(mail_info) +end + def check_send_mail(mail_info) raise 'no/empty subject.' if mail_info['subject'].nil? || mail_info['subject'].empty? raise 'no/empty email_to address.' if mail_info['to'].nil? || mail_info['to'].empty? -- 2.23.0

1 0

[PATCH lkp-tests] tests/cci-makepkg: change to logic condition
by Liu Shaofei 22 Dec '20

22 Dec '20

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/cci-makepkg | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a50a40c20..d48e8495e 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -41,6 +41,8 @@ curl_pkgfile() download_pkgfile() { + [ -n "$PKGBUILD_TAG" ] || return + local pkg_tag_file="PKGBUILD-$PKGBUILD_TAG" curl_pkgfile "pkg/$benchmark" "$pkg_tag_file" @@ -93,7 +95,7 @@ centos_update_gcc() source /opt/rh/devtoolset-8/enable } -[ -n "$PKGBUILD_TAG" ] && download_pkgfile +download_pkgfile distro_install_depends lkp-dev -- 2.23.0

1 0

[PATCH lkp-tests 2/2] lib/env.sh: refactor set-env
by Wei Jihui 22 Dec '20

22 Dec '20

move sbin/set-env.sh functions to lib/env.sh delete sbin/set-env.sh do set-env at sbin/install-dependencies Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/env.sh | 30 ++++++++++++++++++++++++++ sbin/install-dependencies.sh | 10 ++++++++- sbin/set-env.sh | 42 ------------------------------------ 3 files changed, 39 insertions(+), 43 deletions(-) delete mode 100755 sbin/set-env.sh diff --git a/lib/env.sh b/lib/env.sh index 8ac42a83..9f37d8cc 100755 --- a/lib/env.sh +++ b/lib/env.sh @@ -125,3 +125,33 @@ shell_profile() [ -f "$profile" ] || touch $profile echo $profile } + +write_shell_profile() +{ + shell_profile_file=$(shell_profile) + if [ $# -gt 0 ]; then + echo "$@" >> $shell_profile_file + fi + + source $shell_profile_file +} + +write_host() +{ + if is_system "Linux"; then + nr_cpu=$(nproc) + memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') + else + nr_cpu=$(sysctl -n hw.logicalcpu) + memory_total=$(top -l 1 | grep MemRegions | awk '{print $2}') + fi + memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') + memory=$(echo $memory_new | awk '{print int($0)+1}')G + + cat > hosts/$(hostname) <<-EOF + nr_cpu: $nr_cpu + memory: $memory + hdd_partitions: + ssd_partitions: + EOF +} diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 6263dc91..8a4b29cd 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -4,7 +4,6 @@ SCRIPT_DIR=$(dirname $(realpath $0)) PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/env.sh -. $PROJECT_DIR/sbin/set-env.sh # choose install function base on DISTRIBUTION linux_dep() @@ -48,4 +47,13 @@ run() fi } +set_env() +{ + write_host + write_shell_profile "export LKP_SRC=$PWD" + write_shell_profile "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" + source $(shell_profile) +} + +set_env run diff --git a/sbin/set-env.sh b/sbin/set-env.sh deleted file mode 100755 index 8234eaf8..00000000 --- a/sbin/set-env.sh +++ /dev/null @@ -1,42 +0,0 @@ -#!/bin/sh - -SCRIPT_DIR=$(dirname $(realpath $0)) -PROJECT_DIR=$(dirname $SCRIPT_DIR) - -. $PROJECT_DIR/lib/env.sh - -write_shell_profile() -{ - shell_profile_file=$(shell_profile) - if [ $# -gt 0 ]; then - echo $@ >> $shell_profile_file - else - echo "export LKP_SRC=$PWD" >> $shell_profile_file - echo "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" >> $shell_profile_file - fi - - source $shell_profile_file -} - -write_host() -{ - if is_system "Linux"; then - nr_cpu=$(nproc) - memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') - else - nr_cpu=$(sysctl -n hw.logicalcpu) - memory_total=$(top -l 1 | grep MemRegions | awk '{print $2}') - fi - memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') - memory=$(echo $memory_new | awk '{print int($0)+1}')G - - cat > hosts/$(hostname) <<-EOF - nr_cpu: $nr_cpu - memory: $memory - hdd_partitions: - ssd_partitions: - EOF -} - -write_shell_profile -write_host -- 2.23.0

1 0

[PATCH lkp-tests 1/2] sbin/install-dependencies: adjust file format
by Wei Jihui 22 Dec '20

22 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/install-dependencies.sh | 39 ++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 20 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 3a206134..6263dc91 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -6,28 +6,27 @@ PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/env.sh . $PROJECT_DIR/sbin/set-env.sh -# choose install function base on -# DISTRIBUTION +# choose install function base on DISTRIBUTION linux_dep() { -source /etc/os-release -case $ID in -ubuntu|debian) - export DEBIAN_FRONTEND=noninteractive - apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client - ;; -openEuler|fedora|rhel|centos) - if type dnf > /dev/null 2>&1; then - dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - else - yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - fi - sudo gem install -f git activesupport rest-client - ;; -*) - echo "$ID not support! please install dependencies manually." - ;; -esac + source /etc/os-release + case $ID in + ubuntu|debian) + export DEBIAN_FRONTEND=noninteractive + apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client + ;; + openEuler|fedora|rhel|centos) + if type dnf > /dev/null 2>&1; then + dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + else + yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + fi + sudo gem install -f git activesupport rest-client + ;; + *) + echo "$ID not support! please install dependencies manually." + ;; + esac } -- 2.23.0

1 0

[PATCH compass-ci] compare_matrixes.rb : reduce compare time
by Li Yuanchao 22 Dec '20

22 Dec '20

When there is multi object group, don't use one object group to compare. Cause one object groups are too many, they spend too much time. And they are not what we want. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/compare_matrixes.rb | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 6e4c2aa..21bd5c9 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -390,7 +390,10 @@ end def compare_group_matrices(group_matrices, suites_hash, options) result_str = '' group_matrices_array = sort_by_matrix_size(group_matrices) + have_multi_member = group_matrices_array[0][1].size > 1 group_matrices_array.each do |matrice_kv| + next if have_multi_member && matrice_kv[1].size < 2 + result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_hash[matrice_kv[0]], options) end result_str -- 2.23.0

1 0

[PATCH lkp-tests] jobs/bisect.yaml: dc-1g--delimiter => dc-1g
by Cao Xueliang 22 Dec '20

22 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- jobs/bisect.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jobs/bisect.yaml b/jobs/bisect.yaml index 5fab30bfc..11ffc7c05 100644 --- a/jobs/bisect.yaml +++ b/jobs/bisect.yaml @@ -1,4 +1,4 @@ suite: bisect docker_image: delimiter:latest -testbox: dc-1g--delimiter +testbox: dc-1g bisect: -- 2.23.0

1 0

[PATCH compass-ci] doc/tutorial.md: fix abnormal display in Markdown
by Lu Kaiyi 22 Dec '20

22 Dec '20

the middle part in * and * is italics for markdown document, if want show '*' in this condition, need use \* escape '*' character. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- doc/tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..19d5741 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -258,7 +258,7 @@ job yaml ��[YAML](http://yaml.org/YAM 4) �� archlinux �� PKGBUILD �� PKGBUILD �� - PKGBUILD �� /srv/git/archlinux/*/*/PKGBUILD�� + PKGBUILD �� /srv/git/archlinux/\*/\*/PKGBUILD�� ### depends -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: auto_tool => assign_account_client
by Cao Xueliang 22 Dec '20

22 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 5e296a6..1d4ed69 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -4,7 +4,7 @@ require 'json' require 'base64' require_relative 'git' require_relative 'mail_client' -require_relative 'auto_tool' +require_relative 'assign_account_client' # compose and send email for bisect result class MailBisectResult -- 2.23.0

1 0

[PATCH compass-ci] web_backend.rb: optimize var name
by Li Yuanchao 22 Dec '20

22 Dec '20

As the implementation of method combine_group_query_data changes, var name suit_list is ok, cause the method return a hash in fact. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 2405107..0ca71e1 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -186,15 +186,15 @@ end def do_get_groups_matrices(must, dimension, total, size, from) job_list = get_dimension_job_list(dimension, must, size, from) - matrices, suites_list = combine_group_query_data(job_list, dimension) + matrices, suites_hash = combine_group_query_data(job_list, dimension) while matrices.empty? from += size break if from > total job_list = get_dimension_job_list(dimension, must, size, from) - matrices, suites_list = combine_group_query_data(job_list, dimension) + matrices, suites_hash = combine_group_query_data(job_list, dimension) end - [matrices, suites_list] + [matrices, suites_hash] end def get_groups_matrices(conditions, dimension, must, size, from) @@ -209,11 +209,11 @@ end def get_compare_body(params) dimension, conditions = get_dimension_conditions(params) must = get_es_must(params) - groups_matrices, suites_list = get_groups_matrices(conditions, dimension, must, COMPARE_RECORDS_NUMBER, 0) + groups_matrices, suites_hash = get_groups_matrices(conditions, dimension, must, COMPARE_RECORDS_NUMBER, 0) if !groups_matrices || groups_matrices.empty? body = 'No Data.' else - body = compare_group_matrices(groups_matrices, suites_list, { no_print: true }) + body = compare_group_matrices(groups_matrices, suites_hash, { no_print: true }) body = 'No Difference.' if !body || body.empty? end return body -- 2.23.0

1 0

[PATCH lkp-tests] sbin/install-dependencies: adjuest file format
by Wei Jihui 22 Dec '20

22 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/install-dependencies.sh | 39 ++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 20 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 3a206134..6263dc91 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -6,28 +6,27 @@ PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/env.sh . $PROJECT_DIR/sbin/set-env.sh -# choose install function base on -# DISTRIBUTION +# choose install function base on DISTRIBUTION linux_dep() { -source /etc/os-release -case $ID in -ubuntu|debian) - export DEBIAN_FRONTEND=noninteractive - apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client - ;; -openEuler|fedora|rhel|centos) - if type dnf > /dev/null 2>&1; then - dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - else - yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - fi - sudo gem install -f git activesupport rest-client - ;; -*) - echo "$ID not support! please install dependencies manually." - ;; -esac + source /etc/os-release + case $ID in + ubuntu|debian) + export DEBIAN_FRONTEND=noninteractive + apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client + ;; + openEuler|fedora|rhel|centos) + if type dnf > /dev/null 2>&1; then + dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + else + yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + fi + sudo gem install -f git activesupport rest-client + ;; + *) + echo "$ID not support! please install dependencies manually." + ;; + esac } -- 2.23.0

1 0

[PATCH lkp-tests] lib/env.sh: fix the check result of is_virt
by Zhang Yu 22 Dec '20

22 Dec '20

Hypervisor is a software that simulates hardware resources and scheduling on a physical machine. so if grep success, return 1 means it's a physical machine Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- lib/env.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/env.sh b/lib/env.sh index 8ac42a836..a1d5e445b 100755 --- a/lib/env.sh +++ b/lib/env.sh @@ -69,7 +69,8 @@ is_virt() # run as root [ -n "$(virt-what)" ] else - grep -q -w hypervisor /proc/cpuinfo + grep -q -w hypervisor /proc/cpuinfo && return 1 + return 0 fi } -- 2.23.0

1 0

[PATCH compass-ci] 3-code/git: add repository version pointing
by Hu Xuejiao 22 Dec '20

22 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 494107e..2159ddc 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -13,11 +13,18 @@ cd /c || exit # modify and manual run for now: [ -d "compass-ci/.git" ] || { git clone https://gitee.com/wu_fengguang/compass-ci.git - ln -s compass-ci cci + + cd /c/compass-ci || exit + [ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" + + ln -s compass-ci /c/cci } [ -d "lkp-tests/.git" ] || { git clone https://gitee.com/wu_fengguang/lkp-tests.git + + cd /c/lkp-tests || exit + [ -n "$lkp_commitid" ] && git reset --hard "$lkp_commitid" } [ -d "upstream-repos/.git" ] || { -- 2.23.0

1 0

[PATCH lkp-tests] tests/deploy-cci: specifying the repository version
by Hu Xuejiao 22 Dec '20

22 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/deploy-cci | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/deploy-cci b/tests/deploy-cci index e296568f..55835544 100755 --- a/tests/deploy-cci +++ b/tests/deploy-cci @@ -19,6 +19,9 @@ mkdir -p /c cd /c || exit git clone https://gitee.com/wu_fengguang/compass-ci.git +cd /c/compass-ci || exit +[ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" + # Need mount tmpfs to fix compass-ci/container/os-nfs/start error # "exportfs: /exports/result does not support NFS export" mount -t tmpfs none /srv -- 2.23.0

1 0

[PATCH compass-ci] lib/git_bisect: return all errors to bisect mail.
by Cao Xueliang 22 Dec '20

22 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index bc1e035..f789a71 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -82,7 +82,6 @@ class GitBisect result = `git -C #{@work_dir} bisect run #{BISECT_RUN_SCRIPT} #{@bad_job_id} #{(a)error_id.inspect} #{@work_dir}` end Utils.create_bisect_log(@work_dir) - FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) puts "\nbisect result: #{result}" analyse_result(result) end @@ -92,10 +91,16 @@ class GitBisect return nil unless temp[0].include?('is the first bad commit') || temp[-1].include?('bisect run success') first_bad_commit = Utils.parse_first_bad_commit(result) - error_messages = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_errorid(@error_id) + bisect_error = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_errorid(@error_id) + all_errors = Utils.obt_errors(@work_dir, first_bad_commit) + first_bad_commit_result_root = Utils.obt_result_root_by_commit(first_bad_commit) + FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) - return Hash['repo' => @upstream_repo, 'commit' => first_bad_commit, - 'error_messages' => error_messages] + return Hash['repo' => @upstream_repo, + 'commit' => first_bad_commit, + 'bisect_error' => bisect_error, + 'all_errors' => all_errors, + 'first_bad_commit_result_root' => first_bad_commit_result_root] end # first search the good commit in db -- 2.23.0

1 0

[PATCH compass-ci] delimiter/utils: add the get all errors function
by Cao Xueliang 22 Dec '20

22 Dec '20

1. fix some rubocop warnings. 2. add get errors by commits - get the pre_commit by find_parent_commit - get cur_id and pre_id by obt_id_by_commit - get errors by get_compare_result 3. add get result root by commit Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 48 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 41803f3..372e821 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -4,10 +4,12 @@ require 'set' require 'json' +require 'yaml' require 'fileutils' require_relative './constants' require_relative '../../lib/sched_client' +require_relative '../../lib/compare_error_messages' require_relative "#{ENV['LKP_SRC']}/lib/monitor" # a utils module for delimiter service @@ -83,9 +85,20 @@ module Utils es = ESQuery.new new_job = es.query_by_id(new_job_id) - return 'bad' if new_job['stats'].key?(error_id) - return 'good' + status = new_job['stats'].key?(error_id) ? 'bad' : 'good' + puts "new_job_id: #{new_job_id}" + puts "upstream_commit: #{job['upstream_commit']}" + record_jobs(new_job_id, job['upstream_commit']) + + return status + end + + def record_jobs(job_id, job_commit) + FileUtils.mkdir_p TMP_RESULT_ROOT unless File.exist? TMP_RESULT_ROOT + commit_jobs = File.join(TMP_RESULT_ROOT, 'commit_jobs') + content = "#{job_commit}: #{job_id}" + File.open(commit_jobs, 'a+') { |f| f.puts content } end def get_account_info @@ -100,6 +113,8 @@ module Utils account_info = get_account_info raise "query #{DELIMITER_ACCONUT} account info failed!" unless account_info + record_jobs(job['id'], job['upstream_commit']) + job['suite'] = 'bisect' job['my_name'] = account_info['my_name'] job['my_email'] = account_info['my_email'] @@ -133,5 +148,34 @@ module Utils log_content.each { |line| f.puts(line) } end end + + def find_parent_commit(git_dir, commit) + response = %x(git -C #{git_dir} rev-parse #{commit}~1) + return response.chomp + end + + def obt_id_by_commit(commit) + puts "#{TMP_RESULT_ROOT}/commit_jobs" + content = YAML.load(File.open("#{TMP_RESULT_ROOT}/commit_jobs")) + puts content + content[commit] + end + + def obt_errors(git_dir, commit) + pre_commit = find_parent_commit(git_dir, commit) + obt_errors_by_commits(commit, pre_commit) + end + + def obt_errors_by_commits(cur_commit, pre_commit) + cur_id = obt_id_by_commit(cur_commit) + pre_id = obt_id_by_commit(pre_commit) + _, errors = get_compare_result(pre_id, cur_id) + return errors + end + + def obt_result_root_by_commit(commit) + id = obt_id_by_commit(commit) + ESQuery.new.query_by_id(id)['result_root'] + end end end -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_client: add send_mail_encode function
by Cao Xueliang 22 Dec '20

22 Dec '20

If some special characters the payload of post, we use base64 encode to pass the payload. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_client.rb | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/mail_client.rb b/lib/mail_client.rb index 1d0938a..b13492f 100644 --- a/lib/mail_client.rb +++ b/lib/mail_client.rb @@ -2,6 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'base64' require 'rest-client' require_relative 'constants.rb' @@ -18,4 +19,9 @@ class MailClient resource = RestClient::Resource.new("http://#{@host}:#{@port}/send_mail_yaml") resource.post(mail_json) end + + def send_mail_encode(mail_data) + resource = RestClient::Resource.new("http://#{@host}:#{@port}/send_mail_encode") + resource.post(Base64.encode64(mail_data)) + end end -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: delete input job key which is path
by Zhang Yu 22 Dec '20

22 Dec '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- lib/job.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job.rb b/lib/job.rb index 2ed06edc6..7553312ed 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -254,6 +254,7 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end + hash.delete_if { |key, _| key.start_with?('#!') } hash.merge!(@overrides) @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end -- 2.23.0

1 0

[PATCH compass-ci] fix 404 error
by Li Ping 22 Dec '20

22 Dec '20

Signed-off-by: Li Ping <1477412247(a)qq.com> --- README.en.md | 4 ++-- README.zh.md | 4 ++-- ...34\272\350\260\203\346\265\213\344\273\273\345\212\241.md" | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/README.en.md b/README.en.md index 5e9386e..c082167 100644 --- a/README.en.md +++ b/README.en.md @@ -34,13 +34,13 @@ If a new error ID is generated, the bisect is automatically triggered to locate 2. Run the **git push** command to update the repository and automatically trigger the test. -3. You can [view](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/brow… and [compare](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/c… test results on the web page: https://compass-ci.openeuler.org/jobs. +3. You can [view](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/brow… and [compare](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/c… test results on the web page: https://compass-ci.openeuler.org/jobs **Manually Submitting a Test Task** 1. [Install the Compass-CI client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E…. 2. [Compile test cases and manually submit test tasks](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcas…. -3. You can [view](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/brow… and [compare](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/c… test results on the web page: https://compass-ci.openeuler.org/jobs. +3. You can [view](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/brow… and [compare](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/c… test results on the web page: https://compass-ci.openeuler.org/jobs **Logging in to the Test Environment** diff --git a/README.zh.md b/README.zh.md index 0616bec..c2140b0 100644 --- a/README.zh.md +++ b/README.zh.md @@ -37,13 +37,13 @@ Compass-CI 监控很多开源软件 git repos，一旦检测到代码更新， 2. 执行 git push 命令更新仓库，自动触发测试。 -3. 在网页中[查看](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/b… web: https://compass-ci.openeuler.org/jobs。 +3. 在网页中[查看](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/b… web: https://compass-ci.openeuler.org/jobs **手动提交测试任务** 1. [安装 Compass-CI 客户端](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E6%9… 2. [编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testca…, [手动提交测试任务](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… -3. 在网页中[查看](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/b… web: https://compass-ci.openeuler.org/jobs。 +3. 在网页中[查看](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/b… web: https://compass-ci.openeuler.org/jobs **登录测试环境** diff --git "a/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" index b59431e..7a54505 100644 --- "a/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" +++ "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" @@ -3,8 +3,8 @@ # 1. 前提条件： ## 请先学习: * [配置个人邮箱](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/ap… -* [如何申请测试机](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/如… -* [submit命令详解](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… +* [如何申请测试机](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/b… +* [submit命令详解](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… # 2. 操作方法 -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: optimize mail bisect content
by Cao Xueliang 22 Dec '20

22 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 63 ++++++++++++++++++++++++++------------- 1 file changed, 42 insertions(+), 21 deletions(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 090855e..5e296a6 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -1,48 +1,69 @@ # frozen_string_literal: true -require_relative 'mail_client' -require_relative 'git' require 'json' +require 'base64' +require_relative 'git' +require_relative 'mail_client' +require_relative 'auto_tool' # compose and send email for bisect result class MailBisectResult def initialize(bisect_info) - @error_messages = bisect_info['error_messages'] + @bisect_error = bisect_info['bisect_error'] + @all_errors = bisect_info['all_errors'] @repo = bisect_info['repo'] @commit_id = bisect_info['commit'] + @first_bad_commit_result_root = bisect_info['first_bad_commit_result_root'] @git_commit = GitCommit.new(@repo, @commit_id) + # now send mail to review + @to = 'caoxl(a)crystal.ci, caoxl78320(a)163.com, huming15(a)163.com' end def create_send_email - compose_mail - send_mail + send_report_mail(compose_mail) + send_account_mail end def compose_mail - subject = "[Compass-CI][#{(a)repo.split('/')[1]}]: #{(a)error_messages[0].split("\n")[0]}" - job_url = ENV['result_root'] ? "job url: http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}#{ENV['result_root']}\n" : '' - body = <<~BODY - Hi #{(a)git_commit.author_name}, + subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" + prefix_srv = "http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}" + bisect_job_url = ENV['result_root'] ? "bisect job info: #{prefix_srv}#{ENV['result_root']}\n" : '' + first_bad_commit_job_url = "first bad commit job info: #{prefix_srv}#{@first_bad_commit_result_root}\n" - git url: #{(a)git_commit.url} - git commit: #{@commit_id[0..11]} ("#{(a)git_commit.subject}") + data = <<~BODY + To: #{@to} + Subject: #{subject} - gcc version: 7.3.0 + Hi #{(a)git_commit.author_name}, - error_messages: - #{(a)error_messages.join("\n")} + url: #{(a)git_commit.url} + commit: #{@commit_id} ("#{(a)git_commit.subject}") + compiler: gcc (GCC) 7.3.0 - #{job_url} + all errors/warnings: + #{@all_errors} + + #{bisect_job_url} + #{first_bad_commit_job_url} Regards, Compass CI team BODY - to = 'caoxl(a)crystal.ci' - @hash = { 'to' => to, 'body' => body, 'subject' => subject } - # @hash = { 'to' => @git_commit.author_email, 'body' => body, 'subject' => subject } + + return data end - def send_mail - json = @hash.to_json - MailClient.new.send_mail(json) + def send_report_mail(mail_data) + MailClient.new.send_mail_encode(mail_data) + end + + def send_account_mail + user_info = { + 'my_email' => @to, + 'my_name' => @git_commit.author_name, + 'my_commit_url' => "#{@git_commit.url}/commit/#{@commit_id}" + } + + apply_account = AutoAssignAccount.new(user_info) + apply_account.send_account end end -- 2.23.0

1 0