- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 2/2] lib/matrix2.rb remove the *timestamp* form matrix
by Lu Weitao 19 Dec '20

19 Dec '20

[Why] In compare results: the value of *timestamp*(metric) doesn't mean the number of occurrences, it may confuse user like bellow: openeuler centos archLinux metric -------------------- ------------------------------ ------------------------------ ------------------------------ fails:runs change fails:runs change fails:runs | | | | | 3585:13 +6913.7% 344:1 -2.6e+4% 11:1 dmesg.timestamp:last 2350:13 +1.6e+4% 344:1 -1.7e+4% 10:1 kmsg.timestamp:last 2:13 -15.4% 0:1 +84.6% 1:1 last_state.exit_fail [How] remove the metric that include "timestamp" from matrix Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index c712e32..2769f7c 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -87,6 +87,8 @@ def create_matrix(job_list) next unless stats stats.each do |key, value| + next if key.include?('timestamp') + matrix[key] = [] unless matrix[key] matrix[key] << value end -- 2.23.0

1 0

[PATCH compass-ci] lib/commit_status.rb: query latest job_id by commit_id
by Lu Kaiyi 19 Dec '20

19 Dec '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/commit_status.rb | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/lib/commit_status.rb b/lib/commit_status.rb index 81aeb29..f1cbfbc 100644 --- a/lib/commit_status.rb +++ b/lib/commit_status.rb @@ -25,6 +25,17 @@ def query_latest_good_commit(job, error_id) return latest_good_commit end +def query_latest_job_id(commit_id) + items = { + 'upstream_commit' => commit_id + } + jobs_list = query_jobs_from_es(items) + return nil if jobs_list.empty? + + jobs_list.sort_by! { |job| job['end_time'] } + jobs_list.last['id'] +end + def query_jobs_from_es(items) es = ESQuery.new(ES_HOST, ES_PORT) result = es.multi_field_query items -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: check load host config from job
by Wei Jihui 19 Dec '20

19 Dec '20

@defaults will merge to @job, and set @default empty. if load host config from @job, so do not need to load host in advance. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 42b008c7..2ed06edc 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -317,17 +317,6 @@ class Job spec_file_context.each_key { |k| @job.delete k } end - def check_load_hosts_config(defaults_config) - return if @load_hosts_done - - if defaults_config.include?('testbox') - @job['tbox_group'] = tbox_group(defaults_config['testbox']) - elsif defaults_config.include?('tbox_group') - @job['tbox_group'] = defaults_config['tbox_group'] - end - return @job.include?('tbox_group') - end - def load_hosts_config return if @job.include?(:no_defaults) return if @load_hosts_done @@ -354,7 +343,7 @@ class Job def get_lab_hosts_file lab_hosts_file_name = @job['testbox'] - lab_hosts_file = "#{ENV['CCI_REPOS']}/lab-#{@defaults['lab']}/hosts/#{lab_hosts_file_name}" + lab_hosts_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{lab_hosts_file_name}" if File.file?(lab_hosts_file) lab_hosts_file @@ -409,7 +398,6 @@ class Job @defaults[source_file_symkey(file)] = nil revise_hash(@defaults, defaults, true) @defaults.merge!(@overrides) - load_hosts_config if check_load_hosts_config(@defaults) end @file_loaded[file] = true true -- 2.23.0

1 0

[PATCH compass-ci] scheduler-https/start: fix missing env key
by Wu Zhende 19 Dec '20

19 Dec '20

Some key don't define in scheduler-https. Synchronize with scheduler. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler-https/start | 34 +++++++++++++++++++++++---------- 1 file changed, 24 insertions(+), 10 deletions(-) diff --git a/container/scheduler-https/start b/container/scheduler-https/start index 9898c1a..98940d7 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -28,17 +28,32 @@ names = Set.new %w[ INITRD_HTTP_PORT TASKQUEUE_PORT TASKQUEUE_HOST + GIT_SERVER + MASTER_FLUENTD_HOST + MASTER_FLUENTD_PORT + SSHR_PORT + SSHR_PORT_BASE + SSHR_PORT_LEN + lab ] defaults = relevant_defaults(names) +defaults['SSHR_PORT'] ||= 5050 +defaults['SSHR_PORT_BASE'] ||= 50000 +defaults['SSHR_PORT_LEN'] ||= 2000 +defaults['SCHED_PORT'] ||= '3000' +defaults['SCHED_HOST'] ||= '172.17.0.1' +defaults['MASTER_FLUENTD_PORT'] ||= '24224' +defaults['SCHED_HTTPS_PORT'] ||= 11306 +defaults['SCHED_HTTPS_HOST'] ||= 'api.compass-ci.openeuler.org' env = docker_env(defaults) -DEFAULT_LKP = '/c/lkp-tests' -DEFAULT_CCI = '/c/compass-ci' -SCHED_PORT = defaults['SCHED_PORT'] || '3000' -SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' -SCHED_HTTPS_PORT = defaults['SCHED_HTTPS_PORT'] || 11306 -SCHED_HTTPS_HOST = defaults['SCHED_HTTPS_HOST'] || 'api.compass-ci.openeuler.org' +CCI_REPOS = ENV['CCI_REPOS'] || '/c' +DEFAULT_LKP = "#{CCI_REPOS}/lkp-tests" +DEFAULT_CCI = "#{CCI_REPOS}/compass-ci" +LAB = defaults['lab'] || 'z9' +SCHED_HTTPS_PORT = defaults['SCHED_HTTPS_PORT'] +SCHED_HTTPS_HOST = defaults['SCHED_HTTPS_HOST'] docker_rm "s001-alpine-#{SCHED_HTTPS_PORT}" @@ -48,14 +63,13 @@ cmd = %W[ --name s001-alpine-#{SCHED_HTTPS_PORT} -d -u 1090:1090 - -p #{SCHED_HTTPS_PORT}:#{SCHED_PORT} + -p #{SCHED_HTTPS_PORT}:#{defaults['SCHED_PORT']} ] + env + %W[ -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} - -e SCHED_HOST=#{SCHED_HOST} - -e SCHED_PORT=#{SCHED_PORT} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} -v /srv/cci/scheduler/alpine:/srv/cci/scheduler -v /etc/localtime:/etc/localtime:ro -v /srv/result:/srv/result @@ -65,7 +79,7 @@ cmd = %W[ --log-opt mode=non-blocking --log-opt max-buffer-size=4m --log-opt tag=scheduler-#{SCHED_HTTPS_PORT} - sch-ruby-a:v0.00d-#{SCHED_PORT} + sch-ruby-a:v0.00d-#{defaults['SCHED_PORT']} ] cmd += ['sh', '-c', 'umask 002 && ./scheduler --ssl --ssl-key-file /ca/server.key --ssl-cert-file /ca/server.crt'] -- 2.23.0

1 0

[PATCH lkp-tests 4/4] install.sh: grant execute permission
by Wei Jihui 19 Dec '20

19 Dec '20

cd lkp-test ./install.sh && source ~/.${SHELL##*/}rc Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- install.sh | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 install.sh diff --git a/install.sh b/install.sh old mode 100644 new mode 100755 -- 2.23.0

1 0

[PATCH lkp-tests 3/4] install.sh: refactor install flow
by Wei Jihui 19 Dec '20

19 Dec '20

1. when exec install.sh, no $LKP_SRC 2. first enable $LKP_SRC env 3. mv sbin/install-dependencies.sh to install.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- Makefile | 2 -- install.sh | 10 +++++++--- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/Makefile b/Makefile index 0b2d09a2..ae062973 100644 --- a/Makefile +++ b/Makefile @@ -14,8 +14,6 @@ subsystem: install: mkdir -p $(TARGET_DIR_BIN) ln -sf $(shell pwd)/bin/lkp $(TARGET_DIR_BIN)/lkp - bash sbin/set-env.sh - bash sbin/install-dependencies.sh .PHONY: doc doc: diff --git a/install.sh b/install.sh index cea1654f..2bd17d54 100644 --- a/install.sh +++ b/install.sh @@ -1,7 +1,11 @@ #!/bin/bash -. $LKP_SRC/lib/env.sh - -make +. lib/env.sh +write_host +write_shell_profile "export LKP_SRC=$PWD" +write_shell_profile "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" source $(shell_profile) + +. sbin/install-dependencies.sh +make install -- 2.23.0

1 0

[PATCH lkp-tests 1/4] bin/install-dependencies.sh: refactor file
by Wei Jihui 19 Dec '20

19 Dec '20

all dependent functions is move to lib/env.sh, sbin/set-env.sh in not required. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/install-dependencies.sh | 40 +++++++++++++++++------------------- 1 file changed, 19 insertions(+), 21 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 7605a29f..b4d3ad22 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -1,30 +1,28 @@ #!/bin/bash . $LKP_SRC/lib/env.sh -. $LKP_SRC/sbin/set-env.sh -# choose install function base on -# DISTRIBUTION +# choose install function base on DISTRIBUTION linux_dep() { -source /etc/os-release -case $ID in -ubuntu|debian) - export DEBIAN_FRONTEND=noninteractive - apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client - ;; -openEuler|fedora|rhel|centos) - if type dnf > /dev/null 2>&1; then - dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - else - yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - fi - sudo gem install -f git activesupport rest-client - ;; -*) - echo "$ID not support! please install dependencies manually." - ;; -esac + source /etc/os-release + case $ID in + ubuntu|debian) + export DEBIAN_FRONTEND=noninteractive + apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client + ;; + openEuler|fedora|rhel|centos) + if type dnf > /dev/null 2>&1; then + dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + else + yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + fi + sudo gem install -f git activesupport rest-client + ;; + *) + echo "$ID not support! please install dependencies manually." + ;; + esac } -- 2.23.0

1 0

[PATCH compass-ci] result-webdav/nginx.conf: create soft connection after package upload
by Cui Yili 19 Dec '20

19 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 8b54314..f077fca 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -37,6 +37,25 @@ http { end) return resultstrlist end + function symbolic_link(path, upload_file) + level_dir = split(path, "/") + if level_dir[1] ~= "initrd" or upload_file == nil then + return + end + if level_dir[2] == "pkg" or level_dir[2] == "build-pkg" then + link_name = "latest" + else + tmp_name = string.match(upload_file, "(.*)_%d+%.cgz") + if not tmp_name then + link_name = tmp_name + else + return + end + end + link_name = link_name..".cgz" + root_path = "/srv/"..path + io.popen("cd "..root_path.."&& ln -sf "..upload_file.." "..link_name) + end redis_host = os.getenv("REDIS_HOST") redis_port = os.getenv("REDIS_PORT") @@ -65,11 +84,13 @@ http { dirs = split(res, ",") for i = 1, #dirs do if string.find(path, dirs[i], 1, true) then + symbolic_link(path, upload_file) return end end else if string.find(path, res, 1, true) then + symbolic_link(path, upload_file) return end end -- 2.23.0

1 0

[PATCH compass-ci] lib/error_messages.rb: fix error when file is not existed
by Lin Jiaxin 19 Dec '20

19 Dec '20

Traceback (most recent call last): 4: from compare_error_messages.rb:63:in `<main>' 3: from compare_error_messages.rb:20:in `get_compare_result' 2: from /home/linjiaxin/compass-ci/lib/error_messages.rb:33:in `obtain_error_messages' 1: from /home/linjiaxin/compass-ci/lib/error_messages.rb:33:in `open' /home/linjiaxin/compass-ci/lib/error_messages.rb:33:in `initialize': No such file or directory @ rb_sysopen - /srv/result/build-pkg/2020-12-16/vm-2p32g.linjiaxin/openeuler-20.03-aarch64/archlinux-community--fcgi-trunk-HEAD/crystal.193087/build-pkg (Errno::ENOENT) Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/lib/error_messages.rb b/lib/error_messages.rb index 5bbc329..61f8070 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -30,12 +30,14 @@ class ErrorMessages end def obtain_error_messages - File.open((a)file).each_line do |line| - next if extract_error_message(line) - next unless @in_stderr - next unless @error_message['error_line'] =~ /error|warning/i + if FileTest.exist?(@file) + File.open((a)file).each_line do |line| + next if extract_error_message(line) + next unless @in_stderr + next unless @error_message['error_line'] =~ /error|warning/i - @error_messages[@error_message['error_line']] << @error_message['error_line'] + @error_message['error_details'] + @error_messages[@error_message['error_line']] << @error_message['error_line'] + @error_message['error_details'] + end end return @error_messages end -- 2.23.0

1 0

[PATCH v2 compass-ci] lib: add compare_error_messages.rb
by Lin Jiaxin 19 Dec '20

19 Dec '20

Provide formatted error messages and new error number for git bisect. formatted error messages example: previous error: fcgiapp.c:1717:22: warning: comparison of integer expressions of different signedness: ‘int’ and ‘long unsigned int’ [-Wsign-compare] 1717 | if(headerLen < sizeof(header)) { | ^ new error: >> libtool: warning: relinking 'libfcgi++.la' Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 59 +++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 lib/compare_error_messages.rb diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb new file mode 100644 index 0000000..0b51fc3 --- /dev/null +++ b/lib/compare_error_messages.rb @@ -0,0 +1,59 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'terminal-table' +require_relative 'es_query' +require_relative 'error_messages' +require_relative "#{ENV['LKP_SRC']}/lib/common" + +def get_compare_result(previous_job_id, later_job_id) + es = ESQuery.new + + previous_error_ids = es.query_by_id(previous_job_id)['error_ids'] + later_es_result = es.query_by_id(later_job_id) + later_error_ids = later_es_result['error_ids'] + later_result_file = File.join('/srv', later_es_result['result_root'], 'build-pkg') + + new_error_ids = get_new_error_ids(later_error_ids, previous_error_ids) + error_messages = ErrorMessages.new(later_result_file).obtain_error_messages + new_error_number, formatted_error_messages = format_error_messages(error_messages, new_error_ids) + + # new error use ">>" as an identifier in the error header + return new_error_number, formatted_error_messages +end + +def get_new_error_ids(later_error_ids, previous_error_ids) + return nil_to_empty_array(later_error_ids) - nil_to_empty_array(previous_error_ids) +end + +def nil_to_empty_array(array) + array = [] if array.nil? + return array +end + +def format_error_messages(error_messages, new_error_ids) + formatted_error_messages = Terminal::Table.new + formatted_error_messages.style = { border_x: '', border_y: '', border_i: '', padding_left: 0 } + new_error_number = 0 + + error_messages.each do |k, v| + if new_error_ids.include?('build-pkg.' + build_pkg_error_id(k)) + new_error_number += 1 + formatted_error_messages = add_sign(formatted_error_messages, '>>', v) + else + formatted_error_messages = add_sign(formatted_error_messages, ' ', v) + end + end + + return new_error_number, formatted_error_messages.to_s +end + +def add_sign(formatted_error_messages, sign, set) + set.each do |value| + formatted_error_messages.add_row([sign, value]) + end + + return formatted_error_messages +end -- 2.23.0

1 0