December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] tests/build-pkg: add config file support
by Wang Yong 21 Dec '20

21 Dec '20

[Why] config file in some software need storage and recall now we can add config value in yaml and cp ${config} to ${cci_dir}/${benchmark}/ in PKGBUILD so that config file can be uploaded correctly $ cat test.yaml ... config: ${config} ... Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 39 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index addff9f7b..0a143813f 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -5,6 +5,7 @@ # - git_tag # - pkg_ver # upstream_url +# - config # We'll use the below global env vars, but not treat them as parameter to avoid # duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ @@ -20,6 +21,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/distro/common +. $LKP_SRC/lib/http.sh check_vars() { @@ -31,6 +33,29 @@ check_vars() [ -n "$os_mount" ] || die "os_mount is empty" } +check_config_var() +{ + [ -n "$config" ] || return +} + +get_config() +{ + local cci_http_host=${INITRD_HTTP_HOST} + local cci_http_port=${INITRD_HTTP_PORT:-8800} + export cci_dir=/cci/config + local remote_dir="http://${cci_http_host}:${cci_http_port}${cci_dir}/${benchmark}" + + mkdir -p ${cci_dir} + + check_config_var + http_get_file "${remote_dir}/${config}" "${cci_dir}/${benchmark}/${config}" +} + +upload_config() +{ + upload_to_target_dir "${cci_dir}/${benchmark}" +} + mount_dest() { # the same image is mounted to cifs and nfs, the generated cgz files @@ -71,6 +96,13 @@ create_cgz_name() [ -n "$git_tag" ] && { cgz_name="${cgz_path}/${git_tag}.cgz" + # download PKGBUILD-$tag + benchmark="${upstream_repo##*/}" + export benchmark="${benchmark%.git}" + [ -n "$PKGBUILD_TAG" ] && { + get_pkgfile "pkg/$benchmark" "PKGBUILD-$PKGBUILD_TAG" + [ -f "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG not exist" + [ -s "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG is empty" } [ -n "$pkg_ver" ] && { @@ -131,7 +163,11 @@ build_source_pkg() PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf - [ "$?" == 0 ] || exit 1 + local build_state="$?" + + upload_config + + [ "${build_state}" == "0" ] || exit 1 create_softlink upload_to_target_dir ${PKG_MNT}/${pack_to}/${pkgname} } @@ -139,4 +175,5 @@ build_source_pkg() check_vars mount_dest request_pkg +get_config build_source_pkg -- 2.23.0

2 2

[PATCH compass-ci] lib/matrix2.rb: discard the no $suite related job
by Lu Kaiyi 19 Dec '20

19 Dec '20

if job has suite, but no $suite related stats metric, then discard this job when create matrix. --- lib/matrix2.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2507b77..f1b2c78 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,10 +76,15 @@ def create_matrix(job_list) matrix = {} suites = [] job_list.each do |job| - suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats + if job['suite'] + next unless stats.keys.any? { |stat| stat.start_with?(job['suite']) } + + suites << job['suite'] + end + stats.each do |key, value| matrix[key] = [] unless matrix[key] matrix[key] << value -- 2.23.0

2 2

[PATCH compass-ci 2/2] lib/matrix2.rb remove the *timestamp* form matrix
by Lu Weitao 19 Dec '20

19 Dec '20

[Why] In compare results: the value of *timestamp*(metric) doesn't mean the number of occurrences, it may confuse user like bellow: openeuler centos archLinux metric -------------------- ------------------------------ ------------------------------ ------------------------------ fails:runs change fails:runs change fails:runs | | | | | 3585:13 +6913.7% 344:1 -2.6e+4% 11:1 dmesg.timestamp:last 2350:13 +1.6e+4% 344:1 -1.7e+4% 10:1 kmsg.timestamp:last 2:13 -15.4% 0:1 +84.6% 1:1 last_state.exit_fail [How] remove the metric that include "timestamp" from matrix Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index c712e32..2769f7c 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -87,6 +87,8 @@ def create_matrix(job_list) next unless stats stats.each do |key, value| + next if key.include?('timestamp') + matrix[key] = [] unless matrix[key] matrix[key] << value end -- 2.23.0

1 0

[PATCH compass-ci] lib/commit_status.rb: query latest job_id by commit_id
by Lu Kaiyi 19 Dec '20

19 Dec '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/commit_status.rb | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/lib/commit_status.rb b/lib/commit_status.rb index 81aeb29..f1cbfbc 100644 --- a/lib/commit_status.rb +++ b/lib/commit_status.rb @@ -25,6 +25,17 @@ def query_latest_good_commit(job, error_id) return latest_good_commit end +def query_latest_job_id(commit_id) + items = { + 'upstream_commit' => commit_id + } + jobs_list = query_jobs_from_es(items) + return nil if jobs_list.empty? + + jobs_list.sort_by! { |job| job['end_time'] } + jobs_list.last['id'] +end + def query_jobs_from_es(items) es = ESQuery.new(ES_HOST, ES_PORT) result = es.multi_field_query items -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: check load host config from job
by Wei Jihui 19 Dec '20

19 Dec '20

@defaults will merge to @job, and set @default empty. if load host config from @job, so do not need to load host in advance. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 42b008c7..2ed06edc 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -317,17 +317,6 @@ class Job spec_file_context.each_key { |k| @job.delete k } end - def check_load_hosts_config(defaults_config) - return if @load_hosts_done - - if defaults_config.include?('testbox') - @job['tbox_group'] = tbox_group(defaults_config['testbox']) - elsif defaults_config.include?('tbox_group') - @job['tbox_group'] = defaults_config['tbox_group'] - end - return @job.include?('tbox_group') - end - def load_hosts_config return if @job.include?(:no_defaults) return if @load_hosts_done @@ -354,7 +343,7 @@ class Job def get_lab_hosts_file lab_hosts_file_name = @job['testbox'] - lab_hosts_file = "#{ENV['CCI_REPOS']}/lab-#{@defaults['lab']}/hosts/#{lab_hosts_file_name}" + lab_hosts_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{lab_hosts_file_name}" if File.file?(lab_hosts_file) lab_hosts_file @@ -409,7 +398,6 @@ class Job @defaults[source_file_symkey(file)] = nil revise_hash(@defaults, defaults, true) @defaults.merge!(@overrides) - load_hosts_config if check_load_hosts_config(@defaults) end @file_loaded[file] = true true -- 2.23.0

1 0

[PATCH compass-ci] scheduler-https/start: fix missing env key
by Wu Zhende 19 Dec '20

19 Dec '20

Some key don't define in scheduler-https. Synchronize with scheduler. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler-https/start | 34 +++++++++++++++++++++++---------- 1 file changed, 24 insertions(+), 10 deletions(-) diff --git a/container/scheduler-https/start b/container/scheduler-https/start index 9898c1a..98940d7 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -28,17 +28,32 @@ names = Set.new %w[ INITRD_HTTP_PORT TASKQUEUE_PORT TASKQUEUE_HOST + GIT_SERVER + MASTER_FLUENTD_HOST + MASTER_FLUENTD_PORT + SSHR_PORT + SSHR_PORT_BASE + SSHR_PORT_LEN + lab ] defaults = relevant_defaults(names) +defaults['SSHR_PORT'] ||= 5050 +defaults['SSHR_PORT_BASE'] ||= 50000 +defaults['SSHR_PORT_LEN'] ||= 2000 +defaults['SCHED_PORT'] ||= '3000' +defaults['SCHED_HOST'] ||= '172.17.0.1' +defaults['MASTER_FLUENTD_PORT'] ||= '24224' +defaults['SCHED_HTTPS_PORT'] ||= 11306 +defaults['SCHED_HTTPS_HOST'] ||= 'api.compass-ci.openeuler.org' env = docker_env(defaults) -DEFAULT_LKP = '/c/lkp-tests' -DEFAULT_CCI = '/c/compass-ci' -SCHED_PORT = defaults['SCHED_PORT'] || '3000' -SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' -SCHED_HTTPS_PORT = defaults['SCHED_HTTPS_PORT'] || 11306 -SCHED_HTTPS_HOST = defaults['SCHED_HTTPS_HOST'] || 'api.compass-ci.openeuler.org' +CCI_REPOS = ENV['CCI_REPOS'] || '/c' +DEFAULT_LKP = "#{CCI_REPOS}/lkp-tests" +DEFAULT_CCI = "#{CCI_REPOS}/compass-ci" +LAB = defaults['lab'] || 'z9' +SCHED_HTTPS_PORT = defaults['SCHED_HTTPS_PORT'] +SCHED_HTTPS_HOST = defaults['SCHED_HTTPS_HOST'] docker_rm "s001-alpine-#{SCHED_HTTPS_PORT}" @@ -48,14 +63,13 @@ cmd = %W[ --name s001-alpine-#{SCHED_HTTPS_PORT} -d -u 1090:1090 - -p #{SCHED_HTTPS_PORT}:#{SCHED_PORT} + -p #{SCHED_HTTPS_PORT}:#{defaults['SCHED_PORT']} ] + env + %W[ -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} - -e SCHED_HOST=#{SCHED_HOST} - -e SCHED_PORT=#{SCHED_PORT} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} -v /srv/cci/scheduler/alpine:/srv/cci/scheduler -v /etc/localtime:/etc/localtime:ro -v /srv/result:/srv/result @@ -65,7 +79,7 @@ cmd = %W[ --log-opt mode=non-blocking --log-opt max-buffer-size=4m --log-opt tag=scheduler-#{SCHED_HTTPS_PORT} - sch-ruby-a:v0.00d-#{SCHED_PORT} + sch-ruby-a:v0.00d-#{defaults['SCHED_PORT']} ] cmd += ['sh', '-c', 'umask 002 && ./scheduler --ssl --ssl-key-file /ca/server.key --ssl-cert-file /ca/server.crt'] -- 2.23.0

1 0

[PATCH lkp-tests 4/4] install.sh: grant execute permission
by Wei Jihui 19 Dec '20

19 Dec '20

cd lkp-test ./install.sh && source ~/.${SHELL##*/}rc Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- install.sh | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 install.sh diff --git a/install.sh b/install.sh old mode 100644 new mode 100755 -- 2.23.0

1 0

[PATCH lkp-tests 3/4] install.sh: refactor install flow
by Wei Jihui 19 Dec '20

19 Dec '20

1. when exec install.sh, no $LKP_SRC 2. first enable $LKP_SRC env 3. mv sbin/install-dependencies.sh to install.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- Makefile | 2 -- install.sh | 10 +++++++--- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/Makefile b/Makefile index 0b2d09a2..ae062973 100644 --- a/Makefile +++ b/Makefile @@ -14,8 +14,6 @@ subsystem: install: mkdir -p $(TARGET_DIR_BIN) ln -sf $(shell pwd)/bin/lkp $(TARGET_DIR_BIN)/lkp - bash sbin/set-env.sh - bash sbin/install-dependencies.sh .PHONY: doc doc: diff --git a/install.sh b/install.sh index cea1654f..2bd17d54 100644 --- a/install.sh +++ b/install.sh @@ -1,7 +1,11 @@ #!/bin/bash -. $LKP_SRC/lib/env.sh - -make +. lib/env.sh +write_host +write_shell_profile "export LKP_SRC=$PWD" +write_shell_profile "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" source $(shell_profile) + +. sbin/install-dependencies.sh +make install -- 2.23.0

1 0

[PATCH lkp-tests 1/4] bin/install-dependencies.sh: refactor file
by Wei Jihui 19 Dec '20

19 Dec '20

all dependent functions is move to lib/env.sh, sbin/set-env.sh in not required. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/install-dependencies.sh | 40 +++++++++++++++++------------------- 1 file changed, 19 insertions(+), 21 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 7605a29f..b4d3ad22 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -1,30 +1,28 @@ #!/bin/bash . $LKP_SRC/lib/env.sh -. $LKP_SRC/sbin/set-env.sh -# choose install function base on -# DISTRIBUTION +# choose install function base on DISTRIBUTION linux_dep() { -source /etc/os-release -case $ID in -ubuntu|debian) - export DEBIAN_FRONTEND=noninteractive - apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client - ;; -openEuler|fedora|rhel|centos) - if type dnf > /dev/null 2>&1; then - dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - else - yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - fi - sudo gem install -f git activesupport rest-client - ;; -*) - echo "$ID not support! please install dependencies manually." - ;; -esac + source /etc/os-release + case $ID in + ubuntu|debian) + export DEBIAN_FRONTEND=noninteractive + apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client + ;; + openEuler|fedora|rhel|centos) + if type dnf > /dev/null 2>&1; then + dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + else + yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + fi + sudo gem install -f git activesupport rest-client + ;; + *) + echo "$ID not support! please install dependencies manually." + ;; + esac } -- 2.23.0

1 0

[PATCH compass-ci] result-webdav/nginx.conf: create soft connection after package upload
by Cui Yili 19 Dec '20

19 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 8b54314..f077fca 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -37,6 +37,25 @@ http { end) return resultstrlist end + function symbolic_link(path, upload_file) + level_dir = split(path, "/") + if level_dir[1] ~= "initrd" or upload_file == nil then + return + end + if level_dir[2] == "pkg" or level_dir[2] == "build-pkg" then + link_name = "latest" + else + tmp_name = string.match(upload_file, "(.*)_%d+%.cgz") + if not tmp_name then + link_name = tmp_name + else + return + end + end + link_name = link_name..".cgz" + root_path = "/srv/"..path + io.popen("cd "..root_path.."&& ln -sf "..upload_file.." "..link_name) + end redis_host = os.getenv("REDIS_HOST") redis_port = os.getenv("REDIS_PORT") @@ -65,11 +84,13 @@ http { dirs = split(res, ",") for i = 1, #dirs do if string.find(path, dirs[i], 1, true) then + symbolic_link(path, upload_file) return end end else if string.find(path, res, 1, true) then + symbolic_link(path, upload_file) return end end -- 2.23.0

1 0