March 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] lib/git_mirror.rb: change the way to calculate priority for repo
by Li Yuanchao 23 Mar '21

23 Mar '21

Change the way to calculate priority for repo, to make active repo be updated more times and more effective. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 36 +++++++++++++++++++++++++++--------- 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 50a6c75..5ba9154 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -115,7 +115,6 @@ class MirrorMain def initialize @feedback_queue = Queue.new @fork_stat = {} - @priority = 0 @priority_queue = PriorityQueue.new @git_info = {} @defaults = {} @@ -221,11 +220,9 @@ class MirrorMain def push_git_queue return if @git_queue.size >= 1 - fork_key = @priority_queue.delete_min_return_key + fork_key, old_pri = @priority_queue.delete_min do_push(fork_key) - priority_set = @priority > @fork_stat[fork_key][:priority] ? (@priority - @fork_stat[fork_key][:priority]) : 1 - @priority_queue.push fork_key, priority_set - @priority += 1 + @priority_queue.push fork_key, get_repo_priority(fork_key, old_pri) end def main_loop @@ -251,8 +248,7 @@ class MirrorMain @git_info[git_repo] = merge_defaults(git_repo, @git_info[git_repo], belong) fork_stat_init(git_repo) - @priority_queue.push git_repo, @priority - @priority += 1 + @priority_queue.push git_repo, get_repo_priority(git_repo, 0) end def compare_refs(cur_refs, old_refs) @@ -388,8 +384,7 @@ class MirrorMain @git_info[git_repo] = { 'url' => url, 'git_repo' => git_repo, 'is_submodule' => true, 'belong' => belong } fork_stat_init(git_repo) - @priority_queue.push git_repo, @priority - @priority += 1 + @priority_queue.push git_repo, get_repo_priority(git_repo, 0) end end @@ -474,6 +469,8 @@ end # main thread class MirrorMain + WEEK_SECONDS = 604800 + def merge_defaults(object_key, object, belong) return object if object_key == belong @@ -541,4 +538,25 @@ class MirrorMain return true end + + def get_repo_priority(git_repo, old_pri) + old_pri ||= 0 + mirror_dir = "/srv/git/#{@git_info[git_repo]['belong']}/#{git_repo}" + mirror_dir = "#{mirror_dir}.git" unless @git_info[git_repo]['is_submodule'] + + return old_pri + Math.cbrt(WEEK_SECONDS) unless File.directory?(mirror_dir) + + return cal_priority(mirror_dir, old_pri) + end + + def cal_priority(mirror_dir, old_pri) + last_commit_time = %x(git -C #{mirror_dir} log --pretty=format:"%ct" -1 2>/dev/null).to_i + return old_pri + Math.cbrt(WEEK_SECONDS) if last_commit_time.zero? + + t = Time.now.to_i + interval = t - last_commit_time + return old_pri + Math.cbrt(WEEK_SECONDS) if interval <= 0 + + return old_pri + Math.cbrt(interval) + end end -- 2.23.0

1 0

[PATCH compass-ci 1/2] service/sshr: change port
by Wu Zhende 23 Mar '21

23 Mar '21

Restarting the sshr service will disconnect the existing sshr connection and the machine that has been applied for can't be accessed. Therefore, we can't restart the sshr service. Instead, we start a new sshr service and use a new port. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler-https/start | 2 +- container/scheduler/start | 4 ++-- container/ssh-r/start | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/container/scheduler-https/start b/container/scheduler-https/start index ee93e79..4ec84a2 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -38,7 +38,7 @@ names = Set.new %w[ ] defaults = relevant_defaults(names) -defaults['SSHR_PORT'] ||= 5050 +defaults['SSHR_PORT'] ||= 5051 defaults['SSHR_PORT_BASE'] ||= 21000 defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' diff --git a/container/scheduler/start b/container/scheduler/start index e655dd4..8d800f9 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -31,8 +31,8 @@ names = Set.new %w[ ] defaults = relevant_defaults(names) -defaults['SSHR_PORT'] ||= 5050 -defaults['SSHR_PORT_BASE'] ||= 50000 +defaults['SSHR_PORT'] ||= 5051 +defaults['SSHR_PORT_BASE'] ||= 21000 defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' defaults['SCHED_HOST'] ||= '172.17.0.1' diff --git a/container/ssh-r/start b/container/ssh-r/start index 6c7a373..68a63f2 100755 --- a/container/ssh-r/start +++ b/container/ssh-r/start @@ -16,7 +16,7 @@ cmd=( -e TCP_FORWARDING=true -d -p 21000-23999:21000-23999 - -p 5050:22 + -p 5051:22 -v /etc/localtime:/etc/localtime:ro -v /srv/pub/sshr/keys/:/etc/ssh/keys ssh-r:0.001 -- 2.23.0

2 1

[PATCH lkp-tests] stats/build-pkg: update filtering rule
by Lin Jiaxin 23 Mar '21

23 Mar '21

error_ids that are filtered out: /tmp/lkp/clementine-git/src/Clementine/3rdparty/qsqlite/qsql_sqlite.cpp: In function ‘QSqlError qMakeError(sqlite3*, const QString&, QSqlError::ErrorType, int)’: make: *** [makefile.linux:68: makefile.dependencies] Error 1 go: finding github.com/pingcap/errors v0.11.5-0.20201126102027-b0a155152ca3 Cyclomatic Complexity 8 ./include/trace/events/error_report.h:trace_error_report_end etc. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- stats/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/stats/build-pkg b/stats/build-pkg index 90f27656..764594b7 100755 --- a/stats/build-pkg +++ b/stats/build-pkg @@ -20,7 +20,7 @@ while (line = STDIN.gets) seqno = '' end next unless in_stderr - next unless line.downcase =~ /error|warning/ + next unless line.downcase =~ /(error|warning):[^:]/ error_ids << build_pkg_error_id(line) end -- 2.23.0

1 0

[PATCH compass-ci 3/3] sparrow/3-code/dev-env: add upstream config file to fix git-mirror error
by Liu Yinsi 23 Mar '21

23 Mar '21

[error] 2021-03-18 14:24:19.000000000 +0800 git-mirror: ERROR: No upstream-config file 2021-03-18 14:25:20.000000000 +0800 git-mirror: /c/compass-ci/lib/git_mirror.rb:171:in `load_fork_info': undefined method `[]' for nil:NilClass (NoMethodError) 2021-03-18 14:25:20.000000000 +0800 git-mirror: from /c/compass-ci/lib/git_mirror.rb:126:in `initialize' 2021-03-18 14:25:20.000000000 +0800 git-mirror: from ./git-mirror.rb:8:in `new' 2021-03-18 14:25:20.000000000 +0800 git-mirror: from ./git-mirror.rb:8:in `<main>' [why] for user locally deploy compass-ci, if upstream-config file not exists, failed to run git-mirror, so add defaults upstream-config file. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index eb162f7..f16cf80 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -42,6 +42,15 @@ SCHED_PORT: $sched_port lab: $lab EOF +[[ -f /etc/compass-ci/defaults/upstream-config ]] || { +cat > /etc/compass-ci/defaults/upstream-config <<EOF +upstreams: +- url: https://gitee.com/thesamename/upstream-repos.git + location: upstream + git_repo: u/upstream-repos/upstream-repos +EOF +} + mkdir -p /etc/compass-ci/service cat > /etc/compass-ci/service/service-env.yaml <<EOF ES_HOST: $ES_HOST -- 2.23.0

1 0

[PATCH compass-ci 2/3] sparrow/3-code/dev-env: set $REPO_SRC value
by Liu Yinsi 23 Mar '21

23 Mar '21

before: REPO_SRC=/c/upstream-repos after: REPO_SRC=/c/git-repos [why] can refer: [PATCH compass-ci] container/git-mirror: change mount dir "-v #{ENV['REPO_SRC']}:#{DOCKER_REPO}" in $CCI_SRC/container/git-mirror/start script,, so need to change $REPO_SRC value. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index e182b65..eb162f7 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -108,7 +108,7 @@ EOF cat > /etc/profile.d/compass.sh <<'EOF' export LKP_SRC=/c/lkp-tests export CCI_SRC=/c/compass-ci -export REPO_SRC=/c/upstream-repos +export REPO_SRC=/c/git-repos export CCI_REPOS=/c export PATH="$PATH:$CCI_SRC/sbin:$LKP_SRC/sbin:$LKP_SRC/bin" -- 2.23.0

1 0

[PATCH compass-ci 1/3] sparrow/1-storage: add work dir and permission for /c/git-repos
by Liu Yinsi 23 Mar '21

23 Mar '21

the URL of the repository to be tested will store in /c/git-repos, set lkp:committer permission for git-mirror. refer: [PATCH compass-ci] container/git-mirror: change mount dir Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/permission | 2 ++ sparrow/1-storage/tiny | 1 + 2 files changed, 3 insertions(+) diff --git a/sparrow/1-storage/permission b/sparrow/1-storage/permission index 26086d5..3d9bdfa 100755 --- a/sparrow/1-storage/permission +++ b/sparrow/1-storage/permission @@ -11,6 +11,7 @@ chmod_dir=( /srv/cci/serial/fluentd-pos /srv/cci/Maildir /srv/rabbitmq + /c/git-repos ) chmod 775 "${chmod_dir[@]}" @@ -29,6 +30,7 @@ chown_dir=( chown lkp:lkp "${chown_dir[@]}" chown lkp:committer /srv/git +chown lkp:committer /c/git-repos chown mailer:team /srv/cci/Maildir chgrp team /srv/cci/serial/logs diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index 9939d93..0eb7d39 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -20,6 +20,7 @@ dirs=( /srv/cci/libvirt-xml /tftpboot /c + /c/git-repos /etc/qemu /srv/cci/Maildir/.compass-ci/new /srv/cci/Maildir/.compass-ci/cur -- 2.23.0

1 0

[PATCH lkp-tests] daemon/sshd: change sshr service port
by Wu Zhende 23 Mar '21

23 Mar '21

Restarting the sshr service will disconnect the existing sshr connection and the machine that has been applied for can't be accessed. Therefore, we can't restart the sshr service. Instead, we start a new sshr service and use a new port. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- daemon/sshd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/daemon/sshd b/daemon/sshd index 50a7677ab..e0616ea8b 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -13,7 +13,7 @@ PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/email.sh sshr_ip=$LKP_SERVER -[ -n "$sshr_port" ] || sshr_port=5050 +[ -n "$sshr_port" ] || sshr_port=5051 [ -n "$sshr_port_len" ] || sshr_port_len=2000 [ -n "$sshr_port_base" ] || sshr_port_base=21000 -- 2.23.0

1 0

[PATCH compass-ci] docker/docker.rb: fix failed to borrow and run multi-docker
by Lin Jiaxin 23 Mar '21

23 Mar '21

job id=z9.1471951 ==> /tmp/stderr <== /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require': cannot load such file -- ../lib/mq_client (LoadError) from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require' from /c/compass-ci/providers/docker/docker.rb:11:in `<top (required)>' from /c/compass-ci/providers/multi-docker:7:in `require_relative' from /c/compass-ci/providers/multi-docker:7:in `<main>' Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- providers/docker/docker.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 9d6854f..51017f3 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -8,7 +8,7 @@ require 'json' require 'set' require 'fileutils' -require '../lib/mq_client' +require_relative '../../lib/mq_client' require_relative '../../container/defconfig' BASE_DIR = '/srv/dc' -- 2.23.0

1 0

[PATCH compass-ci 2/2] provides: fix require error
by Wu Zhende 23 Mar '21

23 Mar '21

[Error] can't load such file "../lib/mq_client" [How] use absolute path Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/docker.rb | 4 +++- providers/multi-qemu | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 9d6854f..b372a11 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -3,12 +3,14 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +CCI_SRC = ENV['CCI_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + require 'open-uri' require 'json' require 'set' require 'fileutils' -require '../lib/mq_client' +require "#{CCI_SRC}/lib/mq_client" require_relative '../../container/defconfig' BASE_DIR = '/srv/dc' diff --git a/providers/multi-qemu b/providers/multi-qemu index d3f8406..70675d1 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -3,11 +3,13 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +CCI_SRC = ENV['CCI_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + require 'fileutils' require 'optparse' require 'json' -require '../lib/mq_client' +require "#{CCI_SRC}/lib/mq_client" opt = {} options = OptionParser.new do |opts| -- 2.23.0

1 0

[PATCH v4 lkp-tests] tests/ansible_test: ansible test case
by Wang Chenglong 23 Mar '21

23 Mar '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- tests/ansible_test | 81 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100755 tests/ansible_test diff --git a/tests/ansible_test b/tests/ansible_test new file mode 100755 index 000000000..bf75b19b7 --- /dev/null +++ b/tests/ansible_test @@ -0,0 +1,81 @@ +#!/bin/bash +# - ansible_repo + +. $LKP_SRC/lib/debug.sh + +check_vars() +{ + ansible_host=localhost + [ -n "$ansible_repo" ] || die "ansible_repo is empty" + benchmark=${ansible_repo##*/} + + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible +} + +install_ansible() +{ + if [ "$os" == "openeuler" ]; then + yum -y reinstall git + yum -y install python3 + pip3 install ansible + elif [ "$os" == "centos" ]; then + yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm + yum -y reinstall git + yum -y install ansible + elif [ "$os" == "debian" ]; then + apt-get install -y ansible + fi + + command -v ansible > /dev/null && { + ansible localhost -m ping + [ $? -eq 0 ] || { + echo "Ansible failed to install." + exit 1 + } + } +} + +download_ansible_role() +{ + ansible_roles_path=$ansible_path/roles + [ -d $ansible_roles_path ] || mkdir -p $ansible_roles_path + cd $ansible_roles_path + git clone git://github.com/$ansible_repo.git + if [ $? -eq 0 ]; then + echo "$benchmark is successfully cloned" + else + echo "Please check whether your ansible_repo is correct!" + exit 1 + fi +} + +ansible_playbook_run() +{ + cat <<-EOF > "$ansible_path"/site.yml + - hosts: $ansible_host + roles: + - role: $benchmark + EOF + [ $? -eq 0 ] && { + echo "Playbook has been generated!" + } + + ansible-playbook "$ansible_path"/site.yml + + if [ $? -eq 0 ]; then + echo "Your playbook is running successfully!" + else + echo "Your playbook is running failed!" && exit 1 + fi +} + +main() +{ + check_vars + install_ansible + download_ansible_role + ansible_playbook_run +} + +main -- 2.23.0

3 6