- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 3/3] lib/sched.cr: redis use singleton pattern
by Wu Zhende 28 Jan '21

28 Jan '21

[Why] A redis pool is created for each HTTP request. This is wrong. This can cause too many connections to be established. All HTTP requests should share a redis pool. [How] Use singleton pattern Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f46d97c..b90759f 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -32,7 +32,7 @@ class Sched def initialize(env : HTTP::Server::Context) @es = Elasticsearch::Client.new - @redis = Redis::Client.new + @redis = Redis::Client.instance @task_queue = TaskQueueAPI.new @rgc = RemoteGitClient.new @env = env -- 2.23.0

1 0

[PATCH compass-ci 1/3] crystal-shards/shard.yml: add singleton.cr
by Wu Zhende 28 Jan '21

28 Jan '21

In order to use the singleton pattern Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/crystal-shards/shard.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/crystal-shards/shard.yml b/container/crystal-shards/shard.yml index 65a7622..44d175d 100644 --- a/container/crystal-shards/shard.yml +++ b/container/crystal-shards/shard.yml @@ -30,3 +30,5 @@ dependencies: github: anykeyh/json_on_steroids rate_limiter: github: z64/rate_limiter + singleton: + github: waterlink/singleton.cr -- 2.23.0

1 0

[PATCH v4 compass-ci] providers/lib: introduce context.rb
by Xiao Shenwei 28 Jan '21

28 Jan '21

[why] expand erb templates need a class to save context [how] Context class provides a readable property @info, it will save the context required by erb, and a expand_erb method, it can expand a erb templates based on @info. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/lib/context.rb | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 providers/lib/context.rb diff --git a/providers/lib/context.rb b/providers/lib/context.rb new file mode 100644 index 0000000..6f9f579 --- /dev/null +++ b/providers/lib/context.rb @@ -0,0 +1,35 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'erb' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# saves the context required by expand erb templates +class Context + attr_reader :info + + def initialize(mac, hostname, queues) + @info = { + 'mac' => mac, + 'hostname' => hostname, + 'queues' => queues + } + end + + def merge!(hash) + @info.merge!(hash) + end + + # existing variables of @info, can be used by erb + # @info: { 'name' => 'xxx' } + # Templates: + # This is <%= name %> + # After expanding: + # This is xxx + def expand_erb(template, context_hash = {}) + @info.merge!(context_hash) + context = Hashugar.new((a)info).instance_eval { binding } + ERB.new(template, nil, '%').result(context) + end +end -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/cci-depends: fix generate wrong symlink file
by Wang Yong 28 Jan '21

28 Jan '21

[before] $ ll $LKP_SRC/distro/depends/sysbench-cpu sysbench-cpu -> sysbench $ tree /srv/initrd/deps/$os/$os_arch/$os_version/ . ├── sysbench_20200903.cgz ├── sysbench.cgz -> sysbench_20200903.cgz └── sysbench-cpu.cgz -> sysbench.cgz all cgz file under one directory, so we can generate symlink when depends file is a symlink [after] $ tree /srv/initrd/deps/$os/$os_arch/$os_version/sysbench-cpu . ├── sysbench-cpu_20210128.cgz └── sysbench-cpu.cgz -> sysbench-cpu_20210128.cgz now we upload cgz file through curl, it's not appropriate to generate symlink directly, now generate original cgz file and rename it Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/aliyun | 2 +- distro/archlinux | 2 +- distro/common | 12 ++++++++++++ distro/debian | 2 +- tests/cci-depends | 9 ++------- 5 files changed, 17 insertions(+), 10 deletions(-) diff --git a/distro/aliyun b/distro/aliyun index 598243d8e..5c234ada6 100755 --- a/distro/aliyun +++ b/distro/aliyun @@ -80,7 +80,7 @@ yum_download() yum_pack() { local target_dir=/opt/rpms - local date=$(date +"%Y%m%d") + date=$(date +"%Y%m%d") local downloaded_rpms=$(find /var/ -type f -name "*.rpm") diff --git a/distro/archlinux b/distro/archlinux index 3942080a6..3d576555c 100755 --- a/distro/archlinux +++ b/distro/archlinux @@ -50,7 +50,7 @@ pacman_download() pacman_pack() { local target_dir=/opt/pkgs - local date=$(date +"%Y%m%d") + date=$(date +"%Y%m%d") local downloaded_pkgs=$(find_pacman_package) diff --git a/distro/common b/distro/common index 1fd7df4e8..d1a3a5447 100755 --- a/distro/common +++ b/distro/common @@ -94,3 +94,15 @@ upload_to_target_dir() upload_one_curl "${dir}/${file}" "${dir}" >/dev/null } + +rename_symlink_name() +{ + [ "${BM_NAME}" == "${benchmark}" ] && return + + cd "$pack_to" + [ -f ${BM_NAME}.cgz ] || return + rm ${BM_NAME}.cgz + mv ${BM_NAME}_$date.cgz ${benchmark}_$date.cgz + + ln -sf ${benchmark}_$date.cgz ${benchmark}.cgz +} diff --git a/distro/debian b/distro/debian index 8821e3ddd..b07d395b6 100755 --- a/distro/debian +++ b/distro/debian @@ -99,7 +99,7 @@ fixup_perf() pack() { - local date=$(date +"%Y%m%d") + date=$(date +"%Y%m%d") [[ $(type -t fixup_$BM_NAME) = 'function' ]] && fixup_$BM_NAME diff --git a/tests/cci-depends b/tests/cci-depends index 8ac1d7e58..0684a1025 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -38,13 +38,6 @@ umask 002 chown .lkp "$pack_to" } -check_shared_package "$benchmark" && { - ln -sf $BM_NAME.cgz $pack_to/$benchmark.cgz - echo "create symlink for shared package: $benchmark -> $BM_NAME" - - exit 0 -} - share_debian_package distro_install_depends cci-depends @@ -59,4 +52,6 @@ fixup_arch_install $pack_arch pack_benchmark_deps +rename_symlink_name + upload_to_target_dir ${pack_to} -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib: add log.sh
by Liu Yinsi 28 Jan '21

28 Jan '21

Records the run logs of the shell program Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/log.sh | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 lib/log.sh diff --git a/lib/log.sh b/lib/log.sh new file mode 100644 index 0000000..83c330c --- /dev/null +++ b/lib/log.sh @@ -0,0 +1,23 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +log_info() +{ + date +"%F %T $* [INFO]" +} + +log_debug() +{ + date +"%F %T $* [DEBUG]" +} + +log_warn() +{ + date +"%F %T $* [WARN]" >&2 +} + +log_error() +{ + date +"%F %T $* [ERROR]" >&2 +} -- 2.23.0

2 5

[PATCH lkp-tests 3/3] sbin/install-dependencies: replace gem sources
by Liu Yinsi 28 Jan '21

28 Jan '21

Encapsulating functions install_gem_pkg for easy gem sources replacement Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/install-dependencies.sh | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index f2a9155f6..8bd1a1ddb 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -6,6 +6,13 @@ PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/env.sh # choose install function base on DISTRIBUTION + +install_gem_pkg() +{ + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ + sudo gem install -f git activesupport rest-client faye-websocket +} + linux_dep() { source /etc/os-release @@ -13,7 +20,7 @@ linux_dep() ubuntu|debian) export DEBIAN_FRONTEND=noninteractive sudo apt-get install -yqm ruby-dev libssl-dev gcc g++ - sudo gem install -f faye-websocket git activesupport rest-client + install_gem_pkg ;; openEuler|fedora|rhel|centos) if type dnf > /dev/null 2>&1; then @@ -21,7 +28,7 @@ linux_dep() else sudo yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp fi - sudo gem install -f git activesupport rest-client faye-websocket + install_gem_pkg ;; *) echo "$ID not support! please install dependencies manually." @@ -34,7 +41,7 @@ mac_dep() { brew install ruby write_shell_profile "export PATH=/usr/local/opt/ruby/bin:$PATH" - sudo gem install -f git activesupport rest-client + install_gem_pkg } run() -- 2.23.0

1 0

[PATCH lkp-tests 2/3] sbin/install-dependencies: classify gem pkg
by Liu Yinsi 28 Jan '21

28 Jan '21

Install ruby pkg by gem install, not by apt-get install. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/install-dependencies.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index aa069bfcd..f2a9155f6 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -12,8 +12,8 @@ linux_dep() case $ID in ubuntu|debian) export DEBIAN_FRONTEND=noninteractive - sudo apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client ruby-dev libssl-dev gcc g++ - sudo gem install -f faye-websocket + sudo apt-get install -yqm ruby-dev libssl-dev gcc g++ + sudo gem install -f faye-websocket git activesupport rest-client ;; openEuler|fedora|rhel|centos) if type dnf > /dev/null 2>&1; then -- 2.23.0

1 0

[PATCH lkp-tests 1/3] sbin/install-dependencies: fix inconsistent gem command
by Liu Yinsi 28 Jan '21

28 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/install-dependencies.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 6b418e5f4..aa069bfcd 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -13,7 +13,7 @@ linux_dep() ubuntu|debian) export DEBIAN_FRONTEND=noninteractive sudo apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client ruby-dev libssl-dev gcc g++ - sudo gem install faye-websocket + sudo gem install -f faye-websocket ;; openEuler|fedora|rhel|centos) if type dnf > /dev/null 2>&1; then @@ -34,7 +34,7 @@ mac_dep() { brew install ruby write_shell_profile "export PATH=/usr/local/opt/ruby/bin:$PATH" - gem install git activesupport rest-client + sudo gem install -f git activesupport rest-client } run() -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: delete unnecessary update of es
by Li Yuanchao 28 Jan '21

28 Jan '21

It does not need to update es when init, cause nothing changes yet. And update es here would make git-mirror start slowly. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index a111e51..2090f6b 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -214,7 +214,6 @@ class MirrorMain @git_info[git_repo]['git_repo'] = git_repo @git_info[git_repo].merge!(@defaults[project]) if @defaults[project] fork_stat_init(git_repo) - es_repo_update(git_repo) @priority_queue.push git_repo, @priority @priority += 1 end -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor: change ip url to domain name url
by Li Ping 28 Jan '21

28 Jan '21

when the external ip address of host is the same as the srv_http_host, the host can not use the url to get result. --- lib/monitor.rb | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index 9091c92a9..82e48c3e6 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -107,9 +107,7 @@ class Monitor def lftp_mirror @result_root.delete_prefix!('/srv') - srv_http_host = job['SRV_HTTP_HOST'] || 'api.compass-ci.openeuler.org' - srv_http_port = job['SRV_HTTP_PORT'] || '11300' - url = "http://#{srv_http_host}:#{srv_http_port}#{@result_root}" + url = "http://api.compass-ci.openeuler.org:11300#{@result_root}" system "lftp -c mirror #{url} >/dev/null 2>&1" end -- 2.23.0

2 2