- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 2/4] container/web-backend: add api for web search testbox history
by Li Yuanchao 21 Dec '20

21 Dec '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 11ba604..ff8fcff 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -110,3 +110,26 @@ end get '/get_testboxes' do query_testboxes end + +# GET /get_tbox_state?testbox=$testbox +# Response: +# - { +# testbox: $testbox, +# states: { +# "state": "$current_state", +# "job_id": "$current_job_id", +# "history": [ +# { +# "time": "$history_time", +# "mac": "$mac", +# "ip": "$ip", +# "job_id": "$history_job_id", +# "state": "$history_state" +# }, +# ... +# ] +# } +# } +get '/get_tbox_state' do + get_tbox_state(params) +end -- 2.23.0

1 0

[PATCH compass-ci 1/4] sched.cr: put testbox state to es
by Li Yuanchao 21 Dec '20

21 Dec '20

History of testbox would be kept, to be searched by web api. [sample] { "_index" : "testbox", "_type" : "_doc", "_id" : "vm-2p16g.liyuanchao-969257", "_score" : 1.0, "_source" : { "state" : "rebooting", "job_id" : "crystal.190886", "history" : [ { "time" : "2020-12-15 19:52:14", "mac" : "0a-f5-c0-1f-f0-dd", "ip" : "172.18.202.96", "job_id" : "crystal.190886", "state" : "running" }, { "time" : "2020-12-15 19:57:22", "mac" : "0a-f5-c0-1f-f0-dd", "ip" : "172.18.202.96", "job_id" : "crystal.190886", "state" : "rebooting" } ] } } Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/sched.cr | 1 + src/scheduler/elasticsearch_client.cr | 21 +++++++++++++++++++++ 2 files changed, 22 insertions(+) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a1ef28e..ed6e538 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -90,6 +90,7 @@ class Sched end @redis.update_wtmp(testbox, hash) + @es.update_tbox(testbox, hash) # json log hash["testbox"] = testbox diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index 7c92204..b17d7f8 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -87,6 +87,27 @@ class Elasticsearch::Client ) end + def update_tbox(testbox : String, wtmp_hash : Hash) + query = {:index => "testbox", :type => "_doc", :id => testbox} + result = @client.get_source(query) if @client.exists(query) + history = result["history"].as_a? if result.is_a?(JSON::Any) + history ||= [] of JSON::Any + history << JSON.parse(wtmp_hash.to_json) + + @client.create( + { + :index => "testbox", + :type => "_doc", + :id => testbox, + :body => { + :state => wtmp_hash["state"], + :job_id => wtmp_hash["job_id"], + :history => history + } + } + ) + end + private def create(job_content : JSON::Any, job_id : String) return @client.create( { -- 2.23.0

1 0

[PATCH v3 lkp-tests] lib/monitor: set monitor_url based on job
by Wu Zhende 21 Dec '20

21 Dec '20

If @job is exist and monitor_url is not configured, set monitor_url based on @job's SCHED_HOST Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index f65cfccf1..283cd788d 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -24,7 +24,6 @@ class Monitor # 1 means timeout exit @exit_status_code = 0 @defaults = {} - load_default @result = [] @stop_query = {} @reason = nil @@ -33,6 +32,10 @@ class Monitor def load_default return unless @monitor_url == '' + if host = @job["SCHED_HOST"] + return @monitor_url = "ws://#{host}:11310/filter" + end + Dir.glob(['/etc/compass-ci/monitor/*.yaml', "#{ENV['HOME']}/.config/compass-ci/monitor/*.yaml"]).each do |file| next unless File.exist? file @@ -98,6 +101,7 @@ class Monitor def run(timeout = nil) merge_overrides + load_default field_check @query.each do |k, v| -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] lib/matrix2.rb: remove the *timestamp* from matrix
by Lu Weitao 21 Dec '20

21 Dec '20

[Why] In compare results: the value of *timestamp*(metric) doesn't mean the number of occurrences, it may confuse user like bellow: openeuler centos archLinux metric -------------------- ------------------------------ ------------------------------ ------------------------------ fails:runs change fails:runs change fails:runs | | | | | 3585:13 +6913.7% 344:1 -2.6e+4% 11:1 dmesg.timestamp:last 2350:13 +1.6e+4% 344:1 -1.7e+4% 10:1 kmsg.timestamp:last 2:13 -15.4% 0:1 +84.6% 1:1 last_state.exit_fail [How] remove the metric that include "timestamp" from matrix Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index c712e32..2769f7c 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -87,6 +87,8 @@ def create_matrix(job_list) next unless stats stats.each do |key, value| + next if key.include?('timestamp') + matrix[key] = [] unless matrix[key] matrix[key] << value end -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] lib/matrix2.rb: fix rubocop warning
by Lu Weitao 21 Dec '20

21 Dec '20

[Why] lib/matrix2.rb:75:1: C: Metrics/MethodLength: Method has too many lines. [16/15] def create_matrix(job_list) ... Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2507b77..c712e32 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -64,6 +64,12 @@ def samples_fill_missing_zeros(value, size) value.concat([0] * (size - value.size)) end +def matrix_fill_miss_zeros(matrix, col_size) + matrix.each_value do |value| + samples_fill_missing_zeros(value, col_size) + end +end + # input: job_list # return: matrix of Hash(String, Array(Number)) # Eg: matrix: { @@ -86,9 +92,7 @@ def create_matrix(job_list) end end col_size = job_list.size - matrix.each_value do |value| - samples_fill_missing_zeros(value, col_size) - end + matrix_fill_miss_zeros(matrix, col_size) return matrix, suites end -- 2.23.0

1 0

[PATCH v2 lkp-tests 4/4] install.sh: grant execute permission
by Wei Jihui 21 Dec '20

21 Dec '20

cd lkp-test ./install.sh && source ~/.${SHELL##*/}rc Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- install.sh | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 install.sh diff --git a/install.sh b/install.sh old mode 100644 new mode 100755 -- 2.23.0

1 0

[PATCH v2 lkp-tests 3/4] install.sh: refactor install flow
by Wei Jihui 21 Dec '20

21 Dec '20

1. when exec install.sh, no $LKP_SRC 2. first enable $LKP_SRC env 3. mv sbin/install-dependencies.sh to install.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- Makefile | 2 -- install.sh | 10 +++++++--- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/Makefile b/Makefile index 0b2d09a2..ae062973 100644 --- a/Makefile +++ b/Makefile @@ -14,8 +14,6 @@ subsystem: install: mkdir -p $(TARGET_DIR_BIN) ln -sf $(shell pwd)/bin/lkp $(TARGET_DIR_BIN)/lkp - bash sbin/set-env.sh - bash sbin/install-dependencies.sh .PHONY: doc doc: diff --git a/install.sh b/install.sh index cea1654f..2bd17d54 100644 --- a/install.sh +++ b/install.sh @@ -1,7 +1,11 @@ #!/bin/bash -. $LKP_SRC/lib/env.sh - -make +. lib/env.sh +write_host +write_shell_profile "export LKP_SRC=$PWD" +write_shell_profile "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" source $(shell_profile) + +. sbin/install-dependencies.sh +make install -- 2.23.0

1 0

[PATCH compass-ci] doc: update READEME.md
by Liu Yinsi 21 Dec '20

21 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 111 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 86 insertions(+), 25 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index ccb1d15..7c20a53 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -1,26 +1,87 @@ -Deploy a mini instance in one single machine. - -Prepare: - 1. hardware - Server: at least prepare a server - ProductType: ThaiShan200-2280 - Arch: aarch64 - Memory: 8G - CPU: 64 nuclear - DiskSpace: 500G - - 2. software - OS: openEuler-aarch64-20.03 LTS - git: suggest 2.23.0 - - 3. network - Internet is available - - 4. /os for store rootfs - >= 300G - -Steps: - umask 002 +### 本地搭建 compass-ci 服务器节点 + +在 openEuler 系统一键部署 compass-ci 环境，当前已支持 openEuler-aarch64-20.03-LTS 系统环境，以下配置仅供参考。 + +#### 准备工作 +- 硬件 + 服务器类型：ThaiShan200-2280 (建议) + 架构：aarch64 + 内存：>= 32GB + CPU：64 nuclear (建议) + 硬盘：>= 500G (建议划分独立分区) + + >**说明：** + >划分较小独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… + >划分较大独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… + +- 软件 + OS：openEuler-aarch64-20.03 LTS + git：2.23.0版本 (建议) + 预留空间：>= 300G + 网络：可以访问互联网 + + >**说明：** + >openEuler 系统安装详细操作请参考[添加测试用例](https://openeuler.org/zh/docs/20.03_LTS/docs/Installati… + +#### 操作指导 +1. 登录 openEuler 系统 + +2. 创建工作目录并设置文件权限 + ```bash + mkdir demo && cd demo && umask 002 + ``` +3. 克隆 compass-ci 项目代码到 demo 目录 + ```bash git clone https://gitee.com/wu_fengguang/compass-ci.git - cd compass-ci/sparrow - ./install-tiny + ``` +4. 执行一键部署脚本 install-tiny + ```bash + cd compass-ci/sparrow && ./install-tiny + ``` +5. 生成lkp-aarch64.cgz压缩包 + ```bash + cd /c/compass-ci/container/lkp-initrd && ./run + ``` +6. 验证账号 + ```bash + cd /c/compass-ci/sbin && ./build-my-info.rb + 按照提示输入账号名和邮箱即可 + +7. 测试环境是否可以提交job测试 + ```bash + submit iperf.yaml testbox=vm-2p8g + ``` + + 执行上述命令正常情况下会提示信息如下： + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + compass-ci搭建完毕，下面就可以开始进行测试了。 + +8. 下载rootfs（以下载openeuler/aarch64/20.03为例，需要在哪个系统上测试就去对应的/srv/os/目录下使用wget命令下载cgz文件包） + ```bash + mkdir -p /srv/os/openeuler/aarch64/ + cd /srv/os/openeuler/aarch64 + wget http://124.90.34.227:11300/os/test/openeuler/aarch64/20.03.cgz + ``` + +9. 解压cgz包 + ```bash + gzip -dc 20.03.cgz | cpio -idv + ``` + +8. [使用 compass-ci 平台测试开源项目](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… + +9. [编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testca… + +10. 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… + 注意：os_mount 必须指定为cifs + +11. 运行测试任务 + ```bash + cd /c/compass-ci/providers/ && ./my-qemu.sh + ``` + +12. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… id查看output文件 + ```bash + cd /srv/result/iperf/$job_id/output + ``` -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] lib/compare_matrixes.rb: don't remove the field that start with $suite
by Lu Kaiyi 21 Dec '20

21 Dec '20

if the metric field start with $suite, even if the field is unchanged, still keep it in compare result. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_matrixes.rb | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index b7ea5a4..0727797 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -176,11 +176,15 @@ def get_matrixes_values(matrixes_list, options) matrixes_values end -def remove_unchanged_field(matrixes_values) +def remove_unchanged_field(matrixes_values, suite_list) # remove unchanged field from matrixes values and remove :changed key # matrixes_values.each_key do |success| matrixes_values[success].delete_if do |field| + if suite_list.any? { |suite| field.start_with?(suite) } + matrixes_values[success][field][:changed] = true + end + !matrixes_values[success][field].delete(:changed) end end @@ -203,7 +207,7 @@ def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key options = { 'perf-profile': 5, theme: :none, no_print: false }.merge(options) matrixes_values = get_matrixes_values(matrixes_list, options) - remove_unchanged_field(matrixes_values) if matrixes_list.length > 1 + remove_unchanged_field(matrixes_values, suite_list) if matrixes_list.length > 1 no_print = options[:no_print] result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/matrix2.rb: discard the no $suite related job
by Lu Kaiyi 21 Dec '20

21 Dec '20

if job has suite, but no $suite related stats metric, then discard this job when create matrix. --- lib/matrix2.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2507b77..f1b2c78 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,10 +76,15 @@ def create_matrix(job_list) matrix = {} suites = [] job_list.each do |job| - suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats + if job['suite'] + next unless stats.keys.any? { |stat| stat.start_with?(job['suite']) } + + suites << job['suite'] + end + stats.each do |key, value| matrix[key] = [] unless matrix[key] matrix[key] << value -- 2.23.0

1 0

[PATCH v2 lkp-tests] lib/monitor: get host from job
by Wu Zhende 21 Dec '20

21 Dec '20

If @job is exist and monitor_url is not configured, set monitor_url based on SCHED_HOST Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index f65cfccf1..283cd788d 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -24,7 +24,6 @@ class Monitor # 1 means timeout exit @exit_status_code = 0 @defaults = {} - load_default @result = [] @stop_query = {} @reason = nil @@ -33,6 +32,10 @@ class Monitor def load_default return unless @monitor_url == '' + if host = @job["SCHED_HOST"] + return @monitor_url = "ws://#{host}:11310/filter" + end + Dir.glob(['/etc/compass-ci/monitor/*.yaml', "#{ENV['HOME']}/.config/compass-ci/monitor/*.yaml"]).each do |file| next unless File.exist? file @@ -98,6 +101,7 @@ class Monitor def run(timeout = nil) merge_overrides + load_default field_check @query.each do |k, v| -- 2.23.0

2 1

Re: [PATCH v2 compass-ci] lib/matrix2.rb remove the *timestamp* from matrix
by Ren Wen 21 Dec '20

21 Dec '20

Why remove ':' after 'lib/matrix2.rb'? Thanks, RenWen Reply-To: In-Reply-To: <20201221022051.3690974-1-luweitaobe(a)163.com> On Mon, Dec 21, 2020 at 10:20:51AM +0800, Lu Weitao wrote: >[Why] >In compare results: >the value of *timestamp*(metric) doesn't mean the number of occurrences, >it may confuse user like bellow: > > openeuler centos archLinux metric >-------------------- ------------------------------ ------------------------------ ------------------------------ > fails:runs change fails:runs change fails:runs > | | | | | > 3585:13 +6913.7% 344:1 -2.6e+4% 11:1 dmesg.timestamp:last > 2350:13 +1.6e+4% 344:1 -1.7e+4% 10:1 kmsg.timestamp:last > 2:13 -15.4% 0:1 +84.6% 1:1 last_state.exit_fail > >[How] >remove the metric that include "timestamp" from matrix > >Signed-off-by: Lu Weitao <luweitaobe(a)163.com> >--- > lib/matrix2.rb | 2 ++ > 1 file changed, 2 insertions(+) > >diff --git a/lib/matrix2.rb b/lib/matrix2.rb >index c712e32..2769f7c 100644 >--- a/lib/matrix2.rb >+++ b/lib/matrix2.rb >@@ -87,6 +87,8 @@ def create_matrix(job_list) > next unless stats > > stats.each do |key, value| >+ next if key.include?('timestamp') >+ > matrix[key] = [] unless matrix[key] > matrix[key] << value > end >-- >2.23.0 >

2 1

[PATCH lkp-tests 2/4] lib/env.sh: move functions to env.sh
by Wei Jihui 21 Dec '20

21 Dec '20

move functions:write_shell_profile and write_host to lib/env.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/env.sh | 30 ++++++++++++++++++++++++++++++ sbin/set-env.sh | 39 --------------------------------------- 2 files changed, 30 insertions(+), 39 deletions(-) delete mode 100755 sbin/set-env.sh diff --git a/lib/env.sh b/lib/env.sh index 8ac42a83..d360f163 100755 --- a/lib/env.sh +++ b/lib/env.sh @@ -125,3 +125,33 @@ shell_profile() [ -f "$profile" ] || touch $profile echo $profile } + +write_shell_profile() +{ + shell_profile_file=$(shell_profile) + if [ $# -gt 0 ]; then + echo $@ >> $shell_profile_file + fi + + source $shell_profile_file +} + +write_host() +{ + if is_system "Linux"; then + nr_cpu=$(nproc) + memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') + else + nr_cpu=$(sysctl -n hw.logicalcpu) + memory_total=$(top -l 1 | grep MemRegions | awk '{print $2}') + fi + memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') + memory=$(echo $memory_new | awk '{print int($0)+1}')G + + cat > hosts/$(hostname) <<-EOF + nr_cpu: $nr_cpu + memory: $memory + hdd_partitions: + ssd_partitions: + EOF +} diff --git a/sbin/set-env.sh b/sbin/set-env.sh deleted file mode 100755 index cc7f9050..00000000 --- a/sbin/set-env.sh +++ /dev/null @@ -1,39 +0,0 @@ -#!/bin/sh - -. $LKP_SRC/lib/env.sh - -write_shell_profile() -{ - shell_profile_file=$(shell_profile) - if [ $# -gt 0 ]; then - echo $@ >> $shell_profile_file - else - echo "export LKP_SRC=$PWD" >> $shell_profile_file - echo "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" >> $shell_profile_file - fi - - source $shell_profile_file -} - -write_host() -{ - if is_system "Linux"; then - nr_cpu=$(nproc) - memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') - else - nr_cpu=$(sysctl -n hw.logicalcpu) - memory_total=$(top -l 1 | grep MemRegions | awk '{print $2}') - fi - memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') - memory=$(echo $memory_new | awk '{print int($0)+1}')G - - cat > hosts/$(hostname) <<-EOF - nr_cpu: $nr_cpu - memory: $memory - hdd_partitions: - ssd_partitions: - EOF -} - -write_shell_profile -write_host -- 2.23.0

2 1

Re: [PATCH compass-ci 2/2] lib/matrix2.rb remove the *timestamp* form matrix
by Ren Wen 21 Dec '20

21 Dec '20

spell: form -> from Thanks, RenWen Reply-To: In-Reply-To: <20201219133430.1143483-2-luweitaobe(a)163.com> On Sat, Dec 19, 2020 at 09:34:30PM +0800, Lu Weitao wrote: >[Why] >In compare results: >the value of *timestamp*(metric) doesn't mean the number of occurrences, >it may confuse user like bellow: > > openeuler centos archLinux metric >-------------------- ------------------------------ ------------------------------ ------------------------------ > fails:runs change fails:runs change fails:runs > | | | | | > 3585:13 +6913.7% 344:1 -2.6e+4% 11:1 dmesg.timestamp:last > 2350:13 +1.6e+4% 344:1 -1.7e+4% 10:1 kmsg.timestamp:last > 2:13 -15.4% 0:1 +84.6% 1:1 last_state.exit_fail > >[How] >remove the metric that include "timestamp" from matrix > >Signed-off-by: Lu Weitao <luweitaobe(a)163.com> >--- > lib/matrix2.rb | 2 ++ > 1 file changed, 2 insertions(+) > >diff --git a/lib/matrix2.rb b/lib/matrix2.rb >index c712e32..2769f7c 100644 >--- a/lib/matrix2.rb >+++ b/lib/matrix2.rb >@@ -87,6 +87,8 @@ def create_matrix(job_list) > next unless stats > > stats.each do |key, value| >+ next if key.include?('timestamp') >+ > matrix[key] = [] unless matrix[key] > matrix[key] << value > end >-- >2.23.0 >

2 1

[PATCH v2 compass-ci] providers/docker: set the private key of the testbox
by Xiao Shenwei 21 Dec '20

21 Dec '20

[why] login the docker testbox without password [how] Add ssh private key for docker, for automatically establish the ssh reverse tunnel between sut and port map container. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/bin/entrypoint.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/providers/docker/bin/entrypoint.sh b/providers/docker/bin/entrypoint.sh index 63cb605..feb5edf 100755 --- a/providers/docker/bin/entrypoint.sh +++ b/providers/docker/bin/entrypoint.sh @@ -2,4 +2,6 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +mkdir -p /root/.ssh +cp /lkp/lkp/src/rootfs/addon/root/.ssh/id_rsa /root/.ssh/ sh /lkp/lkp/src/rootfs/addon/etc/init.d/lkp-bootstrap -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/matrix2.rb: fix rubocop warning
by Lu Weitao 21 Dec '20

21 Dec '20

[Why] lib/matrix2.rb:75:1: C: Metrics/MethodLength: Method has too many lines. [16/15] def create_matrix(job_list) ... Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2507b77..c712e32 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -64,6 +64,12 @@ def samples_fill_missing_zeros(value, size) value.concat([0] * (size - value.size)) end +def matrix_fill_miss_zeros(matrix, col_size) + matrix.each_value do |value| + samples_fill_missing_zeros(value, col_size) + end +end + # input: job_list # return: matrix of Hash(String, Array(Number)) # Eg: matrix: { @@ -86,9 +92,7 @@ def create_matrix(job_list) end end col_size = job_list.size - matrix.each_value do |value| - samples_fill_missing_zeros(value, col_size) - end + matrix_fill_miss_zeros(matrix, col_size) return matrix, suites end -- 2.23.0

2 1

Re: [PATCH compass-ci 2/3] assign-account-email: remove .bash_profile usage
by Luan Shengde 21 Dec '20

21 Dec '20

On Sat, Dec 19, 2020 at 11:23:43AM +0800, Li Ping wrote: >On Sat, Dec 19, 2020 at 11:09:54AM +0800, Wu Fengguang wrote: >>Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> >>--- >> container/mail-robot/lib/assign-account-email.rb | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >>diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb >>index 77ed4e8..ee3b574 100755 >>--- a/container/mail-robot/lib/assign-account-email.rb >>+++ b/container/mail-robot/lib/assign-account-email.rb >>@@ -37,7 +37,7 @@ def build_apply_account_email(my_info) >> git clone https://gitee.com/wu_fengguang/lkp-tests.git >> cd lkp-tests >> make install >>- source ${HOME}/.bashrc && source ${HOME}/.bash_profile >>+ source ${HOME}/.${SHELL##*/}rc > > use ${HOME} will become root in email Use \\${HOME} will resolve this problem. Thanks, Luan Shengde > >> >> 3) submit job >> reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md >>-- >>2.23.0 >>

1 0

[PATCH lkp-tests] tests/build-pkg: add config file support
by Wang Yong 21 Dec '20

21 Dec '20

[Why] config file in some software need storage and recall now we can add config value in yaml and cp ${config} to ${cci_dir}/${benchmark}/ in PKGBUILD so that config file can be uploaded correctly $ cat test.yaml ... config: ${config} ... Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 39 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index addff9f7b..0a143813f 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -5,6 +5,7 @@ # - git_tag # - pkg_ver # upstream_url +# - config # We'll use the below global env vars, but not treat them as parameter to avoid # duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ @@ -20,6 +21,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/distro/common +. $LKP_SRC/lib/http.sh check_vars() { @@ -31,6 +33,29 @@ check_vars() [ -n "$os_mount" ] || die "os_mount is empty" } +check_config_var() +{ + [ -n "$config" ] || return +} + +get_config() +{ + local cci_http_host=${INITRD_HTTP_HOST} + local cci_http_port=${INITRD_HTTP_PORT:-8800} + export cci_dir=/cci/config + local remote_dir="http://${cci_http_host}:${cci_http_port}${cci_dir}/${benchmark}" + + mkdir -p ${cci_dir} + + check_config_var + http_get_file "${remote_dir}/${config}" "${cci_dir}/${benchmark}/${config}" +} + +upload_config() +{ + upload_to_target_dir "${cci_dir}/${benchmark}" +} + mount_dest() { # the same image is mounted to cifs and nfs, the generated cgz files @@ -71,6 +96,13 @@ create_cgz_name() [ -n "$git_tag" ] && { cgz_name="${cgz_path}/${git_tag}.cgz" + # download PKGBUILD-$tag + benchmark="${upstream_repo##*/}" + export benchmark="${benchmark%.git}" + [ -n "$PKGBUILD_TAG" ] && { + get_pkgfile "pkg/$benchmark" "PKGBUILD-$PKGBUILD_TAG" + [ -f "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG not exist" + [ -s "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG is empty" } [ -n "$pkg_ver" ] && { @@ -131,7 +163,11 @@ build_source_pkg() PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf - [ "$?" == 0 ] || exit 1 + local build_state="$?" + + upload_config + + [ "${build_state}" == "0" ] || exit 1 create_softlink upload_to_target_dir ${PKG_MNT}/${pack_to}/${pkgname} } @@ -139,4 +175,5 @@ build_source_pkg() check_vars mount_dest request_pkg +get_config build_source_pkg -- 2.23.0

2 2

[PATCH compass-ci] lib/matrix2.rb: discard the no $suite related job
by Lu Kaiyi 19 Dec '20

19 Dec '20

if job has suite, but no $suite related stats metric, then discard this job when create matrix. --- lib/matrix2.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2507b77..f1b2c78 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,10 +76,15 @@ def create_matrix(job_list) matrix = {} suites = [] job_list.each do |job| - suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats + if job['suite'] + next unless stats.keys.any? { |stat| stat.start_with?(job['suite']) } + + suites << job['suite'] + end + stats.each do |key, value| matrix[key] = [] unless matrix[key] matrix[key] << value -- 2.23.0

2 2

[PATCH compass-ci 2/2] lib/matrix2.rb remove the *timestamp* form matrix
by Lu Weitao 19 Dec '20

19 Dec '20

[Why] In compare results: the value of *timestamp*(metric) doesn't mean the number of occurrences, it may confuse user like bellow: openeuler centos archLinux metric -------------------- ------------------------------ ------------------------------ ------------------------------ fails:runs change fails:runs change fails:runs | | | | | 3585:13 +6913.7% 344:1 -2.6e+4% 11:1 dmesg.timestamp:last 2350:13 +1.6e+4% 344:1 -1.7e+4% 10:1 kmsg.timestamp:last 2:13 -15.4% 0:1 +84.6% 1:1 last_state.exit_fail [How] remove the metric that include "timestamp" from matrix Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index c712e32..2769f7c 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -87,6 +87,8 @@ def create_matrix(job_list) next unless stats stats.each do |key, value| + next if key.include?('timestamp') + matrix[key] = [] unless matrix[key] matrix[key] << value end -- 2.23.0

1 0

[PATCH compass-ci] lib/commit_status.rb: query latest job_id by commit_id
by Lu Kaiyi 19 Dec '20

19 Dec '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/commit_status.rb | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/lib/commit_status.rb b/lib/commit_status.rb index 81aeb29..f1cbfbc 100644 --- a/lib/commit_status.rb +++ b/lib/commit_status.rb @@ -25,6 +25,17 @@ def query_latest_good_commit(job, error_id) return latest_good_commit end +def query_latest_job_id(commit_id) + items = { + 'upstream_commit' => commit_id + } + jobs_list = query_jobs_from_es(items) + return nil if jobs_list.empty? + + jobs_list.sort_by! { |job| job['end_time'] } + jobs_list.last['id'] +end + def query_jobs_from_es(items) es = ESQuery.new(ES_HOST, ES_PORT) result = es.multi_field_query items -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: check load host config from job
by Wei Jihui 19 Dec '20

19 Dec '20

@defaults will merge to @job, and set @default empty. if load host config from @job, so do not need to load host in advance. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 42b008c7..2ed06edc 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -317,17 +317,6 @@ class Job spec_file_context.each_key { |k| @job.delete k } end - def check_load_hosts_config(defaults_config) - return if @load_hosts_done - - if defaults_config.include?('testbox') - @job['tbox_group'] = tbox_group(defaults_config['testbox']) - elsif defaults_config.include?('tbox_group') - @job['tbox_group'] = defaults_config['tbox_group'] - end - return @job.include?('tbox_group') - end - def load_hosts_config return if @job.include?(:no_defaults) return if @load_hosts_done @@ -354,7 +343,7 @@ class Job def get_lab_hosts_file lab_hosts_file_name = @job['testbox'] - lab_hosts_file = "#{ENV['CCI_REPOS']}/lab-#{@defaults['lab']}/hosts/#{lab_hosts_file_name}" + lab_hosts_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{lab_hosts_file_name}" if File.file?(lab_hosts_file) lab_hosts_file @@ -409,7 +398,6 @@ class Job @defaults[source_file_symkey(file)] = nil revise_hash(@defaults, defaults, true) @defaults.merge!(@overrides) - load_hosts_config if check_load_hosts_config(@defaults) end @file_loaded[file] = true true -- 2.23.0

1 0

[PATCH compass-ci] scheduler-https/start: fix missing env key
by Wu Zhende 19 Dec '20

19 Dec '20

Some key don't define in scheduler-https. Synchronize with scheduler. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler-https/start | 34 +++++++++++++++++++++++---------- 1 file changed, 24 insertions(+), 10 deletions(-) diff --git a/container/scheduler-https/start b/container/scheduler-https/start index 9898c1a..98940d7 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -28,17 +28,32 @@ names = Set.new %w[ INITRD_HTTP_PORT TASKQUEUE_PORT TASKQUEUE_HOST + GIT_SERVER + MASTER_FLUENTD_HOST + MASTER_FLUENTD_PORT + SSHR_PORT + SSHR_PORT_BASE + SSHR_PORT_LEN + lab ] defaults = relevant_defaults(names) +defaults['SSHR_PORT'] ||= 5050 +defaults['SSHR_PORT_BASE'] ||= 50000 +defaults['SSHR_PORT_LEN'] ||= 2000 +defaults['SCHED_PORT'] ||= '3000' +defaults['SCHED_HOST'] ||= '172.17.0.1' +defaults['MASTER_FLUENTD_PORT'] ||= '24224' +defaults['SCHED_HTTPS_PORT'] ||= 11306 +defaults['SCHED_HTTPS_HOST'] ||= 'api.compass-ci.openeuler.org' env = docker_env(defaults) -DEFAULT_LKP = '/c/lkp-tests' -DEFAULT_CCI = '/c/compass-ci' -SCHED_PORT = defaults['SCHED_PORT'] || '3000' -SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' -SCHED_HTTPS_PORT = defaults['SCHED_HTTPS_PORT'] || 11306 -SCHED_HTTPS_HOST = defaults['SCHED_HTTPS_HOST'] || 'api.compass-ci.openeuler.org' +CCI_REPOS = ENV['CCI_REPOS'] || '/c' +DEFAULT_LKP = "#{CCI_REPOS}/lkp-tests" +DEFAULT_CCI = "#{CCI_REPOS}/compass-ci" +LAB = defaults['lab'] || 'z9' +SCHED_HTTPS_PORT = defaults['SCHED_HTTPS_PORT'] +SCHED_HTTPS_HOST = defaults['SCHED_HTTPS_HOST'] docker_rm "s001-alpine-#{SCHED_HTTPS_PORT}" @@ -48,14 +63,13 @@ cmd = %W[ --name s001-alpine-#{SCHED_HTTPS_PORT} -d -u 1090:1090 - -p #{SCHED_HTTPS_PORT}:#{SCHED_PORT} + -p #{SCHED_HTTPS_PORT}:#{defaults['SCHED_PORT']} ] + env + %W[ -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} - -e SCHED_HOST=#{SCHED_HOST} - -e SCHED_PORT=#{SCHED_PORT} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} -v /srv/cci/scheduler/alpine:/srv/cci/scheduler -v /etc/localtime:/etc/localtime:ro -v /srv/result:/srv/result @@ -65,7 +79,7 @@ cmd = %W[ --log-opt mode=non-blocking --log-opt max-buffer-size=4m --log-opt tag=scheduler-#{SCHED_HTTPS_PORT} - sch-ruby-a:v0.00d-#{SCHED_PORT} + sch-ruby-a:v0.00d-#{defaults['SCHED_PORT']} ] cmd += ['sh', '-c', 'umask 002 && ./scheduler --ssl --ssl-key-file /ca/server.key --ssl-cert-file /ca/server.crt'] -- 2.23.0

1 0

[PATCH lkp-tests 4/4] install.sh: grant execute permission
by Wei Jihui 19 Dec '20

19 Dec '20

cd lkp-test ./install.sh && source ~/.${SHELL##*/}rc Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- install.sh | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 install.sh diff --git a/install.sh b/install.sh old mode 100644 new mode 100755 -- 2.23.0

1 0

[PATCH lkp-tests 3/4] install.sh: refactor install flow
by Wei Jihui 19 Dec '20

19 Dec '20

1. when exec install.sh, no $LKP_SRC 2. first enable $LKP_SRC env 3. mv sbin/install-dependencies.sh to install.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- Makefile | 2 -- install.sh | 10 +++++++--- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/Makefile b/Makefile index 0b2d09a2..ae062973 100644 --- a/Makefile +++ b/Makefile @@ -14,8 +14,6 @@ subsystem: install: mkdir -p $(TARGET_DIR_BIN) ln -sf $(shell pwd)/bin/lkp $(TARGET_DIR_BIN)/lkp - bash sbin/set-env.sh - bash sbin/install-dependencies.sh .PHONY: doc doc: diff --git a/install.sh b/install.sh index cea1654f..2bd17d54 100644 --- a/install.sh +++ b/install.sh @@ -1,7 +1,11 @@ #!/bin/bash -. $LKP_SRC/lib/env.sh - -make +. lib/env.sh +write_host +write_shell_profile "export LKP_SRC=$PWD" +write_shell_profile "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" source $(shell_profile) + +. sbin/install-dependencies.sh +make install -- 2.23.0

1 0