- Compass-ci - mailweb.openeuler.org

[PATCH v6 compass-ci 1/3] openeuler: generate the cgz file in container
by Wang Chenglong 30 Nov '20

30 Nov '20

Install packages, set rootfs, pack images package and other actions in docker. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/create-image | 37 ++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/osimage/openeuler/create-image diff --git a/container/osimage/openeuler/create-image b/container/osimage/openeuler/create-image new file mode 100755 index 0000000..631bb14 --- /dev/null +++ b/container/osimage/openeuler/create-image @@ -0,0 +1,37 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +install_packages() +{ + yum -y install --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +setup_rootfs() +{ + ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime + rm -f /etc/machine-id /var/lib/dbus/machine-id + systemd-machine-id-setup +} + +setup_login() { + [ -n "$ROOT_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +pack_cgz() +{ + echo "Packing package. Please wait." + find / ! -path "/tmp/${IMAGE_NAME}" | grep -vf /tmp/files-to-exclude | cpio -o -Hnewc | gzip -9 > /tmp/${IMAGE_NAME} + chmod 644 /tmp/${IMAGE_NAME} +} + +install_packages +setup_rootfs +setup_login +pack_cgz -- 2.23.0

1 0

[PATCH compass-ci] assign-account-email: remove SCHED_HTTPS_HOST
by Wu Fengguang 29 Nov '20

29 Nov '20

It's kind related to lab selection, better set in lab include files. Let's use the default for now. Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- container/mail-robot/lib/assign-account-email.rb | 2 -- 1 file changed, 2 deletions(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index c77a463..0c5f8d4 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -25,8 +25,6 @@ def build_apply_account_email(my_info) my_email: #{my_info['my_email']} my_name: #{my_info['my_name']} my_uuid: #{my_info['my_uuid']} - SCHED_HTTPS_HOST: #{ENV['SCHED_HTTPS_HOST']} - SCHED_HTTPS_PORT: #{ENV['SCHED_HTTPS_PORT']} EOF 2) download lkp-tests and dependencies -- 2.23.0

1 2

[PATCH compass-ci] assign-account-email: minor cleanup
by Wu Fengguang 29 Nov '20

29 Nov '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- container/mail-robot/lib/assign-account-email.rb | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index 4754c4c..c77a463 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -15,8 +15,8 @@ def build_apply_account_email(my_info) You can use the following info to submit jobs: notice: - the followings are ONE-TIME setup - + (1-2) are ONE-TIME setup + 1) setup default config run the following command to add the below setup to default config file @@ -30,7 +30,7 @@ def build_apply_account_email(my_info) EOF 2) download lkp-tests and dependencies - run the following command to install the lkp-test and effect the configuration + run the following command to install and setup lkp-test git clone https://gitee.com/wu_fengguang/lkp-tests.git cd lkp-tests @@ -44,8 +44,8 @@ def build_apply_account_email(my_info) you can also reference to files in lkp-tests/jobs as example. submit jobs, for example: - - submit -m iperf.yaml testbox=vm-2p8g queue=vm-2p8g~\\$USER + + submit -m iperf.yaml testbox=vm-2p8g regards compass-ci -- 2.23.0

1 0

[PATCH compass-ci] doc/manual: optimize indentation for compare.md
by Lu Weitao 28 Nov '20

28 Nov '20

[Why] previous document display confusion on web page Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- ...13\350\257\225\347\273\223\346\236\234.md" | 48 +++++++++---------- 1 file changed, 24 insertions(+), 24 deletions(-) diff --git "a/doc/manual/\345\246\202\344\275\225\346\257\224\350\276\203\346\265\213\350\257\225\347\273\223\346\236\234.md" "b/doc/manual/\345\246\202\344\275\225\346\257\224\350\276\203\346\265\213\350\257\225\347\273\223\346\236\234.md" index 02161f0..a7348d8 100644 --- "a/doc/manual/\345\246\202\344\275\225\346\257\224\350\276\203\346\265\213\350\257\225\347\273\223\346\236\234.md" +++ "b/doc/manual/\345\246\202\344\275\225\346\257\224\350\276\203\346\265\213\350\257\225\347\273\223\346\236\234.md" @@ -23,27 +23,27 @@ dimension: os_version result: -os=openeuler/os_arch=aarch64/pp.iperf.protocol=tcp/pp.iperf.runtime=20/tbox_group=vm-2p8g # other test conditions keep same - - - 20.09 20.03 metric --------------------- ------------------------------ ------------------------------ - fails:runs change fails:runs - | | | - 3:3 -100.0% 0:3 last_state.exit_fail - 3:3 -100.0% 0:3 last_state.is_incomplete_run - 3:3 -100.0% 0:3 last_state.test.iperf.exit_code.127 - 3:3 -100.0% 0:3 stderr.perf_command_failed - - - - 20.09 20.03 metric --------------------- ------------------------------ ------------------------------ - %stddev change %stddev - \ | \ -4.461021e+10 ± 6% -17.4% 3.686392e+10 ± 12% iperf.tcp.receiver.bps -4.461112e+10 ± 6% -17.4% 3.686935e+10 ± 12% iperf.tcp.sender.bps - 94.82 -44.0% 53.10 boot-time.boot - 123.11 -58.4% 51.19 boot-time.idle - 0.00 0 4.87 boot-time.kernel_boot - 4165.50 ± 12% -99.9% 5.00 interrupts.38:GICv3.36.Level.virtio0 + os=openeuler/os_arch=aarch64/pp.iperf.protocol=tcp/pp.iperf.runtime=20/tbox_group=vm-2p8g # other test conditions keep same + + + 20.09 20.03 metric + -------------------- ------------------------------ ------------------------------ + fails:runs change fails:runs + | | | + 3:3 -100.0% 0:3 last_state.exit_fail + 3:3 -100.0% 0:3 last_state.is_incomplete_run + 3:3 -100.0% 0:3 last_state.test.iperf.exit_code.127 + 3:3 -100.0% 0:3 stderr.perf_command_failed + + + + 20.09 20.03 metric + -------------------- ------------------------------ ------------------------------ + %stddev change %stddev + \ | \ + 4.461021e+10 ± 6% -17.4% 3.686392e+10 ± 12% iperf.tcp.receiver.bps + 4.461112e+10 ± 6% -17.4% 3.686935e+10 ± 12% iperf.tcp.sender.bps + 94.82 -44.0% 53.10 boot-time.boot + 123.11 -58.4% 51.19 boot-time.idle + 0.00 0 4.87 boot-time.kernel_boot + 4165.50 ± 12% -99.9% 5.00 interrupts.38:GICv3.36.Level.virtio0 -- 2.23.0

2 4

[PATCH compass-ci] modify README
by Li Ping 28 Nov '20

28 Nov '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- README.md | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 644e9ba..6b96e87 100644 --- a/README.md +++ b/README.md @@ -5,26 +5,30 @@ ### 概念 -Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github, Gitee, Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务。通过 Compass-CI，社区开发者可以将开源软件快速引入 openEuler 社区，并补充更多的测试用例，共同构建一个开放、完整的开源软件生态系统。 +Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github, Gitee, Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务--。Compass-CI 基于开源软件PR进行自动化测试(包括构建测试，软件包自带用例测试等)，共同构建一个开放、完整的开源软件生态测试系统。 ### 功能描述 - **测试服务** - 支持开发者基于本地设备开发，往 github 提交代码，Compass-CI 自动获取代码开展测试，并向开发者反馈测试结果。 + 支持开发者基于本地设备开发，往远端代码仓提交代码时触发Compass-CI 自动获取代码开展测试，并向开发者反馈测试结果。 - **调测环境登录** - Compass-CI 提供 SSH 登录能力，测试过程中如果测出问题，开发者可根据需要登录环境进行调测。 + Compass-CI 提供 SSH 登录能力，测试过程中如果测试任务未执行成功，开发者可根据需要登录环境进行调测。 - **测试结果分析** - Compass-CI 记录历史测试结果，对外提供 web 及命令行接口，支持开发者针对已有的测试结果进行分析，挖掘影响测试结果的因素。 + Compass-CI 提供 web 及命令行接口，支持开发者针对历史测试结果进行分析和比较，挖掘影响测试结果的因素。 + +- **测试结果复现** + + Compass-CI 把测试过程中的各项环境参数保存在job文件中，开发者可以通过重新提交job复现测试结果。 - **辅助定位** - Compass-CI 在测试过程中可以自动识别错误信息，触发基于 git tree 的测试，找出引入问题模块的变化点。 + Compass-CI 可以识别自动化构建测试过程中的错误，触发基于 git tree 的测试，找出引入问题的commit。 ### 架构简介 @@ -35,7 +39,7 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上 - **服务层** - 提供测试服务、环境登录、结果分析、辅助定界、用户项目注册能力，为本项目的主要功能。 + 本项目的主要功能是：提供测试服务、环境登录、结果分析、结果复现、辅助定界。 - **支撑层** @@ -47,7 +51,7 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上 - **资源层** - 提供硬件设备，作为Compass-CI 服务部署的设备以及测试的物理设备、虚拟机、容器资源；提供测试过程中依赖软件仓库及依赖仓库、部署服务器。 + 提供作为Compass-CI 服务部署及测试的物理设备、虚拟设备、容器资源；提供测试过程中的依赖软件及依赖仓库、部署服务器。 ### 应用场景 @@ -64,6 +68,10 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上测试过程中，全面监控系统运行信息（CPU/MEM/IO/网络等），对测试过程中的数据做快照归档，提供多次测试之间快照数据分析对比能力，协助开发者对测试结果开展分析，找出影响测试结果的因素。 +- **复现测试结果** + + 将测试过程中的各项参数(包括环境参数，软件测试参数)完整保留，存放于测试结果文件job.yaml中，支持开发者通过重新提交job复现测试结果。 + - **辅助定界** 测试过程中，发现有 Bug 时，自动触发 Regression 机制，找出首次引入问题的 Commit 信息。 @@ -179,7 +187,7 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上建议用户根据具体需要对 rootfs 进行裁剪。 -## 呼吁合作 +## 欢迎合作 - 增强 git bisect 能力 - 增强数据分析能力 - 增强数据结果可视化能力 -- 2.23.0

1 0

[PATCH v1 compass-ci] delimiter: add delimiter account info to bisect job
by Cao Xueliang 28 Nov '20

28 Nov '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/constants.rb | 1 + src/delimiter/utils.rb | 14 +++++++++++--- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/src/delimiter/constants.rb b/src/delimiter/constants.rb index 93b633d..04f87f4 100644 --- a/src/delimiter/constants.rb +++ b/src/delimiter/constants.rb @@ -6,6 +6,7 @@ START_PROCESS_COUNT = 10 GIT_MIRROR_HOST = ENV['GIT_MIRROR_HOST'] || '172.17.0.1' MONITOR_HOST = ENV['MONITOR_HOST'] || '172.17.0.1' MONITOR_PORT = ENV['MONITOR_PORT'] || '11310' +DELIMITER_ACCONUT = ENV['DELIMITER_ACCONUT'] || 'compass-ci(a)qq.com' TMEP_GIT_BASE = '/c/public_git' DELIMITER_TASK_QUEUE = 'delimiter' BISECT_RUN_SCRIPT = "#{ENV['CCI_SRC']}/src/delimiter/find-commit/bisect_run_script.rb" diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 679c8d9..90c3cce 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -88,16 +88,24 @@ module Utils return 'good' end + def get_account_info + ESQuery.new(index: 'accounts').query_by_id(DELIMITER_ACCONUT) + end + def init_job_content(job_id) es = ESQuery.new job = es.query_by_id(job_id) raise "es query job id: #{job_id} failed!" unless job - job['bad_job_id'] = job_id + account_info = get_account_info + raise "query #{DELIMITER_ACCONUT} account info failed!" unless account_info + job['suite'] = 'bisect' + job['my_uuid'] = account_info['my_uuid'] + job['my_name'] = account_info['my_name'] + job['my_email'] = account_info['my_email'] + job['bad_job_id'] = job_id - job.delete('id') - job.delete('uuid') job.delete('error_ids') job.delete('start_time') job.delete('end_time') -- 2.23.0

1 0

[PATCH compass-ci] delimiter: add common account info to bisect job
by Cao Xueliang 28 Nov '20

28 Nov '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 679c8d9..2be99b5 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -88,16 +88,24 @@ module Utils return 'good' end + def get_account_info + ESQuery.new(index: 'accounts').query_by_id('compass-ci(a)qq.com') + end + def init_job_content(job_id) es = ESQuery.new job = es.query_by_id(job_id) raise "es query job id: #{job_id} failed!" unless job + account_info = get_account_info + raise "query compass-ci(a)qq.com account info failed!" unless account_info + job['bad_job_id'] = job_id job['suite'] = 'bisect' + job['my_email'] = account_info['my_email'] + job['my_name'] = account_info['my_name'] + job['my_uuid'] = account_info['my_uuid'] - job.delete('id') - job.delete('uuid') job.delete('error_ids') job.delete('start_time') job.delete('end_time') -- 2.23.0

2 2

[PATCH v2 compass-ci 2/2] lib/compare_matrixes.rb: refactor format optimization
by Lu Kaiyi 28 Nov '20

28 Nov '20

reduce unnecessary indent to simplify code Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_matrixes.rb | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index d8112e9..7a9870a 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -177,7 +177,7 @@ def get_matrixes_values(matrixes_list, options) end def remove_unchanged_field(matrixes_values) - # remove unchanged field from matrixes valus and remove :changed key + # remove unchanged field from matrixes values and remove :changed key # matrixes_values.each_key do |success| matrixes_values[success].delete_if do |field| @@ -316,21 +316,20 @@ def get_decimal_length(number, length) end def get_suitable_number_str(number, length, format_pattern) - # if number string length can't < target length, + # if number string length is no less than target length, # transform number string to scientific notation string format_str = format(format_pattern, number) return format_str if format_str.length <= length decimal_length = get_decimal_length(number, length) - unless decimal_length.negative? - scientific_str = format("%.#{decimal_length}e", number).sub('e+0', 'e+').sub('e-0', 'e-') - lack_length = length - scientific_str.length - unless lack_length.negative? - return scientific_str + ' ' * lack_length - end - end - format_str + return format_str if decimal_length.negative? + + scientific_str = format("%.#{decimal_length}e", number).sub('e+0', 'e+').sub('e-0', 'e-') + lack_length = length - scientific_str.length + return format_str if lack_length.negative? + + return scientific_str + ' ' * lack_length end # Colorize -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/compare_matrixes.rb: refactor performance optimization
by Lu Kaiyi 28 Nov '20

28 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_matrixes.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index dd9b86e..d8112e9 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -34,7 +34,7 @@ end def fill_missing_with_zeros(value_list, matrix_size) value_list ||= [0] * matrix_size - value_list << 0 while value_list.size < matrix_size + value_list.concat([0] * (matrix_size - value_list.size)) value_list end -- 2.23.0

1 0

[PATCH compass-ci] lib/compare_matrixes.rb: refactor format optimization
by Lu Kaiyi 28 Nov '20

28 Nov '20

reduce unnecessary indent to simplify code Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_matrixes.rb | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index d8112e9..23c7931 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -177,7 +177,7 @@ def get_matrixes_values(matrixes_list, options) end def remove_unchanged_field(matrixes_values) - # remove unchanged field from matrixes valus and remove :changed key + # remove unchanged field from matrixes values and remove :changed key # matrixes_values.each_key do |success| matrixes_values[success].delete_if do |field| @@ -316,21 +316,20 @@ def get_decimal_length(number, length) end def get_suitable_number_str(number, length, format_pattern) - # if number string length can't < target length, + # if number string length no less than target length, # transform number string to scientific notation string format_str = format(format_pattern, number) return format_str if format_str.length <= length decimal_length = get_decimal_length(number, length) - unless decimal_length.negative? - scientific_str = format("%.#{decimal_length}e", number).sub('e+0', 'e+').sub('e-0', 'e-') - lack_length = length - scientific_str.length - unless lack_length.negative? - return scientific_str + ' ' * lack_length - end - end - format_str + return format_str if decimal_length.negative? + + scientific_str = format("%.#{decimal_length}e", number).sub('e+0', 'e+').sub('e-0', 'e-') + lack_length = length - scientific_str.length + return format_str if lack_length.negative? + + return scientific_str + ' ' * lack_length end # Colorize -- 2.23.0

2 1

[PATCH lkp-tests] spec/submit_spec.rb: update submit_job
by Hu Xuejiao 28 Nov '20

28 Nov '20

[why] Originally submit_job is too complicated, it need to be simplified. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit_spec.rb | 55 +++++++++++++++++---------------------------- 1 file changed, 21 insertions(+), 34 deletions(-) diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 4096cb73..e2361b13 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -20,40 +20,27 @@ def traverse_file(output_dir) end end -def submit_job(submit_item) - submit_yaml_path = File.join("#{LKP_SRC}/spec/submit", submit_item) - Dir.glob("#{submit_yaml_path}/*.yaml").each do |yaml_file| - output_dir = File.join(submit_yaml_path, File.basename(yaml_file, '.yaml')) - submit_cmd = [ - "#{LKP_SRC}/sbin/submit", - '-o', output_dir, - '-s', 'lab: spec_lab', - '-s', 'testbox: vm-2p8g--spec_submit', - yaml_file - ] - system(*submit_cmd) - traverse_file(output_dir) +def submit_job() + submit_path = "#{LKP_SRC}/spec/submit" + Dir.foreach(submit_path) do |sub| + next if sub == '.' || sub == '..' + + path = "#{submit_path}/#{sub}" + if File.directory?(path) + Dir.glob("#{path}/*.yaml").each do |yaml_file| + output_dir = File.join(path, File.basename(yaml_file, '.yaml')) + submit_cmd = [ + "#{LKP_SRC}/sbin/submit", + '-o', output_dir, + '-s', 'lab: spec_lab', + '-s', 'testbox: vm-2p8g--spec_submit', + yaml_file + ] + system(*submit_cmd) + traverse_file(output_dir) + end + end end end -describe 'submit job spec' do - it 'link jobs spec' do - submit_job('link_jobs') - end - - it 'link matrix' do - submit_job('matrix') - end - - it 'separate yaml spec' do - submit_job('separate_yaml') - end - - it 'job on fail' do - submit_job('job_on_fail') - end - - it 'merge yaml' do - submit_job('merge_yaml') - end -end +submit_job -- 2.23.0

2 1

[PATCH lkp-tests] submit: fix can not set queue
by Wei Jihui 28 Nov '20

28 Nov '20

[why] it can not set my queue when submit with --my-queue, because user maybe set tbox at config yaml, not using command lines [how] each_jobs will load all config yaml, so move set queue under it. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/submit | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/sbin/submit b/sbin/submit index 23518eb0..cc2d940c 100755 --- a/sbin/submit +++ b/sbin/submit @@ -99,10 +99,6 @@ def find_jobfile(jobfile) exit 1 end -def add_my_queue(job) - job['queue'] = "#{job['tbox_group']}~#{ENV['USER']}" -end - ARGV.each do |jobfile| jobfile = find_jobfile(jobfile) jobs = Job2sh.new @@ -112,12 +108,13 @@ ARGV.each do |jobfile| jobs[:expand_params] = true jobs['testbox'] = opt_set_key_value['testbox'] if opt_set_key_value['testbox'] jobs['tbox_group'] = tbox_group(jobs['testbox']) if jobs.include?('testbox') - add_my_queue(jobs) if opt_my_queue jobs['node_roles'] ||= 'server client' if jobs['cluster'] jobs.each_jobs do |job| raise 'Please configure SCHED_HOST' unless job['SCHED_HOST'] raise 'Please configure SCHED_PORT' unless job['SCHED_PORT'] + job['queue'] = "#{jobs['tbox_group']}~#{ENV['USER']}" if opt_my_queue + job.add_pp job.add_monitors job.add_define_files if opt_auto_define_files -- 2.23.0

1 0

[PATCH compass-ci] lib/compare_matrixes.rb: refactor performance optimization
by Lu Kaiyi 28 Nov '20

28 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_matrixes.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index dd9b86e..d8112e9 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -34,7 +34,7 @@ end def fill_missing_with_zeros(value_list, matrix_size) value_list ||= [0] * matrix_size - value_list << 0 while value_list.size < matrix_size + value_list.concat([0] * (matrix_size - value_list.size)) value_list end -- 2.23.0

1 0

[PATCH compass-ci] web_backend.rb: fix that lost some args
by Li Yuanchao 28 Nov '20

28 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 4846dc5..e548d61 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -148,7 +148,7 @@ def get_es_must(params) must end -def _get_groups_matrices(query, dimension) +def do_get_groups_matrices(query, dimension, total, size, from) result = es_query(query) matrices, suites_list = combine_group_query_data(result, dimension) while matrices.empty? @@ -181,7 +181,7 @@ def get_groups_matrices(conditions, dimension, must, size, from) }] } - matrices, suites_list = _get_groups_matrices(query, dimension) + do_get_groups_matrices(query, dimension, total, size, from) end def get_compare_body(params) -- 2.23.0

2 2

[PATCH lkp-tests] spec/submit_spec.rb: update submit_job
by Hu Xuejiao 28 Nov '20

28 Nov '20

[why] Originally submit_job is too complicated, it need to be simplified. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit_spec.rb | 55 +++++++++++++++++---------------------------- 1 file changed, 21 insertions(+), 34 deletions(-) diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 4096cb73..28aca220 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -20,40 +20,27 @@ def traverse_file(output_dir) end end -def submit_job(submit_item) - submit_yaml_path = File.join("#{LKP_SRC}/spec/submit", submit_item) - Dir.glob("#{submit_yaml_path}/*.yaml").each do |yaml_file| - output_dir = File.join(submit_yaml_path, File.basename(yaml_file, '.yaml')) - submit_cmd = [ - "#{LKP_SRC}/sbin/submit", - '-o', output_dir, - '-s', 'lab: spec_lab', - '-s', 'testbox: vm-2p8g--spec_submit', - yaml_file - ] - system(*submit_cmd) - traverse_file(output_dir) +def submit_job() + submit_path = "#{LKP_SRC}/spec/submit" + Dir.foreach(submit_path) do | sub | + next if sub == '.' || sub == '..' + + path = "#{submit_path}/#{sub}" + if File.directory?(path) + Dir.glob("#{path}/*.yaml").each do |yaml_file| + output_dir = File.join(path, File.basename(yaml_file, '.yaml')) + submit_cmd = [ + "#{LKP_SRC}/sbin/submit", + '-o', output_dir, + '-s', 'lab: spec_lab', + '-s', 'testbox: vm-2p8g--spec_submit', + yaml_file + ] + system(*submit_cmd) + traverse_file(output_dir) + end + end end end -describe 'submit job spec' do - it 'link jobs spec' do - submit_job('link_jobs') - end - - it 'link matrix' do - submit_job('matrix') - end - - it 'separate yaml spec' do - submit_job('separate_yaml') - end - - it 'job on fail' do - submit_job('job_on_fail') - end - - it 'merge yaml' do - submit_job('merge_yaml') - end -end +submit_job -- 2.23.0

1 0

[PATCH v4 compass-ci] answerback-email: fix bugs
by Luan Shengde 28 Nov '20

28 Nov '20

1. set default option -h for tool [why] the tool will show help message if no option specified the tool will get no email address without both '-e' and '-f' [how] add '-h' as default option if no option specified options will be reset to '-h' if both '-e' and '-f' are not specified 2. fix undefined method 'empty' [why] the original value is defined nil for my_email when execute answerback-email with no -e or -f, it will throw error as: raceback (most recent call last): 2: from ./answerback-email.rb:213:in `<main>' 1: from ./answerback-email.rb:184:in `send_account' ./answerback-email.rb:155:in `check_my_email': undefined method `empty?' for nil:NilClass (NoMethodError) [how] use '.nil?' instead of '.empty?' Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index c6bfdd3..badbdea 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -80,7 +80,7 @@ end options = OptionParser.new do |opts| opts.banner = 'Usage: answerback-mail.rb [-e|--email email] [-n|--name name] ' - opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [--login y|n] [--update]\n" + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [-l|--login y|n] [-u|--update]\n" opts.banner += ' answerback-mail.rb [-f|--raw-email email_file] ' opts.banner += "[-g|--gen-sshkey] [--login y|n] [--update]\n" opts.banner += " -e|-f is required when applying account or updating account\n" @@ -121,12 +121,15 @@ options = OptionParser.new do |opts| end opts.on('-l value', '--login value', 'enable/disable login, value: y|n') do |value| - if value.downcase == 'y' + case value + when 'y', 'Y' conf_info['enable_login'] = true - elsif value.downcase == 'n' + when 'n', 'N' conf_info['enable_login'] = false else - raise 'invalid parameter, please use y|n' + puts "-l: bad value #{value}, please use y|n\n\n" + puts opts.banner + return 0 end end @@ -136,6 +139,12 @@ options = OptionParser.new do |opts| end end +if ARGV.empty? + ARGV << '-h' +elsif (['-e', '-f'] - ARGV).eql? ['-e', '-f'] + ARGV.clear + ARGV << '-h' +end options.parse!(ARGV) def apply_account(my_info, conf_info) @@ -152,7 +161,7 @@ def check_my_email(my_info) message += "use -e to add an email address for applying account\n" message += 'or use -f to add an email file' - raise message if my_info['my_email'].empty? + raise message if my_info['my_email'].nil? end def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] answerback-email: fix bugs
by Luan Shengde 28 Nov '20

28 Nov '20

1. set default option -h for tool [why] the tool will show help message if no option specified the tool will get no email address without both '-e' and '-f' [how] add '-h' as default option if no option specified options will be reset to '-h' if both '-e' and '-f' are not specified 2. fix undefined method 'empty' [why] the original data is defined nil, but not empty for my_email when execute answerback-email with no -e or -f, it will throw error as: raceback (most recent call last): 2: from ./answerback-email.rb:213:in `<main>' 1: from ./answerback-email.rb:184:in `send_account' ./answerback-email.rb:155:in `check_my_email': undefined method `empty?' for nil:NilClass (NoMethodError) Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index badbdea..874003e 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -219,4 +219,4 @@ def store_account_info(my_info) es.put_source_by_id(my_info['my_email'], my_info) end -send_account(my_info, conf_info, email_info, my_info_es, stdin_info) +# send_account(my_info, conf_info, email_info, my_info_es, stdin_info) -- 2.23.0

1 1

[PATCH v3 compass-ci 1/2] answerback-email: fix bugs
by Luan Shengde 28 Nov '20

28 Nov '20

1. set default option -h for tool [why] the tool will show help message if no option specified the tool will get no email address without both '-e' and '-f' [how] add '-h' as default option if no option specified options will be reset to '-h' if both '-e' and '-f' are not specified 2. fix undefined method 'empty' [why] the original data is defined nil, but not empty for my_email when execute answerback-email with no -e or -f, it will throw error as: raceback (most recent call last): 2: from ./answerback-email.rb:213:in `<main>' 1: from ./answerback-email.rb:184:in `send_account' ./answerback-email.rb:155:in `check_my_email': undefined method `empty?' for nil:NilClass (NoMethodError) Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index c6bfdd3..badbdea 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -80,7 +80,7 @@ end options = OptionParser.new do |opts| opts.banner = 'Usage: answerback-mail.rb [-e|--email email] [-n|--name name] ' - opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [--login y|n] [--update]\n" + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [-l|--login y|n] [-u|--update]\n" opts.banner += ' answerback-mail.rb [-f|--raw-email email_file] ' opts.banner += "[-g|--gen-sshkey] [--login y|n] [--update]\n" opts.banner += " -e|-f is required when applying account or updating account\n" @@ -121,12 +121,15 @@ options = OptionParser.new do |opts| end opts.on('-l value', '--login value', 'enable/disable login, value: y|n') do |value| - if value.downcase == 'y' + case value + when 'y', 'Y' conf_info['enable_login'] = true - elsif value.downcase == 'n' + when 'n', 'N' conf_info['enable_login'] = false else - raise 'invalid parameter, please use y|n' + puts "-l: bad value #{value}, please use y|n\n\n" + puts opts.banner + return 0 end end @@ -136,6 +139,12 @@ options = OptionParser.new do |opts| end end +if ARGV.empty? + ARGV << '-h' +elsif (['-e', '-f'] - ARGV).eql? ['-e', '-f'] + ARGV.clear + ARGV << '-h' +end options.parse!(ARGV) def apply_account(my_info, conf_info) @@ -152,7 +161,7 @@ def check_my_email(my_info) message += "use -e to add an email address for applying account\n" message += 'or use -f to add an email file' - raise message if my_info['my_email'].empty? + raise message if my_info['my_email'].nil? end def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) -- 2.23.0

1 1

[PATCH v2 compass-ci] answerback-email: fix bugs
by Luan Shengde 28 Nov '20

28 Nov '20

1. set default option -h for tool [why] the tool will show help message if no option specified the tool will get no email address without both '-e' and '-f' [how] add '-h' as default option if no option specified options will be reset to '-h' if both '-e' and '-f' are not specified 2. fix undefined method 'empty' [why] the original data is defined nil, but not empty for my_email when execute answerback-email with no -e or -f, it will throw error as: raceback (most recent call last): 2: from ./answerback-email.rb:213:in `<main>' 1: from ./answerback-email.rb:184:in `send_account' ./answerback-email.rb:155:in `check_my_email': undefined method `empty?' for nil:NilClass (NoMethodError) Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index c6bfdd3..badbdea 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -80,7 +80,7 @@ end options = OptionParser.new do |opts| opts.banner = 'Usage: answerback-mail.rb [-e|--email email] [-n|--name name] ' - opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [--login y|n] [--update]\n" + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [-l|--login y|n] [-u|--update]\n" opts.banner += ' answerback-mail.rb [-f|--raw-email email_file] ' opts.banner += "[-g|--gen-sshkey] [--login y|n] [--update]\n" opts.banner += " -e|-f is required when applying account or updating account\n" @@ -121,12 +121,15 @@ options = OptionParser.new do |opts| end opts.on('-l value', '--login value', 'enable/disable login, value: y|n') do |value| - if value.downcase == 'y' + case value + when 'y', 'Y' conf_info['enable_login'] = true - elsif value.downcase == 'n' + when 'n', 'N' conf_info['enable_login'] = false else - raise 'invalid parameter, please use y|n' + puts "-l: bad value #{value}, please use y|n\n\n" + puts opts.banner + return 0 end end @@ -136,6 +139,12 @@ options = OptionParser.new do |opts| end end +if ARGV.empty? + ARGV << '-h' +elsif (['-e', '-f'] - ARGV).eql? ['-e', '-f'] + ARGV.clear + ARGV << '-h' +end options.parse!(ARGV) def apply_account(my_info, conf_info) @@ -152,7 +161,7 @@ def check_my_email(my_info) message += "use -e to add an email address for applying account\n" message += 'or use -f to add an email file' - raise message if my_info['my_email'].empty? + raise message if my_info['my_email'].nil? end def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) -- 2.23.0

1 1

[PATCH compass-ci] params_group.rb: don't use os_version as filter when dimension is os
by Li Yuanchao 28 Nov '20

28 Nov '20

When compare with dimension 'os', different operating systems can not have the same os_version, so there is no data for compare. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/params_group.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/params_group.rb b/lib/params_group.rb index 48d651b..cde40af 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -98,6 +98,7 @@ def get_group_dimension_params(job, dimensions) dimension_list = [] dimensions.each do |dimension| dimension_list << all_group_params.delete(dimension) if all_group_params.key?(dimension) + all_group_params.delete('os_version') if dimension == 'os' end [all_group_params, dimension_list.join('|')] end -- 2.23.0

1 0

[PATCH compass-ci] lib/matrix2.rb: refactor performance optimization
by Lu Kaiyi 28 Nov '20

28 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 82cf033..2507b77 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -61,9 +61,7 @@ def create_stats(result_root) end def samples_fill_missing_zeros(value, size) - samples = value || [0] * size - samples << 0 while samples.size < size - samples + value.concat([0] * (size - value.size)) end # input: job_list -- 2.23.0

1 0

[PATCH compass-ci] scheduler: fix condition when setting default queue
by Ren Wen 28 Nov '20

28 Nov '20

Hash.*has_key?* returns true although there is no value of the key. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index e836d1f..55bac7f 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -247,7 +247,7 @@ class Job end private def set_queue - return if @hash.has_key?("queue") + return unless self["queue"].empty? # set default value self["queue"] = tbox_group -- 2.23.0

3 3

[PATCH compass-ci] answerback-email: fix bugs
by Luan Shengde 28 Nov '20

28 Nov '20

1. add default option for tool [why] the tool will show help message if no option specified [how] if no option specified, it will add '-h' as default option 2. fix undefined method 'empty' [why] the original data is defined nil, but not empty for my_email when execute answerback-email with no -e or -f, it will throw error as: raceback (most recent call last): 2: from ./answerback-email.rb:213:in `<main>' 1: from ./answerback-email.rb:184:in `send_account' ./answerback-email.rb:155:in `check_my_email': undefined method `empty?' for nil:NilClass (NoMethodError) [how] use '.nil?' instead of '.empty?' Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index c6bfdd3..1aaad75 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -136,7 +136,10 @@ options = OptionParser.new do |opts| end end -options.parse!(ARGV) +if ARGV.empty? + ARGV << '-h' + options.parse!(ARGV) +end def apply_account(my_info, conf_info) apply_info = {} @@ -152,7 +155,7 @@ def check_my_email(my_info) message += "use -e to add an email address for applying account\n" message += 'or use -f to add an email file' - raise message if my_info['my_email'].empty? + raise message if my_info['my_email'].nil? end def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) -- 2.23.0

1 0

[PATCH v2 compass-ci] scheduler: fix the paths to find cluster config file
by Ren Wen 27 Nov '20

27 Nov '20

To find cluster config file at *lab-z9/cluster/* first, *lkp-tests/cluster/* otherwise. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/constants.cr | 3 +++ src/scheduler/submit_job.cr | 14 +++++++++----- 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 1b37333..1afdda0 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -24,6 +24,9 @@ OS_HTTP_PORT = (ENV.has_key?("OS_HTTP_PORT") ? ENV["OS_HTTP_PORT"] : 8000).to_i3 SRV_OS = "/srv/os" SRV_INITRD = "/srv/initrd" +CCI_REPOS = "/c" +LAB_REPO = "lab-z9" + INITRD_HTTP_PREFIX = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" OS_HTTP_PREFIX = "http://#{OS_HTTP_HOST}:#{OS_HTTP_PORT}" SCHED_HTTP_PREFIX = "http://#{SCHED_HOST}:#{SCHED_PORT}" diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 2836275..c60b99d 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -15,7 +15,7 @@ class Sched else cluster_config = get_cluster_config(cluster_file, job.lkp_initrd_user, - job.os_arch) + job.os_arch).not_nil! return submit_cluster_job(job, cluster_config) end rescue ex @@ -152,12 +152,16 @@ class Sched @es.set_job_content(job) end - # get cluster config using own lkp_src cluster file, - # a hash type will be returned def get_cluster_config(cluster_file, lkp_initrd_user, os_arch) lkp_src = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) - cluster_file_path = Path.new(lkp_src, "cluster", cluster_file) - return YAML.parse(File.read(cluster_file_path)) + + cluster_file_paths = [ + Path.new(CCI_REPOS, LAB_REPO, "cluster", cluster_file), + Path.new(lkp_src, "cluster", cluster_file) + ] + cluster_file_paths.each do |f| + return YAML.parse(File.read(f)) if File.file?(f) + end end def get_commit_date(job) -- 2.23.0

1 0

[PATCH v2 compass-ci 4/4] user-client/maintain: send report mail
by Liu Yinsi 27 Nov '20

27 Nov '20

call $CCI_SRC/lib/mail_client.rb to mail. 'LOG' is a file => walk-test.report, job result will write in it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- user-client/maintain/walk-os-test/walk-os-iperf-test | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index 17edb60..923d1ed 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -36,6 +36,17 @@ OS_MOUNT = %w[cifs initramfs].freeze EMAIL_ADDRESS = (ENV['USER']).to_s.freeze +def mail_report + message = File.read(LOG).chomp + data = " + subject: os rootfs test report + to: #{EMAIL_ADDRESS} + body: #{message}" + + email = MailClient.new() + email.send_mail(data) +end + def write_report(report_all) File.open(LOG, 'a') do |f| log = Logger.new(f, 'weekly') -- 2.23.0

1 0