- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] lib/git_bisect: use "--first-parent" option when start bisect
by Cao Xueliang 19 Jan '21

19 Jan '21

1. The alpine 3.13 support to use the latest git which has the "--first-parent" option. 2. This option is useful in avoiding false positives when a merged branch contained broken or non-buildable commits, but the merge itself was OK. Ref: https://git-scm.com/docs/git-bisect Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/Dockerfile | 6 +++--- lib/git_bisect.rb | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/container/delimiter/Dockerfile b/container/delimiter/Dockerfile index ab7519d..cc57320 100644 --- a/container/delimiter/Dockerfile +++ b/container/delimiter/Dockerfile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM alpine:3.11 +FROM alpine:3.13 RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ apk update && \ @@ -9,5 +9,5 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' RUN umask 002 && \ - echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ - gem install rest-client activesupport git json yaml threadpool elasticsearch faye-websocket terminal-table mail io-console + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ + gem install rest-client activesupport git json yaml elasticsearch faye-websocket terminal-table mail io-console diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index 2fee60c..e577afd 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -82,7 +82,7 @@ class GitBisect puts "bad_commit: #{@bad_commit}" puts "good_commit: #{@good_commit}" - result = `git -C #{@work_dir} bisect start #{@bad_commit} #{@good_commit}` + result = `git -C #{@work_dir} bisect start --first-parent #{@bad_commit} #{@good_commit}` temp = result.split(/\n/) if temp[0].include? 'Bisecting' result = `git -C #{@work_dir} bisect run #{BISECT_RUN_SCRIPT} #{@bad_job_id} #{(a)error_id.inspect} #{@work_dir}` -- 2.23.0

1 0

Re: [PATCH lkp-tests] lib/job.sh: modify the generated file suffix
by Lu Weitao 19 Jan '21

19 Jan '21

On Thu, Jan 14, 2021 at 03:44:10PM +0800, Li Ping wrote: >[why] >user extract $program result actually generating a YAML file. >and compass-ci/sbin/result2stats have already support merge yaml file > >Signed-off-by: Li Ping <1477412247(a)qq.com> >--- > lib/job.sh | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > >diff --git a/lib/job.sh b/lib/job.sh >index 03c49e0ad..dc65f68d8 100755 >--- a/lib/job.sh >+++ b/lib/job.sh >@@ -318,5 +318,5 @@ run_test() > run_target_stats() > { > local script_name=$1 >- $LKP_SRC/stats/$script_name < $TMP_RESULT_ROOT/$script_name > $TMP_RESULT_ROOT/$script_name.json >+ $LKP_SRC/stats/$script_name < $TMP_RESULT_ROOT/$script_name > $TMP_RESULT_ROOT/$script_name.yaml it's only write the key-value into $script_name.yaml like: hackbench.throughput: 18277.704125399672 actually we need a matrix with josn file like: { "hackbench.throughput": [ 18277.704125399672 ] } or yaml like: hackbench.throughput: - 18277.704125399672 so we'd better use like: # make sure ENV include RESULT_ROOT # make sure lkp-tests/stats/$script_name exists $LKP_SRC/stats/wrapper $script_name, it will output $RESULT_ROOT/$script_name.json Thanks, Weitao ~ > } >-- >2.23.0 >

2 6

[PATCH v2 compass-ci] doc: sparrow/README.md: update format and content
by Liu Yinsi 18 Jan '21

18 Jan '21

1. git config, this info will be used to build account info, fix verification problem when submit job. 2. setenforce 0, if not set selinux, cci-network.service will not work right. 3. update rootfs download link. 4. division operation instruction Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 66 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 46 insertions(+), 20 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index c7a80e7..89c43d8 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -24,30 +24,49 @@ >openEuler 系统安装详细操作请参考[添加测试用例](https://openeuler.org/zh/docs/20.03_LTS/docs/Installati… ### 操作指导 + +#### 本地搭建compass-ci + 1. 登录 openEuler 系统 -2. 创建工作目录并设置文件权限 +2. 配置git账号 + ```bash + git config --golbal user.name "XXX" + git config --golbal user.email "xxx(a)aa.com" + ``` + +3. 设置文件权限并关闭SELINUX ```bash - mkdir demo && cd demo && umask 002 + umask 002 && setenforce 0 ``` -3. 克隆 compass-ci 项目代码到 demo 目录 + + >**说明：** + >setenforce 0 只是暂时禁用SELINUX，需要修改/etc/selinux/config中的SELINUX=enforcing改为SELINUX=permissive或disabled才能长期有效 + >umask 002 只是暂时设置umask值，需要修改/etc/bashrc中的umask值为002才能长期有效 + +4. 创建工作目录并克隆 compass-ci 项目代码 ```bash + mkdir /c/ && ln -s /c/compass-ci /c/cci git clone https://gitee.com/wu_fengguang/compass-ci.git ``` -4. 执行一键部署脚本 install-tiny + +5. 执行一键部署脚本 install-tiny ```bash cd compass-ci/sparrow && ./install-tiny ``` -5. 生成lkp-aarch64.cgz压缩包 + +#### 提交测试任务前的准备 + +1. 生成lkp-aarch64.cgz压缩包 ```bash cd /c/compass-ci/container/lkp-initrd && ./run ``` -6. 验证账号 +2. 验证账号 ```bash cd /c/compass-ci/sbin && ./build-my-info.rb - 按照提示输入账号名和邮箱即可 + ``` -7. 测试环境是否可以提交job测试 +3. 测试环境是否可以提交job测试 ```bash submit iperf.yaml testbox=vm-2p8g ``` @@ -55,33 +74,40 @@ 执行上述命令正常情况下会提示信息如下: submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - compass-ci搭建完毕，执行步骤8下载所需要的rootfs文件就可以开始进行测试了。 + compass-ci搭建完毕，执行步骤4下载所需要的rootfs文件就可以开始进行测试了。 -8. 下载rootfs（以下载openeuler/aarch64/20.03为例，需要在哪个系统上测试就去对应的/srv/os/目录下使用wget命令下载cgz文件包） +4. 下载rootfs文件（根据所需要的rootfs在[该目录](http://124.90.34.227:11300/os/)下获取对应版本的cgz文件） ```bash - mkdir -p /srv/os/openeuler/aarch64/ - cd /srv/os/openeuler/aarch64 - wget http://124.90.34.227:11300/os/test/openeuler/aarch64/20.03.cgz + mkdir -p /srv/os/openeuler/aarch64/20.03 + cd /srv/os/openeuler/aarch64/20.03 + wget http://124.90.34.227:11300/os/openeuler/aarch64/20.03.cgz ``` -9. 解压cgz包 +5. 解压rootfs cgz 文件 ```bash gzip -dc 20.03.cgz | cpio -idv ``` -10. [使用 compass-ci 平台测试开源项目](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… +#### 提交测试任务到本地compass-ci +本文以测试用例iperf.yaml为例 -11. [编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testca… +1. [使用 compass-ci 平台测试开源项目](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… + +2. [编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testca… + +3. 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… + ```bash + submit iperf.yaml + ``` -12. 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… - 注意：os_mount 必须指定为cifs +#### 运行测试任务并查看任务结果 -13. 运行测试任务 +1. 运行测试任务 ```bash cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -14. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… +2. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… 本地搭建compass-ci用户可在srv/result目录下根据yaml名称/日期/$testbox/$os-$os-version-$os-arch/$job_id查看output文件(可用tab键自动补全多级目录方便查找) 以查看iperf任务结果为例操如下： ```bash -- 2.23.0

1 0

[PATCH compass-ci] compare_error_messages.rb/error_messages.rb: optimize string catenate
by Lin Jiaxin 18 Jan '21

18 Jan '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 2 +- lib/error_messages.rb | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb index d9171bf..237b2c9 100644 --- a/lib/compare_error_messages.rb +++ b/lib/compare_error_messages.rb @@ -73,7 +73,7 @@ def format_error_messages(error_messages, new_error_ids) new_error_number = 0 error_messages.each do |k, v| - if new_error_ids.include?('build-pkg.' + build_pkg_error_id(k)) + if new_error_ids.include?("build-pkg.#{build_pkg_error_id(k)}") new_error_number += 1 formatted_error_messages = add_sign(formatted_error_messages, '>>', v) else diff --git a/lib/error_messages.rb b/lib/error_messages.rb index d83d890..50f9637 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -44,7 +44,7 @@ class ErrorMessages error_messages_by_error_id = [] error_messages = obtain_error_messages error_messages.each do |k, v| - if ('build-pkg.' + build_pkg_error_id(k)) == error_id + if "build-pkg.#{build_pkg_error_id(k)}" == error_id error_messages_by_error_id += return_error_line ? [k] : v.to_a end end -- 2.23.0

1 0

[PATCH v3 compass-ci 3/3] compare_error_messages.rb/error_messages.rb: optimize string catenate
by Lin Jiaxin 18 Jan '21

18 Jan '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 2 +- lib/error_messages.rb | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb index d9171bf..237b2c9 100644 --- a/lib/compare_error_messages.rb +++ b/lib/compare_error_messages.rb @@ -73,7 +73,7 @@ def format_error_messages(error_messages, new_error_ids) new_error_number = 0 error_messages.each do |k, v| - if new_error_ids.include?('build-pkg.' + build_pkg_error_id(k)) + if new_error_ids.include?("build-pkg.#{build_pkg_error_id(k)}") new_error_number += 1 formatted_error_messages = add_sign(formatted_error_messages, '>>', v) else diff --git a/lib/error_messages.rb b/lib/error_messages.rb index d83d890..50f9637 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -44,7 +44,7 @@ class ErrorMessages error_messages_by_error_id = [] error_messages = obtain_error_messages error_messages.each do |k, v| - if ('build-pkg.' + build_pkg_error_id(k)) == error_id + if "build-pkg.#{build_pkg_error_id(k)}" == error_id error_messages_by_error_id += return_error_line ? [k] : v.to_a end end -- 2.23.0

1 0

[PATCH v3 compass-ci 2/3] lib/compare_error_messages.rb: add the credibility check of job
by Lin Jiaxin 18 Jan '21

18 Jan '21

defect: may match the wrong result error_id="build-pkg.pixz.c:warning:'subsuf'defined-but-not-used[-Wunused-function]" filename="pixz.c" correct result: gcc -DHAVE_CONFIG_H -I. -I.. -D_FORTIFY_SOURCE=2 -pthread -Wall -Wno-unknown-pragmas -march=native -O2 -pipe -fstack-protector-strong -fno-plt -Wl,-O1, --sort-common,--as-needed,-z,relro,-z,now -MT pixz-pixz.o -MD -MP -MF .deps/pixz-pixz.Tpo -c -o pixz-pixz.o `test -f 'pixz.c' || echo './'`pixz.c ^^^^^^ wrong result: mv -f .deps/pixz.cpu.Tpo .deps/pixz.cpu.Po ^^^^^^ Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb index 0b51fc3..d9171bf 100644 --- a/lib/compare_error_messages.rb +++ b/lib/compare_error_messages.rb @@ -8,6 +8,40 @@ require_relative 'es_query' require_relative 'error_messages' require_relative "#{ENV['LKP_SRC']}/lib/common" +# previous_job_id's error_ids include error_id +# If later_job_id's output include error filename that extract from error_id, +# the later_job_id is credible. +def credible?(previous_job_id, later_job_id, error_id) + es = ESQuery.new + + previous_result_file = File.join('/srv', es.query_by_id(previous_job_id)['result_root'], 'build-pkg') + later_result_file = File.join('/srv', es.query_by_id(later_job_id)['result_root'], 'build-pkg') + + return false if filenames_check(previous_result_file, later_result_file, error_id).value?(false) + + return true +end + +def filenames_check(previous_result_file, later_result_file, error_id) + filenames = Set.new + filenames_check = Hash.new { |h, k| h[k] = false } + + error_lines = ErrorMessages.new(previous_result_file).obtain_error_messages_by_error_id(error_id, true) + error_lines.each do |error_line| + # "src/ssl_sock.c:1454:104: warning: unused parameter ��al�� [-Wunused-parameter]" => "src/ssl_sock.c" + filenames << $1 if error_line =~ /(.*)(:\d+){2}: (error|warning):/ + end + + File.open(later_result_file).each_line do |line| + filenames.each do |filename| + filenames_check[filename] + filenames_check[filename] = true if line.include?(filename) + end + end + + return filenames_check +end + def get_compare_result(previous_job_id, later_job_id) es = ESQuery.new -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] lib/error_messages.rb: obtain_error_messages_by_error_id function add keyword argument
by Lin Jiaxin 18 Jan '21

18 Jan '21

Determine whether to return the error line or error messages by the value of return_error_line Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/error_messages.rb b/lib/error_messages.rb index 3e1564b..d83d890 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -40,12 +40,12 @@ class ErrorMessages return @error_messages end - def obtain_error_messages_by_error_id(error_id) + def obtain_error_messages_by_error_id(error_id, return_error_line = false) error_messages_by_error_id = [] error_messages = obtain_error_messages error_messages.each do |k, v| if ('build-pkg.' + build_pkg_error_id(k)) == error_id - error_messages_by_error_id += v.to_a + error_messages_by_error_id += return_error_line ? [k] : v.to_a end end error_messages_by_error_id -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/job.cr: auto install dependency of lkp
by Wei Jihui 18 Jan '21

18 Jan '21

Note that the 'lkp' depends should best be pre-installed in each rootfs. So in theory (if that step is well done) only docker image need this, since we cannot modify 3rd party docker image. However to be robust, we can always include this. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 07bfda0..ae1f170 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -556,7 +556,7 @@ class Job private def get_depends_initrd(program_params, initrd_deps_arr, initrd_pkg_arr) initrd_http_prefix = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup - program_params["initramfs-common"] = JSON::Any.new("") if os_mount == "initramfs" + program_params["lkp"] = JSON::Any.new("") program_params.keys.each do |program| if program =~ /^(.*)-\d+$/ -- 2.23.0

1 0

[PATCH lkp-tests] distro/depends: delete initramfs-common
by Wei Jihui 18 Jan '21

18 Jan '21

distro/depends/lkp will pre-installed in each rootfs Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/initramfs-common | 4 ---- 1 file changed, 4 deletions(-) delete mode 100644 distro/depends/initramfs-common diff --git a/distro/depends/initramfs-common b/distro/depends/initramfs-common deleted file mode 100644 index 5f766da6..00000000 --- a/distro/depends/initramfs-common +++ /dev/null @@ -1,4 +0,0 @@ -time -wget -perf -which -- 2.23.0

1 0

[PATCH compass-ci] doc: sparrow/README.md: update format and content
by Liu Yinsi 18 Jan '21

18 Jan '21

1. git config, this info will be used to build account info, fix verification problem when submit job. 2. setenforce 0, if not set selinux, cci-network.service will not work right. 3. update rootfs download link. 4. division operation instruction Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 68 ++++++++++++++++++++++++++++++++--------------- 1 file changed, 47 insertions(+), 21 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index c7a80e7..f0fea3f 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -24,30 +24,49 @@ >openEuler ��[��](https://openeuler.org/zh/docs/20.03_LTS/docs/Installation/%E5%AE%89%E8%A3%85%E5%87%86%E5%A4%87.html) ### �� + +#### ��compass-ci + 1. �� openEuler �� -2. �� +2. ��git�� ```bash - mkdir demo && cd demo && umask 002 + git config --golbal user.name "XXX" + git config --golbal user.email "xxx(a)aa.com" + ``` + +3. ��SELINUX + ``` + umask 002 && setenforce 0 ``` -3. �� compass-ci �� demo �� + + >**��** + >setenforce 0 ��SELINUX��/etc/selinux/config��SELINUX=enforcing��SELINUX=permissive��disabled�� + >umask 002 ��umask��/etc/bashrc��umask��002�� + +4. �� compass-ci �� ```bash + mkdir /c/ && ln -s /c/compass-ci /c/cci git clone https://gitee.com/wu_fengguang/compass-ci.git ``` -4. �� install-tiny + +5. �� install-tiny ```bash cd compass-ci/sparrow && ./install-tiny ``` -5. ��lkp-aarch64.cgz�� + +#### �� + +1. ��lkp-aarch64.cgz�� ```bash cd /c/compass-ci/container/lkp-initrd && ./run ``` -6. �� +2. �� ```bash cd /c/compass-ci/sbin && ./build-my-info.rb - �� + ``` -7. ��job�� +3. ��job�� ```bash submit iperf.yaml testbox=vm-2p8g ``` @@ -55,33 +74,40 @@ ��: submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - compass-ci��8��rootfs�� + compass-ci��4��rootfs�� -8. ��rootfs��openeuler/aarch64/20.03��/srv/os/��wget��cgz�� +4. ��rootfs��rootfs��[��](http://124.90.34.227:11300/os/)��cgz�� ```bash - mkdir -p /srv/os/openeuler/aarch64/ - cd /srv/os/openeuler/aarch64 - wget http://124.90.34.227:11300/os/test/openeuler/aarch64/20.03.cgz + mkdir -p /srv/os/openeuler/aarch64/20.03 + cd /srv/os/openeuler/aarch64/20.03 + wget http://124.90.34.227:11300/os/openeuler/aarch64/20.03.cgz ``` -9. ��cgz�� - ```bash +5. ��rootfs cgz �� + ``` gzip -dc 20.03.cgz | cpio -idv ``` -10. [�� compass-ci ��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/test-oss-project.zh.md) +#### ��compass-ci +��iperf.yaml�� -11. [��](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcase.md) +1. [�� compass-ci ��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/test-oss-project.zh.md) + +2. [��](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcase.md) + +3. ��[submit��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/submit-job.zh.md)�� + ``` + submit iperf.yaml + ``` -12. ��[submit��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/submit-job.zh.md)�� - ��os_mount ��cifs +#### �� -13. �� +1. �� ```bash cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -14. [��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/browse-results.zh.md) +2. [��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/browse-results.zh.md) ��compass-ci��srv/result��yaml��/��/$testbox/$os-$os-version-$os-arch/$job_id��output��(��tab��) ��iperf�� ```bash -- 2.23.0

1 0