- Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci 2/2] send-internet-mail: update start
by Luan Shengde 02 Dec '20

02 Dec '20

attach MAILDIR to the container store mail files that have been sent - MAILDIR/sent Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 483f01a..d6e4e25 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -12,25 +12,28 @@ names = Set.new %w[ ROBOT_EMAIL_ADDRESS ROBOT_EMAIL_PASSWORD SEND_MAIL_PORT + MAILDIR ] defaults = relevant_defaults(names) defaults['CCI_SRC'] = '/c/compass-ci' +defaults['MAILDIR'] ||= '/srv/cci/Maildir/.compass-ci' +defaults['HOST_SERVER'] = ENV['HOSTNAME'] env = docker_env(defaults) -cmd = %W[ +cmd = %w[ docker run --restart=always --name=send-internet-mail -d ] + env + %W[ -p #{defaults['SEND_MAIL_PORT']}:#{defaults['SEND_MAIL_PORT']} + -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw -v #{ENV['CCI_SRC']}:#{defaults['CCI_SRC']} -v /etc/localtime:/etc/localtime:ro -w #{defaults['CCI_SRC']}/container/send-internet-mail - --log-driver json-file send-internet-mail ] -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] send-internet-mail/mail-post: call to store email
by Luan Shengde 02 Dec '20

02 Dec '20

call function store_email to store email to directory sent Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/mail-post.rb b/lib/mail-post.rb index 0b1f94a..df4d255 100755 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -51,5 +51,11 @@ def send_mail(mail_info) to mail_info['to'] body mail_info['body'] end + mail.deliver! + + return if ENV['SEND_MAIL_PORT'].to_s != '49000' + return if ENV['HOST_SERVER'] != 'z9' + + store_email(mail) end -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: fix upload in container
by Wang Yong 02 Dec '20

02 Dec '20

docker can not use mount operation, so now use curl to upload build-pkg cgz file Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index bdaef0355..6e435a6ad 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -18,6 +18,8 @@ ## See: https://wiki.archlinux.org/index.php/Makepkg . $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/env.sh +. $LKP_SRC/distro/common check_vars() { @@ -29,6 +31,13 @@ check_vars() [ -n "$os_mount" ] || die "os_mount is empty" } +mount_pkg_mnt() +{ + [ -n "$LKP_SERVER" ] && { + mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr,port=446 //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" + } +} + mount_dest() { # the same image is mounted to cifs and nfs, the generated cgz files @@ -39,9 +48,7 @@ mount_dest() PKG_MNT=/initrd/build-pkg mkdir -p "$PKG_MNT" - [ -n "$LKP_SERVER" ] && { - mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr,port=446 //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" - } + is_docker || mount_pkg_mnt } get_pkgfile() @@ -117,10 +124,15 @@ build_source_pkg() } cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" + PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 - [ "$?" == 0 ] && create_softlink + [ "$?" == 0 ] || exit 1 + create_softlink + + is_docker || exit 0 + upload_to_target_dir ${PKG_MNT}/${pack_to}/${pkgname} } check_vars -- 2.23.0

1 0

[PATCH v1 compass-ci 2/2] sched: result root add "." and replace "/" to "-"
by Cao Xueliang 02 Dec '20

02 Dec '20

Before: archlinuxaur-zzookeeper-HEAD After: archlinux-aur-z-zookeeper-HEAD Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/pp_params.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/scheduler/pp_params.cr b/src/scheduler/pp_params.cr index 28ec6ef..da8d8f3 100644 --- a/src/scheduler/pp_params.cr +++ b/src/scheduler/pp_params.cr @@ -51,8 +51,8 @@ class Job private def format_string(original_str) temp = [] of String original_str.each_char do |char| - if "#{char}" =~ /-|\w/ - temp << "#{char}" + if "#{char}" =~ /-|\.|\/|\w/ + temp << "#{char}".gsub("\/", "-") end end -- 2.23.0

2 2

[PATCH compass-ci] scheduler: fix: 'lab' not 'LAB' in scheduler environment
by Ren Wen 02 Dec '20

02 Dec '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/constants.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 1b37333..418e7ff 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -10,7 +10,7 @@ JOB_ES_PORT = 9200 JOB_ES_PORT_DEBUG = 9201 JOB_INDEX_TYPE = "jobs/_doc" -LAB = (ENV.has_key?("LAB") ? ENV["LAB"] : "nolab") +LAB = (ENV.has_key?("lab") ? ENV["lab"] : "nolab") SCHED_HOST = (ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1") SCHED_PORT = (ENV.has_key?("SCHED_PORT") ? ENV["SCHED_PORT"] : 3000).to_i32 -- 2.23.0

1 0

[PATCH lkp-tests] spec/submit: add job yaml for changing multi-args style
by Bai Jing 02 Dec '20

02 Dec '20

[test cmd] rspec $LKP_SRC/spec/submit_spec.rb [output] /lkp-tests/spec/submit/matrix/unixbench-spec.yaml => /lkp-tests/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml /lkp-tests/spec/submit/matrix/unixbench-spec.yaml => /lkp-tests/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml Signed-off-by: Bai Jing <799286817(a)qq.com> --- spec/submit/matrix/unixbench-spec.yaml | 6 +++--- .../unixbench-spec-1-300-whetstone-double.yaml | 11 +++++++---- .../unixbench-spec/unixbench-spec-3-300-shell1.yaml | 11 +++++++---- 3 files changed, 17 insertions(+), 11 deletions(-) diff --git a/spec/submit/matrix/unixbench-spec.yaml b/spec/submit/matrix/unixbench-spec.yaml index e17322b4..851f0473 100644 --- a/spec/submit/matrix/unixbench-spec.yaml +++ b/spec/submit/matrix/unixbench-spec.yaml @@ -2,8 +2,8 @@ suite: unixbench runtime: 300s -nr_task test: - - 1 whetstone-double - - 3 shell1 +nr_task|test: + - 1|whetstone-double + - 3|shell1 unixbench: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml index 9896bcef..6a29541f 100644 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml +++ b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml @@ -1,7 +1,7 @@ --- suite: unixbench runtime: 300 -unixbench: +unixbench: lab: spec_lab testbox: vm-2p8g--spec_submit nr_task: '1' @@ -13,7 +13,10 @@ nr_node: 1 nr_cpu: 2 memory: 8G arch: aarch64 -cpufreq_governor: +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 +cpufreq_governor: need_kconfig: - CONFIG_KVM_GUEST=y pp: @@ -21,5 +24,5 @@ pp: runtime: 300 test: whetstone-double nr_task: '1' - cpufreq_governor: -monitors: + cpufreq_governor: +monitors: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml index 912d10ee..3e49255e 100644 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml +++ b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml @@ -1,7 +1,7 @@ --- suite: unixbench runtime: 300 -unixbench: +unixbench: lab: spec_lab testbox: vm-2p8g--spec_submit nr_task: '3' @@ -13,7 +13,10 @@ nr_node: 1 nr_cpu: 2 memory: 8G arch: aarch64 -cpufreq_governor: +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 +cpufreq_governor: need_kconfig: - CONFIG_KVM_GUEST=y pp: @@ -21,5 +24,5 @@ pp: runtime: 300 test: shell1 nr_task: '3' - cpufreq_governor: -monitors: + cpufreq_governor: +monitors: -- 2.23.0

1 0

[PATCH compass-ci] define_files.md: adjust markdown document format
by Xu Xijian 02 Dec '20

02 Dec '20

In markdown documents, text for bash should be contained in block as below: ```bash content ``` Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/define_files.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/doc/job/define_files.md b/doc/job/define_files.md index 2c3457d..e418b4f 100644 --- a/doc/job/define_files.md +++ b/doc/job/define_files.md @@ -2,15 +2,19 @@ Meaning: - We use lkp-tests as client for compass-ci. There are many testcases in lkp-tests. most testcases contain the following files: - distro/depends/$testcase +```SHELL + distro/depends/$testcas distro/depends/$testcase-dev pkg/$testcase/PKGBUILD stats/$testcase tests/$testcase jobs/$testcase.yaml +``` - When user add new testcase files to lkp-tests or change existing testcase files in lkp-tests, these change files that related with test program would be added to define_files field. - The define_files field do not need to be specified by user in the $testcase.yaml. It can be generated automatically when user submit with the option '-a'. User can confirm the define_files field in the job.yaml under result_root. Usage example: +```SHELL - % submit -a $testcase.yaml testbox=vm-2p8g--$USER os=openeuler os_arch=aarch64 os_version=20.03 - % cat /srv/result/$testcase/vm-2p8g--$USER/$date/$job_id/job.yaml +``` -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: unified indent
by Lin Jiaxin 02 Dec '20

02 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 124 ++++++++++++++++++++++++------------------------ 1 file changed, 62 insertions(+), 62 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index bdaef035..5e7e62d4 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -21,60 +21,60 @@ check_vars() { - [ -n "$os" ] || die "os is empty" - [ -n "$os_arch" ] || die "os_arch is empty" - [ -n "$os_version" ] || die "os_version is empty" - [ -n "$pkgbuild_repo" ] || die "pkgbuild_repo is empty" - [ -n "$upstream_commit" ] || die "upstream_commit is empty" - [ -n "$os_mount" ] || die "os_mount is empty" + [ -n "$os" ] || die "os is empty" + [ -n "$os_arch" ] || die "os_arch is empty" + [ -n "$os_version" ] || die "os_version is empty" + [ -n "$pkgbuild_repo" ] || die "pkgbuild_repo is empty" + [ -n "$upstream_commit" ] || die "upstream_commit is empty" + [ -n "$os_mount" ] || die "os_mount is empty" } mount_dest() { - # the same image is mounted to cifs and nfs, the generated cgz files - # are stored in the nfs directory. - [[ "$os_mount" = "cifs" ]] && os_mount="nfs" - pack_to=${os_mount}/${os}/${os_arch}/${os_version} + # the same image is mounted to cifs and nfs, the generated cgz files + # are stored in the nfs directory. + [[ "$os_mount" = "cifs" ]] && os_mount="nfs" + pack_to=${os_mount}/${os}/${os_arch}/${os_version} - PKG_MNT=/initrd/build-pkg - mkdir -p "$PKG_MNT" + PKG_MNT=/initrd/build-pkg + mkdir -p "$PKG_MNT" - [ -n "$LKP_SERVER" ] && { - mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr,port=446 //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" - } + [ -n "$LKP_SERVER" ] && { + mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr,port=446 //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" + } } get_pkgfile() { - curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ - -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "${2##*/}" + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "${2##*/}" } request_pkg() { - local pkgrepo="${pkgbuild_repo%%//*}" - local pkgpath="./" - [[ "$pkgrepo" =~ ^archlinux/(packages|community) ]] && pkgpath="${pkgpath}${pkgbuild_repo#*//}/" - - filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ - -d '{"git_repo": "'${pkgrepo}'", "git_command": ["git-ls-files", "'${pkgpath}'"]}') - - for pkgfile in ${filelist[*]} - do - get_pkgfile "${pkgrepo}" "$pkgfile" - done - - # download PKGBUILD-$tag - benchmark="${upstream_repo##*/}" - benchmark="${benchmark%.git}" - [ -n "$PKGBUILD_TAG" ] && { - get_pkgfile "pkg/$benchmark" "PKGBUILD-$PKGBUILD_TAG" - [ -f "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG not exist" - [ -s "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG is empty" - } - - [ -f PKGBUILD ] || die "PKGBUILD not exist" - [ -s PKGBUILD ] || die "PKGBUILD is empty" + local pkgrepo="${pkgbuild_repo%%//*}" + local pkgpath="./" + [[ "$pkgrepo" =~ ^archlinux/(packages|community) ]] && pkgpath="${pkgpath}${pkgbuild_repo#*//}/" + + filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'${pkgrepo}'", "git_command": ["git-ls-files", "'${pkgpath}'"]}') + + for pkgfile in ${filelist[*]} + do + get_pkgfile "${pkgrepo}" "$pkgfile" + done + + # download PKGBUILD-$tag + benchmark="${upstream_repo##*/}" + benchmark="${benchmark%.git}" + [ -n "$PKGBUILD_TAG" ] && { + get_pkgfile "pkg/$benchmark" "PKGBUILD-$PKGBUILD_TAG" + [ -f "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG not exist" + [ -s "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG is empty" + } + + [ -f PKGBUILD ] || die "PKGBUILD not exist" + [ -s PKGBUILD ] || die "PKGBUILD is empty" } create_softlink() @@ -95,32 +95,32 @@ create_softlink() replace_source() { - sed -i "s|^source=[^)]*|$1|g" "$2" + sed -i "s|^source=[^)]*|$1|g" "$2" } build_source_pkg() { - local repo_dir="" - source PKGBUILD - [ -n "$upstream_repo" ] && [ -n "$upstream_url" ] && { - - if [[ "$source" = *::* ]]; then - repo_dir="${source%%::*}::" - else - repo_dir="$(basename ${upstream_url})" - repo_dir="${repo_dir%.git*}::" - fi - - upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" - replace_source "${upstream_source}" PKGBUILD - [ -n "$PKGBUILD_TAG" ] && replace_source "${upstream_source}" "PKGBUILD-$PKGBUILD_TAG" - } - - cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" - PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 - - [ "$?" == 0 ] && create_softlink + local repo_dir="" + source PKGBUILD + [ -n "$upstream_repo" ] && [ -n "$upstream_url" ] && { + + if [[ "$source" = *::* ]]; then + repo_dir="${source%%::*}::" + else + repo_dir="$(basename ${upstream_url})" + repo_dir="${repo_dir%.git*}::" + fi + + upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" + replace_source "${upstream_source}" PKGBUILD + [ -n "$PKGBUILD_TAG" ] && replace_source "${upstream_source}" "PKGBUILD-$PKGBUILD_TAG" + } + + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" + PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ + $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 + + [ "$?" == 0 ] && create_softlink } check_vars -- 2.23.0

1 0

[PATCH v4 compass-ci] lib: add error_messages.rb
by Lin Jiaxin 02 Dec '20

02 Dec '20

provide error messages for git bisect Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 73 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 lib/error_messages.rb diff --git a/lib/error_messages.rb b/lib/error_messages.rb new file mode 100644 index 0000000..7bf3aa8 --- /dev/null +++ b/lib/error_messages.rb @@ -0,0 +1,73 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' + +# usage: +# foo = ErrorMessages.new(build-pkg) +# build-pkg is a file that includes errors generated by make +# error_messages_hash = foo.obtain_error_messages +# error_messages_hash.each { |k,v| puts k; puts; v.each {|m| puts m } } +# output: +# src/arg.c:117:24: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] +# +# src/arg.c:117:24: warning: comparison between signed and unsigned integer expressions [-Wsign-compare] +# for (nbarg = 0; nbarg < ARGM_NBARGS && ((mask >> (nbarg * ARGT_BITS)) & ARGT_MASK); nbarg++); +# ^ +class ErrorMessages + def initialize(file) + @file = file + @seqno = '' + @in_stderr = false + @error_line = '' + @error_details = '' + @error_message = {} + @error_messages = Hash.new { |h, k| h[k] = Set.new } + end + + def obtain_error_messages + File.open((a)file).each_line do |line| + next if extract_error_message(line) + next unless @in_stderr + next unless @error_message['error_line'] =~ /error|warning/i + + @error_messages[@error_message['error_line']] << @error_message['error_line'] + @error_message['error_details'] + end + return @error_messages + end + + private + + def extract_error_message(line) + if line =~ /^ / + @error_details += line + else + update_error_message(line) + case line + when /^## ______________([0-9.]+):stderr$/ + @in_stderr = true + @seqno = $1 + return true + + when /^## ______________#{@seqno}:enderr$/ + @in_stderr = false + @seqno = '' + add_error_message + end + end + end + + def update_error_message(line) + @error_message['error_line'] = @error_line + @error_message['error_details'] = @error_details + @error_line = line + @error_details = '' + end + + def add_error_message + return unless @error_message['error_line'] =~ /error|warning/i + + @error_messages[@error_message['error_line']] << @error_message['error_line'] + @error_message['error_details'] + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci] scheduler: fix the values diplayed in job.yaml
by Bai Jing 02 Dec '20

02 Dec '20

[original values] initrds_uri: 'initrd http://172.168.131.113:8800/initrd/osimage/openeuler/aarch64/20.03/20201022… initrd http://172.168.131.113:8800/initrd/deps/nfs/debian/aarch64/sid/run-ipconfig… initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/modules-4.19.90-2003… initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/headers-4.19.90-2003… initrd http://172.168.131.113:8800/initrd/deps/initramfs/openeuler/aarch64/20.03/m… initrd http://172.168.131.113:8800/initrd/deps/initramfs/openeuler/aarch64/20.03/s… initrd http://172.168.131.113:8800/initrd/pkg/initramfs/openeuler/aarch64/20.03/mp… initrd http://172.168.131.113:8800/initrd/pkg/initramfs/openeuler/aarch64/20.03/sp… initrd http://172.168.131.113:8800/initrd/lkp/baijing/lkp-aarch64.cgz initrd http://172.168.131.113:3000/job_initrd_tmpfs/crystal.146511/job.cgz ' [now values] initrds_uri: - initrd http://172.168.131.113:8800/initrd/osimage/openeuler/aarch64/20.03/20201022… - initrd http://172.168.131.113:8800/initrd/deps/nfs/debian/aarch64/sid/run-ipconfig… - initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/modules-4.19.90-2003… - initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/headers-4.19.90-2003… - initrd http://172.168.131.113:8800/initrd/deps/initramfs/openeuler/aarch64/20.03/m… - initrd http://172.168.131.113:8800/initrd/deps/initramfs/openeuler/aarch64/20.03/s… - initrd http://172.168.131.113:8800/initrd/pkg/initramfs/openeuler/aarch64/20.03/mp… - initrd http://172.168.131.113:8800/initrd/pkg/initramfs/openeuler/aarch64/20.03/sp… - initrd http://172.168.131.113:8800/initrd/lkp/baijing/lkp-aarch64.cgz - initrd http://172.168.131.113:3087/job_initrd_tmpfs/crystal.147702/job.cgz Signed-off-by: Bai Jing <799286817(a)qq.com> --- src/scheduler/jobfile_operate.cr | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index 98bf011..7ec3ea1 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -76,8 +76,24 @@ module Jobfile::Operate script_lines = JSON.parse(script_lines) end + def self.handle_job_field(job_content, fields) + fields.each do |field| + case field + when "initrds_uri" + values = job_content[field].to_s.split("\n").map(&.strip).reject!(&.empty?) + values = values.map { |value| JSON::Any.new(value) } + job_content[field] = JSON::Any.new(values) + end + end + + return job_content + end + def self.create_job_cpio(job_content : JSON::Any, base_dir : String) job_content = job_content.as_h + + need_to_handle = ["initrds_uri"] + tmp_job_content = handle_job_field(job_content.clone, need_to_handle) # put job2sh in an array if job_content.has_key?("job2sh") @@ -100,8 +116,8 @@ module Jobfile::Operate # - this will create a <'#! jobs/pixz.yaml':> in the yaml file # - but the orange is <#! jobs/pixz.yaml> in the user job.yaml # tested : no effect to job.sh - File.open(temp_yaml, "w") do |file| - YAML.dump(job_content, file) + File.open(temp_yaml, "a") do |file| + YAML.dump(tmp_job_content, file) end # generate unbroken job shell content -- 2.23.0

1 0

[PATCH compass-ci] scheduler: fix the values diplayed in job.yaml
by Bai Jing 02 Dec '20

02 Dec '20

[original values] initrds_uri: 'initrd http://172.168.131.113:8800/initrd/osimage/openeuler/aarch64/20.03/20201022… initrd http://172.168.131.113:8800/initrd/deps/nfs/debian/aarch64/sid/run-ipconfig… initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/modules-4.19.90-2003… initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/headers-4.19.90-2003… initrd http://172.168.131.113:8800/initrd/deps/initramfs/openeuler/aarch64/20.03/m… initrd http://172.168.131.113:8800/initrd/deps/initramfs/openeuler/aarch64/20.03/s… initrd http://172.168.131.113:8800/initrd/pkg/initramfs/openeuler/aarch64/20.03/mp… initrd http://172.168.131.113:8800/initrd/pkg/initramfs/openeuler/aarch64/20.03/sp… initrd http://172.168.131.113:8800/initrd/lkp/baijing/lkp-aarch64.cgz initrd http://172.168.131.113:3000/job_initrd_tmpfs/crystal.146511/job.cgz ' [now values] initrds_uri: - initrd http://172.168.131.113:8800/initrd/osimage/openeuler/aarch64/20.03/20201022… - initrd http://172.168.131.113:8800/initrd/deps/nfs/debian/aarch64/sid/run-ipconfig… - initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/modules-4.19.90-2003… - initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/headers-4.19.90-2003… - initrd http://172.168.131.113:8800/initrd/deps/initramfs/openeuler/aarch64/20.03/m… - initrd http://172.168.131.113:8800/initrd/deps/initramfs/openeuler/aarch64/20.03/s… - initrd http://172.168.131.113:8800/initrd/pkg/initramfs/openeuler/aarch64/20.03/mp… - initrd http://172.168.131.113:8800/initrd/pkg/initramfs/openeuler/aarch64/20.03/sp… - initrd http://172.168.131.113:8800/initrd/lkp/baijing/lkp-aarch64.cgz - initrd http://172.168.131.113:3087/job_initrd_tmpfs/crystal.147702/job.cgz Signed-off-by: Bai Jing <799286817(a)qq.com> --- src/scheduler/jobfile_operate.cr | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index 98bf011..10e93fb 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -76,8 +76,24 @@ module Jobfile::Operate script_lines = JSON.parse(script_lines) end + def self.handle_job_field(job_content, fields) + fields.each do |field| + case field + when "initrds_uri" + values = job_content["initrds_uri"].to_s.split("\n").map(&.strip).reject!(&.empty?) + values = values.map { |value| JSON::Any.new(value) } + job_content["initrds_uri"] = JSON::Any.new(values) + end + end + + return job_content + end + def self.create_job_cpio(job_content : JSON::Any, base_dir : String) job_content = job_content.as_h + + need_to_handle = ["initrds_uri"] + tmp_job_content = handle_job_field(job_content.clone, need_to_handle) # put job2sh in an array if job_content.has_key?("job2sh") @@ -100,8 +116,8 @@ module Jobfile::Operate # - this will create a <'#! jobs/pixz.yaml':> in the yaml file # - but the orange is <#! jobs/pixz.yaml> in the user job.yaml # tested : no effect to job.sh - File.open(temp_yaml, "w") do |file| - YAML.dump(job_content, file) + File.open(temp_yaml, "a") do |file| + YAML.dump(tmp_job_content, file) end # generate unbroken job shell content -- 2.23.0

3 3

[PATCH v2 compass-ci 3/3] send-internet-mail: update start
by Luan Shengde 02 Dec '20

02 Dec '20

attach MAILDIR to the container store mail files that have been sent - MAILDIR/sent Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 483f01a..d6e4e25 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -12,25 +12,28 @@ names = Set.new %w[ ROBOT_EMAIL_ADDRESS ROBOT_EMAIL_PASSWORD SEND_MAIL_PORT + MAILDIR ] defaults = relevant_defaults(names) defaults['CCI_SRC'] = '/c/compass-ci' +defaults['MAILDIR'] ||= '/srv/cci/Maildir/.compass-ci' +defaults['HOST_SERVER'] = ENV['HOSTNAME'] env = docker_env(defaults) -cmd = %W[ +cmd = %w[ docker run --restart=always --name=send-internet-mail -d ] + env + %W[ -p #{defaults['SEND_MAIL_PORT']}:#{defaults['SEND_MAIL_PORT']} + -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw -v #{ENV['CCI_SRC']}:#{defaults['CCI_SRC']} -v /etc/localtime:/etc/localtime:ro -w #{defaults['CCI_SRC']}/container/send-internet-mail - --log-driver json-file send-internet-mail ] -- 2.23.0

3 4

[PATCH v2 compass-ci] 本地安装compass-ci客户端.md: adjust markdown document format
by Xu Xijian 02 Dec '20

02 Dec '20

1. In markdown documents, text for bash should be contained in block as below: ```bash content ``` 2. Add blank line between title and normal text, or it will not line feed. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- ...205compass-ci\345\256\242\346\210\267\347\253\257.md" | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" index 43a6bf5..b986626 100644 --- "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" +++ "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" @@ -9,20 +9,25 @@ Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客 ## Getting started 1. setup default config + run the following command to add the below setup to default config file +```SHELL mkdir -p ~/.config/compass-ci/defaults/ cat >> ~/.config/compass-ci/defaults/${USER}.yaml <<-EOF my_uuid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx my_email: xxx my_name: xx EOF - +``` 2. download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration +```SHELL git clone https://gitee.com/wu_fengguang/lkp-tests.git cd lkp-tests make install source ~/.bashrc && source ~/.bash_profile - +``` 3. submit job + Now try [submitting a job to compass-ci](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] send-internet-mail: add function: store email
by Luan Shengde 02 Dec '20

02 Dec '20

add function: store_email store email to MAILDIR/sent file name: ${email_addr}_${time_now} Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 10 ++++++++++ 1 file changed, 10 insertions(+) mode change 100644 => 100755 container/send-internet-mail/send-internet-mail.rb diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb old mode 100644 new mode 100755 index 3b1d159..851904c --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -19,3 +19,13 @@ smtp = { } Mail.defaults { delivery_method :smtp, smtp } + +def store_email(mail) + time_now = Time.new.strftime('%Y%m%d%H%M%S') + file_name = [mail.to[0], time_now].join('_') + file_full_name = File.join(ENV['MAILDIR'], 'sent', file_name) + File.open(file_full_name, 'w') do |f| + f.puts mail + end + FileUtils.chown_R(1144, 1110, file_full_name) +end -- 2.23.0

2 2

[PATCH v3 compass-ci] container/result-webdav: adjust the value of second_dir
by Cui Yili 02 Dec '20

02 Dec '20

[why] 1. when the uri starts with "//", etc., the previous regular expression did not get the second level directory correctly. 2. when the uploaded data package does not meet the format of "xxxx_yyyymmdd.cgz", uploading is also allowed, but no symbolic link. [how] 1. modify the variable second_dir regular expression. 2. make conditional judgments on link_name. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index c880522..1a4dddc 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -54,17 +54,21 @@ http { path = string.match(uri,"%g*/") upload_file = string.match(uri, ".+/([^/]*%.%w+)$") - second_dir = string.match(uri, "/.+/(.*)/.+") + second_dir = string.match(uri, "/*%w+/(%w+)/.+") - if (second_dir == "pkg") + if (second_dir == "pkg" or second_dir == "build-pkg") then - link_name = "latest.cgz" + link_name = "latest" else - link_name = string.match(upload_file, "(.*)_%d+%.cgz")..".cgz" + link_name = string.match(upload_file, "(.*)_%d+%.cgz") + end + + if (link_name ~= '') + then + link_name = link_name..".cgz" + root_patch = "/srv/"..path + io.popen("cd "..root_patch.."&& ln -sf "..upload_file.." "..link_name) end - - root_patch = "/srv/"..path - io.popen("cd "..root_patch.."&& ln -sf "..upload_file.." "..link_name) } } -- 2.23.0

1 0

[PATCH v2 lkp-tests] jobs/borrow-1h.yaml: set "pub_key" field as level-1 parameter
by Zhang Yale 02 Dec '20

02 Dec '20

[Why] Now the "pub_key" field is not placed under the "sshd" field in yaml, use "pub_key" to determine whether to password-free login. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/borrow-1h.yaml | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/jobs/borrow-1h.yaml b/jobs/borrow-1h.yaml index be22b94a..f2fedd20 100644 --- a/jobs/borrow-1h.yaml +++ b/jobs/borrow-1h.yaml @@ -1,13 +1,13 @@ suite: borrow testcase: borrow +pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> sshd: - pub_key: <%= - begin - File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp - rescue - nil - end - %> # sleep at the bottom sleep: 1h -- 2.23.0

1 0

[PATCH lkp-tests] jobs/borrow-1h.yaml: set "pub_key" field as level-1 parameter
by Zhang Yale 02 Dec '20

02 Dec '20

[Why] Now, the "pub_key" field is not placed under the "sshd" field in yaml, use "pub_key" to determine whether to password-free login. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/borrow-1h.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/jobs/borrow-1h.yaml b/jobs/borrow-1h.yaml index be22b94a..38306441 100644 --- a/jobs/borrow-1h.yaml +++ b/jobs/borrow-1h.yaml @@ -1,13 +1,13 @@ suite: borrow testcase: borrow -sshd: - pub_key: <%= +pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp rescue nil end %> +sshd: # sleep at the bottom sleep: 1h -- 2.23.0

1 1

[PATCH compass-ci] stats_worker.cr: add blank line after next
by Xu Xijian 02 Dec '20

02 Dec '20

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/extract-stats/stats_worker.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index bcf6e52..4a6ce0f 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -102,6 +102,7 @@ class StatsWorker STDERR.puts e.message next end + next if is_exists new_error_ids << error_id -- 2.23.0

2 2

[PATCH v2 compass-ci] container/defconfig.rb: fix rubocop warning
by Liu Yinsi 02 Dec '20

02 Dec '20

defconfig.rb:23:12: Style/BlockDelimiters: Avoid using {...} for multi-line blocks. hash.map { |k, v| Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/container/defconfig.rb b/container/defconfig.rb index dd52fe3..c0768ca 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -20,9 +20,7 @@ end def set_local_env hash = cci_defaults - hash.map { |k, v| - system "export #{k}=#{v}" - } + hash.map { |k, v| system "export #{k}=#{v}" } end def docker_env(hash) -- 2.23.0

1 0

[PATCH v1 compass-ci 1/2] sched: no need set kernel if os mount is container
by Cao Xueliang 02 Dec '20

02 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 44a6ecf..7758e78 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -146,16 +146,22 @@ class Job set_result_root() set_result_service() set_os_mount() + set_kernel() set_depends_initrd() - set_kernel_version() set_initrds_uri() - set_kernel_uri() - set_kernel_params() set_lkp_server() set_sshr_info() set_queue() end + private def set_kernel + return if os_mount == "container" + + set_kernel_version() + set_kernel_uri() + set_kernel_params() + end + private def append_init_field INIT_FIELD.each do |k, v| k = k.to_s -- 2.23.0

1 0

[PATCH compass-ci] sched: result root add "." and replace "/" to "-"
by Cao Xueliang 02 Dec '20

02 Dec '20

Before: archlinuxaur-zzookeeper-HEAD After: archlinux-aur-z-zookeeper-HEAD Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/pp_params.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/scheduler/pp_params.cr b/src/scheduler/pp_params.cr index 28ec6ef..717f25b 100644 --- a/src/scheduler/pp_params.cr +++ b/src/scheduler/pp_params.cr @@ -51,8 +51,8 @@ class Job private def format_string(original_str) temp = [] of String original_str.each_char do |char| - if "#{char}" =~ /-|\w/ - temp << "#{char}" + if "#{char}" =~ /-|\.|\/|\w/ + temp << "#{char}".gusb("\/", "-") end end -- 2.23.0

2 2

[PATCH compass-ci] lib/job.cr: set "pub_key" field as level-1 parameter
by Zhang Yale 02 Dec '20

02 Dec '20

[why] The "pub_key" field in yaml will not be set under sshd, It should be set as level-1 parameter for password-free login, and "sshd" can be placed below the "on_fail" field or elsewhere. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- src/lib/job.cr | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index a4402ab..44a6ecf 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -184,7 +184,7 @@ class Job end private def set_sshr_info - return unless self["sshd"]? + return unless @hash["pub_key"]? self["sshr_port"] = ENV["SSHR_PORT"] self["sshr_port_base"] = ENV["SSHR_PORT_BASE"] @@ -196,8 +196,7 @@ class Job end private def set_my_ssh_pubkey - sshd = @hash["sshd"].as_h? || Hash(String, JSON::Any).new() - pub_key = sshd["pub_key"]?.to_s + pub_key = @hash["pub_key"]?.to_s update_account_my_pub_key(pub_key) @hash["my_ssh_pubkey"] = @account_info["my_ssh_pubkey"] -- 2.23.0

1 0

[PATCH v4 compass-ci 3/3] container/logging-es: set available memory size
by Liu Yinsi 02 Dec '20

02 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/logging-es/Dockerfile | 6 ++++-- container/logging-es/build | 19 +++++++++++-------- 2 files changed, 15 insertions(+), 10 deletions(-) diff --git a/container/logging-es/Dockerfile b/container/logging-es/Dockerfile index 6c01c43..a0acf94 100644 --- a/container/logging-es/Dockerfile +++ b/container/logging-es/Dockerfile @@ -4,6 +4,8 @@ ARG BASE_IMAGE FROM $BASE_IMAGE +ARG MEMORY + # docker image borrowed from hub.docker.com/r/gagara/elasticsearch-oss-arm64 MAINTAINER Wu Zhende <wuzhende666(a)163.com> @@ -14,8 +16,8 @@ RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elastic sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && \ sed -i '$a indices.memory.index_buffer_size: 20%' /usr/share/elasticsearch/config/elasticsearch.yml && \ sed -i '$a thread_pool.write.queue_size: 2000' /usr/share/elasticsearch/config/elasticsearch.yml && \ - sed -i "s/-Xms1g/-Xms30g/g" /usr/share/elasticsearch/config/jvm.options && \ - sed -i "s/-Xmx1g/-Xmx30g/g" /usr/share/elasticsearch/config/jvm.options + sed -i "s/-Xms1g/-Xms${MEMORY}m/g" /usr/share/elasticsearch/config/jvm.options && \ + sed -i "s/-Xmx1g/-Xmx${MEMORY}m/g" /usr/share/elasticsearch/config/jvm.options RUN mkdir /usr/share/elasticsearch/tmp && \ chown -R 1090:1090 /usr/share/elasticsearch diff --git a/container/logging-es/build b/container/logging-es/build index 451041b..6465ace 100755 --- a/container/logging-es/build +++ b/container/logging-es/build @@ -1,14 +1,17 @@ -#!/bin/bash +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -declare -A BASE_IMAGE_DICT +require_relative '../defconfig.rb' -BASE_IMAGE_DICT=( - ["aarch64"]="gagara/elasticsearch-oss-arm64:7.6.2" - ["x86_64"]="elasticsearch:7.6.2" -) +BASE_IMAGE_DICT = { + 'aarch64' => 'gagara/elasticsearch-oss-arm64:7.6.2', + 'x86_64' => 'elasticsearch:7.6.2' +}.freeze -BASE_IMAGE=${BASE_IMAGE_DICT[$(arch)]} +BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] -docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE="$BASE_IMAGE" . +available_memory = get_available_memory + +system "docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} --build-arg MEMORY=#{available_memory} ." -- 2.23.0

1 0

[PATCH v4 compass-ci 2/3] container/es: set available memory size
by Liu Yinsi 02 Dec '20

02 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/es/Dockerfile | 6 ++++-- container/es/build | 8 ++++++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/container/es/Dockerfile b/container/es/Dockerfile index e626dae..ed02490 100644 --- a/container/es/Dockerfile +++ b/container/es/Dockerfile @@ -3,6 +3,8 @@ FROM alpine:3.11 +ARG MEMORY + RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk add --no-cache elasticsearch curl @@ -27,8 +29,8 @@ RUN rm -rf /etc/init.d/elasticsearch \ RUN sed -i 's:#path.data\: /path/to/data:path.data\: /srv/es:' /usr/share/es/config/elasticsearch.yml; RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/es/config/elasticsearch.yml; -RUN sed -i "s/-Xms256m/-Xms20g/g" /usr/share/es/config/jvm.options -RUN sed -i "s/-Xmx256m/-Xmx20g/g" /usr/share/es/config/jvm.options +RUN sed -i "s/-Xms256m/-Xms${MEMORY}m/g" /usr/share/es/config/jvm.options +RUN sed -i "s/-Xmx256m/-Xmx${MEMORY}m/g" /usr/share/es/config/jvm.options WORKDIR /usr/share/es diff --git a/container/es/build b/container/es/build index d5607e0..17a225c 100755 --- a/container/es/build +++ b/container/es/build @@ -1,6 +1,10 @@ -#!/bin/sh +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -docker build -t es643b:alpine311 . --network=host +require_relative '../defconfig.rb' +available_memory = get_available_memory + +system "docker build -t es643b:alpine311 --build-arg MEMORY=#{available_memory} --network=host ." -- 2.23.0

1 0

[PATCH v4 compass-ci 1/3] container/defconfig.rb: limit container memory usage
by Liu Yinsi 02 Dec '20

02 Dec '20

use Math.sqrt(memory) to get container occupied memory according to system memory size, instead of setting fixed size, more reasonable. In addition, '-Xms' not support floating number, so use to_i. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index 05c56db..dd52fe3 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -35,3 +35,11 @@ def docker_rm(container) system "docker stop #{container} && docker rm -f #{container}" end + +def get_available_memory + memory = File.readlines('/proc/meminfo')[0].chomp.split[1].to_f / 1048576 + + # set container available memory size, minimum size is 1024m, maximum size is 30720m, + # take the middle value according to the system memory size. + [1024, 30720, Math.sqrt(memory) * 1024].sort[1].to_i +end -- 2.23.0

1 0

[PATCH v3 compass-ci 3/3] container/logging-es: set available memory size
by Liu Yinsi 02 Dec '20

02 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/logging-es/Dockerfile | 6 ++++-- container/logging-es/build | 19 +++++++++++-------- 2 files changed, 15 insertions(+), 10 deletions(-) diff --git a/container/logging-es/Dockerfile b/container/logging-es/Dockerfile index 6c01c43..a0acf94 100644 --- a/container/logging-es/Dockerfile +++ b/container/logging-es/Dockerfile @@ -4,6 +4,8 @@ ARG BASE_IMAGE FROM $BASE_IMAGE +ARG MEMORY + # docker image borrowed from hub.docker.com/r/gagara/elasticsearch-oss-arm64 MAINTAINER Wu Zhende <wuzhende666(a)163.com> @@ -14,8 +16,8 @@ RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elastic sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && \ sed -i '$a indices.memory.index_buffer_size: 20%' /usr/share/elasticsearch/config/elasticsearch.yml && \ sed -i '$a thread_pool.write.queue_size: 2000' /usr/share/elasticsearch/config/elasticsearch.yml && \ - sed -i "s/-Xms1g/-Xms30g/g" /usr/share/elasticsearch/config/jvm.options && \ - sed -i "s/-Xmx1g/-Xmx30g/g" /usr/share/elasticsearch/config/jvm.options + sed -i "s/-Xms1g/-Xms${MEMORY}m/g" /usr/share/elasticsearch/config/jvm.options && \ + sed -i "s/-Xmx1g/-Xmx${MEMORY}m/g" /usr/share/elasticsearch/config/jvm.options RUN mkdir /usr/share/elasticsearch/tmp && \ chown -R 1090:1090 /usr/share/elasticsearch diff --git a/container/logging-es/build b/container/logging-es/build index 451041b..6465ace 100755 --- a/container/logging-es/build +++ b/container/logging-es/build @@ -1,14 +1,17 @@ -#!/bin/bash +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -declare -A BASE_IMAGE_DICT +require_relative '../defconfig.rb' -BASE_IMAGE_DICT=( - ["aarch64"]="gagara/elasticsearch-oss-arm64:7.6.2" - ["x86_64"]="elasticsearch:7.6.2" -) +BASE_IMAGE_DICT = { + 'aarch64' => 'gagara/elasticsearch-oss-arm64:7.6.2', + 'x86_64' => 'elasticsearch:7.6.2' +}.freeze -BASE_IMAGE=${BASE_IMAGE_DICT[$(arch)]} +BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] -docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE="$BASE_IMAGE" . +available_memory = get_available_memory + +system "docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} --build-arg MEMORY=#{available_memory} ." -- 2.23.0

1 0

[PATCH v3 compass-ci 2/3] container/es: set available memory size
by Liu Yinsi 02 Dec '20

02 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/es/Dockerfile | 6 ++++-- container/es/build | 8 ++++++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/container/es/Dockerfile b/container/es/Dockerfile index e626dae..ed02490 100644 --- a/container/es/Dockerfile +++ b/container/es/Dockerfile @@ -3,6 +3,8 @@ FROM alpine:3.11 +ARG MEMORY + RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk add --no-cache elasticsearch curl @@ -27,8 +29,8 @@ RUN rm -rf /etc/init.d/elasticsearch \ RUN sed -i 's:#path.data\: /path/to/data:path.data\: /srv/es:' /usr/share/es/config/elasticsearch.yml; RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/es/config/elasticsearch.yml; -RUN sed -i "s/-Xms256m/-Xms20g/g" /usr/share/es/config/jvm.options -RUN sed -i "s/-Xmx256m/-Xmx20g/g" /usr/share/es/config/jvm.options +RUN sed -i "s/-Xms256m/-Xms${MEMORY}m/g" /usr/share/es/config/jvm.options +RUN sed -i "s/-Xmx256m/-Xmx${MEMORY}m/g" /usr/share/es/config/jvm.options WORKDIR /usr/share/es diff --git a/container/es/build b/container/es/build index d5607e0..17a225c 100755 --- a/container/es/build +++ b/container/es/build @@ -1,6 +1,10 @@ -#!/bin/sh +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -docker build -t es643b:alpine311 . --network=host +require_relative '../defconfig.rb' +available_memory = get_available_memory + +system "docker build -t es643b:alpine311 --build-arg MEMORY=#{available_memory} --network=host ." -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] container/defconfig.rb: limit container memory usage
by Liu Yinsi 02 Dec '20

02 Dec '20

use Math.sqrt(memory) to get container occupied memory according to system memory size, instead of setting fixed size, more reasonable. In addition, '-Xms' not support floating number, so use to_i. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index 05c56db..e3ca89c 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -35,3 +35,11 @@ def docker_rm(container) system "docker stop #{container} && docker rm -f #{container}" end + +def get_available_memory + memory = File.readlines('/proc/meminfo')[0].chomp.split[1].to_f/1048576 + + # set container available memory size, minimum size is 1024m, maximum size is 30720m, + # take the middle value according to the system memory size. + [1024, 30720, Math.sqrt(memory)*1024].sort[1].to_i +end -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: set "pub_key" field as level-1 parameter for password-free login
by Zhang Yale 02 Dec '20

02 Dec '20

[why] The "pub_key" field in yaml is used as the field to determine whether to password-free login. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- src/lib/job.cr | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index a4402ab..44a6ecf 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -184,7 +184,7 @@ class Job end private def set_sshr_info - return unless self["sshd"]? + return unless @hash["pub_key"]? self["sshr_port"] = ENV["SSHR_PORT"] self["sshr_port_base"] = ENV["SSHR_PORT_BASE"] @@ -196,8 +196,7 @@ class Job end private def set_my_ssh_pubkey - sshd = @hash["sshd"].as_h? || Hash(String, JSON::Any).new() - pub_key = sshd["pub_key"]?.to_s + pub_key = @hash["pub_key"]?.to_s update_account_my_pub_key(pub_key) @hash["my_ssh_pubkey"] = @account_info["my_ssh_pubkey"] -- 2.23.0

2 2

[PATCH compass-ci] define_files.md: text format adapt to markdown
by Xu Xijian 02 Dec '20

02 Dec '20

In markdown documents, text for bash should be contained in block as belows: ```bash content ``` Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/define_files.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/job/define_files.md b/doc/job/define_files.md index 2c3457d..601d498 100644 --- a/doc/job/define_files.md +++ b/doc/job/define_files.md @@ -1,6 +1,7 @@ # define_files Meaning: +```bash - We use lkp-tests as client for compass-ci. There are many testcases in lkp-tests. most testcases contain the following files: distro/depends/$testcase distro/depends/$testcase-dev @@ -10,7 +11,10 @@ Meaning: jobs/$testcase.yaml - When user add new testcase files to lkp-tests or change existing testcase files in lkp-tests, these change files that related with test program would be added to define_files field. - The define_files field do not need to be specified by user in the $testcase.yaml. It can be generated automatically when user submit with the option '-a'. User can confirm the define_files field in the job.yaml under result_root. +``` Usage example: +```bash - % submit -a $testcase.yaml testbox=vm-2p8g--$USER os=openeuler os_arch=aarch64 os_version=20.03 - % cat /srv/result/$testcase/vm-2p8g--$USER/$date/$job_id/job.yaml +``` -- 2.23.0

2 3

[PATCH compass-ci] providers/docker: delete the noexec flag of tmpfs
by Xiao Shenwei 02 Dec '20

02 Dec '20

error messgae: ./configure permission denied [why] tmpfs default option (rw,nosuid,noexec,nodev) can't execute stuff from it cause of the noexec flag refer-to: https://github.com/docker/compose/issues/3425 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/run.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 9edad62..d532a56 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -23,7 +23,7 @@ cmd=( docker run --rm -m $memory - --mount type=tmpfs,destination=/tmp + --tmpfs /tmp:rw,exec,nosuid,nodev -e CCI_SRC=/c/compass-ci -v ${load_path}/lkp:/lkp -v ${DIR}/bin:/root/bin:ro -- 2.23.0

2 3

[PATCH v2 compass-ci 2/3] send-internet-mail/mail-post: call to store email
by Luan Shengde 02 Dec '20

02 Dec '20

call function store_email to store email to directory sent Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/mail-post.rb b/lib/mail-post.rb index 0b1f94a..cac7a76 100755 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -51,5 +51,11 @@ def send_mail(mail_info) to mail_info['to'] body mail_info['body'] end + mail.deliver! + + return unless ENV['SEND_MAIL_PORT'].to_s == '49000' + return unless ENV['HOST_SERVER'] == 'z9' + + store_email(mail) end -- 2.23.0

1 0

[PATCH v2 compass-ci] providers/docker: delete the noexec flag of tmpfs
by Xiao Shenwei 02 Dec '20

02 Dec '20

error messgae: PKGBUILD: ./configure permission denied [why] tmpfs default option (rw,nosuid,noexec,nodev) can't be executed directly cause of the noexec flag refer-to: https://github.com/docker/compose/issues/3425 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/run.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 9edad62..d532a56 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -23,7 +23,7 @@ cmd=( docker run --rm -m $memory - --mount type=tmpfs,destination=/tmp + --tmpfs /tmp:rw,exec,nosuid,nodev -e CCI_SRC=/c/compass-ci -v ${load_path}/lkp:/lkp -v ${DIR}/bin:/root/bin:ro -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] providers/docker: delete the noexec flag of tmpfs
by Xiao Shenwei 02 Dec '20

02 Dec '20

error messgae: ./configure permission denied [why] tmpfs default option (rw,nosuid,noexec,nodev) can't execute stuff from it cause of the noexec flag refer-to: https://github.com/docker/compose/issues/3425 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/run.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 9edad62..d532a56 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -23,7 +23,7 @@ cmd=( docker run --rm -m $memory - --mount type=tmpfs,destination=/tmp + --tmpfs /tmp:rw,exec,nosuid,nodev -e CCI_SRC=/c/compass-ci -v ${load_path}/lkp:/lkp -v ${DIR}/bin:/root/bin:ro -- 2.23.0

1 1

[PATCH v3 compass-ci 2/2] container/netdata-slave: monitoring HW server
by Xiao Shenwei 02 Dec '20

02 Dec '20

[monitored object] file-server : master crystal.ci : slave z9 : slave ibmc-mgnt : slave backup-server: slave jumper-server: slave startup by multi-qemu or multi-docker machine [function] slave node will collect host machine metrics, send to master slave node does not provider data storage, alarm, web services. [usage] config/stream.conf destination = master-HOST[:PORT] api key = uuid refer-to: https://github.com/netdata/netdata/tree/master/streaming Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/netdata-slave/Dockerfile | 10 ++++++ container/netdata-slave/build | 5 +++ container/netdata-slave/config/netdata.conf | 5 +++ container/netdata-slave/config/stream.conf | 10 ++++++ container/netdata-slave/run | 34 +++++++++++++++++++++ 5 files changed, 64 insertions(+) create mode 100755 container/netdata-slave/Dockerfile create mode 100755 container/netdata-slave/build create mode 100644 container/netdata-slave/config/netdata.conf create mode 100644 container/netdata-slave/config/stream.conf create mode 100755 container/netdata-slave/run diff --git a/container/netdata-slave/Dockerfile b/container/netdata-slave/Dockerfile new file mode 100755 index 0000000..5c2c02f --- /dev/null +++ b/container/netdata-slave/Dockerfile @@ -0,0 +1,10 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/netdata/netdata +# SPDX-License-Identifier: GPL-3.0+ + +FROM netdata/netdata + +MAINTAINER Xiao Shenwei <xiaoshenwei96(a)163.com> + +RUN usermod -u 1090 netdata && \ + groupmod -g 1090 netdata diff --git a/container/netdata-slave/build b/container/netdata-slave/build new file mode 100755 index 0000000..1a84550 --- /dev/null +++ b/container/netdata-slave/build @@ -0,0 +1,5 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/netdata/netdata +# SPDX-License-Identifier: GPL-3.0+ + +docker build -t netdata/netdata-slave . diff --git a/container/netdata-slave/config/netdata.conf b/container/netdata-slave/config/netdata.conf new file mode 100644 index 0000000..456fe02 --- /dev/null +++ b/container/netdata-slave/config/netdata.conf @@ -0,0 +1,5 @@ +[global] + memory mode = none + +[health] + enabled = no diff --git a/container/netdata-slave/config/stream.conf b/container/netdata-slave/config/stream.conf new file mode 100644 index 0000000..ac91a7d --- /dev/null +++ b/container/netdata-slave/config/stream.conf @@ -0,0 +1,10 @@ +[stream] + enabled = yes + destination = + api key = + timeout seconds = 60 + default port = 19999 + send charts matching = * + buffer size bytes = 1048576 + reconnect delay seconds = 5 + initial clock resync iterations = 60 diff --git a/container/netdata-slave/run b/container/netdata-slave/run new file mode 100755 index 0000000..e790298 --- /dev/null +++ b/container/netdata-slave/run @@ -0,0 +1,34 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/netdata/netdata +# SPDX-License-Identifier: GPL-3.0+ + +. $CCI_SRC/container/defconfig.sh + +docker_rm netdata-slave + +DIR=$(dirname $(realpath $0)) + +cmd=( + docker run -d + --restart=unless-stopped + --hostname=$HOSTNAME + --name netdata-slave + -p 19999:19999 + -v $DIR/config/netdata.conf:/etc/netdata/netdata.conf + -v $DIR/config/stream.conf:/etc/netdata/stream.conf + -v /etc/passwd:/host/etc/passwd:ro + -v /etc/group:/host/etc/group:ro + -v /proc:/host/proc:ro + -v /sys:/host/sys:ro + -v /etc/localtime:/etc/localtime:ro + -v /etc/os-release:/host//etc/os-release:ro + -v /var/run/docker.sock:/var/run/docker.sock:ro + -v /srv/cache/netdata_cache:/var/cache/netdata + -v /srv/cache/netdata_lib:/var/lib/netdata + --log-driver json-file + --log-opt max-size=1g + --cap-add SYS_PTRACE + --security-opt apparmor=unconfined + netdata/netdata-slave +) +"${cmd[@]}" -- 2.23.0

1 0

[PATCH compass-ci] sched: no need set kernel if os mount is container
by Cao Xueliang 02 Dec '20

02 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index a4402ab..5798ae3 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -146,16 +146,22 @@ class Job set_result_root() set_result_service() set_os_mount() + set_kernel() set_depends_initrd() - set_kernel_version() set_initrds_uri() - set_kernel_uri() - set_kernel_params() set_lkp_server() set_sshr_info() set_queue() end + private def set_kernel + return if os_mount == "container" + + set_kernel_version() + set_kernel_uri() + set_kernel_params() + end + private def append_init_field INIT_FIELD.each do |k, v| k = k.to_s -- 2.23.0

2 1

[PATCH v2 compass-ci] doc/job: add arch-nr_node-nr_cpu-memory.md
by Zhang Yu 01 Dec '20

01 Dec '20

Add a document for introducing the arch, nr_node, nr_cpu, memory fields Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/arch-nr_node-nr_cpu-memory.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 doc/job/arch-nr_node-nr_cpu-memory.md diff --git a/doc/job/arch-nr_node-nr_cpu-memory.md b/doc/job/arch-nr_node-nr_cpu-memory.md new file mode 100644 index 0000000..ba12368 --- /dev/null +++ b/doc/job/arch-nr_node-nr_cpu-memory.md @@ -0,0 +1,18 @@ +# arch, nr_node, nr_cpu, memory + +Meaning: +- Each test machine has some basic information, such as arch, nr_node, nr_cpu, memory + these fields mean: + arch: the architecture of the machine. + nr_node: number of NUMA nodes. + nr_cpu: number of logical CPUs. + memory: the storage space of the machine. + +- These fields are from the file which in $LAB_SRC/hosts. +- Here is an example of these fields in the host file: + arch: aarch64 + nr_node: 4 + nr_cpu: 96 + memory: 256G + ... + -- 2.23.0

2 1

[PATCH v3 compass-ci] lib: add error_messages.rb
by Lin Jiaxin 01 Dec '20

01 Dec '20

provide error messages for git bisect Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 63 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100644 lib/error_messages.rb diff --git a/lib/error_messages.rb b/lib/error_messages.rb new file mode 100644 index 0000000..f59c73a --- /dev/null +++ b/lib/error_messages.rb @@ -0,0 +1,63 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' + +# obtain error messages hash +class ErrorMessages + def initialize(file) + @file = file + @seqno = '' + @in_stderr = false + @error_line = '' + @error_details = '' + @error_message = {} + @error_messages = Hash.new { |h, k| h[k] = Set.new } + end + + def obtain_error_messages + File.open((a)file).each_line do |line| + next if extract_error_message(line) + next unless @in_stderr + next unless @error_message['error_line'].downcase =~ /error|warning/ + + @error_messages[@error_message['error_line']] << @error_message['error_line'] + @error_message['error_details'] + end + return @error_messages + end + + private + + def extract_error_message(line) + if line =~ /^ / + @error_details += line + else + update_error_message(line) + case line + when /^## ______________([0-9.]+):stderr$/ + @in_stderr = true + @seqno = $1 + return true + + when /^## ______________#{@seqno}:enderr$/ + @in_stderr = false + @seqno = '' + add_error_message + end + end + end + + def update_error_message(line) + @error_message['error_line'] = @error_line + @error_message['error_details'] = @error_details + @error_line = line + @error_details = '' + end + + def add_error_message + return unless @error_message['error_line'].downcase =~ /error|warning/ + + @error_messages[@error_message['error_line']] << @error_message['error_line'] + @error_message['error_details'] + end +end -- 2.23.0

2 1

[PATCH v3 compass-ci] 本地安装compass-ci客户端.md: text format adapt to markdown
by Xu Xijian 01 Dec '20

01 Dec '20

In markdown documents, text for bash should be contained in block as below: ```bash content ``` Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- ...50\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" | 2 ++ 1 file changed, 2 insertions(+) diff --git "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" index 43a6bf5..9b89d27 100644 --- "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" +++ "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" @@ -8,6 +8,7 @@ Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客 ## Getting started +```bash 1. setup default config run the following command to add the below setup to default config file mkdir -p ~/.config/compass-ci/defaults/ @@ -23,6 +24,7 @@ Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客 cd lkp-tests make install source ~/.bashrc && source ~/.bash_profile +``` 3. submit job Now try [submitting a job to compass-ci](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… -- 2.23.0

1 0

[PATCH v2 compass-ci] 本地安装compass-ci客户端.md: text format adapt to markdown
by Xu Xijian 01 Dec '20

01 Dec '20

In markdown documents, text for bash should be contained in block as below: ```bash content ``` Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- ...50\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" | 2 ++ 1 file changed, 2 insertions(+) diff --git "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" index 43a6bf5..46fde13 100644 --- "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" +++ "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" @@ -8,6 +8,7 @@ Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客 ## Getting started +```bash 1. setup default config run the following command to add the below setup to default config file mkdir -p ~/.config/compass-ci/defaults/ @@ -26,3 +27,4 @@ Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客 3. submit job Now try [submitting a job to compass-ci](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… +``` -- 2.23.0

1 1

[PATCH compass-ci] 本地安装compass-ci客: text format adapt to markdown
by Xu Xijian 01 Dec '20

01 Dec '20

In markdown documents, text for bash should be contained in block as below: ```bash content ``` Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- ...50\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" | 2 ++ 1 file changed, 2 insertions(+) diff --git "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" index 43a6bf5..46fde13 100644 --- "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" +++ "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" @@ -8,6 +8,7 @@ Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客 ## Getting started +```bash 1. setup default config run the following command to add the below setup to default config file mkdir -p ~/.config/compass-ci/defaults/ @@ -26,3 +27,4 @@ Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客 3. submit job Now try [submitting a job to compass-ci](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… +``` -- 2.23.0

3 5

[PATCH lkp-tests] lib/job.rb: fix submit matrix args by cmd args
by Wei Jihui 01 Dec '20

01 Dec '20

[why] submit "os os_arch os_version"="1 2 3" this submit will split this arg to: os, os_arch, os_version=1, 2, 3 [how] change space to '|', like: submit "os|os_arch|os_version"="1|2|3" ... Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index f9e244a9..2869dcb3 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -288,13 +288,13 @@ class Job def multi_args(hash) jobs_array = [] hash.each { |key, value| - next unless key =~ /^\w.*\s+.*\w$/ + next unless key =~ /^\w.*\|.*\w$/ - next unless value.class == Array + key_array = key.split('|') + [value].flatten.each do |v| + next unless v =~ /^\w.*\|.*\w$/ - key_array = key.split - value.each do |v| - v_array = v.split + v_array = v.split('|') next unless key_array.size == v_array.size hash_job = hash.clone -- 2.23.0

1 0

[PATCH v2 compass-ci 4/4] container: add locate_files in assistant container
by Cao Xueliang 01 Dec '20

01 Dec '20

An interface is provided to support obtaining the real path of the /srv/initrd/deps and pkg program. Example: Input: curl -H 'Content-Type: Application/json' -XPOST '172.17.0.1:8101/locate_files' -d '{"deps_files": ["/srv/initrd/deps/nfs/openeuler/aarch64/20.03/perf/perf.cgz"], "pkg_files": ["/srv/initrd/pkg/nfs/openeuler/aarch64/20.03/plzip/latest.cgz", "/srv/initrd/pkg/nfs/openeuler/aarch64/20.03/stream/latest.cgz"]}' Output: {"deps_files":["/srv/initrd/deps/nfs/openeuler/aarch64/20.03/perf/perf_20201104.cgz"], "pkg_files":["/srv/initrd/pkg/nfs/openeuler/aarch64/20.03/plzip/1-4.cgz", "/srv/initrd/pkg/nfs/openeuler/aarch64/20.03/stream/1-1.cgz"]} Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/routes.rb | 31 +++++++++++++++++++++++ container/assistant/views/locate_files.rb | 19 ++++++++++++++ 2 files changed, 50 insertions(+) create mode 100755 container/assistant/routes.rb create mode 100644 container/assistant/views/locate_files.rb diff --git a/container/assistant/routes.rb b/container/assistant/routes.rb new file mode 100755 index 0000000..54d3670 --- /dev/null +++ b/container/assistant/routes.rb @@ -0,0 +1,31 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'json' +require 'open3' + +require_relative './views/locate_files' + +set :bind, '0.0.0.0' +set :port, 8101 + +post '/locate_files' do + request.body.rewind + + begin + data = JSON.parse request.body.read + rescue JSON::ParserError + return [400, 'parse json params error'] + end + + begin + result = locate_files(data) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + [200, result.to_json] +end diff --git a/container/assistant/views/locate_files.rb b/container/assistant/views/locate_files.rb new file mode 100644 index 0000000..9699128 --- /dev/null +++ b/container/assistant/views/locate_files.rb @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def locate_files(data) + result = {} + + data.each do |key, value| + temp = [] + value.each do |val| + val = val.strip + temp << File.realpath(val) if File.exist?(val) + end + + result.merge!({ "#{key}": temp }) + end + + return result +end -- 2.23.0

1 0

[PATCH v2 compass-ci 3/4] container: start script for assistant container
by Cao Xueliang 01 Dec '20

01 Dec '20

This is a assistant container, it provides some APIs. For now, it provides the locate_files function to give the real path for programs in /srv/initrd/deps and pkg. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/start | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100755 container/assistant/start diff --git a/container/assistant/start b/container/assistant/start new file mode 100755 index 0000000..5f7822e --- /dev/null +++ b/container/assistant/start @@ -0,0 +1,21 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +docker_rm assistant + +cmd=( + docker run + --restart=always + --name assistant + -u nobody + -d + -p 8101:8101 + -v /srv/initrd:/srv/initrd + -v /etc/localtime:/etc/localtime:ro + debian:assistant +) + +"${cmd[@]}" -- 2.23.0

1 0

[PATCH v2 compass-ci 2/4] container: build script for assistant container
by Cao Xueliang 01 Dec '20

01 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/assistant/build diff --git a/container/assistant/build b/container/assistant/build new file mode 100755 index 0000000..7778c9f --- /dev/null +++ b/container/assistant/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t debian:assistant . -- 2.23.0

1 0

[PATCH v2 compass-ci 1/4] container: assistant container Dockerfile
by Cao Xueliang 01 Dec '20

01 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/Dockerfile | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 container/assistant/Dockerfile diff --git a/container/assistant/Dockerfile b/container/assistant/Dockerfile new file mode 100644 index 0000000..f1ad7a2 --- /dev/null +++ b/container/assistant/Dockerfile @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER caoxl78320(a)163.com + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y ruby-json ruby-sinatra curl + +COPY routes.rb /usr/local/bin/ +COPY views /usr/local/bin/views/ + +CMD ["/usr/local/bin/routes.rb"] -- 2.23.0

1 0

[PATCH compass-ci 3/3] scheduler/scheduler.cr: add start_time in before_all
by Wu Zhende 01 Dec '20

01 Dec '20

Record the interface access time for calculating the interface invoking time Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/scheduler.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index 98b8e2e..9f5e670 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -31,8 +31,10 @@ module Scheduler VERSION = "0.2.0" add_context_storage_type(Sched) + add_context_storage_type(Time::Span) before_all do |env| + env.set "start_time", Time.monotonic env.set "sched", Sched.new(env) env.response.headers["Connection"] = "close" end -- 2.23.0

1 0

[PATCH compass-ci 2/3] lib/sched.cr: initialize JSONLogger
by Wu Zhende 01 Dec '20

01 Dec '20

initialize JSONLogger as @log, so can use @log in each interface in Shced. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 26ddd92..a9f893f 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -5,6 +5,7 @@ require "kemal" require "yaml" require "./job" +require "./json_logger" require "./block_helper" require "./taskqueue_api" require "./remote_git_client" @@ -32,6 +33,8 @@ class Sched @block_helper = BlockHelper.new @rgc = RemoteGitClient.new @env = env + @log = JSONLogger.new + @log.set_env(env) end def normalize_mac(mac : String) -- 2.23.0

1 0

[PATCH compass-ci 1/3] lib/json_logger: support env
by Wu Zhende 01 Dec '20

01 Dec '20

API can set env, after set env will add some env info to log. Add: status_code/method/resource/elapsed Like this: {"level_num":1,"level":"INFO","time":"2020-12-01 20:03:19+08:00","message":"test","status_code":200,"method":"POST","resource":"/submit_job","elapsed":"133.68µs"} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/json_logger.cr | 65 ++++++++++++++++++++++++++++++------------ 1 file changed, 47 insertions(+), 18 deletions(-) diff --git a/src/lib/json_logger.cr b/src/lib/json_logger.cr index 371c042..2b5ef6a 100755 --- a/src/lib/json_logger.cr +++ b/src/lib/json_logger.cr @@ -6,27 +6,56 @@ require "json" require "any_merge" class JSONLogger < Logger - FORMATTER = Logger::Formatter.new do | severity, datetime, progname, msg, io| - level_num = severity.to_i32 - logger_hash = JSON.parse(%({"level_num": #{level_num}, - "level": "#{severity}", - "time": "#{datetime}" - })).as_h - - logger_hash.any_merge!({"progname" => progname}) unless progname.empty? - logger_hash.merge!(JSON.parse(%({"caller": #{caller}})).as_h) if level_num >= 2 - - begin - message = JSON.parse(msg).as_h - rescue - message = {"message" => msg} + def initialize(logdev = STDOUT, formatter = my_formatter) + @env = nil + @env_info = Hash(String, String | Int32).new + super(logdev, formatter: formatter) + end + + def my_formatter + Logger::Formatter.new do | severity, datetime, progname, msg, io| + get_env_info(@env.as(HTTP::Server::Context)) if @env + level_num = severity.to_i32 + logger_hash = JSON.parse(%({"level_num": #{level_num}, + "level": "#{severity}", + "time": "#{datetime}" + })).as_h + + logger_hash.any_merge!({"progname" => progname}) unless progname.empty? + logger_hash.merge!(JSON.parse(%({"caller": #{caller}})).as_h) if level_num >= 2 + + begin + message = JSON.parse(msg).as_h + rescue + message = {"message" => msg} + end + logger_hash.any_merge!(message) + logger_hash.any_merge!(@env_info) + + io << logger_hash.to_json end - logger_hash.any_merge!(message) + end - io << logger_hash.to_json + private def get_env_info(env : HTTP::Server::Context) + @env_info["status_code"] = env.response.status_code + @env_info["method"] = env.request.method + @env_info["resource"] = env.request.resource + + elapsed = get_elapsed(env) + @env_info["elapsed"] = elapsed.to_s if elapsed end - def initialize(logdev = STDOUT, formatter = FORMATTER) - super(logdev, formatter: formatter) + private def get_elapsed(env : HTTP::Server::Context) + start_time = env.get?("start_time") + return unless start_time + + elapsed = (Time.monotonic - start_time.as(Time::Span)).total_milliseconds + return "#{elapsed.round(2)}ms" if elapsed >= 1 + + "#{(elapsed * 1000).round(2)}µs" + end + + def set_env(env : HTTP::Server::Context) + @env = env end end -- 2.23.0

1 0

[PATCH v1 compass-ci 4/4] container: add locate_files in assistant container
by Cao Xueliang 01 Dec '20

01 Dec '20

An interface is provided to support obtaining the real path of the /srv/initrd/deps and pkg program. Example: Input: curl -H 'Content-Type: Application/json' -XPOST '172.17.0.1:8101/locate_files' -d '{"deps_files": ["/srv/initrd/deps/nfs/openeuler/aarch64/20.03/perf/perf.cgz"], "pkg_files": ["/srv/initrd/pkg/nfs/openeuler/aarch64/20.03/plzip/latest.cgz", "/srv/initrd/pkg/nfs/openeuler/aarch64/20.03/stream/latest.cgz"]}' Output: {"deps_files":["/srv/initrd/deps/nfs/openeuler/aarch64/20.03/perf/perf_20201104.cgz"], "pkg_files":["/srv/initrd/pkg/nfs/openeuler/aarch64/20.03/plzip/1-4.cgz", "/srv/initrd/pkg/nfs/openeuler/aarch64/20.03/stream/1-1.cgz"]} Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/routes.rb | 31 +++++++++++++++++++++++ container/assistant/views/locate_files.rb | 21 +++++++++++++++ 2 files changed, 52 insertions(+) create mode 100755 container/assistant/routes.rb create mode 100644 container/assistant/views/locate_files.rb diff --git a/container/assistant/routes.rb b/container/assistant/routes.rb new file mode 100755 index 0000000..54d3670 --- /dev/null +++ b/container/assistant/routes.rb @@ -0,0 +1,31 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'json' +require 'open3' + +require_relative './views/locate_files' + +set :bind, '0.0.0.0' +set :port, 8101 + +post '/locate_files' do + request.body.rewind + + begin + data = JSON.parse request.body.read + rescue JSON::ParserError + return [400, 'parse json params error'] + end + + begin + result = locate_files(data) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + [200, result.to_json] +end diff --git a/container/assistant/views/locate_files.rb b/container/assistant/views/locate_files.rb new file mode 100644 index 0000000..5769177 --- /dev/null +++ b/container/assistant/views/locate_files.rb @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def locate_files(data) + result = {} + + data.each do |key, value| + temp = [] + value.each do |val| + val = val.strip + if File.exist?(val) + temp << File.realpath(val) + end + end + + result.merge!({ "#{key}": temp }) + end + + return result +end -- 2.23.0

2 2

[PATCH v1 compass-ci 3/4] container: start script for assistant container
by Cao Xueliang 01 Dec '20

01 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/start | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100755 container/assistant/start diff --git a/container/assistant/start b/container/assistant/start new file mode 100755 index 0000000..5f7822e --- /dev/null +++ b/container/assistant/start @@ -0,0 +1,21 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +docker_rm assistant + +cmd=( + docker run + --restart=always + --name assistant + -u nobody + -d + -p 8101:8101 + -v /srv/initrd:/srv/initrd + -v /etc/localtime:/etc/localtime:ro + debian:assistant +) + +"${cmd[@]}" -- 2.23.0

2 2

[PATCH v1 compass-ci 1/4] container: assistant container Dockerfile
by Cao Xueliang 01 Dec '20

01 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/assistant/Dockerfile diff --git a/container/assistant/Dockerfile b/container/assistant/Dockerfile new file mode 100644 index 0000000..0287747 --- /dev/null +++ b/container/assistant/Dockerfile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y ruby-json ruby-sinatra curl + +COPY routes.rb /usr/local/bin/ +COPY views /usr/local/bin/views/ + +CMD ["/usr/local/bin/routes.rb"] -- 2.23.0

2 2

[PATCH v2 compass-ci 4/4] container/defconfig: fix rubocop warning
by Liu Yinsi 01 Dec '20

01 Dec '20

defconfig.rb:23:12: C: [Corrected] Style/BlockDelimiters: Avoid using {...} for multi-line blocks. hash.map { |k, v| Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/defconfig.rb b/container/defconfig.rb index 026a183..320fba4 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -20,9 +20,9 @@ end def set_local_env hash = cci_defaults - hash.map { |k, v| + hash.map do |k, v| system "export #{k}=#{v}" - } + end end def docker_env(hash) -- 2.23.0

2 2

[PATCH v2 lkp-tests 2/2] sbin: load PKGBUILD.src content
by Liu Shaofei 01 Dec '20

01 Dec '20

The contents of PKGBUILD.src is the following example: source=( "http://172.168.131.113:8800/initrd/build-tar/ansible/ansible-2.9.4.tar.gz" ) Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- sbin/makepkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sbin/makepkg b/sbin/makepkg index 6f8b95af..498b4872 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -895,6 +895,7 @@ update_pkgver() { sed --follow-symlinks -i "s:^pkgrel=[^ ]*:pkgrel=1:" "$BUILDFILE" source_safe "$BUILDFILE" [[ -n $PKGBUILD_TAG ]] && source_safe "$BUILDFILE-$PKGBUILD_TAG" + [[ -f PKGBUILD.src ]] && source_safe "PKGBUILD.src" local fullver=$(get_full_version) msg "$(gettext "Updated version: %s")" "$pkgbase $fullver" else @@ -3699,6 +3700,7 @@ else fi source_buildfile "$BUILDFILE" [[ -n $PKGBUILD_TAG ]] && source_buildfile "$BUILDFILE-$PKGBUILD_TAG" + [[ -f PKGBUILD.src ]] && source_buildfile PKGBUILD.src fi # set defaults if they weren't specified in buildfile -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] tests: replace *.tar.* address with local address
by Liu Shaofei 01 Dec '20

01 Dec '20

When the address of source field is the format of "*.tar.*", we need to replace it with local address. The following example: source=("https://github.com/azukiapp/${pkgname}/archive/v${pkgver}.tar.gz") => source=("http://${LKP_SERVER}:8800/initrd/build-tar/${pkgname}/v${pkgver}.tar.gz") Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index bdaef035..617dc0b7 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -98,6 +98,26 @@ replace_source() sed -i "s|^source=[^)]*|$1|g" "$2" } +# replace *.tar.* address from source field in PKGBUILD +replace_tar_source() +{ + echo "source=(" > PKGBUILD.src + for url in ${source[@]} + do + echo "$url" | egrep '.+\.tar\..+|.+\.tar' && { + local_url="http://${LKP_SERVER}:8800/initrd/build-tar/${pkgname}/${url##*/}" + + # check whether local server exists *.tar.* file + wget -q --spider "${local_url}" + [ "$?" == 0 ] || continue + + url="\"${local_url}\"" + } + echo "$url" >> PKGBUILD.src + done + echo ")" >> PKGBUILD.src +} + build_source_pkg() { local repo_dir="" @@ -116,6 +136,8 @@ build_source_pkg() [ -n "$PKGBUILD_TAG" ] && replace_source "${upstream_source}" "PKGBUILD-$PKGBUILD_TAG" } + replace_tar_source + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 -- 2.23.0

1 0

[PATCH lab-z9] lab-z9: add cluster from a103 to a107
by Wei Jihui 01 Dec '20

01 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- cluster/cs-s1-a105-c4 | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 cluster/cs-s1-a105-c4 diff --git a/cluster/cs-s1-a105-c4 b/cluster/cs-s1-a105-c4 new file mode 100644 index 0000000..fa29b74 --- /dev/null +++ b/cluster/cs-s1-a105-c4 @@ -0,0 +1,22 @@ +switch: Switch-P10 +ip0: 2 +nodes: + taishan200-2280-2s48p-512g--a105: + roles: [ server ] + macs: [ "44:67:47:d7:76:c4" ] + + taishan200-2280-2s48p-256g--a103: + roles: [ client1 ] + macs: [ "44:67:47:c9:d8:b8" ] + + taishan200-2280-2s48p-256g--a104: + roles: [ client2 ] + macs: [ "44:67:47:d7:6d:f0" ] + + taishan200-2280-2s48p-256g--a106: + roles: [ client3 ] + macs: [ "44:67:47:86:1c:30" ] + + taishan200-2280-2s48p-256g--a107: + roles: [ client4 ] + macs: [ "84:46:fe:73:a2:71" ] -- 2.23.0

1 0

[PATCH v5 compass-ci] doc/manual: add how-to-log-in-the-machine.md
by Zhang Yu 01 Dec '20

01 Dec '20

Add a document for introducing how to log in the machine Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/manual/how-to-log-in-the-machine.md | 86 +++++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 doc/manual/how-to-log-in-the-machine.md diff --git a/doc/manual/how-to-log-in-the-machine.md b/doc/manual/how-to-log-in-the-machine.md new file mode 100644 index 0000000..fefd718 --- /dev/null +++ b/doc/manual/how-to-log-in-the-machine.md @@ -0,0 +1,86 @@ +这篇文档将告诉你如何登陆测试环境 + +# 1. 前提条件 +请先学习: +* [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…, 配置个人邮箱 +* [如何申请测试机.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua…, 并在本地生成RSA公私钥对 +* [submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma…, submit 命令详解 + +# 2. 操作方法 +## 2.1 在job.yaml里加上sshd和sleep字段, 以host-info.yaml任务为例： + +```shell + suite: host-info + category: functional + pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + sshd: + # sleep at the bottom + sleep: 1h + + host-info: +``` + +## 2.2 以下有2种方式可以登录到测试机： +### 第一种：使用submit -m -c的方式： + 这种方式提交的任务会在指定的位置sleep并直接登陆到测试机中，适用于登陆测试环境后手动调试 + 命令：submit -m -c host-info.yaml + 任务提交完成后，当测试执行到sshd后会自动登陆到测试机器上： + 效果如下： + +```shell + hi6325@account-vm ~% submit -m -c atomic.yaml + submit atomic.yaml, got job_id=crystal.146528 + query=>{"job_id":["crystal.146528"]} + connect to ws://localhost:11310/filter + {"job_id": "crystal.146528", "result_root": "/srv/result/atomic/2020-12-01/vm-2p8g/openeuler-20.03-aarch64/1-1000/crystal.146528", "job_state": "set result root" + {"job_id": "crystal.146528", "job_state": "boot"} + {"job_id": "crystal.146528", "job_state": "download"} + "time":"2020-12-01 10:12:33","mac":"0a-2d-7b-d9-f8-b1","ip":"172.18.252.12","job_id":"crystal.146528","state":"running","testbox":"vm-2p8g.zhyl-453231"} + {"job_state":"running","job_id":"crystal.146528"} + {"job_id": "crystal.146528", "state": "set ssh port", "ssh_port": "51750", "tbox_name": "vm-2p8g.zhyl-453231"} + + root@vm-2p8g ~# +``` + +### 第二种：根据邮件信息使用ssh方式登录测试机： + 这种是使用submit方式提交的任务完成后，系统自动发送一封邮件提醒您可以在指定时间内登陆到测试环境 + 命令： submit host-info.yaml + 任务执行完成后，系统发送邮件内容如下： + +```shell + Subject: [NOTIFY Compass-ci] vm-2p8g-294828 ready to use + + Dear $my_username: + Thanks for your participation in software ecosystem! + According to your application, vm-2p8g-294828 has been provisioned. + The datails are as follows: + + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch + Regards + Compass-Ci +``` + + 可通过ssh方式登陆到测试环境： + 命令： ssh root(a)api.compass-ci.openeuler.org -p $port + 效果如下： + +```shell + hi6325@account-vm ~% ssh root(a)api.compass-ci.openeuler.org -p 51400 + + root@vm-2p8g ~# +``` -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/compare_matrixes.rb: change stddev to stddev_percent
by Lu Kaiyi 01 Dec '20

01 Dec '20

":stddev" doesn't exists in metric_value, change it to ":stddev_percent" to normally access the value of "standard deviation". Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_matrixes.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 7a9870a..2bb6a61 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -460,7 +460,7 @@ def assign_metric_values(metrics_values, dim, metric, values) metrics_values[metric]['standard_deviation'] ||= {} metric_value = get_values(values, true) metrics_values[metric]['average'][dim] = metric_value[:average] - metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev] || 0 + metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev_percent] || 0 end def assign_metric_change(metrics_values) -- 2.23.0

1 0

[PATCH compass-ci] doc/job: add nr_threads.md
by Bai Jing 01 Dec '20

01 Dec '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- doc/job/nr_threads | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/job/nr_threads diff --git a/doc/job/nr_threads b/doc/job/nr_threads new file mode 100644 index 0000000..487b5f8 --- /dev/null +++ b/doc/job/nr_threads @@ -0,0 +1,16 @@ +# nr_threads + +Meaning: +- The number of threads +- You can specify the number of threads to be started for +the test based on your test requirements. + +Usage example: + +```yaml +suite: spinlock + +nr_threads: +- 1 +- 2 +``` -- 2.23.0

1 0

Re: [PATCH compass-ci] scheduler: move 'auto_submit_idle_job' and add account info
by Ren Wen 01 Dec '20

01 Dec '20

Please ignore this mail. @es.get_account may get a String. Thanks, RenWen >+ def auto_submit_idle_job(tbox_group) >+ account_info = @es.get_account(TEAM_ACCOUNT).as_h > end >-- >2.23.0 >

1 0

[PATCH v4 compass-ci] doc/manual: add how-to-log-in-the-machine.md
by Zhang Yu 01 Dec '20

01 Dec '20

Add a document for introducing how to log in the machine Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/manual/how-to-log-in-the-machine.md | 86 +++++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 doc/manual/how-to-log-in-the-machine.md diff --git a/doc/manual/how-to-log-in-the-machine.md b/doc/manual/how-to-log-in-the-machine.md new file mode 100644 index 0000000..2b84b6d --- /dev/null +++ b/doc/manual/how-to-log-in-the-machine.md @@ -0,0 +1,86 @@ +这篇文档将告诉你如何登陆测试环境 + +# 1. 前提条件 +请先学习: +* [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…, 配置个人邮箱 +* [如何申请测试机.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua…, 并在本地生成RSA公私钥对 +* [submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma…, submit 命令详解 + +# 2. 操作方法 +## 2.1 在job.yaml里加上sshd和sleep字段, 以host-info.yaml任务为例： + +```shell + suite: host-info + category: functional + sshd: + pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + # sleep at the bottom + sleep: 1h + + host-info: +``` + +## 2.2 以下有2种方式可以登录到测试机： +### 第一种：使用submit -m -c的方式： + 这种方式提交的任务会在指定的位置sleep并直接登陆到测试机中，适用于登陆测试环境后手动调试 + 命令：submit -m -c host-info.yaml + 任务提交完成后，当测试执行到sshd后会自动登陆到测试机器上： + 效果如下： + +```shell + hi6325@account-vm ~% submit -m -c atomic.yaml + submit atomic.yaml, got job_id=crystal.146528 + query=>{"job_id":["crystal.146528"]} + connect to ws://localhost:11310/filter + {"job_id": "crystal.146528", "result_root": "/srv/result/atomic/2020-12-01/vm-2p8g/openeuler-20.03-aarch64/1-1000/crystal.146528", "job_state": "set result root" + {"job_id": "crystal.146528", "job_state": "boot"} + {"job_id": "crystal.146528", "job_state": "download"} + "time":"2020-12-01 10:12:33","mac":"0a-2d-7b-d9-f8-b1","ip":"172.18.252.12","job_id":"crystal.146528","state":"running","testbox":"vm-2p8g.zhyl-453231"} + {"job_state":"running","job_id":"crystal.146528"} + {"job_id": "crystal.146528", "state": "set ssh port", "ssh_port": "51750", "tbox_name": "vm-2p8g.zhyl-453231"} + + root@vm-2p8g ~# +``` + +### 第二种：根据邮件信息使用ssh方式登录测试机： + 这种是使用submit方式提交的任务完成后，系统自动发送一封邮件提醒您可以在指定时间内登陆到测试环境 + 命令： submit host-info.yaml + 任务执行完成后，系统发送邮件内容如下： + +```shell + Subject: [NOTIFY Compass-ci] vm-2p8g-294828 ready to use + + Dear $my_username: + Thanks for your participation in software ecosystem! + According to your application, vm-2p8g-294828 has been provisioned. + The datails are as follows: + + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch + Regards + Compass-Ci +``` + + 可通过ssh方式登陆到测试环境： + 命令： ssh root(a)api.compass-ci.openeuler.org -p $port + 效果如下： + +```shell + hi6325@account-vm ~% ssh root(a)api.compass-ci.openeuler.org -p 51400 + + root@vm-2p8g ~# +``` -- 2.23.0

1 0

Re: [PATCH compass-ci] lib/compare_matrixes.rb: fix spell error
by Lu Weitao 01 Dec '20

01 Dec '20

seems it's not spell error, just ":stddev" doesn't exists in metric_value lib/compare.rb have ":stddev", and ":stddev_percent", it will cause confusion we'd better merge them if it's possible in this patch Thanks, Weitao On Tue, Dec 01, 2020 at 03:23:53PM +0800, Lu Kaiyi wrote: >there should be stddev_percent > >Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> >--- > lib/compare_matrixes.rb | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > >diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb >index 7a9870a..2bb6a61 100644 >--- a/lib/compare_matrixes.rb >+++ b/lib/compare_matrixes.rb >@@ -460,7 +460,7 @@ def assign_metric_values(metrics_values, dim, metric, values) > metrics_values[metric]['standard_deviation'] ||= {} > metric_value = get_values(values, true) > metrics_values[metric]['average'][dim] = metric_value[:average] >- metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev] || 0 >+ metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev_percent] || 0 > end > > def assign_metric_change(metrics_values) >-- >2.23.0 >

1 1

[PATCH lkp-tests 1/2] tests: replace *.tar.* address with local address
by Liu Shaofei 01 Dec '20

01 Dec '20

When the address of source field is the format of "*.tar.*", we need to replace it with local address. The following example: source=("https://github.com/azukiapp/${pkgname}/archive/v${pkgver}.tar.gz") => source=("http://${LKP_SERVER}:8800/initrd/build-tar/${pkgname}/v${pkgver}.tar.gz") Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index bdaef035..731ac409 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -98,6 +98,20 @@ replace_source() sed -i "s|^source=[^)]*|$1|g" "$2" } +# replace *.tar.* address from source field in PKGBUILD +replace_tar_source() +{ + echo "source=(" > PKGBUILD.src + for url in ${source[@]} + do + echo "$url" | egrep '.+\.tar\..+|.+\.tar' && { + url="\"http://${LKP_SERVER}:8800/initrd/build-tar/${pkgname}/${url##*/}\"" + } + echo "$url" >> PKGBUILD.src + done + echo ")" >> PKGBUILD.src +} + build_source_pkg() { local repo_dir="" @@ -116,6 +130,8 @@ build_source_pkg() [ -n "$PKGBUILD_TAG" ] && replace_source "${upstream_source}" "PKGBUILD-$PKGBUILD_TAG" } + replace_tar_source + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 -- 2.23.0

2 2

[PATCH compass-ci] doc/manual: modify apply-account.md
by Li Ping 01 Dec '20

01 Dec '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- doc/manual/apply-account.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index 804b6ca..b1a2232 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -22,18 +22,18 @@ Email template: Which contains account information for you: - - my_email - - my_name - - my_uuid - - SCHED_HOST - - SCHED_PORT + my_email + my_name + my_uuid + SCHED_HOST + SCHED_PORT ## step 3: one-time setup -Follow the email to setup your local environment: +**Follow the email** to setup your local environment: - - clone git repos - - write ~/.config/compass-ci/default/account.yaml + git clone https://gitee.com/wu_fengguang/lkp-tests.git + vi ~/.config/compass-ci/defaults/account.yaml ## step 4: try it out -- 2.23.0

1 0

[PATCH compass-ci] lib/compare_matrixes.rb: fix spell error
by Lu Kaiyi 01 Dec '20

01 Dec '20

there should be stddev_percent Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_matrixes.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 7a9870a..2bb6a61 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -460,7 +460,7 @@ def assign_metric_values(metrics_values, dim, metric, values) metrics_values[metric]['standard_deviation'] ||= {} metric_value = get_values(values, true) metrics_values[metric]['average'][dim] = metric_value[:average] - metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev] || 0 + metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev_percent] || 0 end def assign_metric_change(metrics_values) -- 2.23.0

1 0

[PATCH compass-ci] scheduler: move 'auto_submit_idle_job' and add account info
by Ren Wen 01 Dec '20

01 Dec '20

Move `auto_submit_idle_job` from `sched.cr` to `find_job_boot.cr`, this is a step for finding job when there's no job in conventional queues. When submitting idle job, queue is: tbox_group/idle Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 14 -------------- src/scheduler/constants.cr | 2 ++ src/scheduler/find_job_boot.cr | 23 +++++++++++++++++++++++ 3 files changed, 25 insertions(+), 14 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 26ddd92..46ded9c 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -54,20 +54,6 @@ class Sched @redis.hash_del("sched/host2queues", hostname) end - def auto_submit_idle_job(tbox_group) - full_path_patterns = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/*.yaml" - extra_job_fields = [ - "idle_job=true", - "MASTER_FLUENTD_HOST=#{ENV["MASTER_FLUENTD_HOST"]}", - "MASTER_FLUENTD_PORT=#{ENV["MASTER_FLUENTD_PORT"]}", - ] - - Jobfile::Operate.auto_submit_job( - full_path_patterns, - "testbox: #{tbox_group}", - extra_job_fields) if Dir.glob(full_path_patterns).size > 0 - end - def update_tbox_wtmp testbox = "" hash = Hash(String, String).new diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 1b37333..5e6d0ab 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -29,3 +29,5 @@ OS_HTTP_PREFIX = "http://#{OS_HTTP_HOST}:#{OS_HTTP_PORT}" SCHED_HTTP_PREFIX = "http://#{SCHED_HOST}:#{SCHED_PORT}" DEMO_JOB = %({"suite":"pixz","testcase":"pixz","category":"benchmark","nr_threads":1,"pixz":null,"job_origin":"jobs/pixz.yaml","testbox":"wfg-e595","arch":"x86_64","tbox_group":"wfg-e595","id":"100","kmsg":null,"boot-time":null,"uptime":null,"iostat":null,"heartbeat":null,"vmstat":null,"numa-numastat":null,"numa-vmstat":null,"numa-meminfo":null,"proc-vmstat":null,"proc-stat":null,"meminfo":null,"slabinfo":null,"interrupts":null,"kconfig":"x86_64-rhel-7.6","compiler":"gcc-7"}) + +TEAM_ACCOUNT = "team(a)crystal.ci" diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 9805464..772e82f 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -217,4 +217,27 @@ class Sched return job end + + def auto_submit_idle_job(tbox_group) + account_info = @es.get_account(TEAM_ACCOUNT).as_h + raise "Failed to verify the account: #{TEAM_ACCOUNT}" unless account_info["found"].as_bool? + my_email = account_info["my_email"] + my_name = account_info["my_name"] + my_uuid = account_info["my_uuid"] + + full_path_patterns = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/*.yaml" + extra_job_fields = [ + "queue=#{tbox_group}/idle", + "my_email=#{my_email}", + "my_name=#{my_name}", + "my_uuid=#{my_uuid}", + "MASTER_FLUENTD_HOST=#{ENV["MASTER_FLUENTD_HOST"]}", + "MASTER_FLUENTD_PORT=#{ENV["MASTER_FLUENTD_PORT"]}", + ] + + Jobfile::Operate.auto_submit_job( + full_path_patterns, + "testbox: #{tbox_group}", + extra_job_fields) if Dir.glob(full_path_patterns).size > 0 + end end -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/manual: add 如何登陆测试机.md
by Zhang Yu 01 Dec '20

01 Dec '20

Add a document for introducing how to log in test machine Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- ...06\346\265\213\350\257\225\346\234\272.md" | 80 +++++++++++++++++++ 1 file changed, 80 insertions(+) create mode 100644 "doc/manual/\345\246\202\344\275\225\347\231\273\351\231\206\346\265\213\350\257\225\346\234\272.md" diff --git "a/doc/manual/\345\246\202\344\275\225\347\231\273\351\231\206\346\265\213\350\257\225\346\234\272.md" "b/doc/manual/\345\246\202\344\275\225\347\231\273\351\231\206\346\265\213\350\257\225\346\234\272.md" new file mode 100644 index 0000000..774488d --- /dev/null +++ "b/doc/manual/\345\246\202\344\275\225\347\231\273\351\231\206\346\265\213\350\257\225\346\234\272.md" @@ -0,0 +1,80 @@ +这篇文档将告诉你如何登陆测试环境 + +# 1 前提条件 +请先学习: +apply-account.md，配置个人邮箱： +参考文档：[apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md) +如何申请测试机，并在本地生成RSA公私钥对： +参考文档：[如何申请测试机.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/如何申请测试机.md) +submit 命令详解： +参考文档：[submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/submit命令详解.md) + +# 2. 操作方法 +## 2.1 在job.yaml里加上sshd和sleep字段, 以host-info.yaml任务为例： + suite: host-info + category: functional + + sshd: + pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + # sleep at the bottom + sleep: 1h + + host-info: + +## 2.2 提交任务的方式有2种，适用于不同的登陆需求： +### 第一种：使用submit -m -c的方式提交任务： + 这种方式提交的任务会在指定的位置sleep并直接登陆到测试机中，适用于登陆测试环境后手动调试 + 命令：submit -m -c host-info.yaml + 任务提交完成后，当测试执行到sshd后会自动登陆到测试机器上： + 效果如下： + hi6325@account-vm ~% submit -m -c atomic.yaml + submit atomic.yaml, got job_id=crystal.146528 + query=>{"job_id":["crystal.146528"]} + connect to ws://localhost:11310/filter + {"job_id": "crystal.146528", "result_root": "/srv/result/atomic/2020-12-01/vm-2p8g/openeuler-20.03-aarch64/1-1000/crystal.146528", "job_state": "set result root" + {"job_id": "crystal.146528", "job_state": "boot"} + {"job_id": "crystal.146528", "job_state": "download"} + {"time":"2020-12-01 10:12:33","mac":"0a-2d-7b-d9-f8-b1","ip":"172.18.252.12","job_id":"crystal.146528","state":"running","testbox":"vm-2p8g.zhyl-453231"} + {"job_state":"running","job_id":"crystal.146528"} + {"job_id": "crystal.146528", "state": "set ssh port", "ssh_port": "51750", "tbox_name": "vm-2p8g.zhyl-453231"} + + root@vm-2p8g ~# + +### 第二种：使用submit的方式提交任务： + 这种方式提交的任务会在测试执行完成后，系统自动发送一封邮件提醒您可以在指定时间内登陆到测试环境 + 命令： submit host-info.yaml + 任务执行完成后，系统发送邮件如下： + Subject: [NOTIFY Compass-ci] vm-2p8g-294828 ready to use + + Dear $my_username: + Thanks for your participation in software ecosystem! + According to your application, vm-2p8g-294828 has been provisioned. + The datails are as follows: + + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch + + Regards + Compass-Ci + + 可通过ssh方式登陆到测试环境： + 命令： ssh root(a)api.compass-ci.openeuler.org -p $port + 效果如下： + hi6325@account-vm ~% ssh root(a)api.compass-ci.openeuler.org -p 51400 + + + root@vm-2p8g ~# -- 2.23.0

2 2

[PATCH v3 compass-ci] doc/manual: add how-to-log-in-the-machine.md
by Zhang Yu 01 Dec '20

01 Dec '20

Add a document for introducing how to log in the machine Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/manual/how-to-log-in-the-machine.md | 89 +++++++++++++++++++++++++ 1 file changed, 89 insertions(+) create mode 100644 doc/manual/how-to-log-in-the-machine.md diff --git a/doc/manual/how-to-log-in-the-machine.md b/doc/manual/how-to-log-in-the-machine.md new file mode 100644 index 0000000..969257a --- /dev/null +++ b/doc/manual/how-to-log-in-the-machine.md @@ -0,0 +1,89 @@ +这篇文档将告诉你如何登陆测试环境 + +# 1. 前提条件 +请先学习: +apply-account.md, 配置个人邮箱： +参考文档：[apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md) +如何申请测试机，并在本地生成RSA公私钥对： +参考文档：[如何申请测试机.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/如何申请测试机.md) +submit 命令详解： +参考文档：[submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/submit命令详解.md) + +# 2. 操作方法 +## 2.1 在job.yaml里加上sshd和sleep字段, 以host-info.yaml任务为例： + + ```shell + suite: host-info + category: functional + sshd: + pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + # sleep at the bottom + sleep: 1h + + host-info: + ``` + +## 2.2 提交任务的方式有2种，适用于不同的登陆需求： +### 第一种：使用submit -m -c的方式提交任务： + 这种方式提交的任务会在指定的位置sleep并直接登陆到测试机中，适用于登陆测试环境后手动调试 + 命令：submit -m -c host-info.yaml + 任务提交完成后，当测试执行到sshd后会自动登陆到测试机器上： + 效果如下： + + ```shell + hi6325@account-vm ~% submit -m -c atomic.yaml + submit atomic.yaml, got job_id=crystal.146528 + query=>{"job_id":["crystal.146528"]} + connect to ws://localhost:11310/filter + {"job_id": "crystal.146528", "result_root": "/srv/result/atomic/2020-12-01/vm-2p8g/openeuler-20.03-aarch64/1-1000/crystal.146528", "job_state": "set result root" + {"job_id": "crystal.146528", "job_state": "boot"} + {"job_id": "crystal.146528", "job_state": "download"} + "time":"2020-12-01 10:12:33","mac":"0a-2d-7b-d9-f8-b1","ip":"172.18.252.12","job_id":"crystal.146528","state":"running","testbox":"vm-2p8g.zhyl-453231"} + {"job_state":"running","job_id":"crystal.146528"} + {"job_id": "crystal.146528", "state": "set ssh port", "ssh_port": "51750", "tbox_name": "vm-2p8g.zhyl-453231"} + + root@vm-2p8g ~# + ``` + +### 第二种：使用submit的方式提交任务： + 这种方式提交的任务会在测试执行完成后，系统自动发送一封邮件提醒您可以在指定时间内登陆到测试环境 + 命令： submit host-info.yaml + 任务执行完成后，系统发送邮件内容如下： + + ```shell + Subject: [NOTIFY Compass-ci] vm-2p8g-294828 ready to use + + Dear $my_username: + Thanks for your participation in software ecosystem! + According to your application, vm-2p8g-294828 has been provisioned. + The datails are as follows: + + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch + Regards + Compass-Ci + ``` + + 可通过ssh方式登陆到测试环境： + 命令： ssh root(a)api.compass-ci.openeuler.org -p $port + 效果如下： + + ```shell + hi6325@account-vm ~% ssh root(a)api.compass-ci.openeuler.org -p 51400 + + root@vm-2p8g ~# + ``` -- 2.23.0

1 0

[PATCH v1 compass-ci 2/4] container: build script for assistant container
by Cao Xueliang 01 Dec '20

01 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/assistant/build diff --git a/container/assistant/build b/container/assistant/build new file mode 100755 index 0000000..7778c9f --- /dev/null +++ b/container/assistant/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t debian:assistant . -- 2.23.0

1 0

[PATCH compass-ci] doc/manual: fix submit命令详解.md
by Wu Zhende 01 Dec '20

01 Dec '20

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- ...submit\345\221\275\344\273\244\350\257\246\350\247\243.md" | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git "a/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.md" "b/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.md" index 71d9dfd..0edb045 100644 --- "a/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.md" +++ "b/doc/manual/submit\345\221\275\344\273\244\350\257\246\350\247\243.md" @@ -36,7 +36,7 @@ submit iperf.yaml failed, got job_id=0, error: Missing required job key: 'testbo submit iperf.yaml testbox=vm-2p8g ``` -testbox 字段的值指定需要的测试机，可以使用 `ls` 命令查看 `lkp-tests/hosts` 路径下的可选测试机。如下图所示： +testbox 字段的值指定需要的测试机，可以使用 `ll` 命令查看 `lkp-tests/hosts` 路径下的可选测试机。如下所示： ```shell hi8109@account-vm ~/lkp-tests/hosts% ll @@ -83,7 +83,7 @@ total 120K ### 高级用法 -submit 命令的选项如下图所示： +submit 命令的选项如下所示： ```shell hi8109@account-vm ~% submit -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] kernel_params.cr: refactor if block to one line
by Xu Xijian 01 Dec '20

01 Dec '20

In crystal language, if the last line of a function has no value, this function will return nil, so the function kernel_console can be refactored into one line. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/kernel_params.cr | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index a558735..2cc8172 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -26,11 +26,7 @@ class Job end private def kernel_console - if os_arch == "x86_64" - return "console=ttyS0,115200 console=tty0" - else - return "" - end + return "console=ttyS0,115200 console=tty0" if os_arch == "x86_64" end private def set_kernel_params -- 2.23.0

1 1

[PATCH v2 compass-ci] lib: add error_messages.rb
by Lin Jiaxin 01 Dec '20

01 Dec '20

provide error messages for git bisect Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 62 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100644 lib/error_messages.rb diff --git a/lib/error_messages.rb b/lib/error_messages.rb new file mode 100644 index 0000000..dee0fdf --- /dev/null +++ b/lib/error_messages.rb @@ -0,0 +1,62 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' + +# get error messages hash +class ErrorMessages + def initialize(file) + @file = file + @seqno = '' + @in_stderr = false + @error_line = '' + @error_details = '' + @error_message = {} + @error_messages = Hash.new { |h, k| h[k] = Set.new } + end + + def extract_error_messages + File.open((a)file).each_line do |line| + next if extract_error_message(line) + next unless @in_stderr + next unless @error_message['error_line'].downcase =~ /error|warning/ + + @error_messages[@error_message['error_line']] << @error_message['error_line'] + @error_message['error_details'] + end + return @error_messages + end + + private + + def extract_error_message(line) + if line =~ /^ / + @error_details += line + else + update_error_message(line) + case line + when /^## ______________([0-9.]+):stderr$/ + @in_stderr = true + @seqno = $1 + return true + when /^## ______________#{@seqno}:enderr$/ + update_error_messages + end + end + end + + def update_error_message(line) + @error_message['error_line'] = @error_line + @error_message['error_details'] = @error_details + @error_line = line + @error_details = '' + end + + def update_error_messages + @in_stderr = false + @seqno = '' + return unless @error_message['error_line'].downcase =~ /error|warning/ + + @error_messages[@error_message['error_line']] << @error_message['error_line'] + @error_message['error_details'] + end +end -- 2.23.0

2 1

[PATCH compass-ci 1/3] container/defconfig.rb: limit container memory usage
by Liu Yinsi 01 Dec '20

01 Dec '20

use Math.sqrt(memory) to set container occupied memory according to system memory size, instead of setting fixed size, more reasonable. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index 05c56db..ef3ab96 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -35,3 +35,11 @@ def docker_rm(container) system "docker stop #{container} && docker rm -f #{container}" end + + +def set_available_memory + memory = `awk '($1 == "MemTotal:"){print $2/1048576}' /proc/meminfo`.to_i + + # set container available memory size, minimum size is 1g, maximum size is 30g, take the middle value according to the system memory size. + [1, 30, Math.sqrt(memory)].sort[1].to_i +end -- 2.23.0

2 2

[PATCH compass-ci] user-client/maintain: fix variable error
by Liu Yinsi 01 Dec '20

01 Dec '20

result_dict -> result_array Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- user-client/maintain/walk-os-test/walk-os-iperf-test | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index c4ffbd4..ed97999 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -70,7 +70,7 @@ def monitor_state(job_id) %w[boot extract_finished].each do |state| result_array = monitor({ 'job_id' => job_id, 'job_state' => state }, { 'stop' => true }, '1800').result - time_list.push(Time.parse(result_dict[0]['time'])) unless result_array.empty? + time_list.push(Time.parse(result_array[0]['time'])) unless result_array.empty? end time_list end -- 2.23.0

1 0

[PATCH compass-ci] doc/manual: translate file name
by Li Ping 01 Dec '20

01 Dec '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- doc/manual/how-to-apply-for-machine.md | 251 ++++++++++++++++++ doc/manual/how-to-compare-test-results.md | 49 ++++ ...to-use-cci-to-test-open-source-projects.md | 68 +++++ doc/manual/how-to-view-test-results.md | 24 ++ doc/manual/how-to-write-pkgbuild.md | 81 ++++++ doc/manual/install-cci-client-locally.md | 28 ++ doc/manual/submit-usage.md | 211 +++++++++++++++ 7 files changed, 712 insertions(+) create mode 100644 doc/manual/how-to-apply-for-machine.md create mode 100644 doc/manual/how-to-compare-test-results.md create mode 100644 doc/manual/how-to-use-cci-to-test-open-source-projects.md create mode 100644 doc/manual/how-to-view-test-results.md create mode 100644 doc/manual/how-to-write-pkgbuild.md create mode 100644 doc/manual/install-cci-client-locally.md create mode 100644 doc/manual/submit-usage.md diff --git a/doc/manual/how-to-apply-for-machine.md b/doc/manual/how-to-apply-for-machine.md new file mode 100644 index 0000000..d70136f --- /dev/null +++ b/doc/manual/how-to-apply-for-machine.md @@ -0,0 +1,251 @@ +# 前提条件 + +请确认您已按照 [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… +- send apply account email. +- receive email from compass-ci(a)qq.com. +- local environment configuration. + + +# 申请测试机（虚拟机） + +1. 生成本地RSA公私钥对 + + ```shell + hi684@account-vm ~% ssh-keygen -t rsa + Generating public/private rsa key pair. + Enter file in which to save the key (/home/hi684/.ssh/id_rsa): + Created directory '/home/hi684/.ssh'. + Enter passphrase (empty for no passphrase): + Enter same passphrase again: + Your identification has been saved in /home/hi684/.ssh/id_rsa. + Your public key has been saved in /home/hi684/.ssh/id_rsa.pub. + The key fingerprint is: + SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx hi684@account-vm + The key's randomart image is: + +---[RSA 2048]----+ + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + +----[SHA256]-----+ + hi684@account-vm ~% ls -hla .ssh + total 16K + drwx------. 2 hi684 hi684 4.0K Nov 26 16:37 . + drwx------. 7 hi684 hi684 4.0K Nov 26 16:37 .. + -rw-------. 1 hi684 hi684 1.8K Nov 26 16:37 id_rsa + -rw-r--r--. 1 hi684 hi684 398 Nov 26 16:37 id_rsa.pub + ``` + +2. 根据需求选择yaml + + 每位用户`/home/${USER}`目录下面都存放了一个 lkp-tests 的文件夹。 + + ```shell + hi684@account-vm ~% cd lkp-tests/jobs + hi684@account-vm ~/lkp-tests/jobs% ls -hl borrow-* + -rw-r--r--. 1 root root 53 Nov 2 14:54 borrow-10d.yaml + -rw-r--r--. 1 root root 64 Nov 2 14:54 borrow-1d.yaml + -rw-r--r--. 1 root root 235 Nov 19 15:27 borrow-1h.yaml + ``` + +3. 提交yaml并连接测试机（虚拟机） + + ```shell + hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml + 2020-11-26 14:44:25 +0800 WARN -- skip non-executable /home/hi684/lkp-tests/monitors/plain/wait + submit borrow-1h.yaml, got job_id=z9.170593 + query=>{"job_id":["z9.170593"]} + connect to ws://172.168.131.2:11310/filter + {"job_id":"z9.170593","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-26/vm-2p8g/openeuler-20.03-aarch64/3600/z9.170593"} + {"job_id": "z9.170593", "result_root": "/srv/result/borrow/2020-11-26/vm-2p8g/openeuler-20.03-aarch64/3600/z9.170593", "job_state": "set result root"} + {"job_id": "z9.170593", "job_state": "boot"} + {"job_id": "z9.170593", "job_state": "download"} + {"time":"2020-11-26 14:45:06","mac":"0a-1f-0d-3c-91-5c","ip":"172.18.156.13","job_id":"z9.170593","state":"running","testbox":"vm-2p8g.taishan200-2280-2s64p-256g--a38-12"} + {"job_state":"running","job_id":"z9.170593"} + {"job_id": "z9.170593", "state": "set ssh port", "ssh_port": "51840", "tbox_name": "vm-2p8g.taishan200-2280-2s64p-256g--a38-12"} + Host 172.168.131.2 not found in /home/hi684/.ssh/known_hosts + Warning: Permanently added '[172.168.131.2]:51840' (ECDSA) to the list of known hosts. + Last login: Wed Sep 23 11:10:58 2020 + + + Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 + + System information as of time: Thu Nov 26 06:44:18 CST 2020 + + System load: 0.83 + Processes: 107 + Memory used: 6.1% + Swap used: 0.0% + Usage On: 89% + IP address: 172.18.156.13 + Users online: 1 + + + + root@vm-2p8g ~# + ``` + + 更多关于`submit命令如何使用`、`testbox都有什么可选项`、`如何borrow指定的操作系统`，请参见文章末尾FAQ。 + +4. 使用完毕退还测试机（虚拟机） + + ```shell + root@vm-2p8g ~# reboot + Connection to 172.168.131.2 closed by remote host. + Connection to 172.168.131.2 closed. + hi684@account-vm ~/lkp-tests/jobs% + ``` + + +# 申请测试机（物理机） + + +1. 生成本地RSA公私钥对 + + ```shell + hi684@account-vm ~% ssh-keygen -t rsa + Generating public/private rsa key pair. + Enter file in which to save the key (/home/hi684/.ssh/id_rsa): + Created directory '/home/hi684/.ssh'. + Enter passphrase (empty for no passphrase): + Enter same passphrase again: + Your identification has been saved in /home/hi684/.ssh/id_rsa. + Your public key has been saved in /home/hi684/.ssh/id_rsa.pub. + The key fingerprint is: + SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx hi684@account-vm + The key's randomart image is: + +---[RSA 2048]----+ + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + +----[SHA256]-----+ + hi684@account-vm ~% ls -hla .ssh + total 16K + drwx------. 2 hi684 hi684 4.0K Nov 26 16:37 . + drwx------. 7 hi684 hi684 4.0K Nov 26 16:37 .. + -rw-------. 1 hi684 hi684 1.8K Nov 26 16:37 id_rsa + -rw-r--r--. 1 hi684 hi684 398 Nov 26 16:37 id_rsa.pub + ``` + +2. 根据需求选择yaml + + 每位用户`/home/${USER}`目录下面都存放了一个 lkp-tests 的文件夹。 + + ```shell + hi684@account-vm ~% cd lkp-tests/jobs + hi684@account-vm ~/lkp-tests/jobs% ls -hl borrow-* + -rw-r--r--. 1 root root 53 Nov 2 14:54 borrow-10d.yaml + -rw-r--r--. 1 root root 64 Nov 2 14:54 borrow-1d.yaml + -rw-r--r--. 1 root root 235 Nov 19 15:27 borrow-1h.yaml + ``` + +3. 提交yaml并连接测试机（物理机） + + ```shell + hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=taishan200-2280-2s64p-256g borrow-1h.yaml + 2020-11-26 14:49:13 +0800 WARN -- skip non-executable /home/hi684/lkp-tests/monitors/plain/wait + submit borrow-1h.yaml, got job_id=z9.170594 + query=>{"job_id":["z9.170594"]} + connect to ws://172.168.131.2:11310/filter + {"job_id":"z9.170594","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-26/taishan200-2280-2s64p-256g/openeuler-20.03-aarch64/3600/z9.170594"} + {"job_id": "z9.170594", "result_root": "/srv/result/borrow/2020-11-26/taishan200-2280-2s64p-256g/openeuler-20.03-aarch64/3600/z9.170594", "job_state": "set result root"} + {"job_id": "z9.170594", "job_state": "boot"} + {"job_id": "z9.170594", "job_state": "download"} + {"time":"2020-11-26 14:51:56","mac":"84-46-fe-26-d3-47","ip":"172.168.178.48","job_id":"z9.170594","state":"running","testbox":"taishan200-2280-2s64p-256g--a5"} + {"job_state":"running","job_id":"z9.170594"} + {"job_id": "z9.170594", "state": "set ssh port", "ssh_port": "50420", "tbox_name": "taishan200-2280-2s64p-256g--a5"} + Host 172.168.131.2 not found in /home/hi684/.ssh/known_hosts + Warning: Permanently added '[172.168.131.2]:50420' (ECDSA) to the list of known hosts. + Last login: Wed Sep 23 11:10:58 2020 + + + Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 + + System information as of time: Thu Nov 26 14:51:59 CST 2020 + + System load: 1.31 + Processes: 1020 + Memory used: 5.1% + Swap used: 0.0% + Usage On: 3% + IP address: 172.168.178.48 + Users online: 1 + + + + root@taishan200-2280-2s64p-256g--a5 ~# + ``` + + 更多关于`submit命令如何使用`、`testbox都有什么可选项`、`如何borrow指定的操作系统`，请参见文章末尾FAQ。 + +4. 使用完毕退还测试机（物理机） + + ```shell + root@taishan200-2280-2s64p-256g--a5 ~# reboot + Connection to 172.168.131.2 closed by remote host. + Connection to 172.168.131.2 closed. + hi684@account-vm ~/lkp-tests/jobs% + ``` + + +# FAQ + + +* 如何自行修改申请时长 + + ```shell + hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml + suite: borrow + testcase: borrow + + sshd: + pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + # sleep at the bottom + sleep: 1h + hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml + sleep: 1h + # 使用vim来修改你的sleep字段的值 + hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml + # 修改完毕后重新submit即可 + hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml + ``` + +* Submit命令指导 + + 参考文档：[submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/… + +* testbox有什么可选项 + + testbox可选项请参考：https://gitee.com/wu_fengguang/lab-z9/tree/master/hosts + + >![](./../public_sys-resources/icon-note.gif) **说明：** + > + > 虚拟机的testbox : vm-xxx + > 物理机的testbox : taishan200-2280-xxx + + + + >![](./../public_sys-resources/icon-notice.gif) **注意：** + > - 物理机的testbox若选择以`--axx`结尾的，则表示指定到了具体的某一个物理机。若此物理机任务队列中已经有任务在排队，则需要等待队列中前面的任务执行完毕后，才会轮到你提交的borrow任务。 + > - 物理机的testbox若不选择以`-axx`结尾的，表示不指定具体的某一个物理机。则此时集群中的空闲物理机会即时被分配执行你的borrow任务。 + +* 如何 borrow 指定的操作系统 + + 关于支持的`os`, `os_arch`, `os_version`，参见：[os-os_verison-os_arch.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/job/os-os_verison-os_arch.md) diff --git a/doc/manual/how-to-compare-test-results.md b/doc/manual/how-to-compare-test-results.md new file mode 100644 index 0000000..a7348d8 --- /dev/null +++ b/doc/manual/how-to-compare-test-results.md @@ -0,0 +1,49 @@ +# Introduction of compare feature + The compare feature used to analyze the results of different jobs, showing the performance waves and changes + under different influencing factors, for users to analyze performance influencing factor. + +## url: https://compass-ci.openeuler.org/compare + +## options directions +### filters: + - suite: iperf, netperf, mysql, ... + - OS: openeuler 20.03, centos 7.6, ... + - os_arch: aarch64, x86 + - tbox_group: vm-2p8g, taishan200-2880-2s48p-256g, ... + we can combine the above options arbitrarily to limit compare scope. + choose at least one option as filter. + +### dimension + Dimension can select: os, os_version, os_arch, suite, tbox_group. + Within filter, we will compare all different job result by dimension + and keep other test conditions are same. + +## example: + filter: suite = iperf + dimension: os_version + + result: + os=openeuler/os_arch=aarch64/pp.iperf.protocol=tcp/pp.iperf.runtime=20/tbox_group=vm-2p8g # other test conditions keep same + + + 20.09 20.03 metric + -------------------- ------------------------------ ------------------------------ + fails:runs change fails:runs + | | | + 3:3 -100.0% 0:3 last_state.exit_fail + 3:3 -100.0% 0:3 last_state.is_incomplete_run + 3:3 -100.0% 0:3 last_state.test.iperf.exit_code.127 + 3:3 -100.0% 0:3 stderr.perf_command_failed + + + + 20.09 20.03 metric + -------------------- ------------------------------ ------------------------------ + %stddev change %stddev + \ | \ + 4.461021e+10 ± 6% -17.4% 3.686392e+10 ± 12% iperf.tcp.receiver.bps + 4.461112e+10 ± 6% -17.4% 3.686935e+10 ± 12% iperf.tcp.sender.bps + 94.82 -44.0% 53.10 boot-time.boot + 123.11 -58.4% 51.19 boot-time.idle + 0.00 0 4.87 boot-time.kernel_boot + 4165.50 ± 12% -99.9% 5.00 interrupts.38:GICv3.36.Level.virtio0 diff --git a/doc/manual/how-to-use-cci-to-test-open-source-projects.md b/doc/manual/how-to-use-cci-to-test-open-source-projects.md new file mode 100644 index 0000000..e73b9f9 --- /dev/null +++ b/doc/manual/how-to-use-cci-to-test-open-source-projects.md @@ -0,0 +1,68 @@ +# 使用 compass-ci 平台测试开源项目 + +本文介绍了使用 compass-ci 平台测试开源项目的操作。 + +### 添加待测试仓库 URL 到 upstream-repos 仓库 + +执行以下步骤，将想要测试的代码仓信息以 yaml 文件的方式添加到 upstream-repos 仓库（https://gitee.com/wu_fengguang/upstream-repos ）。 + +1. Fork 要测试的代码仓库并 git clone 到本地，本文以 blacklight 仓库（https://github.com/baskerville/backlight ）为例说明。 + +![](./../pictures/fork_blacklight.png) + +2. 执行以下命令，以首字母和仓库名创建文件路径。 + + ``` + mkdir -p b/backlight + ``` + +3. 执行以下命令，在该目录下新建同名文件 backlight。 + ``` + cd b/backlight + touch backlight + ``` + +4. 执行以下命令，将 backlight 仓库 url 信息写入 backlight 文件。 + + ``` + vim backlight + ``` + 内容格式为 + + ``` + --- + url: + - https://github.com/baskerville/backlight + ``` + + >![](./../public_sys-resources/icon-notice.gif) **注意：** + > + >可参考 upstream-repos 仓库中已有文件格式,请保持格式一致。 + +5. 通过 Pull Request 命令将新增的 backlight 文件提交到 blacklight 仓库。 + + +### 提交测试任务到 compass-ci 平台 + +1. 准备测试用例 + + 测试用例可以自己编写并添加到 lkp-tests 仓库，也可以直接使用 lkp-tests 仓库（https://gitee.com/wu_fengguang/lkp-tests ）的 jobs 目录下已有的测试用例。 + + * 使用仓库中已经适配好的测试用例 + 如果 lkp-tests 仓库中正好有你想要的测试用例，你可以直接使用。以 iperf.yaml 文件为例说明如下： + iperf.yaml 是一个已经适配好的测试用例，它位于 lkp-tests 仓库的 jobs 目录下，其中有一些基本的测试参数。 + + * 编写测试用例并添加到仓库 + + 请参考：[如何添加测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… ) + +2. 配置 auto_submit.yaml 文件，提交测试任务 + + 你只需要在 compass-ci 仓库下面的 sbin/auto_submit.yaml 文件中添加配置信息，如： + ``` + b/backlight/backlight: + - testbox=vm-2p8g os=openEuler os_version=20.03 os_mount=initramfs os_arch=aarch64 iperf.yaml + ``` + 通过 Pull Request 的方式将修改好的 auto_submit.yaml 文件提交到 compass-ci 仓库，就可以使用 compass-ci 测试你的项目了。 + + auto_submit.yaml 文件的参数配置请参考 https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job 。 diff --git a/doc/manual/how-to-view-test-results.md b/doc/manual/how-to-view-test-results.md new file mode 100644 index 0000000..36caaeb --- /dev/null +++ b/doc/manual/how-to-view-test-results.md @@ -0,0 +1,24 @@ +# 如何查看任务结果 + +测试用例执行完成后，可以登录到 https://compass-ci.openeuler.org/jobs 查看执行结果。 + +可以根据提交时定义的 suite 或者提交任务返回的 id 在网站上找到自己的任务，点击任务对应的 job_state 列即可查看任务结果。 + +![](./../pictures/jobs.png) + +## 结果文件介绍 + +**job.yaml文件** + + +job.yaml 文件中部分字段是用户提交上来的，其他字段是平台根据提交的 job 自动添加进来的。此文件包含了测试任务需要的所有参数。 + +**output文件** + +output 文件记录了用例的执行过程，文件最后部分一般会有 check_exit_code 这个状态码，非 0 代表测试用例错误。 + +**stats.json** + +测试用例执行完成会生成一个与测试用例同名的文件，记录它们的测试命令及标准化输出结果。compass-ci 会对这些文件进行解析，生成后缀名是 .json 的文件。 + +stats.json 是所有的 json 文件的汇总，所有测试命令的关键结果都会统计到这个文件中，便于后续的比较和分析。 diff --git a/doc/manual/how-to-write-pkgbuild.md b/doc/manual/how-to-write-pkgbuild.md new file mode 100644 index 0000000..66a1833 --- /dev/null +++ b/doc/manual/how-to-write-pkgbuild.md @@ -0,0 +1,81 @@ +# PKGBUILD是什么 + +PKGBUILD 是一个 shell 脚本，makepkg 通过 PKGBUILD 中包含的指令，生成包含二进制文件和安装指令的软件包。 + +# PKGBUILD包含什么 + +PKGBUILD包含两部分内容：变量和函数。 + +## 定义变量 + +- pkgname: 必须定义，表示软件包的名称; +- pkgver: 必须定义，表示软件包的版本号； +- pkgrel: 必须定义，表示软件包的发布号； +- arch: 必须定义，表示软件包使用的架构序列； +- depends: 可选字段，软件测试运行时需要的依赖包名称； +- makedepends: 可选字段，构建软件包时需要的文件列表； +- source: 可选字段，指定每个源文件的MD5哈希值，用于构建过程中验证源文件的完整性。 + +## 定义函数 + +- package函数 + package函数必须定义，此函数用于将文件安装到将成为构建包的根目录的目录中； +- prepare函数 + 定义可选的prepare函数，在其中执行用于准备构建源码的操作； +- build函数 + 定义可选的build函数，用于编译和/或构建源码； +- check函数 + 定义可选的check函数，用于运行程序包的测试套件。 + +>![](./../public_sys-resources/icon-notice.gif) **注意：** +> +> srcdir 是提取或复制源文件的目录，所有打包功能都在 srcdir 目录内部运行；pkgdir 是构建软件包的根目录，仅在 package 函数中使用。 + + +# 如何编写PKGBUILD + + +1. 执行如下命令创建PKGBUILD，文件名必须以“PKGBUILD”命名。 + + ```shell + touch PKGBUILD + ``` + + +2. 使用 vim 打开 PKGBUILD，编写 PKGBUILD 文件内容，一个 PKGBUILD 示例如下： + + ```shell + pkgname=zstd + pkgver=1.4.4 + pkgrel=2 + arch=('i686' 'x86_64' 'aarch64') + url='https://github.com/facebook/zstd' + license=('custom:BSD3' 'GPL2') + depends=('xz' 'zlib' 'lz4') + makedepends=('git') + source=('git://github.com/facebook/zstd.git#branch=dev') + md5sums=('SKIP') + + pkgver() { + cd "$srcdir/$pkgname" + git describe --long --tags | sed 's/$[^-]*-g$/r\1/;s/-/./g;s/^v//g' + } + + build() { + cd "$srcdir/$pkgname" + make + make -C contrib/pzstd + } + + package() { + cd "$srcdir/$pkgname" + make PREFIX="/usr" DESTDIR="$pkgdir/" install + install -D -m755 contrib/pzstd/pzstd "$pkgdir/usr/bin/pzstd" + install -D -m644 LICENSE "${pkgdir}/usr/share/licenses/${pkgname}/LICENSE" + } + ``` + +# 参考 + +- [PKGBUILD(5) Manual Page](https://www.archlinux.org/pacman/PKGBUILD.5.html) +- [pkgbuild demo文件](https://git.archlinux.org/pacman.git/plain/proto/PKGBUILD.proto) diff --git a/doc/manual/install-cci-client-locally.md b/doc/manual/install-cci-client-locally.md new file mode 100644 index 0000000..43a6bf5 --- /dev/null +++ b/doc/manual/install-cci-client-locally.md @@ -0,0 +1,28 @@ +# 本地安装compass-ci客户端 + +Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客户端，通过本地安装 lkp-tests 可以手动提交测试任务 + +:exclamation: 前提条件：按照 [apply-ssh-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master… 完成如下动作： +- send apply account email +- receive email from compass-ci(a)qq.com + +## Getting started + +1. setup default config + run the following command to add the below setup to default config file + mkdir -p ~/.config/compass-ci/defaults/ + cat >> ~/.config/compass-ci/defaults/${USER}.yaml <<-EOF + my_uuid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx + my_email: xxx + my_name: xx + EOF + +2. download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source ~/.bashrc && source ~/.bash_profile + +3. submit job + Now try [submitting a job to compass-ci](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… diff --git a/doc/manual/submit-usage.md b/doc/manual/submit-usage.md new file mode 100644 index 0000000..71d9dfd --- /dev/null +++ b/doc/manual/submit-usage.md @@ -0,0 +1,211 @@ +# submit 命令详解 + +### 前提条件 + +已经按照[本地安装compass-ci客户端.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/本地安装compass-ci客户端.md )完成安装 + +### 概述 + +submit 命令的作用是提交测试任务。该命令提供了多个选项帮助用户更灵活的提交任务，您可以在命令行直接输入 summit 命令来查看帮助信息，并根据实际需求灵活使用。 + + +### 基本用法 + +测试任务以 yaml 文件的方式提交，因此，您需要事先准备好测试任务的 yaml 文件（本文以 iperf.yaml 为例）。使用如下命令提交测试任务： + +``` +submit iperf.yaml +``` +如下所示： + +```shell +hi8109@account-vm ~% submit iperf.yaml +submit iperf.yaml, got job_id=z9.173924 +``` + +本文所示的 yaml 文件均已添加 testbox 字段，若您提交的 yaml 文件不包含该字段将报错： + +```shell +hi8109@account-vm ~% submit iperf.yaml +submit iperf.yaml failed, got job_id=0, error: Missing required job key: 'testbox' +``` + +您可以在 yaml 文件中添加 testbox 字段，或使用如下命令： + +``` +submit iperf.yaml testbox=vm-2p8g +``` + +testbox 字段的值指定需要的测试机，可以使用 `ls` 命令查看 `lkp-tests/hosts` 路径下的可选测试机。如下图所示： + +```shell +hi8109@account-vm ~/lkp-tests/hosts% ll +total 120K +-rw-r--r--. 1 root root 76 2020-11-02 14:54 vm-snb +-rw-r--r--. 1 root root 64 2020-11-02 14:54 vm-pxe-hi1620-2p8g +-rw-r--r--. 1 root root 64 2020-11-02 14:54 vm-pxe-hi1620-2p4g +-rw-r--r--. 1 root root 64 2020-11-02 14:54 vm-pxe-hi1620-2p1g +-rw-r--r--. 1 root root 64 2020-11-02 14:54 vm-pxe-hi1620-1p1g +-rw-r--r--. 1 root root 75 2020-11-02 14:54 vm-hi1620-2p8g +-rw-r--r--. 1 root root 75 2020-11-02 14:54 vm-hi1620-2p4g +-rw-r--r--. 1 root root 75 2020-11-02 14:54 vm-hi1620-2p1g +-rw-r--r--. 1 root root 75 2020-11-02 14:54 vm-hi1620-1p1g +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p8g-pxe +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p8g +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p4g-pxe +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p4g +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p1g-pxe +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p1g +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-1p1g-pxe +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-1p1g +-rw-r--r--. 1 root root 14 2020-11-02 14:54 taishan200-2280-2s64p-256g +-rw-r--r--. 1 root root 497 2020-11-02 14:54 lkp-skl-d01 +-rw-r--r--. 1 root root 574 2020-11-02 14:54 lkp-ivb-d04 +-rw-r--r--. 1 root root 606 2020-11-02 14:54 lkp-ivb-d02 +-rw-r--r--. 1 root root 384 2020-11-02 14:54 lkp-ivb-d01 +-rw-r--r--. 1 root root 388 2020-11-02 14:54 lkp-hsw-d01 +-rw-r--r--. 1 root root 385 2020-11-02 14:54 lkp-bdw-de1 +-rw-r--r--. 1 root root 11 2020-11-02 14:54 dc-8g +-rw-r--r--. 1 root root 11 2020-11-02 14:54 dc-4g +-rw-r--r--. 1 root root 11 2020-11-02 14:54 dc-2g +-rw-r--r--. 1 root root 11 2020-11-02 14:54 dc-1g +-rw-r--r--. 1 root root 13 2020-11-02 14:54 2288hv5-2s64p +-rw-r--r--. 1 root root 74 2020-11-02 14:54 vm-snb-i386 +``` + +>![](./../public_sys-resources/icon-note.gif) **说明：** +> +>使用 "=" 更新 yaml 中的字段，"=" 在命令行中的位置不同优先级不同：. +> * submit iperf.yaml testbox=vm-2p8g 命令中 "=" 定义在 yaml 文件之后，则 "=" 的优先级高于 yaml 文件,testbox=vm-2p8g 会覆盖 yaml 文件中已经定义的 testbox 字段。 +> * submit testbox=vm-2p8g iperf.yaml 命令中 "=" 定义在 yaml 文件之前，则 "=" 的优先级低于 yaml 文件,testbox=vm-2p8g 不会覆盖 yaml 文件中已经定义的 testbox 字段，只有当 yaml 文件中不存在 testbox 字段才会赋值。 + + + +### 高级用法 + +submit 命令的选项如下图所示： + +```shell +hi8109@account-vm ~% submit +Usage: submit [options] job1.yaml job2.yaml ... + submit test jobs to the scheduler + +options: + -s, --set 'KEY: VALUE' add YAML hash to job + -o, --output DIR save job yaml to DIR/ + -a, --auto-define-files auto add define_files + -c, --connect auto connect to the host + -m, --monitor monitor job status: use -m 'KEY: VALUE' to add rule + --my-queue add to my queue +``` + +* **-s的用法** + + 使用 -s 'KEY:VALUE' 参数可以将键值对更新到提交的任务当中。示例如下所示： + ``` + submit -s 'testbox: vm-2p8g' iperf.yaml + ``` + + + * 如果 iperf.yaml 中不存在 testbox：vm-2p8g ，最终提交的任务将会加上该信息。 + * 如果 iperf.yaml 中存在 testbox 字段，但是值不为 vm-2p8g ，最终提交的任务中 testbox 的值将会被替换为vm-2p8g。 + +* **-o的用法** + + 使用-o DIR 命令可以将最终生成的yaml文件保存到指定目录 DIR 下。示例如下所示： + + ``` + submit -o ~/iperf.yaml + ``` + + 运行命令之后会在指定目录生成经过 submit 处理过的 yaml 文件。 + +* **-a的用法** + + 如果你的测试用例对客户端的lkp-tests 做了更改，需要使用 -a 选项来适配。将客户端的 lkp-tests 下做的更改，同步到服务端，并在测试机上生成你的测试脚本。 + 示例命令如下： + + ``` + submit -a iperf.yaml + ``` + +* **-m的用法** + + 使用 -m 参数可以启动任务监控功能，并将任务执行过程中的各种状态信息打印到控制台上，方便用户实时监控测试任务的执行过程。 + 示例命令如下： + + ``` + submit -m iperf.yaml + ``` + + 控制台显示如下： + + ```shell + hi8109@account-vm ~% submit -m iperf.yaml + submit iperf.yaml, got job_id=z9.173923 + query=>{"job_id":["z9.173923"]} + connect to ws://172.168.131.2:11310/filter + {"job_id":"z9.173923","message":"","job_state":"submit","result_root":"/srv/result/iperf/2020-11-30/vm-2p8g/openeuler-20.03-aarch6 + {"job_id": "z9.173923", "result_root": "/srv/result/iperf/2020-11-30/vm-2p8g/openeuler-20.03-aarch64/tcp-30/z9.173923", "job_state + {"job_id": "z9.173923", "job_state": "boot"} + {"job_id": "z9.173923", "job_state": "download"} + {"time":"2020-11-30 20:28:16","mac":"0a-f5-9f-83-62-ea","ip":"172.18.192.21","job_id":"z9.173923","state":"running","testbox":"vm- + {"job_state":"running","job_id":"z9.173923"} + {"job_state":"post_run","job_id":"z9.173923"} + {"start_time":"2020-11-30 12:25:15","end_time":"2020-11-30 12:25:45","loadavg":"1.12 0.38 0.14 1/105 1956","job_id":"z9.173923"} + {"job_state":"finished","job_id":"z9.173923"} + {"job_id": "z9.173923", "job_state": "complete"} + {"time":"2020-11-30 20:28:54","mac":"0a-f5-9f-83-62-ea","ip":"172.18.192.21","job_id":"z9.173923","state":"rebooting","testbox":"v + {"job_id": "z9.173923", "job_state": "extract_finished"} + connection closed: normal + ``` + +* **-c的用法** + + -c 参数需要搭配 -m 参数来使用，可以使申请设备的任务实现自动登入功能。 + + 示例命令如下： + + ``` + submit -m -c borrow-1h.yaml + ``` + 当我们提交一个申请设备的任务后，会获取到返回的登陆信息，如 `ssh ip -p port`，添加 -c 参数之后不需要我们手动输入 ssh 登陆命令来进入执行机。 + + 控制台显示如下： + + ```shell + hi8109@account-vm ~% submit -m -c borrow-1h.yaml + submit borrow-1h.yaml, got job_id=z9.173925 + query=>{"job_id":["z9.173925"]} + connect to ws://172.168.131.2:11310/filter + {"job_id":"z9.173925","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-30/vm-2p8g/openeuler-20.03-aarch + {"job_id": "z9.173925", "result_root": "/srv/result/borrow/2020-11-30/vm-2p8g/openeuler-20.03-aarch64/3600/z9.173925", "job_state" + {"job_id": "z9.173925", "job_state": "boot"} + {"job_id": "z9.173925", "job_state": "download"} + {"time":"2020-11-30 20:35:04","mac":"0a-24-5d-c8-aa-d0","ip":"172.18.101.4","job_id":"z9.173925","state":"running","testbox":"vm-2 + {"job_state":"running","job_id":"z9.173925"} + {"job_id": "z9.173925", "state": "set ssh port", "ssh_port": "50200", "tbox_name": "vm-2p8g.taishan200-2280-2s48p-256g--a52-7"} + Host 172.168.131.2 not found in /home/hi8109/.ssh/known_hosts + Warning: Permanently added '[172.168.131.2]:50200' (ECDSA) to the list of known hosts. + Last login: Wed Sep 23 11:10:58 2020 + + + Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 + + System information as of time: Mon Nov 30 12:32:04 CST 2020 + + System load: 0.50 + Processes: 105 + Memory used: 6.1% + Swap used: 0.0% + Usage On: 89% + IP address: 172.17.0.1 + Users online: 1 + + + + root@vm-2p8g ~# + ``` + + 已经成功登陆执行机。 + -- 2.23.0

2 1

[PATCH compass-ci] doc/manual: translation file name
by Li Ping 01 Dec '20

01 Dec '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- doc/manual/how-to-apply-for-machine.md | 251 ++++++++++++++++++ doc/manual/how-to-compare-test-results.md | 49 ++++ ...to-use-cci-to-test-open-source-projects.md | 68 +++++ doc/manual/how-to-view-test-results.md | 24 ++ doc/manual/how-to-write-pkgbuild.md | 81 ++++++ doc/manual/install-cci-client-locally.md | 28 ++ doc/manual/submit-usage.md | 211 +++++++++++++++ 7 files changed, 712 insertions(+) create mode 100644 doc/manual/how-to-apply-for-machine.md create mode 100644 doc/manual/how-to-compare-test-results.md create mode 100644 doc/manual/how-to-use-cci-to-test-open-source-projects.md create mode 100644 doc/manual/how-to-view-test-results.md create mode 100644 doc/manual/how-to-write-pkgbuild.md create mode 100644 doc/manual/install-cci-client-locally.md create mode 100644 doc/manual/submit-usage.md diff --git a/doc/manual/how-to-apply-for-machine.md b/doc/manual/how-to-apply-for-machine.md new file mode 100644 index 0000000..d70136f --- /dev/null +++ b/doc/manual/how-to-apply-for-machine.md @@ -0,0 +1,251 @@ +# 前提条件 + +请确认您已按照 [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… +- send apply account email. +- receive email from compass-ci(a)qq.com. +- local environment configuration. + + +# 申请测试机（虚拟机） + +1. 生成本地RSA公私钥对 + + ```shell + hi684@account-vm ~% ssh-keygen -t rsa + Generating public/private rsa key pair. + Enter file in which to save the key (/home/hi684/.ssh/id_rsa): + Created directory '/home/hi684/.ssh'. + Enter passphrase (empty for no passphrase): + Enter same passphrase again: + Your identification has been saved in /home/hi684/.ssh/id_rsa. + Your public key has been saved in /home/hi684/.ssh/id_rsa.pub. + The key fingerprint is: + SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx hi684@account-vm + The key's randomart image is: + +---[RSA 2048]----+ + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + +----[SHA256]-----+ + hi684@account-vm ~% ls -hla .ssh + total 16K + drwx------. 2 hi684 hi684 4.0K Nov 26 16:37 . + drwx------. 7 hi684 hi684 4.0K Nov 26 16:37 .. + -rw-------. 1 hi684 hi684 1.8K Nov 26 16:37 id_rsa + -rw-r--r--. 1 hi684 hi684 398 Nov 26 16:37 id_rsa.pub + ``` + +2. 根据需求选择yaml + + 每位用户`/home/${USER}`目录下面都存放了一个 lkp-tests 的文件夹。 + + ```shell + hi684@account-vm ~% cd lkp-tests/jobs + hi684@account-vm ~/lkp-tests/jobs% ls -hl borrow-* + -rw-r--r--. 1 root root 53 Nov 2 14:54 borrow-10d.yaml + -rw-r--r--. 1 root root 64 Nov 2 14:54 borrow-1d.yaml + -rw-r--r--. 1 root root 235 Nov 19 15:27 borrow-1h.yaml + ``` + +3. 提交yaml并连接测试机（虚拟机） + + ```shell + hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml + 2020-11-26 14:44:25 +0800 WARN -- skip non-executable /home/hi684/lkp-tests/monitors/plain/wait + submit borrow-1h.yaml, got job_id=z9.170593 + query=>{"job_id":["z9.170593"]} + connect to ws://172.168.131.2:11310/filter + {"job_id":"z9.170593","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-26/vm-2p8g/openeuler-20.03-aarch64/3600/z9.170593"} + {"job_id": "z9.170593", "result_root": "/srv/result/borrow/2020-11-26/vm-2p8g/openeuler-20.03-aarch64/3600/z9.170593", "job_state": "set result root"} + {"job_id": "z9.170593", "job_state": "boot"} + {"job_id": "z9.170593", "job_state": "download"} + {"time":"2020-11-26 14:45:06","mac":"0a-1f-0d-3c-91-5c","ip":"172.18.156.13","job_id":"z9.170593","state":"running","testbox":"vm-2p8g.taishan200-2280-2s64p-256g--a38-12"} + {"job_state":"running","job_id":"z9.170593"} + {"job_id": "z9.170593", "state": "set ssh port", "ssh_port": "51840", "tbox_name": "vm-2p8g.taishan200-2280-2s64p-256g--a38-12"} + Host 172.168.131.2 not found in /home/hi684/.ssh/known_hosts + Warning: Permanently added '[172.168.131.2]:51840' (ECDSA) to the list of known hosts. + Last login: Wed Sep 23 11:10:58 2020 + + + Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 + + System information as of time: Thu Nov 26 06:44:18 CST 2020 + + System load: 0.83 + Processes: 107 + Memory used: 6.1% + Swap used: 0.0% + Usage On: 89% + IP address: 172.18.156.13 + Users online: 1 + + + + root@vm-2p8g ~# + ``` + + 更多关于`submit命令如何使用`、`testbox都有什么可选项`、`如何borrow指定的操作系统`，请参见文章末尾FAQ。 + +4. 使用完毕退还测试机（虚拟机） + + ```shell + root@vm-2p8g ~# reboot + Connection to 172.168.131.2 closed by remote host. + Connection to 172.168.131.2 closed. + hi684@account-vm ~/lkp-tests/jobs% + ``` + + +# 申请测试机（物理机） + + +1. 生成本地RSA公私钥对 + + ```shell + hi684@account-vm ~% ssh-keygen -t rsa + Generating public/private rsa key pair. + Enter file in which to save the key (/home/hi684/.ssh/id_rsa): + Created directory '/home/hi684/.ssh'. + Enter passphrase (empty for no passphrase): + Enter same passphrase again: + Your identification has been saved in /home/hi684/.ssh/id_rsa. + Your public key has been saved in /home/hi684/.ssh/id_rsa.pub. + The key fingerprint is: + SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx hi684@account-vm + The key's randomart image is: + +---[RSA 2048]----+ + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + |xxxxxxxxxxxxxxxxx| + +----[SHA256]-----+ + hi684@account-vm ~% ls -hla .ssh + total 16K + drwx------. 2 hi684 hi684 4.0K Nov 26 16:37 . + drwx------. 7 hi684 hi684 4.0K Nov 26 16:37 .. + -rw-------. 1 hi684 hi684 1.8K Nov 26 16:37 id_rsa + -rw-r--r--. 1 hi684 hi684 398 Nov 26 16:37 id_rsa.pub + ``` + +2. 根据需求选择yaml + + 每位用户`/home/${USER}`目录下面都存放了一个 lkp-tests 的文件夹。 + + ```shell + hi684@account-vm ~% cd lkp-tests/jobs + hi684@account-vm ~/lkp-tests/jobs% ls -hl borrow-* + -rw-r--r--. 1 root root 53 Nov 2 14:54 borrow-10d.yaml + -rw-r--r--. 1 root root 64 Nov 2 14:54 borrow-1d.yaml + -rw-r--r--. 1 root root 235 Nov 19 15:27 borrow-1h.yaml + ``` + +3. 提交yaml并连接测试机（物理机） + + ```shell + hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=taishan200-2280-2s64p-256g borrow-1h.yaml + 2020-11-26 14:49:13 +0800 WARN -- skip non-executable /home/hi684/lkp-tests/monitors/plain/wait + submit borrow-1h.yaml, got job_id=z9.170594 + query=>{"job_id":["z9.170594"]} + connect to ws://172.168.131.2:11310/filter + {"job_id":"z9.170594","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-26/taishan200-2280-2s64p-256g/openeuler-20.03-aarch64/3600/z9.170594"} + {"job_id": "z9.170594", "result_root": "/srv/result/borrow/2020-11-26/taishan200-2280-2s64p-256g/openeuler-20.03-aarch64/3600/z9.170594", "job_state": "set result root"} + {"job_id": "z9.170594", "job_state": "boot"} + {"job_id": "z9.170594", "job_state": "download"} + {"time":"2020-11-26 14:51:56","mac":"84-46-fe-26-d3-47","ip":"172.168.178.48","job_id":"z9.170594","state":"running","testbox":"taishan200-2280-2s64p-256g--a5"} + {"job_state":"running","job_id":"z9.170594"} + {"job_id": "z9.170594", "state": "set ssh port", "ssh_port": "50420", "tbox_name": "taishan200-2280-2s64p-256g--a5"} + Host 172.168.131.2 not found in /home/hi684/.ssh/known_hosts + Warning: Permanently added '[172.168.131.2]:50420' (ECDSA) to the list of known hosts. + Last login: Wed Sep 23 11:10:58 2020 + + + Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 + + System information as of time: Thu Nov 26 14:51:59 CST 2020 + + System load: 1.31 + Processes: 1020 + Memory used: 5.1% + Swap used: 0.0% + Usage On: 3% + IP address: 172.168.178.48 + Users online: 1 + + + + root@taishan200-2280-2s64p-256g--a5 ~# + ``` + + 更多关于`submit命令如何使用`、`testbox都有什么可选项`、`如何borrow指定的操作系统`，请参见文章末尾FAQ。 + +4. 使用完毕退还测试机（物理机） + + ```shell + root@taishan200-2280-2s64p-256g--a5 ~# reboot + Connection to 172.168.131.2 closed by remote host. + Connection to 172.168.131.2 closed. + hi684@account-vm ~/lkp-tests/jobs% + ``` + + +# FAQ + + +* 如何自行修改申请时长 + + ```shell + hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml + suite: borrow + testcase: borrow + + sshd: + pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + # sleep at the bottom + sleep: 1h + hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml + sleep: 1h + # 使用vim来修改你的sleep字段的值 + hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml + # 修改完毕后重新submit即可 + hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml + ``` + +* Submit命令指导 + + 参考文档：[submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/… + +* testbox有什么可选项 + + testbox可选项请参考：https://gitee.com/wu_fengguang/lab-z9/tree/master/hosts + + >![](./../public_sys-resources/icon-note.gif) **说明：** + > + > 虚拟机的testbox : vm-xxx + > 物理机的testbox : taishan200-2280-xxx + + + + >![](./../public_sys-resources/icon-notice.gif) **注意：** + > - 物理机的testbox若选择以`--axx`结尾的，则表示指定到了具体的某一个物理机。若此物理机任务队列中已经有任务在排队，则需要等待队列中前面的任务执行完毕后，才会轮到你提交的borrow任务。 + > - 物理机的testbox若不选择以`-axx`结尾的，表示不指定具体的某一个物理机。则此时集群中的空闲物理机会即时被分配执行你的borrow任务。 + +* 如何 borrow 指定的操作系统 + + 关于支持的`os`, `os_arch`, `os_version`，参见：[os-os_verison-os_arch.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/job/os-os_verison-os_arch.md) diff --git a/doc/manual/how-to-compare-test-results.md b/doc/manual/how-to-compare-test-results.md new file mode 100644 index 0000000..a7348d8 --- /dev/null +++ b/doc/manual/how-to-compare-test-results.md @@ -0,0 +1,49 @@ +# Introduction of compare feature + The compare feature used to analyze the results of different jobs, showing the performance waves and changes + under different influencing factors, for users to analyze performance influencing factor. + +## url: https://compass-ci.openeuler.org/compare + +## options directions +### filters: + - suite: iperf, netperf, mysql, ... + - OS: openeuler 20.03, centos 7.6, ... + - os_arch: aarch64, x86 + - tbox_group: vm-2p8g, taishan200-2880-2s48p-256g, ... + we can combine the above options arbitrarily to limit compare scope. + choose at least one option as filter. + +### dimension + Dimension can select: os, os_version, os_arch, suite, tbox_group. + Within filter, we will compare all different job result by dimension + and keep other test conditions are same. + +## example: + filter: suite = iperf + dimension: os_version + + result: + os=openeuler/os_arch=aarch64/pp.iperf.protocol=tcp/pp.iperf.runtime=20/tbox_group=vm-2p8g # other test conditions keep same + + + 20.09 20.03 metric + -------------------- ------------------------------ ------------------------------ + fails:runs change fails:runs + | | | + 3:3 -100.0% 0:3 last_state.exit_fail + 3:3 -100.0% 0:3 last_state.is_incomplete_run + 3:3 -100.0% 0:3 last_state.test.iperf.exit_code.127 + 3:3 -100.0% 0:3 stderr.perf_command_failed + + + + 20.09 20.03 metric + -------------------- ------------------------------ ------------------------------ + %stddev change %stddev + \ | \ + 4.461021e+10 ± 6% -17.4% 3.686392e+10 ± 12% iperf.tcp.receiver.bps + 4.461112e+10 ± 6% -17.4% 3.686935e+10 ± 12% iperf.tcp.sender.bps + 94.82 -44.0% 53.10 boot-time.boot + 123.11 -58.4% 51.19 boot-time.idle + 0.00 0 4.87 boot-time.kernel_boot + 4165.50 ± 12% -99.9% 5.00 interrupts.38:GICv3.36.Level.virtio0 diff --git a/doc/manual/how-to-use-cci-to-test-open-source-projects.md b/doc/manual/how-to-use-cci-to-test-open-source-projects.md new file mode 100644 index 0000000..e73b9f9 --- /dev/null +++ b/doc/manual/how-to-use-cci-to-test-open-source-projects.md @@ -0,0 +1,68 @@ +# 使用 compass-ci 平台测试开源项目 + +本文介绍了使用 compass-ci 平台测试开源项目的操作。 + +### 添加待测试仓库 URL 到 upstream-repos 仓库 + +执行以下步骤，将想要测试的代码仓信息以 yaml 文件的方式添加到 upstream-repos 仓库（https://gitee.com/wu_fengguang/upstream-repos ）。 + +1. Fork 要测试的代码仓库并 git clone 到本地，本文以 blacklight 仓库（https://github.com/baskerville/backlight ）为例说明。 + +![](./../pictures/fork_blacklight.png) + +2. 执行以下命令，以首字母和仓库名创建文件路径。 + + ``` + mkdir -p b/backlight + ``` + +3. 执行以下命令，在该目录下新建同名文件 backlight。 + ``` + cd b/backlight + touch backlight + ``` + +4. 执行以下命令，将 backlight 仓库 url 信息写入 backlight 文件。 + + ``` + vim backlight + ``` + 内容格式为 + + ``` + --- + url: + - https://github.com/baskerville/backlight + ``` + + >![](./../public_sys-resources/icon-notice.gif) **注意：** + > + >可参考 upstream-repos 仓库中已有文件格式,请保持格式一致。 + +5. 通过 Pull Request 命令将新增的 backlight 文件提交到 blacklight 仓库。 + + +### 提交测试任务到 compass-ci 平台 + +1. 准备测试用例 + + 测试用例可以自己编写并添加到 lkp-tests 仓库，也可以直接使用 lkp-tests 仓库（https://gitee.com/wu_fengguang/lkp-tests ）的 jobs 目录下已有的测试用例。 + + * 使用仓库中已经适配好的测试用例 + 如果 lkp-tests 仓库中正好有你想要的测试用例，你可以直接使用。以 iperf.yaml 文件为例说明如下： + iperf.yaml 是一个已经适配好的测试用例，它位于 lkp-tests 仓库的 jobs 目录下，其中有一些基本的测试参数。 + + * 编写测试用例并添加到仓库 + + 请参考：[如何添加测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… ) + +2. 配置 auto_submit.yaml 文件，提交测试任务 + + 你只需要在 compass-ci 仓库下面的 sbin/auto_submit.yaml 文件中添加配置信息，如： + ``` + b/backlight/backlight: + - testbox=vm-2p8g os=openEuler os_version=20.03 os_mount=initramfs os_arch=aarch64 iperf.yaml + ``` + 通过 Pull Request 的方式将修改好的 auto_submit.yaml 文件提交到 compass-ci 仓库，就可以使用 compass-ci 测试你的项目了。 + + auto_submit.yaml 文件的参数配置请参考 https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job 。 diff --git a/doc/manual/how-to-view-test-results.md b/doc/manual/how-to-view-test-results.md new file mode 100644 index 0000000..36caaeb --- /dev/null +++ b/doc/manual/how-to-view-test-results.md @@ -0,0 +1,24 @@ +# 如何查看任务结果 + +测试用例执行完成后，可以登录到 https://compass-ci.openeuler.org/jobs 查看执行结果。 + +可以根据提交时定义的 suite 或者提交任务返回的 id 在网站上找到自己的任务，点击任务对应的 job_state 列即可查看任务结果。 + +![](./../pictures/jobs.png) + +## 结果文件介绍 + +**job.yaml文件** + + +job.yaml 文件中部分字段是用户提交上来的，其他字段是平台根据提交的 job 自动添加进来的。此文件包含了测试任务需要的所有参数。 + +**output文件** + +output 文件记录了用例的执行过程，文件最后部分一般会有 check_exit_code 这个状态码，非 0 代表测试用例错误。 + +**stats.json** + +测试用例执行完成会生成一个与测试用例同名的文件，记录它们的测试命令及标准化输出结果。compass-ci 会对这些文件进行解析，生成后缀名是 .json 的文件。 + +stats.json 是所有的 json 文件的汇总，所有测试命令的关键结果都会统计到这个文件中，便于后续的比较和分析。 diff --git a/doc/manual/how-to-write-pkgbuild.md b/doc/manual/how-to-write-pkgbuild.md new file mode 100644 index 0000000..66a1833 --- /dev/null +++ b/doc/manual/how-to-write-pkgbuild.md @@ -0,0 +1,81 @@ +# PKGBUILD是什么 + +PKGBUILD 是一个 shell 脚本，makepkg 通过 PKGBUILD 中包含的指令，生成包含二进制文件和安装指令的软件包。 + +# PKGBUILD包含什么 + +PKGBUILD包含两部分内容：变量和函数。 + +## 定义变量 + +- pkgname: 必须定义，表示软件包的名称; +- pkgver: 必须定义，表示软件包的版本号； +- pkgrel: 必须定义，表示软件包的发布号； +- arch: 必须定义，表示软件包使用的架构序列； +- depends: 可选字段，软件测试运行时需要的依赖包名称； +- makedepends: 可选字段，构建软件包时需要的文件列表； +- source: 可选字段，指定每个源文件的MD5哈希值，用于构建过程中验证源文件的完整性。 + +## 定义函数 + +- package函数 + package函数必须定义，此函数用于将文件安装到将成为构建包的根目录的目录中； +- prepare函数 + 定义可选的prepare函数，在其中执行用于准备构建源码的操作； +- build函数 + 定义可选的build函数，用于编译和/或构建源码； +- check函数 + 定义可选的check函数，用于运行程序包的测试套件。 + +>![](./../public_sys-resources/icon-notice.gif) **注意：** +> +> srcdir 是提取或复制源文件的目录，所有打包功能都在 srcdir 目录内部运行；pkgdir 是构建软件包的根目录，仅在 package 函数中使用。 + + +# 如何编写PKGBUILD + + +1. 执行如下命令创建PKGBUILD，文件名必须以“PKGBUILD”命名。 + + ```shell + touch PKGBUILD + ``` + + +2. 使用 vim 打开 PKGBUILD，编写 PKGBUILD 文件内容，一个 PKGBUILD 示例如下： + + ```shell + pkgname=zstd + pkgver=1.4.4 + pkgrel=2 + arch=('i686' 'x86_64' 'aarch64') + url='https://github.com/facebook/zstd' + license=('custom:BSD3' 'GPL2') + depends=('xz' 'zlib' 'lz4') + makedepends=('git') + source=('git://github.com/facebook/zstd.git#branch=dev') + md5sums=('SKIP') + + pkgver() { + cd "$srcdir/$pkgname" + git describe --long --tags | sed 's/$[^-]*-g$/r\1/;s/-/./g;s/^v//g' + } + + build() { + cd "$srcdir/$pkgname" + make + make -C contrib/pzstd + } + + package() { + cd "$srcdir/$pkgname" + make PREFIX="/usr" DESTDIR="$pkgdir/" install + install -D -m755 contrib/pzstd/pzstd "$pkgdir/usr/bin/pzstd" + install -D -m644 LICENSE "${pkgdir}/usr/share/licenses/${pkgname}/LICENSE" + } + ``` + +# 参考 + +- [PKGBUILD(5) Manual Page](https://www.archlinux.org/pacman/PKGBUILD.5.html) +- [pkgbuild demo文件](https://git.archlinux.org/pacman.git/plain/proto/PKGBUILD.proto) diff --git a/doc/manual/install-cci-client-locally.md b/doc/manual/install-cci-client-locally.md new file mode 100644 index 0000000..43a6bf5 --- /dev/null +++ b/doc/manual/install-cci-client-locally.md @@ -0,0 +1,28 @@ +# 本地安装compass-ci客户端 + +Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客户端，通过本地安装 lkp-tests 可以手动提交测试任务 + +:exclamation: 前提条件：按照 [apply-ssh-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master… 完成如下动作： +- send apply account email +- receive email from compass-ci(a)qq.com + +## Getting started + +1. setup default config + run the following command to add the below setup to default config file + mkdir -p ~/.config/compass-ci/defaults/ + cat >> ~/.config/compass-ci/defaults/${USER}.yaml <<-EOF + my_uuid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx + my_email: xxx + my_name: xx + EOF + +2. download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source ~/.bashrc && source ~/.bash_profile + +3. submit job + Now try [submitting a job to compass-ci](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… diff --git a/doc/manual/submit-usage.md b/doc/manual/submit-usage.md new file mode 100644 index 0000000..71d9dfd --- /dev/null +++ b/doc/manual/submit-usage.md @@ -0,0 +1,211 @@ +# submit 命令详解 + +### 前提条件 + +已经按照[本地安装compass-ci客户端.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/本地安装compass-ci客户端.md )完成安装 + +### 概述 + +submit 命令的作用是提交测试任务。该命令提供了多个选项帮助用户更灵活的提交任务，您可以在命令行直接输入 summit 命令来查看帮助信息，并根据实际需求灵活使用。 + + +### 基本用法 + +测试任务以 yaml 文件的方式提交，因此，您需要事先准备好测试任务的 yaml 文件（本文以 iperf.yaml 为例）。使用如下命令提交测试任务： + +``` +submit iperf.yaml +``` +如下所示： + +```shell +hi8109@account-vm ~% submit iperf.yaml +submit iperf.yaml, got job_id=z9.173924 +``` + +本文所示的 yaml 文件均已添加 testbox 字段，若您提交的 yaml 文件不包含该字段将报错： + +```shell +hi8109@account-vm ~% submit iperf.yaml +submit iperf.yaml failed, got job_id=0, error: Missing required job key: 'testbox' +``` + +您可以在 yaml 文件中添加 testbox 字段，或使用如下命令： + +``` +submit iperf.yaml testbox=vm-2p8g +``` + +testbox 字段的值指定需要的测试机，可以使用 `ls` 命令查看 `lkp-tests/hosts` 路径下的可选测试机。如下图所示： + +```shell +hi8109@account-vm ~/lkp-tests/hosts% ll +total 120K +-rw-r--r--. 1 root root 76 2020-11-02 14:54 vm-snb +-rw-r--r--. 1 root root 64 2020-11-02 14:54 vm-pxe-hi1620-2p8g +-rw-r--r--. 1 root root 64 2020-11-02 14:54 vm-pxe-hi1620-2p4g +-rw-r--r--. 1 root root 64 2020-11-02 14:54 vm-pxe-hi1620-2p1g +-rw-r--r--. 1 root root 64 2020-11-02 14:54 vm-pxe-hi1620-1p1g +-rw-r--r--. 1 root root 75 2020-11-02 14:54 vm-hi1620-2p8g +-rw-r--r--. 1 root root 75 2020-11-02 14:54 vm-hi1620-2p4g +-rw-r--r--. 1 root root 75 2020-11-02 14:54 vm-hi1620-2p1g +-rw-r--r--. 1 root root 75 2020-11-02 14:54 vm-hi1620-1p1g +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p8g-pxe +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p8g +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p4g-pxe +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p4g +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p1g-pxe +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-2p1g +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-1p1g-pxe +-rw-r--r--. 1 root root 61 2020-11-02 14:54 vm-1p1g +-rw-r--r--. 1 root root 14 2020-11-02 14:54 taishan200-2280-2s64p-256g +-rw-r--r--. 1 root root 497 2020-11-02 14:54 lkp-skl-d01 +-rw-r--r--. 1 root root 574 2020-11-02 14:54 lkp-ivb-d04 +-rw-r--r--. 1 root root 606 2020-11-02 14:54 lkp-ivb-d02 +-rw-r--r--. 1 root root 384 2020-11-02 14:54 lkp-ivb-d01 +-rw-r--r--. 1 root root 388 2020-11-02 14:54 lkp-hsw-d01 +-rw-r--r--. 1 root root 385 2020-11-02 14:54 lkp-bdw-de1 +-rw-r--r--. 1 root root 11 2020-11-02 14:54 dc-8g +-rw-r--r--. 1 root root 11 2020-11-02 14:54 dc-4g +-rw-r--r--. 1 root root 11 2020-11-02 14:54 dc-2g +-rw-r--r--. 1 root root 11 2020-11-02 14:54 dc-1g +-rw-r--r--. 1 root root 13 2020-11-02 14:54 2288hv5-2s64p +-rw-r--r--. 1 root root 74 2020-11-02 14:54 vm-snb-i386 +``` + +>![](./../public_sys-resources/icon-note.gif) **说明：** +> +>使用 "=" 更新 yaml 中的字段，"=" 在命令行中的位置不同优先级不同：. +> * submit iperf.yaml testbox=vm-2p8g 命令中 "=" 定义在 yaml 文件之后，则 "=" 的优先级高于 yaml 文件,testbox=vm-2p8g 会覆盖 yaml 文件中已经定义的 testbox 字段。 +> * submit testbox=vm-2p8g iperf.yaml 命令中 "=" 定义在 yaml 文件之前，则 "=" 的优先级低于 yaml 文件,testbox=vm-2p8g 不会覆盖 yaml 文件中已经定义的 testbox 字段，只有当 yaml 文件中不存在 testbox 字段才会赋值。 + + + +### 高级用法 + +submit 命令的选项如下图所示： + +```shell +hi8109@account-vm ~% submit +Usage: submit [options] job1.yaml job2.yaml ... + submit test jobs to the scheduler + +options: + -s, --set 'KEY: VALUE' add YAML hash to job + -o, --output DIR save job yaml to DIR/ + -a, --auto-define-files auto add define_files + -c, --connect auto connect to the host + -m, --monitor monitor job status: use -m 'KEY: VALUE' to add rule + --my-queue add to my queue +``` + +* **-s的用法** + + 使用 -s 'KEY:VALUE' 参数可以将键值对更新到提交的任务当中。示例如下所示： + ``` + submit -s 'testbox: vm-2p8g' iperf.yaml + ``` + + + * 如果 iperf.yaml 中不存在 testbox：vm-2p8g ，最终提交的任务将会加上该信息。 + * 如果 iperf.yaml 中存在 testbox 字段，但是值不为 vm-2p8g ，最终提交的任务中 testbox 的值将会被替换为vm-2p8g。 + +* **-o的用法** + + 使用-o DIR 命令可以将最终生成的yaml文件保存到指定目录 DIR 下。示例如下所示： + + ``` + submit -o ~/iperf.yaml + ``` + + 运行命令之后会在指定目录生成经过 submit 处理过的 yaml 文件。 + +* **-a的用法** + + 如果你的测试用例对客户端的lkp-tests 做了更改，需要使用 -a 选项来适配。将客户端的 lkp-tests 下做的更改，同步到服务端，并在测试机上生成你的测试脚本。 + 示例命令如下： + + ``` + submit -a iperf.yaml + ``` + +* **-m的用法** + + 使用 -m 参数可以启动任务监控功能，并将任务执行过程中的各种状态信息打印到控制台上，方便用户实时监控测试任务的执行过程。 + 示例命令如下： + + ``` + submit -m iperf.yaml + ``` + + 控制台显示如下： + + ```shell + hi8109@account-vm ~% submit -m iperf.yaml + submit iperf.yaml, got job_id=z9.173923 + query=>{"job_id":["z9.173923"]} + connect to ws://172.168.131.2:11310/filter + {"job_id":"z9.173923","message":"","job_state":"submit","result_root":"/srv/result/iperf/2020-11-30/vm-2p8g/openeuler-20.03-aarch6 + {"job_id": "z9.173923", "result_root": "/srv/result/iperf/2020-11-30/vm-2p8g/openeuler-20.03-aarch64/tcp-30/z9.173923", "job_state + {"job_id": "z9.173923", "job_state": "boot"} + {"job_id": "z9.173923", "job_state": "download"} + {"time":"2020-11-30 20:28:16","mac":"0a-f5-9f-83-62-ea","ip":"172.18.192.21","job_id":"z9.173923","state":"running","testbox":"vm- + {"job_state":"running","job_id":"z9.173923"} + {"job_state":"post_run","job_id":"z9.173923"} + {"start_time":"2020-11-30 12:25:15","end_time":"2020-11-30 12:25:45","loadavg":"1.12 0.38 0.14 1/105 1956","job_id":"z9.173923"} + {"job_state":"finished","job_id":"z9.173923"} + {"job_id": "z9.173923", "job_state": "complete"} + {"time":"2020-11-30 20:28:54","mac":"0a-f5-9f-83-62-ea","ip":"172.18.192.21","job_id":"z9.173923","state":"rebooting","testbox":"v + {"job_id": "z9.173923", "job_state": "extract_finished"} + connection closed: normal + ``` + +* **-c的用法** + + -c 参数需要搭配 -m 参数来使用，可以使申请设备的任务实现自动登入功能。 + + 示例命令如下： + + ``` + submit -m -c borrow-1h.yaml + ``` + 当我们提交一个申请设备的任务后，会获取到返回的登陆信息，如 `ssh ip -p port`，添加 -c 参数之后不需要我们手动输入 ssh 登陆命令来进入执行机。 + + 控制台显示如下： + + ```shell + hi8109@account-vm ~% submit -m -c borrow-1h.yaml + submit borrow-1h.yaml, got job_id=z9.173925 + query=>{"job_id":["z9.173925"]} + connect to ws://172.168.131.2:11310/filter + {"job_id":"z9.173925","message":"","job_state":"submit","result_root":"/srv/result/borrow/2020-11-30/vm-2p8g/openeuler-20.03-aarch + {"job_id": "z9.173925", "result_root": "/srv/result/borrow/2020-11-30/vm-2p8g/openeuler-20.03-aarch64/3600/z9.173925", "job_state" + {"job_id": "z9.173925", "job_state": "boot"} + {"job_id": "z9.173925", "job_state": "download"} + {"time":"2020-11-30 20:35:04","mac":"0a-24-5d-c8-aa-d0","ip":"172.18.101.4","job_id":"z9.173925","state":"running","testbox":"vm-2 + {"job_state":"running","job_id":"z9.173925"} + {"job_id": "z9.173925", "state": "set ssh port", "ssh_port": "50200", "tbox_name": "vm-2p8g.taishan200-2280-2s48p-256g--a52-7"} + Host 172.168.131.2 not found in /home/hi8109/.ssh/known_hosts + Warning: Permanently added '[172.168.131.2]:50200' (ECDSA) to the list of known hosts. + Last login: Wed Sep 23 11:10:58 2020 + + + Welcome to 4.19.90-2003.4.0.0036.oe1.aarch64 + + System information as of time: Mon Nov 30 12:32:04 CST 2020 + + System load: 0.50 + Processes: 105 + Memory used: 6.1% + Swap used: 0.0% + Usage On: 89% + IP address: 172.17.0.1 + Users online: 1 + + + + root@vm-2p8g ~# + ``` + + 已经成功登陆执行机。 + -- 2.23.0

1 1

[PATCH compass-ci] os-os_verison-os_arch.md: add queue value when submit a job
by Xu Xijian 01 Dec '20

01 Dec '20

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/os-os_verison-os_arch.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/job/os-os_verison-os_arch.md b/doc/job/os-os_verison-os_arch.md index 9939414..bc900e3 100644 --- a/doc/job/os-os_verison-os_arch.md +++ b/doc/job/os-os_verison-os_arch.md @@ -13,4 +13,4 @@ Meaning: ... Usage example: -- submit iperf.yaml testbox=vm-2p8g os=openeuler os_arch=aarch64 os_version=20.03 +- submit iperf.yaml testbox=vm-2p8g queue=vm-2p8g~$USER os=openeuler os_arch=aarch64 os_version=20.03 -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] kernel_version.md: text format adapt to markdown
by Xu Xijian 01 Dec '20

01 Dec '20

In markdown documents, text for bash should be contained in block as below: ```bash content ``` If not, the display in gitee will be strange, for example, $text$ will show as a ltalics text, some lines with special character will erasure line breaks. Thus the old text will show as follows: ├── headers-4.19.90-2003.4.0.0036.oe1.aarch64.cgz├── modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz├── vmlinuz-4.19.90-2003.4.0.0036.oe1.aarch64└── vmlinuz -> vmlinuz-4.19.90-2003.4.0.0036.oe1.aarch64 Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/kernel_version.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/job/kernel_version.md b/doc/job/kernel_version.md index 4a193fb..b73d42e 100644 --- a/doc/job/kernel_version.md +++ b/doc/job/kernel_version.md @@ -26,6 +26,7 @@ Meaning: centos/aarch64/8.1.1911 - 4.18.0-147.el8(default) +```bash Related files: - In initramfs boot process, every kernel version is related with a vmlinuz, module and headers. - Files like below under $boot_dir, an example $boot_dir can be "/srv/os/openeuler/aarch64/20.03/boot". @@ -33,6 +34,7 @@ Related files: ├── modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz ├── vmlinuz-4.19.90-2003.4.0.0036.oe1.aarch64 └── vmlinuz -> vmlinuz-4.19.90-2003.4.0.0036.oe1.aarch64 +``` Usage example: - submit iperf.yaml testbox=vm-2p8g queue=vm-2p8g~$USER os=openeuler os_arch=aarch64 os_version=20.03 runtime=20 kernel_version=4.19.90-2003.4.0.0036.oe1.aarch64 -- 2.23.0

1 0

[PATCH v2 compass-ci 3/4] container/logging-es: set available memory size
by Liu Yinsi 01 Dec '20

01 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/logging-es/Dockerfile | 6 ++++-- container/logging-es/build | 19 +++++++++++-------- 2 files changed, 15 insertions(+), 10 deletions(-) diff --git a/container/logging-es/Dockerfile b/container/logging-es/Dockerfile index 6c01c43..97dd81c 100644 --- a/container/logging-es/Dockerfile +++ b/container/logging-es/Dockerfile @@ -4,6 +4,8 @@ ARG BASE_IMAGE FROM $BASE_IMAGE +ARG MEMORY + # docker image borrowed from hub.docker.com/r/gagara/elasticsearch-oss-arm64 MAINTAINER Wu Zhende <wuzhende666(a)163.com> @@ -14,8 +16,8 @@ RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elastic sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && \ sed -i '$a indices.memory.index_buffer_size: 20%' /usr/share/elasticsearch/config/elasticsearch.yml && \ sed -i '$a thread_pool.write.queue_size: 2000' /usr/share/elasticsearch/config/elasticsearch.yml && \ - sed -i "s/-Xms1g/-Xms30g/g" /usr/share/elasticsearch/config/jvm.options && \ - sed -i "s/-Xmx1g/-Xmx30g/g" /usr/share/elasticsearch/config/jvm.options + sed -i "s/-Xms1g/-Xms${MEMORY}g/g" /usr/share/elasticsearch/config/jvm.options && \ + sed -i "s/-Xmx1g/-Xmx${MEMORY}g/g" /usr/share/elasticsearch/config/jvm.options RUN mkdir /usr/share/elasticsearch/tmp && \ chown -R 1090:1090 /usr/share/elasticsearch diff --git a/container/logging-es/build b/container/logging-es/build index 451041b..50077d8 100755 --- a/container/logging-es/build +++ b/container/logging-es/build @@ -1,14 +1,17 @@ -#!/bin/bash +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -declare -A BASE_IMAGE_DICT +require_relative '../defconfig.rb' -BASE_IMAGE_DICT=( - ["aarch64"]="gagara/elasticsearch-oss-arm64:7.6.2" - ["x86_64"]="elasticsearch:7.6.2" -) +BASE_IMAGE_DICT = { + 'aarch64' => 'gagara/elasticsearch-oss-arm64:7.6.2', + 'x86_64' => 'elasticsearch:7.6.2' +}.freeze -BASE_IMAGE=${BASE_IMAGE_DICT[$(arch)]} +BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] -docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE="$BASE_IMAGE" . +available_memory = set_available_memory + +system "docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} --build-arg MEMORY=#{available_memory} ." -- 2.23.0

1 0

[PATCH v2 compass-ci 2/4] container/es: set available memory size
by Liu Yinsi 01 Dec '20

01 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/es/Dockerfile | 6 ++++-- container/es/build | 8 ++++++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/container/es/Dockerfile b/container/es/Dockerfile index e626dae..c54f3cf 100644 --- a/container/es/Dockerfile +++ b/container/es/Dockerfile @@ -3,6 +3,8 @@ FROM alpine:3.11 +ARG MEMORY + RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk add --no-cache elasticsearch curl @@ -27,8 +29,8 @@ RUN rm -rf /etc/init.d/elasticsearch \ RUN sed -i 's:#path.data\: /path/to/data:path.data\: /srv/es:' /usr/share/es/config/elasticsearch.yml; RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/es/config/elasticsearch.yml; -RUN sed -i "s/-Xms256m/-Xms20g/g" /usr/share/es/config/jvm.options -RUN sed -i "s/-Xmx256m/-Xmx20g/g" /usr/share/es/config/jvm.options +RUN sed -i "s/-Xms256m/-Xms${MEMORY}g/g" /usr/share/es/config/jvm.options +RUN sed -i "s/-Xmx256m/-Xmx${MEMORY}g/g" /usr/share/es/config/jvm.options WORKDIR /usr/share/es diff --git a/container/es/build b/container/es/build index d5607e0..1693e7f 100755 --- a/container/es/build +++ b/container/es/build @@ -1,6 +1,10 @@ -#!/bin/sh +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -docker build -t es643b:alpine311 . --network=host +require_relative '../defconfig.rb' +available_memory = set_available_memory + +system "docker build -t es643b:alpine311 --build-arg MEMORY=#{available_memory}. --network=host" -- 2.23.0

1 0

[PATCH compass-ci] doc/manual: add 如何登陆测试机.md
by Zhang Yu 01 Dec '20

01 Dec '20

Add a document for introducing how to log in test machine Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- ...06\346\265\213\350\257\225\346\234\272.md" | 80 +++++++++++++++++++ 1 file changed, 80 insertions(+) create mode 100644 "doc/manual/\345\246\202\344\275\225\347\231\273\351\231\206\346\265\213\350\257\225\346\234\272.md" diff --git "a/doc/manual/\345\246\202\344\275\225\347\231\273\351\231\206\346\265\213\350\257\225\346\234\272.md" "b/doc/manual/\345\246\202\344\275\225\347\231\273\351\231\206\346\265\213\350\257\225\346\234\272.md" new file mode 100644 index 0000000..8855695 --- /dev/null +++ "b/doc/manual/\345\246\202\344\275\225\347\231\273\351\231\206\346\265\213\350\257\225\346\234\272.md" @@ -0,0 +1,80 @@ +这篇文档将告诉你如何登陆测试环境 + +# 1 前提条件 +请先学习: +submit 命令详解： +参考文档：[submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/submit命令详解.md) +如何申请测试机，并在本地生成RSA公私钥对： +参考文档：[submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/如何申请测试机.md) +apply-account.md，配置个人邮箱： +参考文档：[submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md) + +# 2. 操作方法 +## 2.1 在job.yaml里加上sshd和sleep字段, 以host-info.yaml任务为例： + suite: host-info + category: functional + + sshd: + pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + # sleep at the bottom + sleep: 1h + + host-info: + +## 2.2 提交任务的方式有2种，适用于不同的登陆需求： +### 第一种：使用submit -m -c的方式提交任务： + 这种方式提交的任务会在指定的位置sleep并直接登陆到测试机中，适用于登陆测试环境后手动调试 + 命令：submit -m -c host-info.yaml + 任务提交完成后，当测试执行到sshd后会自动登陆到测试机器上： + 效果如下： + hi6325@account-vm ~% submit -m -c atomic.yaml + submit atomic.yaml, got job_id=crystal.146528 + query=>{"job_id":["crystal.146528"]} + connect to ws://localhost:11310/filter + {"job_id": "crystal.146528", "result_root": "/srv/result/atomic/2020-12-01/vm-2p8g/openeuler-20.03-aarch64/1-1000/crystal.146528", "job_state": "set result root" + {"job_id": "crystal.146528", "job_state": "boot"} + {"job_id": "crystal.146528", "job_state": "download"} + {"time":"2020-12-01 10:12:33","mac":"0a-2d-7b-d9-f8-b1","ip":"172.18.252.12","job_id":"crystal.146528","state":"running","testbox":"vm-2p8g.zhyl-453231"} + {"job_state":"running","job_id":"crystal.146528"} + {"job_id": "crystal.146528", "state": "set ssh port", "ssh_port": "51750", "tbox_name": "vm-2p8g.zhyl-453231"} + + root@vm-2p8g ~# + +### 第二种：使用submit的方式提交任务： + 这种方式提交的任务会在测试执行完成后，系统自动发送一封邮件提醒您可以在指定时间内登陆到测试环境 + 命令： submit host-info.yaml + 任务执行完成后，系统发送邮件如下： + Subject: [NOTIFY Compass-ci] vm-2p8g-294828 ready to use + + Dear $my_username: + Thanks for your participation in software ecosystem! + According to your application, vm-2p8g-294828 has been provisioned. + The datails are as follows: + + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch + + Regards + Compass-Ci + + 可通过ssh方式登陆到测试环境： + 命令： ssh root(a)api.compass-ci.openeuler.org -p $port + 效果如下： + hi6325@account-vm ~% ssh root(a)api.compass-ci.openeuler.org -p 51400 + + + root@vm-2p8g ~# -- 2.23.0

2 3

[PATCH compass-ci] lib: add error_messages.rb
by Lin Jiaxin 01 Dec '20

01 Dec '20

provide error messages for git bisect Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 64 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 lib/error_messages.rb diff --git a/lib/error_messages.rb b/lib/error_messages.rb new file mode 100644 index 0000000..113a7c7 --- /dev/null +++ b/lib/error_messages.rb @@ -0,0 +1,64 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' + +# get error messages hash +class ErrorMessages + def initialize(file) + @file = file + @seqno = '' + @in_stderr = false + @error_line = '' + @error_details = '' + @error_message = {} + @error_messages = Hash.new { |h, k| h[k] = Set.new } + end + + def extract_error_messages + File.open((a)file).each_line do |line| + is_next = extract_error_message(line) + + next if is_next + next unless @in_stderr + next unless @error_message['error_line'].downcase =~ /error|warning/ + + @error_messages[@error_message['error_line']] << @error_message['error_line'] + @error_message['error_details'] + end + return @error_messages + end + + private + + def extract_error_message(line) + if line =~ /^ / + @error_details += line + else + update_error_message(line) + case line + when /^## ______________([0-9.]+):stderr$/ + @in_stderr = true + @seqno = $1 + return true + when /^## ______________#{@seqno}:enderr$/ + update_error_messages + end + end + end + + def update_error_message(line) + @error_message['error_line'] = @error_line + @error_message['error_details'] = @error_details + @error_line = line + @error_details = '' + end + + def update_error_messages + @in_stderr = false + @seqno = '' + return unless @error_message['error_line'].downcase =~ /error|warning/ + + @error_messages[@error_message['error_line']] << @error_message['error_line'] + @error_message['error_details'] + end +end -- 2.23.0

1 0

[PATCH compass-ci] doc/tutorial: standalone partitions and increase memory
by Liu Yinsi 01 Dec '20

01 Dec '20

8g memory not enough, and should set standalone partitions before deploy CCI, because run container will takes up /var/lib/docker a lot of space. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- doc/tutorial.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index 03f1342..585be6f 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -288,9 +288,13 @@ job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAM - 硬件服务器类型：ThaiShan200-2280 (建议) 架构：aarch64 - 内存：>= 8GB + 内存：>= 32GB CPU：64 nuclear (建议) - 硬盘：>= 500G + 硬盘：>= 500G (建议划分独立分区) + + >**说明：** + >划分较小独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… + >划分较大独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… - 软件 OS：openEuler-aarch64-20.03 LTS -- 2.23.0

1 0

[PATCH compass-ci] doc/tutorial: standalone partitions and increase memory
by Liu Yinsi 01 Dec '20

01 Dec '20

8g memory not enough, and should set standalone partitions before deploy CCI, because run container will takes up /var/lib/docker a lot of space. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- doc/tutorial.md | 670 ++++++++++++++++++++++++------------------------ 1 file changed, 337 insertions(+), 333 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index 03f1342..a32d117 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -1,333 +1,337 @@ -# Compass-CI 用户指南 - -## 简介 - - -### 概念 - -Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github, Gitee, Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务。通过 Compass-CI，社区开发者可以将开源软件快速引入 openEuler 社区，并补充更多的测试用例，共同构建一个开放、完整的开源软件生态系统。 - - -### 功能描述 - -- 测试服务 - - 支持开发者基于本地设备开发，往 github 提交代码，Compass-CI 自动获取代码开展测试，并向开发者反馈测试结果。 - -- 调测环境登录 - - Compass-CI 提供 SSH 登录能力，测试过程中如果测出问题，开发者可根据需要登录环境进行调测。 - -- 测试结果分析 - - Compass-CI 记录历史测试结果，对外提供 web 及命令行接口，支持开发者针对已有的测试结果进行分析，挖掘影响测试结果的因素。 - -- 辅助定位 - - Compass-CI 在测试过程中可以自动识别错误信息，触发基于 git tree 的测试，找出引入问题模块的变化点。 - -### 应用场景 - -- 聚合开发者测试用例 - - 开发者向代码托管平台提交代码、测试用例、测试工具时，Compass-CI 自动获取提交的代码开展构建测试，同时获取开发者编写到开源软件包的测试用例进行自动化测试，并反馈测试结果。 - -- 登录环境随时调测 - - 测试过程中，发现有 Bug 时，可随时提供调测资源服务，登录到环境进行复现、调试。 - -- 快照数据分析对比 - - 测试过程中，全面监控系统运行信息（CPU/MEM/IO/网络等），对测试过程中的数据做快照归档，提供多次测试之间快照数据分析对比能力，协助开发者对测试结果开展分析，找出影响测试结果的因素。 - -- 辅助定界 - - 测试过程中，发现有 Bug 时，自动触发 Regression 机制，找出首次引入问题 Commit 信息。 - - - -## 使用 Compass-CI -> 当您注册之后，便可以编写 yaml 文件并通过我们的工具上传任务以进行自定义测试，测试功能将尽快上线。 - - -- 注册自己的仓库 - - 如果您想在 `git push` 的时候, 自动触发测试, 那么需要把您的公开 git url 添加到如下仓库 [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos)。 - ```bash - git clone https://gitee.com/wu_fengguang/upstream-repos.git - less upstream-repos/README.md - ``` - -### 编写 job yaml 文件 - -#### job yaml 简介 -job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAML_for_ruby.html)的格式编写，所有 job 文件位于```$LKP_SRC/jobs```路径下。[$LKP_SRC](https://gitee.com/wu_fengguang/lkp-te… - -#### job yaml 的结构 - -- 标识头部 Header（必选） - - 每一个 job 文件的开始部分，都有一些基础的描述信息，称之为 yaml 的 Header。头部主要有 suite 和 category，suite 是测试套的名称。category 是测试类型，包括 benchmarch（性能测试）/functional（功能测试）/noise-benchmark（不常用）。 - - ```yaml - suite: netperf \\测试套名称 - category: benchmark \\测试类型：benchmarch|functional|noise-benchmark - ``` - -- 测试脚本和参数（必选） - - - 测试脚本位于```$LKP_SRC/tests```，```$LKP_SRC/daemon```路径下。job yaml 中写入与上述路径中脚本相匹配的文件，将其视为可执行脚本。 - - ``` - $LKP_SRC/setup - $LKP_SRC/monitors - $LKP_SRC/daemon - $LKP_SRC/tests - ``` - - 参数值作为环境变量传递，每个测试脚本将在其文件头的注释中记录可接受的参数。 - - ```yaml - netserver: - netperf: - runtime: 10s - test: - - TCP_STREAM - - UDP_STREAM - send_size: - - 1024 - - 2048 - ``` -- 测试资源相关变量（必选） - - SUT/schduler/部署等资源相关变量。 - ```yaml - testbox: vm-2p8g - os: openeuler - os_version: 20.03 - os_arch: aarch64 - ``` -- 系统设置脚本（可选） - - 位于```$LKP_SRC/setup```路径下。设置脚本将在测试脚本之前运行，主要是用于启停一些依赖性服务或工具，或者配置测试所需的参数等。 - - ```yaml - cgroup2: #$LKP_SRC/setup/cgroup2 executable script - memory.high: 90% - memory.low: 50% - memory.max: max - memory.swap.max: - io.max: - io.weight: - rdma.max: - ``` -- 监控脚本（可选） - - 位于```$LKP_SRC/monitors```路径下。监视器在运行基准测试时可以捕获性能统计数据，对性能分析和回归根源有价值。 - - ```yaml - proc-vmstats: - nterval: 5 - ``` - -**了解更多 job yaml 相关知识** [Job definition to execution](https://gitee.com/wu_fengguang/lkp-tests/blob/master/jobs/README… - - -### 提交测试任务 - -### 查看测试结果 - -### 比较测试结果 - -### 提交 borrow 任务 - -### 提交 bisect 任务 - -## 高级功能 - -### 添加 OS 支持 - -制作一个 initramfs 启动的 cgz 镜像，当系统内核启动时，直接从打包的 cgz 镜像中导出 rootfs，在内存中展开文件系统，利用磁盘的高速缓存机制让系统直接在内存中读写文件，以提升系统的 I/O 性能。 - -#### 操作步骤 - - 1. 获取对应 os 版本的 rootfs（以 openEuler 为例） - - 通过 docker 获取 rootfs - - 1) 下载 openEuler 官方提供的 docker 镜像压缩包 - - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) 加载 docker 镜像 - ```bash - docker load -i openEuler-docker.aarch64 - ``` - - 3) 启动 openEuler 容器 - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) 拷贝 docker 的 rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - - 通过 qemu.img(qcow2格式)获取 rootfs (openEuler为例) - - - 1) 下载 openEule r官方网站提供的 qcow2 格式镜像 - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… - ``` - 2) 使用{compass-ci}/container/qcow2rootfs 制作rootfs - ```bash - cd {compass-ci}/container/qcow2rootfs - ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs - ``` - 2. 定制rootfs - 1. 使用chroot命令切换到 rootfs (此步骤需要 root 权限) - ```bash - chroot openEuler-rootfs - ``` - 2. 根据个人需要安装并配置服务 - - a. 修改 root 密码 - b. 配置 ssh 服务 - c. 检查系统时间 - d. 如果使用 docker 制作 osimage 还需要以下操作： - > 1. 安装所需版本内核 - > 2. 从 centos 官方网站下载内核rpm包 - > 3. 使用 yum 进行安装 - > 4. 删除 docker 环境变量文件 - - 3. 退出 rootfs，并打包 - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` -#### FAQ -1. 日志报错 “Unable to mount root fs on unknown-block” - - 问题现象 - ```bash - [ 0.390437] List of all partitions: - [ 0.390806] No filesystem could mount root, tried: - [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) - ... - [ 0.399404] Memory Limit: none - [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- - ``` - - 解决方法 - - 1）启动内存不足，增加内存可解决。 - 2）内核文件权限不足，给予 644 权限。 - -2. 系统运行缓慢 - - - 问题现象 - - 打包镜像体积过大，会消耗很大内存 - - 解决方法 - - 建议用户根据具体需要对 rootfs 进行裁剪。 - - -### 添加测试用例 - 参考文档详见：[添加测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/ad… - - -### PKGBUILD 构建 - -1. 概述： - - 使用 PKGBUILD 完成上游项目源代码的编译构建，同一项目可以使用不同 PKGBUILD 完成项目编译构建过程。 - PKGBUILD 相关参考文档：https://wiki.archlinux.org/index.php/PKGBUILD。 - -2. 步骤： - - 1) 注册仓库 - 如果您想在 git push 的时候, 自动触发测试, 那么需要把您的公开 git url 添加到如下仓库 [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos)。 - ```bash - git clone https://gitee.com/wu_fengguang/upstream-repos.git - less upstream-repos/README.md - ``` - - 2) 执行构建测试 - 项目代码更新后自动触发构建任务，无需其他操作。 - - 3) 查看结果 - web: https://compass-ci.openeuler.org/jobs。 - - 4) 备注 - 目前提供 archlinux 下已有 PKGBUILD 项目构建测试，暂不支持自主导入 PKGBUILD 文件。 - PKGBUILD 存放路径： /srv/git/archlinux/*/*/PKGBUILD。 - - -### depends - -- 依赖包构建 - 流程: 以下默认使用 debian:sid 版本进行,如果使用其他OS跑测试用例，则需要在 lkp-tests/distro/adaptation/ 下将包名以键值对的形式写入对应 OS 的文件中 - > 例： `echo 'apache2: httpd' >> lkp-tests/distro/adaptation/openeuler` - - 1) 在lkp-tests/distro/depends/下新建一个与测试用例同名的文件 - `vim lkp-tests/distro/depends/test` - 2) 将测试用例所需debian下的包名填入创建的文件中 - `echo "apache2" >> lkp-tests/distro/depends/test` - 3) 生成lkp-aarch64.cgz - `./crystal-ci/container/lkp-initrd/run` - 4) 提交测试用例 - `submit test.yaml` - 5) 查看结果 - https://compass-ci.openeuler.org/jobs - - - -### 本地搭建 compass-ci 服务器节点 - -在 openEuler 系统一键部署 compass-ci 环境，当前已支持 openEuler-aarch64-20.03-LTS 系统环境，以下配置仅供参考。 - -#### 准备工作 -- 硬件 - 服务器类型：ThaiShan200-2280 (建议) - 架构：aarch64 - 内存：>= 8GB - CPU：64 nuclear (建议) - 硬盘：>= 500G - -- 软件 - OS：openEuler-aarch64-20.03 LTS - git：2.23.0版本 (建议) - 预留空间：>= 300G - 网络：可以访问互联网 - - >**说明：** - >openEuler 系统安装详细操作请参考[添加测试用例](https://openeuler.org/zh/docs/20.03_LTS/docs/Installati… - - -#### 操作指导 -1. 登录 openEuler 系统 - -2. 创建工作目录并设置文件权限 - ```bash - mkdir demo && cd demo && umask 002 - ``` -3. 克隆 compass-ci 项目代码到 demo 目录 - ```bash - git clone https://gitee.com/wu_fengguang/compass-ci.git - ``` -4. 执行一键部署脚本 install-tiny - ```bash - cd compass-ci/sparrow && ./install-tiny - ``` - - -## 呼吁合作 - - 增强 git bisect 能力 - - 增强数据分析能力 - - 增强数据结果可视化能力 - -## 下一步计划 - -1. 优化 git bisect 精准率及效率。 -2. 优化数据分析联动能力，让数据分析更加聚焦。 -3. 数据可视化优化，更加友好展示数据比较结果。 -4. 增强登录认证机制，如：GPG 认证。 -5. 优化部署效率。 +# Compass-CI 用户指南 + +## 简介 + + +### 概念 + +Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github, Gitee, Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务。通过 Compass-CI，社区开发者可以将开源软件快速引入 openEuler 社区，并补充更多的测试用例，共同构建一个开放、完整的开源软件生态系统。 + + +### 功能描述 + +- 测试服务 + + 支持开发者基于本地设备开发，往 github 提交代码，Compass-CI 自动获取代码开展测试，并向开发者反馈测试结果。 + +- 调测环境登录 + + Compass-CI 提供 SSH 登录能力，测试过程中如果测出问题，开发者可根据需要登录环境进行调测。 + +- 测试结果分析 + + Compass-CI 记录历史测试结果，对外提供 web 及命令行接口，支持开发者针对已有的测试结果进行分析，挖掘影响测试结果的因素。 + +- 辅助定位 + + Compass-CI 在测试过程中可以自动识别错误信息，触发基于 git tree 的测试，找出引入问题模块的变化点。 + +### 应用场景 + +- 聚合开发者测试用例 + + 开发者向代码托管平台提交代码、测试用例、测试工具时，Compass-CI 自动获取提交的代码开展构建测试，同时获取开发者编写到开源软件包的测试用例进行自动化测试，并反馈测试结果。 + +- 登录环境随时调测 + + 测试过程中，发现有 Bug 时，可随时提供调测资源服务，登录到环境进行复现、调试。 + +- 快照数据分析对比 + + 测试过程中，全面监控系统运行信息（CPU/MEM/IO/网络等），对测试过程中的数据做快照归档，提供多次测试之间快照数据分析对比能力，协助开发者对测试结果开展分析，找出影响测试结果的因素。 + +- 辅助定界 + + 测试过程中，发现有 Bug 时，自动触发 Regression 机制，找出首次引入问题 Commit 信息。 + + + +## 使用 Compass-CI +> 当您注册之后，便可以编写 yaml 文件并通过我们的工具上传任务以进行自定义测试，测试功能将尽快上线。 + + +- 注册自己的仓库 + + 如果您想在 `git push` 的时候, 自动触发测试, 那么需要把您的公开 git url 添加到如下仓库 [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos)。 + ```bash + git clone https://gitee.com/wu_fengguang/upstream-repos.git + less upstream-repos/README.md + ``` + +### 编写 job yaml 文件 + +#### job yaml 简介 +job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAML_for_ruby.html)的格式编写，所有 job 文件位于```$LKP_SRC/jobs```路径下。[$LKP_SRC](https://gitee.com/wu_fengguang/lkp-te… + +#### job yaml 的结构 + +- 标识头部 Header（必选） + + 每一个 job 文件的开始部分，都有一些基础的描述信息，称之为 yaml 的 Header。头部主要有 suite 和 category，suite 是测试套的名称。category 是测试类型，包括 benchmarch（性能测试）/functional（功能测试）/noise-benchmark（不常用）。 + + ```yaml + suite: netperf \\测试套名称 + category: benchmark \\测试类型：benchmarch|functional|noise-benchmark + ``` + +- 测试脚本和参数（必选） + + - 测试脚本位于```$LKP_SRC/tests```，```$LKP_SRC/daemon```路径下。job yaml 中写入与上述路径中脚本相匹配的文件，将其视为可执行脚本。 + + ``` + $LKP_SRC/setup + $LKP_SRC/monitors + $LKP_SRC/daemon + $LKP_SRC/tests + ``` + - 参数值作为环境变量传递，每个测试脚本将在其文件头的注释中记录可接受的参数。 + + ```yaml + netserver: + netperf: + runtime: 10s + test: + - TCP_STREAM + - UDP_STREAM + send_size: + - 1024 + - 2048 + ``` +- 测试资源相关变量（必选） + + SUT/schduler/部署等资源相关变量。 + ```yaml + testbox: vm-2p8g + os: openeuler + os_version: 20.03 + os_arch: aarch64 + ``` +- 系统设置脚本（可选） + + 位于```$LKP_SRC/setup```路径下。设置脚本将在测试脚本之前运行，主要是用于启停一些依赖性服务或工具，或者配置测试所需的参数等。 + + ```yaml + cgroup2: #$LKP_SRC/setup/cgroup2 executable script + memory.high: 90% + memory.low: 50% + memory.max: max + memory.swap.max: + io.max: + io.weight: + rdma.max: + ``` +- 监控脚本（可选） + + 位于```$LKP_SRC/monitors```路径下。监视器在运行基准测试时可以捕获性能统计数据，对性能分析和回归根源有价值。 + + ```yaml + proc-vmstats: + nterval: 5 + ``` + +**了解更多 job yaml 相关知识** [Job definition to execution](https://gitee.com/wu_fengguang/lkp-tests/blob/master/jobs/README… + + +### 提交测试任务 + +### 查看测试结果 + +### 比较测试结果 + +### 提交 borrow 任务 + +### 提交 bisect 任务 + +## 高级功能 + +### 添加 OS 支持 + +制作一个 initramfs 启动的 cgz 镜像，当系统内核启动时，直接从打包的 cgz 镜像中导出 rootfs，在内存中展开文件系统，利用磁盘的高速缓存机制让系统直接在内存中读写文件，以提升系统的 I/O 性能。 + +#### 操作步骤 + + 1. 获取对应 os 版本的 rootfs（以 openEuler 为例） + - 通过 docker 获取 rootfs + + 1) 下载 openEuler 官方提供的 docker 镜像压缩包 + + ```bash + wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… + ``` + 2) 加载 docker 镜像 + ```bash + docker load -i openEuler-docker.aarch64 + ``` + + 3) 启动 openEuler 容器 + ```bash + docker run -id openeuler-20.03-lts + ``` + 4) 拷贝 docker 的 rootfs + ```bash + docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs + ``` + - 通过 qemu.img(qcow2格式)获取 rootfs (openEuler为例) + + + 1) 下载 openEule r官方网站提供的 qcow2 格式镜像 + ```bash + wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… + ``` + 2) 使用{compass-ci}/container/qcow2rootfs 制作rootfs + ```bash + cd {compass-ci}/container/qcow2rootfs + ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs + ``` + 2. 定制rootfs + 1. 使用chroot命令切换到 rootfs (此步骤需要 root 权限) + ```bash + chroot openEuler-rootfs + ``` + 2. 根据个人需要安装并配置服务 + + a. 修改 root 密码 + b. 配置 ssh 服务 + c. 检查系统时间 + d. 如果使用 docker 制作 osimage 还需要以下操作： + > 1. 安装所需版本内核 + > 2. 从 centos 官方网站下载内核rpm包 + > 3. 使用 yum 进行安装 + > 4. 删除 docker 环境变量文件 + + 3. 退出 rootfs，并打包 + ```bash + cd $rootfs + find . | coip -o -Hnewc |gzip -9 > $os_name.cgz + ``` +#### FAQ +1. 日志报错 “Unable to mount root fs on unknown-block” + - 问题现象 + ```bash + [ 0.390437] List of all partitions: + [ 0.390806] No filesystem could mount root, tried: + [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) + ... + [ 0.399404] Memory Limit: none + [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- + ``` + - 解决方法 + + 1）启动内存不足，增加内存可解决。 + 2）内核文件权限不足，给予 644 权限。 + +2. 系统运行缓慢 + + - 问题现象 + + 打包镜像体积过大，会消耗很大内存 + - 解决方法 + + 建议用户根据具体需要对 rootfs 进行裁剪。 + + +### 添加测试用例 + 参考文档详见：[添加测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/ad… + + +### PKGBUILD 构建 + +1. 概述： + + 使用 PKGBUILD 完成上游项目源代码的编译构建，同一项目可以使用不同 PKGBUILD 完成项目编译构建过程。 + PKGBUILD 相关参考文档：https://wiki.archlinux.org/index.php/PKGBUILD。 + +2. 步骤： + + 1) 注册仓库 + 如果您想在 git push 的时候, 自动触发测试, 那么需要把您的公开 git url 添加到如下仓库 [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos)。 + ```bash + git clone https://gitee.com/wu_fengguang/upstream-repos.git + less upstream-repos/README.md + ``` + + 2) 执行构建测试 + 项目代码更新后自动触发构建任务，无需其他操作。 + + 3) 查看结果 + web: https://compass-ci.openeuler.org/jobs。 + + 4) 备注 + 目前提供 archlinux 下已有 PKGBUILD 项目构建测试，暂不支持自主导入 PKGBUILD 文件。 + PKGBUILD 存放路径： /srv/git/archlinux/*/*/PKGBUILD。 + + +### depends + +- 依赖包构建 + 流程: 以下默认使用 debian:sid 版本进行,如果使用其他OS跑测试用例，则需要在 lkp-tests/distro/adaptation/ 下将包名以键值对的形式写入对应 OS 的文件中 + > 例： `echo 'apache2: httpd' >> lkp-tests/distro/adaptation/openeuler` + + 1) 在lkp-tests/distro/depends/下新建一个与测试用例同名的文件 + `vim lkp-tests/distro/depends/test` + 2) 将测试用例所需debian下的包名填入创建的文件中 + `echo "apache2" >> lkp-tests/distro/depends/test` + 3) 生成lkp-aarch64.cgz + `./crystal-ci/container/lkp-initrd/run` + 4) 提交测试用例 + `submit test.yaml` + 5) 查看结果 + https://compass-ci.openeuler.org/jobs + + + +### 本地搭建 compass-ci 服务器节点 + +在 openEuler 系统一键部署 compass-ci 环境，当前已支持 openEuler-aarch64-20.03-LTS 系统环境，以下配置仅供参考。 + +#### 准备工作 +- 硬件 + 服务器类型：ThaiShan200-2280 (建议) + 架构：aarch64 + 内存：>= 32GB + CPU：64 nuclear (建议) + 硬盘：>= 500G (建议划分独立分区) + + >**说明：** + >划分较小独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… + >划分较大独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… + +- 软件 + OS：openEuler-aarch64-20.03 LTS + git：2.23.0版本 (建议) + 预留空间：>= 300G + 网络：可以访问互联网 + + >**说明：** + >openEuler 系统安装详细操作请参考[添加测试用例](https://openeuler.org/zh/docs/20.03_LTS/docs/Installati… + + +#### 操作指导 +1. 登录 openEuler 系统 + +2. 创建工作目录并设置文件权限 + ```bash + mkdir demo && cd demo && umask 002 + ``` +3. 克隆 compass-ci 项目代码到 demo 目录 + ```bash + git clone https://gitee.com/wu_fengguang/compass-ci.git + ``` +4. 执行一键部署脚本 install-tiny + ```bash + cd compass-ci/sparrow && ./install-tiny + ``` + + +## 呼吁合作 + - 增强 git bisect 能力 + - 增强数据分析能力 + - 增强数据结果可视化能力 + +## 下一步计划 + +1. 优化 git bisect 精准率及效率。 +2. 优化数据分析联动能力，让数据分析更加聚焦。 +3. 数据可视化优化，更加友好展示数据比较结果。 +4. 增强登录认证机制，如：GPG 认证。 +5. 优化部署效率。 -- 2.23.0

1 1

[PATCH compass-ci] lib/job.cr: no need update sshd
by Wu Zhende 01 Dec '20

01 Dec '20

Only need set my_ssh_pubkey, lkp-test/daemon/sshd use my_ssh_pubkey to set authorized_keys. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 6f81172..a4402ab 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -192,16 +192,14 @@ class Job return if @account_info["found"]? == false - update_sshd + set_my_ssh_pubkey end - private def update_sshd + private def set_my_ssh_pubkey sshd = @hash["sshd"].as_h? || Hash(String, JSON::Any).new() pub_key = sshd["pub_key"]?.to_s update_account_my_pub_key(pub_key) - sshd["pub_key"] = @account_info["my_ssh_pubkey"] - @hash["sshd"] = JSON.parse(sshd.to_json) @hash["my_ssh_pubkey"] = @account_info["my_ssh_pubkey"] end -- 2.23.0

1 0

[PATCH compass-ci] doc: introduction of architecture
by Liu Yinsi 01 Dec '20

01 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- doc/introduction-of-architecture.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 doc/introduction-of-architecture.md diff --git a/doc/introduction-of-architecture.md b/doc/introduction-of-architecture.md new file mode 100644 index 0000000..f97ffc6 --- /dev/null +++ b/doc/introduction-of-architecture.md @@ -0,0 +1,23 @@ +# 架构简介 + +![](https://gitee.com/wu_fengguang/compass-ci/raw/master/doc/pictures/compass-ci-architecture.png) + +## 接入层 + +成为衔接 Compass-CI 服务于开发者的纽带，同时提供内部设备管理的平台，提供[web](https://compass-ci.openeuler.org/)门户查阅测试结果、… [环境登录](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E5%… + +## 服务层 + +提供[测试服务](https://gitee.com/wu_fengguang/compass-ci)、[环境登录](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E5%A6%82%E4%BD%95%E7%94%B3%E8%AF%B7%E6%B5%8B%E8%AF%95%E6%9C%BA.md)、[结果分析](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E5%A6%82%E4%BD%95%E6%AF%94%E8%BE%83%E6%B5%8B%E8%AF%95%E7%BB%93%E6%9E%9C.md)、辅助定界、用户[项目注册](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E5%A6%82%E4%BD%95%E4%BD%BF%E7%94%A8compass-ci%E6%B5%8B%E8%AF%95%E5%BC%80%E6%BA%90%E9%A1%B9%E7%9B%AE.md)能力，为本项目的主要功能。 + +## 支撑层 + +作为项目主体功能实现，详细描述资源编排、[部署](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/README.md)、[构建](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E5%A6%82%E4%BD%95%E7%BC%96%E5%86%99PKGBUILD.md)、[测试](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/submit%E5%91%BD%E4%BB%A4%E8%AF%A6%E8%A7%A3.md) 等详细实现。 + +## 数据层 + +为项目提供数据处理服务，包括测试结果格式化及归档。 + +## 资源层 + +提供硬件设备，作为Compass-CI 服务部署的设备以及测试的物理设备、虚拟机、容器资源；提供测试过程中依赖软件仓库及依赖仓库、部署服务器。 -- 2.23.0

1 0

[PATCH compass-ci] doc/manual: fix Invalid hyperlink
by Liu Yinsi 01 Dec '20

01 Dec '20

apply-ssh-account -> apply-account Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- ...\224\263\350\257\267\346\265\213\350\257\225\346\234\272.md" | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git "a/doc/manual/\345\246\202\344\275\225\347\224\263\350\257\267\346\265\213\350\257\225\346\234\272.md" "b/doc/manual/\345\246\202\344\275\225\347\224\263\350\257\267\346\265\213\350\257\225\346\234\272.md" index 470dadb..35fff1d 100644 --- "a/doc/manual/\345\246\202\344\275\225\347\224\263\350\257\267\346\265\213\350\257\225\346\234\272.md" +++ "b/doc/manual/\345\246\202\344\275\225\347\224\263\350\257\267\346\265\213\350\257\225\346\234\272.md" @@ -1,7 +1,7 @@ # 1. 前提条件 ------------- -已经按照[apply-ssh-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-ssh-account.md)完成如下动作： +已经按照[apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md)完成如下动作： - send apply account email. - receive email from compass-ci(a)qq.com. - local environment configuration. -- 2.23.0

1 0

[PATCH compass-ci 2/2] container/es: set es indices
by Liu Yinsi 01 Dec '20

01 Dec '20

even es indices have exist, call set_es_indices just warning "$index already exists" && exit, not have bad influence. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/es/start | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/es/start b/container/es/start index c268a34..e43d01b 100755 --- a/container/es/start +++ b/container/es/start @@ -19,3 +19,5 @@ cmd=( ) "${cmd[@]}" + +set_es_indices -- 2.23.0

1 0

[PATCH compass-ci 1/2] container/defconfig.sh: set es indices
by Liu Yinsi 01 Dec '20

01 Dec '20

when firstly deploy es service, need execute $CCI_SRC/sbin/es-*-mapping.sh after execute es/start, define function set_es_indices() to call, instead of doing this step manually. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/container/defconfig.sh b/container/defconfig.sh index 8a5d08c..410ed75 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -21,3 +21,8 @@ docker_rm() docker stop $container docker rm -f $container } + +set_es_indices() +{ + find $CCI_SRC/sbin/ -name "es-*-mapping.sh" -exec sh {} \; +} -- 2.23.0

1 0

[PATCH compass-ci] README: minor improvements
by Wu Fengguang 01 Dec '20

01 Dec '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- README.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 314a8c6..55b8583 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,8 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上 **测试服务** -使用Compass-CI 基于开源软件 PR 触发[自动化测试](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… job](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/submi… +Compass-CI 监控很多开源软件 git repos，一旦检测到代码更新，会自动触发[自动化测试](https://gitee.com/wu_fengguang/compass-ci/blob… +开发者也可以[手动提交测试 job](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/submi… **调测环境登录** @@ -22,11 +23,12 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上 **测试结果复现** -通过历史测试生成 job.yaml 文件，重新提交 job.yaml 还原测试环境，复现测试结果。 +一次测试运行的所有决定性参数会在 job.yaml 文件中保留完整记录。 +重新提交该 job.yaml 即可在一样的软硬件环境下，重跑同一测试。 **辅助定位** -Compass-CI 可以识别自动化构建测试过程中的错误，触发基于 git tree 的测试，找出引入问题的 commit。 +如果出现新的 error id，就会自动触发bisect，定位引入该 error id 的 commit。 ## Getting started @@ -46,7 +48,7 @@ Compass-CI 可以识别自动化构建测试过程中的错误，触发基于 gi **登录测试环境** -1. 向 Compass-CI(a)qq.com 发送邮件[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual… +1. 向 compass-ci(a)qq.com 发送邮件[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual… 2. 根据邮件反馈内容[完成环境配置](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/… 3. 在测试任务中添加 sshd 字段，提交相应的任务，[登录测试环境](https://gitee.com/wu_fengguang/compass-ci/blob/master/d… -- 2.23.0

1 0

[PATCH compass-ci 3/3] container/logging-es: set available memory size
by Liu Yinsi 01 Dec '20

01 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/logging-es/Dockerfile | 6 ++++-- container/logging-es/build | 18 ++++++++++-------- 2 files changed, 14 insertions(+), 10 deletions(-) diff --git a/container/logging-es/Dockerfile b/container/logging-es/Dockerfile index 6c01c43..97dd81c 100644 --- a/container/logging-es/Dockerfile +++ b/container/logging-es/Dockerfile @@ -4,6 +4,8 @@ ARG BASE_IMAGE FROM $BASE_IMAGE +ARG MEMORY + # docker image borrowed from hub.docker.com/r/gagara/elasticsearch-oss-arm64 MAINTAINER Wu Zhende <wuzhende666(a)163.com> @@ -14,8 +16,8 @@ RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elastic sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && \ sed -i '$a indices.memory.index_buffer_size: 20%' /usr/share/elasticsearch/config/elasticsearch.yml && \ sed -i '$a thread_pool.write.queue_size: 2000' /usr/share/elasticsearch/config/elasticsearch.yml && \ - sed -i "s/-Xms1g/-Xms30g/g" /usr/share/elasticsearch/config/jvm.options && \ - sed -i "s/-Xmx1g/-Xmx30g/g" /usr/share/elasticsearch/config/jvm.options + sed -i "s/-Xms1g/-Xms${MEMORY}g/g" /usr/share/elasticsearch/config/jvm.options && \ + sed -i "s/-Xmx1g/-Xmx${MEMORY}g/g" /usr/share/elasticsearch/config/jvm.options RUN mkdir /usr/share/elasticsearch/tmp && \ chown -R 1090:1090 /usr/share/elasticsearch diff --git a/container/logging-es/build b/container/logging-es/build index 451041b..bd05c9c 100755 --- a/container/logging-es/build +++ b/container/logging-es/build @@ -1,14 +1,16 @@ -#!/bin/bash +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -declare -A BASE_IMAGE_DICT +require_relative '../defconfig.rb' -BASE_IMAGE_DICT=( - ["aarch64"]="gagara/elasticsearch-oss-arm64:7.6.2" - ["x86_64"]="elasticsearch:7.6.2" -) +BASE_IMAGE_DICT = { + 'aarch64' => 'gagara/elasticsearch-oss-arm64:7.6.2', + 'x86_64' => 'elasticsearch:7.6.2'}.freeze -BASE_IMAGE=${BASE_IMAGE_DICT[$(arch)]} +BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] -docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE="$BASE_IMAGE" . +available_memory = set_available_memory + +system "docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} --build-arg MEMORY=#{available_memory} ." -- 2.23.0

1 0

[PATCH compass-ci 2/3] container/es: set available memory size
by Liu Yinsi 01 Dec '20

01 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/es/Dockerfile | 6 ++++-- container/es/build | 8 ++++++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/container/es/Dockerfile b/container/es/Dockerfile index e626dae..c54f3cf 100644 --- a/container/es/Dockerfile +++ b/container/es/Dockerfile @@ -3,6 +3,8 @@ FROM alpine:3.11 +ARG MEMORY + RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk add --no-cache elasticsearch curl @@ -27,8 +29,8 @@ RUN rm -rf /etc/init.d/elasticsearch \ RUN sed -i 's:#path.data\: /path/to/data:path.data\: /srv/es:' /usr/share/es/config/elasticsearch.yml; RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/es/config/elasticsearch.yml; -RUN sed -i "s/-Xms256m/-Xms20g/g" /usr/share/es/config/jvm.options -RUN sed -i "s/-Xmx256m/-Xmx20g/g" /usr/share/es/config/jvm.options +RUN sed -i "s/-Xms256m/-Xms${MEMORY}g/g" /usr/share/es/config/jvm.options +RUN sed -i "s/-Xmx256m/-Xmx${MEMORY}g/g" /usr/share/es/config/jvm.options WORKDIR /usr/share/es diff --git a/container/es/build b/container/es/build index d5607e0..7cc7d33 100755 --- a/container/es/build +++ b/container/es/build @@ -1,6 +1,10 @@ -#!/bin/sh +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -docker build -t es643b:alpine311 . --network=host +require_relative '../defconfig.rb' +available_memory = set_available_memory + +system "docker build -t es643b:alpine311 --build-arg MEMORY=#{availale_memory}. --network=host" -- 2.23.0

1 0

[PATCH v3 compass-ci] compare_matrixes.rb: put compare matrix with more members first
by Li Yuanchao 01 Dec '20

01 Dec '20

As hash table is disordered, the output of compare matrixes sometimes would put one member matrix first, that is not a good comparison. We should put good comparison first. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/compare_matrixes.rb | 33 +++++++++++++++++++++------------ 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index d8112e9..9e4850d 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -383,24 +383,33 @@ end # compare each matrices_list within pre dimension of group matrices # input: group matrices # output: pre compare result of each group +# the result with more comparison objects first def compare_group_matrices(group_matrices, suites_list, options) result_str = '' - group_matrices.each do |k, v| - matrices_list = [] - matrices_titles = [] - v.each do |dim, matrix| - matrices_titles << dim - matrices_list << matrix - end - if options[:no_print] - result_str += compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) - else - print compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) - end + group_matrices_array = sort_by_matrix_size(group_matrices) + group_matrices_array.each do |matrice_kv| + result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_list, options) end result_str end +def get_matrix_str(matrice_key, matrice_value, suites_list, options) + m_list = [] + m_titles = [] + matrice_value.each do |dim, matrix| + m_titles << dim + m_list << matrix + end + return compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) if options[:no_print] + + print compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) +end + +# big size first +def sort_by_matrix_size(group_matrices) + group_matrices.sort { |a, b| b[1].size <=> a[1].size } +end + # input: groups_matrices # { # group_key_1 => { -- 2.23.0

1 0

[PATCH compass-ci 4/4] container: add locate_files in assistant container
by Cao Xueliang 01 Dec '20

01 Dec '20

An interface is provided to support obtaining the real path of the /srv/initrd/deps and pkg program. Example: Input: curl -H 'Content-Type: Application/json' -XPOST '172.17.0.1:8101/locate_files' -d '{"deps_files": ["/srv/initrd/deps/nfs/openeuler/aarch64/20.03/perf/perf.cgz"], "pkg_files": ["/srv/initrd/pkg/nfs/openeuler/aarch64/20.03/plzip/latest.cgz", "/srv/initrd/pkg/nfs/openeuler/aarch64/20.03/stream/latest.cgz"]}' Output: {"deps_files":["/srv/initrd/deps/nfs/openeuler/aarch64/20.03/perf/perf_20201104.cgz"], "pkg_files":["/srv/initrd/pkg/nfs/openeuler/aarch64/20.03/plzip/1-4.cgz", "/srv/initrd/pkg/nfs/openeuler/aarch64/20.03/stream/1-1.cgz"]} Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/routes.rb | 31 +++++++++++++++++++++++ container/assistant/views/locate_files.rb | 21 +++++++++++++++ 2 files changed, 52 insertions(+) create mode 100755 container/assistant/routes.rb create mode 100644 container/assistant/views/locate_files.rb diff --git a/container/assistant/routes.rb b/container/assistant/routes.rb new file mode 100755 index 0000000..6623fd2 --- /dev/null +++ b/container/assistant/routes.rb @@ -0,0 +1,31 @@ +#!/usr/bin/ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'json' +require 'open3' + +require_relative './views/locate_files' + +set :bind, '0.0.0.0' +set :port, 8101 + +post '/locate_files' do + request.body.rewind + + begin + data = JSON.parse request.body.read + rescue JSON::ParserError + return [400, 'parse json params error'] + end + + begin + result = locate_files(data) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + [200, result.to_json] +end diff --git a/container/assistant/views/locate_files.rb b/container/assistant/views/locate_files.rb new file mode 100644 index 0000000..5769177 --- /dev/null +++ b/container/assistant/views/locate_files.rb @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def locate_files(data) + result = {} + + data.each do |key, value| + temp = [] + value.each do |val| + val = val.strip + if File.exist?(val) + temp << File.realpath(val) + end + end + + result.merge!({ "#{key}": temp }) + end + + return result +end -- 2.23.0

1 0

[PATCH compass-ci 3/4] container: start script for assistant container
by Cao Xueliang 01 Dec '20

01 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/start | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100755 container/assistant/start diff --git a/container/assistant/start b/container/assistant/start new file mode 100755 index 0000000..5f7822e --- /dev/null +++ b/container/assistant/start @@ -0,0 +1,21 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +docker_rm assistant + +cmd=( + docker run + --restart=always + --name assistant + -u nobody + -d + -p 8101:8101 + -v /srv/initrd:/srv/initrd + -v /etc/localtime:/etc/localtime:ro + debian:assistant +) + +"${cmd[@]}" -- 2.23.0

1 0

[PATCH compass-ci 2/4] container: build script for assistant container
by Cao Xueliang 01 Dec '20

01 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/assistant/build diff --git a/container/assistant/build b/container/assistant/build new file mode 100755 index 0000000..7778c9f --- /dev/null +++ b/container/assistant/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t debian:assistant . -- 2.23.0

1 0

[PATCH compass-ci 1/4] container: assistant container Dockerfile
by Cao Xueliang 01 Dec '20

01 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/assistant/Dockerfile diff --git a/container/assistant/Dockerfile b/container/assistant/Dockerfile new file mode 100644 index 0000000..0287747 --- /dev/null +++ b/container/assistant/Dockerfile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y ruby-json ruby-sinatra curl + +COPY routes.rb /usr/local/bin/ +COPY views /usr/local/bin/views/ + +CMD ["/usr/local/bin/routes.rb"] -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] compare_matrixes.rb: put compare matrix with more members first
by Li Yuanchao 01 Dec '20

01 Dec '20

As hash table is disordered, the output of compare matrixes sometimes would put one member matrix first, that is not a good comparison. We should put good comparison first. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/compare_matrixes.rb | 33 +++++++++++++++++++++------------ 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index d8112e9..8232513 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -383,24 +383,33 @@ end # compare each matrices_list within pre dimension of group matrices # input: group matrices # output: pre compare result of each group +# the result with more comparison objects first def compare_group_matrices(group_matrices, suites_list, options) result_str = '' - group_matrices.each do |k, v| - matrices_list = [] - matrices_titles = [] - v.each do |dim, matrix| - matrices_titles << dim - matrices_list << matrix - end - if options[:no_print] - result_str += compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) - else - print compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) - end + group_matrices_array = sort_by_matrice_size(group_matrices) + group_matrices_array.each do |matrice_kv| + result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_list, options) end result_str end +def get_matrix_str(matrice_key, matrice_value, suites_list, options) + m_list = [] + m_titles = [] + matrice_value.each do |dim, matrix| + m_titles << dim + m_list << matrix + end + return compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) if options[:no_print] + + print compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) +end + +# big size first +def sort_by_matrice_size(group_matrices) + group_matrices.sort { |a, b| b[1].size <=> a[1].size } +end + # input: groups_matrices # { # group_key_1 => { -- 2.23.0

2 2

[PATCH lkp-tests 1/2] depends/lkp-dev: add package needed in docker
by Wang Yong 01 Dec '20

01 Dec '20

[error] /lkp/lkp/src/sbin/makepkg: line 403: file: command not found Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/depends/lkp-dev | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/lkp-dev b/distro/depends/lkp-dev index 0e49807e8..555124f96 100644 --- a/distro/depends/lkp-dev +++ b/distro/depends/lkp-dev @@ -27,3 +27,4 @@ flex bison libssl-dev libipc-run-perl +file -- 2.23.0

2 2

[PATCH v2 compass-ci 1/2] delete source and add getting started
by Li Ping 01 Dec '20

01 Dec '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- README.md | 37 +++++++++++++++---------------------- 1 file changed, 15 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index 80e0766..128eba0 100644 --- a/README.md +++ b/README.md @@ -10,50 +10,43 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上 - **测试服务** - 使用Compass-CI 基于开源软件 PR 触发[自动化测试](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… +使用Compass-CI 基于开源软件 PR 触发[自动化测试](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… - **调测环境登录** - 使用 SSH [登录测试环境进行调测](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… +使用 SSH [登录测试环境进行调测](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… - **测试结果分析** - 通过 [web](https://compass-ci.openeuler.org) 接口，对历史测试结果进行分析和比较。 +通过 [Web](https://compass-ci.openeuler.org) 接口，对历史测试结果进行分析和比较。 - **测试结果复现** - Compass-CI 把测试过程中的各项环境参数保存在任务结果输出的job.yaml文件中，开发者可以通过重新提交job.yaml复现测试结果。 +Compass-CI 把测试过程中的各项环境参数保存在任务结果输出的job.yaml文件中，开发者可以通过重新提交job.yaml复现测试结果。 - **辅助定位** - Compass-CI 可以识别自动化构建测试过程中的错误，触发基于 git tree 的测试，找出引入问题的commit。 +Compass-CI 可以识别自动化构建测试过程中的错误，触发基于 git tree 的测试，找出引入问题的commit。 -## 使用 Compass-CI -> 您只需要先注册自己的仓库，当您的仓库有 commit 提交时，构建测试会自动执行，并且可在我们的网站中查看结果。 +## Getting started -- 注册自己的仓库 +- **自动化测试** - 如果您想在 `git push` 的时候, 自动触发测试, 那么需要把您的公开 git url 添加到如下仓库 [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos)。 - ```bash - git clone https://gitee.com/wu_fengguang/upstream-repos.git - less upstream-repos/README.md - ``` +1. 添加待测试仓库 URL 到 [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos.git) 仓库，[编写测试用例并添加到仓库](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/…, 详细流程请查看[这篇文档](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/man… -- `git push` - - 更新仓库，自动触发测试。 +2. git push 更新仓库，自动触发测试。 -- 在网页中搜索并查看结果 - - web: https://compass-ci.openeuler.org/jobs +3. 在网页中搜索并查看结果 web: https://compass-ci.openeuler.org/jobs -## Source +- **手动提交测试任务** -我们最新的，最完整的源码都存放在 [Gitee](https://gitee.com/wu_fengguang/compass-ci.git) 上，Fork 我们吧！ +1. [安装Compass-CI客户端](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/… +2. [编写测试用例](https/blob/master/doc/add-testcase.md), [手动提交测试任务](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… +3. 在网页中搜索并查看结果 web: https://compass-ci.openeuler.org/jobs ## Contributing to Compass-CI -我们非常欢迎有新的贡献者，我们也很乐意为我们的贡献者提供一些指导 +我们非常欢迎有新的贡献者，我们也很乐意为我们的贡献者提供一些指导，Compass-CI 主要是使用 Ruby 和 Crystal 开发的一个项目，我们使用 rubocop和 ameba 来规范我d的代码风格，具体说明可以查看 [rococop源码仓](https://github.com/rubocop-hq/rubocop)和[ameba源码仓](https://githu… ## Website -- 2.23.0

2 1