December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] sbin: optimize build-my-info.rb
by Luan Shengde 24 Dec '20

24 Dec '20

mv config part to lib/build_my_info_client.rb lib/build_my_info_client will enable been called to build my_info Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/build-my-info.rb | 42 +++++------------------------------------- 1 file changed, 5 insertions(+), 37 deletions(-) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index d75f767..40193af 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -3,48 +3,16 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -require 'json' -require 'fileutils' require 'io/console' require_relative '../lib/es_client' +require_relative '../lib/build_my_info_client' print 'email: ' my_email = $stdin.echo = gets.chomp print 'name: ' my_name = $stdin.echo = gets.chomp -my_token = %x(uuidgen).chomp +print 'lab: ' +lab = $stdin.echo = gets.chomp -my_info = { - 'my_name' => my_name, - 'my_email' => my_email, - 'my_token' => my_token -} - -def store_account_info(my_info) - es = ESClient.new(index: 'accounts') - es.put_source_by_id(my_info['my_email'], my_info) -end - -def config_default_yaml(my_info) - yaml_dir = "#{ENV['HOME']}/.config/compass-ci/defaults" - FileUtils.mkdir_p yaml_dir unless File.directory? yaml_dir - yaml_file = "#{yaml_dir}/account.yaml" - FileUtils.touch(yaml_file) unless File.exist? yaml_file - - yaml_info = YAML.load_file(yaml_file) || {} - yaml_info.update my_info - - File.open(yaml_file, 'w') do |f| - f.puts yaml_info.to_yaml - end -end - -def complete_my_info(my_info) - my_info['my_login_name'] = '' - my_info['my_commit_url'] = '' - my_info['my_ssh_pubkey'] = [] -end - -config_default_yaml(my_info) -complete_my_info(my_info) -store_account_info(my_info) +build_my_info = BuildMyInfo.new(my_email, my_name, lab) +build_my_info.config_my_info -- 2.23.0

1 0

[PATCH lkp-tests] fix: extra empty line
by Ren Wen 24 Dec '20

24 Dec '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- tests/host-info | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/host-info b/tests/host-info index 0ab8e6c97..e0c866257 100755 --- a/tests/host-info +++ b/tests/host-info @@ -106,7 +106,6 @@ get_ipmi_ip() is_docker && return local ip=$(ipmitool lan print | awk 'NR==3{print $4}') [ -n "$ip" ] && echo "ipmi_ip: $ip" - } main() -- 2.23.0

2 2

[PATCH lkp-tests] tests: fix return value
by Zhang Yu 24 Dec '20

24 Dec '20

[why] if /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0 both not exit, the value of return should be 0 not 1 Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/host-info b/tests/host-info index 3984727b4..fe55d1ff2 100755 --- a/tests/host-info +++ b/tests/host-info @@ -102,7 +102,7 @@ get_ipmi_ip() # 802.1q VLAN ID : Disabled # RMCP+ Cipher Suites : 0,1,2,3,17 # Cipher Suite Priv Max : XuuaXXXXXXXXXXX - [ -e /dev/ipmi0 ] || [ -e /dev/ipmi/0 ] || [ -e /dev/ipmidev/0 ] || return + [ -e /dev/ipmi0 ] || [ -e /dev/ipmi/0 ] || [ -e /dev/ipmidev/0 ] || return 0 local ip=$(ipmitool lan print | awk 'NR==3{print $4}') [ -n "$ip" ] && echo "ipmi_ip: $ip" } -- 2.23.0

3 4

[PATCH v3 compass-ci] doc/tutorial.md: fix abnormal display in Markdown
by Lu Kaiyi 24 Dec '20

24 Dec '20

the middle part in * and * is italics for markdown document, if want show '*' in this condition, add ` in both sides of path. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- doc/tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..bf72630 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -258,7 +258,7 @@ job yaml ��[YAML](http://yaml.org/YAM 4) �� archlinux �� PKGBUILD �� PKGBUILD �� - PKGBUILD �� /srv/git/archlinux/*/*/PKGBUILD�� + PKGBUILD �� `/srv/git/archlinux/*/*/PKGBUILD`�� ### depends -- 2.23.0

1 0

[PATCH compass-ci] lib: add build_my_info_client
by Luan Shengde 24 Dec '20

24 Dec '20

purpose: for user build their infos when installing compassci - store my_info to ES - write my_info to default/lab yaml file if not, job will fail for verification problems for user private environment Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/build_my_info_client.rb | 79 +++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100755 lib/build_my_info_client.rb diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb new file mode 100755 index 0000000..9e44c00 --- /dev/null +++ b/lib/build_my_info_client.rb @@ -0,0 +1,79 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'fileutils' +require 'io/console' +require_relative '../lib/es_client' + +# purpose: +# build my_info +# store my_info to ES +# write my_info to default/lab yaml file +# for user install compass-ci at their local server +# user meet verification problems when submit jobs +# add calling in installing script to meet the verification function +# usage: +# require_relative 'build_my_info_client' +# +# build_my_info = BuildMyInfo.new(email, name, lab) +# build_user_info.config_my_info +class BuildMyInfo + def initialize(my_email, my_name, lab) + @lab = lab || 'z9' + @my_info = { + 'my_email' => my_email, + 'my_name' => my_name, + 'my_token' => %x(uuidgen).chomp + } + end + + def config_default_yaml + default_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/defaults" + FileUtils.mkdir_p default_yaml_dir unless File.directory? default_yaml_dir + default_yaml_file = "#{default_yaml_dir}/account.yaml" + FileUtils.touch(default_yaml_file) unless File.exist? default_yaml_file + + default_yaml_info = YAML.load_file(default_yaml_file) || {} + default_yaml_info['my_email'] = @my_info['my_email'] + default_yaml_info['my_name'] = @my_info['my_name'] + + File.open(default_yaml_file, 'w') do |f| + f.puts default_yaml_info.to_yaml + end + end + + def config_lab_yaml + lab_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/include/lab" + FileUtils.mkdir_p lab_yaml_dir unless File.directory? lab_yaml_dir + lab_yaml_file = "#{lab_yaml_dir}/#{(a)lab}.yaml" + FileUtils.touch(lab_yaml_file) unless File.exist? lab_yaml_file + + lab_yaml_info = YAML.load_file(lab_yaml_file) || {} + lab_yaml_info['my_token'] = @my_info['my_token'] + + File.open(lab_yaml_file, 'w') do |f| + f.puts lab_yaml_info.to_yaml + end + end + + def complete_my_info + @my_info['my_login_name'] = '' + @my_info['my_commit_url'] = '' + @my_info['my_ssh_pubkey'] = [] + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def config_my_info + config_default_yaml + config_lab_yaml + complete_my_info + store_account_info + end +end -- 2.23.0

1 0

[PATCH lkp-tests] fix shebang in crystal script
by Lin Jiaxin 24 Dec '20

24 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/constant.cr | 2 +- lib/dmesg.cr | 2 +- lib/run_env.cr | 2 +- lib/string_ext.cr | 2 +- lib/yaml.cr | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/constant.cr b/lib/constant.cr index b5adcf9d..c17a0f36 100644 --- a/lib/constant.cr +++ b/lib/constant.cr @@ -1,4 +1,4 @@ -#!/usr/bin/env ruby +#!/usr/bin/env crystal require "./run_env" diff --git a/lib/dmesg.cr b/lib/dmesg.cr index 9cbe6751..25834e4b 100644 --- a/lib/dmesg.cr +++ b/lib/dmesg.cr @@ -1,4 +1,4 @@ -#!/usr/bin/env ruby +#!/usr/bin/env crystal LKP_SRC = ENV["LKP_SRC"] || File.dirname(__DIR__) diff --git a/lib/run_env.cr b/lib/run_env.cr index cdccaf98..af0ec39b 100644 --- a/lib/run_env.cr +++ b/lib/run_env.cr @@ -1,4 +1,4 @@ -#!/usr/bin/env ruby +#!/usr/bin/env crystal LOCAL_RUN_ENV = "LKP_LOCAL_RUN".freeze require "yaml" diff --git a/lib/string_ext.cr b/lib/string_ext.cr index 41fcaf1a..398ab7e0 100644 --- a/lib/string_ext.cr +++ b/lib/string_ext.cr @@ -1,4 +1,4 @@ -#!/usr/bin/env ruby +#!/usr/bin/env crystal REGEX_ANSI_COLOR = /\e\[([0-9;]+m|[mK])/.freeze diff --git a/lib/yaml.cr b/lib/yaml.cr index b9fa1a94..19691d1d 100644 --- a/lib/yaml.cr +++ b/lib/yaml.cr @@ -1,4 +1,4 @@ -#!/usr/bin/env ruby +#!/usr/bin/env crystal require "./common" require "./log" -- 2.23.0

2 2

[PATCH v7 compass-ci] tutorial.md: update initramfs tutorial.md
by Wang Chenglong 24 Dec '20

24 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/tutorial.md | 80 +++++++++++++------------------------------------ 1 file changed, 21 insertions(+), 59 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..b8207ba 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -150,78 +150,40 @@ job yaml ��[YAML](http://yaml.org/YAM #### �� - 1. �� os �� rootfs�� openEuler �� - - �� docker �� rootfs - - 1) �� openEuler �� docker �� - - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) �� docker �� - ```bash - docker load -i openEuler-docker.aarch64 - ``` - - 3) �� openEuler �� - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) �� docker �� rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - - �� qemu.img(qcow2��)�� rootfs (openEuler��) - - - 1) �� openEule r�� qcow2 �� - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… - ``` - 2) ��{compass-ci}/container/qcow2rootfs ��rootfs - ```bash - cd {compass-ci}/container/qcow2rootfs - ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs - ``` - 2. ��rootfs - 1. ��chroot�� rootfs (�� root ��) - ```bash - chroot openEuler-rootfs - ``` - 2. �� - - a. �� root �� - b. �� ssh �� - c. �� - d. �� docker �� osimage �� - > 1. �� - > 2. �� centos ��rpm�� - > 3. �� yum �� - > 4. �� docker �� - - 3. �� rootfs�� - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` + 1. �� openEuler �� + ```bash + cd {compass-ci}/rootfs/initramfs/openeuler/aarch64/20.03-LTS/ + ./run + ``` + + 2. ��/�� + �� {compass-ci} ��rootfs�� rootfs ��/�� + ```bash + mkdir rootfs + cd rootfs + zcat ../$os_name.cgz | cpio -idmv + find ./lib/modules/* | cpio -o -Hnewc | gzip -9 > $HOME/modules-$os_name.cgz + find ./usr/src/kernels/* | cpio -o -Hnewc | gzip -9 > $HOME/headers-$os_name.cgz + ``` + #### FAQ -1. �� Unable to mount root fs on unknown-block�� +1. �� Unable to mount root fs on unknown-block�� - �� ```bash [ 0.390437] List of all partitions: - [ 0.390806] No filesystem could mount root, tried: + [ 0.390806] No filesystem could mount root, tried: [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ... [ 0.399404] Memory Limit: none [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- ``` - �� - + 1�� 2�� 644 �� - + 2. �� - + - �� -- 2.23.0

1 0

[PATCH v3 compass-ci] fix: qcow2rootfs/bin/common: unable to understand vmlinux
by Wang Chenglong 24 Dec '20

24 Dec '20

[error] [WARNNING] Restoring the owner of qcow2 file Finding vmlinuz under /tmp/a/boot ... /c/compass-ci/container/qcow2rootfs/bin/common: line 135: ROOTFS_VMLINUZ_FILE: unbound variable [why] some kernels are called vmlinx instead of vmlinz. --- container/qcow2rootfs/bin/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/qcow2rootfs/bin/common b/container/qcow2rootfs/bin/common index da39f52..baf936c 100755 --- a/container/qcow2rootfs/bin/common +++ b/container/qcow2rootfs/bin/common @@ -127,7 +127,7 @@ get_rootfs_kernel() { exit 5 } local vmlinuz_file vmlinuz kernel - vmlinuz_file=$(find ./boot -name "vmlinuz-*" |grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz_file=$(find ./boot -name "vmlinu[z|x]-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel -- 2.23.0

2 2

[PATCH v5 compass-ci] docker-rootf: add different install methods
by Wang Chenglong 24 Dec '20

24 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image index 787a42c..ffe4937 100755 --- a/container/docker-rootfs/setup-image +++ b/container/docker-rootfs/setup-image @@ -4,8 +4,15 @@ install_packages() { - yum install -y --skip-broken $(</tmp/packages-to-install) - yum clean all + if [ $(command -v yum) >/dev/null ]; then + yum install -y --skip-broken $(</tmp/packages-to-install) + yum clean all + elif [ $(command -v apt-get) >/dev/null ]; then + apt update + apt-get install -y --fix-missing $(</tmp/packages-to-install) + elif [ $(command -v pacman) >/dev/null ]; then + pacman -Syu --noconfirm $(</tmp/packages-to-install) + fi } remove_file() -- 2.23.0

2 2

[PATCH v4 compass-ci] docker-rootf: add different install methods
by Wang Chenglong 24 Dec '20

24 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image index 787a42c..05e5a24 100755 --- a/container/docker-rootfs/setup-image +++ b/container/docker-rootfs/setup-image @@ -4,8 +4,15 @@ install_packages() { - yum install -y --skip-broken $(</tmp/packages-to-install) - yum clean all + if [ $(command -v yum) >/dev/null ]; then + yum install -y --skip-broken $(</tmp/packages-to-install) + yum clean all + elif [ $(command -v apt-get) >/dev/null ]; then + apt update + apt-get install -y --fix-missing $(</tmp/packages-to-install) + elif [ $(command -v pacman) >/dev/null ]; then + pacman -Sy --noconfirm $(</tmp/packages-to-install) + fi } remove_file() -- 2.23.0

1 1

[PATCH v2 lkp-tests] lib/job.rb: fix job symbol key check
by Wei Jihui 24 Dec '20

24 Dec '20

load job from yaml maybe has symbol key symbol key does not has start_with function. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 7553312e..35e0ceee 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -254,7 +254,6 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end - hash.delete_if { |key, _| key.start_with?('#!') } hash.merge!(@overrides) @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end @@ -290,6 +289,12 @@ class Job jobs_array = [] hash.each { |key, value| next unless key =~ /^\w.*\|.*\w$/ + next unless key.is_a?(String) + + if key.start_with?('#! ') + hash.delete(key) + next + end key_array = key.split('|').map(&:strip) [value].flatten.each do |v| -- 2.23.0

3 3

[PATCH v2 lkp-tests] jobs: optimize yaml file
by Liu Shaofei 23 Dec '20

23 Dec '20

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- jobs/build-pkg.yaml | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/jobs/build-pkg.yaml b/jobs/build-pkg.yaml index b17fd2ade..7a47bd724 100644 --- a/jobs/build-pkg.yaml +++ b/jobs/build-pkg.yaml @@ -1,9 +1,8 @@ -suite: build-pkg -category: functional - -# pkgbuild_repo examples +# the field of "pkgbuild_repo" refers to the path of PKGBUILD, the examples are as follow: # pkgbuild_repo: archlinux/aur-j/java8-openjdk # pkgbuild_repo: archlinux/community//keepalived/trunk # pkgbuild_repo: archlinux/packages//prometheus-node-exporter/trunk +suite: build-pkg +category: functional build-pkg: -- 2.23.0

1 0

[PATCH compass-ci] container/sub-fluentd: the dependency file is missing
by Hu Xuejiao 23 Dec '20

23 Dec '20

[why] container is failed to start in test-machine because of missing file Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- container/sub-fluentd/build-depends | 1 + 1 file changed, 1 insertion(+) create mode 100755 container/sub-fluentd/build-depends diff --git a/container/sub-fluentd/build-depends b/container/sub-fluentd/build-depends new file mode 100755 index 0000000..29ee9d3 --- /dev/null +++ b/container/sub-fluentd/build-depends @@ -0,0 +1 @@ +fluentd-base -- 2.23.0

1 0

[PATCH lkp-tests] sbin/run_job: Automatic task execution
by Hu Xuejiao 23 Dec '20

23 Dec '20

[purpose] Verify that a local job can be consumed properly Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100755 sbin/run_job diff --git a/sbin/run_job b/sbin/run_job new file mode 100755 index 00000000..2b02eb77 --- /dev/null +++ b/sbin/run_job @@ -0,0 +1,76 @@ +#!/bin/bash +# tbox_group +# testbox +# nr_vm +# nr_container +# queues + +cci_makepkg_file="$LKP_SRC/jobs/cci-makepkg.yaml" +grep "atomic" $cci_makepkg_file || sed -i 's|benchmark:|& atomic|g' $cci_makepkg_file + +submit_one_job() +{ + cmd=( + submit -m $yaml + os=${single_arg[0]} + os_arch=${single_arg[1]} + os_version=${single_arg[2]} + os_mount=${single_arg[3]} + testbox=$testbox + queue="$tbox_group.$HOSTNAME" + ) + output=$("${cmd[@]}") + echo "$output" | grep -o -E 'job_state=finished[^\"]*' +} + +submit_job() +{ + yaml_args=("$cci_makepkg_file" "host-info.yaml") + for yaml in "${yaml_args[@]}" + do + submit_one_job & + done +} + +run_qemu() +{ + local nr_vm=4 + local testbox="vm-2p8g" + local tbox_group="vm-2p8g" + local queues="$tbox_group.$HOSTNAME" + + . $LKP_SRC/daemon/multi-qemu + + qemu_args=( + "openeuler aarch64 20.03 cifs" + "centos aarch64 7.6.1810 initramfs" + ) + for qemu_arg in "${qemu_args[@]}" + do + local single_arg=($(echo $qemu_arg)) + submit_job + done +} + +run_docker() +{ + local nr_container=2 + local tbox_group="dc-8g" + local testbox="dc-8g" + local queues="$tbox_group.$HOSTNAME" + + . $LKP_SRC/daemon/multi-docker + + docker_args=( + "centos aarch64 7.6.1810 container" + ) + for docker_arg in "${docker_args[@]}" + do + local single_arg=($(echo $docker_arg)) + submit_job + done +} + +run_qemu +run_docker +wait -- 2.23.0

1 0

[PATCH v3 compass-ci] docker-rootf: add different install methods
by Wang Chenglong 23 Dec '20

23 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image index 787a42c..05e5a24 100755 --- a/container/docker-rootfs/setup-image +++ b/container/docker-rootfs/setup-image @@ -4,8 +4,15 @@ install_packages() { - yum install -y --skip-broken $(</tmp/packages-to-install) - yum clean all + if [ $(command -v yum) >/dev/null ]; then + yum install -y --skip-broken $(</tmp/packages-to-install) + yum clean all + elif [ $(command -v apt-get) >/dev/null ]; then + apt update + apt-get install -y --fix-missing $(</tmp/packages-to-install) + elif [ $(command -v pacman) >/dev/null ]; then + pacman -Sy --noconfirm $(</tmp/packages-to-install) + fi } remove_file() -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: fix job symbol key check
by Wei Jihui 23 Dec '20

23 Dec '20

load job from yaml maybe has symbol key symbol key does not has start_with function. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 7553312e..5ae67e6b 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -254,7 +254,6 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end - hash.delete_if { |key, _| key.start_with?('#!') } hash.merge!(@overrides) @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end @@ -289,7 +288,12 @@ class Job def multi_args(hash) jobs_array = [] hash.each { |key, value| - next unless key =~ /^\w.*\|.*\w$/ + next unless key =~ /^\w.*\|.*\w$/ || key.is_a?(String) + + if key.start_with?('#! ') + hash.delete(key) + next + end key_array = key.split('|').map(&:strip) [value].flatten.each do |v| -- 2.23.0

1 0

[PATCH v2 compass-ci] docker-rootf: add different install methods
by Wang Chenglong 23 Dec '20

23 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image index 787a42c..7274e2e 100755 --- a/container/docker-rootfs/setup-image +++ b/container/docker-rootfs/setup-image @@ -4,8 +4,15 @@ install_packages() { - yum install -y --skip-broken $(</tmp/packages-to-install) - yum clean all + if [ $(command -v yum) ]; then + yum install -y --skip-broken $(</tmp/packages-to-install) + yum clean all + elif [ $(command -v apt-get) ]; then + apt update + apt-get install -y --fix-missing $(</tmp/packages-to-install) + elif [ $(command -v pacman) ]; then + pacman -Sy --noconfirm $(</tmp/packages-to-install) + fi } remove_file() -- 2.23.0

1 0

[PATCH compass-ci] docker-rootf: add different install methods
by Wang Chenglong 23 Dec '20

23 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image index 787a42c..6a121d3 100755 --- a/container/docker-rootfs/setup-image +++ b/container/docker-rootfs/setup-image @@ -4,8 +4,15 @@ install_packages() { - yum install -y --skip-broken $(</tmp/packages-to-install) - yum clean all + if [ $(command -v yum) ]; then + yum install -y --skip-broken $(</tmp/packages-to-install) + yum clean all + elif [ $(command -v apt-get) ]; then + apt update + apt-get install -y --no-install-recommends $(</tmp/packages-to-install) + elif [ $(command -v pacman) ]; then + pacman -Sy --noconfirm $(</tmp/packages-to-install) + fi } remove_file() -- 2.23.0

2 2

[PATCH lkp-tests] lkp-tests: delete useless file
by Wei Jihui 23 Dec '20

23 Dec '20

install.sh is useless now install method is: make install && source ~/.${SHELL##*/}rc Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- install.sh | 7 ------- 1 file changed, 7 deletions(-) delete mode 100644 install.sh diff --git a/install.sh b/install.sh deleted file mode 100644 index cea1654f..00000000 --- a/install.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/bin/bash - -. $LKP_SRC/lib/env.sh - -make - -source $(shell_profile) -- 2.23.0

1 0

[PATCH compass-ci] scheduler: fix block error
by Ren Wen 23 Dec '20

23 Dec '20

[why] Every request to scheduler will initialize a new 'Sched', each 'Sched' will new a 'BlockHelper'. For each node in cluster, it can't release when blocking at scheduler because they use different locks. [how] Set an instance of 'BlockHelper' to class variable. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 3 ++- src/scheduler/request_cluster_state.cr | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f7220f9..25e3708 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -28,11 +28,12 @@ class Sched property redis property block_helper + @@block_helper = BlockHelper.new + def initialize(env : HTTP::Server::Context) @es = Elasticsearch::Client.new @redis = Redis::Client.new @task_queue = TaskQueueAPI.new - @block_helper = BlockHelper.new @rgc = RemoteGitClient.new @env = env @log = env.log.as(JSONLogger) diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index 1782797..0703792 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -25,7 +25,7 @@ class Sched update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) when "wait_ready" update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) - @block_helper.block_until_finished(cluster_id) { + @@block_helper.block_until_finished(cluster_id) { cluster_state = sync_cluster_state(cluster_id, job_id, states[request_state]) cluster_state == "ready" || cluster_state == "abort" } -- 2.23.0

1 0

[PATCH lkp-tests] Makefile: delete set-env in Makefile
by Wei Jihui 23 Dec '20

23 Dec '20

set-env is move to install-dependencies.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- Makefile | 1 - 1 file changed, 1 deletion(-) diff --git a/Makefile b/Makefile index 0b2d09a2..236bea25 100644 --- a/Makefile +++ b/Makefile @@ -14,7 +14,6 @@ subsystem: install: mkdir -p $(TARGET_DIR_BIN) ln -sf $(shell pwd)/bin/lkp $(TARGET_DIR_BIN)/lkp - bash sbin/set-env.sh bash sbin/install-dependencies.sh .PHONY: doc -- 2.23.0

1 0

[PATCH compass-ci] service/scheduler: capturing exceptions
by Wu Zhende 23 Dec '20

23 Dec '20

For API-level errors, use warn. For service-level errors, use error. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 14 ++++++++++++++ src/scheduler.cr | 9 ++++++++- src/scheduler/close_job.cr | 2 ++ src/scheduler/download_file.cr | 2 ++ src/scheduler/find_job_boot.cr | 2 ++ src/scheduler/find_next_job_boot.cr | 2 ++ src/scheduler/request_cluster_state.cr | 2 ++ src/scheduler/update_job_parameter.cr | 2 ++ 8 files changed, 34 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f7220f9..be8cac5 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -45,6 +45,8 @@ class Sched def alive(version) debug_message("Env= {\n#{`export`}}") "LKP Alive! The time is #{Time.local}, version = #{version}" + rescue e + @log.warn(e) end def normalize_mac(mac : String) @@ -59,6 +61,8 @@ class Sched else "No yet!" end + rescue e + @log.warn(e) end def del_host_mac @@ -69,6 +73,8 @@ class Sched else "No yet!" end + rescue e + @log.warn(e) end def set_host2queues @@ -79,6 +85,8 @@ class Sched else "No yet!" end + rescue e + @log.warn(e) end def del_host2queues @@ -89,6 +97,8 @@ class Sched else "No yet!" end + rescue e + @log.warn(e) end def update_tbox_wtmp @@ -118,6 +128,8 @@ class Sched # json log hash["testbox"] = testbox @log.info(hash.to_json) + rescue e + @log.warn(e) end def report_ssh_port @@ -130,6 +142,8 @@ class Sched end @log.info(%({"job_id": "#{job_id}", "state": "set ssh port", "ssh_port": "#{ssh_port}", "tbox_name": "#{testbox}"})) + rescue e + @log.warn(e) end private def query_consumable_keys(shortest_queue_name) diff --git a/src/scheduler.cr b/src/scheduler.cr index 41702e7..07b38a3 100644 --- a/src/scheduler.cr +++ b/src/scheduler.cr @@ -3,7 +3,14 @@ require "scheduler/scheduler" require "./scheduler/constants.cr" +require "./lib/json_logger" module Scheduler - Kemal.run(SCHED_PORT) + log = JSONLogger.new + + begin + Kemal.run(SCHED_PORT) + rescue e + log.error(e) + end end diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index d8f80d7..9b4da6a 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -24,5 +24,7 @@ class Sched @redis.remove_finished_job(job_id) @log.info(%({"job_id": "#{job_id}", "job_state": "complete"})) + rescue e + @log.warn(e) end end diff --git a/src/scheduler/download_file.cr b/src/scheduler/download_file.cr index e1f0c9a..50e27ba 100644 --- a/src/scheduler/download_file.cr +++ b/src/scheduler/download_file.cr @@ -10,5 +10,7 @@ class Sched @log.info(%({"job_id": "#{job_id}", "job_state": "download"})) send_file @env, file_path + rescue e + @log.warn(e) end end diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 9b3db88..5f71b32 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -21,6 +21,8 @@ class Sched @log.info(%({"job_id": "#{job_id}", "job_state": "boot"})) if job_id response + rescue e + @log.warn(e) end # auto submit a job to collect the host information diff --git a/src/scheduler/find_next_job_boot.cr b/src/scheduler/find_next_job_boot.cr index 0bdfe5b..091c41a 100644 --- a/src/scheduler/find_next_job_boot.cr +++ b/src/scheduler/find_next_job_boot.cr @@ -14,5 +14,7 @@ class Sched @log.info(%({"job_id": "#{job_id}", "job_state": "boot"})) if job_id response + rescue e + @log.warn(e) end end diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index 1782797..06e3329 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -68,6 +68,8 @@ class Sched # show cluster state return @redis.hash_get("sched/cluster_state", cluster_id) + rescue e + @log.warn(e) end # node_state: "finish" | "ready" diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index da528c1..744274a 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -30,5 +30,7 @@ class Sched log = job_content.dup log["job_id"] = log.delete("id").not_nil! @log.info(log.to_json) + rescue e + @log.warn(e) end end -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/tutorial.md: fix abnormal display in Markdown
by Lu Kaiyi 23 Dec '20

23 Dec '20

the middle part in * and * is italics for markdown document, if want show '*' in this condition, add ` in both sides of path. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- doc/tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..bf72630 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -258,7 +258,7 @@ job yaml ��[YAML](http://yaml.org/YAM 4) �� archlinux �� PKGBUILD �� PKGBUILD �� - PKGBUILD �� /srv/git/archlinux/*/*/PKGBUILD�� + PKGBUILD �� `/srv/git/archlinux/*/*/PKGBUILD`�� ### depends -- 2.23.0

1 0

[PATCH v6 compass-ci] tutorial.md: update initramfs tutorial.md
by Wang Chenglong 23 Dec '20

23 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/tutorial.md | 80 +++++++++++++------------------------------------ 1 file changed, 21 insertions(+), 59 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..b8207ba 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -150,78 +150,40 @@ job yaml ��[YAML](http://yaml.org/YAM #### �� - 1. �� os �� rootfs�� openEuler �� - - �� docker �� rootfs - - 1) �� openEuler �� docker �� - - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) �� docker �� - ```bash - docker load -i openEuler-docker.aarch64 - ``` - - 3) �� openEuler �� - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) �� docker �� rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - - �� qemu.img(qcow2��)�� rootfs (openEuler��) - - - 1) �� openEule r�� qcow2 �� - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… - ``` - 2) ��{compass-ci}/container/qcow2rootfs ��rootfs - ```bash - cd {compass-ci}/container/qcow2rootfs - ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs - ``` - 2. ��rootfs - 1. ��chroot�� rootfs (�� root ��) - ```bash - chroot openEuler-rootfs - ``` - 2. �� - - a. �� root �� - b. �� ssh �� - c. �� - d. �� docker �� osimage �� - > 1. �� - > 2. �� centos ��rpm�� - > 3. �� yum �� - > 4. �� docker �� - - 3. �� rootfs�� - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` + 1. �� openEuler �� + ```bash + cd {compass-ci}/rootfs/initramfs/openeuler/aarch64/20.03-LTS/ + ./run + ``` + + 2. ��/�� + �� {compass-ci} ��rootfs�� rootfs ��/�� + ```bash + mkdir rootfs + cd rootfs + zcat ../$os_name.cgz | cpio -idmv + find ./lib/modules/* | cpio -o -Hnewc | gzip -9 > $HOME/modules-$os_name.cgz + find ./usr/src/kernels/* | cpio -o -Hnewc | gzip -9 > $HOME/headers-$os_name.cgz + ``` + #### FAQ -1. �� Unable to mount root fs on unknown-block�� +1. �� Unable to mount root fs on unknown-block�� - �� ```bash [ 0.390437] List of all partitions: - [ 0.390806] No filesystem could mount root, tried: + [ 0.390806] No filesystem could mount root, tried: [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ... [ 0.399404] Memory Limit: none [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- ``` - �� - + 1�� 2�� 644 �� - + 2. �� - + - �� -- 2.23.0

1 0

[PATCH v2 compass-ci] fix: qcow2rootfs/bin/common: unable to understand vmlinux
by Wang Chenglong 23 Dec '20

23 Dec '20

[error] [WARNNING] Restoring the owner of qcow2 file Finding vmlinuz under /tmp/a/boot ... /c/compass-ci/container/qcow2rootfs/bin/common: line 135: ROOTFS_VMLINUZ_FILE: unbound variable [why] some kernels are called vmlinx instead of vmlinz. --- container/qcow2rootfs/bin/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/qcow2rootfs/bin/common b/container/qcow2rootfs/bin/common index da39f52..baf936c 100755 --- a/container/qcow2rootfs/bin/common +++ b/container/qcow2rootfs/bin/common @@ -127,7 +127,7 @@ get_rootfs_kernel() { exit 5 } local vmlinuz_file vmlinuz kernel - vmlinuz_file=$(find ./boot -name "vmlinuz-*" |grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz_file=$(find ./boot -name "vmlinu[z|x]-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel -- 2.23.0

1 0

[PATCH compass-ci] fix: qcow2rootfs/bin/common: unable to understand vmlinux
by Wang Chenglong 23 Dec '20

23 Dec '20

[error] [WARNNING] Restoring the owner of qcow2 file Finding vmlinuz under /tmp/a/boot ... /c/compass-ci/container/qcow2rootfs/bin/common: line 135: ROOTFS_VMLINUZ_FILE: unbound variable [why] some kernels are called vmlinx instead of vmlinz. --- container/qcow2rootfs/bin/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/qcow2rootfs/bin/common b/container/qcow2rootfs/bin/common index da39f52..336602e 100755 --- a/container/qcow2rootfs/bin/common +++ b/container/qcow2rootfs/bin/common @@ -127,7 +127,7 @@ get_rootfs_kernel() { exit 5 } local vmlinuz_file vmlinuz kernel - vmlinuz_file=$(find ./boot -name "vmlinuz-*" |grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz_file=$(find ./boot -name "vmlinu[z|x]-*" |grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel -- 2.23.0

2 2

Re: [PATCH lkp-tests] Makefile: delete set-env in makefile
by Liu Shaofei 22 Dec '20

22 Dec '20

Reply-To: In-Reply-To: <20201222125413.2442750-1-weijihuiall(a)163.com> The Titel: makefile => Makefile? Thanks, Liushaofei On Tue, Dec 22, 2020 at 08:54:13PM +0800, Wei Jihui wrote: >set-env is move to install-dependencies.sh > >Signed-off-by: Wei Jihui <weijihuiall(a)163.com> >--- > Makefile | 1 - > 1 file changed, 1 deletion(-) > >diff --git a/Makefile b/Makefile >index 0b2d09a2..236bea25 100644 >--- a/Makefile >+++ b/Makefile >@@ -14,7 +14,6 @@ subsystem: > install: > mkdir -p $(TARGET_DIR_BIN) > ln -sf $(shell pwd)/bin/lkp $(TARGET_DIR_BIN)/lkp >- bash sbin/set-env.sh > bash sbin/install-dependencies.sh > > .PHONY: doc >-- >2.23.0 >

1 0

[PATCH compass-ci] result-webdav: if judgment error and some file symlinks are not created
by Cui Yili 22 Dec '20

22 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index f077fca..1feddd4 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -46,7 +46,7 @@ http { link_name = "latest" else tmp_name = string.match(upload_file, "(.*)_%d+%.cgz") - if not tmp_name then + if tmp_name then link_name = tmp_name else return -- 2.23.0

2 1

[PATCH lkp-tests] Makefile: delete set-env in makefile
by Wei Jihui 22 Dec '20

22 Dec '20

set-env is move to install-dependencies.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- Makefile | 1 - 1 file changed, 1 deletion(-) diff --git a/Makefile b/Makefile index 0b2d09a2..236bea25 100644 --- a/Makefile +++ b/Makefile @@ -14,7 +14,6 @@ subsystem: install: mkdir -p $(TARGET_DIR_BIN) ln -sf $(shell pwd)/bin/lkp $(TARGET_DIR_BIN)/lkp - bash sbin/set-env.sh bash sbin/install-dependencies.sh .PHONY: doc -- 2.23.0

1 0

[PATCH lkp-tests] jobs: fix to report errors when submit jobs
by Liu Shaofei 22 Dec '20

22 Dec '20

[why] /home/liushaofei/lkp-tests/lib/job.rb:257:in `block (2 levels) in load': undefined method `start_with?' for :"# pkgbuild_repo examples":Symbol (NoMethodError) ERROR -- /home/liushaofei/lkp-tests/jobs/build-pkg.yaml: undefined method `start_with?' for :"# pkgbuild_repo examples":Symbol Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- jobs/build-pkg.yaml | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/jobs/build-pkg.yaml b/jobs/build-pkg.yaml index b17fd2ade..7a47bd724 100644 --- a/jobs/build-pkg.yaml +++ b/jobs/build-pkg.yaml @@ -1,9 +1,8 @@ -suite: build-pkg -category: functional - -# pkgbuild_repo examples +# the field of "pkgbuild_repo" refers to the path of PKGBUILD, the examples are as follow: # pkgbuild_repo: archlinux/aur-j/java8-openjdk # pkgbuild_repo: archlinux/community//keepalived/trunk # pkgbuild_repo: archlinux/packages//prometheus-node-exporter/trunk +suite: build-pkg +category: functional build-pkg: -- 2.23.0

1 0

[PATCH compass-ci] sparrow/4-docker/buildall: skip ssh-r
by Wu Zhende 22 Dec '20

22 Dec '20

Do not restart the ssh-r container randomly. Otherwise, the established connection will be disconnected, and the requested hosts can't be logged in. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sparrow/4-docker/buildall | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 98d3ce4..32801d1 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -31,6 +31,7 @@ do_one() mkdir $tmpdir/$container_name 2>/dev/null && ( cd $container + [ "$container_name" == 'ssh-r' ] && exit if [ "$action" != "reboot" ]; then [ -x build ] && ./build [ -x install ] && ./install -- 2.23.0

1 0

[PATCH compass-ci] lib/constants: MAIL_HOST=>SEND_MAIL_HOST, MAIL_PORT=>SEND_MAIL_PORT
by Cao Xueliang 22 Dec '20

22 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/constants.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/constants.rb b/lib/constants.rb index 3d1ef1d..f0938f2 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -8,8 +8,8 @@ config = cci_defaults ES_HOST = config['ES_HOST'] || '172.17.0.1' ES_PORT = config['ES_PORT'] || 9200 -MAIL_HOST = config['MAIL_HOST'] || '172.17.0.1' -MAIL_PORT = config['MAIL_PORT'] || 11311 +MAIL_HOST = config['SEND_MAIL_HOST'] || '172.17.0.1' +MAIL_PORT = config['SEND_MAIL_PORT'] || 49000 SRV_HTTP_HOST = config['SRV_HTTP_HOST'] || ENV['SRV_HTTP_HOST'] || '172.17.0.1' SRV_HTTP_PORT = config['SRV_HTTP_PORT'] || ENV['SRV_HTTP_PORT'] || 11300 -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib: add assign_account_client
by Luan Shengde 22 Dec '20

22 Dec '20

add assign_account_client for code calling to assign account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign_account_client.rb | 83 ++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100755 lib/assign_account_client.rb diff --git a/lib/assign_account_client.rb b/lib/assign_account_client.rb new file mode 100755 index 0000000..d7e0623 --- /dev/null +++ b/lib/assign_account_client.rb @@ -0,0 +1,83 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require 'set' +require 'optparse' +require_relative '../container/defconfig' +require_relative 'es_client' +require_relative '../container/mail-robot/lib/assign-account-email' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_HOST + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) +JUMPER_HOST = defaults['JUMPER_HOST'] +JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 +LAB = ENV['lab'] + +# used for other codes calling to assign account for user +class AutoAssignAccount + def initialize(user_info) + @my_info = user_info + + @my_info_es = {} + end + + def update_from_es + account_infos = ESQuery.new(index: 'accounts') + my_account_info = account_infos.query_by_id(@my_info['my_email']) || {} + + my_account_info.update @my_info + @my_info.update my_account_info + @my_info['my_ssh_pubkey'] = [] unless my_account_info['my_ssh_pubkey'] + + @my_info['my_token'] = %x(uuidgen).chomp if @my_info['my_login_name'].nil? + end + + def apply_account + apply_info = {} + + apply_info.update @my_info + apply_info['enable_login'] = true + apply_info['is_update_account'] = true unless apply_info['my_login_name'].nil? + apply_info['lab'] = LAB + + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') + JSON.parse account_info_str + end + + def update_my_info_from_account_info + account_info = apply_account + + @my_info['my_login_name'] = account_info['my_login_name'] + @my_info['my_ssh_pubkey'] << account_info['my_jumper_pubkey'] unless account_info['my_jumper_pubkey'].nil? + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def send_mail + message = build_apply_account_email(@my_info) + %x(curl -XPOST "#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text" -d "#{message}") + end + + def send_account + update_from_es + update_my_info_from_account_info + + store_account_info + send_mail + end +end -- 2.23.0

1 1

[PATCH compass-ci 2/2] lib/mail-post: add send_mail_encode function
by Luan Shengde 22 Dec '20

22 Dec '20

[why] the http protocol will parse in fail if the request data has special characters. or it's hard to analysis the data if encoded or not on the send mail service. add stand alone interface for it Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/lib/mail-post.rb b/lib/mail-post.rb index df4d255..ac455fa 100644 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -8,6 +8,7 @@ require 'json' require 'yaml' require 'open3' require 'mail' +require 'base64' set :bind, '0.0.0.0' set :port, ENV['SEND_MAIL_PORT'] @@ -37,6 +38,19 @@ post '/send_mail_text' do send_mail(mail_info) end +post '/send_mail_encode' do + data_decode = Base64.decode64(request.body.read) + data = Mail.read_from_string(data_decode) + + mail_info = { + 'subject' => data.subject, + 'to' => data.to, + 'body' => data.body.decoded + } + + send_mail(mail_info) +end + def check_send_mail(mail_info) raise 'no/empty subject.' if mail_info['subject'].nil? || mail_info['subject'].empty? raise 'no/empty email_to address.' if mail_info['to'].nil? || mail_info['to'].empty? -- 2.23.0

1 0

[PATCH lkp-tests] tests/cci-makepkg: change to logic condition
by Liu Shaofei 22 Dec '20

22 Dec '20

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/cci-makepkg | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a50a40c20..d48e8495e 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -41,6 +41,8 @@ curl_pkgfile() download_pkgfile() { + [ -n "$PKGBUILD_TAG" ] || return + local pkg_tag_file="PKGBUILD-$PKGBUILD_TAG" curl_pkgfile "pkg/$benchmark" "$pkg_tag_file" @@ -93,7 +95,7 @@ centos_update_gcc() source /opt/rh/devtoolset-8/enable } -[ -n "$PKGBUILD_TAG" ] && download_pkgfile +download_pkgfile distro_install_depends lkp-dev -- 2.23.0

1 0

[PATCH lkp-tests 2/2] lib/env.sh: refactor set-env
by Wei Jihui 22 Dec '20

22 Dec '20

move sbin/set-env.sh functions to lib/env.sh delete sbin/set-env.sh do set-env at sbin/install-dependencies Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/env.sh | 30 ++++++++++++++++++++++++++ sbin/install-dependencies.sh | 10 ++++++++- sbin/set-env.sh | 42 ------------------------------------ 3 files changed, 39 insertions(+), 43 deletions(-) delete mode 100755 sbin/set-env.sh diff --git a/lib/env.sh b/lib/env.sh index 8ac42a83..9f37d8cc 100755 --- a/lib/env.sh +++ b/lib/env.sh @@ -125,3 +125,33 @@ shell_profile() [ -f "$profile" ] || touch $profile echo $profile } + +write_shell_profile() +{ + shell_profile_file=$(shell_profile) + if [ $# -gt 0 ]; then + echo "$@" >> $shell_profile_file + fi + + source $shell_profile_file +} + +write_host() +{ + if is_system "Linux"; then + nr_cpu=$(nproc) + memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') + else + nr_cpu=$(sysctl -n hw.logicalcpu) + memory_total=$(top -l 1 | grep MemRegions | awk '{print $2}') + fi + memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') + memory=$(echo $memory_new | awk '{print int($0)+1}')G + + cat > hosts/$(hostname) <<-EOF + nr_cpu: $nr_cpu + memory: $memory + hdd_partitions: + ssd_partitions: + EOF +} diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 6263dc91..8a4b29cd 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -4,7 +4,6 @@ SCRIPT_DIR=$(dirname $(realpath $0)) PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/env.sh -. $PROJECT_DIR/sbin/set-env.sh # choose install function base on DISTRIBUTION linux_dep() @@ -48,4 +47,13 @@ run() fi } +set_env() +{ + write_host + write_shell_profile "export LKP_SRC=$PWD" + write_shell_profile "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" + source $(shell_profile) +} + +set_env run diff --git a/sbin/set-env.sh b/sbin/set-env.sh deleted file mode 100755 index 8234eaf8..00000000 --- a/sbin/set-env.sh +++ /dev/null @@ -1,42 +0,0 @@ -#!/bin/sh - -SCRIPT_DIR=$(dirname $(realpath $0)) -PROJECT_DIR=$(dirname $SCRIPT_DIR) - -. $PROJECT_DIR/lib/env.sh - -write_shell_profile() -{ - shell_profile_file=$(shell_profile) - if [ $# -gt 0 ]; then - echo $@ >> $shell_profile_file - else - echo "export LKP_SRC=$PWD" >> $shell_profile_file - echo "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" >> $shell_profile_file - fi - - source $shell_profile_file -} - -write_host() -{ - if is_system "Linux"; then - nr_cpu=$(nproc) - memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') - else - nr_cpu=$(sysctl -n hw.logicalcpu) - memory_total=$(top -l 1 | grep MemRegions | awk '{print $2}') - fi - memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') - memory=$(echo $memory_new | awk '{print int($0)+1}')G - - cat > hosts/$(hostname) <<-EOF - nr_cpu: $nr_cpu - memory: $memory - hdd_partitions: - ssd_partitions: - EOF -} - -write_shell_profile -write_host -- 2.23.0

1 0

[PATCH lkp-tests 1/2] sbin/install-dependencies: adjust file format
by Wei Jihui 22 Dec '20

22 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/install-dependencies.sh | 39 ++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 20 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 3a206134..6263dc91 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -6,28 +6,27 @@ PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/env.sh . $PROJECT_DIR/sbin/set-env.sh -# choose install function base on -# DISTRIBUTION +# choose install function base on DISTRIBUTION linux_dep() { -source /etc/os-release -case $ID in -ubuntu|debian) - export DEBIAN_FRONTEND=noninteractive - apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client - ;; -openEuler|fedora|rhel|centos) - if type dnf > /dev/null 2>&1; then - dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - else - yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - fi - sudo gem install -f git activesupport rest-client - ;; -*) - echo "$ID not support! please install dependencies manually." - ;; -esac + source /etc/os-release + case $ID in + ubuntu|debian) + export DEBIAN_FRONTEND=noninteractive + apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client + ;; + openEuler|fedora|rhel|centos) + if type dnf > /dev/null 2>&1; then + dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + else + yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + fi + sudo gem install -f git activesupport rest-client + ;; + *) + echo "$ID not support! please install dependencies manually." + ;; + esac } -- 2.23.0

1 0

[PATCH compass-ci] compare_matrixes.rb : reduce compare time
by Li Yuanchao 22 Dec '20

22 Dec '20

When there is multi object group, don't use one object group to compare. Cause one object groups are too many, they spend too much time. And they are not what we want. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/compare_matrixes.rb | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 6e4c2aa..21bd5c9 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -390,7 +390,10 @@ end def compare_group_matrices(group_matrices, suites_hash, options) result_str = '' group_matrices_array = sort_by_matrix_size(group_matrices) + have_multi_member = group_matrices_array[0][1].size > 1 group_matrices_array.each do |matrice_kv| + next if have_multi_member && matrice_kv[1].size < 2 + result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_hash[matrice_kv[0]], options) end result_str -- 2.23.0

1 0

[PATCH lkp-tests] jobs/bisect.yaml: dc-1g--delimiter => dc-1g
by Cao Xueliang 22 Dec '20

22 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- jobs/bisect.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jobs/bisect.yaml b/jobs/bisect.yaml index 5fab30bfc..11ffc7c05 100644 --- a/jobs/bisect.yaml +++ b/jobs/bisect.yaml @@ -1,4 +1,4 @@ suite: bisect docker_image: delimiter:latest -testbox: dc-1g--delimiter +testbox: dc-1g bisect: -- 2.23.0

1 0

[PATCH compass-ci] doc/tutorial.md: fix abnormal display in Markdown
by Lu Kaiyi 22 Dec '20

22 Dec '20

the middle part in * and * is italics for markdown document, if want show '*' in this condition, need use \* escape '*' character. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- doc/tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..19d5741 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -258,7 +258,7 @@ job yaml ��[YAML](http://yaml.org/YAM 4) �� archlinux �� PKGBUILD �� PKGBUILD �� - PKGBUILD �� /srv/git/archlinux/*/*/PKGBUILD�� + PKGBUILD �� /srv/git/archlinux/\*/\*/PKGBUILD�� ### depends -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: auto_tool => assign_account_client
by Cao Xueliang 22 Dec '20

22 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 5e296a6..1d4ed69 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -4,7 +4,7 @@ require 'json' require 'base64' require_relative 'git' require_relative 'mail_client' -require_relative 'auto_tool' +require_relative 'assign_account_client' # compose and send email for bisect result class MailBisectResult -- 2.23.0

1 0

[PATCH compass-ci] web_backend.rb: optimize var name
by Li Yuanchao 22 Dec '20

22 Dec '20

As the implementation of method combine_group_query_data changes, var name suit_list is ok, cause the method return a hash in fact. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 2405107..0ca71e1 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -186,15 +186,15 @@ end def do_get_groups_matrices(must, dimension, total, size, from) job_list = get_dimension_job_list(dimension, must, size, from) - matrices, suites_list = combine_group_query_data(job_list, dimension) + matrices, suites_hash = combine_group_query_data(job_list, dimension) while matrices.empty? from += size break if from > total job_list = get_dimension_job_list(dimension, must, size, from) - matrices, suites_list = combine_group_query_data(job_list, dimension) + matrices, suites_hash = combine_group_query_data(job_list, dimension) end - [matrices, suites_list] + [matrices, suites_hash] end def get_groups_matrices(conditions, dimension, must, size, from) @@ -209,11 +209,11 @@ end def get_compare_body(params) dimension, conditions = get_dimension_conditions(params) must = get_es_must(params) - groups_matrices, suites_list = get_groups_matrices(conditions, dimension, must, COMPARE_RECORDS_NUMBER, 0) + groups_matrices, suites_hash = get_groups_matrices(conditions, dimension, must, COMPARE_RECORDS_NUMBER, 0) if !groups_matrices || groups_matrices.empty? body = 'No Data.' else - body = compare_group_matrices(groups_matrices, suites_list, { no_print: true }) + body = compare_group_matrices(groups_matrices, suites_hash, { no_print: true }) body = 'No Difference.' if !body || body.empty? end return body -- 2.23.0

1 0

[PATCH lkp-tests] sbin/install-dependencies: adjuest file format
by Wei Jihui 22 Dec '20

22 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/install-dependencies.sh | 39 ++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 20 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 3a206134..6263dc91 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -6,28 +6,27 @@ PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/env.sh . $PROJECT_DIR/sbin/set-env.sh -# choose install function base on -# DISTRIBUTION +# choose install function base on DISTRIBUTION linux_dep() { -source /etc/os-release -case $ID in -ubuntu|debian) - export DEBIAN_FRONTEND=noninteractive - apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client - ;; -openEuler|fedora|rhel|centos) - if type dnf > /dev/null 2>&1; then - dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - else - yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - fi - sudo gem install -f git activesupport rest-client - ;; -*) - echo "$ID not support! please install dependencies manually." - ;; -esac + source /etc/os-release + case $ID in + ubuntu|debian) + export DEBIAN_FRONTEND=noninteractive + apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client + ;; + openEuler|fedora|rhel|centos) + if type dnf > /dev/null 2>&1; then + dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + else + yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + fi + sudo gem install -f git activesupport rest-client + ;; + *) + echo "$ID not support! please install dependencies manually." + ;; + esac } -- 2.23.0

1 0

[PATCH lkp-tests] lib/env.sh: fix the check result of is_virt
by Zhang Yu 22 Dec '20

22 Dec '20

Hypervisor is a software that simulates hardware resources and scheduling on a physical machine. so if grep success, return 1 means it's a physical machine Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- lib/env.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/env.sh b/lib/env.sh index 8ac42a836..a1d5e445b 100755 --- a/lib/env.sh +++ b/lib/env.sh @@ -69,7 +69,8 @@ is_virt() # run as root [ -n "$(virt-what)" ] else - grep -q -w hypervisor /proc/cpuinfo + grep -q -w hypervisor /proc/cpuinfo && return 1 + return 0 fi } -- 2.23.0

1 0

[PATCH compass-ci] 3-code/git: add repository version pointing
by Hu Xuejiao 22 Dec '20

22 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 494107e..2159ddc 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -13,11 +13,18 @@ cd /c || exit # modify and manual run for now: [ -d "compass-ci/.git" ] || { git clone https://gitee.com/wu_fengguang/compass-ci.git - ln -s compass-ci cci + + cd /c/compass-ci || exit + [ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" + + ln -s compass-ci /c/cci } [ -d "lkp-tests/.git" ] || { git clone https://gitee.com/wu_fengguang/lkp-tests.git + + cd /c/lkp-tests || exit + [ -n "$lkp_commitid" ] && git reset --hard "$lkp_commitid" } [ -d "upstream-repos/.git" ] || { -- 2.23.0

1 0

[PATCH lkp-tests] tests/deploy-cci: specifying the repository version
by Hu Xuejiao 22 Dec '20

22 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/deploy-cci | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/deploy-cci b/tests/deploy-cci index e296568f..55835544 100755 --- a/tests/deploy-cci +++ b/tests/deploy-cci @@ -19,6 +19,9 @@ mkdir -p /c cd /c || exit git clone https://gitee.com/wu_fengguang/compass-ci.git +cd /c/compass-ci || exit +[ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" + # Need mount tmpfs to fix compass-ci/container/os-nfs/start error # "exportfs: /exports/result does not support NFS export" mount -t tmpfs none /srv -- 2.23.0

1 0

[PATCH compass-ci] lib/git_bisect: return all errors to bisect mail.
by Cao Xueliang 22 Dec '20

22 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index bc1e035..f789a71 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -82,7 +82,6 @@ class GitBisect result = `git -C #{@work_dir} bisect run #{BISECT_RUN_SCRIPT} #{@bad_job_id} #{(a)error_id.inspect} #{@work_dir}` end Utils.create_bisect_log(@work_dir) - FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) puts "\nbisect result: #{result}" analyse_result(result) end @@ -92,10 +91,16 @@ class GitBisect return nil unless temp[0].include?('is the first bad commit') || temp[-1].include?('bisect run success') first_bad_commit = Utils.parse_first_bad_commit(result) - error_messages = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_errorid(@error_id) + bisect_error = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_errorid(@error_id) + all_errors = Utils.obt_errors(@work_dir, first_bad_commit) + first_bad_commit_result_root = Utils.obt_result_root_by_commit(first_bad_commit) + FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) - return Hash['repo' => @upstream_repo, 'commit' => first_bad_commit, - 'error_messages' => error_messages] + return Hash['repo' => @upstream_repo, + 'commit' => first_bad_commit, + 'bisect_error' => bisect_error, + 'all_errors' => all_errors, + 'first_bad_commit_result_root' => first_bad_commit_result_root] end # first search the good commit in db -- 2.23.0

1 0

[PATCH compass-ci] delimiter/utils: add the get all errors function
by Cao Xueliang 22 Dec '20

22 Dec '20

1. fix some rubocop warnings. 2. add get errors by commits - get the pre_commit by find_parent_commit - get cur_id and pre_id by obt_id_by_commit - get errors by get_compare_result 3. add get result root by commit Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 48 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 41803f3..372e821 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -4,10 +4,12 @@ require 'set' require 'json' +require 'yaml' require 'fileutils' require_relative './constants' require_relative '../../lib/sched_client' +require_relative '../../lib/compare_error_messages' require_relative "#{ENV['LKP_SRC']}/lib/monitor" # a utils module for delimiter service @@ -83,9 +85,20 @@ module Utils es = ESQuery.new new_job = es.query_by_id(new_job_id) - return 'bad' if new_job['stats'].key?(error_id) - return 'good' + status = new_job['stats'].key?(error_id) ? 'bad' : 'good' + puts "new_job_id: #{new_job_id}" + puts "upstream_commit: #{job['upstream_commit']}" + record_jobs(new_job_id, job['upstream_commit']) + + return status + end + + def record_jobs(job_id, job_commit) + FileUtils.mkdir_p TMP_RESULT_ROOT unless File.exist? TMP_RESULT_ROOT + commit_jobs = File.join(TMP_RESULT_ROOT, 'commit_jobs') + content = "#{job_commit}: #{job_id}" + File.open(commit_jobs, 'a+') { |f| f.puts content } end def get_account_info @@ -100,6 +113,8 @@ module Utils account_info = get_account_info raise "query #{DELIMITER_ACCONUT} account info failed!" unless account_info + record_jobs(job['id'], job['upstream_commit']) + job['suite'] = 'bisect' job['my_name'] = account_info['my_name'] job['my_email'] = account_info['my_email'] @@ -133,5 +148,34 @@ module Utils log_content.each { |line| f.puts(line) } end end + + def find_parent_commit(git_dir, commit) + response = %x(git -C #{git_dir} rev-parse #{commit}~1) + return response.chomp + end + + def obt_id_by_commit(commit) + puts "#{TMP_RESULT_ROOT}/commit_jobs" + content = YAML.load(File.open("#{TMP_RESULT_ROOT}/commit_jobs")) + puts content + content[commit] + end + + def obt_errors(git_dir, commit) + pre_commit = find_parent_commit(git_dir, commit) + obt_errors_by_commits(commit, pre_commit) + end + + def obt_errors_by_commits(cur_commit, pre_commit) + cur_id = obt_id_by_commit(cur_commit) + pre_id = obt_id_by_commit(pre_commit) + _, errors = get_compare_result(pre_id, cur_id) + return errors + end + + def obt_result_root_by_commit(commit) + id = obt_id_by_commit(commit) + ESQuery.new.query_by_id(id)['result_root'] + end end end -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_client: add send_mail_encode function
by Cao Xueliang 22 Dec '20

22 Dec '20

If some special characters the payload of post, we use base64 encode to pass the payload. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_client.rb | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/mail_client.rb b/lib/mail_client.rb index 1d0938a..b13492f 100644 --- a/lib/mail_client.rb +++ b/lib/mail_client.rb @@ -2,6 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'base64' require 'rest-client' require_relative 'constants.rb' @@ -18,4 +19,9 @@ class MailClient resource = RestClient::Resource.new("http://#{@host}:#{@port}/send_mail_yaml") resource.post(mail_json) end + + def send_mail_encode(mail_data) + resource = RestClient::Resource.new("http://#{@host}:#{@port}/send_mail_encode") + resource.post(Base64.encode64(mail_data)) + end end -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: delete input job key which is path
by Zhang Yu 22 Dec '20

22 Dec '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- lib/job.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job.rb b/lib/job.rb index 2ed06edc6..7553312ed 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -254,6 +254,7 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end + hash.delete_if { |key, _| key.start_with?('#!') } hash.merge!(@overrides) @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end -- 2.23.0

1 0

[PATCH compass-ci] fix 404 error
by Li Ping 22 Dec '20

22 Dec '20

Signed-off-by: Li Ping <1477412247(a)qq.com> --- README.en.md | 4 ++-- README.zh.md | 4 ++-- ...34\272\350\260\203\346\265\213\344\273\273\345\212\241.md" | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/README.en.md b/README.en.md index 5e9386e..c082167 100644 --- a/README.en.md +++ b/README.en.md @@ -34,13 +34,13 @@ If a new error ID is generated, the bisect is automatically triggered to locate 2. Run the **git push** command to update the repository and automatically trigger the test. -3. You can [view](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/brow… and [compare](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/c… test results on the web page: https://compass-ci.openeuler.org/jobs. +3. You can [view](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/brow… and [compare](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/c… test results on the web page: https://compass-ci.openeuler.org/jobs **Manually Submitting a Test Task** 1. [Install the Compass-CI client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E…. 2. [Compile test cases and manually submit test tasks](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcas…. -3. You can [view](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/brow… and [compare](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/c… test results on the web page: https://compass-ci.openeuler.org/jobs. +3. You can [view](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/brow… and [compare](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/c… test results on the web page: https://compass-ci.openeuler.org/jobs **Logging in to the Test Environment** diff --git a/README.zh.md b/README.zh.md index 0616bec..c2140b0 100644 --- a/README.zh.md +++ b/README.zh.md @@ -37,13 +37,13 @@ Compass-CI 监控很多开源软件 git repos，一旦检测到代码更新， 2. 执行 git push 命令更新仓库，自动触发测试。 -3. 在网页中[查看](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/b… web: https://compass-ci.openeuler.org/jobs。 +3. 在网页中[查看](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/b… web: https://compass-ci.openeuler.org/jobs **手动提交测试任务** 1. [安装 Compass-CI 客户端](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E6%9… 2. [编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testca…, [手动提交测试任务](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… -3. 在网页中[查看](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/b… web: https://compass-ci.openeuler.org/jobs。 +3. 在网页中[查看](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/b… web: https://compass-ci.openeuler.org/jobs **登录测试环境** diff --git "a/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" index b59431e..7a54505 100644 --- "a/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" +++ "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" @@ -3,8 +3,8 @@ # 1. 前提条件： ## 请先学习: * [配置个人邮箱](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/ap… -* [如何申请测试机](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/如… -* [submit命令详解](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… +* [如何申请测试机](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/b… +* [submit命令详解](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… # 2. 操作方法 -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: optimize mail bisect content
by Cao Xueliang 22 Dec '20

22 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 63 ++++++++++++++++++++++++++------------- 1 file changed, 42 insertions(+), 21 deletions(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 090855e..5e296a6 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -1,48 +1,69 @@ # frozen_string_literal: true -require_relative 'mail_client' -require_relative 'git' require 'json' +require 'base64' +require_relative 'git' +require_relative 'mail_client' +require_relative 'auto_tool' # compose and send email for bisect result class MailBisectResult def initialize(bisect_info) - @error_messages = bisect_info['error_messages'] + @bisect_error = bisect_info['bisect_error'] + @all_errors = bisect_info['all_errors'] @repo = bisect_info['repo'] @commit_id = bisect_info['commit'] + @first_bad_commit_result_root = bisect_info['first_bad_commit_result_root'] @git_commit = GitCommit.new(@repo, @commit_id) + # now send mail to review + @to = 'caoxl(a)crystal.ci, caoxl78320(a)163.com, huming15(a)163.com' end def create_send_email - compose_mail - send_mail + send_report_mail(compose_mail) + send_account_mail end def compose_mail - subject = "[Compass-CI][#{(a)repo.split('/')[1]}]: #{(a)error_messages[0].split("\n")[0]}" - job_url = ENV['result_root'] ? "job url: http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}#{ENV['result_root']}\n" : '' - body = <<~BODY - Hi #{(a)git_commit.author_name}, + subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" + prefix_srv = "http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}" + bisect_job_url = ENV['result_root'] ? "bisect job info: #{prefix_srv}#{ENV['result_root']}\n" : '' + first_bad_commit_job_url = "first bad commit job info: #{prefix_srv}#{@first_bad_commit_result_root}\n" - git url: #{(a)git_commit.url} - git commit: #{@commit_id[0..11]} ("#{(a)git_commit.subject}") + data = <<~BODY + To: #{@to} + Subject: #{subject} - gcc version: 7.3.0 + Hi #{(a)git_commit.author_name}, - error_messages: - #{(a)error_messages.join("\n")} + url: #{(a)git_commit.url} + commit: #{@commit_id} ("#{(a)git_commit.subject}") + compiler: gcc (GCC) 7.3.0 - #{job_url} + all errors/warnings: + #{@all_errors} + + #{bisect_job_url} + #{first_bad_commit_job_url} Regards, Compass CI team BODY - to = 'caoxl(a)crystal.ci' - @hash = { 'to' => to, 'body' => body, 'subject' => subject } - # @hash = { 'to' => @git_commit.author_email, 'body' => body, 'subject' => subject } + + return data end - def send_mail - json = @hash.to_json - MailClient.new.send_mail(json) + def send_report_mail(mail_data) + MailClient.new.send_mail_encode(mail_data) + end + + def send_account_mail + user_info = { + 'my_email' => @to, + 'my_name' => @git_commit.author_name, + 'my_commit_url' => "#{@git_commit.url}/commit/#{@commit_id}" + } + + apply_account = AutoAssignAccount.new(user_info) + apply_account.send_account end end -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: temporarily cancel account verification
by Wu Zhende 22 Dec '20

22 Dec '20

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index e6f7e6a..2554db1 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -347,13 +347,14 @@ class Job error_msg = "Failed to verify the account.\n" error_msg += "Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco…" account_info = @es.get_account(self["my_email"]) - raise account_info unless account_info.is_a?(JSON::Any) + # raise account_info unless account_info.is_a?(JSON::Any) - @account_info = account_info.as_h + # @account_info = account_info.as_h + @account_info = account_info.as_h if account_info.is_a?(JSON::Any) - raise error_msg if @account_info["found"]? == false - raise error_msg unless self["my_name"] == @account_info["my_name"].to_s - raise error_msg unless self["my_token"] == @account_info["my_token"] + # raise error_msg if @account_info["found"]? == false + # raise error_msg unless self["my_name"] == @account_info["my_name"].to_s + # raise error_msg unless self["my_token"] == @account_info["my_token"] @hash.delete("my_uuid") @hash.delete("my_token") -- 2.23.0

1 0

[PATCH lkp-tests] adaptation/archlinux: add archlinux mapping
by Wang Yong 22 Dec '20

22 Dec '20

add mapping in order to run cci-depends job for build-pkg Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/adaptation/archlinux | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/distro/adaptation/archlinux b/distro/adaptation/archlinux index 89163cf41..c52212e08 100644 --- a/distro/adaptation/archlinux +++ b/distro/adaptation/archlinux @@ -72,3 +72,11 @@ libtirpc: libtirpc1 initramfs-tools: openssh-server: openssh openssh-client: openssh +autopoint: gettext +g++: gcc +libarchive13: libarchive +libbz2-1.0: bzip2 +libssl: openssl +libtool-bin: libtool +linux-perf: perf +ruby-dev: ruby -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests: fix return value
by Zhang Yu 22 Dec '20

22 Dec '20

[why] if /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0 both not exit, the value of return should be 0 not 1 Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/host-info b/tests/host-info index 3984727b4..fe55d1ff2 100755 --- a/tests/host-info +++ b/tests/host-info @@ -102,7 +102,7 @@ get_ipmi_ip() # 802.1q VLAN ID : Disabled # RMCP+ Cipher Suites : 0,1,2,3,17 # Cipher Suite Priv Max : XuuaXXXXXXXXXXX - [ -e /dev/ipmi0 ] || [ -e /dev/ipmi/0 ] || [ -e /dev/ipmidev/0 ] || return + [ -e /dev/ipmi0 ] || [ -e /dev/ipmi/0 ] || [ -e /dev/ipmidev/0 ] || return 0 local ip=$(ipmitool lan print | awk 'NR==3{print $4}') [ -n "$ip" ] && echo "ipmi_ip: $ip" } -- 2.23.0

1 0

[PATCH v4 compass-ci 1/6] fetch-mail: config file for fetchmail
by Luan Shengde 22 Dec '20

22 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/fetchmailrc.template | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 container/fetch-mail/fetchmailrc.template diff --git a/container/fetch-mail/fetchmailrc.template b/container/fetch-mail/fetchmailrc.template new file mode 100644 index 0000000..8c7e82b --- /dev/null +++ b/container/fetch-mail/fetchmailrc.template @@ -0,0 +1,5 @@ +poll imap.qq.com with proto IMAP +username "FETCH_MAIL_ADDRESS" +password "FETCH_MAIL_AUTH_CODE" +mda "/usr/bin/procmail -f %F -d %T" +nokeep -- 2.23.0

2 2

[PATCH v4 compass-ci 2/6] fetch-mail: config file for procmail
by Luan Shengde 22 Dec '20

22 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/procmailrc.template | 25 ++++++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 container/fetch-mail/procmailrc.template diff --git a/container/fetch-mail/procmailrc.template b/container/fetch-mail/procmailrc.template new file mode 100644 index 0000000..9cea4bc --- /dev/null +++ b/container/fetch-mail/procmailrc.template @@ -0,0 +1,25 @@ +DROPPRIVS=yes +PATH=$HOME/bin:/usr/local/bin:/usr/bin:/bin +MAILDIR=FETCH_MAIL_DIR +MAILLIST=${MAILDIR} +DEFAULT=${MAILDIR}/FETCH_MAIL_BOX/ +ORGMAIL=${DEFAULT}/FETCH_MAIL_BOX/ +LOGFILE=${MAILDIR}/log/procmail # recommended + +UMASK=002 + +DISPLAY=:0 +XAUTHORITY=$HOME/.Xauthority + +# One often receives duplicate messages for cross-postings to mailing +# lists, or explicit replies to a post from you in addition to the +# mailing list message. +# +# The following recipe eliminates all duplicate messages. It tells +# formail(1) to keep a cache file of the Message-IDs of the most recent +# messages you've received. Since Message-IDs are guaranteed unique for +# distinct messages, they are ideally suited to weed out duplicates. + +# LOCKFILE=$MAILDIR/.msgid.lock +:0 Wh: $MAILDIR/.msgid.lock +| formail -D 655360 $MAILDIR/log/msgid -- 2.23.0

2 2

[PATCH v2 compass-ci] lib/matrix2.rb: fix rubocop warning
by Lu Weitao 22 Dec '20

22 Dec '20

[Why] lib/matrix2.rb:81:1: C: Metrics/AbcSize: Assignment Branch Condition size for create_matrix is too high. [<5, 16, 5> 17.49/15] def create_matrix(job_list) ... ^^^^^^^^^^^^^^^^^^^^^^^^^^^ lib/matrix2.rb:81:1: C: Metrics/CyclomaticComplexity: Cyclomatic complexity for create_matrix is too high. [9/7] def create_matrix(job_list) ... ^^^^^^^^^^^^^^^^^^^^^^^^^^^ lib/matrix2.rb:81:1: C: Metrics/MethodLength: Method has too many lines. [18/15] def create_matrix(job_list) ... ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index a7b83c6..19eb8f1 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -83,13 +83,7 @@ def create_matrix(job_list) suites = [] job_list.each do |job| stats = job['stats'] - next unless stats - - if job['suite'] - next unless stats.keys.any? { |stat| stat.start_with?(job['suite']) } - - suites << job['suite'] - end + next unless stats && assign_suites(suites, job, stats) stats.each do |key, value| next if key.include?('timestamp') @@ -103,6 +97,12 @@ def create_matrix(job_list) return matrix, suites end +def assign_suites(suites, job, stats) + return unless job['suite'] && stats.keys.any? { |stat| stat.start_with?(job['suite']) } + + suites << job['suite'] +end + # input: query results from es_query # return: matrix def combine_query_data(query_data) -- 2.23.0

1 0

[PATCH compass-ci] lib/matrix2.rb: fix rubocop offenses
by Lu Kaiyi 22 Dec '20

22 Dec '20

[error] Inspecting 1 file C Offenses: matrix2.rb:160:35: C: Style/TrailingUnderscoreVariable: Do not use trailing _s in parallel assignment. Prefer groups[group_key][dim_key], = create_matrix(jobs). groups[group_key][dim_key], _ = create_matrix(jobs) ^^ Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index dfc606e..5f538bd 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -157,7 +157,7 @@ def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) groups.each do |group_key, dims| dims.each do |dim_key, jobs| - groups[group_key][dim_key], _ = create_matrix(jobs) + groups[group_key][dim_key], = create_matrix(jobs) end end -- 2.23.0

1 0

[PATCH compass-ci] lib/matrix2.rb: fix rubocop warning
by Lu Weitao 22 Dec '20

22 Dec '20

[Why] lib/matrix2.rb:81:1: C: Metrics/AbcSize: Assignment Branch Condition size for create_matrix is too high. [<5, 16, 5> 17.49/15] def create_matrix(job_list) ... ^^^^^^^^^^^^^^^^^^^^^^^^^^^ lib/matrix2.rb:81:1: C: Metrics/CyclomaticComplexity: Cyclomatic complexity for create_matrix is too high. [9/7] def create_matrix(job_list) ... ^^^^^^^^^^^^^^^^^^^^^^^^^^^ lib/matrix2.rb:81:1: C: Metrics/MethodLength: Method has too many lines. [18/15] def create_matrix(job_list) ... ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index a7b83c6..fc805d8 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -85,12 +85,7 @@ def create_matrix(job_list) stats = job['stats'] next unless stats - if job['suite'] - next unless stats.keys.any? { |stat| stat.start_with?(job['suite']) } - - suites << job['suite'] - end - + assign_suites(suites, job, stats) stats.each do |key, value| next if key.include?('timestamp') @@ -103,6 +98,12 @@ def create_matrix(job_list) return matrix, suites end +def assign_suites(suites, job, stats) + return unless job['suite'] && stats.keys.any? { |stat| stat.start_with?(job['suite']) } + + suites << job['suite'] +end + # input: query results from es_query # return: matrix def combine_query_data(query_data) -- 2.23.0

1 1

[PATCH v2 compass-ci] lib/matrix2.rb remove the *timestamp* from matrix
by Lu Weitao 22 Dec '20

22 Dec '20

[Why] In compare results: the value of *timestamp*(metric) doesn't mean the number of occurrences, it may confuse user like bellow: openeuler centos archLinux metric -------------------- ------------------------------ ------------------------------ ------------------------------ fails:runs change fails:runs change fails:runs | | | | | 3585:13 +6913.7% 344:1 -2.6e+4% 11:1 dmesg.timestamp:last 2350:13 +1.6e+4% 344:1 -1.7e+4% 10:1 kmsg.timestamp:last 2:13 -15.4% 0:1 +84.6% 1:1 last_state.exit_fail [How] remove the metric that include "timestamp" from matrix Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index c712e32..2769f7c 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -87,6 +87,8 @@ def create_matrix(job_list) next unless stats stats.each do |key, value| + next if key.include?('timestamp') + matrix[key] = [] unless matrix[key] matrix[key] << value end -- 2.23.0

3 3

[PATCH compass-ci] lib/matrix2.rb: remove duplicate code
by Lu Weitao 22 Dec '20

22 Dec '20

[Why] Had handle empty group in lib/params_group.rb def filter_groups(groups) groups.each do |group_key, value| value.each_key do |dim_key| value.delete(dim_key) if value[dim_key].empty? end groups.delete(group_key) if groups[group_key].empty? end end Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 4 ---- 1 file changed, 4 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index dfc606e..a7b83c6 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -124,10 +124,6 @@ def combine_group_query_data(job_list, dims) suites_hash = {} groups = auto_group(job_list, dims) groups.each do |group_key, value| - if value.empty? - groups.delete(group_key) - next - end suite_list = [] value.each do |dimension_key, jobs| groups[group_key][dimension_key], suites = create_matrix(jobs) -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests/build-pkg: add config file support
by Wang Yong 22 Dec '20

22 Dec '20

[Why] config file in some software need storage and recall now we can add config value in yaml and cp ${config} to ${cci_dir}/${pkgname}/ in PKGBUILD so that config file can be uploaded correctly $ cat test.yaml ... config: ${config} ... Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index addff9f7b..06a58d498 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -4,6 +4,7 @@ # - upstream_commit # - git_tag # - pkg_ver +# - config # upstream_url # We'll use the below global env vars, but not treat them as parameter to avoid @@ -20,6 +21,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/distro/common +. $LKP_SRC/lib/http.sh check_vars() { @@ -31,6 +33,29 @@ check_vars() [ -n "$os_mount" ] || die "os_mount is empty" } +check_config_var() +{ + [ -n "$config" ] || return +} + +get_config() +{ + local cci_http_host=${INITRD_HTTP_HOST} + local cci_http_port=${INITRD_HTTP_PORT:-8800} + export cci_dir=/cci/build-config + local remote_dir="http://${cci_http_host}:${cci_http_port}${cci_dir}/${pkgname}" + + mkdir -p ${cci_dir} + + check_config_var + http_get_file "${remote_dir}/${config}" "${cci_dir}/${pkgname}/${config}" +} + +upload_config() +{ + upload_to_target_dir "${cci_dir}/${pkgname}" +} + mount_dest() { # the same image is mounted to cifs and nfs, the generated cgz files @@ -46,7 +71,7 @@ get_pkgfile() { curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "${2##*/}" - } +} request_pkg() { @@ -125,13 +150,17 @@ build_source_pkg() fi } replace_source PKGBUILD.src - create_cgz_name + get_config PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf - [ "$?" == 0 ] || exit 1 + local build_state="$?" + + upload_config + + [ "${build_state}" == "0" ] || exit 1 create_softlink upload_to_target_dir ${PKG_MNT}/${pack_to}/${pkgname} } -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/job.cr: check account info
by Wu Zhende 22 Dec '20

22 Dec '20

If submit yaml in /srv/result, will not check account info. Fix it Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4d0b5df..e6f7e6a 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -51,17 +51,23 @@ class Job def initialize(job_content : JSON::Any, id) @hash = job_content.as_h + @es = Elasticsearch::Client.new + @account_info = Hash(String, JSON::Any).new # init job with "-1", or use the original job_content["id"] id = "-1" if "#{id}" == "" if initialized? - return if @hash["id"] == "#{id}" + if @hash["id"] == "#{id}" + return unless @hash.has_key?("my_uuid") || @hash.has_key?("my_token") + + check_account_info() + return + end end - @es = Elasticsearch::Client.new @hash["id"] = JSON::Any.new("#{id}") - @account_info = Hash(String, JSON::Any).new + check_required_keys() check_account_info() set_defaults() -- 2.23.0

2 1

[PATCH v2 lkp-tests] daemon: fix ssh login testbox for docker
by Xiao Shenwei 22 Dec '20

22 Dec '20

[why] systemctl is not supported when use docker [how] use the sshd command Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- daemon/sshd | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/daemon/sshd b/daemon/sshd index 41fa99adf..6bfda467e 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -6,19 +6,31 @@ # sshr_port_base . $LKP_SRC/lib/http.sh +. $LKP_SRC/lib/env.sh sshr_ip=$LKP_SERVER [ -n "$sshr_port" ] || sshr_port=5050 [ -n "$sshr_port_len" ] || sshr_port_len=2000 [ -n "$sshr_port_base" ] || sshr_port_base=50000 +generate_key_file() +{ + echo -e "y\n" | ssh-keygen -t rsa -b 2048 -N '' -f /etc/ssh/ssh_host_rsa_key +} + run_ssh() { [ -n "$my_ssh_pubkey" ] || return umask 0077 mkdir -p /root/.ssh echo "$my_ssh_pubkey" > /root/.ssh/authorized_keys - systemctl start sshd + if is_docker; then + generate_key_file + sshd_path=$(cmd_path sshd) + $sshd_path -q + else + systemctl start sshd + fi } data_success() -- 2.23.0

4 5

Re: [PATCH lkp-tests] fix lkp make install error
by Wu Fengguang 22 Dec '20

22 Dec '20

TO ALL Best practice is to use realpath. >-. $LKP_SRC/lib/env.sh >-. $LKP_SRC/sbin/set-env.sh >+. lib/env.sh >+. sbin/set-env.sh SCRIPT_DIR=$(dirname $(realpath $0)) PROJECT_DIR=$(dirname $SCRIPT_DIR) $PROJECT_DIR/lib/env.sh $PROJECT_DIR/sbin/set-env.sh Thanks, Fengguang

2 1

[PATCH lkp-tests] daemon/multi-qemu: add $CACHE_DIR judgment
by Hu Xuejiao 22 Dec '20

22 Dec '20

$CACHE_DIR is not created in crystal Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- daemon/multi-qemu | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/daemon/multi-qemu b/daemon/multi-qemu index 90197dae..87b40758 100755 --- a/daemon/multi-qemu +++ b/daemon/multi-qemu @@ -13,7 +13,7 @@ multi_qemu() export CCI_SRC=${CCI_SRC:-"/c/compass-ci"} local hostname=$tbox_group.$HOSTNAME - cd "$CACHE_DIR" + [ -n "$CACHE_DIR" ] && cd "$CACHE_DIR" $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" } -- 2.23.0

3 2

[PATCH compass-ci] lib: add assign_account_client
by Luan Shengde 22 Dec '20

22 Dec '20

add assign_account_client for code calling to assign account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign_account_client.rb | 83 ++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100755 lib/assign_account_client.rb diff --git a/lib/assign_account_client.rb b/lib/assign_account_client.rb new file mode 100755 index 0000000..d7e0623 --- /dev/null +++ b/lib/assign_account_client.rb @@ -0,0 +1,83 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require 'set' +require 'optparse' +require_relative '../container/defconfig' +require_relative 'es_client' +require_relative '../container/mail-robot/lib/assign-account-email' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_HOST + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) +JUMPER_HOST = defaults['JUMPER_HOST'] +JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 +LAB = ENV['lab'] + +# used for other codes calling to assign account for user +class AutoAssignAccount + def initialize(user_info) + @my_info = user_info + + @my_info_es = {} + end + + def update_from_es + account_infos = ESQuery.new(index: 'accounts') + my_account_info = account_infos.query_by_id(@my_info['my_email']) || {} + + my_account_info.update @my_info + @my_info.update my_account_info + @my_info['my_ssh_pubkey'] = [] unless my_account_info['my_ssh_pubkey'] + + @my_info['my_token'] = %x(uuidgen).chomp if @my_info['my_login_name'].nil? + end + + def apply_account + apply_info = {} + + apply_info.update @my_info + apply_info['enable_login'] = true + apply_info['is_update_account'] = true unless apply_info['my_login_name'].nil? + apply_info['lab'] = LAB + + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') + JSON.parse account_info_str + end + + def update_my_info_from_account_info + account_info = apply_account + + @my_info['my_login_name'] = account_info['my_login_name'] + @my_info['my_ssh_pubkey'] << account_info['my_jumper_pubkey'] unless account_info['my_jumper_pubkey'].nil? + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def send_mail + message = build_apply_account_email(@my_info) + %x(curl -XPOST "#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text" -d "#{message}") + end + + def send_account + update_from_es + update_my_info_from_account_info + + store_account_info + send_mail + end +end -- 2.23.0

2 2

[PATCH v2 lkp-tests 1/4] bin/install-dependencies.sh: refactor file
by Wei Jihui 22 Dec '20

22 Dec '20

all dependent functions is move to lib/env.sh, sbin/set-env.sh in not required. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/install-dependencies.sh | 40 +++++++++++++++++------------------- 1 file changed, 19 insertions(+), 21 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 7605a29f..b4d3ad22 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -1,30 +1,28 @@ #!/bin/bash . $LKP_SRC/lib/env.sh -. $LKP_SRC/sbin/set-env.sh -# choose install function base on -# DISTRIBUTION +# choose install function base on DISTRIBUTION linux_dep() { -source /etc/os-release -case $ID in -ubuntu|debian) - export DEBIAN_FRONTEND=noninteractive - apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client - ;; -openEuler|fedora|rhel|centos) - if type dnf > /dev/null 2>&1; then - dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - else - yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - fi - sudo gem install -f git activesupport rest-client - ;; -*) - echo "$ID not support! please install dependencies manually." - ;; -esac + source /etc/os-release + case $ID in + ubuntu|debian) + export DEBIAN_FRONTEND=noninteractive + apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client + ;; + openEuler|fedora|rhel|centos) + if type dnf > /dev/null 2>&1; then + dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + else + yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + fi + sudo gem install -f git activesupport rest-client + ;; + *) + echo "$ID not support! please install dependencies manually." + ;; + esac } -- 2.23.0

3 3

[PATCH v2 lkp-tests] tests/build-pkg: add config file support
by Wang Yong 21 Dec '20

21 Dec '20

[Why] config file in some software need storage and recall now we can add config value in yaml and cp ${config} to ${cci_dir}/${benchmark}/ in PKGBUILD so that config file can be uploaded correctly $ cat test.yaml ... config: ${config} ... Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index addff9f7b..e539be923 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -4,6 +4,7 @@ # - upstream_commit # - git_tag # - pkg_ver +# - config # upstream_url # We'll use the below global env vars, but not treat them as parameter to avoid @@ -20,6 +21,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/distro/common +. $LKP_SRC/lib/http.sh check_vars() { @@ -31,6 +33,29 @@ check_vars() [ -n "$os_mount" ] || die "os_mount is empty" } +check_config_var() +{ + [ -n "$config" ] || return +} + +get_config() +{ + local cci_http_host=${INITRD_HTTP_HOST} + local cci_http_port=${INITRD_HTTP_PORT:-8800} + export cci_dir=/cci/config + local remote_dir="http://${cci_http_host}:${cci_http_port}${cci_dir}/${pkgname}" + + mkdir -p ${cci_dir} + + check_config_var + http_get_file "${remote_dir}/${config}" "${cci_dir}/${pkgname}/${config}" +} + +upload_config() +{ + upload_to_target_dir "${cci_dir}/${pkgname}" +} + mount_dest() { # the same image is mounted to cifs and nfs, the generated cgz files @@ -46,7 +71,7 @@ get_pkgfile() { curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "${2##*/}" - } +} request_pkg() { @@ -125,13 +150,17 @@ build_source_pkg() fi } replace_source PKGBUILD.src - create_cgz_name + get_config PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf - [ "$?" == 0 ] || exit 1 + local build_state="$?" + + upload_config + + [ "${build_state}" == "0" ] || exit 1 create_softlink upload_to_target_dir ${PKG_MNT}/${pack_to}/${pkgname} } -- 2.23.0

1 0

[PATCH lkp-tests] daemon/multi-docker: add default value of queues
by Hu Xuejiao 21 Dec '20

21 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- daemon/multi-docker | 2 ++ 1 file changed, 2 insertions(+) diff --git a/daemon/multi-docker b/daemon/multi-docker index 257645b8..7759fdf3 100755 --- a/daemon/multi-docker +++ b/daemon/multi-docker @@ -6,6 +6,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +: ${queues:="$tbox_group.$HOSTNAME,$tbox_group.$(arch)"} + multi_docker() { export CCI_SRC=${CCI_SRC:-"/c/compass-ci"} -- 2.23.0

1 0

[PATCH v4 compass-ci 3/3] lib: use suites_hash instead of suites_list to keep consistent
by Lu Kaiyi 21 Dec '20

21 Dec '20

[why] suite_list maybe not consistent with group key if sort group matrixes. [how] use suites_hash instead of suites_list to keep correct congruent relationship with same group_key. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 4 ++-- lib/compare_matrixes.rb | 10 +++++----- lib/matrix2.rb | 6 +++--- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index da761ef..9090c3b 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -62,8 +62,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices, suites_list = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, suites_list, options) + groups_matrices, suites_hash = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_hash, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 0727797..6e4c2aa 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -387,25 +387,25 @@ end # input: group matrices # output: pre compare result of each group # the result with more comparison objects first -def compare_group_matrices(group_matrices, suites_list, options) +def compare_group_matrices(group_matrices, suites_hash, options) result_str = '' group_matrices_array = sort_by_matrix_size(group_matrices) group_matrices_array.each do |matrice_kv| - result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_list, options) + result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_hash[matrice_kv[0]], options) end result_str end -def get_matrix_str(matrice_key, matrice_value, suites_list, options) +def get_matrix_str(matrice_key, matrice_value, suite_list, options) m_list = [] m_titles = [] matrice_value.each do |dim, matrix| m_titles << dim m_list << matrix end - return compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) if options[:no_print] + return compare_matrixes(m_list, suite_list, m_titles, matrice_key, options: options) if options[:no_print] - print compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) + print compare_matrixes(m_list, suite_list, m_titles, matrice_key, options: options) return '' end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 4393a65..dfc606e 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -121,7 +121,7 @@ end # ... # } def combine_group_query_data(job_list, dims) - suites_list = [] + suites_hash = {} groups = auto_group(job_list, dims) groups.each do |group_key, value| if value.empty? @@ -133,10 +133,10 @@ def combine_group_query_data(job_list, dims) groups[group_key][dimension_key], suites = create_matrix(jobs) suite_list.concat(suites) end - suites_list << suite_list + suites_hash[group_key] = suite_list end - return groups, suites_list + return groups, suites_hash end # input: -- 2.23.0

1 0

[PATCH v4 compass-ci 2/3] lib/compare_matrixes.rb: don't remove the field that start with $suite
by Lu Kaiyi 21 Dec '20

21 Dec '20

if the metric field start with $suite, even if the field is unchanged, still keep it in compare result. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_matrixes.rb | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index b7ea5a4..0727797 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -176,11 +176,15 @@ def get_matrixes_values(matrixes_list, options) matrixes_values end -def remove_unchanged_field(matrixes_values) +def remove_unchanged_field(matrixes_values, suite_list) # remove unchanged field from matrixes values and remove :changed key # matrixes_values.each_key do |success| matrixes_values[success].delete_if do |field| + if suite_list.any? { |suite| field.start_with?(suite) } + matrixes_values[success][field][:changed] = true + end + !matrixes_values[success][field].delete(:changed) end end @@ -203,7 +207,7 @@ def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key options = { 'perf-profile': 5, theme: :none, no_print: false }.merge(options) matrixes_values = get_matrixes_values(matrixes_list, options) - remove_unchanged_field(matrixes_values) if matrixes_list.length > 1 + remove_unchanged_field(matrixes_values, suite_list) if matrixes_list.length > 1 no_print = options[:no_print] result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( -- 2.23.0

1 0

[PATCH v4 compass-ci 1/3] lib/matrix2.rb: discard the no $suite related job
by Lu Kaiyi 21 Dec '20

21 Dec '20

if job has suite, but no $suite related stats metric, then discard this job when create matrix. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index bff4eee..4393a65 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -82,10 +82,15 @@ def create_matrix(job_list) matrix = {} suites = [] job_list.each do |job| - suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats + if job['suite'] + next unless stats.keys.any? { |stat| stat.start_with?(job['suite']) } + + suites << job['suite'] + end + stats.each do |key, value| next if key.include?('timestamp') -- 2.23.0

1 0

[PATCH compass-ci] service/scheduler: move the service logic to the Sched class
by Wu Zhende 21 Dec '20

21 Dec '20

Keep scheduler.cr neat and do not perform operations related to env. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 40 +++++++++++++++++++++++++++------- src/scheduler/find_job_boot.cr | 2 +- src/scheduler/scheduler.cr | 32 ++++----------------------- 3 files changed, 37 insertions(+), 37 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a1ef28e..f7220f9 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -51,20 +51,44 @@ class Sched mac.gsub(":", "-") end - def set_host_mac(mac : String, hostname : String) - @redis.hash_set("sched/mac2host", normalize_mac(mac), hostname) + def set_host_mac + if (hostname = @env.params.query["hostname"]?) && (mac = @env.params.query["mac"]?) + @redis.hash_set("sched/mac2host", normalize_mac(mac), hostname) + + "Done" + else + "No yet!" + end end - def del_host_mac(mac : String) - @redis.hash_del("sched/mac2host", normalize_mac(mac)) + def del_host_mac + if mac = @env.params.query["mac"]? + @redis.hash_del("sched/mac2host", normalize_mac(mac)) + + "Done" + else + "No yet!" + end end - def set_host2queues(hostname : String, queues : String) - @redis.hash_set("sched/host2queues", hostname, queues) + def set_host2queues + if (queues = @env.params.query["queues"]?) && (hostname = @env.params.query["host"]?) + @redis.hash_set("sched/host2queues", hostname, queues) + + "Done" + else + "No yet!" + end end - def del_host2queues(hostname : String) - @redis.hash_del("sched/host2queues", hostname) + def del_host2queues + if hostname = @env.params.query["host"]? + @redis.hash_del("sched/host2queues", hostname) + + "Done" + else + "No yet!" + end end def update_tbox_wtmp diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 7565375..9b3db88 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -29,7 +29,7 @@ class Sched # then hostname will be "sut-00-01-02-03-04" !!! def submit_host_info_job(mac) host = "sut-#{mac}" - set_host_mac(mac, host) + @redis.hash_set("sched/mac2host", normalize_mac(mac), host) Jobfile::Operate.auto_submit_job( "#{ENV["LKP_SRC"]}/jobs/host-info.yaml", diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index 7ec149c..2103f7c 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -79,46 +79,22 @@ module Scheduler # # curl -X PUT "http://localhost:3000/set_host_mac?hostname=wfg&mac=00-01-02-03-04-05" put "/set_host_mac" do |env| - if (client_hostname = env.params.query["hostname"]?) && (client_mac = env.params.query["mac"]?) - env.sched.set_host_mac(client_mac, client_hostname) - - "Done" - else - "No yet!" - end + env.sched.set_host_mac end # curl -X PUT "http://localhost:3000/set_host2queues?queues=vm-2p8g.aarch64&host=vm-2p8g.a…" put "/set_host2queues" do |env| - if (client_queues = env.params.query["queues"]?) && (client_host = env.params.query["host"]?) - env.sched.set_host2queues(client_host, client_queues) - - "Done" - else - "No yet" - end + env.sched.set_host2queues end # curl -X PUT "http://localhost:3000/del_host_mac?mac=00-01-02-03-04-05" put "/del_host_mac" do |env| - if client_mac = env.params.query["mac"]? - env.sched.del_host_mac(client_mac) - - "Done" - else - "No yet!" - end + env.sched.del_host_mac end # curl -X PUT "http://localhost:3000/del_host2queues?host=vm-2p8g.aarch64" put "/del_host2queues" do |env| - if client_host = env.params.query["host"]? - env.sched.del_host2queues(client_host) - - "Done" - else - "No yet!" - end + env.sched.del_host2queues end # client(runner) report job's status -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: check account info
by Wu Zhende 21 Dec '20

21 Dec '20

If submit yaml in /srv/result, will not check account info. Fix it Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4d0b5df..e6f7e6a 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -51,17 +51,23 @@ class Job def initialize(job_content : JSON::Any, id) @hash = job_content.as_h + @es = Elasticsearch::Client.new + @account_info = Hash(String, JSON::Any).new # init job with "-1", or use the original job_content["id"] id = "-1" if "#{id}" == "" if initialized? - return if @hash["id"] == "#{id}" + if @hash["id"] == "#{id}" + return unless @hash.has_key?("my_uuid") || @hash.has_key?("my_token") + + check_account_info() + return + end end - @es = Elasticsearch::Client.new @hash["id"] = JSON::Any.new("#{id}") - @account_info = Hash(String, JSON::Any).new + check_required_keys() check_account_info() set_defaults() -- 2.23.0

1 0

[PATCH compass-ci 4/4] web_backend.rb: optimize the way of query data for compare
by Li Yuanchao 21 Dec '20

21 Dec '20

Before: When do compare with dimension, first query 100(default size) jobs by filters, then group these jobs by dimension. But if one kind of job is done too many, other kinds of jobs may can not be got. After: Fisrt query how many kinds of dimensions in es, take dimension = 'os' as an example, the result can be ["openeuler", "archlinux", "centos", "debian", "kylin", "archLinux"] Then query 100 jobs by each os, then group all these jobs. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/compare.rb | 2 +- lib/matrix2.rb | 3 +- src/lib/web_backend.rb | 64 ++++++++++++++++++++++++++++-------------- 3 files changed, 45 insertions(+), 24 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index db264f8..da761ef 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -69,7 +69,7 @@ end def create_groups_matrices_list(conditions, dims) es = ESQuery.new(ES_HOST, ES_PORT) query_results = es.multi_field_query(conditions) - combine_group_query_data(query_results, dims) + combine_group_query_data(query_results['hits']['hits'], dims) end # ------------------------------------------------------------------------------------------- diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2507b77..b6cfa79 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -109,9 +109,8 @@ end # group2_key => {...} # ... # } -def combine_group_query_data(query_data, dims) +def combine_group_query_data(job_list, dims) suites_list = [] - job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| if value.empty? diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index c4182a3..a680b8a 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -34,7 +34,7 @@ ALL_FIELDS = FIELDS + NOT_SHOW_FIELDS NOT_NEED_EXIST_FIELDS = %w[error_ids upstream_repo].freeze PREFIX_SEARCH_FIELDS = ['tbox_group'].freeze ES_CLIENT = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") -COMPARE_RECORDS_NUMBER = 50 +COMPARE_RECORDS_NUMBER = 100 def es_query(query) ES_CLIENT.search index: 'jobs*', body: query @@ -148,30 +148,21 @@ def get_es_must(params) must end -def do_get_groups_matrices(query, dimension, total, size, from) - result = es_query(query) - matrices, suites_list = combine_group_query_data(result, dimension) - while matrices.empty? - from += size - break if from > total - - query[:from] = from - result = es_query(query) - matrices, suites_list = combine_group_query_data(result, dimension) - end - [matrices, suites_list] +def get_dimension_list(dimension) + query = { size: 0, aggs: { dims: { terms: { size: 10000, field: dimension[0] } } } } + buckets = es_query(query)['aggregations']['dims']['buckets'] + dimension_list = [] + buckets.each { |dims_agg| dimension_list << dims_agg['key'] } + return dimension_list end -def get_groups_matrices(conditions, dimension, must, size, from) - must += build_multi_field_subquery_body(conditions) - count_query = { query: { bool: { must: must } } } - total = es_count(count_query) - return {} if total < 1 - +def query_dimension(dim_field, dim_value, must, size, from) + must_dim = Array.new(must) + must_dim << { term: { dim_field => dim_value } } query = { query: { bool: { - must: must + must: must_dim } }, size: size, @@ -180,8 +171,39 @@ def get_groups_matrices(conditions, dimension, must, size, from) start_time: { order: 'desc' } }] } + es_query(query)['hits']['hits'] +end + +def get_dimension_job_list(dimension, must, size, from) + dimension_list = get_dimension_list(dimension) + job_list = [] + dimension_list.each do |dim| + job_list += query_dimension(dimension[0], dim, must, size, from) + end + job_list +end + +def do_get_groups_matrices(must, dimension, total, size, from) + job_list = get_dimension_job_list(dimension, must, size, from) + + matrices, suites_list = combine_group_query_data(job_list, dimension) + while matrices.empty? + from += size + break if from > total + + job_list = get_dimension_job_list(dimension, must, size, from) + matrices, suites_list = combine_group_query_data(job_list, dimension) + end + [matrices, suites_list] +end + +def get_groups_matrices(conditions, dimension, must, size, from) + must += build_multi_field_subquery_body(conditions) + count_query = { query: { bool: { must: must } } } + total = es_count(count_query) + return {} if total < 1 - do_get_groups_matrices(query, dimension, total, size, from) + do_get_groups_matrices(must, dimension, total, size, from) end def get_compare_body(params) -- 2.23.0

2 1

[PATCH v2 lkp-tests 2/4] lib/env.sh: move functions to env.sh
by Wei Jihui 21 Dec '20

21 Dec '20

move functions:write_shell_profile and write_host to lib/env.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/env.sh | 30 ++++++++++++++++++++++++++++++ sbin/set-env.sh | 39 --------------------------------------- 2 files changed, 30 insertions(+), 39 deletions(-) delete mode 100755 sbin/set-env.sh diff --git a/lib/env.sh b/lib/env.sh index 8ac42a83..9f37d8cc 100755 --- a/lib/env.sh +++ b/lib/env.sh @@ -125,3 +125,33 @@ shell_profile() [ -f "$profile" ] || touch $profile echo $profile } + +write_shell_profile() +{ + shell_profile_file=$(shell_profile) + if [ $# -gt 0 ]; then + echo "$@" >> $shell_profile_file + fi + + source $shell_profile_file +} + +write_host() +{ + if is_system "Linux"; then + nr_cpu=$(nproc) + memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') + else + nr_cpu=$(sysctl -n hw.logicalcpu) + memory_total=$(top -l 1 | grep MemRegions | awk '{print $2}') + fi + memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') + memory=$(echo $memory_new | awk '{print int($0)+1}')G + + cat > hosts/$(hostname) <<-EOF + nr_cpu: $nr_cpu + memory: $memory + hdd_partitions: + ssd_partitions: + EOF +} diff --git a/sbin/set-env.sh b/sbin/set-env.sh deleted file mode 100755 index cc7f9050..00000000 --- a/sbin/set-env.sh +++ /dev/null @@ -1,39 +0,0 @@ -#!/bin/sh - -. $LKP_SRC/lib/env.sh - -write_shell_profile() -{ - shell_profile_file=$(shell_profile) - if [ $# -gt 0 ]; then - echo $@ >> $shell_profile_file - else - echo "export LKP_SRC=$PWD" >> $shell_profile_file - echo "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" >> $shell_profile_file - fi - - source $shell_profile_file -} - -write_host() -{ - if is_system "Linux"; then - nr_cpu=$(nproc) - memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') - else - nr_cpu=$(sysctl -n hw.logicalcpu) - memory_total=$(top -l 1 | grep MemRegions | awk '{print $2}') - fi - memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') - memory=$(echo $memory_new | awk '{print int($0)+1}')G - - cat > hosts/$(hostname) <<-EOF - nr_cpu: $nr_cpu - memory: $memory - hdd_partitions: - ssd_partitions: - EOF -} - -write_shell_profile -write_host -- 2.23.0

2 1

[PATCH compass-ci] container/delimiter: install util-linux, terminal-table, mail
by Cao Xueliang 21 Dec '20

21 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/delimiter/Dockerfile b/container/delimiter/Dockerfile index 3db5890..93c52c6 100644 --- a/container/delimiter/Dockerfile +++ b/container/delimiter/Dockerfile @@ -6,8 +6,8 @@ FROM alpine:3.11 RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ apk update && \ apk add --no-cache 'ruby-dev' \ - 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' RUN umask 002 && \ echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ - gem install rest-client activesupport git json yaml threadpool elasticsearch faye-websocket + gem install rest-client activesupport git json yaml threadpool elasticsearch faye-websocket terminal-table mail -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] lib/matrix2.rb: discard the no $suite related job
by Lu Kaiyi 21 Dec '20

21 Dec '20

if job has suite, but no $suite related stats metric, then discard this job when create matrix. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2769f7c..ef5d935 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -82,10 +82,15 @@ def create_matrix(job_list) matrix = {} suites = [] job_list.each do |job| - suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats + if job['suite'] + next unless stats.keys.any? { |stat| stat.start_with?(job['suite']) } + + suites << job['suite'] + end + stats.each do |key, value| next if key.include?('timestamp') -- 2.23.0

2 1

[PATCH v3 compass-ci 2/3] lib/compare_matrixes.rb: don't remove the field that start with $suite
by Lu Kaiyi 21 Dec '20

21 Dec '20

if the metric field start with $suite, even if the field is unchanged, still keep it in compare result. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_matrixes.rb | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index b7ea5a4..0727797 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -176,11 +176,15 @@ def get_matrixes_values(matrixes_list, options) matrixes_values end -def remove_unchanged_field(matrixes_values) +def remove_unchanged_field(matrixes_values, suite_list) # remove unchanged field from matrixes values and remove :changed key # matrixes_values.each_key do |success| matrixes_values[success].delete_if do |field| + if suite_list.any? { |suite| field.start_with?(suite) } + matrixes_values[success][field][:changed] = true + end + !matrixes_values[success][field].delete(:changed) end end @@ -203,7 +207,7 @@ def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key options = { 'perf-profile': 5, theme: :none, no_print: false }.merge(options) matrixes_values = get_matrixes_values(matrixes_list, options) - remove_unchanged_field(matrixes_values) if matrixes_list.length > 1 + remove_unchanged_field(matrixes_values, suite_list) if matrixes_list.length > 1 no_print = options[:no_print] result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( -- 2.23.0

2 1

[PATCH v3 compass-ci 3/3] lib: use suites_hash instead of suites_list to keep consistent
by Lu Kaiyi 21 Dec '20

21 Dec '20

[why] suite_list maybe not consistent with group key if sort group matrixes. [how] use suites_hash instead of suites_list to keep correct congruent relationship with same group_key. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 4 ++-- lib/compare_matrixes.rb | 10 +++++----- lib/matrix2.rb | 6 +++--- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index db264f8..b5251fc 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -62,8 +62,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices, suites_list = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, suites_list, options) + groups_matrices, suites_hash = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_hash, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 0727797..6e4c2aa 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -387,25 +387,25 @@ end # input: group matrices # output: pre compare result of each group # the result with more comparison objects first -def compare_group_matrices(group_matrices, suites_list, options) +def compare_group_matrices(group_matrices, suites_hash, options) result_str = '' group_matrices_array = sort_by_matrix_size(group_matrices) group_matrices_array.each do |matrice_kv| - result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_list, options) + result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_hash[matrice_kv[0]], options) end result_str end -def get_matrix_str(matrice_key, matrice_value, suites_list, options) +def get_matrix_str(matrice_key, matrice_value, suite_list, options) m_list = [] m_titles = [] matrice_value.each do |dim, matrix| m_titles << dim m_list << matrix end - return compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) if options[:no_print] + return compare_matrixes(m_list, suite_list, m_titles, matrice_key, options: options) if options[:no_print] - print compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) + print compare_matrixes(m_list, suite_list, m_titles, matrice_key, options: options) return '' end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index ef5d935..891c3b0 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -121,7 +121,7 @@ end # ... # } def combine_group_query_data(query_data, dims) - suites_list = [] + suites_hash = {} job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| @@ -134,10 +134,10 @@ def combine_group_query_data(query_data, dims) groups[group_key][dimension_key], suites = create_matrix(jobs) suite_list.concat(suites) end - suites_list << suite_list + suites_hash[group_key] = suite_list end - return groups, suites_list + return groups, suites_hash end # input: -- 2.23.0

1 0

[PATCH compass-ci 3/4] src/lib/web_backend.rb: add testbox api implementation methods
by Li Yuanchao 21 Dec '20

21 Dec '20

Add implementation methods for web api GET /get_tbox_state?testbox=$testbox Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 07e2e9e..c4182a3 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -461,3 +461,29 @@ def query_testboxes end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def get_tbox_state_body(params) + query = { + "query": { + "match": { + "_id": params[:testbox] + } + } + } + body = ES_CLIENT.search(index: 'testbox', body: query)['hits']['hits'][0]['_source'] + + { + testbox: params[:testbox], + states: body + }.to_json +end + +def get_tbox_state(params) + begin + body = get_tbox_state_body(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get testbox state error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

1 0

[PATCH compass-ci 2/4] container/web-backend: add api for web search testbox history
by Li Yuanchao 21 Dec '20

21 Dec '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 11ba604..ff8fcff 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -110,3 +110,26 @@ end get '/get_testboxes' do query_testboxes end + +# GET /get_tbox_state?testbox=$testbox +# Response: +# - { +# testbox: $testbox, +# states: { +# "state": "$current_state", +# "job_id": "$current_job_id", +# "history": [ +# { +# "time": "$history_time", +# "mac": "$mac", +# "ip": "$ip", +# "job_id": "$history_job_id", +# "state": "$history_state" +# }, +# ... +# ] +# } +# } +get '/get_tbox_state' do + get_tbox_state(params) +end -- 2.23.0

1 0

[PATCH compass-ci 1/4] sched.cr: put testbox state to es
by Li Yuanchao 21 Dec '20

21 Dec '20

History of testbox would be kept, to be searched by web api. [sample] { "_index" : "testbox", "_type" : "_doc", "_id" : "vm-2p16g.liyuanchao-969257", "_score" : 1.0, "_source" : { "state" : "rebooting", "job_id" : "crystal.190886", "history" : [ { "time" : "2020-12-15 19:52:14", "mac" : "0a-f5-c0-1f-f0-dd", "ip" : "172.18.202.96", "job_id" : "crystal.190886", "state" : "running" }, { "time" : "2020-12-15 19:57:22", "mac" : "0a-f5-c0-1f-f0-dd", "ip" : "172.18.202.96", "job_id" : "crystal.190886", "state" : "rebooting" } ] } } Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/sched.cr | 1 + src/scheduler/elasticsearch_client.cr | 21 +++++++++++++++++++++ 2 files changed, 22 insertions(+) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a1ef28e..ed6e538 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -90,6 +90,7 @@ class Sched end @redis.update_wtmp(testbox, hash) + @es.update_tbox(testbox, hash) # json log hash["testbox"] = testbox diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index 7c92204..b17d7f8 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -87,6 +87,27 @@ class Elasticsearch::Client ) end + def update_tbox(testbox : String, wtmp_hash : Hash) + query = {:index => "testbox", :type => "_doc", :id => testbox} + result = @client.get_source(query) if @client.exists(query) + history = result["history"].as_a? if result.is_a?(JSON::Any) + history ||= [] of JSON::Any + history << JSON.parse(wtmp_hash.to_json) + + @client.create( + { + :index => "testbox", + :type => "_doc", + :id => testbox, + :body => { + :state => wtmp_hash["state"], + :job_id => wtmp_hash["job_id"], + :history => history + } + } + ) + end + private def create(job_content : JSON::Any, job_id : String) return @client.create( { -- 2.23.0

1 0

[PATCH v3 lkp-tests] lib/monitor: set monitor_url based on job
by Wu Zhende 21 Dec '20

21 Dec '20

If @job is exist and monitor_url is not configured, set monitor_url based on @job's SCHED_HOST Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index f65cfccf1..283cd788d 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -24,7 +24,6 @@ class Monitor # 1 means timeout exit @exit_status_code = 0 @defaults = {} - load_default @result = [] @stop_query = {} @reason = nil @@ -33,6 +32,10 @@ class Monitor def load_default return unless @monitor_url == '' + if host = @job["SCHED_HOST"] + return @monitor_url = "ws://#{host}:11310/filter" + end + Dir.glob(['/etc/compass-ci/monitor/*.yaml', "#{ENV['HOME']}/.config/compass-ci/monitor/*.yaml"]).each do |file| next unless File.exist? file @@ -98,6 +101,7 @@ class Monitor def run(timeout = nil) merge_overrides + load_default field_check @query.each do |k, v| -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] lib/matrix2.rb: remove the *timestamp* from matrix
by Lu Weitao 21 Dec '20

21 Dec '20

[Why] In compare results: the value of *timestamp*(metric) doesn't mean the number of occurrences, it may confuse user like bellow: openeuler centos archLinux metric -------------------- ------------------------------ ------------------------------ ------------------------------ fails:runs change fails:runs change fails:runs | | | | | 3585:13 +6913.7% 344:1 -2.6e+4% 11:1 dmesg.timestamp:last 2350:13 +1.6e+4% 344:1 -1.7e+4% 10:1 kmsg.timestamp:last 2:13 -15.4% 0:1 +84.6% 1:1 last_state.exit_fail [How] remove the metric that include "timestamp" from matrix Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index c712e32..2769f7c 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -87,6 +87,8 @@ def create_matrix(job_list) next unless stats stats.each do |key, value| + next if key.include?('timestamp') + matrix[key] = [] unless matrix[key] matrix[key] << value end -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] lib/matrix2.rb: fix rubocop warning
by Lu Weitao 21 Dec '20

21 Dec '20

[Why] lib/matrix2.rb:75:1: C: Metrics/MethodLength: Method has too many lines. [16/15] def create_matrix(job_list) ... Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2507b77..c712e32 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -64,6 +64,12 @@ def samples_fill_missing_zeros(value, size) value.concat([0] * (size - value.size)) end +def matrix_fill_miss_zeros(matrix, col_size) + matrix.each_value do |value| + samples_fill_missing_zeros(value, col_size) + end +end + # input: job_list # return: matrix of Hash(String, Array(Number)) # Eg: matrix: { @@ -86,9 +92,7 @@ def create_matrix(job_list) end end col_size = job_list.size - matrix.each_value do |value| - samples_fill_missing_zeros(value, col_size) - end + matrix_fill_miss_zeros(matrix, col_size) return matrix, suites end -- 2.23.0

1 0

[PATCH v2 lkp-tests 4/4] install.sh: grant execute permission
by Wei Jihui 21 Dec '20

21 Dec '20

cd lkp-test ./install.sh && source ~/.${SHELL##*/}rc Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- install.sh | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 install.sh diff --git a/install.sh b/install.sh old mode 100644 new mode 100755 -- 2.23.0

1 0

[PATCH v2 lkp-tests 3/4] install.sh: refactor install flow
by Wei Jihui 21 Dec '20

21 Dec '20

1. when exec install.sh, no $LKP_SRC 2. first enable $LKP_SRC env 3. mv sbin/install-dependencies.sh to install.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- Makefile | 2 -- install.sh | 10 +++++++--- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/Makefile b/Makefile index 0b2d09a2..ae062973 100644 --- a/Makefile +++ b/Makefile @@ -14,8 +14,6 @@ subsystem: install: mkdir -p $(TARGET_DIR_BIN) ln -sf $(shell pwd)/bin/lkp $(TARGET_DIR_BIN)/lkp - bash sbin/set-env.sh - bash sbin/install-dependencies.sh .PHONY: doc doc: diff --git a/install.sh b/install.sh index cea1654f..2bd17d54 100644 --- a/install.sh +++ b/install.sh @@ -1,7 +1,11 @@ #!/bin/bash -. $LKP_SRC/lib/env.sh - -make +. lib/env.sh +write_host +write_shell_profile "export LKP_SRC=$PWD" +write_shell_profile "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" source $(shell_profile) + +. sbin/install-dependencies.sh +make install -- 2.23.0

1 0

[PATCH compass-ci] doc: update READEME.md
by Liu Yinsi 21 Dec '20

21 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 111 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 86 insertions(+), 25 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index ccb1d15..7c20a53 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -1,26 +1,87 @@ -Deploy a mini instance in one single machine. - -Prepare: - 1. hardware - Server: at least prepare a server - ProductType: ThaiShan200-2280 - Arch: aarch64 - Memory: 8G - CPU: 64 nuclear - DiskSpace: 500G - - 2. software - OS: openEuler-aarch64-20.03 LTS - git: suggest 2.23.0 - - 3. network - Internet is available - - 4. /os for store rootfs - >= 300G - -Steps: - umask 002 +### 本地搭建 compass-ci 服务器节点 + +在 openEuler 系统一键部署 compass-ci 环境，当前已支持 openEuler-aarch64-20.03-LTS 系统环境，以下配置仅供参考。 + +#### 准备工作 +- 硬件 + 服务器类型：ThaiShan200-2280 (建议) + 架构：aarch64 + 内存：>= 32GB + CPU：64 nuclear (建议) + 硬盘：>= 500G (建议划分独立分区) + + >**说明：** + >划分较小独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… + >划分较大独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… + +- 软件 + OS：openEuler-aarch64-20.03 LTS + git：2.23.0版本 (建议) + 预留空间：>= 300G + 网络：可以访问互联网 + + >**说明：** + >openEuler 系统安装详细操作请参考[添加测试用例](https://openeuler.org/zh/docs/20.03_LTS/docs/Installati… + +#### 操作指导 +1. 登录 openEuler 系统 + +2. 创建工作目录并设置文件权限 + ```bash + mkdir demo && cd demo && umask 002 + ``` +3. 克隆 compass-ci 项目代码到 demo 目录 + ```bash git clone https://gitee.com/wu_fengguang/compass-ci.git - cd compass-ci/sparrow - ./install-tiny + ``` +4. 执行一键部署脚本 install-tiny + ```bash + cd compass-ci/sparrow && ./install-tiny + ``` +5. 生成lkp-aarch64.cgz压缩包 + ```bash + cd /c/compass-ci/container/lkp-initrd && ./run + ``` +6. 验证账号 + ```bash + cd /c/compass-ci/sbin && ./build-my-info.rb + 按照提示输入账号名和邮箱即可 + +7. 测试环境是否可以提交job测试 + ```bash + submit iperf.yaml testbox=vm-2p8g + ``` + + 执行上述命令正常情况下会提示信息如下： + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + compass-ci搭建完毕，下面就可以开始进行测试了。 + +8. 下载rootfs（以下载openeuler/aarch64/20.03为例，需要在哪个系统上测试就去对应的/srv/os/目录下使用wget命令下载cgz文件包） + ```bash + mkdir -p /srv/os/openeuler/aarch64/ + cd /srv/os/openeuler/aarch64 + wget http://124.90.34.227:11300/os/test/openeuler/aarch64/20.03.cgz + ``` + +9. 解压cgz包 + ```bash + gzip -dc 20.03.cgz | cpio -idv + ``` + +8. [使用 compass-ci 平台测试开源项目](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… + +9. [编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testca… + +10. 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… + 注意：os_mount 必须指定为cifs + +11. 运行测试任务 + ```bash + cd /c/compass-ci/providers/ && ./my-qemu.sh + ``` + +12. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… id查看output文件 + ```bash + cd /srv/result/iperf/$job_id/output + ``` -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] lib/compare_matrixes.rb: don't remove the field that start with $suite
by Lu Kaiyi 21 Dec '20

21 Dec '20

if the metric field start with $suite, even if the field is unchanged, still keep it in compare result. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_matrixes.rb | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index b7ea5a4..0727797 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -176,11 +176,15 @@ def get_matrixes_values(matrixes_list, options) matrixes_values end -def remove_unchanged_field(matrixes_values) +def remove_unchanged_field(matrixes_values, suite_list) # remove unchanged field from matrixes values and remove :changed key # matrixes_values.each_key do |success| matrixes_values[success].delete_if do |field| + if suite_list.any? { |suite| field.start_with?(suite) } + matrixes_values[success][field][:changed] = true + end + !matrixes_values[success][field].delete(:changed) end end @@ -203,7 +207,7 @@ def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key options = { 'perf-profile': 5, theme: :none, no_print: false }.merge(options) matrixes_values = get_matrixes_values(matrixes_list, options) - remove_unchanged_field(matrixes_values) if matrixes_list.length > 1 + remove_unchanged_field(matrixes_values, suite_list) if matrixes_list.length > 1 no_print = options[:no_print] result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/matrix2.rb: discard the no $suite related job
by Lu Kaiyi 21 Dec '20

21 Dec '20

if job has suite, but no $suite related stats metric, then discard this job when create matrix. --- lib/matrix2.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2507b77..f1b2c78 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,10 +76,15 @@ def create_matrix(job_list) matrix = {} suites = [] job_list.each do |job| - suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats + if job['suite'] + next unless stats.keys.any? { |stat| stat.start_with?(job['suite']) } + + suites << job['suite'] + end + stats.each do |key, value| matrix[key] = [] unless matrix[key] matrix[key] << value -- 2.23.0

1 0

[PATCH v2 lkp-tests] lib/monitor: get host from job
by Wu Zhende 21 Dec '20

21 Dec '20

If @job is exist and monitor_url is not configured, set monitor_url based on SCHED_HOST Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index f65cfccf1..283cd788d 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -24,7 +24,6 @@ class Monitor # 1 means timeout exit @exit_status_code = 0 @defaults = {} - load_default @result = [] @stop_query = {} @reason = nil @@ -33,6 +32,10 @@ class Monitor def load_default return unless @monitor_url == '' + if host = @job["SCHED_HOST"] + return @monitor_url = "ws://#{host}:11310/filter" + end + Dir.glob(['/etc/compass-ci/monitor/*.yaml', "#{ENV['HOME']}/.config/compass-ci/monitor/*.yaml"]).each do |file| next unless File.exist? file @@ -98,6 +101,7 @@ class Monitor def run(timeout = nil) merge_overrides + load_default field_check @query.each do |k, v| -- 2.23.0

2 1

Re: [PATCH v2 compass-ci] lib/matrix2.rb remove the *timestamp* from matrix
by Ren Wen 21 Dec '20

21 Dec '20

Why remove ':' after 'lib/matrix2.rb'? Thanks, RenWen Reply-To: In-Reply-To: <20201221022051.3690974-1-luweitaobe(a)163.com> On Mon, Dec 21, 2020 at 10:20:51AM +0800, Lu Weitao wrote: >[Why] >In compare results: >the value of *timestamp*(metric) doesn't mean the number of occurrences, >it may confuse user like bellow: > > openeuler centos archLinux metric >-------------------- ------------------------------ ------------------------------ ------------------------------ > fails:runs change fails:runs change fails:runs > | | | | | > 3585:13 +6913.7% 344:1 -2.6e+4% 11:1 dmesg.timestamp:last > 2350:13 +1.6e+4% 344:1 -1.7e+4% 10:1 kmsg.timestamp:last > 2:13 -15.4% 0:1 +84.6% 1:1 last_state.exit_fail > >[How] >remove the metric that include "timestamp" from matrix > >Signed-off-by: Lu Weitao <luweitaobe(a)163.com> >--- > lib/matrix2.rb | 2 ++ > 1 file changed, 2 insertions(+) > >diff --git a/lib/matrix2.rb b/lib/matrix2.rb >index c712e32..2769f7c 100644 >--- a/lib/matrix2.rb >+++ b/lib/matrix2.rb >@@ -87,6 +87,8 @@ def create_matrix(job_list) > next unless stats > > stats.each do |key, value| >+ next if key.include?('timestamp') >+ > matrix[key] = [] unless matrix[key] > matrix[key] << value > end >-- >2.23.0 >

2 1

[PATCH lkp-tests 2/4] lib/env.sh: move functions to env.sh
by Wei Jihui 21 Dec '20

21 Dec '20

move functions:write_shell_profile and write_host to lib/env.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/env.sh | 30 ++++++++++++++++++++++++++++++ sbin/set-env.sh | 39 --------------------------------------- 2 files changed, 30 insertions(+), 39 deletions(-) delete mode 100755 sbin/set-env.sh diff --git a/lib/env.sh b/lib/env.sh index 8ac42a83..d360f163 100755 --- a/lib/env.sh +++ b/lib/env.sh @@ -125,3 +125,33 @@ shell_profile() [ -f "$profile" ] || touch $profile echo $profile } + +write_shell_profile() +{ + shell_profile_file=$(shell_profile) + if [ $# -gt 0 ]; then + echo $@ >> $shell_profile_file + fi + + source $shell_profile_file +} + +write_host() +{ + if is_system "Linux"; then + nr_cpu=$(nproc) + memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') + else + nr_cpu=$(sysctl -n hw.logicalcpu) + memory_total=$(top -l 1 | grep MemRegions | awk '{print $2}') + fi + memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') + memory=$(echo $memory_new | awk '{print int($0)+1}')G + + cat > hosts/$(hostname) <<-EOF + nr_cpu: $nr_cpu + memory: $memory + hdd_partitions: + ssd_partitions: + EOF +} diff --git a/sbin/set-env.sh b/sbin/set-env.sh deleted file mode 100755 index cc7f9050..00000000 --- a/sbin/set-env.sh +++ /dev/null @@ -1,39 +0,0 @@ -#!/bin/sh - -. $LKP_SRC/lib/env.sh - -write_shell_profile() -{ - shell_profile_file=$(shell_profile) - if [ $# -gt 0 ]; then - echo $@ >> $shell_profile_file - else - echo "export LKP_SRC=$PWD" >> $shell_profile_file - echo "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" >> $shell_profile_file - fi - - source $shell_profile_file -} - -write_host() -{ - if is_system "Linux"; then - nr_cpu=$(nproc) - memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') - else - nr_cpu=$(sysctl -n hw.logicalcpu) - memory_total=$(top -l 1 | grep MemRegions | awk '{print $2}') - fi - memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') - memory=$(echo $memory_new | awk '{print int($0)+1}')G - - cat > hosts/$(hostname) <<-EOF - nr_cpu: $nr_cpu - memory: $memory - hdd_partitions: - ssd_partitions: - EOF -} - -write_shell_profile -write_host -- 2.23.0

2 1

Re: [PATCH compass-ci 2/2] lib/matrix2.rb remove the *timestamp* form matrix
by Ren Wen 21 Dec '20

21 Dec '20

spell: form -> from Thanks, RenWen Reply-To: In-Reply-To: <20201219133430.1143483-2-luweitaobe(a)163.com> On Sat, Dec 19, 2020 at 09:34:30PM +0800, Lu Weitao wrote: >[Why] >In compare results: >the value of *timestamp*(metric) doesn't mean the number of occurrences, >it may confuse user like bellow: > > openeuler centos archLinux metric >-------------------- ------------------------------ ------------------------------ ------------------------------ > fails:runs change fails:runs change fails:runs > | | | | | > 3585:13 +6913.7% 344:1 -2.6e+4% 11:1 dmesg.timestamp:last > 2350:13 +1.6e+4% 344:1 -1.7e+4% 10:1 kmsg.timestamp:last > 2:13 -15.4% 0:1 +84.6% 1:1 last_state.exit_fail > >[How] >remove the metric that include "timestamp" from matrix > >Signed-off-by: Lu Weitao <luweitaobe(a)163.com> >--- > lib/matrix2.rb | 2 ++ > 1 file changed, 2 insertions(+) > >diff --git a/lib/matrix2.rb b/lib/matrix2.rb >index c712e32..2769f7c 100644 >--- a/lib/matrix2.rb >+++ b/lib/matrix2.rb >@@ -87,6 +87,8 @@ def create_matrix(job_list) > next unless stats > > stats.each do |key, value| >+ next if key.include?('timestamp') >+ > matrix[key] = [] unless matrix[key] > matrix[key] << value > end >-- >2.23.0 >

2 1

[PATCH v2 compass-ci] providers/docker: set the private key of the testbox
by Xiao Shenwei 21 Dec '20

21 Dec '20

[why] login the docker testbox without password [how] Add ssh private key for docker, for automatically establish the ssh reverse tunnel between sut and port map container. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/bin/entrypoint.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/providers/docker/bin/entrypoint.sh b/providers/docker/bin/entrypoint.sh index 63cb605..feb5edf 100755 --- a/providers/docker/bin/entrypoint.sh +++ b/providers/docker/bin/entrypoint.sh @@ -2,4 +2,6 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +mkdir -p /root/.ssh +cp /lkp/lkp/src/rootfs/addon/root/.ssh/id_rsa /root/.ssh/ sh /lkp/lkp/src/rootfs/addon/etc/init.d/lkp-bootstrap -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/matrix2.rb: fix rubocop warning
by Lu Weitao 21 Dec '20

21 Dec '20

[Why] lib/matrix2.rb:75:1: C: Metrics/MethodLength: Method has too many lines. [16/15] def create_matrix(job_list) ... Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2507b77..c712e32 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -64,6 +64,12 @@ def samples_fill_missing_zeros(value, size) value.concat([0] * (size - value.size)) end +def matrix_fill_miss_zeros(matrix, col_size) + matrix.each_value do |value| + samples_fill_missing_zeros(value, col_size) + end +end + # input: job_list # return: matrix of Hash(String, Array(Number)) # Eg: matrix: { @@ -86,9 +92,7 @@ def create_matrix(job_list) end end col_size = job_list.size - matrix.each_value do |value| - samples_fill_missing_zeros(value, col_size) - end + matrix_fill_miss_zeros(matrix, col_size) return matrix, suites end -- 2.23.0

2 1

Re: [PATCH compass-ci 2/3] assign-account-email: remove .bash_profile usage
by Luan Shengde 21 Dec '20

21 Dec '20

On Sat, Dec 19, 2020 at 11:23:43AM +0800, Li Ping wrote: >On Sat, Dec 19, 2020 at 11:09:54AM +0800, Wu Fengguang wrote: >>Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> >>--- >> container/mail-robot/lib/assign-account-email.rb | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >>diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb >>index 77ed4e8..ee3b574 100755 >>--- a/container/mail-robot/lib/assign-account-email.rb >>+++ b/container/mail-robot/lib/assign-account-email.rb >>@@ -37,7 +37,7 @@ def build_apply_account_email(my_info) >> git clone https://gitee.com/wu_fengguang/lkp-tests.git >> cd lkp-tests >> make install >>- source ${HOME}/.bashrc && source ${HOME}/.bash_profile >>+ source ${HOME}/.${SHELL##*/}rc > > use ${HOME} will become root in email Use \\${HOME} will resolve this problem. Thanks, Luan Shengde > >> >> 3) submit job >> reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md >>-- >>2.23.0 >>

1 0

[PATCH lkp-tests] tests/build-pkg: add config file support
by Wang Yong 21 Dec '20

21 Dec '20

[Why] config file in some software need storage and recall now we can add config value in yaml and cp ${config} to ${cci_dir}/${benchmark}/ in PKGBUILD so that config file can be uploaded correctly $ cat test.yaml ... config: ${config} ... Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 39 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index addff9f7b..0a143813f 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -5,6 +5,7 @@ # - git_tag # - pkg_ver # upstream_url +# - config # We'll use the below global env vars, but not treat them as parameter to avoid # duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ @@ -20,6 +21,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/distro/common +. $LKP_SRC/lib/http.sh check_vars() { @@ -31,6 +33,29 @@ check_vars() [ -n "$os_mount" ] || die "os_mount is empty" } +check_config_var() +{ + [ -n "$config" ] || return +} + +get_config() +{ + local cci_http_host=${INITRD_HTTP_HOST} + local cci_http_port=${INITRD_HTTP_PORT:-8800} + export cci_dir=/cci/config + local remote_dir="http://${cci_http_host}:${cci_http_port}${cci_dir}/${benchmark}" + + mkdir -p ${cci_dir} + + check_config_var + http_get_file "${remote_dir}/${config}" "${cci_dir}/${benchmark}/${config}" +} + +upload_config() +{ + upload_to_target_dir "${cci_dir}/${benchmark}" +} + mount_dest() { # the same image is mounted to cifs and nfs, the generated cgz files @@ -71,6 +96,13 @@ create_cgz_name() [ -n "$git_tag" ] && { cgz_name="${cgz_path}/${git_tag}.cgz" + # download PKGBUILD-$tag + benchmark="${upstream_repo##*/}" + export benchmark="${benchmark%.git}" + [ -n "$PKGBUILD_TAG" ] && { + get_pkgfile "pkg/$benchmark" "PKGBUILD-$PKGBUILD_TAG" + [ -f "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG not exist" + [ -s "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG is empty" } [ -n "$pkg_ver" ] && { @@ -131,7 +163,11 @@ build_source_pkg() PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf - [ "$?" == 0 ] || exit 1 + local build_state="$?" + + upload_config + + [ "${build_state}" == "0" ] || exit 1 create_softlink upload_to_target_dir ${PKG_MNT}/${pack_to}/${pkgname} } @@ -139,4 +175,5 @@ build_source_pkg() check_vars mount_dest request_pkg +get_config build_source_pkg -- 2.23.0

2 2

[PATCH compass-ci] lib/matrix2.rb: discard the no $suite related job
by Lu Kaiyi 19 Dec '20

19 Dec '20

if job has suite, but no $suite related stats metric, then discard this job when create matrix. --- lib/matrix2.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2507b77..f1b2c78 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,10 +76,15 @@ def create_matrix(job_list) matrix = {} suites = [] job_list.each do |job| - suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats + if job['suite'] + next unless stats.keys.any? { |stat| stat.start_with?(job['suite']) } + + suites << job['suite'] + end + stats.each do |key, value| matrix[key] = [] unless matrix[key] matrix[key] << value -- 2.23.0

2 2

[PATCH compass-ci 2/2] lib/matrix2.rb remove the *timestamp* form matrix
by Lu Weitao 19 Dec '20

19 Dec '20

[Why] In compare results: the value of *timestamp*(metric) doesn't mean the number of occurrences, it may confuse user like bellow: openeuler centos archLinux metric -------------------- ------------------------------ ------------------------------ ------------------------------ fails:runs change fails:runs change fails:runs | | | | | 3585:13 +6913.7% 344:1 -2.6e+4% 11:1 dmesg.timestamp:last 2350:13 +1.6e+4% 344:1 -1.7e+4% 10:1 kmsg.timestamp:last 2:13 -15.4% 0:1 +84.6% 1:1 last_state.exit_fail [How] remove the metric that include "timestamp" from matrix Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index c712e32..2769f7c 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -87,6 +87,8 @@ def create_matrix(job_list) next unless stats stats.each do |key, value| + next if key.include?('timestamp') + matrix[key] = [] unless matrix[key] matrix[key] << value end -- 2.23.0

1 0

[PATCH compass-ci] lib/commit_status.rb: query latest job_id by commit_id
by Lu Kaiyi 19 Dec '20

19 Dec '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/commit_status.rb | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/lib/commit_status.rb b/lib/commit_status.rb index 81aeb29..f1cbfbc 100644 --- a/lib/commit_status.rb +++ b/lib/commit_status.rb @@ -25,6 +25,17 @@ def query_latest_good_commit(job, error_id) return latest_good_commit end +def query_latest_job_id(commit_id) + items = { + 'upstream_commit' => commit_id + } + jobs_list = query_jobs_from_es(items) + return nil if jobs_list.empty? + + jobs_list.sort_by! { |job| job['end_time'] } + jobs_list.last['id'] +end + def query_jobs_from_es(items) es = ESQuery.new(ES_HOST, ES_PORT) result = es.multi_field_query items -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: check load host config from job
by Wei Jihui 19 Dec '20

19 Dec '20

@defaults will merge to @job, and set @default empty. if load host config from @job, so do not need to load host in advance. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 42b008c7..2ed06edc 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -317,17 +317,6 @@ class Job spec_file_context.each_key { |k| @job.delete k } end - def check_load_hosts_config(defaults_config) - return if @load_hosts_done - - if defaults_config.include?('testbox') - @job['tbox_group'] = tbox_group(defaults_config['testbox']) - elsif defaults_config.include?('tbox_group') - @job['tbox_group'] = defaults_config['tbox_group'] - end - return @job.include?('tbox_group') - end - def load_hosts_config return if @job.include?(:no_defaults) return if @load_hosts_done @@ -354,7 +343,7 @@ class Job def get_lab_hosts_file lab_hosts_file_name = @job['testbox'] - lab_hosts_file = "#{ENV['CCI_REPOS']}/lab-#{@defaults['lab']}/hosts/#{lab_hosts_file_name}" + lab_hosts_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{lab_hosts_file_name}" if File.file?(lab_hosts_file) lab_hosts_file @@ -409,7 +398,6 @@ class Job @defaults[source_file_symkey(file)] = nil revise_hash(@defaults, defaults, true) @defaults.merge!(@overrides) - load_hosts_config if check_load_hosts_config(@defaults) end @file_loaded[file] = true true -- 2.23.0

1 0

[PATCH compass-ci] scheduler-https/start: fix missing env key
by Wu Zhende 19 Dec '20

19 Dec '20

Some key don't define in scheduler-https. Synchronize with scheduler. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler-https/start | 34 +++++++++++++++++++++++---------- 1 file changed, 24 insertions(+), 10 deletions(-) diff --git a/container/scheduler-https/start b/container/scheduler-https/start index 9898c1a..98940d7 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -28,17 +28,32 @@ names = Set.new %w[ INITRD_HTTP_PORT TASKQUEUE_PORT TASKQUEUE_HOST + GIT_SERVER + MASTER_FLUENTD_HOST + MASTER_FLUENTD_PORT + SSHR_PORT + SSHR_PORT_BASE + SSHR_PORT_LEN + lab ] defaults = relevant_defaults(names) +defaults['SSHR_PORT'] ||= 5050 +defaults['SSHR_PORT_BASE'] ||= 50000 +defaults['SSHR_PORT_LEN'] ||= 2000 +defaults['SCHED_PORT'] ||= '3000' +defaults['SCHED_HOST'] ||= '172.17.0.1' +defaults['MASTER_FLUENTD_PORT'] ||= '24224' +defaults['SCHED_HTTPS_PORT'] ||= 11306 +defaults['SCHED_HTTPS_HOST'] ||= 'api.compass-ci.openeuler.org' env = docker_env(defaults) -DEFAULT_LKP = '/c/lkp-tests' -DEFAULT_CCI = '/c/compass-ci' -SCHED_PORT = defaults['SCHED_PORT'] || '3000' -SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' -SCHED_HTTPS_PORT = defaults['SCHED_HTTPS_PORT'] || 11306 -SCHED_HTTPS_HOST = defaults['SCHED_HTTPS_HOST'] || 'api.compass-ci.openeuler.org' +CCI_REPOS = ENV['CCI_REPOS'] || '/c' +DEFAULT_LKP = "#{CCI_REPOS}/lkp-tests" +DEFAULT_CCI = "#{CCI_REPOS}/compass-ci" +LAB = defaults['lab'] || 'z9' +SCHED_HTTPS_PORT = defaults['SCHED_HTTPS_PORT'] +SCHED_HTTPS_HOST = defaults['SCHED_HTTPS_HOST'] docker_rm "s001-alpine-#{SCHED_HTTPS_PORT}" @@ -48,14 +63,13 @@ cmd = %W[ --name s001-alpine-#{SCHED_HTTPS_PORT} -d -u 1090:1090 - -p #{SCHED_HTTPS_PORT}:#{SCHED_PORT} + -p #{SCHED_HTTPS_PORT}:#{defaults['SCHED_PORT']} ] + env + %W[ -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} - -e SCHED_HOST=#{SCHED_HOST} - -e SCHED_PORT=#{SCHED_PORT} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} -v /srv/cci/scheduler/alpine:/srv/cci/scheduler -v /etc/localtime:/etc/localtime:ro -v /srv/result:/srv/result @@ -65,7 +79,7 @@ cmd = %W[ --log-opt mode=non-blocking --log-opt max-buffer-size=4m --log-opt tag=scheduler-#{SCHED_HTTPS_PORT} - sch-ruby-a:v0.00d-#{SCHED_PORT} + sch-ruby-a:v0.00d-#{defaults['SCHED_PORT']} ] cmd += ['sh', '-c', 'umask 002 && ./scheduler --ssl --ssl-key-file /ca/server.key --ssl-cert-file /ca/server.crt'] -- 2.23.0

1 0

[PATCH lkp-tests 4/4] install.sh: grant execute permission
by Wei Jihui 19 Dec '20

19 Dec '20

cd lkp-test ./install.sh && source ~/.${SHELL##*/}rc Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- install.sh | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 install.sh diff --git a/install.sh b/install.sh old mode 100644 new mode 100755 -- 2.23.0

1 0

[PATCH lkp-tests 3/4] install.sh: refactor install flow
by Wei Jihui 19 Dec '20

19 Dec '20

1. when exec install.sh, no $LKP_SRC 2. first enable $LKP_SRC env 3. mv sbin/install-dependencies.sh to install.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- Makefile | 2 -- install.sh | 10 +++++++--- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/Makefile b/Makefile index 0b2d09a2..ae062973 100644 --- a/Makefile +++ b/Makefile @@ -14,8 +14,6 @@ subsystem: install: mkdir -p $(TARGET_DIR_BIN) ln -sf $(shell pwd)/bin/lkp $(TARGET_DIR_BIN)/lkp - bash sbin/set-env.sh - bash sbin/install-dependencies.sh .PHONY: doc doc: diff --git a/install.sh b/install.sh index cea1654f..2bd17d54 100644 --- a/install.sh +++ b/install.sh @@ -1,7 +1,11 @@ #!/bin/bash -. $LKP_SRC/lib/env.sh - -make +. lib/env.sh +write_host +write_shell_profile "export LKP_SRC=$PWD" +write_shell_profile "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" source $(shell_profile) + +. sbin/install-dependencies.sh +make install -- 2.23.0

1 0

[PATCH lkp-tests 1/4] bin/install-dependencies.sh: refactor file
by Wei Jihui 19 Dec '20

19 Dec '20

all dependent functions is move to lib/env.sh, sbin/set-env.sh in not required. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/install-dependencies.sh | 40 +++++++++++++++++------------------- 1 file changed, 19 insertions(+), 21 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 7605a29f..b4d3ad22 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -1,30 +1,28 @@ #!/bin/bash . $LKP_SRC/lib/env.sh -. $LKP_SRC/sbin/set-env.sh -# choose install function base on -# DISTRIBUTION +# choose install function base on DISTRIBUTION linux_dep() { -source /etc/os-release -case $ID in -ubuntu|debian) - export DEBIAN_FRONTEND=noninteractive - apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client - ;; -openEuler|fedora|rhel|centos) - if type dnf > /dev/null 2>&1; then - dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - else - yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - fi - sudo gem install -f git activesupport rest-client - ;; -*) - echo "$ID not support! please install dependencies manually." - ;; -esac + source /etc/os-release + case $ID in + ubuntu|debian) + export DEBIAN_FRONTEND=noninteractive + apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client + ;; + openEuler|fedora|rhel|centos) + if type dnf > /dev/null 2>&1; then + dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + else + yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + fi + sudo gem install -f git activesupport rest-client + ;; + *) + echo "$ID not support! please install dependencies manually." + ;; + esac } -- 2.23.0

1 0

[PATCH compass-ci] result-webdav/nginx.conf: create soft connection after package upload
by Cui Yili 19 Dec '20

19 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 8b54314..f077fca 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -37,6 +37,25 @@ http { end) return resultstrlist end + function symbolic_link(path, upload_file) + level_dir = split(path, "/") + if level_dir[1] ~= "initrd" or upload_file == nil then + return + end + if level_dir[2] == "pkg" or level_dir[2] == "build-pkg" then + link_name = "latest" + else + tmp_name = string.match(upload_file, "(.*)_%d+%.cgz") + if not tmp_name then + link_name = tmp_name + else + return + end + end + link_name = link_name..".cgz" + root_path = "/srv/"..path + io.popen("cd "..root_path.."&& ln -sf "..upload_file.." "..link_name) + end redis_host = os.getenv("REDIS_HOST") redis_port = os.getenv("REDIS_PORT") @@ -65,11 +84,13 @@ http { dirs = split(res, ",") for i = 1, #dirs do if string.find(path, dirs[i], 1, true) then + symbolic_link(path, upload_file) return end end else if string.find(path, res, 1, true) then + symbolic_link(path, upload_file) return end end -- 2.23.0

1 0

[PATCH compass-ci] lib/error_messages.rb: fix error when file is not existed
by Lin Jiaxin 19 Dec '20

19 Dec '20

Traceback (most recent call last): 4: from compare_error_messages.rb:63:in `<main>' 3: from compare_error_messages.rb:20:in `get_compare_result' 2: from /home/linjiaxin/compass-ci/lib/error_messages.rb:33:in `obtain_error_messages' 1: from /home/linjiaxin/compass-ci/lib/error_messages.rb:33:in `open' /home/linjiaxin/compass-ci/lib/error_messages.rb:33:in `initialize': No such file or directory @ rb_sysopen - /srv/result/build-pkg/2020-12-16/vm-2p32g.linjiaxin/openeuler-20.03-aarch64/archlinux-community--fcgi-trunk-HEAD/crystal.193087/build-pkg (Errno::ENOENT) Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/lib/error_messages.rb b/lib/error_messages.rb index 5bbc329..61f8070 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -30,12 +30,14 @@ class ErrorMessages end def obtain_error_messages - File.open((a)file).each_line do |line| - next if extract_error_message(line) - next unless @in_stderr - next unless @error_message['error_line'] =~ /error|warning/i + if FileTest.exist?(@file) + File.open((a)file).each_line do |line| + next if extract_error_message(line) + next unless @in_stderr + next unless @error_message['error_line'] =~ /error|warning/i - @error_messages[@error_message['error_line']] << @error_message['error_line'] + @error_message['error_details'] + @error_messages[@error_message['error_line']] << @error_message['error_line'] + @error_message['error_details'] + end end return @error_messages end -- 2.23.0

1 0

[PATCH v2 compass-ci] lib: add compare_error_messages.rb
by Lin Jiaxin 19 Dec '20

19 Dec '20

Provide formatted error messages and new error number for git bisect. formatted error messages example: previous error: fcgiapp.c:1717:22: warning: comparison of integer expressions of different signedness: ‘int’ and ‘long unsigned int’ [-Wsign-compare] 1717 | if(headerLen < sizeof(header)) { | ^ new error: >> libtool: warning: relinking 'libfcgi++.la' Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 59 +++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 lib/compare_error_messages.rb diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb new file mode 100644 index 0000000..0b51fc3 --- /dev/null +++ b/lib/compare_error_messages.rb @@ -0,0 +1,59 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'terminal-table' +require_relative 'es_query' +require_relative 'error_messages' +require_relative "#{ENV['LKP_SRC']}/lib/common" + +def get_compare_result(previous_job_id, later_job_id) + es = ESQuery.new + + previous_error_ids = es.query_by_id(previous_job_id)['error_ids'] + later_es_result = es.query_by_id(later_job_id) + later_error_ids = later_es_result['error_ids'] + later_result_file = File.join('/srv', later_es_result['result_root'], 'build-pkg') + + new_error_ids = get_new_error_ids(later_error_ids, previous_error_ids) + error_messages = ErrorMessages.new(later_result_file).obtain_error_messages + new_error_number, formatted_error_messages = format_error_messages(error_messages, new_error_ids) + + # new error use ">>" as an identifier in the error header + return new_error_number, formatted_error_messages +end + +def get_new_error_ids(later_error_ids, previous_error_ids) + return nil_to_empty_array(later_error_ids) - nil_to_empty_array(previous_error_ids) +end + +def nil_to_empty_array(array) + array = [] if array.nil? + return array +end + +def format_error_messages(error_messages, new_error_ids) + formatted_error_messages = Terminal::Table.new + formatted_error_messages.style = { border_x: '', border_y: '', border_i: '', padding_left: 0 } + new_error_number = 0 + + error_messages.each do |k, v| + if new_error_ids.include?('build-pkg.' + build_pkg_error_id(k)) + new_error_number += 1 + formatted_error_messages = add_sign(formatted_error_messages, '>>', v) + else + formatted_error_messages = add_sign(formatted_error_messages, ' ', v) + end + end + + return new_error_number, formatted_error_messages.to_s +end + +def add_sign(formatted_error_messages, sign, set) + set.each do |value| + formatted_error_messages.add_row([sign, value]) + end + + return formatted_error_messages +end -- 2.23.0

1 0

[PATCH compass-ci] lib: add compare_error_messages.rb
by Lin Jiaxin 19 Dec '20

19 Dec '20

Provide formatted error messages and new error number for git bisect. formatted error messages example: previous error: fcgiapp.c:1717:22: warning: comparison of integer expressions of different signedness: ‘int’ and ‘long unsigned int’ [-Wsign-compare] 1717 | if(headerLen < sizeof(header)) { | ^ new error: >> libtool: warning: relinking 'libfcgi++.la' Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 50 +++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 lib/compare_error_messages.rb diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb new file mode 100644 index 0000000..c96edef --- /dev/null +++ b/lib/compare_error_messages.rb @@ -0,0 +1,50 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'terminal-table' +require_relative 'es_query' +require_relative 'error_messages' +require_relative "#{ENV['LKP_SRC']}/lib/common" + +def get_compare_result(previous_job_id, later_job_id) + es = ESQuery.new + + previous_error_ids = es.query_by_id(previous_job_id)['error_ids'] + later_es_result = es.query_by_id(later_job_id) + later_error_ids = later_es_result['error_ids'] + later_result_file = File.join('/srv', later_es_result['result_root'], 'build-pkg') + + new_error_ids = later_error_ids - previous_error_ids + error_messages = ErrorMessages.new(later_result_file).obtain_error_messages + new_error_number, formatted_error_messages = format_error_messages(error_messages, new_error_ids) + + # new error use ">>" as an identifier in the error header + return new_error_number, formatted_error_messages +end + +def format_error_messages(error_messages, new_error_ids) + formatted_error_messages = Terminal::Table.new + formatted_error_messages.style = { border_x: '', border_y: '', border_i: '', padding_left: 0 } + new_error_number = 0 + + error_messages.each do |k, v| + if new_error_ids.include?('build-pkg.' + build_pkg_error_id(k)) + new_error_number += 1 + formatted_error_messages = add_sign(formatted_error_messages, '>>', v) + else + formatted_error_messages = add_sign(formatted_error_messages, ' ', v) + end + end + + return new_error_number, formatted_error_messages.to_s +end + +def add_sign(formatted_error_messages, sign, set) + set.each do |value| + formatted_error_messages.add_row([sign, value]) + end + + return formatted_error_messages +end -- 2.23.0

1 0

[PATCH v5 compass-ci 2/2] container: http-initrd container support job read /srv/cci/build-config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/initrd-http/root/etc/nginx/conf.d/default.conf | 4 ++++ container/initrd-http/start | 1 + 2 files changed, 5 insertions(+) diff --git a/container/initrd-http/root/etc/nginx/conf.d/default.conf b/container/initrd-http/root/etc/nginx/conf.d/default.conf index f2834c7..18813a2 100644 --- a/container/initrd-http/root/etc/nginx/conf.d/default.conf +++ b/container/initrd-http/root/etc/nginx/conf.d/default.conf @@ -13,4 +13,8 @@ server { location /initrd { autoindex on; } + + location /cci { + autoindex on; + } } diff --git a/container/initrd-http/start b/container/initrd-http/start index ed5d443..8341050 100755 --- a/container/initrd-http/start +++ b/container/initrd-http/start @@ -14,6 +14,7 @@ cmd=( --name initrd-http -p ${INITRD_HTTP_PORT:-8800}:80 -v /srv/initrd:/usr/share/nginx/html/initrd:ro + -v /srv/cci:/usr/share/nginx/html/cci:ro -v /etc/localtime:/etc/localtime:ro -d initrd-http -- 2.23.0

1 0

[PATCH v5 compass-ci 1/2] sched: refactor set_upload_dirs and add config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

When build-pkg job has config in job.yaml need support cci/build-config dir write. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/result-webdav/start | 1 + src/lib/job.cr | 25 ++++++++++++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-) diff --git a/container/result-webdav/start b/container/result-webdav/start index bbbbb24..2852f24 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -21,6 +21,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ + -v /srv/cci/build-config/:/srv/cci/build-config/ result-webdav ) diff --git a/src/lib/job.cr b/src/lib/job.cr index a23475d..4d0b5df 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -233,18 +233,33 @@ class Job set_upload_dirs() end - def set_upload_dirs + def get_package_dir + if @hash["cci-makepkg"]? || @hash["cci-depends"]? || @hash["build-pkg"]? + mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup + common_dir = "#{mount_type}/#{os}/#{os_arch}/#{os_version}" + end + if @hash["cci-makepkg"]? - package_dir = ",/initrd/pkg" + package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" elsif @hash["cci-depends"]? - package_dir = ",/initrd/deps" + package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["build-pkg"]? - package_dir = ",/initrd/build-pkg" + if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] + else + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-1] + end + package_dir = ",/initrd/build-pkg/#{common_dir}/#{package_name}" + package_dir += ",/cci/build-config" if @hash["config"]? else package_dir = "" end - self["upload_dirs"] = "#{result_root}#{package_dir}" + return package_dir + end + + def set_upload_dirs + self["upload_dirs"] = "#{result_root}#{get_package_dir}" end private def set_result_service -- 2.23.0

1 0

[PATCH v4 compass-ci 2/2] container: http-initrd container support job write cci/build-config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/initrd-http/root/etc/nginx/conf.d/default.conf | 4 ++++ container/initrd-http/start | 1 + 2 files changed, 5 insertions(+) diff --git a/container/initrd-http/root/etc/nginx/conf.d/default.conf b/container/initrd-http/root/etc/nginx/conf.d/default.conf index f2834c7..546e824 100644 --- a/container/initrd-http/root/etc/nginx/conf.d/default.conf +++ b/container/initrd-http/root/etc/nginx/conf.d/default.conf @@ -13,4 +13,8 @@ server { location /initrd { autoindex on; } + + location /build-config { + autoindex on; + } } diff --git a/container/initrd-http/start b/container/initrd-http/start index ed5d443..ac8506e 100755 --- a/container/initrd-http/start +++ b/container/initrd-http/start @@ -14,6 +14,7 @@ cmd=( --name initrd-http -p ${INITRD_HTTP_PORT:-8800}:80 -v /srv/initrd:/usr/share/nginx/html/initrd:ro + -v /srv/cci/build-config:/usr/share/nginx/html/build-config:ro -v /etc/localtime:/etc/localtime:ro -d initrd-http -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] sched: refactor set_upload_dirs and add config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

When build-pkg job has config in job.yaml need support cci/build-config dir write. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/result-webdav/start | 1 + src/lib/job.cr | 25 ++++++++++++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-) diff --git a/container/result-webdav/start b/container/result-webdav/start index bbbbb24..2852f24 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -21,6 +21,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ + -v /srv/cci/build-config/:/srv/cci/build-config/ result-webdav ) diff --git a/src/lib/job.cr b/src/lib/job.cr index a23475d..4d0b5df 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -233,18 +233,33 @@ class Job set_upload_dirs() end - def set_upload_dirs + def get_package_dir + if @hash["cci-makepkg"]? || @hash["cci-depends"]? || @hash["build-pkg"]? + mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup + common_dir = "#{mount_type}/#{os}/#{os_arch}/#{os_version}" + end + if @hash["cci-makepkg"]? - package_dir = ",/initrd/pkg" + package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" elsif @hash["cci-depends"]? - package_dir = ",/initrd/deps" + package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["build-pkg"]? - package_dir = ",/initrd/build-pkg" + if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] + else + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-1] + end + package_dir = ",/initrd/build-pkg/#{common_dir}/#{package_name}" + package_dir += ",/cci/build-config" if @hash["config"]? else package_dir = "" end - self["upload_dirs"] = "#{result_root}#{package_dir}" + return package_dir + end + + def set_upload_dirs + self["upload_dirs"] = "#{result_root}#{get_package_dir}" end private def set_result_service -- 2.23.0

1 0

[PATCH compass-ci] doc: update documents
by Luan Shengde 19 Dec '20

19 Dec '20

update documents: my_uuid => my_token Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 6 +++--- doc/manual/apply-account.md | 2 +- ...43\205compass-ci\345\256\242\346\210\267\347\253\257.md" | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/mail-robot.md b/doc/mail-robot.md index 06e4e87..1e8d618 100644 --- a/doc/mail-robot.md +++ b/doc/mail-robot.md @@ -58,14 +58,14 @@ attahcment first attachment - apply_my_account - my_uuid - generate uuid + my_token + generate my_token apply_account invoke ApplyJumperAccount to apply new account with my_info and pub_key my_info: - my_email - my_name - - my_uuid + - my_token apply_jumper_account required data: pub_key complete my_info diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index d9a03c8..8ae6c6d 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -25,7 +25,7 @@ Which contains account information for you: my_email my_name - my_uuid + my_token ## step 3: one-time setup diff --git "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" index 5a02431..160fdd0 100644 --- "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" +++ "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" @@ -14,7 +14,7 @@ Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客 ```SHELL mkdir -p ~/.config/compass-ci/defaults/ cat >> ~/.config/compass-ci/defaults/${USER}.yaml <<-EOF - my_uuid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx + my_token: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx my_email: xxx my_name: xx EOF -- 2.23.0

2 2

[PATCH v3 compass-ci] sched: refactor set_upload_dirs and add config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

When build-pkg job has config in job.yaml need support cci/build-config dir write. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/result-webdav/start | 1 + src/lib/job.cr | 25 ++++++++++++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-) diff --git a/container/result-webdav/start b/container/result-webdav/start index bbbbb24..2852f24 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -21,6 +21,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ + -v /srv/cci/build-config/:/srv/cci/build-config/ result-webdav ) diff --git a/src/lib/job.cr b/src/lib/job.cr index a23475d..d5e8ad5 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -233,18 +233,33 @@ class Job set_upload_dirs() end - def set_upload_dirs + def get_package_dir + if @hash["cci-makepkg"]? || @hash["cci-depends"]? || @hash["build-pkg"]? + mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup + common_dir = "#{mount_type}/#{os}/#{os_arch}/#{os_version}" + end + if @hash["cci-makepkg"]? - package_dir = ",/initrd/pkg" + package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" elsif @hash["cci-depends"]? - package_dir = ",/initrd/deps" + package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["build-pkg"]? - package_dir = ",/initrd/build-pkg" + if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] + else + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-1] + end + package_dir = ",/initrd/build-pkg/#{common_dir}/#{package_name}" + package_dir += ",/cci/config" if @hash["config"]? else package_dir = "" end - self["upload_dirs"] = "#{result_root}#{package_dir}" + return package_dir + end + + def set_upload_dirs + self["upload_dirs"] = "#{result_root}#{get_package_dir}" end private def set_result_service -- 2.23.0

1 1

[PATCH v2 compass-ci] sched: refactor set_upload_dirs and add config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

When build-pkg job has config in job.yaml need support cci/build-config dir write. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/result-webdav/start | 1 + src/lib/job.cr | 25 ++++++++++++++++++++----- 2 files changed, 21 insertions(+), 5 deletions(-) diff --git a/container/result-webdav/start b/container/result-webdav/start index bbbbb24..a719806 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -21,6 +21,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ + -v /srv/cci/config/:/srv/cci/config/ result-webdav ) diff --git a/src/lib/job.cr b/src/lib/job.cr index a23475d..d5e8ad5 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -233,18 +233,33 @@ class Job set_upload_dirs() end - def set_upload_dirs + def get_package_dir + if @hash["cci-makepkg"]? || @hash["cci-depends"]? || @hash["build-pkg"]? + mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup + common_dir = "#{mount_type}/#{os}/#{os_arch}/#{os_version}" + end + if @hash["cci-makepkg"]? - package_dir = ",/initrd/pkg" + package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" elsif @hash["cci-depends"]? - package_dir = ",/initrd/deps" + package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["build-pkg"]? - package_dir = ",/initrd/build-pkg" + if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] + else + package_name = @hash["pkgbuild_repo"].to_s.split("/")[-1] + end + package_dir = ",/initrd/build-pkg/#{common_dir}/#{package_name}" + package_dir += ",/cci/config" if @hash["config"]? else package_dir = "" end - self["upload_dirs"] = "#{result_root}#{package_dir}" + return package_dir + end + + def set_upload_dirs + self["upload_dirs"] = "#{result_root}#{get_package_dir}" end private def set_result_service -- 2.23.0

1 1

[PATCH compass-ci] modify options for auto submit api-avx2neon.yaml
by Li Ping 19 Dec '20

19 Dec '20

Signed-off-by: Li Ping <1477412247(a)qq.com> --- sbin/auto_submit.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index d0f4fbe..e1eb39b 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -1,5 +1,5 @@ a/AvxToNeon/AvxToNeon: -- testbox=vm-2p8g os=centos os_version=7.6.1810 os_mount=initramfs os_arch=aarch64 api-avx2neon.yaml +- testbox=vm-2p8g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 api-avx2neon.yaml c/compass-ci/compass-ci: - testbox=taishan200-2280-2s64p-256g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml - testbox=taishan200-2280-2s64p-256g os=centos os_version=7.6.1810 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml -- 2.23.0

1 0

[PATCH v1 compass-ci] sched: refactor set_upload_dirs and add config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

When build-pkg job has config in job.yaml need support cci/config dir read and write. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/result-webdav/start | 1 + src/lib/job.cr | 19 ++++++++++++++----- 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/container/result-webdav/start b/container/result-webdav/start index bbbbb24..a719806 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -21,6 +21,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ + -v /srv/cci/config/:/srv/cci/config/ result-webdav ) diff --git a/src/lib/job.cr b/src/lib/job.cr index a23475d..85159f5 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -233,18 +233,27 @@ class Job set_upload_dirs() end - def set_upload_dirs + def get_package_dir + if @hash["cci-makepkg"]? or @hash["cci-depends"]? or @hash["build-pkg"]? + common_dir = "#{os_mount}/#{os}/#{os_arch}/#{os_version}" + end + if @hash["cci-makepkg"]? - package_dir = ",/initrd/pkg" + package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" elsif @hash["cci-depends"]? - package_dir = ",/initrd/deps" + package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["build-pkg"]? - package_dir = ",/initrd/build-pkg" + package_dir = ",/initrd/build-pkg/#{common_dir}/#{@hash["pkgbuild_repo"].split("/")[-1]}" + package_dir += ",/cci/config" if @hash["config"]? else package_dir = "" end - self["upload_dirs"] = "#{result_root}#{package_dir}" + return package_dir + end + + def set_upload_dirs + self["upload_dirs"] = "#{result_root}#{get_package_dir}" end private def set_result_service -- 2.23.0

2 3

[PATCH lkp-tests] lib/monitor: get host from job
by Wu Zhende 19 Dec '20

19 Dec '20

If @job is exist and monitor_url is not configured, set monitor_url based on SCHED_HOST Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index f65cfccf1..283cd788d 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -24,7 +24,6 @@ class Monitor # 1 means timeout exit @exit_status_code = 0 @defaults = {} - load_default @result = [] @stop_query = {} @reason = nil @@ -33,6 +32,10 @@ class Monitor def load_default return unless @monitor_url == '' + if host = @job["SCHED_HOST"] + return @monitor_url = "ws://#{host}:11310/filter" + end + Dir.glob(['/etc/compass-ci/monitor/*.yaml', "#{ENV['HOME']}/.config/compass-ci/monitor/*.yaml"]).each do |file| next unless File.exist? file @@ -98,6 +101,7 @@ class Monitor def run(timeout = nil) merge_overrides + load_default field_check @query.each do |k, v| -- 2.23.0

1 0

[PATCH compass-ci] modify assign-account-email
by Li Ping 19 Dec '20

19 Dec '20

[why]: source /root/.bashrc && /root/.bash_profile follow this step can not success. Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/mail-robot/lib/assign-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index 77ed4e8..dccbd40 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -37,7 +37,7 @@ def build_apply_account_email(my_info) git clone https://gitee.com/wu_fengguang/lkp-tests.git cd lkp-tests make install - source ${HOME}/.bashrc && source ${HOME}/.bash_profile + source ~/.${SHELL##*/}rc 3) submit job reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md -- 2.23.0

2 1

[PATCH compass-ci] sched: refactor set_upload_dirs and add config dir
by Cao Xueliang 19 Dec '20

19 Dec '20

When build-pkg job has config in job.yaml need support cci/config dir read and write. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/result-webdav/start | 1 + src/lib/job.cr | 17 ++++++++++++----- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/container/result-webdav/start b/container/result-webdav/start index bbbbb24..a719806 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -21,6 +21,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ + -v /srv/cci/config/:/srv/cci/config/ result-webdav ) diff --git a/src/lib/job.cr b/src/lib/job.cr index a23475d..5644ddf 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -233,18 +233,25 @@ class Job set_upload_dirs() end - def set_upload_dirs + def get_package_dir + if @hash["cci-makepkg"]? or @hash["cci-depends"]? or @hash["build-pkg"]? + common_dir = "#{os_mount}/#{os}/#{os_arch}/#{os_version}" + end + if @hash["cci-makepkg"]? - package_dir = ",/initrd/pkg" + package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" elsif @hash["cci-depends"]? - package_dir = ",/initrd/deps" + package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["build-pkg"]? - package_dir = ",/initrd/build-pkg" + package_dir = ",/initrd/build-pkg/#{common_dir}/#{@hash["pkgbuild_repo"].split("/")[-1]}" + package_dir += ",/cci/config" if @hash["config"]? else package_dir = "" end + end - self["upload_dirs"] = "#{result_root}#{package_dir}" + def set_upload_dirs + self["upload_dirs"] = "#{result_root}#{get_package_dir}" end private def set_result_service -- 2.23.0

1 1

[PATCH compass-ci 3/3] rename my_uuid to my_token
by Wu Fengguang 19 Dec '20

19 Dec '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/mail-robot.md | 6 +++--- doc/manual/apply-account.md | 4 ++-- sbin/build-my-info.rb | 6 +++--- sbin/es-accounts-mapping.sh | 2 +- src/delimiter/utils.rb | 4 ++-- 5 files changed, 11 insertions(+), 11 deletions(-) diff --git a/doc/mail-robot.md b/doc/mail-robot.md index 06e4e87..d3778e7 100644 --- a/doc/mail-robot.md +++ b/doc/mail-robot.md @@ -58,14 +58,14 @@ attahcment first attachment - apply_my_account - my_uuid + my_token generate uuid apply_account invoke ApplyJumperAccount to apply new account with my_info and pub_key my_info: - - my_email - my_name - - my_uuid + - my_email + - my_token apply_jumper_account required data: pub_key complete my_info diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index d9a03c8..7e1a6a6 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -23,9 +23,9 @@ Email template: Which contains account information for you: - my_email my_name - my_uuid + my_email + my_token ## step 3: one-time setup diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index 73f7390..d75f767 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -12,12 +12,12 @@ print 'email: ' my_email = $stdin.echo = gets.chomp print 'name: ' my_name = $stdin.echo = gets.chomp -my_uuid = %x(uuidgen).chomp +my_token = %x(uuidgen).chomp my_info = { - 'my_email' => my_email, 'my_name' => my_name, - 'my_uuid' => my_uuid + 'my_email' => my_email, + 'my_token' => my_token } def store_account_info(my_info) diff --git a/sbin/es-accounts-mapping.sh b/sbin/es-accounts-mapping.sh index d5566dc..d04b0f3 100755 --- a/sbin/es-accounts-mapping.sh +++ b/sbin/es-accounts-mapping.sh @@ -18,7 +18,7 @@ else "my_commit_url": { "type": "keyword" }, - "my_uuid": { + "my_token": { "type": "keyword" }, "my_email": { diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index e87bbbe..41803f3 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -101,9 +101,9 @@ module Utils raise "query #{DELIMITER_ACCONUT} account info failed!" unless account_info job['suite'] = 'bisect' - job['my_uuid'] = account_info['my_uuid'] job['my_name'] = account_info['my_name'] job['my_email'] = account_info['my_email'] + job['my_token'] = account_info['my_token'] job['bad_job_id'] = job_id job.delete('error_ids') @@ -127,7 +127,7 @@ module Utils def create_bisect_log(git_dir) FileUtils.mkdir_p TMP_RESULT_ROOT unless File.exist? TMP_RESULT_ROOT - log_file = File.join(TMP_RESULT_ROOT, "bisect.log") + log_file = File.join(TMP_RESULT_ROOT, 'bisect.log') log_content = parse_bisect_log(git_dir) File.open(log_file, 'w') do |f| log_content.each { |line| f.puts(line) } -- 2.23.0

1 0

[PATCH compass-ci 2/3] assign-account-email: remove .bash_profile usage
by Wu Fengguang 19 Dec '20

19 Dec '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- container/mail-robot/lib/assign-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index 77ed4e8..ee3b574 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -37,7 +37,7 @@ def build_apply_account_email(my_info) git clone https://gitee.com/wu_fengguang/lkp-tests.git cd lkp-tests make install - source ${HOME}/.bashrc && source ${HOME}/.bash_profile + source ${HOME}/.${SHELL##*/}rc 3) submit job reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md -- 2.23.0

1 0

[PATCH compass-ci 1/3] doc/manual: improve install client document
by Wu Fengguang 19 Dec '20

19 Dec '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- ...ci\345\256\242\346\210\267\347\253\257.md" | 27 +++++++------------ 1 file changed, 10 insertions(+), 17 deletions(-) diff --git "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" index 5a02431..dd115f7 100644 --- "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" +++ "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" @@ -2,32 +2,25 @@ Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客户端，通过本地安装 lkp-tests 可以手动提交测试任务 +## 申请帐号 + :exclamation: 前提条件：按照 [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… 完成如下动作： - send apply account email - receive email from compass-ci(a)qq.com +- get reply email and follow its instructions to + - setup default config -## Getting started +## 下载安装 lkp-tests -1. setup default config +Run the following command to install/setup lkp-test: - run the following command to add the below setup to default config file -```SHELL - mkdir -p ~/.config/compass-ci/defaults/ - cat >> ~/.config/compass-ci/defaults/${USER}.yaml <<-EOF - my_uuid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx - my_email: xxx - my_name: xx - EOF -``` -2. download lkp-tests and dependencies - - run the following command to install the lkp-test and effect the configuration ```SHELL git clone https://gitee.com/wu_fengguang/lkp-tests.git cd lkp-tests make install - source ~/.bashrc && source ~/.bash_profile + source ~/.${SHELL##*/}rc ``` -3. submit job - Now try [submitting a job to compass-ci](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… +## submit job + +Now try [submitting a job to compass-ci](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… -- 2.23.0

1 0

[PATCH compass-ci] fetch-mail: update Dockerfile
by Luan Shengde 19 Dec '20

19 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/Dockerfile | 1 + 1 file changed, 1 insertion(+) diff --git a/container/fetch-mail/Dockerfile b/container/fetch-mail/Dockerfile index 0434e09..99abf86 100644 --- a/container/fetch-mail/Dockerfile +++ b/container/fetch-mail/Dockerfile @@ -5,6 +5,7 @@ FROM centos:latest MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive +RUN groupadd -g 1147 mailer RUN useradd -u 1144 -g 1147 mailer RUN yum update -y RUN yum install -y procmail fetchmail -- 2.23.0

1 0

[PATCH lkp-tests] daemon: fix ssh login testbox for docker
by Xiao Shenwei 18 Dec '20

18 Dec '20

[why] systemctl is not supported when use docker [how] use the sshd command Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- daemon/sshd | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/daemon/sshd b/daemon/sshd index 41fa99adf..6bfda467e 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -6,19 +6,31 @@ # sshr_port_base . $LKP_SRC/lib/http.sh +. $LKP_SRC/lib/env.sh sshr_ip=$LKP_SERVER [ -n "$sshr_port" ] || sshr_port=5050 [ -n "$sshr_port_len" ] || sshr_port_len=2000 [ -n "$sshr_port_base" ] || sshr_port_base=50000 +generate_key_file() +{ + echo -e "y\n" | ssh-keygen -t rsa -b 2048 -N '' -f /etc/ssh/ssh_host_rsa_key +} + run_ssh() { [ -n "$my_ssh_pubkey" ] || return umask 0077 mkdir -p /root/.ssh echo "$my_ssh_pubkey" > /root/.ssh/authorized_keys - systemctl start sshd + if is_docker; then + generate_key_file + sshd_path=$(cmd_path sshd) + $sshd_path -q + else + systemctl start sshd + fi } data_success() -- 2.23.0

1 0

[PATCH compass-ci] providers/docker: set the private key of the testbox
by Xiao Shenwei 18 Dec '20

18 Dec '20

[why] ssh password-free login for testbox of docker [how] Add ssh private key for docker, for automatically establish the ssh reverse tunnel between sut and port map container. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/bin/entrypoint.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/providers/docker/bin/entrypoint.sh b/providers/docker/bin/entrypoint.sh index 63cb605..feb5edf 100755 --- a/providers/docker/bin/entrypoint.sh +++ b/providers/docker/bin/entrypoint.sh @@ -2,4 +2,6 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +mkdir -p /root/.ssh +cp /lkp/lkp/src/rootfs/addon/root/.ssh/id_rsa /root/.ssh/ sh /lkp/lkp/src/rootfs/addon/etc/init.d/lkp-bootstrap -- 2.23.0

1 0

help star
by Wu Fengguang 18 Dec '20

18 Dec '20

Wei Jihui is our help star, thanks Wei Jihui! 24.0 Wu Fengguang 22.8 Wei Jihui 21.4 Yu Chuan 18.7 Cao Xueliang 12.0 Wu Zhende 10.6 Xiao Shenwei 9.2 Lu Weitao 8.5 Hu Xuejiao 7.8 Wang Yong 7.4 Liu Shaofei 6.4 Luan Shengde 5.9 Xu Xijian 5.7 Cui Yili 5.3 Zhang Yu 5.3 Zhang Dewan 5.2 Zhang Yale 5.0 Bai Jing 4.8 Liu Yinsi 4.4 Lin Jiaxin 4.3 Li Ping 4.0 Wang Chenglong 2.7 Sun Yukui 2.6 Lu Kaiyi 2.4 Ren Wen 2.2 Du Kaitian 1.3 Li Yuanchao 0.0 Tong Qunfeng 0.0 Wang Shuai 0.0 Cui Xucui

1 0

git count
by Wu Fengguang 18 Dec '20

18 Dec '20

Shengde is our code star again, thanks Shengde! wfg@crystal ~/todo% bin/git-count 14 Luan Shengde 13 Cao Xueliang 12 Wu Fengguang 10 Liu Yinsi 10 Sun Yukui 10 Wu Zhende 8 Ren Wen 7 Zhang Yu 7 Li Yuanchao 6 Cui Yili 6 Wang Chenglong 6 Lu Kaiyi 5 Liu Shaofei 5 Lu Weitao 5 Wei Jihui 5 Yu Chuan 5 Wang Yong 4 Hu Xuejiao 3 Bai Jing 3 Xu Xijian 3 Zhang Yale 3 Lin Jiaxin 3 Li Ping 2 Xiao Shenwei 0 Tong Qunfeng 0 Du Kaitian 0 Zhang Dewan

1 0

review star
by Wu Fengguang 18 Dec '20

18 Dec '20

Xueliang is our review star, thanks Xueliang! wfg@crystal ~/todo% bin/review-star Review email records in the last 7 days total review paragraph point people 121 21 24 24 Cao Xueliang 48 30 57 20 Wu Fengguang 74 16 23 15 Wei Jihui 69 7 20 10 Luan Shengde 40 13 11 7 Liu Shaofei 42 14 20 7 Liu Yinsi 29 4 13 5 Yu Chuan 8 2 6 4 Xu Xijian 36 8 7 4 Lu Kaiyi 26 4 5 4 Xiao Shenwei 36 6 4 3 Zhang Yu 22 6 7 3 Lin Jiaxin 13 4 12 2 Sun Yukui 21 5 12 2 Ren Wen 14 4 2 2 Cui Yili 6 2 1 1 Bai Jing 18 3 2 1 Hu Xuejiao 30 4 4 1 Wang Chenglong 38 1 1 1 Wu Zhende 7 2 0 0 Zhang Yale 15 3 4 0 Li Ping 9 0 0 0 Li Yuanchao 6 2 2 0 Zhang Dewan 10 0 0 0 Wang Yong 25 2 4 0 Lu Weitao

1 0

[PATCH compass-ci] lib/es_query: add query filed from index
by Lu Weitao 18 Dec '20

18 Dec '20

[Why] We need only get specific field instead of entire document. it's like SQL: SELECT DISTINCT column FROM table Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/es_query.rb | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 3bea624..86c078f 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -61,6 +61,30 @@ class ESQuery return result[0]['_source'] end + + # select doc_field from index + # input: + # eg: suite (@index: jobs) + # output: + # [ + # {"key"=>"build-pkg", "doc_count"=>90841}, + # {"key"=>"cci-depends", "doc_count"=>4636}, + # {"key"=>"cci-makepkg", "doc_count"=>3647}, + # ... + # ] + def query_specific_fields(field) + query = { + aggs: { + "all_#{field}" => { + terms: { field: field, size: 1000 } + } + }, + size: 0 + } + result = @client.search index: @index + '*', body: query + + result ['aggregations']["all_#{field}"]['buckets'] + end end # Range Query Example: -- 2.23.0

1 1

[PATCH compass-ci] mail-robot/assign-account-email: add config for lab
by Wei Jihui 18 Dec '20

18 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- container/mail-robot/lib/assign-account-email.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index 834e96f..77ed4e8 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -24,6 +24,7 @@ def build_apply_account_email(my_info) cat >> ~/.config/compass-ci/defaults/account.yaml <<-EOF my_email: #{my_info['my_email']} my_name: #{my_info['my_name']} + lab: #{ENV['lab']} EOF mkdir -p ~/.config/compass-ci/include/lab cat >> ~/.config/compass-ci/include/lab/#{ENV['lab']}.yaml <<-EOF -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/2] distro/adaptation: add sshd mapping
by Wang Yong 18 Dec '20

18 Dec '20

add sshd mapping in different OS Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/adaptation/archlinux | 2 ++ distro/adaptation/centos | 1 + distro/adaptation/fedora | 1 + distro/adaptation/kylin | 2 ++ distro/adaptation/openeuler | 2 ++ distro/adaptation/opensuse | 1 + distro/adaptation/oracle | 1 + 7 files changed, 10 insertions(+) diff --git a/distro/adaptation/archlinux b/distro/adaptation/archlinux index 53474d6c4..89163cf41 100644 --- a/distro/adaptation/archlinux +++ b/distro/adaptation/archlinux @@ -70,3 +70,5 @@ xutils-dev: imake xz-utils: xz libtirpc: libtirpc1 initramfs-tools: +openssh-server: openssh +openssh-client: openssh diff --git a/distro/adaptation/centos b/distro/adaptation/centos index e7fabf36d..2cfe3877c 100644 --- a/distro/adaptation/centos +++ b/distro/adaptation/centos @@ -334,3 +334,4 @@ junit4: junit liboath-dev: liboath-devel lsb-release: redhat-lsb linux-base: +openssh-client: openssh-clients diff --git a/distro/adaptation/fedora b/distro/adaptation/fedora index 38b7e9dc4..6686bc8c2 100644 --- a/distro/adaptation/fedora +++ b/distro/adaptation/fedora @@ -162,3 +162,4 @@ network-manager: NetworkManager javahelper: junit4: junit linux-base: +openssh-client: openssh-clients diff --git a/distro/adaptation/kylin b/distro/adaptation/kylin index f143fde96..e9286effd 100644 --- a/distro/adaptation/kylin +++ b/distro/adaptation/kylin @@ -147,3 +147,5 @@ lsb-release: openjdk: java-1.8.0-openjdk ruby-full: ruby ruby-devel linux-base: +openssh-server: openssh +openssh-client: openssh diff --git a/distro/adaptation/openeuler b/distro/adaptation/openeuler index e72f34fd2..09120489b 100644 --- a/distro/adaptation/openeuler +++ b/distro/adaptation/openeuler @@ -293,3 +293,5 @@ lsb-release: openjdk: java-1.8.0-openjdk ruby-full: ruby ruby-devel rubygem-rdoc linux-base: +openssh-server: openssh +openssh-client: openssh diff --git a/distro/adaptation/opensuse b/distro/adaptation/opensuse index 868d06b0d..f061c0894 100644 --- a/distro/adaptation/opensuse +++ b/distro/adaptation/opensuse @@ -13,3 +13,4 @@ libtirpc: libtirpc3 # Go get it yourself: https://github.com/schacon/ruby-git # We have no package for it yet ruby-git: +openssh-client: openssh-clients diff --git a/distro/adaptation/oracle b/distro/adaptation/oracle index af8415ec3..522b3d869 100644 --- a/distro/adaptation/oracle +++ b/distro/adaptation/oracle @@ -135,3 +135,4 @@ initramfs-tools: init-system-helpers: javahelper: linux-base: +openssh-client: openssh-clients -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] distro/depends: add sshd depends file
by Wang Yong 18 Dec '20

18 Dec '20

add depends file for sshd Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/depends/sshd | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 distro/depends/sshd diff --git a/distro/depends/sshd b/distro/depends/sshd new file mode 100644 index 000000000..85ba955ae --- /dev/null +++ b/distro/depends/sshd @@ -0,0 +1,2 @@ +openssh-server +openssh-client -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: fix error for submit remote job
by Wu Zhende 18 Dec '20

18 Dec '20

[Error] submit iperf.yaml failed, got job_id=0, error: Missing uuid for remote job [How] Don't need to check uuid for remote job now. All jobs use my_token Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index e7b9faf..dc43cbf 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -185,18 +185,6 @@ class Job # handle by me, then keep connect to me self["LKP_SERVER"] = SCHED_HOST self["LKP_CGI_PORT"] = SCHED_PORT.to_s - - # need further uuid check (validate? exist? no need?) - if self["SCHED_HOST"] != SCHED_HOST # remote submited job - # ?further fix to 127.0.0.1 (from remote ssh port forwarding) - # ?even set self["SCHED_HOST"] and self["SCHED_PORT"] - - if self["uuid"] == "" - puts "Job's SCHED_HOST is #{self["SCHED_HOST"]}, " + - "current scheduler IP is: #{SCHED_HOST}" - raise "Missing uuid for remote job" - end - end end private def set_sshr_info -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: add reference url
by Wu Zhende 18 Dec '20

18 Dec '20

submit iperf.yaml testbox=vm-2p8g submit iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_token'. Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index e7b9faf..e046072 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -325,7 +325,11 @@ class Job private def check_required_keys REQUIRED_KEYS.each do |key| if !@hash[key]? - raise "Missing required job key: '#{key}'" + error_msg = "Missing required job key: '#{key}'." + if ["my_email", "my_name", "my_token"].includes?(key) + error_msg += "\nPlease refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco…" + end + raise error_msg end end end -- 2.23.0

1 0

[PATCH compass-ci] container/initrd-http: support "/srv/cci/" dir download
by Cao Xueliang 18 Dec '20

18 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/initrd-http/root/etc/nginx/conf.d/default.conf | 4 ++++ container/initrd-http/start | 1 + 2 files changed, 5 insertions(+) diff --git a/container/initrd-http/root/etc/nginx/conf.d/default.conf b/container/initrd-http/root/etc/nginx/conf.d/default.conf index f2834c7..18813a2 100644 --- a/container/initrd-http/root/etc/nginx/conf.d/default.conf +++ b/container/initrd-http/root/etc/nginx/conf.d/default.conf @@ -13,4 +13,8 @@ server { location /initrd { autoindex on; } + + location /cci { + autoindex on; + } } diff --git a/container/initrd-http/start b/container/initrd-http/start index ed5d443..8341050 100755 --- a/container/initrd-http/start +++ b/container/initrd-http/start @@ -14,6 +14,7 @@ cmd=( --name initrd-http -p ${INITRD_HTTP_PORT:-8800}:80 -v /srv/initrd:/usr/share/nginx/html/initrd:ro + -v /srv/cci:/usr/share/nginx/html/cci:ro -v /etc/localtime:/etc/localtime:ro -d initrd-http -- 2.23.0

1 0

[PATCH v2 compass-ci] doc: and runtime field in borrow-machine.en.md and borrow-machine.zh.md
by Zhang Yale 18 Dec '20

18 Dec '20

User can specify the sleep time in "runtime" field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/manual/borrow-machine.en.md | 11 ++++++----- doc/manual/borrow-machine.zh.md | 11 ++++++----- 2 files changed, 12 insertions(+), 10 deletions(-) diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index b9c2bc4..0277984 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -202,7 +202,8 @@ Ensure that you have performed the following operations according to the [apply- hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml suite: borrow testcase: borrow - + + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -212,10 +213,10 @@ Ensure that you have performed the following operations according to the [apply- %> sshd: # sleep at the bottom - sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # Use the VIM editor to change the value of the sleep field. + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml + runtime: 1h + # Use the VIM editor to change the value of the runtime field. hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # After changing the value, submit the request again. hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 06d4db6..5bc951d 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -206,7 +206,8 @@ hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml suite: borrow testcase: borrow - + + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -216,10 +217,10 @@ %> sshd: # sleep at the bottom - sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # ��vim��sleep�� + runtime: 1h + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml + runtime: 1h + # ��vim��runtime�� hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # ��submit�� hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/2] tests/build-pkg: disable 2>&1 when run makepkg
by Lin Jiaxin 18 Dec '20

18 Dec '20

Make error will redirect itself to stdout now. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 59625299..addff9f7 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -129,7 +129,7 @@ build_source_pkg() create_cgz_name PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ - $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf 2>&1 + $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf [ "$?" == 0 ] || exit 1 create_softlink -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] revoke commit: 2dad3102dc5c11d02e66eab9840bcdffb11e93f0
by Lin Jiaxin 18 Dec '20

18 Dec '20

[Why] We modify the make source code let it's stderr to stdout when compile error. So no need for this commit. before: all stderr => stdout => file: build-pkg => stats/build-pkg -- when run makepkg But stats/build-pkg can only analyze make error. So it ignores some other errors. after: unknown error => stderr => file: stderr => stats/stderr make stderr => stdout => file: build-pkg => stats/build-pkg Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/makepkg | 18 +++--------------- tests/build-pkg | 4 ++-- 2 files changed, 5 insertions(+), 17 deletions(-) diff --git a/sbin/makepkg b/sbin/makepkg index 131ae690..498b4872 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -1761,19 +1761,11 @@ run_build() { [[ -d /usr/lib/ccache/bin ]] && export PATH="/usr/lib/ccache/bin:$PATH" fi - if [[ $STDERR_REDIR == y ]];then - run_function_safe "build" 2>&1 - else - run_function_safe "build" - fi + run_function_safe "build" } run_check() { - if [[ $STDERR_REDIR == y ]];then - run_function_safe "check" 2>&1 - else - run_function_safe "check" - fi + run_function_safe "check" } run_package() { @@ -1784,11 +1776,7 @@ run_package() { pkgfunc="package_$1" fi - if [[ $STDERR_REDIR == y ]];then - run_function_safe "$pkgfunc" 2>&1 - else - run_function_safe "$pkgfunc" - fi + run_function_safe "$pkgfunc" } build_id() { diff --git a/tests/build-pkg b/tests/build-pkg index dd1ee120..59625299 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -128,8 +128,8 @@ build_source_pkg() create_cgz_name - PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" STDERR_REDIR=y\ - $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf + PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ + $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf 2>&1 [ "$?" == 0 ] || exit 1 create_softlink -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: rename my_uuid to my_token
by Wu Zhende 18 Dec '20

18 Dec '20

my_token is more expressive Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1d44559..fa8e2dc 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -319,7 +319,7 @@ class Job testbox my_email my_name - my_uuid + my_token ] private def check_required_keys @@ -340,9 +340,10 @@ class Job raise error_msg if @account_info["found"]? == false raise error_msg unless self["my_name"] == @account_info["my_name"].to_s - raise error_msg unless self["my_uuid"] == @account_info["my_uuid"] + raise error_msg unless self["my_token"] == @account_info["my_token"] @hash.delete("my_uuid") + @hash.delete("my_token") end private def get_initialized_keys @@ -366,7 +367,7 @@ class Job "SCHED_HOST", "SCHED_PORT"] - initialized_keys -= ["my_uuid", + initialized_keys -= ["my_token", "kernel_version", "kernel_uri", "kernel_params", -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: fix incorrect git_tag value
by Liu Shaofei 18 Dec '20

18 Dec '20

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index fd2697e25..dd1ee120f 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -97,7 +97,7 @@ replace_source() echo "$url" | egrep 'git\+|\.git$' && { [ -n "$repo_dir" ] && { url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}" - [ -n ${git_tag} ] && { + [ -n "$git_tag" ] && { url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#tag=${git_tag}" } } -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: fix incorrect git_tag value
by Liu Shaofei 18 Dec '20

18 Dec '20

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index fd2697e25..9cb51ff6f 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -17,7 +17,6 @@ ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, ## make customizations, generate meta-info, and package everything together. ## See: https://wiki.archlinux.org/index.php/Makepkg - . $LKP_SRC/lib/debug.sh . $LKP_SRC/distro/common @@ -97,7 +96,7 @@ replace_source() echo "$url" | egrep 'git\+|\.git$' && { [ -n "$repo_dir" ] && { url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}" - [ -n ${git_tag} ] && { + [ -n "$git_tag" ] && { url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#tag=${git_tag}" } } -- 2.23.0

1 1

[PATCH compass-ci] fix: idle job for subqueue: submit idle jobs when finding no job
by Ren Wen 17 Dec '20

17 Dec '20

[Processes] There're 2 types of queues to consume: 1. user queue : sched/<taishan-a12>/<my_email> 2. idle queue : sched/<taishan-a12>/idle Consume Queue 1 at first, next Queue2 if no job found in Queue1. If there's no job in all Queues, auto submit idle jobs to idle queue. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 7565375..567016e 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -51,31 +51,38 @@ class Sched end def get_queues(host) - queues = [] of String + default_queues = [] of String + sub_queues = [] of String queues_str = @redis.hash_get("sched/host2queues", host) - return queues unless queues_str + return default_queues, sub_queues unless queues_str queues_str.split(',', remove_empty: true) do |item| - queues << item.strip + default_queues << item.strip end - sub_queues = [] of String - queues.each do |queue| - matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") + default_queues.each do |queue| + matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") matched_queues.each do |mq| match_data = "#{mq}".match(%r(^#{QUEUE_NAME_BASE}/sched/(#{queue}/.+)/ready$)) sub_queues << match_data[1] if match_data end end - return rand_queues(sub_queues) + return default_queues, sub_queues end def get_job_from_queues(queues, testbox) + default_queues, sub_queues = queues + all_queues = rand_queues(sub_queues) + + default_queues.each do |queue| + all_queues << "#{queue}/idle" + end + job = nil - queues.each do |queue| + all_queues.each do |queue| job = prepare_job("sched/#{queue}", testbox) return job if job end @@ -89,6 +96,8 @@ class Sched if job create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) + else + spawn { auto_submit_idle_job(host) } end return boot_content(job, boot_type) -- 2.23.0

1 0

[PATCH compass-ci] src/lib/web_backend.rb remove non-existent os_version
by Lu Weitao 17 Dec '20

17 Dec '20

remove non-existent os_version of compare_candidates Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/lib/web_backend.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 07e2e9e..cacafde 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -97,7 +97,7 @@ def compare_candidates_body suite: all_suite, OS: [ { os: 'openeuler', os_version: ['1.0', '20.03'] }, - { os: 'centos', os_version: ['7.6', '7.8', '8.1', 'sid'] }, + { os: 'centos', os_version: ['7.6', '7.8', '8.1'] }, { os: 'debian', os_version: %w[10 sid] }, { os: 'archlinux', os_version: ['5.5.0-1'] } ], -- 2.23.0

1 0

[PATCH lab-z9] hosts: add taishan200-2280-2s64p-256g--a121
by Wei Jihui 17 Dec '20

17 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- hosts/taishan200-2280-2s64p-256g--a121 | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 hosts/taishan200-2280-2s64p-256g--a121 diff --git a/hosts/taishan200-2280-2s64p-256g--a121 b/hosts/taishan200-2280-2s64p-256g--a121 new file mode 100644 index 0000000..9c87315 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a121 @@ -0,0 +1,22 @@ +memory: 256G +nr_ssd_partitions: 1 +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43943 +mac_addr: + - 44:67:47:e9:76:95 + - 44:67:47:e9:76:96 + - 44:67:47:e9:76:97 + - 44:67:47:e9:76:98 + - ac:b3:b5:1e:d3:ab + - ac:b3:b5:1e:d3:ac + - ac:b3:b5:1e:d3:ad + - ac:b3:b5:1e:d3:ae + - ac:b3:b5:1e:d6:0b + - ac:b3:b5:1e:d6:0c + - ac:b3:b5:1e:d6:0d + - ac:b3:b5:1e:d6:0e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.12.2 -- 2.23.0

1 0

[PATCH compass-ci] container/web-backend: install terminal-table
by Li Yuanchao 17 Dec '20

17 Dec '20

As code changes, 'terminal-table' is needed in this container. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/web-backend/Dockerfile b/container/web-backend/Dockerfile index 4fb6610..2ac5148 100644 --- a/container/web-backend/Dockerfile +++ b/container/web-backend/Dockerfile @@ -11,7 +11,7 @@ COPY conf/sources.list /etc/apt/ RUN apt-get update && \ apt-get install -y git ruby-json ruby-rails ruby-sinatra && \ gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ - gem install elasticsearch redis git + gem install elasticsearch redis git terminal-table COPY web-backend /usr/local/bin/ -- 2.23.0

1 0

[PATCH compass-ci] fix: idle job for subqueue: also hand over job from idle queue
by Ren Wen 17 Dec '20

17 Dec '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/close_job.cr | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index d8f80d7..a389a5c 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -14,8 +14,11 @@ class Sched raise "es set job content fail!" end + subqueue = job.subqueue + queue = subqueue == "idle" ? job.queue : "#{job.queue}/#{job.subqueue}" + response = @task_queue.hand_over_task( - "sched/#{job.queue}/#{job.subqueue}", "extract_stats", job_id + "sched/#{queue}", "extract_stats", job_id ) if response[0] != 201 raise "#{response}" -- 2.23.0

1 0

[PATCH compass-ci] fix: idle job for subqueue: "idle" is a value of subqueue not a field
by Ren Wen 17 Dec '20

17 Dec '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 7 ------- 1 file changed, 7 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index c322d0f..c09d145 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -114,13 +114,6 @@ class Sched def submit_single_job(job) queue = "#{job.queue}/#{job.subqueue}" - # only single job will has "idle job" and "execute rate limiter" - if job["idle_job"].empty? - queue += "#{job.get_uuid_tag}" - else - queue = "#{queue}/idle" - end - job_id = add_task(queue, job.lab) return [{ "job_id" => "0", -- 2.23.0

1 0

[PATCH compass-ci] fix: idle job for subqueue: subqueue can be "idle"
by Ren Wen 17 Dec '20

17 Dec '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1d44559..9fc3bf7 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -274,7 +274,7 @@ class Job end private def set_subqueue - self["subqueue"] = self["my_email"] + self["subqueue"] = self["my_email"] unless self["subqueue"] == "idle" end # if not assign tbox_group, set it to a match result from testbox -- 2.23.0

1 0

[PATCH compass-ci] lib: optimize es query_by_id to adapt to change the es mapping
by Cao Xueliang 17 Dec '20

17 Dec '20

When we change es mapping, we can't use es.get_source, because the jobs is a alias for new jobs index, the old data at the old index. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/es_query.rb | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 68c70b5..3bea624 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -54,9 +54,12 @@ class ESQuery end def query_by_id(id) - @client.get_source({ index: @index, type: '_doc', id: id }) - rescue Elasticsearch::Transport::Transport::Errors::NotFound - nil + result = @client.search(index: @index + '*', + body: { query: { bool: { must: { term: { '_id' => id } } } }, + size: 1 })['hits']['hits'] + return nil unless result.size == 1 + + return result[0]['_source'] end end -- 2.23.0

1 0

[PATCH compass-ci] doc: and runtime field in borrow-machine.en.md and borrow-machine.zh.md
by Zhang Yale 17 Dec '20

17 Dec '20

User can specify the sleep time in "runtime" field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/manual/borrow-machine.en.md | 11 ++++++----- doc/manual/borrow-machine.zh.md | 11 ++++++----- 2 files changed, 12 insertions(+), 10 deletions(-) diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index b9c2bc4..0277984 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -202,7 +202,8 @@ Ensure that you have performed the following operations according to the [apply- hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml suite: borrow testcase: borrow - + + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -212,10 +213,10 @@ Ensure that you have performed the following operations according to the [apply- %> sshd: # sleep at the bottom - sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # Use the VIM editor to change the value of the sleep field. + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml + runtime: 1h + # Use the VIM editor to change the value of the runtime field. hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # After changing the value, submit the request again. hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 06d4db6..5bc951d 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -206,7 +206,8 @@ hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml suite: borrow testcase: borrow - + + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -216,10 +217,10 @@ %> sshd: # sleep at the bottom - sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # ��vim��sleep�� + runtime: 1h + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml + runtime: 1h + # ��vim��runtime�� hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # ��submit�� hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

1 0

[PATCH lkp-tests] distro/depends: add sshd depends file
by Wang Yong 17 Dec '20

17 Dec '20

add depends file for sshd Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/depends/sshd | 1 + 1 file changed, 1 insertion(+) create mode 100644 distro/depends/sshd diff --git a/distro/depends/sshd b/distro/depends/sshd new file mode 100644 index 000000000..dd21b5aa9 --- /dev/null +++ b/distro/depends/sshd @@ -0,0 +1 @@ +openssh-server -- 2.23.0

2 2

[PATCH v2 compass-ci] scheduler: submit cluster job to "queue/subqueue"
by Ren Wen 17 Dec '20

17 Dec '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 4303b1d..c322d0f 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -37,6 +37,7 @@ class Sched def submit_cluster_job(job, cluster_config) job_messages = Array(Hash(String, String)).new lab = job.lab + subqueue = job.subqueue # collect all job ids job_ids = [] of String @@ -52,7 +53,7 @@ class Sched # steps for each host cluster_config["nodes"].as_h.each do |host, config| queue = host.to_s - job_id = add_task(queue, lab) + job_id = add_task("#{queue}/#{subqueue}", lab) # return when job_id is '0' # 2 Questions: -- 2.23.0

1 0

[PATCH lkp-tests] jobs: add runtime field in borrow-*.yaml
by Zhang Yale 17 Dec '20

17 Dec '20

[why] 1. We need "$runtime" field to determine "Due time" value in email template received by users. 2. And the "$runtime" field is used in sleep script too. Therefore, sleep can be left blank, and pp.sleep can be parse successfully now. 3. User can specify the sleep time in "runtime" field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/borrow-10d.yaml | 3 ++- jobs/borrow-1d.yaml | 3 ++- jobs/borrow-1h.yaml | 3 ++- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/jobs/borrow-10d.yaml b/jobs/borrow-10d.yaml index e0e35e17..88a137e0 100644 --- a/jobs/borrow-10d.yaml +++ b/jobs/borrow-10d.yaml @@ -1,6 +1,7 @@ suite: borrow testcase: borrow +runtime: 10d ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -10,4 +11,4 @@ ssh_pub_key: <%= %> sshd: # sleep at the bottom -sleep: 10d +sleep: diff --git a/jobs/borrow-1d.yaml b/jobs/borrow-1d.yaml index c31d7ebd..f79b440b 100644 --- a/jobs/borrow-1d.yaml +++ b/jobs/borrow-1d.yaml @@ -3,6 +3,7 @@ testcase: borrow timeout: 1d +runtime: 1d ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -12,4 +13,4 @@ ssh_pub_key: <%= %> sshd: # sleep at the bottom -sleep: 1d +sleep: diff --git a/jobs/borrow-1h.yaml b/jobs/borrow-1h.yaml index d6b72c43..d13af910 100644 --- a/jobs/borrow-1h.yaml +++ b/jobs/borrow-1h.yaml @@ -1,6 +1,7 @@ suite: borrow testcase: borrow +runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -10,4 +11,4 @@ ssh_pub_key: <%= %> sshd: # sleep at the bottom -sleep: 1h +sleep: -- 2.23.0

2 2

[PATCH compass-ci] result-webdav/nginx.conf: error 500 at the second request
by Cui Yili 17 Dec '20

17 Dec '20

[why] the table in the function split is used as a variable elsewhere. error: Attempt to call field 'insert' (a nil value) [how] table => dirs Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index af27653..8b54314 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -61,13 +61,13 @@ http { ngx.exit(ngx.HTTP_FORBIDDEN) end - if string.find(res, ",") then - table = split(res,",") - for i = 1, #table do - if string.find(path, table[i]) then + if string.find(res, ",", 1, true) then + dirs = split(res, ",") + for i = 1, #dirs do + if string.find(path, dirs[i], 1, true) then return end - end + end else if string.find(path, res, 1, true) then return -- 2.23.0

2 1

[PATCH v1 compass-ci] delimiter: optimize delimiter utils function
by Cao Xueliang 17 Dec '20

17 Dec '20

1. Delete redundant bad job fields. 2. Add to make /tmp/lkp/result dir. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index b60ada0..e87bbbe 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -108,8 +108,11 @@ module Utils job.delete('error_ids') job.delete('start_time') + job.delete('job_state') job.delete('end_time') job.delete('loadavg') + job.delete('stats') + job.delete('id') return job end @@ -123,6 +126,7 @@ module Utils end def create_bisect_log(git_dir) + FileUtils.mkdir_p TMP_RESULT_ROOT unless File.exist? TMP_RESULT_ROOT log_file = File.join(TMP_RESULT_ROOT, "bisect.log") log_content = parse_bisect_log(git_dir) File.open(log_file, 'w') do |f| -- 2.23.0

1 0

[PATCH compass-ci] delimiter: optimize delimiter utils function
by Cao Xueliang 17 Dec '20

17 Dec '20

1. Delete redundant bad job fields. 2. Add to make /tmp/lkp/result dir. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index b60ada0..e87bbbe 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -108,8 +108,11 @@ module Utils job.delete('error_ids') job.delete('start_time') + job.delete('job_state') job.delete('end_time') job.delete('loadavg') + job.delete('stats') + job.delete('id') return job end @@ -123,6 +126,7 @@ module Utils end def create_bisect_log(git_dir) + FileUtils.mkdir_p TMP_RESULT_ROOT unless File.exist? TMP_RESULT_ROOT log_file = File.join(TMP_RESULT_ROOT, "bisect.log") log_content = parse_bisect_log(git_dir) File.open(log_file, 'w') do |f| -- 2.23.0

1 1

[PATCH v2 lkp-tests] tests: fix error when running host-info in vm
by Zhang Yu 17 Dec '20

17 Dec '20

Error: Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/tests/host-info b/tests/host-info index 0ab8e6c97..d1d0df449 100755 --- a/tests/host-info +++ b/tests/host-info @@ -102,11 +102,9 @@ get_ipmi_ip() # 802.1q VLAN ID : Disabled # RMCP+ Cipher Suites : 0,1,2,3,17 # Cipher Suite Priv Max : XuuaXXXXXXXXXXX - is_virt && return - is_docker && return + [ -e /dev/ipmi0 ] || [ -e /dev/ipmi/0 ] || [ -e /dev/ipmidev/0 ] && return local ip=$(ipmitool lan print | awk 'NR==3{print $4}') [ -n "$ip" ] && echo "ipmi_ip: $ip" - } main() -- 2.23.0

2 2

[PATCH v1 compass-ci] delimiter: error id use inspect to avoid special characters
by Cao Xueliang 17 Dec '20

17 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 2 +- src/delimiter/delimiter.rb | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index 97ad467..bc1e035 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -79,7 +79,7 @@ class GitBisect result = `git -C #{@work_dir} bisect start #{@bad_commit} #{@good_commit}` temp = result.split(/\n/) if temp[0].include? 'Bisecting' - result = `git -C #{@work_dir} bisect run #{BISECT_RUN_SCRIPT} #{@bad_job_id} "#{@error_id}" #{@work_dir}` + result = `git -C #{@work_dir} bisect run #{BISECT_RUN_SCRIPT} #{@bad_job_id} #{(a)error_id.inspect} #{@work_dir}` end Utils.create_bisect_log(@work_dir) FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) diff --git a/src/delimiter/delimiter.rb b/src/delimiter/delimiter.rb index c1922e2..bf36db3 100644 --- a/src/delimiter/delimiter.rb +++ b/src/delimiter/delimiter.rb @@ -25,7 +25,7 @@ class Delimiter next end - %x(#{LKP_SRC}/sbin/submit bad_job_id=#{task['job_id']} error_id=#{task['error_id']} bisect.yaml) + %x(#{LKP_SRC}/sbin/submit bad_job_id=#{task['job_id']} error_id=#{task['error_id'].inspect} bisect.yaml) rescue StandardError => e puts e sleep(60) -- 2.23.0

1 0

[PATCH compass-ci] delimiter: error id use inspect to avoid special characters
by Cao Xueliang 17 Dec '20

17 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 2 +- src/delimiter/delimiter.rb | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index 97ad467..bc1e035 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -79,7 +79,7 @@ class GitBisect result = `git -C #{@work_dir} bisect start #{@bad_commit} #{@good_commit}` temp = result.split(/\n/) if temp[0].include? 'Bisecting' - result = `git -C #{@work_dir} bisect run #{BISECT_RUN_SCRIPT} #{@bad_job_id} "#{@error_id}" #{@work_dir}` + result = `git -C #{@work_dir} bisect run #{BISECT_RUN_SCRIPT} #{@bad_job_id} #{(a)error_id.inspect} #{@work_dir}` end Utils.create_bisect_log(@work_dir) FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) diff --git a/src/delimiter/delimiter.rb b/src/delimiter/delimiter.rb index c1922e2..cb2ed6c 100644 --- a/src/delimiter/delimiter.rb +++ b/src/delimiter/delimiter.rb @@ -25,7 +25,7 @@ class Delimiter next end - %x(#{LKP_SRC}/sbin/submit bad_job_id=#{task['job_id']} error_id=#{task['error_id']} bisect.yaml) + %x(#{LKP_SRC}/sbin/submit bad_job_id=#{task['job_id']} error_id=\'#{task['error_id']}\' bisect.yaml) rescue StandardError => e puts e sleep(60) -- 2.23.0

1 1

Re: [PATCH lkp-tests] daemon: add the local execution permission
by Bai Jing 17 Dec '20

17 Dec '20

title can be : add the execute permission Thanks, Baijing Reply-To: In-Reply-To: <20201217061232.1480336-1-1034502035(a)qq.com> On Thu, Dec 17, 2020 at 02:12:32PM +0800, Hu Xuejiao wrote: >Signed-off-by: Hu XueJiao <1034502035(a)qq.com> >--- > daemon/multi-docker | 2 +- > daemon/multi-qemu | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) > >diff --git a/daemon/multi-docker b/daemon/multi-docker >index b39506b6..257645b8 100755 >--- a/daemon/multi-docker >+++ b/daemon/multi-docker >@@ -8,7 +8,7 @@ > > multi_docker() > { >- export CCI_SRC=/c/compass-ci >+ export CCI_SRC=${CCI_SRC:-"/c/compass-ci"} > local hostname=$tbox_group.$HOSTNAME > $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues > } >diff --git a/daemon/multi-qemu b/daemon/multi-qemu >index aa2348b4..55b96c8e 100755 >--- a/daemon/multi-qemu >+++ b/daemon/multi-qemu >@@ -10,7 +10,7 @@ > > multi_qemu() > { >- export CCI_SRC=/c/compass-ci >+ export CCI_SRC=${CCI_SRC:-"/c/compass-ci"} > local hostname=$tbox_group.$HOSTNAME > > cd "$CACHE_DIR" >-- >2.23.0 >

1 0

[PATCH v2 compass-ci] sbin/es-jobs-mapping.sh: update pp.sleep mapping
by Lu Weitao 17 Dec '20

17 Dec '20

[Why] bellow doc will be store to ES { ... "pp":{ "sleep": 3600 } } or { .. "pp": { "sleep": {"runtime": 3600} } } both type doc will cause mapper-parsing-exceptions Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-jobs-mapping.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 5cb9b9c..b43625e 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -79,6 +79,10 @@ else "dynamic": true, "properties": {} }, + "pp.sleep":{ + "type": "object", + "enabled": false + }, "suite": { "type": "keyword" }, -- 2.23.0

1 0

[PATCH compass-ci] sbin/es-jobs-mapping.sh: update
by Lu Weitao 17 Dec '20

17 Dec '20

[Why] bellow doc will be store to ES { ... "pp":{ "sleep": 3600 } } or { .. "pp": { "sleep": {"runtime": 3600} } } both type doc will cause mapper-parsing-exceptions Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-jobs-mapping.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 5cb9b9c..5713cbf 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -79,6 +79,10 @@ else "dynamic": true, "properties": {} }, + "pp.sleep":{ + "type": "object", + "enabled": false + }, "suite": { "type": "keyword" }, -- 2.23.0

3 4

[PATCH v2 lkp-tests] daemon: add the local execution permission
by Hu Xuejiao 17 Dec '20

17 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- daemon/multi-docker | 2 +- daemon/multi-qemu | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/daemon/multi-docker b/daemon/multi-docker index b39506b6..257645b8 100755 --- a/daemon/multi-docker +++ b/daemon/multi-docker @@ -8,7 +8,7 @@ multi_docker() { - export CCI_SRC=/c/compass-ci + export CCI_SRC=${CCI_SRC:-"/c/compass-ci"} local hostname=$tbox_group.$HOSTNAME $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues } diff --git a/daemon/multi-qemu b/daemon/multi-qemu index aa2348b4..55b96c8e 100755 --- a/daemon/multi-qemu +++ b/daemon/multi-qemu @@ -10,7 +10,7 @@ multi_qemu() { - export CCI_SRC=/c/compass-ci + export CCI_SRC=${CCI_SRC:-"/c/compass-ci"} local hostname=$tbox_group.$HOSTNAME cd "$CACHE_DIR" -- 2.23.0

1 0

[PATCH lkp-tests] daemon: add the local execution permission
by Hu Xuejiao 17 Dec '20

17 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- daemon/multi-docker | 2 +- daemon/multi-qemu | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/daemon/multi-docker b/daemon/multi-docker index b39506b6..257645b8 100755 --- a/daemon/multi-docker +++ b/daemon/multi-docker @@ -8,7 +8,7 @@ multi_docker() { - export CCI_SRC=/c/compass-ci + export CCI_SRC=${CCI_SRC:-"/c/compass-ci"} local hostname=$tbox_group.$HOSTNAME $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues } diff --git a/daemon/multi-qemu b/daemon/multi-qemu index aa2348b4..55b96c8e 100755 --- a/daemon/multi-qemu +++ b/daemon/multi-qemu @@ -10,7 +10,7 @@ multi_qemu() { - export CCI_SRC=/c/compass-ci + export CCI_SRC=${CCI_SRC:-"/c/compass-ci"} local hostname=$tbox_group.$HOSTNAME cd "$CACHE_DIR" -- 2.23.0

1 0

[PATCH v5 compass-ci 6/6] fetch-mail: start file
by Luan Shengde 17 Dec '20

17 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/start | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 container/fetch-mail/start diff --git a/container/fetch-mail/start b/container/fetch-mail/start new file mode 100755 index 0000000..98cbd8f --- /dev/null +++ b/container/fetch-mail/start @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig' + +docker_rm 'fetch-mail' + +names = Set.new %w[ + FETCH_MAIL_ADDRESS + FETCH_MAIL_AUTH_CODE + FETCH_MAIL_DIR + FETCH_MAIL_BOX +] + +defaults = relevant_defaults(names) + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + -u mailer + --name=fetch-mail + -d +] + env + %W[ + -v #{defaults['FETCH_MAIL_DIR']}:#{defaults['FETCH_MAIL_DIR']}:rw + -w /home/mailer/ + fetch-mail +] + +cmd += ['sh', '-c', './run.sh'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v5 compass-ci 5/6] fetch-mail: build file
by Luan Shengde 17 Dec '20

17 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/fetch-mail/build diff --git a/container/fetch-mail/build b/container/fetch-mail/build new file mode 100755 index 0000000..e72bed8 --- /dev/null +++ b/container/fetch-mail/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t fetch-mail:latest . -- 2.23.0

1 0

[PATCH v5 compass-ci 4/6] fetch-mail: Dockerfile
by Luan Shengde 17 Dec '20

17 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/Dockerfile | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 container/fetch-mail/Dockerfile diff --git a/container/fetch-mail/Dockerfile b/container/fetch-mail/Dockerfile new file mode 100644 index 0000000..0434e09 --- /dev/null +++ b/container/fetch-mail/Dockerfile @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM centos:latest +MAINTAINER Luan Shengde <shdluan(a)163.com> +ENV DEBIAN_FRONTEND noninteractive + +RUN useradd -u 1144 -g 1147 mailer +RUN yum update -y +RUN yum install -y procmail fetchmail + +COPY fetchmailrc.template /home/mailer/.fetchmailrc +COPY procmailrc.template /home/mailer/.procmailrc +COPY run.sh /home/mailer/ + +RUN chmod 600 /home/mailer/.fetchmailrc +RUN chmod 600 /home/mailer/.procmailrc +RUN chown -R mailer:mailer /home/mailer +RUN chmod +x /home/mailer/run.sh -- 2.23.0

1 0

[PATCH v5 compass-ci 3/6] fetch-mail: runner for fetch-mail service
by Luan Shengde 17 Dec '20

17 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/run.sh | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100755 container/fetch-mail/run.sh diff --git a/container/fetch-mail/run.sh b/container/fetch-mail/run.sh new file mode 100755 index 0000000..c79b054 --- /dev/null +++ b/container/fetch-mail/run.sh @@ -0,0 +1,14 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +mail_conf () +{ + sed -i "s|FETCH_MAIL_ADDRESS|${FETCH_MAIL_ADDRESS}|g" '.fetchmailrc' + sed -i "s|FETCH_MAIL_AUTH_CODE|${FETCH_MAIL_AUTH_CODE}|g" '.fetchmailrc' + sed -i "s|FETCH_MAIL_DIR|${FETCH_MAIL_DIR}|g" '.procmailrc' + sed -i "s|FETCH_MAIL_BOX|${FETCH_MAIL_BOX}|g" '.procmailrc' +} + +mail_conf +fetchmail -d 100 -- 2.23.0

1 0

[PATCH v5 compass-ci 2/6] fetch-mail: config file for procmail
by Luan Shengde 17 Dec '20

17 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/procmailrc.template | 25 ++++++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 container/fetch-mail/procmailrc.template diff --git a/container/fetch-mail/procmailrc.template b/container/fetch-mail/procmailrc.template new file mode 100644 index 0000000..9cea4bc --- /dev/null +++ b/container/fetch-mail/procmailrc.template @@ -0,0 +1,25 @@ +DROPPRIVS=yes +PATH=$HOME/bin:/usr/local/bin:/usr/bin:/bin +MAILDIR=FETCH_MAIL_DIR +MAILLIST=${MAILDIR} +DEFAULT=${MAILDIR}/FETCH_MAIL_BOX/ +ORGMAIL=${DEFAULT}/FETCH_MAIL_BOX/ +LOGFILE=${MAILDIR}/log/procmail # recommended + +UMASK=002 + +DISPLAY=:0 +XAUTHORITY=$HOME/.Xauthority + +# One often receives duplicate messages for cross-postings to mailing +# lists, or explicit replies to a post from you in addition to the +# mailing list message. +# +# The following recipe eliminates all duplicate messages. It tells +# formail(1) to keep a cache file of the Message-IDs of the most recent +# messages you've received. Since Message-IDs are guaranteed unique for +# distinct messages, they are ideally suited to weed out duplicates. + +# LOCKFILE=$MAILDIR/.msgid.lock +:0 Wh: $MAILDIR/.msgid.lock +| formail -D 655360 $MAILDIR/log/msgid -- 2.23.0

1 0

[PATCH v5 compass-ci 1/6] fetch-mail: config file for fetchmail
by Luan Shengde 17 Dec '20

17 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/fetchmailrc.template | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 container/fetch-mail/fetchmailrc.template diff --git a/container/fetch-mail/fetchmailrc.template b/container/fetch-mail/fetchmailrc.template new file mode 100644 index 0000000..8c7e82b --- /dev/null +++ b/container/fetch-mail/fetchmailrc.template @@ -0,0 +1,5 @@ +poll imap.qq.com with proto IMAP +username "FETCH_MAIL_ADDRESS" +password "FETCH_MAIL_AUTH_CODE" +mda "/usr/bin/procmail -f %F -d %T" +nokeep -- 2.23.0

1 0

[PATCH lkp-tests] daemon: add the local execution permission to the script
by Hu Xuejiao 17 Dec '20

17 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- daemon/multi-docker | 2 +- daemon/multi-qemu | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/daemon/multi-docker b/daemon/multi-docker index b39506b6..49c8c968 100755 --- a/daemon/multi-docker +++ b/daemon/multi-docker @@ -8,7 +8,7 @@ multi_docker() { - export CCI_SRC=/c/compass-ci + [ -n "$CCI_SRC" ] || export CCI_SRC=/c/compass-ci local hostname=$tbox_group.$HOSTNAME $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues } diff --git a/daemon/multi-qemu b/daemon/multi-qemu index aa2348b4..99f63583 100755 --- a/daemon/multi-qemu +++ b/daemon/multi-qemu @@ -10,9 +10,9 @@ multi_qemu() { - export CCI_SRC=/c/compass-ci + [ -n "$CCI_SRC" ] || export CCI_SRC=~/compass-ci local hostname=$tbox_group.$HOSTNAME - + cd "$CACHE_DIR" $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" } -- 2.23.0

4 3

[PATCH v2 compass-ci] git_mirror.rb: improve how to get filenames
by Ren Wen 17 Dec '20

17 Dec '20

Dir.children returns filenames except for “.” and “..”. Signed-off-by: Ren Wen <15991987063(a)163.com> --- lib/git_mirror.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 389b185..156d5bb 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -128,7 +128,7 @@ class MirrorMain def traverse_repodir(repodir) if File.directory? repodir load_defaults(repodir) - entry_list = Dir.entries(repodir) - Array['.', '..', 'DEFAULTS', '.ignore', '.git'] + entry_list = Dir.children(repodir) - ['DEFAULTS'] entry_list.each do |entry| next if entry.start_with? '.' -- 2.23.0

1 0

[PATCH compass-ci] container/srv-http: import h5ai into container
by Lu Weitao 17 Dec '20

17 Dec '20

[Why] Optimize result page style to make it more modern Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/build | 17 +++++++++++++++++ .../srv-http/root/etc/nginx/conf.d/default.conf | 5 ++--- 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/container/srv-http/build b/container/srv-http/build index 3a500ea..6580c55 100755 --- a/container/srv-http/build +++ b/container/srv-http/build @@ -2,4 +2,21 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +[[ $H5AI_SRC ]] || H5AI_SRC=/c/h5ai + +WORKSPACE=$(dirname $(realpath $0)) + +# npm build h5ai package +cd $H5AI_SRC +npm install +npm run build + +# add h5ai package in ./root/srv to add it into image +cd $WORKSPACE +mkdir ./root/srv +cp -r $H5AI_SRC/build/_h5ai ./root/srv + docker build -t srv-http . + +# clean h5ai package +rm -rf ./root/srv diff --git a/container/srv-http/root/etc/nginx/conf.d/default.conf b/container/srv-http/root/etc/nginx/conf.d/default.conf index 676418a..646eee3 100644 --- a/container/srv-http/root/etc/nginx/conf.d/default.conf +++ b/container/srv-http/root/etc/nginx/conf.d/default.conf @@ -3,8 +3,8 @@ server { server_name "srv-http"; server_tokens off; - root /usr/share/nginx/html; - index index.html; + root /srv; + index index.html index.php /_h5ai/public/index.php; charset utf-8,gbk; @@ -24,7 +24,6 @@ server { autoindex on; autoindex_localtime on; allow all; - root /srv; location ~* /[-_a-z]+$ { add_header Content-Type text/plain; } -- 2.23.0

1 1

[PATCH compass-ci] providers: mount "/srv/result" for docker
by Cao Xueliang 17 Dec '20

17 Dec '20

Now use mount "/srv/result" to get error messages for bisect job by container, next will get the error messages from the assistant container API. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- providers/docker/run.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index d532a56..127cf5b 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -29,6 +29,7 @@ cmd=( -v ${DIR}/bin:/root/bin:ro -v $CCI_SRC:/c/compass-ci -v /srv/git:/srv/git:ro + -v /srv/result:/srv/result:ro -v ${busybox_path}:/usr/local/bin/busybox --oom-score-adj="-1000" ${docker_image} -- 2.23.0

1 0

[PATCH compass-ci] code-spec.md: adjust markdown document format
by Xu Xijian 17 Dec '20

17 Dec '20

In markdown documents, text for bash should be contained in block as belows: ```SHELL content SHELL Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/code-spec.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/code-spec.md b/doc/code-spec.md index d44c6fe..a717f64 100644 --- a/doc/code-spec.md +++ b/doc/code-spec.md @@ -21,10 +21,12 @@ code注意事项 - puts/printf字符串应当容易搜索(grep)。特别注意包含足够长的固定内容substring. 例如： +```SHELL die "test yaml$${test_yaml}$ not exist in ${LKP_SRC}/jobs !!!" ==> die "cannot find test yaml ($test_yaml) in $LKP_SRC/jobs" ^^^^^^^^^^^^^^^^^^^^^ grep-friendly fixed substring +``` 重构友好编码 ============ -- 2.23.0

2 2

[PATCH compass-ci] lib: add the add_task function to taskqueue client
by Cao Xueliang 17 Dec '20

17 Dec '20

1. Modify the default host to 172.17.0.1. 2. Add the add_task function to taskqueue client. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/taskqueue_client.rb | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/lib/taskqueue_client.rb b/lib/taskqueue_client.rb index 91b72dc..2c22990 100644 --- a/lib/taskqueue_client.rb +++ b/lib/taskqueue_client.rb @@ -6,7 +6,7 @@ require 'rest-client' # taskqueue client class TaskQueueClient - HOST = (ENV.key?('TASKQUEUE_HOST') ? ENV['TASKQUEUE_HOST'] : '127.0.0.1') + HOST = (ENV.key?('TASKQUEUE_HOST') ? ENV['TASKQUEUE_HOST'] : '172.17.0.1') PORT = (ENV.key?('TASKQUEUE_PORT') ? ENV['TASKQUEUE_PORT'] : 3060).to_i def initialize(host = HOST, port = PORT) @host = host @@ -20,4 +20,14 @@ class TaskQueueClient url: url ) end + + def add_task(queue_path, json_data) + url = "http://#{@host}:#{@port}/add?queue=#{queue_path}" + RestClient::Request.execute( + method: :post, + url: url, + payload: json_data, + headers: { content_type: 'application/json' } + ) + end end -- 2.23.0

1 0

[PATCH compass-ci] container: add curl to delimiter container for uploading files
by Cao Xueliang 17 Dec '20

17 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/delimiter/Dockerfile b/container/delimiter/Dockerfile index 4163cea..3db5890 100644 --- a/container/delimiter/Dockerfile +++ b/container/delimiter/Dockerfile @@ -6,7 +6,7 @@ FROM alpine:3.11 RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ apk update && \ apk add --no-cache 'ruby-dev' \ - 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' RUN umask 002 && \ echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ -- 2.23.0

1 0

[PATCH v1 compass-ci] lib: fix string literal in condition
by Cao Xueliang 17 Dec '20

17 Dec '20

1. Fix warning string literal in condition 2. Display the first error message in email subject Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index c0a2731..090855e 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -19,8 +19,8 @@ class MailBisectResult end def compose_mail - subject = "[Compass-CI][#{(a)repo.split('/')[1]}]: #{@error_messages[0]}" - job_url = "job url: #{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}/#{ENV['result_root']}\n" ? ENV['result_root'] : '' + subject = "[Compass-CI][#{(a)repo.split('/')[1]}]: #{(a)error_messages[0].split("\n")[0]}" + job_url = ENV['result_root'] ? "job url: http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}#{ENV['result_root']}\n" : '' body = <<~BODY Hi #{(a)git_commit.author_name}, @@ -28,6 +28,7 @@ class MailBisectResult git commit: #{@commit_id[0..11]} ("#{(a)git_commit.subject}") gcc version: 7.3.0 + error_messages: #{(a)error_messages.join("\n")} -- 2.23.0

1 0

[PATCH compass-ci] lib: fix string literal in condition
by Cao Xueliang 17 Dec '20

17 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index c0a2731..090855e 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -19,8 +19,8 @@ class MailBisectResult end def compose_mail - subject = "[Compass-CI][#{(a)repo.split('/')[1]}]: #{@error_messages[0]}" - job_url = "job url: #{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}/#{ENV['result_root']}\n" ? ENV['result_root'] : '' + subject = "[Compass-CI][#{(a)repo.split('/')[1]}]: #{(a)error_messages[0].split("\n")[0]}" + job_url = ENV['result_root'] ? "job url: http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}#{ENV['result_root']}\n" : '' body = <<~BODY Hi #{(a)git_commit.author_name}, @@ -28,6 +28,7 @@ class MailBisectResult git commit: #{@commit_id[0..11]} ("#{(a)git_commit.subject}") gcc version: 7.3.0 + error_messages: #{(a)error_messages.join("\n")} -- 2.23.0

1 0

[PATCH v6 compass-ci 4/4] container/assign-account/get_account_info.rb: refactor sort for files
by Lu Kaiyi 17 Dec '20

17 Dec '20

files.sort make no difference for files itself Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- container/assign-account/get_account_info.rb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 67c4c46..af08d91 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -76,11 +76,11 @@ class AccountStorage end def build_account_name(available_dir, files) - files.sort + files.sort! account_info = [] - account_info.push files[0] - source_file = File.join(available_dir, files[0]) - account_info.push File.readlines(source_file)[0].chomp + account_info.push files.first + source_file = File.join(available_dir, files.first) + account_info.push File.readlines(source_file).first.chomp dest_dir = File.join(@account_dir, 'assigned-users') FileUtils.mv(source_file, dest_dir) -- 2.23.0

1 0

[PATCH v6 compass-ci 3/4] lib/compare_data_format.rb: limit the number of table columns in a single row
by Lu Kaiyi 17 Dec '20

17 Dec '20

[example] compare -t compare_template.yaml in z9 [input] a compare_template.yaml like below: compare_metrics: - meminfo.Active - meminfo.Memused filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: centos os_version: 7.6 - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s [output] +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | Hackbench Performance Testing (unit: KB/s, x_name: bs|test_size) | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | meminfo.Active | 4k|1G | 4k|10G | 4k|80G | 4k|100G | 16k|1G | 16k|10G | 16k|100G | 32k|1G | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | average openeuler 20.03 | 1051645.34 | 704958.00 | 706798.00 | 657669.27 | 1030484.85 | 979289.00 | 948576.00 | 1262591.17 | | standard_deviation openeuler 20.03 | 74.00 | 0.00 | 0.00 | 8.00 | 46.00 | 0.00 | 2.00 | 42.00 | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | meminfo.Active | 64k|1G | 64k|100G | 128k|1G | 128k|100G | 256k|1G | 256k|100G | 512k|1G | 512k|100G | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | average openeuler 20.03 | 1418731.53 | 1854231.33 | 2004321.76 | 2960171.75 | 2979326.50 | 4775340.00 | 5003555.53 | 5630845.75 | | standard_deviation openeuler 20.03 | 47.00 | 10.00 | 65.00 | 18.00 | 88.00 | 8.00 | 108.00 | 15.00 | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | meminfo.Active | 1024k|1G | 1024k|100G | 2048k|100G | 4096k|100G | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | average openeuler 20.03 | 8974190.91 | 9641039.00 | 5480763.67 | 7302509.67 | | standard_deviation openeuler 20.03 | 118.00 | 36.00 | 86.00 | 67.00 | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 41 +++++++++++++++++++++++++++++--------- 1 file changed, 32 insertions(+), 9 deletions(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index c1f7023..0f6bdc0 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -72,19 +72,20 @@ end # format compare template results into a table format # class FormatTableData - def initialize(result_hash) + def initialize(result_hash, row_size = 8) @title = result_hash['title'] @tables = result_hash['tables'] @unit = result_hash['unit'] @x_name = result_hash['x_name'] + @row_size = row_size end def show_table @tables.each do |table_title, table| @tb = Terminal::Table.new set_table_title - set_field_names(table_title, table) - add_rows(table) + row_num = get_row_num(table) + split_data_column(table_title, table, row_num) set_align_column print_table end @@ -94,14 +95,36 @@ class FormatTableData @tb.title = "#{@title} (unit: #{@unit}, x_name: #{@x_name})" end - def set_field_names(table_title, table) + def get_row_num(table) + data_column_size = table['average']['source'][0].size + unless @row_size.positive? + warn('row size must be positive!') + exit + end + (data_column_size / @row_size.to_f).ceil + end + + def split_data_column(table_title, table, row_num) + row_num.times do |row| + starts = 1 + row * @row_size + ends = starts + @row_size + set_field_names(table_title, table, starts, ends) + add_rows(table, starts, ends) + break if row == row_num - 1 + + @tb.add_separator + @tb.add_separator + end + end + + def set_field_names(table_title, table, starts, ends) field_names = [table_title] - field_names.concat(table['average']['source'][0]) + field_names.concat(table['average']['source'][0][starts - 1...ends - 1]) @tb.add_row(field_names) @tb.add_separator end - def add_rows(table) + def add_rows(table, starts, ends) row_names = %w[average standard_deviation change] max_size = row_names.map(&:size).max row_names.each do |row_name| @@ -109,15 +132,15 @@ class FormatTableData dimensions_size = table[row_name]['dimensions'].size (1...dimensions_size).each do |index| - add_row(table, row_name, index, max_size) + add_row(table, row_name, index, max_size, starts, ends) end end end - def add_row(table, row_name, index, max_size) + def add_row(table, row_name, index, max_size, starts, ends) row = table[row_name]['source'][index] row_title = [row_name + ' ' * (max_size - row_name.size), row[0]].join(' ') - format_data_row = row[1..-1] + format_data_row = row[starts...ends] if row_name == 'change' format_data_row.map! { |data| format('%.1f%%', data) } else -- 2.23.0

1 0

[PATCH v6 compass-ci 2/4] add options for compare_by_template
by Lu Kaiyi 17 Dec '20

17 Dec '20

if want json format results, use command like below: compare -t compare_template.yaml --theme json otherwise, print table format results by default: compare -t compare_template.yaml Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 4 ++-- lib/compare_matrixes.rb | 10 +++++++--- sbin/compare | 2 +- 3 files changed, 10 insertions(+), 6 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index f34e239..db264f8 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -96,11 +96,11 @@ end # unit: KB/s # -def compare_by_template(template) +def compare_by_template(template, options) template_params = load_template(template) groups_matrices = create_groups_matrices(template_params) compare_results = compare_metrics_values(groups_matrices) - show_compare_result(compare_results, template_params) + show_compare_result(compare_results, template_params, options) end def load_template(template) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 9bf676f..b7ea5a4 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -508,11 +508,15 @@ def get_dimensions_combination(dimension_list) dims end -def show_compare_result(metrics_compare_results, template_params) +def show_compare_result(metrics_compare_results, template_params, options) formatter = FormatEchartData.new(metrics_compare_results, template_params) echart_results = formatter.format_for_echart - table_results = FormatTableData.new(echart_results) - table_results.show_table + if options[:theme] == 'json' + print JSON.pretty_generate(echart_results) + else + table_results = FormatTableData.new(echart_results) + table_results.show_table + end end # Format Fields diff --git a/sbin/compare b/sbin/compare index c13ec1a..e8d1ac4 100755 --- a/sbin/compare +++ b/sbin/compare @@ -67,7 +67,7 @@ opt_parser.parse!(argv) options = { theme: colorful } if colorful if template - compare_by_template(template) + compare_by_template(template, options) elsif is_group compare_group(argv, dimensions, options) else -- 2.23.0

1 0

[PATCH v6 compass-ci 1/4] lib: friendly show compare template result in command line
by Lu Kaiyi 17 Dec '20

17 Dec '20

[why] friendly show compare template result in command line. [how] rehandle the results of "compare -t <compare_template.yaml>" with terminal-table in Ruby. [example] compare -t compare_template.yaml in z9 [input] a compare_template.yaml like below: compare_metrics: - fio.write_iops - fio.read_iops filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: centos os_version: 7.6 - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s [output] +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | Hackbench Performance Testing (unit: KB/s, x_name: bs|test_size) | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | fio.read_iops | 4k|1G | 4k|80G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | average openeuler 20.03 | 144076.29 | 11601.10 | 37865.30 | 21145.10 | 14010.34 | 6701.24 | 3205.08 | 1367.48 | 673.33 | | standard_deviation openeuler 20.03 | 195.00 | 0.00 | 214.00 | 205.00 | 188.00 | 183.00 | 180.00 | 191.00 | 191.00 | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | Hackbench Performance Testing (unit: KB/s, x_name: bs|test_size) | +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | fio.write_iops | 4k|1G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | average centos 7.6 | 345243.03 | 142698.79 | 62108.35 | 34747.73 | 26330.19 | 10317.85 | 7471.71 | 3558.30 | | average openeuler 20.03 | 122003.54 | 33528.53 | 31469.06 | 13870.14 | 8249.71 | 4329.45 | 1976.54 | 1141.00 | | standard_deviation centos 7.6 | 97.00 | 95.00 | 122.00 | 125.00 | 100.00 | 130.00 | 101.00 | 103.00 | | standard_deviation openeuler 20.03 | 174.00 | 188.00 | 171.00 | 197.00 | 181.00 | 175.00 | 170.00 | 176.00 | | change centos 7.6 vs openeuler 20.03 | 183.0% | 325.6% | 97.4% | 150.5% | 219.2% | 138.3% | 278.0% | 211.9% | +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 73 ++++++++++++++++++++++++++++++++++++++ lib/compare_matrixes.rb | 3 +- 2 files changed, 75 insertions(+), 1 deletion(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index bf4013b..c1f7023 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -2,6 +2,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'terminal-table' + # ---------------------------------------------------------------------------------------------------- # format compare results for a specific format # @@ -66,6 +68,77 @@ class FormatEchartData end end +# ---------------------------------------------------------------------------------------------------- +# format compare template results into a table format +# +class FormatTableData + def initialize(result_hash) + @title = result_hash['title'] + @tables = result_hash['tables'] + @unit = result_hash['unit'] + @x_name = result_hash['x_name'] + end + + def show_table + @tables.each do |table_title, table| + @tb = Terminal::Table.new + set_table_title + set_field_names(table_title, table) + add_rows(table) + set_align_column + print_table + end + end + + def set_table_title + @tb.title = "#{@title} (unit: #{@unit}, x_name: #{@x_name})" + end + + def set_field_names(table_title, table) + field_names = [table_title] + field_names.concat(table['average']['source'][0]) + @tb.add_row(field_names) + @tb.add_separator + end + + def add_rows(table) + row_names = %w[average standard_deviation change] + max_size = row_names.map(&:size).max + row_names.each do |row_name| + next unless table[row_name] + + dimensions_size = table[row_name]['dimensions'].size + (1...dimensions_size).each do |index| + add_row(table, row_name, index, max_size) + end + end + end + + def add_row(table, row_name, index, max_size) + row = table[row_name]['source'][index] + row_title = [row_name + ' ' * (max_size - row_name.size), row[0]].join(' ') + format_data_row = row[1..-1] + if row_name == 'change' + format_data_row.map! { |data| format('%.1f%%', data) } + else + format_data_row.map! { |data| format('%.2f', data) } + end + @tb.add_row([row_title, *format_data_row]) + end + + def set_align_column + @tb.number_of_columns.times do |index| + @tb.align_column(index + 1, :right) + end + @tb.align_column(0, :left) + end + + def print_table + puts @tb + puts + end +end + # input: x_params_list # eg: ["1G|4K", "1G|1024k", "1G|128k", 2G|4k] # output: diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 5069939..9bf676f 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -511,7 +511,8 @@ end def show_compare_result(metrics_compare_results, template_params) formatter = FormatEchartData.new(metrics_compare_results, template_params) echart_results = formatter.format_for_echart - print JSON.pretty_generate(echart_results) + table_results = FormatTableData.new(echart_results) + table_results.show_table end # Format Fields -- 2.23.0

1 0

[PATCH compass-ci] lib: change the file permission
by Zhang Yu 17 Dec '20

17 Dec '20

The file permission should be consistent with the other file in this directory Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- lib/mail-post.rb | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100755 => 100644 lib/mail-post.rb diff --git a/lib/mail-post.rb b/lib/mail-post.rb old mode 100755 new mode 100644 -- 2.23.0

1 0

[PATCH compass-ci] sparrow/0-package: install hostname
by Liu Yinsi 17 Dec '20

17 Dec '20

sparrow/3-code/dev-env will use hostname command, so add it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/centos | 1 + sparrow/0-package/os/debian | 1 + sparrow/0-package/os/openEuler | 1 + 3 files changed, 3 insertions(+) diff --git a/sparrow/0-package/os/centos b/sparrow/0-package/os/centos index 6e3ed42..8df36c0 100755 --- a/sparrow/0-package/os/centos +++ b/sparrow/0-package/os/centos @@ -21,6 +21,7 @@ pkgs=( lockfile-progs busybox iproute + hostname ) yum install -y --skip-broken "${pkgs[@]}" diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index 8ee6d7e..6a58a0c 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -19,6 +19,7 @@ pkgs=( bridge-utils busybox iproute2 + hostname ) apt-get install -y --fix-missing "${pkgs[@]}" diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 13e0875..a63964f 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -20,6 +20,7 @@ pkgs=( wget busybox iproute + hostname ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests: fix error when running host-info in vm
by Zhang Yu 17 Dec '20

17 Dec '20

Error: Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/tests/host-info b/tests/host-info index 0ab8e6c97..3984727b4 100755 --- a/tests/host-info +++ b/tests/host-info @@ -102,11 +102,9 @@ get_ipmi_ip() # 802.1q VLAN ID : Disabled # RMCP+ Cipher Suites : 0,1,2,3,17 # Cipher Suite Priv Max : XuuaXXXXXXXXXXX - is_virt && return - is_docker && return + [ -e /dev/ipmi0 ] || [ -e /dev/ipmi/0 ] || [ -e /dev/ipmidev/0 ] || return local ip=$(ipmitool lan print | awk 'NR==3{print $4}') [ -n "$ip" ] && echo "ipmi_ip: $ip" - } main() -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sparrow/3-code: use defaults/* to be loaded as job defaults
by Liu Yinsi 16 Dec '20

16 Dec '20

before: /etc/compass-ci/account/autosubmit.yaml after: /etc/compass-ci/defaults/admin-account.yaml Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 750738f..7eb6166 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -41,8 +41,7 @@ LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS MAILDIR: /srv/cci/Maildir/.compass-ci EOF -mkdir -p /etc/compass-ci/account -cat > /etc/compass-ci/account/autosubmit.yaml <<EOF +cat > /etc/compass-ci/defaults/admin-account.yaml <<EOF my_name: Auto Submit Robot my_email: autosubmit@localhost my_token: $(uuidgen) -- 2.23.0

2 1

[PATCH compass-ci] container/assign-account/get_account_info.rb: refactor sort for files
by Lu Kaiyi 16 Dec '20

16 Dec '20

files.sort make no difference for files itself Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- container/assign-account/get_account_info.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 9a7acda..0ce8cad 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -76,7 +76,7 @@ class AccountStorage end def build_account_name(available_dir, files) - files.sort + files.sort! account_info = [] account_info.push files[0] source_file = File.join(available_dir, files[0]) -- 2.23.0

2 1

[PATCH v5 compass-ci 3/3] lib/compare_data_format.rb: limit the number of table columns in a single row
by Lu Kaiyi 16 Dec '20

16 Dec '20

[example] compare -t compare_template.yaml in z9 [input] a compare_template.yaml like below: compare_metrics: - meminfo.Active - meminfo.Memused filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: centos os_version: 7.6 - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s [output] +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | Hackbench Performance Testing (unit: KB/s, x_name: bs|test_size) | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | meminfo.Active | 4k|1G | 4k|10G | 4k|80G | 4k|100G | 16k|1G | 16k|10G | 16k|100G | 32k|1G | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | average openeuler 20.03 | 1051645.34 | 704958.00 | 706798.00 | 657669.27 | 1030484.85 | 979289.00 | 948576.00 | 1262591.17 | | standard_deviation openeuler 20.03 | 74.00 | 0.00 | 0.00 | 8.00 | 46.00 | 0.00 | 2.00 | 42.00 | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | meminfo.Active | 64k|1G | 64k|100G | 128k|1G | 128k|100G | 256k|1G | 256k|100G | 512k|1G | 512k|100G | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | average openeuler 20.03 | 1418731.53 | 1854231.33 | 2004321.76 | 2960171.75 | 2979326.50 | 4775340.00 | 5003555.53 | 5630845.75 | | standard_deviation openeuler 20.03 | 47.00 | 10.00 | 65.00 | 18.00 | 88.00 | 8.00 | 108.00 | 15.00 | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | meminfo.Active | 1024k|1G | 1024k|100G | 2048k|100G | 4096k|100G | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | average openeuler 20.03 | 8974190.91 | 9641039.00 | 5480763.67 | 7302509.67 | | standard_deviation openeuler 20.03 | 118.00 | 36.00 | 86.00 | 67.00 | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 41 +++++++++++++++++++++++++++++--------- 1 file changed, 32 insertions(+), 9 deletions(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index c1f7023..0f6bdc0 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -72,19 +72,20 @@ end # format compare template results into a table format # class FormatTableData - def initialize(result_hash) + def initialize(result_hash, row_size = 8) @title = result_hash['title'] @tables = result_hash['tables'] @unit = result_hash['unit'] @x_name = result_hash['x_name'] + @row_size = row_size end def show_table @tables.each do |table_title, table| @tb = Terminal::Table.new set_table_title - set_field_names(table_title, table) - add_rows(table) + row_num = get_row_num(table) + split_data_column(table_title, table, row_num) set_align_column print_table end @@ -94,14 +95,36 @@ class FormatTableData @tb.title = "#{@title} (unit: #{@unit}, x_name: #{@x_name})" end - def set_field_names(table_title, table) + def get_row_num(table) + data_column_size = table['average']['source'][0].size + unless @row_size.positive? + warn('row size must be positive!') + exit + end + (data_column_size / @row_size.to_f).ceil + end + + def split_data_column(table_title, table, row_num) + row_num.times do |row| + starts = 1 + row * @row_size + ends = starts + @row_size + set_field_names(table_title, table, starts, ends) + add_rows(table, starts, ends) + break if row == row_num - 1 + + @tb.add_separator + @tb.add_separator + end + end + + def set_field_names(table_title, table, starts, ends) field_names = [table_title] - field_names.concat(table['average']['source'][0]) + field_names.concat(table['average']['source'][0][starts - 1...ends - 1]) @tb.add_row(field_names) @tb.add_separator end - def add_rows(table) + def add_rows(table, starts, ends) row_names = %w[average standard_deviation change] max_size = row_names.map(&:size).max row_names.each do |row_name| @@ -109,15 +132,15 @@ class FormatTableData dimensions_size = table[row_name]['dimensions'].size (1...dimensions_size).each do |index| - add_row(table, row_name, index, max_size) + add_row(table, row_name, index, max_size, starts, ends) end end end - def add_row(table, row_name, index, max_size) + def add_row(table, row_name, index, max_size, starts, ends) row = table[row_name]['source'][index] row_title = [row_name + ' ' * (max_size - row_name.size), row[0]].join(' ') - format_data_row = row[1..-1] + format_data_row = row[starts...ends] if row_name == 'change' format_data_row.map! { |data| format('%.1f%%', data) } else -- 2.23.0

2 1

[PATCH v5 compass-ci 2/3] add options for compare_by_template
by Lu Kaiyi 16 Dec '20

16 Dec '20

if want json format results, use command like below: compare -t compare_template.yaml --theme json otherwise, print table format results by default: compare -t compare_template.yaml Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 4 ++-- lib/compare_matrixes.rb | 11 ++++++++--- sbin/compare | 2 +- 3 files changed, 11 insertions(+), 6 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index f34e239..db264f8 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -96,11 +96,11 @@ end # unit: KB/s # -def compare_by_template(template) +def compare_by_template(template, options) template_params = load_template(template) groups_matrices = create_groups_matrices(template_params) compare_results = compare_metrics_values(groups_matrices) - show_compare_result(compare_results, template_params) + show_compare_result(compare_results, template_params, options) end def load_template(template) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 9bf676f..e8f52f1 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -508,11 +508,16 @@ def get_dimensions_combination(dimension_list) dims end -def show_compare_result(metrics_compare_results, template_params) +def show_compare_result(metrics_compare_results, template_params, options) + theme = options[:theme] formatter = FormatEchartData.new(metrics_compare_results, template_params) echart_results = formatter.format_for_echart - table_results = FormatTableData.new(echart_results) - table_results.show_table + if theme == 'json' + print JSON.pretty_generate(echart_results) + else + table_results = FormatTableData.new(echart_results) + table_results.show_table + end end # Format Fields diff --git a/sbin/compare b/sbin/compare index c13ec1a..e8d1ac4 100755 --- a/sbin/compare +++ b/sbin/compare @@ -67,7 +67,7 @@ opt_parser.parse!(argv) options = { theme: colorful } if colorful if template - compare_by_template(template) + compare_by_template(template, options) elsif is_group compare_group(argv, dimensions, options) else -- 2.23.0

2 1

[PATCH lkp-tests] tests/build-pkg: build different version software packages
by Liu Shaofei 16 Dec '20

16 Dec '20

[why] When building different version software, we need to pack out different *.cgz. The field of PKGBUILD_PKGVE can be appended job.yaml file, which specifies the version number. The parts of contents for job.yaml file is as follow: job.yaml: ... pkgbuild_repo: archlinux/packages//fcgi/trunk PKGBUILD_PKGVER: 2.4.2 ... [how] -rw-rw-r--. 1 lkp lkp 46K 2020-12-15 17:17 2.4.2.cgz lrwxrwxrwx. 1 lkp lkp 9 2020-12-15 17:17 latest.cgz -> 2.4.2.cgz Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 25 ++++++++++--------------- 1 file changed, 10 insertions(+), 15 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index aafc120ed..c3909d1b9 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -3,6 +3,7 @@ # - upstream_repo # - upstream_commit # - PKGBUILD_TAG +# - PKGBUILD_PKGVER # upstream_url # We'll use the below global env vars, but not treat them as parameter to avoid @@ -60,18 +61,6 @@ request_pkg() do get_pkgfile "${pkgrepo}" "$pkgfile" done - - # download PKGBUILD-$tag - benchmark="${upstream_repo##*/}" - benchmark="${benchmark%.git}" - [ -n "$PKGBUILD_TAG" ] && { - get_pkgfile "pkg/$benchmark" "PKGBUILD-$PKGBUILD_TAG" - [ -f "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG not exist" - [ -s "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG is empty" - } - - [ -f PKGBUILD ] || die "PKGBUILD not exist" - [ -s PKGBUILD ] || die "PKGBUILD is empty" } create_softlink() @@ -109,6 +98,7 @@ replace_source() build_source_pkg() { + [ -n "$PKGBUILD_PKGVER" ] && sed -i "s|^pkgver=.*|pkgver=${PKGBUILD_PKGVER}|g" PKGBUILD repo_dir="" source PKGBUILD [ -n "$upstream_repo" ] && [ -n "$upstream_url" ] && { @@ -122,11 +112,16 @@ build_source_pkg() } replace_source PKGBUILD.src - [ -n "$PKGBUILD_TAG" ] && replace_source "PKGBUILD-$PKGBUILD_TAG" - cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" + pkg_args="-A --check --skippgpcheck" + + [ -n ${PKGBUILD_PKGVER} ] && { + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${pkgver}.cgz" + pkg_args="${pkg_args} --skipchecksums --skipinteg" + } + PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" STDERR_REDIR=y\ - $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf + $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf [ "$?" == 0 ] || exit 1 create_softlink -- 2.23.0

2 3

[PATCH lkp-tests 1/3] tests/build-pkg: fix format
by Lin Jiaxin 16 Dec '20

16 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index aafc120e..dbde3e54 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -2,8 +2,8 @@ # - pkgbuild_repo # - upstream_repo # - upstream_commit +# - upstream_url # - PKGBUILD_TAG -# upstream_url # We'll use the below global env vars, but not treat them as parameter to avoid # duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ -- 2.23.0

2 1

Re: [PATCH lkp-tests] lib/upload: change the key of upload results
by Cao Xueliang 16 Dec '20

16 Dec '20

On Wed, Dec 16, 2020 at 07:56:34PM +0800, Li Ping wrote: >On Wed, Dec 16, 2020 at 03:37:09PM +0800, Wei Jihui wrote: >>the key of upload results is changed from access_key to job id >> >>Signed-off-by: Wei Jihui <weijihuiall(a)163.com> >>--- >> lib/job-init.sh | 2 +- >> lib/upload.sh | 18 +++++++++--------- >> 2 files changed, 10 insertions(+), 10 deletions(-) >> >>diff --git a/lib/job-init.sh b/lib/job-init.sh >>index 40534af3..dc4a65d9 100755 >>--- a/lib/job-init.sh >>+++ b/lib/job-init.sh >>@@ -55,7 +55,7 @@ setup_result_service() >> >> supports_raw_upload && result_service=raw_upload && return >> >>- is_docker || [ -n "$access_key" ] || { >>+ is_docker || [ -n "$id" ] || { >> if [ "$os_mount" = 'nfs' ] || [ -z "$os_mount" ]; then >> supports_netfs 'nfs' && result_service=$RESULT_SERVER:/result && return >> elif [ "$os_mount" = 'cifs' ] || [ -z "$os_mount" ]; then >>diff --git a/lib/upload.sh b/lib/upload.sh >>index a5daceda..0115e350 100755 >>--- a/lib/upload.sh >>+++ b/lib/upload.sh >>@@ -78,9 +78,9 @@ upload_one_curl() >> ( >> cd $(dirname "$1") >> dir=$(basename "$1") >>- if [ -n "$access_key" ]; then >>- find "$dir" -type d -exec curl -sSf -X MKCOL "$http_url/{}" --cookie "ACCESSKEY=$access_key" \; >>- find "$dir" -type f -size +0 -exec curl -sSf -T '{}' "$http_url/{}" --cookie "ACCESSKEY=$access_key" \; >>+ if [ -n "$id" ]; then >>+ find "$dir" -type d -exec curl -sSf -X MKCOL "$http_url/{}" --cookie "JOBID=$id" \; >>+ find "$dir" -type f -size +0 -exec curl -sSf -T '{}' "$http_url/{}" --cookie "JOBID=$id" \; > > users can upload files or folders regardless > of whether the job id exists. what is the > function of determing whether the job id > exist? Get the id from the env var, when we run a job lkp will export all the job yaml as the env var. Thanks, Xueliang > > Thanks, > Ping > >> else >> find "$dir" -type d -exec curl -sSf -X MKCOL "$http_url/{}" \; >> find "$dir" -type f -size +0 -exec curl -sSf -T '{}' "$http_url/{}" \; >>@@ -88,8 +88,8 @@ upload_one_curl() >> ) >> else >> [ -s "$src" ] || return >>- if [ -n "$$access_key" ]; then >>- curl -sSf -T "$src" $http_url/ --cookie "ACCESSKEY=$access_key" >>+ if [ -n "$id" ]; then >>+ curl -sSf -T "$src" $http_url/ --cookie "JOBID=$id" >> else >> curl -sSf -T "$src" $http_url/ >> fi >>@@ -111,8 +111,8 @@ upload_files_curl() >> for dir in $(echo $target_directory | tr '/' ' ') >> do >> job_result_root=$job_result_root/$dir/ >>- if [ -n "$$access_key" ]; then >>- curl -sSf -X MKCOL "$http_host$job_result_root" --cookie "ACCESSKEY=$access_key" >/dev/null >>+ if [ -n "$id" ]; then >>+ curl -sSf -X MKCOL "$http_host$job_result_root" --cookie "JOBID=$id" >/dev/null >> else >> curl -sSf -X MKCOL "$http_host$job_result_root" >/dev/null >> fi >>@@ -178,12 +178,12 @@ upload_files() >> upload_files_copy "$@" >> return >> } >>- if has_cmd rsync && has_rsync_server && [ -z "$access_key" ]; then >>+ if has_cmd rsync && has_rsync_server && [ -z "$id" ]; then >> upload_files_rsync "$@" >> return >> fi >> >>- if has_cmd lftp && [ -z "$access_key" ]; then >>+ if has_cmd lftp && [ -z "$id" ]; then >> upload_files_lftp "$@" >> return >> fi >>-- >>2.23.0 >>

1 0

[PATCH v2 lkp-tests] tests/build-pkg: build different version software packages
by Liu Shaofei 16 Dec '20

16 Dec '20

[why] When building different version software, we need to pack out different *.cgz. The field of "git_tag" and "pkg_ver" can be appended job.yaml file, which specifies the version number. The parts of contents for job.yaml file is as follow: job.yaml: ... pkgbuild_repo: archlinux/packages//fcgi/trunk pkg_ver: 2.4.2 ... [how] -rw-rw-r--. 1 lkp lkp 46K 2020-12-15 17:17 2.4.2.cgz lrwxrwxrwx. 1 lkp lkp 9 2020-12-15 17:17 latest.cgz -> 2.4.2.cgz Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 43 +++++++++++++++++++++++-------------------- 1 file changed, 23 insertions(+), 20 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index aafc120ed..fd2697e25 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -2,7 +2,8 @@ # - pkgbuild_repo # - upstream_repo # - upstream_commit -# - PKGBUILD_TAG +# - git_tag +# - pkg_ver # upstream_url # We'll use the below global env vars, but not treat them as parameter to avoid @@ -60,34 +61,32 @@ request_pkg() do get_pkgfile "${pkgrepo}" "$pkgfile" done +} + +create_cgz_name() +{ + cgz_path="$PKG_MNT/${pack_to}/${pkgname}" + cgz_name="${cgz_path}/${upstream_commit}.cgz" + pkg_args="-A --check --skippgpcheck" - # download PKGBUILD-$tag - benchmark="${upstream_repo##*/}" - benchmark="${benchmark%.git}" - [ -n "$PKGBUILD_TAG" ] && { - get_pkgfile "pkg/$benchmark" "PKGBUILD-$PKGBUILD_TAG" - [ -f "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG not exist" - [ -s "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG is empty" + [ -n "$git_tag" ] && { + cgz_name="${cgz_path}/${git_tag}.cgz" } - [ -f PKGBUILD ] || die "PKGBUILD not exist" - [ -s PKGBUILD ] || die "PKGBUILD is empty" + [ -n "$pkg_ver" ] && { + cgz_name="${cgz_path}/${pkg_ver}.cgz" + pkg_args="${pkg_args} --skipchecksums --skipinteg" + } } create_softlink() { [ -e "$cgz_name" ] || return 0 - local soft_path="$PKG_MNT/${pack_to}/${pkgname}" local bm_name=$(basename $(realpath ${cgz_name})) - if [ -n "$PKGBUILD_TAG" ]; then - echo "create soft link: ${PKGBUILD_TAG}.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/${PKGBUILD_TAG}.cgz" - fi - echo "create soft link: latest.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/latest.cgz" + ln -sf "${bm_name}" "${cgz_path}/latest.cgz" } replace_source() @@ -98,6 +97,9 @@ replace_source() echo "$url" | egrep 'git\+|\.git$' && { [ -n "$repo_dir" ] && { url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}" + [ -n ${git_tag} ] && { + url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#tag=${git_tag}" + } } } url=$(echo "$url" | sed 's|https://|http://|g') @@ -109,6 +111,8 @@ replace_source() build_source_pkg() { + [ -n "$pkg_ver" ] && sed -i "s|^pkgver=.*|pkgver=${pkg_ver}|g" PKGBUILD + repo_dir="" source PKGBUILD [ -n "$upstream_repo" ] && [ -n "$upstream_url" ] && { @@ -122,11 +126,10 @@ build_source_pkg() } replace_source PKGBUILD.src - [ -n "$PKGBUILD_TAG" ] && replace_source "PKGBUILD-$PKGBUILD_TAG" + create_cgz_name - cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" STDERR_REDIR=y\ - $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf + $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf [ "$?" == 0 ] || exit 1 create_softlink -- 2.23.0

2 2