- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] lib: add build_my_info_client
by Luan Shengde 24 Dec '20

24 Dec '20

purpose: for user build their infos when installing compassci - store my_info to ES - write my_info to default/lab yaml file if not, job will fail for verification problems for user private environment Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/build_my_info_client.rb | 79 +++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100755 lib/build_my_info_client.rb diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb new file mode 100755 index 0000000..9e44c00 --- /dev/null +++ b/lib/build_my_info_client.rb @@ -0,0 +1,79 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'fileutils' +require 'io/console' +require_relative '../lib/es_client' + +# purpose: +# build my_info +# store my_info to ES +# write my_info to default/lab yaml file +# for user install compass-ci at their local server +# user meet verification problems when submit jobs +# add calling in installing script to meet the verification function +# usage: +# require_relative 'build_my_info_client' +# +# build_my_info = BuildMyInfo.new(email, name, lab) +# build_user_info.config_my_info +class BuildMyInfo + def initialize(my_email, my_name, lab) + @lab = lab || 'z9' + @my_info = { + 'my_email' => my_email, + 'my_name' => my_name, + 'my_token' => %x(uuidgen).chomp + } + end + + def config_default_yaml + default_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/defaults" + FileUtils.mkdir_p default_yaml_dir unless File.directory? default_yaml_dir + default_yaml_file = "#{default_yaml_dir}/account.yaml" + FileUtils.touch(default_yaml_file) unless File.exist? default_yaml_file + + default_yaml_info = YAML.load_file(default_yaml_file) || {} + default_yaml_info['my_email'] = @my_info['my_email'] + default_yaml_info['my_name'] = @my_info['my_name'] + + File.open(default_yaml_file, 'w') do |f| + f.puts default_yaml_info.to_yaml + end + end + + def config_lab_yaml + lab_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/include/lab" + FileUtils.mkdir_p lab_yaml_dir unless File.directory? lab_yaml_dir + lab_yaml_file = "#{lab_yaml_dir}/#{(a)lab}.yaml" + FileUtils.touch(lab_yaml_file) unless File.exist? lab_yaml_file + + lab_yaml_info = YAML.load_file(lab_yaml_file) || {} + lab_yaml_info['my_token'] = @my_info['my_token'] + + File.open(lab_yaml_file, 'w') do |f| + f.puts lab_yaml_info.to_yaml + end + end + + def complete_my_info + @my_info['my_login_name'] = '' + @my_info['my_commit_url'] = '' + @my_info['my_ssh_pubkey'] = [] + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def config_my_info + config_default_yaml + config_lab_yaml + complete_my_info + store_account_info + end +end -- 2.23.0

1 0

[PATCH lkp-tests] fix shebang in crystal script
by Lin Jiaxin 24 Dec '20

24 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/constant.cr | 2 +- lib/dmesg.cr | 2 +- lib/run_env.cr | 2 +- lib/string_ext.cr | 2 +- lib/yaml.cr | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/constant.cr b/lib/constant.cr index b5adcf9d..c17a0f36 100644 --- a/lib/constant.cr +++ b/lib/constant.cr @@ -1,4 +1,4 @@ -#!/usr/bin/env ruby +#!/usr/bin/env crystal require "./run_env" diff --git a/lib/dmesg.cr b/lib/dmesg.cr index 9cbe6751..25834e4b 100644 --- a/lib/dmesg.cr +++ b/lib/dmesg.cr @@ -1,4 +1,4 @@ -#!/usr/bin/env ruby +#!/usr/bin/env crystal LKP_SRC = ENV["LKP_SRC"] || File.dirname(__DIR__) diff --git a/lib/run_env.cr b/lib/run_env.cr index cdccaf98..af0ec39b 100644 --- a/lib/run_env.cr +++ b/lib/run_env.cr @@ -1,4 +1,4 @@ -#!/usr/bin/env ruby +#!/usr/bin/env crystal LOCAL_RUN_ENV = "LKP_LOCAL_RUN".freeze require "yaml" diff --git a/lib/string_ext.cr b/lib/string_ext.cr index 41fcaf1a..398ab7e0 100644 --- a/lib/string_ext.cr +++ b/lib/string_ext.cr @@ -1,4 +1,4 @@ -#!/usr/bin/env ruby +#!/usr/bin/env crystal REGEX_ANSI_COLOR = /\e\[([0-9;]+m|[mK])/.freeze diff --git a/lib/yaml.cr b/lib/yaml.cr index b9fa1a94..19691d1d 100644 --- a/lib/yaml.cr +++ b/lib/yaml.cr @@ -1,4 +1,4 @@ -#!/usr/bin/env ruby +#!/usr/bin/env crystal require "./common" require "./log" -- 2.23.0

2 2

[PATCH v7 compass-ci] tutorial.md: update initramfs tutorial.md
by Wang Chenglong 24 Dec '20

24 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/tutorial.md | 80 +++++++++++++------------------------------------ 1 file changed, 21 insertions(+), 59 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..b8207ba 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -150,78 +150,40 @@ job yaml ��[YAML](http://yaml.org/YAM #### �� - 1. �� os �� rootfs�� openEuler �� - - �� docker �� rootfs - - 1) �� openEuler �� docker �� - - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) �� docker �� - ```bash - docker load -i openEuler-docker.aarch64 - ``` - - 3) �� openEuler �� - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) �� docker �� rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - - �� qemu.img(qcow2��)�� rootfs (openEuler��) - - - 1) �� openEule r�� qcow2 �� - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… - ``` - 2) ��{compass-ci}/container/qcow2rootfs ��rootfs - ```bash - cd {compass-ci}/container/qcow2rootfs - ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs - ``` - 2. ��rootfs - 1. ��chroot�� rootfs (�� root ��) - ```bash - chroot openEuler-rootfs - ``` - 2. �� - - a. �� root �� - b. �� ssh �� - c. �� - d. �� docker �� osimage �� - > 1. �� - > 2. �� centos ��rpm�� - > 3. �� yum �� - > 4. �� docker �� - - 3. �� rootfs�� - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` + 1. �� openEuler �� + ```bash + cd {compass-ci}/rootfs/initramfs/openeuler/aarch64/20.03-LTS/ + ./run + ``` + + 2. ��/�� + �� {compass-ci} ��rootfs�� rootfs ��/�� + ```bash + mkdir rootfs + cd rootfs + zcat ../$os_name.cgz | cpio -idmv + find ./lib/modules/* | cpio -o -Hnewc | gzip -9 > $HOME/modules-$os_name.cgz + find ./usr/src/kernels/* | cpio -o -Hnewc | gzip -9 > $HOME/headers-$os_name.cgz + ``` + #### FAQ -1. �� Unable to mount root fs on unknown-block�� +1. �� Unable to mount root fs on unknown-block�� - �� ```bash [ 0.390437] List of all partitions: - [ 0.390806] No filesystem could mount root, tried: + [ 0.390806] No filesystem could mount root, tried: [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ... [ 0.399404] Memory Limit: none [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- ``` - �� - + 1�� 2�� 644 �� - + 2. �� - + - �� -- 2.23.0

1 0

[PATCH v3 compass-ci] fix: qcow2rootfs/bin/common: unable to understand vmlinux
by Wang Chenglong 24 Dec '20

24 Dec '20

[error] [WARNNING] Restoring the owner of qcow2 file Finding vmlinuz under /tmp/a/boot ... /c/compass-ci/container/qcow2rootfs/bin/common: line 135: ROOTFS_VMLINUZ_FILE: unbound variable [why] some kernels are called vmlinx instead of vmlinz. --- container/qcow2rootfs/bin/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/qcow2rootfs/bin/common b/container/qcow2rootfs/bin/common index da39f52..baf936c 100755 --- a/container/qcow2rootfs/bin/common +++ b/container/qcow2rootfs/bin/common @@ -127,7 +127,7 @@ get_rootfs_kernel() { exit 5 } local vmlinuz_file vmlinuz kernel - vmlinuz_file=$(find ./boot -name "vmlinuz-*" |grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz_file=$(find ./boot -name "vmlinu[z|x]-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel -- 2.23.0

2 2

[PATCH v5 compass-ci] docker-rootf: add different install methods
by Wang Chenglong 24 Dec '20

24 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image index 787a42c..ffe4937 100755 --- a/container/docker-rootfs/setup-image +++ b/container/docker-rootfs/setup-image @@ -4,8 +4,15 @@ install_packages() { - yum install -y --skip-broken $(</tmp/packages-to-install) - yum clean all + if [ $(command -v yum) >/dev/null ]; then + yum install -y --skip-broken $(</tmp/packages-to-install) + yum clean all + elif [ $(command -v apt-get) >/dev/null ]; then + apt update + apt-get install -y --fix-missing $(</tmp/packages-to-install) + elif [ $(command -v pacman) >/dev/null ]; then + pacman -Syu --noconfirm $(</tmp/packages-to-install) + fi } remove_file() -- 2.23.0

2 2

[PATCH v4 compass-ci] docker-rootf: add different install methods
by Wang Chenglong 24 Dec '20

24 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image index 787a42c..05e5a24 100755 --- a/container/docker-rootfs/setup-image +++ b/container/docker-rootfs/setup-image @@ -4,8 +4,15 @@ install_packages() { - yum install -y --skip-broken $(</tmp/packages-to-install) - yum clean all + if [ $(command -v yum) >/dev/null ]; then + yum install -y --skip-broken $(</tmp/packages-to-install) + yum clean all + elif [ $(command -v apt-get) >/dev/null ]; then + apt update + apt-get install -y --fix-missing $(</tmp/packages-to-install) + elif [ $(command -v pacman) >/dev/null ]; then + pacman -Sy --noconfirm $(</tmp/packages-to-install) + fi } remove_file() -- 2.23.0

1 1

[PATCH v2 lkp-tests] lib/job.rb: fix job symbol key check
by Wei Jihui 24 Dec '20

24 Dec '20

load job from yaml maybe has symbol key symbol key does not has start_with function. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 7553312e..35e0ceee 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -254,7 +254,6 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end - hash.delete_if { |key, _| key.start_with?('#!') } hash.merge!(@overrides) @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end @@ -290,6 +289,12 @@ class Job jobs_array = [] hash.each { |key, value| next unless key =~ /^\w.*\|.*\w$/ + next unless key.is_a?(String) + + if key.start_with?('#! ') + hash.delete(key) + next + end key_array = key.split('|').map(&:strip) [value].flatten.each do |v| -- 2.23.0

3 3

[PATCH v2 lkp-tests] jobs: optimize yaml file
by Liu Shaofei 23 Dec '20

23 Dec '20

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- jobs/build-pkg.yaml | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/jobs/build-pkg.yaml b/jobs/build-pkg.yaml index b17fd2ade..7a47bd724 100644 --- a/jobs/build-pkg.yaml +++ b/jobs/build-pkg.yaml @@ -1,9 +1,8 @@ -suite: build-pkg -category: functional - -# pkgbuild_repo examples +# the field of "pkgbuild_repo" refers to the path of PKGBUILD, the examples are as follow: # pkgbuild_repo: archlinux/aur-j/java8-openjdk # pkgbuild_repo: archlinux/community//keepalived/trunk # pkgbuild_repo: archlinux/packages//prometheus-node-exporter/trunk +suite: build-pkg +category: functional build-pkg: -- 2.23.0

1 0

[PATCH compass-ci] container/sub-fluentd: the dependency file is missing
by Hu Xuejiao 23 Dec '20

23 Dec '20

[why] container is failed to start in test-machine because of missing file Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- container/sub-fluentd/build-depends | 1 + 1 file changed, 1 insertion(+) create mode 100755 container/sub-fluentd/build-depends diff --git a/container/sub-fluentd/build-depends b/container/sub-fluentd/build-depends new file mode 100755 index 0000000..29ee9d3 --- /dev/null +++ b/container/sub-fluentd/build-depends @@ -0,0 +1 @@ +fluentd-base -- 2.23.0

1 0

[PATCH lkp-tests] sbin/run_job: Automatic task execution
by Hu Xuejiao 23 Dec '20

23 Dec '20

[purpose] Verify that a local job can be consumed properly Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100755 sbin/run_job diff --git a/sbin/run_job b/sbin/run_job new file mode 100755 index 00000000..2b02eb77 --- /dev/null +++ b/sbin/run_job @@ -0,0 +1,76 @@ +#!/bin/bash +# tbox_group +# testbox +# nr_vm +# nr_container +# queues + +cci_makepkg_file="$LKP_SRC/jobs/cci-makepkg.yaml" +grep "atomic" $cci_makepkg_file || sed -i 's|benchmark:|& atomic|g' $cci_makepkg_file + +submit_one_job() +{ + cmd=( + submit -m $yaml + os=${single_arg[0]} + os_arch=${single_arg[1]} + os_version=${single_arg[2]} + os_mount=${single_arg[3]} + testbox=$testbox + queue="$tbox_group.$HOSTNAME" + ) + output=$("${cmd[@]}") + echo "$output" | grep -o -E 'job_state=finished[^\"]*' +} + +submit_job() +{ + yaml_args=("$cci_makepkg_file" "host-info.yaml") + for yaml in "${yaml_args[@]}" + do + submit_one_job & + done +} + +run_qemu() +{ + local nr_vm=4 + local testbox="vm-2p8g" + local tbox_group="vm-2p8g" + local queues="$tbox_group.$HOSTNAME" + + . $LKP_SRC/daemon/multi-qemu + + qemu_args=( + "openeuler aarch64 20.03 cifs" + "centos aarch64 7.6.1810 initramfs" + ) + for qemu_arg in "${qemu_args[@]}" + do + local single_arg=($(echo $qemu_arg)) + submit_job + done +} + +run_docker() +{ + local nr_container=2 + local tbox_group="dc-8g" + local testbox="dc-8g" + local queues="$tbox_group.$HOSTNAME" + + . $LKP_SRC/daemon/multi-docker + + docker_args=( + "centos aarch64 7.6.1810 container" + ) + for docker_arg in "${docker_args[@]}" + do + local single_arg=($(echo $docker_arg)) + submit_job + done +} + +run_qemu +run_docker +wait -- 2.23.0

1 0

[PATCH v3 compass-ci] docker-rootf: add different install methods
by Wang Chenglong 23 Dec '20

23 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image index 787a42c..05e5a24 100755 --- a/container/docker-rootfs/setup-image +++ b/container/docker-rootfs/setup-image @@ -4,8 +4,15 @@ install_packages() { - yum install -y --skip-broken $(</tmp/packages-to-install) - yum clean all + if [ $(command -v yum) >/dev/null ]; then + yum install -y --skip-broken $(</tmp/packages-to-install) + yum clean all + elif [ $(command -v apt-get) >/dev/null ]; then + apt update + apt-get install -y --fix-missing $(</tmp/packages-to-install) + elif [ $(command -v pacman) >/dev/null ]; then + pacman -Sy --noconfirm $(</tmp/packages-to-install) + fi } remove_file() -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: fix job symbol key check
by Wei Jihui 23 Dec '20

23 Dec '20

load job from yaml maybe has symbol key symbol key does not has start_with function. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 7553312e..5ae67e6b 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -254,7 +254,6 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end - hash.delete_if { |key, _| key.start_with?('#!') } hash.merge!(@overrides) @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end @@ -289,7 +288,12 @@ class Job def multi_args(hash) jobs_array = [] hash.each { |key, value| - next unless key =~ /^\w.*\|.*\w$/ + next unless key =~ /^\w.*\|.*\w$/ || key.is_a?(String) + + if key.start_with?('#! ') + hash.delete(key) + next + end key_array = key.split('|').map(&:strip) [value].flatten.each do |v| -- 2.23.0

1 0

[PATCH v2 compass-ci] docker-rootf: add different install methods
by Wang Chenglong 23 Dec '20

23 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image index 787a42c..7274e2e 100755 --- a/container/docker-rootfs/setup-image +++ b/container/docker-rootfs/setup-image @@ -4,8 +4,15 @@ install_packages() { - yum install -y --skip-broken $(</tmp/packages-to-install) - yum clean all + if [ $(command -v yum) ]; then + yum install -y --skip-broken $(</tmp/packages-to-install) + yum clean all + elif [ $(command -v apt-get) ]; then + apt update + apt-get install -y --fix-missing $(</tmp/packages-to-install) + elif [ $(command -v pacman) ]; then + pacman -Sy --noconfirm $(</tmp/packages-to-install) + fi } remove_file() -- 2.23.0

1 0

[PATCH compass-ci] docker-rootf: add different install methods
by Wang Chenglong 23 Dec '20

23 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image index 787a42c..6a121d3 100755 --- a/container/docker-rootfs/setup-image +++ b/container/docker-rootfs/setup-image @@ -4,8 +4,15 @@ install_packages() { - yum install -y --skip-broken $(</tmp/packages-to-install) - yum clean all + if [ $(command -v yum) ]; then + yum install -y --skip-broken $(</tmp/packages-to-install) + yum clean all + elif [ $(command -v apt-get) ]; then + apt update + apt-get install -y --no-install-recommends $(</tmp/packages-to-install) + elif [ $(command -v pacman) ]; then + pacman -Sy --noconfirm $(</tmp/packages-to-install) + fi } remove_file() -- 2.23.0

2 2

[PATCH lkp-tests] lkp-tests: delete useless file
by Wei Jihui 23 Dec '20

23 Dec '20

install.sh is useless now install method is: make install && source ~/.${SHELL##*/}rc Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- install.sh | 7 ------- 1 file changed, 7 deletions(-) delete mode 100644 install.sh diff --git a/install.sh b/install.sh deleted file mode 100644 index cea1654f..00000000 --- a/install.sh +++ /dev/null @@ -1,7 +0,0 @@ -#!/bin/bash - -. $LKP_SRC/lib/env.sh - -make - -source $(shell_profile) -- 2.23.0

1 0

[PATCH compass-ci] scheduler: fix block error
by Ren Wen 23 Dec '20

23 Dec '20

[why] Every request to scheduler will initialize a new 'Sched', each 'Sched' will new a 'BlockHelper'. For each node in cluster, it can't release when blocking at scheduler because they use different locks. [how] Set an instance of 'BlockHelper' to class variable. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 3 ++- src/scheduler/request_cluster_state.cr | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f7220f9..25e3708 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -28,11 +28,12 @@ class Sched property redis property block_helper + @@block_helper = BlockHelper.new + def initialize(env : HTTP::Server::Context) @es = Elasticsearch::Client.new @redis = Redis::Client.new @task_queue = TaskQueueAPI.new - @block_helper = BlockHelper.new @rgc = RemoteGitClient.new @env = env @log = env.log.as(JSONLogger) diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index 1782797..0703792 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -25,7 +25,7 @@ class Sched update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) when "wait_ready" update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) - @block_helper.block_until_finished(cluster_id) { + @@block_helper.block_until_finished(cluster_id) { cluster_state = sync_cluster_state(cluster_id, job_id, states[request_state]) cluster_state == "ready" || cluster_state == "abort" } -- 2.23.0

1 0

[PATCH lkp-tests] Makefile: delete set-env in Makefile
by Wei Jihui 23 Dec '20

23 Dec '20

set-env is move to install-dependencies.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- Makefile | 1 - 1 file changed, 1 deletion(-) diff --git a/Makefile b/Makefile index 0b2d09a2..236bea25 100644 --- a/Makefile +++ b/Makefile @@ -14,7 +14,6 @@ subsystem: install: mkdir -p $(TARGET_DIR_BIN) ln -sf $(shell pwd)/bin/lkp $(TARGET_DIR_BIN)/lkp - bash sbin/set-env.sh bash sbin/install-dependencies.sh .PHONY: doc -- 2.23.0

1 0

[PATCH compass-ci] service/scheduler: capturing exceptions
by Wu Zhende 23 Dec '20

23 Dec '20

For API-level errors, use warn. For service-level errors, use error. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 14 ++++++++++++++ src/scheduler.cr | 9 ++++++++- src/scheduler/close_job.cr | 2 ++ src/scheduler/download_file.cr | 2 ++ src/scheduler/find_job_boot.cr | 2 ++ src/scheduler/find_next_job_boot.cr | 2 ++ src/scheduler/request_cluster_state.cr | 2 ++ src/scheduler/update_job_parameter.cr | 2 ++ 8 files changed, 34 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f7220f9..be8cac5 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -45,6 +45,8 @@ class Sched def alive(version) debug_message("Env= {\n#{`export`}}") "LKP Alive! The time is #{Time.local}, version = #{version}" + rescue e + @log.warn(e) end def normalize_mac(mac : String) @@ -59,6 +61,8 @@ class Sched else "No yet!" end + rescue e + @log.warn(e) end def del_host_mac @@ -69,6 +73,8 @@ class Sched else "No yet!" end + rescue e + @log.warn(e) end def set_host2queues @@ -79,6 +85,8 @@ class Sched else "No yet!" end + rescue e + @log.warn(e) end def del_host2queues @@ -89,6 +97,8 @@ class Sched else "No yet!" end + rescue e + @log.warn(e) end def update_tbox_wtmp @@ -118,6 +128,8 @@ class Sched # json log hash["testbox"] = testbox @log.info(hash.to_json) + rescue e + @log.warn(e) end def report_ssh_port @@ -130,6 +142,8 @@ class Sched end @log.info(%({"job_id": "#{job_id}", "state": "set ssh port", "ssh_port": "#{ssh_port}", "tbox_name": "#{testbox}"})) + rescue e + @log.warn(e) end private def query_consumable_keys(shortest_queue_name) diff --git a/src/scheduler.cr b/src/scheduler.cr index 41702e7..07b38a3 100644 --- a/src/scheduler.cr +++ b/src/scheduler.cr @@ -3,7 +3,14 @@ require "scheduler/scheduler" require "./scheduler/constants.cr" +require "./lib/json_logger" module Scheduler - Kemal.run(SCHED_PORT) + log = JSONLogger.new + + begin + Kemal.run(SCHED_PORT) + rescue e + log.error(e) + end end diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index d8f80d7..9b4da6a 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -24,5 +24,7 @@ class Sched @redis.remove_finished_job(job_id) @log.info(%({"job_id": "#{job_id}", "job_state": "complete"})) + rescue e + @log.warn(e) end end diff --git a/src/scheduler/download_file.cr b/src/scheduler/download_file.cr index e1f0c9a..50e27ba 100644 --- a/src/scheduler/download_file.cr +++ b/src/scheduler/download_file.cr @@ -10,5 +10,7 @@ class Sched @log.info(%({"job_id": "#{job_id}", "job_state": "download"})) send_file @env, file_path + rescue e + @log.warn(e) end end diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 9b3db88..5f71b32 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -21,6 +21,8 @@ class Sched @log.info(%({"job_id": "#{job_id}", "job_state": "boot"})) if job_id response + rescue e + @log.warn(e) end # auto submit a job to collect the host information diff --git a/src/scheduler/find_next_job_boot.cr b/src/scheduler/find_next_job_boot.cr index 0bdfe5b..091c41a 100644 --- a/src/scheduler/find_next_job_boot.cr +++ b/src/scheduler/find_next_job_boot.cr @@ -14,5 +14,7 @@ class Sched @log.info(%({"job_id": "#{job_id}", "job_state": "boot"})) if job_id response + rescue e + @log.warn(e) end end diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index 1782797..06e3329 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -68,6 +68,8 @@ class Sched # show cluster state return @redis.hash_get("sched/cluster_state", cluster_id) + rescue e + @log.warn(e) end # node_state: "finish" | "ready" diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index da528c1..744274a 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -30,5 +30,7 @@ class Sched log = job_content.dup log["job_id"] = log.delete("id").not_nil! @log.info(log.to_json) + rescue e + @log.warn(e) end end -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/tutorial.md: fix abnormal display in Markdown
by Lu Kaiyi 23 Dec '20

23 Dec '20

the middle part in * and * is italics for markdown document, if want show '*' in this condition, add ` in both sides of path. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- doc/tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..bf72630 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -258,7 +258,7 @@ job yaml ��[YAML](http://yaml.org/YAM 4) �� archlinux �� PKGBUILD �� PKGBUILD �� - PKGBUILD �� /srv/git/archlinux/*/*/PKGBUILD�� + PKGBUILD �� `/srv/git/archlinux/*/*/PKGBUILD`�� ### depends -- 2.23.0

1 0

[PATCH v6 compass-ci] tutorial.md: update initramfs tutorial.md
by Wang Chenglong 23 Dec '20

23 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/tutorial.md | 80 +++++++++++++------------------------------------ 1 file changed, 21 insertions(+), 59 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..b8207ba 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -150,78 +150,40 @@ job yaml ��[YAML](http://yaml.org/YAM #### �� - 1. �� os �� rootfs�� openEuler �� - - �� docker �� rootfs - - 1) �� openEuler �� docker �� - - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) �� docker �� - ```bash - docker load -i openEuler-docker.aarch64 - ``` - - 3) �� openEuler �� - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) �� docker �� rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - - �� qemu.img(qcow2��)�� rootfs (openEuler��) - - - 1) �� openEule r�� qcow2 �� - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… - ``` - 2) ��{compass-ci}/container/qcow2rootfs ��rootfs - ```bash - cd {compass-ci}/container/qcow2rootfs - ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs - ``` - 2. ��rootfs - 1. ��chroot�� rootfs (�� root ��) - ```bash - chroot openEuler-rootfs - ``` - 2. �� - - a. �� root �� - b. �� ssh �� - c. �� - d. �� docker �� osimage �� - > 1. �� - > 2. �� centos ��rpm�� - > 3. �� yum �� - > 4. �� docker �� - - 3. �� rootfs�� - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` + 1. �� openEuler �� + ```bash + cd {compass-ci}/rootfs/initramfs/openeuler/aarch64/20.03-LTS/ + ./run + ``` + + 2. ��/�� + �� {compass-ci} ��rootfs�� rootfs ��/�� + ```bash + mkdir rootfs + cd rootfs + zcat ../$os_name.cgz | cpio -idmv + find ./lib/modules/* | cpio -o -Hnewc | gzip -9 > $HOME/modules-$os_name.cgz + find ./usr/src/kernels/* | cpio -o -Hnewc | gzip -9 > $HOME/headers-$os_name.cgz + ``` + #### FAQ -1. �� Unable to mount root fs on unknown-block�� +1. �� Unable to mount root fs on unknown-block�� - �� ```bash [ 0.390437] List of all partitions: - [ 0.390806] No filesystem could mount root, tried: + [ 0.390806] No filesystem could mount root, tried: [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ... [ 0.399404] Memory Limit: none [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- ``` - �� - + 1�� 2�� 644 �� - + 2. �� - + - �� -- 2.23.0

1 0

[PATCH v2 compass-ci] fix: qcow2rootfs/bin/common: unable to understand vmlinux
by Wang Chenglong 23 Dec '20

23 Dec '20

[error] [WARNNING] Restoring the owner of qcow2 file Finding vmlinuz under /tmp/a/boot ... /c/compass-ci/container/qcow2rootfs/bin/common: line 135: ROOTFS_VMLINUZ_FILE: unbound variable [why] some kernels are called vmlinx instead of vmlinz. --- container/qcow2rootfs/bin/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/qcow2rootfs/bin/common b/container/qcow2rootfs/bin/common index da39f52..baf936c 100755 --- a/container/qcow2rootfs/bin/common +++ b/container/qcow2rootfs/bin/common @@ -127,7 +127,7 @@ get_rootfs_kernel() { exit 5 } local vmlinuz_file vmlinuz kernel - vmlinuz_file=$(find ./boot -name "vmlinuz-*" |grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz_file=$(find ./boot -name "vmlinu[z|x]-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel -- 2.23.0

1 0

[PATCH compass-ci] fix: qcow2rootfs/bin/common: unable to understand vmlinux
by Wang Chenglong 23 Dec '20

23 Dec '20

[error] [WARNNING] Restoring the owner of qcow2 file Finding vmlinuz under /tmp/a/boot ... /c/compass-ci/container/qcow2rootfs/bin/common: line 135: ROOTFS_VMLINUZ_FILE: unbound variable [why] some kernels are called vmlinx instead of vmlinz. --- container/qcow2rootfs/bin/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/qcow2rootfs/bin/common b/container/qcow2rootfs/bin/common index da39f52..336602e 100755 --- a/container/qcow2rootfs/bin/common +++ b/container/qcow2rootfs/bin/common @@ -127,7 +127,7 @@ get_rootfs_kernel() { exit 5 } local vmlinuz_file vmlinuz kernel - vmlinuz_file=$(find ./boot -name "vmlinuz-*" |grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz_file=$(find ./boot -name "vmlinu[z|x]-*" |grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel -- 2.23.0

2 2

Re: [PATCH lkp-tests] Makefile: delete set-env in makefile
by Liu Shaofei 22 Dec '20

22 Dec '20

Reply-To: In-Reply-To: <20201222125413.2442750-1-weijihuiall(a)163.com> The Titel: makefile => Makefile? Thanks, Liushaofei On Tue, Dec 22, 2020 at 08:54:13PM +0800, Wei Jihui wrote: >set-env is move to install-dependencies.sh > >Signed-off-by: Wei Jihui <weijihuiall(a)163.com> >--- > Makefile | 1 - > 1 file changed, 1 deletion(-) > >diff --git a/Makefile b/Makefile >index 0b2d09a2..236bea25 100644 >--- a/Makefile >+++ b/Makefile >@@ -14,7 +14,6 @@ subsystem: > install: > mkdir -p $(TARGET_DIR_BIN) > ln -sf $(shell pwd)/bin/lkp $(TARGET_DIR_BIN)/lkp >- bash sbin/set-env.sh > bash sbin/install-dependencies.sh > > .PHONY: doc >-- >2.23.0 >

1 0

[PATCH compass-ci] result-webdav: if judgment error and some file symlinks are not created
by Cui Yili 22 Dec '20

22 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index f077fca..1feddd4 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -46,7 +46,7 @@ http { link_name = "latest" else tmp_name = string.match(upload_file, "(.*)_%d+%.cgz") - if not tmp_name then + if tmp_name then link_name = tmp_name else return -- 2.23.0

2 1

[PATCH lkp-tests] Makefile: delete set-env in makefile
by Wei Jihui 22 Dec '20

22 Dec '20

set-env is move to install-dependencies.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- Makefile | 1 - 1 file changed, 1 deletion(-) diff --git a/Makefile b/Makefile index 0b2d09a2..236bea25 100644 --- a/Makefile +++ b/Makefile @@ -14,7 +14,6 @@ subsystem: install: mkdir -p $(TARGET_DIR_BIN) ln -sf $(shell pwd)/bin/lkp $(TARGET_DIR_BIN)/lkp - bash sbin/set-env.sh bash sbin/install-dependencies.sh .PHONY: doc -- 2.23.0

1 0

[PATCH lkp-tests] jobs: fix to report errors when submit jobs
by Liu Shaofei 22 Dec '20

22 Dec '20

[why] /home/liushaofei/lkp-tests/lib/job.rb:257:in `block (2 levels) in load': undefined method `start_with?' for :"# pkgbuild_repo examples":Symbol (NoMethodError) ERROR -- /home/liushaofei/lkp-tests/jobs/build-pkg.yaml: undefined method `start_with?' for :"# pkgbuild_repo examples":Symbol Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- jobs/build-pkg.yaml | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/jobs/build-pkg.yaml b/jobs/build-pkg.yaml index b17fd2ade..7a47bd724 100644 --- a/jobs/build-pkg.yaml +++ b/jobs/build-pkg.yaml @@ -1,9 +1,8 @@ -suite: build-pkg -category: functional - -# pkgbuild_repo examples +# the field of "pkgbuild_repo" refers to the path of PKGBUILD, the examples are as follow: # pkgbuild_repo: archlinux/aur-j/java8-openjdk # pkgbuild_repo: archlinux/community//keepalived/trunk # pkgbuild_repo: archlinux/packages//prometheus-node-exporter/trunk +suite: build-pkg +category: functional build-pkg: -- 2.23.0

1 0

[PATCH compass-ci] sparrow/4-docker/buildall: skip ssh-r
by Wu Zhende 22 Dec '20

22 Dec '20

Do not restart the ssh-r container randomly. Otherwise, the established connection will be disconnected, and the requested hosts can't be logged in. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sparrow/4-docker/buildall | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 98d3ce4..32801d1 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -31,6 +31,7 @@ do_one() mkdir $tmpdir/$container_name 2>/dev/null && ( cd $container + [ "$container_name" == 'ssh-r' ] && exit if [ "$action" != "reboot" ]; then [ -x build ] && ./build [ -x install ] && ./install -- 2.23.0

1 0

[PATCH compass-ci] lib/constants: MAIL_HOST=>SEND_MAIL_HOST, MAIL_PORT=>SEND_MAIL_PORT
by Cao Xueliang 22 Dec '20

22 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/constants.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/constants.rb b/lib/constants.rb index 3d1ef1d..f0938f2 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -8,8 +8,8 @@ config = cci_defaults ES_HOST = config['ES_HOST'] || '172.17.0.1' ES_PORT = config['ES_PORT'] || 9200 -MAIL_HOST = config['MAIL_HOST'] || '172.17.0.1' -MAIL_PORT = config['MAIL_PORT'] || 11311 +MAIL_HOST = config['SEND_MAIL_HOST'] || '172.17.0.1' +MAIL_PORT = config['SEND_MAIL_PORT'] || 49000 SRV_HTTP_HOST = config['SRV_HTTP_HOST'] || ENV['SRV_HTTP_HOST'] || '172.17.0.1' SRV_HTTP_PORT = config['SRV_HTTP_PORT'] || ENV['SRV_HTTP_PORT'] || 11300 -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib: add assign_account_client
by Luan Shengde 22 Dec '20

22 Dec '20

add assign_account_client for code calling to assign account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign_account_client.rb | 83 ++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100755 lib/assign_account_client.rb diff --git a/lib/assign_account_client.rb b/lib/assign_account_client.rb new file mode 100755 index 0000000..d7e0623 --- /dev/null +++ b/lib/assign_account_client.rb @@ -0,0 +1,83 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require 'set' +require 'optparse' +require_relative '../container/defconfig' +require_relative 'es_client' +require_relative '../container/mail-robot/lib/assign-account-email' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_HOST + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) +JUMPER_HOST = defaults['JUMPER_HOST'] +JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 +LAB = ENV['lab'] + +# used for other codes calling to assign account for user +class AutoAssignAccount + def initialize(user_info) + @my_info = user_info + + @my_info_es = {} + end + + def update_from_es + account_infos = ESQuery.new(index: 'accounts') + my_account_info = account_infos.query_by_id(@my_info['my_email']) || {} + + my_account_info.update @my_info + @my_info.update my_account_info + @my_info['my_ssh_pubkey'] = [] unless my_account_info['my_ssh_pubkey'] + + @my_info['my_token'] = %x(uuidgen).chomp if @my_info['my_login_name'].nil? + end + + def apply_account + apply_info = {} + + apply_info.update @my_info + apply_info['enable_login'] = true + apply_info['is_update_account'] = true unless apply_info['my_login_name'].nil? + apply_info['lab'] = LAB + + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') + JSON.parse account_info_str + end + + def update_my_info_from_account_info + account_info = apply_account + + @my_info['my_login_name'] = account_info['my_login_name'] + @my_info['my_ssh_pubkey'] << account_info['my_jumper_pubkey'] unless account_info['my_jumper_pubkey'].nil? + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def send_mail + message = build_apply_account_email(@my_info) + %x(curl -XPOST "#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text" -d "#{message}") + end + + def send_account + update_from_es + update_my_info_from_account_info + + store_account_info + send_mail + end +end -- 2.23.0

1 1

[PATCH compass-ci 2/2] lib/mail-post: add send_mail_encode function
by Luan Shengde 22 Dec '20

22 Dec '20

[why] the http protocol will parse in fail if the request data has special characters. or it's hard to analysis the data if encoded or not on the send mail service. add stand alone interface for it Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/lib/mail-post.rb b/lib/mail-post.rb index df4d255..ac455fa 100644 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -8,6 +8,7 @@ require 'json' require 'yaml' require 'open3' require 'mail' +require 'base64' set :bind, '0.0.0.0' set :port, ENV['SEND_MAIL_PORT'] @@ -37,6 +38,19 @@ post '/send_mail_text' do send_mail(mail_info) end +post '/send_mail_encode' do + data_decode = Base64.decode64(request.body.read) + data = Mail.read_from_string(data_decode) + + mail_info = { + 'subject' => data.subject, + 'to' => data.to, + 'body' => data.body.decoded + } + + send_mail(mail_info) +end + def check_send_mail(mail_info) raise 'no/empty subject.' if mail_info['subject'].nil? || mail_info['subject'].empty? raise 'no/empty email_to address.' if mail_info['to'].nil? || mail_info['to'].empty? -- 2.23.0

1 0

[PATCH lkp-tests] tests/cci-makepkg: change to logic condition
by Liu Shaofei 22 Dec '20

22 Dec '20

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/cci-makepkg | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a50a40c20..d48e8495e 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -41,6 +41,8 @@ curl_pkgfile() download_pkgfile() { + [ -n "$PKGBUILD_TAG" ] || return + local pkg_tag_file="PKGBUILD-$PKGBUILD_TAG" curl_pkgfile "pkg/$benchmark" "$pkg_tag_file" @@ -93,7 +95,7 @@ centos_update_gcc() source /opt/rh/devtoolset-8/enable } -[ -n "$PKGBUILD_TAG" ] && download_pkgfile +download_pkgfile distro_install_depends lkp-dev -- 2.23.0

1 0

[PATCH lkp-tests 2/2] lib/env.sh: refactor set-env
by Wei Jihui 22 Dec '20

22 Dec '20

move sbin/set-env.sh functions to lib/env.sh delete sbin/set-env.sh do set-env at sbin/install-dependencies Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/env.sh | 30 ++++++++++++++++++++++++++ sbin/install-dependencies.sh | 10 ++++++++- sbin/set-env.sh | 42 ------------------------------------ 3 files changed, 39 insertions(+), 43 deletions(-) delete mode 100755 sbin/set-env.sh diff --git a/lib/env.sh b/lib/env.sh index 8ac42a83..9f37d8cc 100755 --- a/lib/env.sh +++ b/lib/env.sh @@ -125,3 +125,33 @@ shell_profile() [ -f "$profile" ] || touch $profile echo $profile } + +write_shell_profile() +{ + shell_profile_file=$(shell_profile) + if [ $# -gt 0 ]; then + echo "$@" >> $shell_profile_file + fi + + source $shell_profile_file +} + +write_host() +{ + if is_system "Linux"; then + nr_cpu=$(nproc) + memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') + else + nr_cpu=$(sysctl -n hw.logicalcpu) + memory_total=$(top -l 1 | grep MemRegions | awk '{print $2}') + fi + memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') + memory=$(echo $memory_new | awk '{print int($0)+1}')G + + cat > hosts/$(hostname) <<-EOF + nr_cpu: $nr_cpu + memory: $memory + hdd_partitions: + ssd_partitions: + EOF +} diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 6263dc91..8a4b29cd 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -4,7 +4,6 @@ SCRIPT_DIR=$(dirname $(realpath $0)) PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/env.sh -. $PROJECT_DIR/sbin/set-env.sh # choose install function base on DISTRIBUTION linux_dep() @@ -48,4 +47,13 @@ run() fi } +set_env() +{ + write_host + write_shell_profile "export LKP_SRC=$PWD" + write_shell_profile "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" + source $(shell_profile) +} + +set_env run diff --git a/sbin/set-env.sh b/sbin/set-env.sh deleted file mode 100755 index 8234eaf8..00000000 --- a/sbin/set-env.sh +++ /dev/null @@ -1,42 +0,0 @@ -#!/bin/sh - -SCRIPT_DIR=$(dirname $(realpath $0)) -PROJECT_DIR=$(dirname $SCRIPT_DIR) - -. $PROJECT_DIR/lib/env.sh - -write_shell_profile() -{ - shell_profile_file=$(shell_profile) - if [ $# -gt 0 ]; then - echo $@ >> $shell_profile_file - else - echo "export LKP_SRC=$PWD" >> $shell_profile_file - echo "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" >> $shell_profile_file - fi - - source $shell_profile_file -} - -write_host() -{ - if is_system "Linux"; then - nr_cpu=$(nproc) - memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') - else - nr_cpu=$(sysctl -n hw.logicalcpu) - memory_total=$(top -l 1 | grep MemRegions | awk '{print $2}') - fi - memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') - memory=$(echo $memory_new | awk '{print int($0)+1}')G - - cat > hosts/$(hostname) <<-EOF - nr_cpu: $nr_cpu - memory: $memory - hdd_partitions: - ssd_partitions: - EOF -} - -write_shell_profile -write_host -- 2.23.0

1 0

[PATCH lkp-tests 1/2] sbin/install-dependencies: adjust file format
by Wei Jihui 22 Dec '20

22 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/install-dependencies.sh | 39 ++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 20 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 3a206134..6263dc91 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -6,28 +6,27 @@ PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/env.sh . $PROJECT_DIR/sbin/set-env.sh -# choose install function base on -# DISTRIBUTION +# choose install function base on DISTRIBUTION linux_dep() { -source /etc/os-release -case $ID in -ubuntu|debian) - export DEBIAN_FRONTEND=noninteractive - apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client - ;; -openEuler|fedora|rhel|centos) - if type dnf > /dev/null 2>&1; then - dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - else - yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - fi - sudo gem install -f git activesupport rest-client - ;; -*) - echo "$ID not support! please install dependencies manually." - ;; -esac + source /etc/os-release + case $ID in + ubuntu|debian) + export DEBIAN_FRONTEND=noninteractive + apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client + ;; + openEuler|fedora|rhel|centos) + if type dnf > /dev/null 2>&1; then + dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + else + yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + fi + sudo gem install -f git activesupport rest-client + ;; + *) + echo "$ID not support! please install dependencies manually." + ;; + esac } -- 2.23.0

1 0

[PATCH compass-ci] compare_matrixes.rb : reduce compare time
by Li Yuanchao 22 Dec '20

22 Dec '20

When there is multi object group, don't use one object group to compare. Cause one object groups are too many, they spend too much time. And they are not what we want. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/compare_matrixes.rb | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 6e4c2aa..21bd5c9 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -390,7 +390,10 @@ end def compare_group_matrices(group_matrices, suites_hash, options) result_str = '' group_matrices_array = sort_by_matrix_size(group_matrices) + have_multi_member = group_matrices_array[0][1].size > 1 group_matrices_array.each do |matrice_kv| + next if have_multi_member && matrice_kv[1].size < 2 + result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_hash[matrice_kv[0]], options) end result_str -- 2.23.0

1 0

[PATCH lkp-tests] jobs/bisect.yaml: dc-1g--delimiter => dc-1g
by Cao Xueliang 22 Dec '20

22 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- jobs/bisect.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jobs/bisect.yaml b/jobs/bisect.yaml index 5fab30bfc..11ffc7c05 100644 --- a/jobs/bisect.yaml +++ b/jobs/bisect.yaml @@ -1,4 +1,4 @@ suite: bisect docker_image: delimiter:latest -testbox: dc-1g--delimiter +testbox: dc-1g bisect: -- 2.23.0

1 0

[PATCH compass-ci] doc/tutorial.md: fix abnormal display in Markdown
by Lu Kaiyi 22 Dec '20

22 Dec '20

the middle part in * and * is italics for markdown document, if want show '*' in this condition, need use \* escape '*' character. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- doc/tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..19d5741 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -258,7 +258,7 @@ job yaml ��[YAML](http://yaml.org/YAM 4) �� archlinux �� PKGBUILD �� PKGBUILD �� - PKGBUILD �� /srv/git/archlinux/*/*/PKGBUILD�� + PKGBUILD �� /srv/git/archlinux/\*/\*/PKGBUILD�� ### depends -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: auto_tool => assign_account_client
by Cao Xueliang 22 Dec '20

22 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 5e296a6..1d4ed69 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -4,7 +4,7 @@ require 'json' require 'base64' require_relative 'git' require_relative 'mail_client' -require_relative 'auto_tool' +require_relative 'assign_account_client' # compose and send email for bisect result class MailBisectResult -- 2.23.0

1 0

[PATCH compass-ci] web_backend.rb: optimize var name
by Li Yuanchao 22 Dec '20

22 Dec '20

As the implementation of method combine_group_query_data changes, var name suit_list is ok, cause the method return a hash in fact. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 2405107..0ca71e1 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -186,15 +186,15 @@ end def do_get_groups_matrices(must, dimension, total, size, from) job_list = get_dimension_job_list(dimension, must, size, from) - matrices, suites_list = combine_group_query_data(job_list, dimension) + matrices, suites_hash = combine_group_query_data(job_list, dimension) while matrices.empty? from += size break if from > total job_list = get_dimension_job_list(dimension, must, size, from) - matrices, suites_list = combine_group_query_data(job_list, dimension) + matrices, suites_hash = combine_group_query_data(job_list, dimension) end - [matrices, suites_list] + [matrices, suites_hash] end def get_groups_matrices(conditions, dimension, must, size, from) @@ -209,11 +209,11 @@ end def get_compare_body(params) dimension, conditions = get_dimension_conditions(params) must = get_es_must(params) - groups_matrices, suites_list = get_groups_matrices(conditions, dimension, must, COMPARE_RECORDS_NUMBER, 0) + groups_matrices, suites_hash = get_groups_matrices(conditions, dimension, must, COMPARE_RECORDS_NUMBER, 0) if !groups_matrices || groups_matrices.empty? body = 'No Data.' else - body = compare_group_matrices(groups_matrices, suites_list, { no_print: true }) + body = compare_group_matrices(groups_matrices, suites_hash, { no_print: true }) body = 'No Difference.' if !body || body.empty? end return body -- 2.23.0

1 0

[PATCH lkp-tests] sbin/install-dependencies: adjuest file format
by Wei Jihui 22 Dec '20

22 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/install-dependencies.sh | 39 ++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 20 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 3a206134..6263dc91 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -6,28 +6,27 @@ PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/env.sh . $PROJECT_DIR/sbin/set-env.sh -# choose install function base on -# DISTRIBUTION +# choose install function base on DISTRIBUTION linux_dep() { -source /etc/os-release -case $ID in -ubuntu|debian) - export DEBIAN_FRONTEND=noninteractive - apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client - ;; -openEuler|fedora|rhel|centos) - if type dnf > /dev/null 2>&1; then - dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - else - yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - fi - sudo gem install -f git activesupport rest-client - ;; -*) - echo "$ID not support! please install dependencies manually." - ;; -esac + source /etc/os-release + case $ID in + ubuntu|debian) + export DEBIAN_FRONTEND=noninteractive + apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client + ;; + openEuler|fedora|rhel|centos) + if type dnf > /dev/null 2>&1; then + dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + else + yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + fi + sudo gem install -f git activesupport rest-client + ;; + *) + echo "$ID not support! please install dependencies manually." + ;; + esac } -- 2.23.0

1 0

[PATCH lkp-tests] lib/env.sh: fix the check result of is_virt
by Zhang Yu 22 Dec '20

22 Dec '20

Hypervisor is a software that simulates hardware resources and scheduling on a physical machine. so if grep success, return 1 means it's a physical machine Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- lib/env.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/env.sh b/lib/env.sh index 8ac42a836..a1d5e445b 100755 --- a/lib/env.sh +++ b/lib/env.sh @@ -69,7 +69,8 @@ is_virt() # run as root [ -n "$(virt-what)" ] else - grep -q -w hypervisor /proc/cpuinfo + grep -q -w hypervisor /proc/cpuinfo && return 1 + return 0 fi } -- 2.23.0

1 0

[PATCH compass-ci] 3-code/git: add repository version pointing
by Hu Xuejiao 22 Dec '20

22 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 494107e..2159ddc 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -13,11 +13,18 @@ cd /c || exit # modify and manual run for now: [ -d "compass-ci/.git" ] || { git clone https://gitee.com/wu_fengguang/compass-ci.git - ln -s compass-ci cci + + cd /c/compass-ci || exit + [ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" + + ln -s compass-ci /c/cci } [ -d "lkp-tests/.git" ] || { git clone https://gitee.com/wu_fengguang/lkp-tests.git + + cd /c/lkp-tests || exit + [ -n "$lkp_commitid" ] && git reset --hard "$lkp_commitid" } [ -d "upstream-repos/.git" ] || { -- 2.23.0

1 0

[PATCH lkp-tests] tests/deploy-cci: specifying the repository version
by Hu Xuejiao 22 Dec '20

22 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/deploy-cci | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/deploy-cci b/tests/deploy-cci index e296568f..55835544 100755 --- a/tests/deploy-cci +++ b/tests/deploy-cci @@ -19,6 +19,9 @@ mkdir -p /c cd /c || exit git clone https://gitee.com/wu_fengguang/compass-ci.git +cd /c/compass-ci || exit +[ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" + # Need mount tmpfs to fix compass-ci/container/os-nfs/start error # "exportfs: /exports/result does not support NFS export" mount -t tmpfs none /srv -- 2.23.0

1 0

[PATCH compass-ci] lib/git_bisect: return all errors to bisect mail.
by Cao Xueliang 22 Dec '20

22 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index bc1e035..f789a71 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -82,7 +82,6 @@ class GitBisect result = `git -C #{@work_dir} bisect run #{BISECT_RUN_SCRIPT} #{@bad_job_id} #{(a)error_id.inspect} #{@work_dir}` end Utils.create_bisect_log(@work_dir) - FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) puts "\nbisect result: #{result}" analyse_result(result) end @@ -92,10 +91,16 @@ class GitBisect return nil unless temp[0].include?('is the first bad commit') || temp[-1].include?('bisect run success') first_bad_commit = Utils.parse_first_bad_commit(result) - error_messages = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_errorid(@error_id) + bisect_error = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_errorid(@error_id) + all_errors = Utils.obt_errors(@work_dir, first_bad_commit) + first_bad_commit_result_root = Utils.obt_result_root_by_commit(first_bad_commit) + FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) - return Hash['repo' => @upstream_repo, 'commit' => first_bad_commit, - 'error_messages' => error_messages] + return Hash['repo' => @upstream_repo, + 'commit' => first_bad_commit, + 'bisect_error' => bisect_error, + 'all_errors' => all_errors, + 'first_bad_commit_result_root' => first_bad_commit_result_root] end # first search the good commit in db -- 2.23.0

1 0

[PATCH compass-ci] delimiter/utils: add the get all errors function
by Cao Xueliang 22 Dec '20

22 Dec '20

1. fix some rubocop warnings. 2. add get errors by commits - get the pre_commit by find_parent_commit - get cur_id and pre_id by obt_id_by_commit - get errors by get_compare_result 3. add get result root by commit Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 48 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 41803f3..372e821 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -4,10 +4,12 @@ require 'set' require 'json' +require 'yaml' require 'fileutils' require_relative './constants' require_relative '../../lib/sched_client' +require_relative '../../lib/compare_error_messages' require_relative "#{ENV['LKP_SRC']}/lib/monitor" # a utils module for delimiter service @@ -83,9 +85,20 @@ module Utils es = ESQuery.new new_job = es.query_by_id(new_job_id) - return 'bad' if new_job['stats'].key?(error_id) - return 'good' + status = new_job['stats'].key?(error_id) ? 'bad' : 'good' + puts "new_job_id: #{new_job_id}" + puts "upstream_commit: #{job['upstream_commit']}" + record_jobs(new_job_id, job['upstream_commit']) + + return status + end + + def record_jobs(job_id, job_commit) + FileUtils.mkdir_p TMP_RESULT_ROOT unless File.exist? TMP_RESULT_ROOT + commit_jobs = File.join(TMP_RESULT_ROOT, 'commit_jobs') + content = "#{job_commit}: #{job_id}" + File.open(commit_jobs, 'a+') { |f| f.puts content } end def get_account_info @@ -100,6 +113,8 @@ module Utils account_info = get_account_info raise "query #{DELIMITER_ACCONUT} account info failed!" unless account_info + record_jobs(job['id'], job['upstream_commit']) + job['suite'] = 'bisect' job['my_name'] = account_info['my_name'] job['my_email'] = account_info['my_email'] @@ -133,5 +148,34 @@ module Utils log_content.each { |line| f.puts(line) } end end + + def find_parent_commit(git_dir, commit) + response = %x(git -C #{git_dir} rev-parse #{commit}~1) + return response.chomp + end + + def obt_id_by_commit(commit) + puts "#{TMP_RESULT_ROOT}/commit_jobs" + content = YAML.load(File.open("#{TMP_RESULT_ROOT}/commit_jobs")) + puts content + content[commit] + end + + def obt_errors(git_dir, commit) + pre_commit = find_parent_commit(git_dir, commit) + obt_errors_by_commits(commit, pre_commit) + end + + def obt_errors_by_commits(cur_commit, pre_commit) + cur_id = obt_id_by_commit(cur_commit) + pre_id = obt_id_by_commit(pre_commit) + _, errors = get_compare_result(pre_id, cur_id) + return errors + end + + def obt_result_root_by_commit(commit) + id = obt_id_by_commit(commit) + ESQuery.new.query_by_id(id)['result_root'] + end end end -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_client: add send_mail_encode function
by Cao Xueliang 22 Dec '20

22 Dec '20

If some special characters the payload of post, we use base64 encode to pass the payload. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_client.rb | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/mail_client.rb b/lib/mail_client.rb index 1d0938a..b13492f 100644 --- a/lib/mail_client.rb +++ b/lib/mail_client.rb @@ -2,6 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'base64' require 'rest-client' require_relative 'constants.rb' @@ -18,4 +19,9 @@ class MailClient resource = RestClient::Resource.new("http://#{@host}:#{@port}/send_mail_yaml") resource.post(mail_json) end + + def send_mail_encode(mail_data) + resource = RestClient::Resource.new("http://#{@host}:#{@port}/send_mail_encode") + resource.post(Base64.encode64(mail_data)) + end end -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: delete input job key which is path
by Zhang Yu 22 Dec '20

22 Dec '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- lib/job.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job.rb b/lib/job.rb index 2ed06edc6..7553312ed 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -254,6 +254,7 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end + hash.delete_if { |key, _| key.start_with?('#!') } hash.merge!(@overrides) @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end -- 2.23.0

1 0

[PATCH compass-ci] fix 404 error
by Li Ping 22 Dec '20

22 Dec '20

Signed-off-by: Li Ping <1477412247(a)qq.com> --- README.en.md | 4 ++-- README.zh.md | 4 ++-- ...34\272\350\260\203\346\265\213\344\273\273\345\212\241.md" | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/README.en.md b/README.en.md index 5e9386e..c082167 100644 --- a/README.en.md +++ b/README.en.md @@ -34,13 +34,13 @@ If a new error ID is generated, the bisect is automatically triggered to locate 2. Run the **git push** command to update the repository and automatically trigger the test. -3. You can [view](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/brow… and [compare](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/c… test results on the web page: https://compass-ci.openeuler.org/jobs. +3. You can [view](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/brow… and [compare](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/c… test results on the web page: https://compass-ci.openeuler.org/jobs **Manually Submitting a Test Task** 1. [Install the Compass-CI client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E…. 2. [Compile test cases and manually submit test tasks](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcas…. -3. You can [view](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/brow… and [compare](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/c… test results on the web page: https://compass-ci.openeuler.org/jobs. +3. You can [view](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/brow… and [compare](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/c… test results on the web page: https://compass-ci.openeuler.org/jobs **Logging in to the Test Environment** diff --git a/README.zh.md b/README.zh.md index 0616bec..c2140b0 100644 --- a/README.zh.md +++ b/README.zh.md @@ -37,13 +37,13 @@ Compass-CI 监控很多开源软件 git repos，一旦检测到代码更新， 2. 执行 git push 命令更新仓库，自动触发测试。 -3. 在网页中[查看](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/b… web: https://compass-ci.openeuler.org/jobs。 +3. 在网页中[查看](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/b… web: https://compass-ci.openeuler.org/jobs **手动提交测试任务** 1. [安装 Compass-CI 客户端](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E6%9… 2. [编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testca…, [手动提交测试任务](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… -3. 在网页中[查看](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/b… web: https://compass-ci.openeuler.org/jobs。 +3. 在网页中[查看](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/b… web: https://compass-ci.openeuler.org/jobs **登录测试环境** diff --git "a/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" index b59431e..7a54505 100644 --- "a/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" +++ "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" @@ -3,8 +3,8 @@ # 1. 前提条件： ## 请先学习: * [配置个人邮箱](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/ap… -* [如何申请测试机](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/如… -* [submit命令详解](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… +* [如何申请测试机](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/b… +* [submit命令详解](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… # 2. 操作方法 -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: optimize mail bisect content
by Cao Xueliang 22 Dec '20

22 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 63 ++++++++++++++++++++++++++------------- 1 file changed, 42 insertions(+), 21 deletions(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 090855e..5e296a6 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -1,48 +1,69 @@ # frozen_string_literal: true -require_relative 'mail_client' -require_relative 'git' require 'json' +require 'base64' +require_relative 'git' +require_relative 'mail_client' +require_relative 'auto_tool' # compose and send email for bisect result class MailBisectResult def initialize(bisect_info) - @error_messages = bisect_info['error_messages'] + @bisect_error = bisect_info['bisect_error'] + @all_errors = bisect_info['all_errors'] @repo = bisect_info['repo'] @commit_id = bisect_info['commit'] + @first_bad_commit_result_root = bisect_info['first_bad_commit_result_root'] @git_commit = GitCommit.new(@repo, @commit_id) + # now send mail to review + @to = 'caoxl(a)crystal.ci, caoxl78320(a)163.com, huming15(a)163.com' end def create_send_email - compose_mail - send_mail + send_report_mail(compose_mail) + send_account_mail end def compose_mail - subject = "[Compass-CI][#{(a)repo.split('/')[1]}]: #{(a)error_messages[0].split("\n")[0]}" - job_url = ENV['result_root'] ? "job url: http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}#{ENV['result_root']}\n" : '' - body = <<~BODY - Hi #{(a)git_commit.author_name}, + subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" + prefix_srv = "http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}" + bisect_job_url = ENV['result_root'] ? "bisect job info: #{prefix_srv}#{ENV['result_root']}\n" : '' + first_bad_commit_job_url = "first bad commit job info: #{prefix_srv}#{@first_bad_commit_result_root}\n" - git url: #{(a)git_commit.url} - git commit: #{@commit_id[0..11]} ("#{(a)git_commit.subject}") + data = <<~BODY + To: #{@to} + Subject: #{subject} - gcc version: 7.3.0 + Hi #{(a)git_commit.author_name}, - error_messages: - #{(a)error_messages.join("\n")} + url: #{(a)git_commit.url} + commit: #{@commit_id} ("#{(a)git_commit.subject}") + compiler: gcc (GCC) 7.3.0 - #{job_url} + all errors/warnings: + #{@all_errors} + + #{bisect_job_url} + #{first_bad_commit_job_url} Regards, Compass CI team BODY - to = 'caoxl(a)crystal.ci' - @hash = { 'to' => to, 'body' => body, 'subject' => subject } - # @hash = { 'to' => @git_commit.author_email, 'body' => body, 'subject' => subject } + + return data end - def send_mail - json = @hash.to_json - MailClient.new.send_mail(json) + def send_report_mail(mail_data) + MailClient.new.send_mail_encode(mail_data) + end + + def send_account_mail + user_info = { + 'my_email' => @to, + 'my_name' => @git_commit.author_name, + 'my_commit_url' => "#{@git_commit.url}/commit/#{@commit_id}" + } + + apply_account = AutoAssignAccount.new(user_info) + apply_account.send_account end end -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: temporarily cancel account verification
by Wu Zhende 22 Dec '20

22 Dec '20

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index e6f7e6a..2554db1 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -347,13 +347,14 @@ class Job error_msg = "Failed to verify the account.\n" error_msg += "Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco…" account_info = @es.get_account(self["my_email"]) - raise account_info unless account_info.is_a?(JSON::Any) + # raise account_info unless account_info.is_a?(JSON::Any) - @account_info = account_info.as_h + # @account_info = account_info.as_h + @account_info = account_info.as_h if account_info.is_a?(JSON::Any) - raise error_msg if @account_info["found"]? == false - raise error_msg unless self["my_name"] == @account_info["my_name"].to_s - raise error_msg unless self["my_token"] == @account_info["my_token"] + # raise error_msg if @account_info["found"]? == false + # raise error_msg unless self["my_name"] == @account_info["my_name"].to_s + # raise error_msg unless self["my_token"] == @account_info["my_token"] @hash.delete("my_uuid") @hash.delete("my_token") -- 2.23.0

1 0

[PATCH lkp-tests] adaptation/archlinux: add archlinux mapping
by Wang Yong 22 Dec '20

22 Dec '20

add mapping in order to run cci-depends job for build-pkg Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/adaptation/archlinux | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/distro/adaptation/archlinux b/distro/adaptation/archlinux index 89163cf41..c52212e08 100644 --- a/distro/adaptation/archlinux +++ b/distro/adaptation/archlinux @@ -72,3 +72,11 @@ libtirpc: libtirpc1 initramfs-tools: openssh-server: openssh openssh-client: openssh +autopoint: gettext +g++: gcc +libarchive13: libarchive +libbz2-1.0: bzip2 +libssl: openssl +libtool-bin: libtool +linux-perf: perf +ruby-dev: ruby -- 2.23.0

1 0