- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 3/3] libvirt/lib: request scheduler to consume job
by Xiao Shenwei 22 Jan '21

22 Jan '21

request url: http://#{@sched_host}:#{@sched_port}/boot.libvirt/mac/#{@info['mac']} Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/consumer.rb | 85 +++++++++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 providers/libvirt/lib/consumer.rb diff --git a/providers/libvirt/lib/consumer.rb b/providers/libvirt/lib/consumer.rb new file mode 100644 index 0000000..f1477fe --- /dev/null +++ b/providers/libvirt/lib/consumer.rb @@ -0,0 +1,85 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'yaml' +require 'json' +require 'rest-client' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +# register hostname mac queues +# request scheduler +class Consumer + attr_reader :info + + def initialize(hostname, queues, logger) + @logger = logger + @info = { 'hostname' => hostname, 'queues' => queues } + host_exists + parse_mac_from_hostname + defconfig + register_info + end + + def connect + res = RestClient.get("http://#{@sched_host}:#{@sched_port}/boot.libvirt/mac/#{@info['mac']}") + if res.empty? + @logger.error('Can not connect scheduler') + raise 'Can not connect scheduler' + end + JSON.parse(res) + end + + def clean_info + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/del_host_mac?mac=#{@info['mac']}", {} + ) + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/del_host2queues?host=#{@info['hostname']}", {} + ) + end + + private + + def defconfig + names = Set.new %w[ + SCHED_HOST + SCHED_PORT + ] + defaults = relevant_defaults(names) + @sched_host = defaults['SCHED_HOST'] || '172.17.0.1' + @sched_port = defaults['SCHED_PORT'] || 3000 + @logger.info("SCHED_HOST: #{@sched_host}") + @logger.info("SCHED_PORT: #{@sched_port}") + end + + def parse_mac_from_hostname + # TODO + # use digest/md5 replace + cmd = %(echo #{@info['hostname']} | + md5sum | + sed "s/^\$..\$\$..\$\$..\$\$..\$\$..\$.*$/0a-\\1-\\2-\\3-\\4-\\5/" + ) + @info['mac'] = %x(#{cmd}).chomp + @logger.info("Mac address: #{@info['mac']}") + end + + def register_info + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/set_host_mac?hostname=#{@info['hostname']}&mac=#{@info['mac']}", {} + ) + RestClient.put( + "http://#{@sched_host}:#{@sched_port}/set_host2queues?host=#{@info['hostname']}&queues=#{@info['queues']}", {} + ) + end + + def host_exists + @info['host'] = @info['hostname'].split('.')[0] + host_file = "#{ENV['LKP_SRC']}/hosts/#{@info['host']}" + return if FileTest.exists?(host_file) + + @logger.error("The file does't exist: #{host_file}") + raise "The file does't exist: #{host_file}" + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] providers/libvirt: add a libvirt provider
by Xiao Shenwei 22 Jan '21

22 Jan '21

libvirt is a vm management tool, it based on a domain.xml to run vm each element of xml can be parsed qemu-kvm parameter. so we can configure xml file to run different type of vm Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/client.rb | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100755 providers/libvirt/client.rb diff --git a/providers/libvirt/client.rb b/providers/libvirt/client.rb new file mode 100755 index 0000000..29cdaf3 --- /dev/null +++ b/providers/libvirt/client.rb @@ -0,0 +1,23 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative 'lib/mylog' +require_relative 'lib/consumer' + +def main(hostname, queues) + puts "cat #{hostname}.log" + logger = Mylog.new("#{hostname}.log") + consumer = Consumer.new(hostname, queues, logger) + response = consumer.connect + if response.nil? || response['job_id'].empty? + logger.warn('no job now') + puts '----------' + puts 'no job now' + puts '----------' + return + end + ## clean env + consumer.clean_info +end -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow: use nokogiri to parse xml file
by Xiao Shenwei 22 Jan '21

22 Jan '21

[why] libvirt based on xml file to run vm, so we need parse xml file nokogiri is a excellent tool of parse xml Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sparrow/0-package/common | 2 +- sparrow/0-package/os/openEuler | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 2d7c9e3..1eb175c 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -2,7 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch +gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch nokogiri grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^mailer:' /etc/passwd || useradd -u 1144 mailer diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 61a37c4..e815fd1 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -24,6 +24,7 @@ pkgs=( libvirt curl sudo + zlib-devel ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

2 2

[PATCH v6 lkp-tests] reset machine testbox by job
by Zhang Dewan 22 Jan '21

22 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 10 ++++++++++ tests/reset-tbox | 46 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+) create mode 100644 jobs/reset-tbox.yaml create mode 100755 tests/reset-tbox diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml new file mode 100644 index 00000000..52ea76b7 --- /dev/null +++ b/jobs/reset-tbox.yaml @@ -0,0 +1,10 @@ +suite: reset-tbox +category: functional + +testbox: dc-8g +os_mount: container +docker_image: reset-tbox:latest +queue: reset-tbox + +reset-tbox: +host_name: diff --git a/tests/reset-tbox b/tests/reset-tbox new file mode 100755 index 00000000..7f64faee --- /dev/null +++ b/tests/reset-tbox @@ -0,0 +1,46 @@ +#!/bin/bash +# host_name + +account_file=/etc/lab/.account.info +hosts_file=/etc/lab/hosts_file +command_file=/usr/bin/ipmitool + +file_check() +{ + for i in $account_file $hosts_file $command_file + do + [ -f $i ] || { + echo "File not found: $i" + exit 0 + } + done + source $account_file +} + +tbox_check() +{ + ibmc_ip=$(grep -w $host_name $hosts_file | awk '{print $1}') + + [[ "$ibmc_ip" =~ ^[0-9]+.* ]] || { + echo "Testbox not found: $host_name" + exit 1 + } +} + +reset_tbox() +{ + echo "reboot the tesetbox..." + ipmitool -I lanplus -H $ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset +} + +main() +{ + file_check + tbox_check + reset_tbox || { + echo "Failed to reboot the testbox: $host_name" + exit 2 + } +} + +main -- 2.23.0

1 0

[PATCH v5 lkp-tests] reset machine testbox by job
by Zhang Dewan 22 Jan '21

22 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 10 +++++++++ tests/reset-tbox | 52 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+) create mode 100644 jobs/reset-tbox.yaml create mode 100755 tests/reset-tbox diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml new file mode 100644 index 00000000..52ea76b7 --- /dev/null +++ b/jobs/reset-tbox.yaml @@ -0,0 +1,10 @@ +suite: reset-tbox +category: functional + +testbox: dc-8g +os_mount: container +docker_image: reset-tbox:latest +queue: reset-tbox + +reset-tbox: +host_name: diff --git a/tests/reset-tbox b/tests/reset-tbox new file mode 100755 index 00000000..ae3cbe52 --- /dev/null +++ b/tests/reset-tbox @@ -0,0 +1,52 @@ +#!/bin/bash +# host_name + +account_file=/etc/lab/.account.info +hosts_file=/etc/lab/hosts_file +command_file=/usr/bin/ipmitool + +file_check() +{ + for i in $account_file $hosts_file $command_file + do + [ -f $i ] || { + echo "File not found: $i" + exit 0 + } + done + source $account_file +} + +tbox_check() +{ + ibmc_ip=$(grep -w $host_name $hosts_file | awk '{print $1}') + + [[ "$ibmc_ip" =~ ^[0-9]+.* ]] || { + echo "Testbox not found: $host_name" + exit 1 + } +} + +reset_tbox() +{ + echo "reboot the tesetbox..." + ipmitool -I lanplus -H $ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset +} + +result_check() +{ + if [ $? -ne 0 ] ; then + echo "Failed to reboot the testbox: $host_name" + fi + echo "done." +} + +main() +{ + file_check + tbox_check + reset_tbox + result_check +} + +main -- 2.23.0

2 2

[PATCH compass-ci] sparrow/1-storage: fix failed to set permission
by Liu Yinsi 22 Jan '21

22 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 ~/compass-ci/sparrow/1-storage# ./permission chown: changing ownership of '/srv/initrd': Read-only file system [why] mount -t cifs //$LKP_SERVER/initrd /srv/initrd -o port=446,guest,ro,hard,vers=1.0,noacl,nouser_xattr because mount permission for /srv/initrd is read-only, cause set permission failed. [how] for user install compass-ci at their local server, no need to mount directory, they will download rootfs by wget link, so remove it. mount rootfs can move to selftest own tests deploy-cci. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/tiny | 3 --- 1 file changed, 3 deletions(-) diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index eb7270d..e589937 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -26,6 +26,3 @@ dirs=( ) mkdir -p "${dirs[@]}" - -mount -t cifs //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr -mount -t cifs //$LKP_SERVER/initrd /srv/initrd -o port=446,guest,ro,hard,vers=1.0,noacl,nouser_xattr -- 2.23.0

2 2

[PATCH compass-ci] add a comment, make it easier to understand
by Zhang Dewan 22 Jan '21

22 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- providers/qemu/kvm.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index b4e87fc..9034ccf 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -89,6 +89,7 @@ set_initrd() set_bios() { bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd + # when arch='x86_64', use the follwing file: [ -f "$bios" ] || bios=/usr/share/ovmf/OVMF.fd } -- 2.23.0

2 4

[PATCH compass-ci] doc/manual: optimize submit-job.zh.md
by Wu Zhende 22 Jan '21

22 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- doc/manual/submit-job.zh.md | 34 ++++++++++++---------------------- 1 file changed, 12 insertions(+), 22 deletions(-) diff --git a/doc/manual/submit-job.zh.md b/doc/manual/submit-job.zh.md index 540d91f..92faa5c 100644 --- a/doc/manual/submit-job.zh.md +++ b/doc/manual/submit-job.zh.md @@ -8,32 +8,22 @@ submit 命令的作用是提交测试任务。该命令提供了多个选项帮助用户更灵活的提交任务，您可以在命令行直接输入 summit 命令来查看帮助信息，并根据实际需求灵活使用。 - ### 基本用法 -测试任务以 yaml 文件的方式提交，因此，您需要事先准备好测试任务的 yaml 文件（本文以 iperf.yaml 为例）。使用如下命令提交测试任务： - -``` -submit iperf.yaml -``` -如下所示： - -```shell -hi8109@account-vm ~% submit iperf.yaml -submit iperf.yaml, got job_id=z9.173924 -``` +测试任务以 yaml 文件的方式提交，因此，您需要事先准备好测试任务的 yaml 文件（本文以 iperf.yaml 为例）。 -本文所示的 yaml 文件均已添加 testbox 字段，若您提交的 yaml 文件不包含该字段将报错： +若您的yaml文件不包含testbox字段，直接提交将会报错： ```shell hi8109@account-vm ~% submit iperf.yaml submit iperf.yaml failed, got job_id=0, error: Missing required job key: 'testbox' ``` -您可以在 yaml 文件中添加 testbox 字段，或使用如下命令： +因为testbox是必填字段，您可以在 yaml 文件中添加 testbox 字段，或使用如下命令： -``` -submit iperf.yaml testbox=vm-2p8g +```shell +hi8109@account-vm ~% submit iperf.yaml testbox=vm-2p8g +submit iperf.yaml, got job_id=z9.173924 ``` testbox 字段的值指定需要的测试机，可以使用 `ll` 命令查看 `lkp-tests/hosts` 路径下的可选测试机。如下所示： @@ -115,7 +105,7 @@ options: 使用-o DIR 命令可以将最终生成的yaml文件保存到指定目录 DIR 下。示例如下所示： ``` - submit -o ~/iperf.yaml + submit -o ~/iperf.yaml testbox=vm-2p8g ``` 运行命令之后会在指定目录生成经过 submit 处理过的 yaml 文件。 @@ -126,7 +116,7 @@ options: 示例命令如下： ``` - submit -a iperf.yaml + submit -a iperf.yaml testbox=vm-2p8g ``` ``` @@ -141,13 +131,13 @@ options: 示例命令如下： ``` - submit -m iperf.yaml + submit -m iperf.yaml testbox=vm-2p8g ``` 控制台显示如下： ```shell - hi8109@account-vm ~% submit -m iperf.yaml + hi8109@account-vm ~% submit -m iperf.yaml testbox=vm-2p8g submit iperf.yaml, got job_id=z9.173923 query=>{"job_id":["z9.173923"]} connect to ws://172.168.131.2:11310/filter @@ -173,14 +163,14 @@ options: 示例命令如下： ``` - submit -m -c borrow-1h.yaml + submit -m -c borrow-1h.yaml testbox=vm-2p8g ``` 当我们提交一个申请设备的任务后，会获取到返回的登陆信息，如 `ssh ip -p port`，添加 -c 参数之后不需要我们手动输入 ssh 登陆命令来进入执行机。控制台显示如下： ```shell - hi8109@account-vm ~% submit -m -c borrow-1h.yaml + hi8109@account-vm ~% submit -m -c borrow-1h.yaml testbox=vm-2p8g submit borrow-1h.yaml, got job_id=z9.173925 query=>{"job_id":["z9.173925"]} connect to ws://172.168.131.2:11310/filter -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/job.cr: use an independent interface for submit
by Wu Zhende 22 Jan '21

22 Jan '21

When we use Job.new, we need to do some extra processing only when we submit the job. Use interface "initialized?" to determine whether to submit a job. However, there is a vulnerability. If a job with all fields initialized is submitted, the job can be successfully submitted without checking the account. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 29 ++++++++++------------------- src/scheduler/submit_job.cr | 1 + 2 files changed, 11 insertions(+), 19 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 9b52d88..964c520 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -55,25 +55,6 @@ class Job @es = Elasticsearch::Client.new @account_info = Hash(String, JSON::Any).new @log = JSONLogger.new - - # init job with "-1", or use the original job_content["id"] - id = "-1" if "#{id}" == "" - - if initialized? - if @hash["id"] == "#{id}" - return unless @hash.has_key?("my_uuid") || @hash.has_key?("my_token") - - check_account_info() - set_sshr_info() - return - end - end - - @hash["id"] = JSON::Any.new("#{id}") - - check_required_keys() - check_account_info() - set_defaults() end METHOD_KEYS = %w( @@ -155,6 +136,16 @@ class Job end end + def submit(id = nil) + # init job with "-1", or use the original job_content["id"] + id = "-1" if "#{id}" == "" + @hash["id"] = JSON::Any.new("#{id}") + + check_required_keys() + check_account_info() + set_defaults() + end + private def set_defaults append_init_field() set_docker_os() diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index bcb265f..1837cb1 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -7,6 +7,7 @@ class Sched job_content = JSON.parse(body) job = Job.new(job_content, job_content["id"]?) + job.submit(job_content["id"]?) job["commit_date"] = get_commit_date(job) cluster_file = job["cluster"] -- 2.23.0

1 0

[PATCH v2 compass-ci] add a comment, make it easier to understand
by Zhang Dewan 21 Jan '21

21 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- providers/qemu/kvm.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index b4e87fc..d445248 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -89,6 +89,7 @@ set_initrd() set_bios() { bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd + # when arch='x86_64', use the following file: [ -f "$bios" ] || bios=/usr/share/ovmf/OVMF.fd } -- 2.23.0

1 0

[PATCH lkp-tests] expose/report error on install_proxy()
by Wu Fengguang 21 Jan '21

21 Jan '21

Carefully preserve error code from external commands, and return error code to function callers. Check/report error, and fail it early. Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- bin/lkp-setup-rootfs | 28 +++++++++++++--------------- lib/http.sh | 8 ++++++++ 2 files changed, 21 insertions(+), 15 deletions(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index cca1b53be..cf48c8f6b 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -103,8 +103,6 @@ wget_install_cgz() install_proxy_yum() { - [ -d "/etc/yum/" ] || return - echo "proxy=http://$SQUID_HOST:$SQUID_PORT" >> /etc/yum.conf if ls /etc/yum.repos.d/*.repo >/dev/null 2>&1; then sed -i 's/https:/http:/g' /etc/yum.repos.d/*.repo @@ -115,8 +113,6 @@ install_proxy_yum() install_proxy_apt() { - [ -d "/etc/apt/" ] || return - cat << EOF > /etc/apt/apt.conf.d/proxy Acquire::http::proxy "http://$SQUID_HOST:$SQUID_PORT/"; Acquire::ftp::proxy "ftp://$SQUID_HOST:$SQUID_PORT/"; @@ -133,8 +129,8 @@ EOF install_docker_proxy() { - [ -n "$DOCKER_PROXY_HOST" ] || return - [ -n "$DOCKER_PROXY_PORT" ] || return + [ -n "$DOCKER_PROXY_HOST" ] || return 0 + [ -n "$DOCKER_PROXY_PORT" ] || return 0 mkdir -p /etc/systemd/system/docker.service.d cat << EOF > /etc/systemd/system/docker.service.d/http-proxy.conf @@ -156,30 +152,32 @@ EOF modify_gem_source() { echo ':sources: ["http://rubygems.org"]' >> /root/.gemrc - - return 0 } install_goproxy() { - mkdir -p /root/.config/go/ + mkdir -p /root/.config/go/ || return echo "GOPROXY=https://goproxy.io,direct" >> /root/.config/go/env - - return 0 } install_proxy() { - modify_gem_source + modify_gem_source || return install_docker_proxy - install_goproxy + install_goproxy || return [ -z "$SQUID_HOST" ] && return [ -z "$SQUID_PORT" ] && return - install_proxy_apt || install_proxy_yum + if [ -d "/etc/yum/" ]; then + install_proxy_yum + elif [ -d "/etc/apt/" ]; then + install_proxy_apt + else + : + fi } # Save the result to @@ -251,7 +249,7 @@ chmod_root_ssh() chmod 600 /root/.ssh/id_rsa } -install_proxy +install_proxy || exit_with_job_state 'error:proxy' chmod_root_ssh while true; do diff --git a/lib/http.sh b/lib/http.sh index a0cb33fc2..523c86333 100755 --- a/lib/http.sh +++ b/lib/http.sh @@ -60,8 +60,16 @@ jobfile_append_var() http_get_cgi "cgi-bin/lkp-jobfile-append-var?$query_str" } +exit_with_job_state() +{ + local exit_code=$? + set_job_state "$1" + exit $exit_code +} + set_job_state() { + [ -n "$1" ] || return jobfile_append_var "job_state=$1" } -- 2.23.0

1 0

help star
by Wu Fengguang 21 Jan '21

21 Jan '21

Xueliang is our help star again, thanks Xueliang! 23.2 Wu Fengguang 15.9 Cao Xueliang 13.7 Yu Chuan 13.4 Wei Jihui 12.3 Xiao Shenwei 10.9 Luan Shengde 8.4 Liu Yinsi 8.4 Liu Shaofei 7.9 Lin Jiaxin 7.6 Cui Yili 7.5 Wu Zhende 6.9 Zhang Dewan 6.8 Lu Weitao 4.7 Zhang Yale 4.4 Ren Wen 4.2 Bai Jing 3.9 Du Kaitian 3.9 Xu Xijian 3.8 Li Yuanchao 3.4 Wang Yong 3.2 Hu Xuejiao 2.2 Lu Kaiyi 2.1 Li Ping 1.3 Wang Chenglong 0.0 Tong Qunfeng 0.0 Wang Shuai 0.0 Cui Xucui

1 0

review star
by Wu Fengguang 21 Jan '21

21 Jan '21

Jihui is our review star again, thanks Jihui! Review email records in the last 7 days total review paragraph point people 61 48 122 39 Wu Fengguang 96 31 48 36 Wei Jihui 93 19 24 33 Cao Xueliang 35 18 33 17 Xiao Shenwei 38 19 52 15 Luan Shengde 58 13 16 6 Ren Wen 29 5 9 5 Lin Jiaxin 41 8 14 5 Liu Yinsi 23 2 4 4 Yu Chuan 12 5 6 4 Xu Xijian 13 7 12 4 Liu Shaofei 3 3 4 4 Wang Yong 33 3 9 3 Zhang Yale 45 5 10 3 Wu Zhende 35 2 7 2 Wang Chenglong 39 2 4 2 Lu Kaiyi 34 4 3 2 Zhang Dewan 15 2 2 1 Hu Xuejiao 10 2 2 1 Bai Jing 9 1 1 1 Li Yuanchao 28 5 4 1 Cui Yili 11 0 0 0 Li Ping 8 4 9 0 Lu Weitao

1 0

git count
by Wu Fengguang 21 Jan '21

21 Jan '21

Xueliang is our code star, thanks Xueliang! 9 Cao Xueliang 8 Wu Zhende 7 Liu Yinsi 6 Wei Jihui 6 Ren Wen 4 Xu Xijian 4 Liu Shaofei 4 Zhang Yale 3 Yu Chuan 3 Lin Jiaxin 2 Bai Jing 2 Hu Xuejiao 2 Lu Weitao 2 Xiao Shenwei 2 Wang Chenglong 2 Li Yuanchao 1 Wu Fengguang 1 Wang Yong 1 Luan Shengde 1 Li Ping 1 Lu Kaiyi 0 Tong Qunfeng 0 Du Kaitian 0 Sun Yukui 0 Zhang Yu 0 Cui Yili 0 Zhang Dewan 0 Shi Zhichao

1 0

Re: [PATCH v4 lkp-tests] add option '-l' for submit
by Wu Fengguang 21 Jan '21

21 Jan '21

On Tue, Jan 05, 2021 at 10:03:22AM +0800, Li Ping wrote: >[Usage]: >submit -m -l job.yaml > ==> lftp -c mirror http://SRV_HTTP_HOST:SRV_HTTP_PORT/RESULT_ROOT -l = lftp? "download results" can be more meaningful feature name. lftp is just one of the clients. Thanks, Fengguang

2 1

[PATCH lkp-tests] daptation/debian: add pgbench dependency
by Hu Xuejiao 21 Jan '21

21 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- distro/adaptation/debian | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/adaptation/debian b/distro/adaptation/debian index 90260f1d..15706cf6 100644 --- a/distro/adaptation/debian +++ b/distro/adaptation/debian @@ -21,3 +21,4 @@ libperl5: libperl5.24 perl-modules-5: perl-modules-5.24 libgcc-dev: libgcc-6-dev openjdk: openjdk-8-jdk +postgresql-server-dev-all: postgresql -- 2.23.0

1 0

Re: [PATCH v4 lkp-tests] add option '-l' for submit
by Wu Fengguang 21 Jan '21

21 Jan '21

TO ALL >+ srv_http_host = job['SRV_HTTP_HOST'] || '124.160.11.58' >+ srv_http_port = job['SRV_HTTP_PORT'] || '11300' DON't use IP address. Use domain name instead. Thanks, Fengguang

2 1

[PATCH lkp-tests] adaptation/debian: add pgbench dependency
by Hu Xuejiao 21 Jan '21

21 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- distro/adaptation/centos | 1 + distro/adaptation/debian | 1 + 2 files changed, 2 insertions(+) diff --git a/distro/adaptation/centos b/distro/adaptation/centos index 2cfe3877..8fbae666 100644 --- a/distro/adaptation/centos +++ b/distro/adaptation/centos @@ -335,3 +335,4 @@ liboath-dev: liboath-devel lsb-release: redhat-lsb linux-base: openssh-client: openssh-clients +postgresql-server: postgresql diff --git a/distro/adaptation/debian b/distro/adaptation/debian index 90260f1d..15706cf6 100644 --- a/distro/adaptation/debian +++ b/distro/adaptation/debian @@ -21,3 +21,4 @@ libperl5: libperl5.24 perl-modules-5: perl-modules-5.24 libgcc-dev: libgcc-6-dev openjdk: openjdk-8-jdk +postgresql-server-dev-all: postgresql -- 2.23.0

1 0

[PATCH v1 compass-ci 2/4] providers/libvirt: customizing logger format
by Xiao Shenwei 21 Jan '21

21 Jan '21

format: datetime-severity: message such as: 2021-01-20 16:04:03 +0800-INFO: xxxxxx Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/mylog.rb | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 providers/libvirt/lib/mylog.rb diff --git a/providers/libvirt/lib/mylog.rb b/providers/libvirt/lib/mylog.rb new file mode 100644 index 0000000..69061ca --- /dev/null +++ b/providers/libvirt/lib/mylog.rb @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' + +# define a logger format +class Mylog < Logger + def initialize(filename) + @name = filename + if FileTest.exist?(filename) + File.delete(filename) + end + super(filename) + self.datetime_format = '%Y-%m-%d %H:%M:%s' + self.formatter = proc do |severity, datetime, _, msg| + "#{datetime}-#{severity}: #{msg}\n" + end + end +end -- 2.23.0

3 7

[PATCH compass-ci] lib/job.cr: use an independent interface for submit
by Wu Zhende 21 Jan '21

21 Jan '21

When we use Job.new, we need to do some extra processing only when we submit the job. Use interface "initialized?" to determine whether to submit a job. However, there is a vulnerability. If a job with all fields initialized is submitted, the job can be successfully submitted without checking the account. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 29 ++++++++++------------------- src/scheduler/submit_job.cr | 1 + 2 files changed, 11 insertions(+), 19 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 9b52d88..964c520 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -55,25 +55,6 @@ class Job @es = Elasticsearch::Client.new @account_info = Hash(String, JSON::Any).new @log = JSONLogger.new - - # init job with "-1", or use the original job_content["id"] - id = "-1" if "#{id}" == "" - - if initialized? - if @hash["id"] == "#{id}" - return unless @hash.has_key?("my_uuid") || @hash.has_key?("my_token") - - check_account_info() - set_sshr_info() - return - end - end - - @hash["id"] = JSON::Any.new("#{id}") - - check_required_keys() - check_account_info() - set_defaults() end METHOD_KEYS = %w( @@ -155,6 +136,16 @@ class Job end end + def submit(id = nil) + # init job with "-1", or use the original job_content["id"] + id = "-1" if "#{id}" == "" + @hash["id"] = JSON::Any.new("#{id}") + + check_required_keys() + check_account_info() + set_defaults() + end + private def set_defaults append_init_field() set_docker_os() diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index bcb265f..1837cb1 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -7,6 +7,7 @@ class Sched job_content = JSON.parse(body) job = Job.new(job_content, job_content["id"]?) + job.submit(job_content["id"]?) job["commit_date"] = get_commit_date(job) cluster_file = job["cluster"] -- 2.23.0

2 1

[PATCH v3 lkp-tests] sbin/submit: add -f option for uploading libvirt config
by Ren Wen 21 Jan '21

21 Jan '21

An option for 'submit': -f, --upload-file FILE upload an XML template of libvirt FILE's extension should be '.xml'. Example: case1: submit -f existent_file.xml => existent_file/<md5sum>.xml case2: submit -f nonexistent_file.xml => Find no file: nonexistent_file.xml Signed-off-by: Ren Wen <15991987063(a)163.com> --- lib/job.rb | 11 +++++++++++ sbin/submit | 31 +++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 631ebfc98..812f8c299 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -16,6 +16,17 @@ require 'set' require 'pp' require 'English' +# copied from compass-ci/container/defconfig.rb +def cci_defaults + hash = {} + Dir.glob(['/etc/compass-ci/defaults/*.yaml', + '/etc/compass-ci/accounts/*.yaml', + "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| + hash.update YAML.load_file(file) || {} + end + hash +end + def restore(ah, copy) if ah.class == Hash ah.clear.merge!(copy) diff --git a/sbin/submit b/sbin/submit index 39cdcf576..1d56369ad 100755 --- a/sbin/submit +++ b/sbin/submit @@ -17,6 +17,7 @@ opt_auto_define_files = false opt_monitor = false opt_monitor_query = {} opt_my_queue = false +opt_upload_files = [] actions = ['output', 'stop'] options = OptionParser.new do |opts| @@ -64,10 +65,40 @@ options = OptionParser.new do |opts| opt_my_queue = true end + opts.on('-f FILE', '--upload-file FILE', "upload an XML template of libvirt. File: *.xml") do |file| + unless File.exist? file + puts "Find no file: #{file}" + exit 1 + end + + opt_upload_files << file + end + end options.parse!(ARGV) +def upload_configs(files) + defaults = cci_defaults() + upload_host = defaults['LKP_SERVER'] + upload_port = defaults['UPLOAD_LIBVIRT_PORT'] || 3081 + + files.each do |file| + response = `curl -sSf -T #{file} http://#{upload_host}:#{upload_port}/` + if $? == 0 + puts response + else + # TODO: how to deal with uploaded file when one failing + puts response + exit 1 + end + end + + exit 0 +end + +upload_configs(opt_upload_files) unless opt_upload_files.empty? + seen_job_yaml = false ARGV.delete_if do |arg| if arg.index '=' -- 2.23.0

3 4

[PATCH v4 lkp-tests] sbin/submit: add option -i for submit
by Wei Jihui 21 Jan '21

21 Jan '21

submit -i is for add other yamls content to job.yaml [usage] case 1: submit -i 1.yaml -i 2.yaml job.yaml var priority: 1.yaml < 2.yaml < job.yaml case 2: submit -i 1.yaml job.yaml -i 2.yaml var priority: 1.yaml < job.yaml < 2.yaml case 3: submit -i 1.yaml job1.yaml -i 2.yaml job2.yaml -i 3.yaml var priority: 1.yaml < job1.yaml < 2.yaml < 3.yaml var priority: 1.yaml < 2.yaml < job2.yaml < 3.yaml Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 17 +++++++++++++---- sbin/submit | 30 ++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 4 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 631ebfc9..640b25a5 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -215,6 +215,8 @@ class Job attr_accessor :overrides attr_accessor :defaults attr_accessor :cmdline_defaults + attr_accessor :default_yamls + attr_accessor :override_yamls def initialize(job = {}, defaults = {}, overrides = {}, cmdline_defaults={}) @job = job @@ -222,6 +224,8 @@ class Job @overrides = overrides # from command line @cmdline_defaults = cmdline_defaults # from command line @available_programs = {} + @default_yamls = [] + @override_yamls = [] end def source_file_symkey(file) @@ -254,8 +258,11 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end - hash.merge!(@overrides) - hash.merge!(fault_reproduction) if hash.has_key?("id") + + revise_hash(hash, load_include_yamls(@default_yamls), false) unless @default_yamls.empty? + revise_hash(hash, load_include_yamls(@override_yamls), true) unless @override_yamls.empty? + revise_hash(hash, @overrides, true) unless @overrides.empty? + @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end rescue StandardError => e @@ -286,8 +293,10 @@ class Job @jobfile = jobfile end - def fault_reproduction - wait = load_yaml("#{lkp_src}/jobs/fault_reproduction.yaml", {}) + def load_include_yamls(include_yamls) + include_hash = {} + include_hash[comment_to_symbol("#{include_yamls.join(' ')}")] = nil + include_hash.merge!(load_yaml_merge(include_yamls, {})) end def multi_args(hash) diff --git a/sbin/submit b/sbin/submit index 39cdcf57..e8b17648 100755 --- a/sbin/submit +++ b/sbin/submit @@ -14,6 +14,7 @@ opt_set_key_value = {} opt_cmdline_defaults = {} opt_output_dir = nil opt_auto_define_files = false +opt_include_yamls = [] opt_monitor = false opt_monitor_query = {} opt_my_queue = false @@ -43,6 +44,10 @@ options = OptionParser.new do |opts| opt_auto_define_files = true end + opts.on('-i include.yaml', '--include include.yaml', 'include other job yamls') do |yaml| + opt_include_yamls << yaml + end + opts.on('-c', '--connect', 'auto connect to the host') do actions << 'connect' end @@ -66,6 +71,7 @@ options = OptionParser.new do |opts| end +ARGV_CLONE = ARGV.clone options.parse!(ARGV) seen_job_yaml = false @@ -91,6 +97,19 @@ end job_ids = [] job_hash_list = [] +def prioritize_include_yaml(include_yamls, jobfile) + default_yamls, override_yamls = [], [] + jobfile_index = ARGV_CLONE.index(jobfile) + include_yamls.each do |yaml| + if ARGV_CLONE.index(yaml) < jobfile_index + default_yamls << yaml + else + override_yamls << yaml + end + end + return default_yamls, override_yamls +end + def find_jobfile(jobfile) return jobfile if File.exist?(jobfile) @@ -103,14 +122,25 @@ def find_jobfile(jobfile) exit 1 end +def find_jobfiles(jobfile_list) + search_jobfile_list = [] + jobfile_list.each do |jobfile| + search_jobfile_list << find_jobfile(jobfile) + end + return search_jobfile_list +end + submit_id = %x(uuidgen).chomp puts "submit_id=#{submit_id}" ARGV.each do |jobfile| + default_yamls, override_yamls = prioritize_include_yaml(opt_include_yamls, jobfile) jobfile = find_jobfile(jobfile) jobs = Job2sh.new jobs.cmdline_defaults = opt_cmdline_defaults jobs.overrides = opt_set_key_value + jobs.default_yamls = find_jobfiles(default_yamls) + jobs.override_yamls = find_jobfiles(override_yamls) jobs.load(jobfile, true) || next jobs[:expand_params] = true jobs['testbox'] = opt_set_key_value['testbox'] if opt_set_key_value['testbox'] -- 2.23.0

1 0

[PATCH compass-ci] container/add-software: if cd failed, don't go on exec rm cmd
by Xu Xijian 21 Jan '21

21 Jan '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/add-software/run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/add-software/run b/container/add-software/run index aab659b..8a1d9ea 100755 --- a/container/add-software/run +++ b/container/add-software/run @@ -42,7 +42,7 @@ install_cmds=( addsoftware:auto bash -c " yum install -y --skip-broken --installroot=/mnt/root $packages - cd /mnt/root + cd /mnt/root && rm -rf srv/{*,.*} opt/{*,.*} mnt/{*,.*} media/{*,.*} home/{*,.*} lost+found/{*,.*} dev/{*,.*} proc/{*,.*} run/{*,.*} sys/{*,.*} > /dev/null 2>&1 " ) -- 2.23.0

2 2

[PATCH compass-ci] delimiter: use check_job_credible API to replace credible? function
by Cao Xueliang 21 Jan '21

21 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 2c6549c..41b65de 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -78,6 +78,7 @@ module Utils # monitor the job id and job state query job stats when job state is extract_finished # according to the job stats return good/bad/nil def get_job_status(job, error_id) + bad_job_id = job['bad_job_id'] new_job_id = submit_job(job) puts "new job id: #{new_job_id}" return nil unless new_job_id @@ -86,7 +87,9 @@ module Utils extract_finished = monitor_run_stop(query) return nil unless extract_finished.zero? - raise "the job is incredible for bisect: #{new_job_id}" unless credible?(job['bad_job_id'], new_job_id, error_id) + check_result = AssistResult.new.check_job_credible(bad_job_id, new_job_id, error_id) + raise "check job credible failed: #{bad_job_id}, #{new_job_id}, #{error_id}" if check_result == nil + raise "the job is incredible for bisect: #{new_job_id}" unless check_result['credible'] stats = query_stats(new_job_id, 10) raise "es cant query #{new_job_id} stats field!" unless stats -- 2.23.0

1 0

[PATCH v4 lkp-tests 4/5] new reset-tbox job yaml and script
by Zhang Dewan 21 Jan '21

21 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 2 +- tests/reset-tbox | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml index 216a3852..52ea76b7 100644 --- a/jobs/reset-tbox.yaml +++ b/jobs/reset-tbox.yaml @@ -7,4 +7,4 @@ docker_image: reset-tbox:latest queue: reset-tbox reset-tbox: - host_name: +host_name: diff --git a/tests/reset-tbox b/tests/reset-tbox index 36aa37d5..4d2c191e 100755 --- a/tests/reset-tbox +++ b/tests/reset-tbox @@ -1,5 +1,5 @@ #!/bin/bash -# - host_nameset +# host_name account_file=/etc/lab/.account.info hosts_file=/etc/lab/hosts_file -- 2.23.0

2 2