January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH v6 compass-ci 3/7] send-internet-mail: add timing work
by Luan Shengde 12 Jan '21

12 Jan '21

daily timing work: reset email rate limit queues everyday at 00:00 Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_init.rb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100755 container/send-internet-mail/email_init.rb diff --git a/container/send-internet-mail/email_init.rb b/container/send-internet-mail/email_init.rb new file mode 100755 index 0000000..d3f8323 --- /dev/null +++ b/container/send-internet-mail/email_init.rb @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'rufus-scheduler' +require 'redis' + +redis = Redis.new(host: REDIS_HOST, port: REDIS_PORT) +email_init = Rufus::Scheduler.new + +# Timing work for email counting everyday. +# the email counts queues will be reset everyday +# to ensure can send mail in the new day to user. +email_init.cron '0 0 * * *' do + redis.del 'email_in_limit' + redis.del 'email_out_limit' +end -- 2.23.0

1 0

[PATCH v6 compass-ci 2/7] send-internet-mail: optimize
by Luan Shengde 12 Jan '21

12 Jan '21

add init redis host/port add invoke code files Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb index 8fdaa0d..7b357f8 100755 --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -6,7 +6,13 @@ require 'mail' require 'sinatra' -require "#{ENV['CCI_SRC']}/lib/mail-post" +REDIS_HOST = %x(/sbin/ip route | awk '/default/ {print $3}').chomp +REDIS_PORT = ENV['REDIS_PORT'] + +require_relative 'email_init' +require_relative 'email_limit_queue' +require_relative 'email_mapping' +require_relative '../../lib/mail-post' smtp = { address: 'smtp.qq.com', -- 2.23.0

1 0

[PATCH v6 compass-ci 1/7] send-internet-mail/Dockerfile: add install gem resources
by Luan Shengde 12 Jan '21

12 Jan '21

[why] redis: used to read/store email counts from/to redis queue rufus-scheduler: used to reset the queue as a timing task Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index d151bda..03e3327 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -6,4 +6,4 @@ MAINTAINER shdluan(a)163.com ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y ruby-mail ruby-json ruby-sinatra + apt-get install -y ruby-mail ruby-json ruby-sinatra ruby-redis ruby-rufus-scheduler -- 2.23.0

1 0

[PATCH v2 compass-ci] container/auto-submit: fix submit job failed
by Liu Yinsi 12 Jan '21

12 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/container/es# docker exec -it auto-submit sh /c/lkp-tests/sbin $ ./submit iperf.yaml testbox=vm-2p8g submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_email'. Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_email'. Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… [how] for user intall compass-ci at their local server, need to verify when submit job in auto-submit container, so store autosubmit_info to ES, and mount yaml file in container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/start | 2 +- lib/build_my_info_client.rb | 4 ++-- sbin/build-my-info.rb | 4 ++++ 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/container/auto-submit/start b/container/auto-submit/start index c25fbdd..e66cf57 100755 --- a/container/auto-submit/start +++ b/container/auto-submit/start @@ -21,7 +21,7 @@ cmd = %W[ -v #{ENV['LKP_SRC']}:#{DOCKER_LKP} -v /etc/localtime:/etc/localtime:ro -v /etc/compass-ci/defaults:/etc/compass-ci/defaults:ro - -v /etc/compass-ci/account:/etc/compass-ci/account:ro + -v /etc/compass-ci/account:/home/lkp/.config/compass-ci/defaults:ro -v /srv/git:/srv/git -w /c/compass-ci/sbin alpine:auto-submit diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb index 162bf9f..d0d4b77 100755 --- a/lib/build_my_info_client.rb +++ b/lib/build_my_info_client.rb @@ -21,12 +21,12 @@ require_relative '../lib/es_client' # build_my_info = BuildMyInfo.new(email, name, lab) # build_my_info.config_my_info class BuildMyInfo - def initialize(my_email, my_name, lab) + def initialize(my_email, my_name, lab, my_token = %x(uuidgen).chomp) @lab = lab || 'nolab' @my_info = { 'my_email' => my_email, 'my_name' => my_name, - 'my_token' => %x(uuidgen).chomp, + 'my_token' => my_token, 'my_login_name' => nil, 'my_commit_url' => nil, 'my_ssh_pubkey' => [] diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index fd75f51..d68932e 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -31,3 +31,7 @@ options.parse! build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) build_my_info.config_my_info + +autosubmit_info = YAML.load_file('/etc/compass-ci/account/autosubmit.yaml') +build_autosubmit_info = BuildMyInfo.new(autosubmit_info['my_email'], autosubmit_info['my_name'], autosubmit_info['lab'], autosubmit_info['my_token']) +build_autosubmit_info.store_account_info -- 2.23.0

2 2

[PATCH v3 lkp-tests] lib/bootstrap: fix failed to get MAC and IP
by Liu Yinsi 12 Jan '21

12 Jan '21

[error] LKP: HOSTNAME vm-2p8g.2288hv5-2s44p-384g--b5-1573861, MAC , IP , kernel 5.9.0-5-riscv64 1, serial console /dev/ttyS0 [why] [ 93.815004] + network_ok [ 93.874456] ++ cat /sys/class/net/eth0//operstate [ 93.879278] + '[' up = up ']' [ 93.882356] + return 0 [ 93.941910] + export_ip_mac [ 93.956875] + cat /sys/class/net/eth0/operstate [ 93.960236] ++ route -n [ 93.968810] + route='Kernel IP routing table [ 93.988935] Destination GatewayGenmaskFlags Metric Ref Use Iface' [ 94.011625] + echo Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface [ 94.018548] + has_cmd ip [ 94.022832] + command -v ip [ 94.029742] ++ awk '{print $5; exit}' [ 94.036137] ++ ip route get 1.2.3.4 [ 94.045622] ++ /usr/bin/mawk '{print $5; exit}' [ 94.056504] RTNETLINK answers: Network is unreachable [ 94.061418] + export PUB_NIC= [ 94.064587] + PUB_NIC= export_mac_ip: PUB_NIC=$(ip route get 1.2.3.4 | awk '{print $5; exit}') PUB_IP=$( ip route get 1.2.3.4 | awk '{print $7; exit}') network_ok: [ "$(cat $i/operstate)" = 'up' ]&& return 0 when cat /sys/class/net/eth0/operstate is up, network is ok, then it will not sleep and immediately execute export_ip_mac(), but actually route is not ready, it can't get mac and ip by search route, so use ping can ensure save enough time for route. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/bootstrap.sh | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index b04ea3dd4..df25e99e0 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -87,13 +87,7 @@ net_devices_link() network_ok() { - local i - for i in /sys/class/net/*/ - do - [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 - done + ping -c 3 $LKP_SERVER > /dev/null && return 0 is_clearlinux && { net_devices_link up -- 2.23.0

3 4

[PATCH compass-ci] sparrow/3-code: add RESULT_WEBDAV_PORT field
by Xiao Shenwei 12 Jan '21

12 Jan '21

[why] $LKP_SRC/lib/upload.sh will upload some result file by result-webdav container [how] display definition in default profile Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 1b311f0..f2366c2 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -19,6 +19,7 @@ server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} : ${ASSIST_RESULT_PORT:=8102} +: ${RESULT_WEBDAV_PORT:=3080} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF @@ -43,6 +44,7 @@ LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS MAILDIR: /srv/cci/Maildir/.compass-ci ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT +RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT EOF mkdir -p /etc/compass-ci/account -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: set the upload path of package based on os attribute of top-level package style
by Cao Xueliang 12 Jan '21

12 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 218e2e1..1c6eb1a 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -249,10 +249,37 @@ class Job set_upload_dirs() end + def get_pkg_common_dir + common_dir = "" + + if @hash["cci-makepkg"]? + pkg_style = "cci-makepkg" + elsif @hash["cci-depends"]? + pkg_style = "cci-depends" + elsif @hash["build-pkg"]? + pkg_style = "build-pkg" + else + pkg_style = nil + end + + if pkg_style + tmp = @hash["#{pkg_style}"] + tmp_os = tmp["os"]? ? tmp["os"].to_s : "#{os}" + tmp_os_arch = tmp["os_arch"]? ? tmp["os_arch"].to_s : "#{os_arch}" + tmp_os_version = tmp["os_version"]? ? tmp["os_version"].to_s : "#{os_version}" + + tmp_os_mount = tmp["os_mount"]? ? tmp["os_mount"].to_s : "#{os_mount}" + mount_type = tmp_os_mount == "cifs" ? "nfs" : tmp_os_mount.dup + + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + end + + return common_dir + end + def get_package_dir if @hash["cci-makepkg"]? || @hash["cci-depends"]? || @hash["build-pkg"]? - mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup - common_dir = "#{mount_type}/#{os}/#{os_arch}/#{os_version}" + common_dir = get_pkg_common_dir end if @hash["cci-makepkg"]? -- 2.23.0

2 2

[PATCH lkp-tests] reset-tbox to reboot machine testbox
by Zhang Dewan 11 Jan '21

11 Jan '21

[why] sumbit a job to reboot machine testbox is better. [how] cp ~/lkp-test/jobs/reset-tbox-yaml . submit reset-tbox.yaml host_name="full-name-of-machine-testbox" Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 10 ++++++++++ tests/reset-tbox | 17 +++++++++++++++++ tests/reset-tbox.yaml | 12 ++++++++++++ 3 files changed, 39 insertions(+) create mode 100644 jobs/reset-tbox.yaml create mode 100755 tests/reset-tbox create mode 100644 tests/reset-tbox.yaml diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml new file mode 100644 index 00000000..9c3f4f01 --- /dev/null +++ b/jobs/reset-tbox.yaml @@ -0,0 +1,10 @@ +suite: reset-tbox +testcase: reset-tbox + +testbox: dc-8g +os_mount: container +docker_image: 172.168.131.2:5001/reset-tbox +queue: reset + +host_name: +reset-tbox: diff --git a/tests/reset-tbox b/tests/reset-tbox new file mode 100755 index 00000000..b0f645d7 --- /dev/null +++ b/tests/reset-tbox @@ -0,0 +1,17 @@ +#!/bin/bash +# host_name + +source "/etc/lab/.account.info" +hosts_file=/etc/lab/hosts_file + +# require $host_name +host_ibmc_ip=$(grep -w $host_name $hosts_file | awk '{print $1}') + +if [[ "$host_ibmc_ip" =~ ^[0-9]+.* ]]; then + echo ">>> exec ipmitool commamd!" + ipmitool -I lanplus -H $host_ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset + [ $? -eq 0 ] && echo "succeed!" +else + echo "specify host cannot found0!" + exit +fi diff --git a/tests/reset-tbox.yaml b/tests/reset-tbox.yaml new file mode 100644 index 00000000..f557085f --- /dev/null +++ b/tests/reset-tbox.yaml @@ -0,0 +1,12 @@ +short_description: > + To reboot machine testbox like taishan200-2280-2s64p-256g--aN etc. + +description: | + reset-tbox executed by a special docker which running on ibmc-mgnt, where you can access all server's iBMC-os by ipmitool, and then reboot the host. + +homepage: + +parameters: + host_name: see it on the first line about 'latbox aN' + +results: -- 2.23.0

3 4

[PATCH v4 compass-ci 3/4] scheduler: fix response of 'boot.ipxe'
by Ren Wen 11 Jan '21

11 Jan '21

Fix this to adapt the changes to job fields: - 'kernel_uri' - 'kernel_params' - 'initrds_uri' Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b2f0802..5e6044a 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -158,9 +158,13 @@ class Sched private def get_boot_ipxe(job : Job) response = "#!ipxe\n\n" - response += Array(String).from_json(job.initrds_uri).join("\n") + "\n" - response += job.kernel_uri + " " - response += Array(String).from_json(job.kernel_params).join(" ") + + _initrds_uri = Array(String).from_json(job.initrds_uri).map { |uri| "initrd #{uri}" } + response += _initrds_uri.join("\n") + "\n" + + _kernel_params = ["kernel #{job.kernel_uri}"] + Array(String).from_json(job.kernel_params) + Array(String).from_json(job.ipxe_kernel_params) + response += _kernel_params.join(" ") + response += "\nboot\n" return response -- 2.23.0

1 1

[PATCH v4 compass-ci 2/4] scheduler: remove prefix of 'kernel_uri' and 'initrds_uri'
by Ren Wen 11 Jan '21

11 Jan '21

Prefix 'kernel ' and 'initrd ' of 'kernel_uri' and 'initrds_uri' are just used by 'boot.ipxe'. Need to make this independent to adapt other boot types. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 260b1ea..59a4c57 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -433,8 +433,7 @@ class Job end private def set_kernel_uri - self["kernel_uri"] = "kernel #{OS_HTTP_PREFIX}" + - JobHelper.service_path("#{linux_vmlinuz_path}") + self["kernel_uri"] = "#{OS_HTTP_PREFIX}" + JobHelper.service_path("#{linux_vmlinuz_path}") end private def common_initrds @@ -497,7 +496,7 @@ class Job initrds_uri_values = [] of JSON::Any get_initrds().each do |initrd| - initrds_uri_values << JSON::Any.new("initrd #{initrd}") + initrds_uri_values << JSON::Any.new("#{initrd}") end @hash["initrds_uri"] = JSON::Any.new(initrds_uri_values) -- 2.23.0

1 1