- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] bin/lkp-setup-rootfs: add goproxy
by Cui Yili 20 Oct '20

20 Oct '20

Solve go download timeout. Signed-off-by: cuiyili <2268260388(a)qq.com> --- bin/lkp-setup-rootfs | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index 15002feb..4aacb970 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -157,12 +157,22 @@ modify_gem_source() return 0 } +install_goproxy() +{ + mkdir -p /root/.config/go/env + echo "GOPROXY=https://goproxy.io,direct" >> /root/.config/go/env + + return 0 +} + install_proxy() { modify_gem_source install_docker_proxy + install_goproxy + [ -z "$SQUID_HOST" ] && return [ -z "$SQUID_PORT" ] && return -- 2.23.0

3 2

[PATCH compass-ci] git_mirror: repo updates frequently will be updated first
by Li Yuanchao 20 Oct '20

20 Oct '20

PriorityQueue will pop whose priority is minimum first. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index e80b7ca..593d1eb 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -155,6 +155,7 @@ class MirrorMain @fork_stat[git_repo][:queued] = false return unless feedback_info[:possible_new_refs] + @fork_stat[git_repo][:priority] += 1 return reload_fork_info if git_repo == 'upstream-repos/upstream-repos' new_refs = check_new_refs(git_repo) @@ -164,16 +165,18 @@ class MirrorMain send_message(feedback_info) end + def do_push(fork_key) + @fork_stat[fork_key][:queued] = true + @git_info[fork_key][:cur_refs] = get_cur_refs(fork_key) if @git_info[fork_key][:cur_refs].nil? + @git_queue.push(@git_info[fork_key]) + end + def push_git_queue return if @git_queue.size >= 1 fork_key = @priority_queue.delete_min_return_key - unless @fork_stat[fork_key][:queued] - @fork_stat[fork_key][:queued] = true - @git_info[fork_key][:cur_refs] = get_cur_refs(fork_key) if @git_info[fork_key][:cur_refs].nil? - @git_queue.push(@git_info[fork_key]) - end - @priority_queue.push fork_key, @priority + do_push(fork_key) unless @fork_stat[fork_key][:queued] + @priority_queue.push fork_key, (@priority - @fork_stat[fork_key][:priority]) @priority += 1 end -- 2.23.0

1 0

[PATCH compass-ci] lib: fix ameba warning
by Bai Jing 20 Oct '20

20 Oct '20

error: Style/UnlessElse: Favour if over unless with else Signed-off-by: Bai Jing <799286817(a)qq.com> --- src/lib/sched.cr | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a4ddcec..5076366 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -267,10 +267,10 @@ class Sched }] unless tbox_group # only single job will has "idle job" and "execute rate limiter" - unless job["idle_job"].empty? - tbox_group = "#{tbox_group}/idle" - else + if job["idle_job"].empty? tbox_group += "#{job.get_uuid_tag}" + else + tbox_group = "#{tbox_group}/idle" end job_id = add_task(tbox_group, job.lab) -- 2.23.0

1 0

[PATCH compass-ci 1/3] user-client/maintain: submit test rootfs job in z9 tbox
by Liu Yinsi 20 Oct '20

20 Oct '20

1. submit job in z9 tbox, remove my-qemu.sh. 2. submit 'key=value' instead of -s. 3. remove job yaml redundant field. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- .../maintain/walk-os-test/iperf-walk-os.yaml | 4 - .../maintain/walk-os-test/walk-os-iperf-test | 119 +++++++++--------- 2 files changed, 58 insertions(+), 65 deletions(-) diff --git a/user-client/maintain/walk-os-test/iperf-walk-os.yaml b/user-client/maintain/walk-os-test/iperf-walk-os.yaml index 647220c..237d006 100644 --- a/user-client/maintain/walk-os-test/iperf-walk-os.yaml +++ b/user-client/maintain/walk-os-test/iperf-walk-os.yaml @@ -7,9 +7,5 @@ if role server: if role client: iperf: protocol: tcp -testbox: vm-2p8g--USER -os: OS -os_arch: OS_ARCH -os_version: OS_VER node_roles: server client LKP_DEBUG_PREFIX: bash -x diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index 21e1428..9910723 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -1,62 +1,59 @@ -#!/bin/bash -e +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ - -# os test rounds -test_times=1 - -test_logs=walk-test.report -test_yaml=iperf-walk-os.yaml -test_os=( - 'openeuler aarch64 20' - 'openeuler aarch64 20.03' - 'centos aarch64 7.6' - 'centos aarch64 7.8' - 'centos aarch64 8.1.1911' - 'debian aarch64 sid' -) - -trap 'echo User Cancelled! && exit' SIGINT - -sec2date() { - date -d@$1 +"%Y%m%d_%H:%M:%S" -} - -logging() { - if [ 'new' == "$1" ] ; then - printf "%-5s %-8s %-25s %-20s %-20s %-8s %-10s\n" ${@:2} > $test_logs - else - printf "%-5s %-8s %-25s %-20s %-20s %-8s %-10s\n" $@ >> $test_logs - fi -} - -create_test_yaml() { - local hwarch os os_arch os_ver cur_os - hwarch=$(arch) - cur_os="$@" - os=$(echo $cur_os |cut -d' ' -f1) - os_arch=$(echo $cur_os |cut -d' ' -f2) - os_ver=$(echo $cur_os |cut -d' ' -f3) - sed "s/USER/$USER/" $test_yaml > test.yaml - sed -i "s/OS_ARCH/$os_arch/" test.yaml - sed -i "s/OS_VER/$os_ver/" test.yaml - sed -i "s/OS/$os/" test.yaml - echo "$hwarch $os/$os_arch/$os_ver" -} - -job_test() { - local job_id bef aft - job_id=$(submit test.yaml |awk '{print $NF}') - bef=$(date +%s) - ./my-qemu.sh - aft=$(date +%s) - logging "$1" "$2" "$3" "$(sec2date $bef)" "$(sec2date $aft)" "$((aft - bef))" "$job_id" -} - -logging new "Index" "hwArch" "OS" "Begin" "End" "Cost/sec" "Job_ID" -for round in $(seq $test_times) -do - for idx in "${!test_os[@]}" - do - job_test $round $(create_test_yaml "${test_os[$idx]}") - done -done +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +# rootfs test via job, including cifs, initramfs files. +# boot way including PXE and kvm. +# input: job yaml, os, os_version, os_arch, os_mount, testbox +# output: a test report, including job ID, os, runtime, job state and url to check logs, +# and send mail. +# +# Eg: test report format: +# +# Job_ID OS Begin +# crystal.104748 openeuler/aarch64/20.03/initramfs 2020-10-19/18:42:41 +# +# End Cost/sec Job_State Result_Url +# 2020-10-19/18:44:48 127 finished #{SRV_HTTP_HOST}:#{SRV_HTTP_PORT}/#{result_root} + +TEST_OS = [ + 'openeuler aarch64 20.03', + 'centos aarch64 7.6.1810', + 'centos aarch64 7.8.2003', + 'centos aarch64 8.1.1911', + 'debian aarch64 sid' +].freeze + +MOUNT_TYPE = %w[cifs initramfs].freeze +TESTBOX = %w[vm-2p8g vm-pxe-2p8g].freeze +TEST_REPORT = 'walk-test.report' +TEST_YAML = 'iperf-walk-os.yaml' + +def submit_job(os_array, testbox) + os = os_array[0] + os_arch = os_array[1] + os_ver = os_array[2] + os_mount = os_array[3] + message = %(submit 'testbox=#{testbox}' 'os=#{os}' 'os_arch=#{os_arch}' 'os_version=#{os_ver}' 'os_mount=#{os_mount}' #{TEST_YAML}) + return if message.include? 'error' + + # if job not running on tbox, add this the task will be consumed. + # Process.fork do + # %x(./my-qemu.sh >/dev/null 2>&1) + # end +end + +def get_os_array(os_mount, testbox) + TEST_OS.each do |item| + os_array = item.split.push(os_mount) + + submit_job(os_array, testbox) + end +end + +def get_testbox(os_mount) + TESTBOX.each { |testbox| get_os_array(os_mount, testbox) } +end + +MOUNT_TYPE.each { |os_mount| get_testbox(os_mount) } -- 2.23.0

1 0

[PATCH v3 compass-ci 1/5] container/send-internet-mail: mail-post.rb
by Luan Shengde 20 Oct '20

20 Oct '20

get request for send internet mail analysis email data invoke send-internet-mail to send the mail Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- lib/mail-post.rb | 50 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100755 lib/mail-post.rb diff --git a/lib/mail-post.rb b/lib/mail-post.rb new file mode 100755 index 0000000..ea8bf56 --- /dev/null +++ b/lib/mail-post.rb @@ -0,0 +1,50 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'json' +require 'yaml' +require 'open3' +require_relative "#{ENV['CCI_SRC']}/container/send-internet-mail/send-internet-mail.rb" + +set :bind, '0.0.0.0' +set :port, 11312 + +post '/send_mail_yaml' do + data = YAML.safe_load request.body.read + raise TypeError, data, 'request data type error' unless data.class.eql? Hash + + mail_info = { + 'references' => data['references'], + 'from' => data['from'], + 'subject' => data['subject'] || nil, + 'to' => data['to'] || nil, + 'body' => data['body'] || nil + } + check_send_mail(mail_info) +end + +post '/send_mail_text' do + data = Mail.read_from_string(request.body.read) + + mail_info = { + 'references' => data.references, + 'from' => data.from, + 'subject' => data.subject || nil, + 'to' => data.to || nil, + 'body' => data.body.decoded || nil + } + check_send_mail(mail_info) +end + +def check_send_mail(mail_info) + raise 'empty subject.' if mail_info['subject'].nil? + raise 'empty email address.' if mail_info['to'].nil? + raise 'empty email content.' if mail_info['body'].nil? + + puts ENV['EMAIL_ADDRESS'] + puts ENV['EMAIL_PASSWORD'] + send_mail(mail_info) +end -- 2.23.0

3 6

[PATCH v3 compass-ci 5/5] container/send-internet-mail: start file
by Luan Shengde 20 Oct '20

20 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-internet-mail/start | 72 ++++++++++++++++++++++++++++++ 1 file changed, 72 insertions(+) create mode 100755 container/send-internet-mail/start diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start new file mode 100755 index 0000000..e4d641d --- /dev/null +++ b/container/send-internet-mail/start @@ -0,0 +1,72 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig' + +docker_rm 'send-internet-mail' + +names = Set.new %w[ + EMAIL_PASSWORD + EMAIL_ADDRESS +] + +defaults = relevant_defaults(names) + +EMAIL_ADDRESS = defaults['EMAIL_ADDRESS'] +EMAIL_PASSWORD = defaults['EMAIL_PASSWORD'] + +cmd = %W[ + docker run + --restart=always + --name=send-internet-mail + -d + -e EMAIL_ADDRESS=#{EMAIL_ADDRESS} + -e EMAIL_PASSWORD=#{EMAIL_PASSWORD} + -e CCI_SRC=#{ENV['CCI_SRC']} + -p 11312:11312 + -v #{ENV['CCI_SRC']}:#{ENV['CCI_SRC']} + -v /etc/localtime:/etc/localtime:ro + --log-driver json-file + send-internet-mail +] + +cmd += ['bash', '-c', "#{ENV['CCI_SRC']}/lib/mail-post.rb"] + +system(*cmd) + + +puts <<EOF +port: +--- +data='{ +"subject": "email subject", +"to": "email_to_addr", +"body": "email message" +}' + +or + +data=" +subject: email subject +to: email_to_addr +body: email message" + +Usage: + curl -XPOST 'send-mail-server:11312/send_mail_yaml' -d "$data" + +--- +data=" +To: email_to_addr +Subject: email_subject + +mail_msg_line1 +mail_msg_line2 +... +" + +Usage: + curl -XPOST 'send-mail-server:11312/send_mail_text' -d "$data" +EOF -- 2.23.0

2 1

[PATCH compass-ci] git_mirror: feedback_info[:git_repo] => git_repo
by Li Yuanchao 20 Oct '20

20 Oct '20

simplify the code Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index fef4aca..e80b7ca 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -151,12 +151,13 @@ class MirrorMain return if @feedback_queue.empty? feedback_info = @feedback_queue.pop(true) - @fork_stat[feedback_info[:git_repo]][:queued] = false + git_repo = feedback_info[:git_repo] + @fork_stat[git_repo][:queued] = false return unless feedback_info[:possible_new_refs] - return reload_fork_info if feedback_info[:git_repo] == 'upstream-repos/upstream-repos' + return reload_fork_info if git_repo == 'upstream-repos/upstream-repos' - new_refs = check_new_refs(feedback_info[:git_repo]) + new_refs = check_new_refs(git_repo) return if new_refs[:heads].empty? feedback_info[:new_refs] = new_refs -- 2.23.0

1 0

[PATCH compass-ci] container/fluentd: use workers to improve concurrency
by Wu Zhende 20 Oct '20

20 Oct '20

[Why] Currently,only one worker is used to process all logs. As a result,logs can't be processed when there are a large number of concurrent logs. So use 10 workers. But tail plugin don't support multi-process. Use worker 0 for tail plugin. Use worker 1-9 for forward plugin. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/fluentd/docker-fluentd.conf | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/container/fluentd/docker-fluentd.conf b/container/fluentd/docker-fluentd.conf index ac73dfb..23c7585 100644 --- a/container/fluentd/docker-fluentd.conf +++ b/container/fluentd/docker-fluentd.conf @@ -1,8 +1,8 @@ -<source> - @type forward - bind 0.0.0.0 -</source> +<system> + workers 10 +</system> +<worker 0> <source> @type tail path /srv/cci/serial/logs/* @@ -37,6 +37,21 @@ </format> </store> </match> +</worker> + +<worker 1-9> +<source> + @type forward + bind 0.0.0.0 +</source> + +<filter **> + @type record_transformer + enable_ruby + <record> + time ${time.strftime('%Y-%m-%d %H:%M:%S%z')} + </record> +</filter> <filter **> @type parser @@ -74,6 +89,7 @@ </server> </store> </match> +</worker> <label @FLUENT_LOG> <match fluent.*> -- 2.23.0

1 0

[PATCH compass-ci] container/monitoring: fix Missing ENV key: "LKP_SRC"
by Cao Xueliang 20 Oct '20

20 Oct '20

[why] The monitor container require job.cr, and job.cr require jobfile_operate.cr which need to get the "LKP_SRC" from ENV. So, need add env variable "LKP_SRC" and mount lkp src to the monitor container. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/monitoring/start | 3 +++ 1 file changed, 3 insertions(+) diff --git a/container/monitoring/start b/container/monitoring/start index 199f091..2e2946c 100755 --- a/container/monitoring/start +++ b/container/monitoring/start @@ -16,6 +16,7 @@ defaults = relevant_defaults(names) env = docker_env(defaults) DEFAULT_CCI = '/c/cci' +DEFAULT_LKP = '/c/lkp-tests' MONITOR_PORT = defaults['MONITOR_PORT'] || '11310' docker_rm "monitoring-#{MONITOR_PORT}" @@ -28,7 +29,9 @@ cmd = %W[ -p #{MONITOR_PORT}:#{MONITOR_PORT} ] + env + %W[ -e CCI_SRC=#{DEFAULT_CCI} + -e LKP_SRC=#{DEFAULT_LKP} -v /etc/localtime:/etc/localtime:ro + -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v /srv/result:/srv/result monitoring-#{MONITOR_PORT} ] -- 2.23.0

1 0

[PATCH v4 compass-ci] service/scheduler: add sshr port to job content
by Wu Zhende 20 Oct '20

20 Oct '20

[Why] Network users need to use sshr to login testbox. So testbox should set sshr. And testbox get sshr info from job content. Scheduler needs to add sshr info to job content Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler/start | 6 ++++++ src/lib/job.cr | 9 +++++++++ 2 files changed, 15 insertions(+) diff --git a/container/scheduler/start b/container/scheduler/start index 81e7139..305916e 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -24,10 +24,16 @@ names = Set.new %w[ GIT_SERVER FLUENTD_SERVER_HOST FLUENTD_SERVER_PORT + SSHR_PORT + SSHR_PORT_BASE + SSHR_PORT_LEN lab ] defaults = relevant_defaults(names) +defaults['SSHR_PORT'] ||= 5050 +defaults['SSHR_PORT_BASE'] ||= 50000 +defaults['SSHR_PORT_LEN'] ||= 2000 env = docker_env(defaults) CCI_REPOS = ENV['CCI_REPOS'] || '/c' diff --git a/src/lib/job.cr b/src/lib/job.cr index 15dc03e..12d3980 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -139,6 +139,7 @@ class Job set_kernel_append_root() set_kernel_params() set_lkp_server() + set_sshr_port() end private def append_init_field @@ -168,6 +169,14 @@ class Job end end + private def set_sshr_port + return unless self["sshd"]? + + self["sshr_port"] = ENV["SSHR_PORT"] + self["sshr_port_base"] = ENV["SSHR_PORT_BASE"] + self["sshr_port_len"] = ENV["SSHR_PORT_LEN"] + end + private def set_os_dir self["os_dir"] = "#{os}/#{os_arch}/#{os_version}" end -- 2.23.0

1 0