- Compass-ci - mailweb.openeuler.org

[PATCH v8 compass-ci 1/5] send-internet-mail: optimize
by Luan Shengde 14 Jan '21

14 Jan '21

add init redis host/port Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb index 8fdaa0d..6003028 100755 --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -6,7 +6,10 @@ require 'mail' require 'sinatra' -require "#{ENV['CCI_SRC']}/lib/mail-post" +REDIS_HOST = %x(/sbin/ip route | awk '/default/ {print $3}').chomp +REDIS_PORT = ENV['REDIS_PORT'] + +require_relative '../../lib/mail-post' smtp = { address: 'smtp.qq.com', -- 2.23.0

2 4

[PATCH lkp-tests] daemon/sshd: refactor sshd
by Wei Jihui 14 Jan '21

14 Jan '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- daemon/sshd | 61 +++++++----------------------------------------- lib/email.sh | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+), 53 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index e9f38006..7ba471a8 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -10,6 +10,7 @@ PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/http.sh . $PROJECT_DIR/lib/env.sh +. $PROJECT_DIR/lib/email.sh sshr_ip=$LKP_SERVER [ -n "$sshr_port" ] || sshr_port=5050 @@ -36,56 +37,6 @@ run_ssh() fi } -data_success() -{ - data="To: $my_email -Subject: [NOTIFY Compass-CI] Machine application successful - -Dear $my_username, - - Your test machine has been provisioned. - - Login: - ssh root(a)api.compass-ci.openeuler.org -p $port - Due time: - $deadline - HW: - nr_cpu: $nr_cpu - memory: $memory - testbox: $testbox - OS: - $os $os_version $os_arch - -Regards -Compass-CI -" -} - -data_failure() -{ - data="To: $my_email -Subject: [NOTIFY Compass-ci] Machine application failed - -Dear $my_username: - - Sorry to inform you that your application failed. - You may need to wait a moment, or check whether your pub_key exists. - -Regards -Compass-Ci -" -} - -compose_email() -{ - deadline=$(date -d "$runtime seconds" +"%Y-%m-%d %H:%M:%S") - if [ -n "$port" ]; then - data_success - else - data_failure - fi -} - set_port() { [ -n "$my_ssh_pubkey" ] || return @@ -103,9 +54,13 @@ set_port() run_email() { [ -n "$my_email" ] || return 0 - my_username=$(echo "$my_email" | awk -F '@' '{print $1}') - compose_email - curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$data" + + deadline=$(date -d "$runtime seconds" +"%Y-%m-%d %H:%M:%S") + if [ -n "$port" ]; then + send_email "borrow_success" + else + send_email "borrow_failed" + fi } run_ssh diff --git a/lib/email.sh b/lib/email.sh index d7c2c44a..ed75e0ce 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -15,6 +15,72 @@ Compass-CI " } +job_debug_content() +{ + email_content="To: $my_email +Subject: [NOTIFY Compass-CI] Test job debug: $id + +Dear $my_name, + + Your test job is ready to debug and test machine has been provisioned. + + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch + +Regards +Compass-CI +" +} + + +borrow_success_content() +{ + email_content="To: $my_email +Subject: [NOTIFY Compass-CI] Machine application successful: $id + +Dear $my_name, + + Your test machine has been provisioned. + + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch + +Regards +Compass-CI +" +} + +borrow_failed_content() +{ + email_content="To: $my_email +Subject: [NOTIFY Compass-CI] Machine application failed: $id + +Dear $my_name: + + Sorry to inform you that your application failed. + You may need to wait a moment, or check whether your pub_key exists. + +Regards +Compass-CI +" +} + send_email() { local subject=$1 -- 2.23.0

1 0

[PATCH v4 compass-ci] sparrow/4-docker: fix logical error
by Liu Yinsi 14 Jan '21

14 Jan '21

before: docker images exist, then build images after: docker images not exist, then build images [error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/sparrow/4-docker# ./buildall 2021-01-14 03:32:28 INFO -- start build /c/compass-ci/container/archlinux/. Unable to find image 'archlinux:testbed' locally docker: Error response from daemon: manifest for archlinux:testbed not found. See 'docker run --help'. [why] when docker images not exist, will not execute ./build, so failed to build images. [how] when docker images not exist, then execute ./build. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index f24226c..77fe43a 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -33,7 +33,7 @@ do_one() cd "$container" [ "$container_name" == 'ssh-r' ] && exit docker images | grep -wq "$container_name" - if [ "$?" == 0 ] && [ "$action" != "reboot" ]; then + if [ "$?" -ne 0 ] || [ "$action" != "reboot" ]; then [ -x build ] && ./build [ -x install ] && ./install fi -- 2.23.0

1 1

[PATCH v6 lkp-tests] lib/bootstrap: fix failed to get MAC and IP
by Liu Yinsi 14 Jan '21

14 Jan '21

before: network interfaces "up"|"1" -> network is ok after: network interfaces "up"|"1" && export_ip_mac ok -> network is ok [error] LKP: HOSTNAME vm-2p8g.2288hv5-2s44p-384g--b5-1573861, MAC , IP , kernel 5.9.0-5-riscv64 1, serial console /dev/ttyS0 [why] [ 93.815004] + network_ok [ 93.874456] ++ cat /sys/class/net/eth0//operstate [ 93.879278] + '[' up = up ']' [ 93.882356] + return 0 [ 93.941910] + export_ip_mac [ 94.018548] + has_cmd ip [ 94.022832] + command -v ip [ 94.029742] ++ awk '{print $5; exit}' [ 94.036137] ++ ip route get 1.2.3.4 [ 94.045622] ++ /usr/bin/mawk '{print $5; exit}' [ 94.056504] RTNETLINK answers: Network is unreachable [ 94.061418] + export PUB_NIC= [ 94.064587] + PUB_NIC= even network interfaces ready, but routing not, get ip by routing failed, so add a check export_ip_mac to ensure network is ok. test: /srv/result/host-info/2021-01-13/vm-2p8g/debian-sid-aarch64/z9.351368 /srv/result/host-info/2021-01-13/vm-2p8g/debian-sid-riscv64/z9.351388 /srv/result/host-info/2021-01-13/vm-2p8g/debian-sid-x86_64/z9.351369 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/bootstrap.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index b04ea3dd4..2969544a6 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -91,8 +91,8 @@ network_ok() for i in /sys/class/net/*/ do [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 + [ "$(cat $i/operstate)" = 'up' ] && export_ip_mac && return 0 + [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && export_ip_mac && return 0 done is_clearlinux && { @@ -296,6 +296,8 @@ export_ip_mac() export PUB_IP=$(ifconfig $PUB_NIC | awk '/inet / {print $2}') fi + [ -n "$PUB_NIC" ] || return 1 + export PUB_MAC=$(cat /sys/class/net/$PUB_NIC/address) [ -n "$SCHED_HOST" ] && PUB_MAC=$(echo "$PUB_MAC" | tr : -) } @@ -746,7 +748,6 @@ boot_init() setup_network run_ntpdate - export_ip_mac announce_bootup -- 2.23.0

2 2

[PATCH compass-ci 2/2] src/delimiter: delete unused requires
by Cao Xueliang 14 Jan '21

14 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/delimiter.rb | 2 -- 1 file changed, 2 deletions(-) diff --git a/src/delimiter/delimiter.rb b/src/delimiter/delimiter.rb index 1afffbc..a66f4fb 100644 --- a/src/delimiter/delimiter.rb +++ b/src/delimiter/delimiter.rb @@ -5,9 +5,7 @@ require 'json' require_relative './constants' -require_relative '../../lib/git_bisect' require_relative '../../lib/taskqueue_client' -require_relative '../../lib/mail_bisect_result' # consume assister task queue class Delimiter -- 2.23.0

1 0

[PATCH compass-ci 1/2] container/delimiter: no need mount "/srv/git" and delete some ENV vars
by Cao Xueliang 14 Jan '21

14 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/start | 8 -------- 1 file changed, 8 deletions(-) diff --git a/container/delimiter/start b/container/delimiter/start index cb44d84..669923b 100755 --- a/container/delimiter/start +++ b/container/delimiter/start @@ -7,15 +7,8 @@ require 'set' require_relative '../defconfig.rb' names = Set.new %w[ - ES_HOST - ES_PORT - SEND_MAIL_HOST - SEND_MAIL_PORT - MONITOR_HOST - MONITOR_PORT TASKQUEUE_HOST TASKQUEUE_PORT - GIT_MIRROR_HOST ] defaults = relevant_defaults(names) @@ -38,7 +31,6 @@ cmd = %w[ -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro -v /etc/localtime:/etc/localtime:ro - -v /srv/git:/srv/git:ro -w #{DEFAULT_CCI}/src delimiter ] -- 2.23.0

1 0

[PATCH v8 compass-ci 5/5] send-internet-mail: email rate limit
by Luan Shengde 14 Jan '21

14 Jan '21

add email_in_limit to store email address that send count less than limit value. add email_out_limit to store email address that send count beyond limit value. the email queues email_in_limit/email_out_limit will be reset everyday at 00:00, every time send mail to an email address, the value for the mail address in email_in_limit will +1, until the value up to the limited value, the email address will be moved to email_out_limit, and the email send to the address will be throw away. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post/email_limit_queue.rb | 62 ++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100755 lib/mail-post/email_limit_queue.rb diff --git a/lib/mail-post/email_limit_queue.rb b/lib/mail-post/email_limit_queue.rb new file mode 100755 index 0000000..c7e3954 --- /dev/null +++ b/lib/mail-post/email_limit_queue.rb @@ -0,0 +1,62 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# check if the emails count has beyond the limit. +# email_in_limit is used to store email address email count +# less than the limit value. +# email_out_limit is used to store email address email count +# beyond the limit. +# case the email is the first one for the day, the email will +# be add to email_in_limit with value 1. +# case the email in email_in_limit but email count less than the +# limit value, its value will +1. +# case the email's email count up to the limit value, the email +# will be moved to email_out_limit. +# when the email is in email_out_limit, if send mail to the email +# address, the email will be kicked out from the email address bar. +class EmailRateLimit + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_counts + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + @mail_info['to'] = check_emails(email_to) + @mail_info['cc'] = check_emails(email_cc) + + return @mail_info + end + + def check_emails(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + if @redis.hexists 'email_out_limit', email + mail_list -= [email] + next + elsif @redis.hexists 'email_in_limit', email + email_account = @redis.hget 'email_in_limit', email + @redis.hset 'email_in_limit', email, email_account.to_i + 1 + else + @redis.hset 'email_in_limit', email, 1 + end + + change_queue(email) + end + return mail_list + end + + def change_queue(email) + return unless (@redis.hget 'email_in_limit', email).to_i >= 10 + + @redis.hdel 'email_in_limit', email + @redis.hset 'email_out_limit', email, 10 + end +end -- 2.23.0

1 0

[PATCH v8 compass-ci 4/5] send-internet-mail: add email mapping
by Luan Shengde 14 Jan '21

14 Jan '21

example: name => email_addr tag => email_addr email_addr_r => email_addr_d an email address can be set multi mappings, you can set a name or email to a specified email address. when you write either a name or an email address in the send address bar, the email address will be converted to the mapped email address. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post/email_mapping.rb | 50 ++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100755 lib/mail-post/email_mapping.rb diff --git a/lib/mail-post/email_mapping.rb b/lib/mail-post/email_mapping.rb new file mode 100755 index 0000000..02bfc6b --- /dev/null +++ b/lib/mail-post/email_mapping.rb @@ -0,0 +1,50 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# email address translation according to the email_mapping queue. +# email_mapping is used to set email mapping, example: +# email_mapping: +# name => email_address_d +# tag => email_address_d +# email_address_r => email_address_d +# ... +# +# can set multi keys for an email address, the key can be a name, +# an email, or some something else like a tag. +# case add the key is added to email address bar, the key will be +# transferred to the mapped email address. +class EmailAddrMapping + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_mapping + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + email_bcc = @mail_info['bcc'].clone + @mail_info['to'] = email_mapping(email_to) + @mail_info['cc'] = email_mapping(email_cc) + @mail_info['bcc'] = email_mapping(email_bcc) + + return @mail_info + end + + def email_mapping(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + next unless @redis.hexists 'email_mapping', email + + mapped_email = @redis.hget 'email_mapping', email + mail_list -= [email] + mail_list << mapped_email unless mail_list.include? mapped_email + end + return mail_list + end +end -- 2.23.0

1 0

[PATCH v8 compass-ci 3/5] send-internet-mail: add timing work
by Luan Shengde 14 Jan '21

14 Jan '21

daily timing work: reset email rate limit queues everyday at 00:00 Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post/email_init.rb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100755 lib/mail-post/email_init.rb diff --git a/lib/mail-post/email_init.rb b/lib/mail-post/email_init.rb new file mode 100755 index 0000000..d3f8323 --- /dev/null +++ b/lib/mail-post/email_init.rb @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'rufus-scheduler' +require 'redis' + +redis = Redis.new(host: REDIS_HOST, port: REDIS_PORT) +email_init = Rufus::Scheduler.new + +# Timing work for email counting everyday. +# the email counts queues will be reset everyday +# to ensure can send mail in the new day to user. +email_init.cron '0 0 * * *' do + redis.del 'email_in_limit' + redis.del 'email_out_limit' +end -- 2.23.0

1 0

[PATCH v8 compass-ci 2/5] lib/mail-post: add email counts/mapping checks
by Luan Shengde 14 Jan '21

14 Jan '21

add check email count: only email addresses that email count within the limited value can be sent emails add check email mapping: the address will be converted to the mapping value if it has been set an email mapping value Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) mode change 100644 => 100755 lib/mail-post.rb diff --git a/lib/mail-post.rb b/lib/mail-post.rb old mode 100644 new mode 100755 index 9d3e601..4eb5800 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -10,6 +10,10 @@ require 'open3' require 'mail' require 'base64' +require_relative 'mail-post/email_init' +require_relative 'mail-post/email_limit_queue' +require_relative 'mail-post/email_mapping' + set :bind, '0.0.0.0' set :port, ENV['SEND_MAIL_PORT'] @@ -57,14 +61,32 @@ post '/send_mail_encode' do send_mail(mail_info) end +def check_email_limit(mail_info) + email_limit = EmailRateLimit.new(mail_info) + email_limit.check_email_counts +end + +def check_email_mapping(mail_info) + email_mapping = EmailAddrMapping.new(mail_info) + email_mapping.check_email_mapping +end + def check_send_mail(mail_info) raise 'no/empty subject.' if mail_info['subject'].nil? || mail_info['subject'].empty? raise 'no/empty email_to address.' if mail_info['to'].nil? || mail_info['to'].empty? raise 'no/empty email content.' if mail_info['body'].nil? || mail_info['body'].empty? + + return mail_info unless ENV['SEND_MAIL_PORT'].to_s == '49000' + + mail_info = check_email_mapping(mail_info.clone) + mail_info = check_email_limit(mail_info.clone) + return mail_info end def send_mail(mail_info) - check_send_mail(mail_info) + mail_info = check_send_mail(mail_info) + return if mail_info['to'].empty? + mail = Mail.new do from ENV['ROBOT_EMAIL_ADDRESS'] subject mail_info['subject'] @@ -79,7 +101,6 @@ def send_mail(mail_info) end def check_to_store_email(mail) - return if ENV['SEND_MAIL_PORT'].to_s != '49000' return if ENV['HOST_SERVER'] != 'z9' -- 2.23.0

1 0

[PATCH v3 lkp-tests] stats: add ansible_test stats script
by Zhang Yale 14 Jan '21

14 Jan '21

[why] Output needs to be displayed in the stats.json file, so this output need to processed as a key-value pair [input] PLAY RECAP ********************************************************************* localhost : ok=2 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 playbook_run_on_fail: 1 [output] ansible_test.total.nr_ok: 2 ansible_test.total.nr_changed: 0 ansible_test.total.nr_failed: 1 ansible_test.total.nr_skipped: 0 ansible_test.total.nr_rescued: 0 ansible_test.total.nr_ignored: 0 playbook_run_on_fail: 1 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/ansible_test | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100755 stats/ansible_test diff --git a/stats/ansible_test b/stats/ansible_test new file mode 100755 index 00000000..ec7b332c --- /dev/null +++ b/stats/ansible_test @@ -0,0 +1,15 @@ +#!/usr/bin/env ruby + +while (line = STDIN.gets) + case line.chomp! + when /: ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ + puts "ansible_test.total.nr_ok: #{$1}" + puts "ansible_test.total.nr_changed: #{$2}" + puts "ansible_test.total.nr_failed: #{$4}" + puts "ansible_test.total.nr_skipped: #{$5}" + puts "ansible_test.total.nr_rescued: #{$6}" + puts "ansible_test.total.nr_ignored: #{$7}" + when /.*playbook_run_on_fail/ + puts line + end +end -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats: add ansible_test stats script
by Zhang Yale 14 Jan '21

14 Jan '21

[input] PLAY RECAP ********************************************************************* localhost : ok=2 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 playbook_run_on_fail: 1 [output] ansible_test.total.nr_ok: 2 ansible_test.total.nr_changed: 0 ansible_test.total.nr_failed: 1 ansible_test.total.nr_skipped: 0 ansible_test.total.nr_rescued: 0 ansible_test.total.nr_ignored: 0 playbook_run_on_fail: 1 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/ansible_test | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100755 stats/ansible_test diff --git a/stats/ansible_test b/stats/ansible_test new file mode 100755 index 00000000..74ac32f1 --- /dev/null +++ b/stats/ansible_test @@ -0,0 +1,16 @@ +#!/usr/bin/env ruby + +while (line = STDIN.gets) + case line.chomp! + when /\: ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ + puts "ansible_test.total.nr_ok: #{$1}" + puts "ansible_test.total.nr_changed: #{$2}" + puts "ansible_test.total.nr_failed: #{$4}" + puts "ansible_test.total.nr_skipped: #{$5}" + puts "ansible_test.total.nr_rescued: #{$6}" + puts "ansible_test.total.nr_ignored: #{$7}" + when /.*playbook_run_on_fail/ + puts line + end +end + -- 2.23.0

4 8

[PATCH v7 compass-ci 12/12] send-internet-mail: email rate limit
by Luan Shengde 14 Jan '21

14 Jan '21

add email_in_limit to store email address that send count less than limit value. add email_out_limit to store email address that send count beyond limit value. the email queues email_in_limit/email_out_limit will be reset everyday at 00:00, every time send mail to an email address, the value for the mail address in email_in_limit will +1, until the value up to the limited value, the email address will be moved to email_out_limit, and the email send to the address will be throw away. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post/email_limit_queue.rb | 62 ++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100755 lib/mail-post/email_limit_queue.rb diff --git a/lib/mail-post/email_limit_queue.rb b/lib/mail-post/email_limit_queue.rb new file mode 100755 index 0000000..c7e3954 --- /dev/null +++ b/lib/mail-post/email_limit_queue.rb @@ -0,0 +1,62 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# check if the emails count has beyond the limit. +# email_in_limit is used to store email address email count +# less than the limit value. +# email_out_limit is used to store email address email count +# beyond the limit. +# case the email is the first one for the day, the email will +# be add to email_in_limit with value 1. +# case the email in email_in_limit but email count less than the +# limit value, its value will +1. +# case the email's email count up to the limit value, the email +# will be moved to email_out_limit. +# when the email is in email_out_limit, if send mail to the email +# address, the email will be kicked out from the email address bar. +class EmailRateLimit + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_counts + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + @mail_info['to'] = check_emails(email_to) + @mail_info['cc'] = check_emails(email_cc) + + return @mail_info + end + + def check_emails(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + if @redis.hexists 'email_out_limit', email + mail_list -= [email] + next + elsif @redis.hexists 'email_in_limit', email + email_account = @redis.hget 'email_in_limit', email + @redis.hset 'email_in_limit', email, email_account.to_i + 1 + else + @redis.hset 'email_in_limit', email, 1 + end + + change_queue(email) + end + return mail_list + end + + def change_queue(email) + return unless (@redis.hget 'email_in_limit', email).to_i >= 10 + + @redis.hdel 'email_in_limit', email + @redis.hset 'email_out_limit', email, 10 + end +end -- 2.23.0

1 1

[PATCH v7 compass-ci 11/12] send-internet-mail: add email mapping
by Luan Shengde 14 Jan '21

14 Jan '21

example: name => email_addr tag => email_addr email_addr_r => email_addr_d an email address can be set multi mappings, you can set a name or email to a specified email address. when you write either a name or an email address in the send address bar, the email address will be converted to the mapped email address. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post/email_mapping.rb | 50 ++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100755 lib/mail-post/email_mapping.rb diff --git a/lib/mail-post/email_mapping.rb b/lib/mail-post/email_mapping.rb new file mode 100755 index 0000000..02bfc6b --- /dev/null +++ b/lib/mail-post/email_mapping.rb @@ -0,0 +1,50 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# email address translation according to the email_mapping queue. +# email_mapping is used to set email mapping, example: +# email_mapping: +# name => email_address_d +# tag => email_address_d +# email_address_r => email_address_d +# ... +# +# can set multi keys for an email address, the key can be a name, +# an email, or some something else like a tag. +# case add the key is added to email address bar, the key will be +# transferred to the mapped email address. +class EmailAddrMapping + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_mapping + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + email_bcc = @mail_info['bcc'].clone + @mail_info['to'] = email_mapping(email_to) + @mail_info['cc'] = email_mapping(email_cc) + @mail_info['bcc'] = email_mapping(email_bcc) + + return @mail_info + end + + def email_mapping(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + next unless @redis.hexists 'email_mapping', email + + mapped_email = @redis.hget 'email_mapping', email + mail_list -= [email] + mail_list << mapped_email unless mail_list.include? mapped_email + end + return mail_list + end +end -- 2.23.0

1 1

[PATCH v7 compass-ci 10/12] send-internet-mail: add timing work
by Luan Shengde 14 Jan '21

14 Jan '21

daily timing work: reset email rate limit queues everyday at 00:00 Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post/email_init.rb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100755 lib/mail-post/email_init.rb diff --git a/lib/mail-post/email_init.rb b/lib/mail-post/email_init.rb new file mode 100755 index 0000000..d3f8323 --- /dev/null +++ b/lib/mail-post/email_init.rb @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'rufus-scheduler' +require 'redis' + +redis = Redis.new(host: REDIS_HOST, port: REDIS_PORT) +email_init = Rufus::Scheduler.new + +# Timing work for email counting everyday. +# the email counts queues will be reset everyday +# to ensure can send mail in the new day to user. +email_init.cron '0 0 * * *' do + redis.del 'email_in_limit' + redis.del 'email_out_limit' +end -- 2.23.0

1 1

[PATCH v7 compass-ci 09/12] lib/mail-post: add email counts/mapping checks
by Luan Shengde 14 Jan '21

14 Jan '21

add check email count: only email addresses that email count within the limited value can be sent emails add check email mapping: the address will be converted to the mapping value if it has been set an email mapping value Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/mail-post.rb b/lib/mail-post.rb index 803a134..4eb5800 100755 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -10,6 +10,10 @@ require 'open3' require 'mail' require 'base64' +require_relative 'mail-post/email_init' +require_relative 'mail-post/email_limit_queue' +require_relative 'mail-post/email_mapping' + set :bind, '0.0.0.0' set :port, ENV['SEND_MAIL_PORT'] -- 2.23.0

1 1

[PATCH v7 compass-ci 08/12] send-internet-mail: optimize
by Luan Shengde 14 Jan '21

14 Jan '21

add init redis host/port Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 3 --- 1 file changed, 3 deletions(-) diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb index 7b357f8..6003028 100755 --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -9,9 +9,6 @@ require 'sinatra' REDIS_HOST = %x(/sbin/ip route | awk '/default/ {print $3}').chomp REDIS_PORT = ENV['REDIS_PORT'] -require_relative 'email_init' -require_relative 'email_limit_queue' -require_relative 'email_mapping' require_relative '../../lib/mail-post' smtp = { -- 2.23.0

1 1

[PATCH v7 compass-ci 07/12] send-internet-mail/start: add key REDIS_PORT
by Luan Shengde 14 Jan '21

14 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 02c6ab6..136435d 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -13,6 +13,7 @@ names = Set.new %w[ ROBOT_EMAIL_PASSWORD SEND_MAIL_PORT SENT_MAILDIR + REDIS_PORT ] defaults = relevant_defaults(names) -- 2.23.0

1 1

[PATCH v7 compass-ci 06/12] lib/mail-post: add email counts/mapping checks
by Luan Shengde 14 Jan '21

14 Jan '21

add check email count: only email addresses that email count within the limited value can be sent emails add check email mapping: the address will be converted to the mapping value if it has been set an email mapping value Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) mode change 100644 => 100755 lib/mail-post.rb diff --git a/lib/mail-post.rb b/lib/mail-post.rb old mode 100644 new mode 100755 index 9d3e601..803a134 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -57,14 +57,32 @@ post '/send_mail_encode' do send_mail(mail_info) end +def check_email_limit(mail_info) + email_limit = EmailRateLimit.new(mail_info) + email_limit.check_email_counts +end + +def check_email_mapping(mail_info) + email_mapping = EmailAddrMapping.new(mail_info) + email_mapping.check_email_mapping +end + def check_send_mail(mail_info) raise 'no/empty subject.' if mail_info['subject'].nil? || mail_info['subject'].empty? raise 'no/empty email_to address.' if mail_info['to'].nil? || mail_info['to'].empty? raise 'no/empty email content.' if mail_info['body'].nil? || mail_info['body'].empty? + + return mail_info unless ENV['SEND_MAIL_PORT'].to_s == '49000' + + mail_info = check_email_mapping(mail_info.clone) + mail_info = check_email_limit(mail_info.clone) + return mail_info end def send_mail(mail_info) - check_send_mail(mail_info) + mail_info = check_send_mail(mail_info) + return if mail_info['to'].empty? + mail = Mail.new do from ENV['ROBOT_EMAIL_ADDRESS'] subject mail_info['subject'] @@ -79,7 +97,6 @@ def send_mail(mail_info) end def check_to_store_email(mail) - return if ENV['SEND_MAIL_PORT'].to_s != '49000' return if ENV['HOST_SERVER'] != 'z9' -- 2.23.0

1 1

[PATCH v7 compass-ci 05/12] send-internet-mail: email rate limit
by Luan Shengde 14 Jan '21

14 Jan '21

add email_in_limit to store email address that send count less than limit value. add email_out_limit to store email address that send count beyond limit value the email queues email_in_limit/email_out_limit will be reset everyday at 00:00, every time send mail to an email address, the value for the email address in email_in_limit will +1, until the value up to the limited value, the email address will be moved to email_out_limit, and the email send to the address will be throw away. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../send-internet-mail/email_limit_queue.rb | 62 +++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100755 container/send-internet-mail/email_limit_queue.rb diff --git a/container/send-internet-mail/email_limit_queue.rb b/container/send-internet-mail/email_limit_queue.rb new file mode 100755 index 0000000..c7e3954 --- /dev/null +++ b/container/send-internet-mail/email_limit_queue.rb @@ -0,0 +1,62 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# check if the emails count has beyond the limit. +# email_in_limit is used to store email address email count +# less than the limit value. +# email_out_limit is used to store email address email count +# beyond the limit. +# case the email is the first one for the day, the email will +# be add to email_in_limit with value 1. +# case the email in email_in_limit but email count less than the +# limit value, its value will +1. +# case the email's email count up to the limit value, the email +# will be moved to email_out_limit. +# when the email is in email_out_limit, if send mail to the email +# address, the email will be kicked out from the email address bar. +class EmailRateLimit + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_counts + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + @mail_info['to'] = check_emails(email_to) + @mail_info['cc'] = check_emails(email_cc) + + return @mail_info + end + + def check_emails(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + if @redis.hexists 'email_out_limit', email + mail_list -= [email] + next + elsif @redis.hexists 'email_in_limit', email + email_account = @redis.hget 'email_in_limit', email + @redis.hset 'email_in_limit', email, email_account.to_i + 1 + else + @redis.hset 'email_in_limit', email, 1 + end + + change_queue(email) + end + return mail_list + end + + def change_queue(email) + return unless (@redis.hget 'email_in_limit', email).to_i >= 10 + + @redis.hdel 'email_in_limit', email + @redis.hset 'email_out_limit', email, 10 + end +end -- 2.23.0

1 1

[PATCH v7 compass-ci 04/12] send-internet-mail: add email mapping
by Luan Shengde 14 Jan '21

14 Jan '21

example: name => email_addr tag => email_addr email_addr_r => email_addr_d an email address can be set multi mappings, you can set a name or email to a specified email address. when you write either a name or an email address in the send address bar, the email address will be converted to the mapped email address. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_mapping.rb | 50 +++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100755 container/send-internet-mail/email_mapping.rb diff --git a/container/send-internet-mail/email_mapping.rb b/container/send-internet-mail/email_mapping.rb new file mode 100755 index 0000000..02bfc6b --- /dev/null +++ b/container/send-internet-mail/email_mapping.rb @@ -0,0 +1,50 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# email address translation according to the email_mapping queue. +# email_mapping is used to set email mapping, example: +# email_mapping: +# name => email_address_d +# tag => email_address_d +# email_address_r => email_address_d +# ... +# +# can set multi keys for an email address, the key can be a name, +# an email, or some something else like a tag. +# case add the key is added to email address bar, the key will be +# transferred to the mapped email address. +class EmailAddrMapping + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_mapping + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + email_bcc = @mail_info['bcc'].clone + @mail_info['to'] = email_mapping(email_to) + @mail_info['cc'] = email_mapping(email_cc) + @mail_info['bcc'] = email_mapping(email_bcc) + + return @mail_info + end + + def email_mapping(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + next unless @redis.hexists 'email_mapping', email + + mapped_email = @redis.hget 'email_mapping', email + mail_list -= [email] + mail_list << mapped_email unless mail_list.include? mapped_email + end + return mail_list + end +end -- 2.23.0

1 1

[PATCH v7 compass-ci 03/12] send-internet-mail: add timing work
by Luan Shengde 14 Jan '21

14 Jan '21

daily timing work: reset email rate limit queues everyday at 00:00 Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_init.rb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100755 container/send-internet-mail/email_init.rb diff --git a/container/send-internet-mail/email_init.rb b/container/send-internet-mail/email_init.rb new file mode 100755 index 0000000..d3f8323 --- /dev/null +++ b/container/send-internet-mail/email_init.rb @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'rufus-scheduler' +require 'redis' + +redis = Redis.new(host: REDIS_HOST, port: REDIS_PORT) +email_init = Rufus::Scheduler.new + +# Timing work for email counting everyday. +# the email counts queues will be reset everyday +# to ensure can send mail in the new day to user. +email_init.cron '0 0 * * *' do + redis.del 'email_in_limit' + redis.del 'email_out_limit' +end -- 2.23.0

1 1

[PATCH compass-ci 3/3] container/auto-submit: delete unnecessary mount volume
by Liu Yinsi 14 Jan '21

14 Jan '21

[why] $CCI_SRC/sbin/auto-submit will call build_my_info_client, and auto output ~/.config/compass-ci/default/account.yaml in container, so mount /etc/compass-ci/account is unnecessary. delete code in sparrow/3-code/dev-env, reason is same as above. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/start | 1 - sparrow/3-code/dev-env | 7 ------- 2 files changed, 8 deletions(-) diff --git a/container/auto-submit/start b/container/auto-submit/start index c25fbdd..4632ab0 100755 --- a/container/auto-submit/start +++ b/container/auto-submit/start @@ -21,7 +21,6 @@ cmd = %W[ -v #{ENV['LKP_SRC']}:#{DOCKER_LKP} -v /etc/localtime:/etc/localtime:ro -v /etc/compass-ci/defaults:/etc/compass-ci/defaults:ro - -v /etc/compass-ci/account:/etc/compass-ci/account:ro -v /srv/git:/srv/git -w /c/compass-ci/sbin alpine:auto-submit diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index ca91da5..4bafa80 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -52,13 +52,6 @@ ES_HOST: $ES_HOST ES_PORT: $ES_PORT EOF -mkdir -p /etc/compass-ci/account -cat > /etc/compass-ci/account/autosubmit.yaml <<EOF -my_name: Auto Submit Robot -my_email: autosubmit@localhost -my_token: $(uuidgen) -EOF - cat > /etc/profile.d/compass.sh <<'EOF' export LKP_SRC=/c/lkp-tests export CCI_SRC=/c/compass-ci -- 2.23.0

1 0

[PATCH compass-ci 2/3] container/auto-submit: add package
by Liu Yinsi 14 Jan '21

14 Jan '21

[why] util-linux, io-console, elasticsearch will be used in $CCI_SRC/sbin/auto-submit Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/auto-submit/Dockerfile b/container/auto-submit/Dockerfile index 52cd996..5804642 100644 --- a/container/auto-submit/Dockerfile +++ b/container/auto-submit/Dockerfile @@ -9,9 +9,9 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou RUN apk update && \ apk upgrade && \ apk add --no-cache git && \ - apk add ruby-dev make gcc g++ + apk add ruby-dev make gcc g++ util-linux RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc RUN umask 002 && \ - gem install bunny json activesupport git rest-client + gem install bunny json activesupport git rest-client io-console elasticsearch -- 2.23.0

1 0

[PATCH v7 compass-ci 02/12] send-internet-mail: optimize
by Luan Shengde 14 Jan '21

14 Jan '21

add init redis host/port add invoke code files Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb index 8fdaa0d..7b357f8 100755 --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -6,7 +6,13 @@ require 'mail' require 'sinatra' -require "#{ENV['CCI_SRC']}/lib/mail-post" +REDIS_HOST = %x(/sbin/ip route | awk '/default/ {print $3}').chomp +REDIS_PORT = ENV['REDIS_PORT'] + +require_relative 'email_init' +require_relative 'email_limit_queue' +require_relative 'email_mapping' +require_relative '../../lib/mail-post' smtp = { address: 'smtp.qq.com', -- 2.23.0

1 1

[PATCH v7 compass-ci 01/12] send-internet-mail/Dockerfile: add install gem resources
by Luan Shengde 14 Jan '21

14 Jan '21

[why] redis: used to read/store email counts from/to redis queue rufus-scheduler: used to reset the queue as a timing task Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index d151bda..03e3327 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -6,4 +6,4 @@ MAINTAINER shdluan(a)163.com ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y ruby-mail ruby-json ruby-sinatra + apt-get install -y ruby-mail ruby-json ruby-sinatra ruby-redis ruby-rufus-scheduler -- 2.23.0

1 1

Re: [PATCH v2 lkp-tests] jobs: and ansible_test.yaml
by Liu Shaofei 14 Jan '21

14 Jan '21

^---- and => add Reply-To: In-Reply-To: <20210114021509.109179-1-ylzhangah(a)qq.com> On Thu, Jan 14, 2021 at 10:15:09AM +0800, Zhang Yale wrote: >Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> >--- > jobs/ansible_test.yaml | 6 ++++++ > 1 file changed, 6 insertions(+) > create mode 100644 jobs/ansible_test.yaml > >diff --git a/jobs/ansible_test.yaml b/jobs/ansible_test.yaml >new file mode 100644 >index 00000000..49f31562 >--- /dev/null >+++ b/jobs/ansible_test.yaml >@@ -0,0 +1,6 @@ >+suite: ansible_test >+category: functional >+ >+ansible_test: >+ ansible_repo: >+# - geerlingguy/ansible-role-jenkins ^---- whether to remove '#'? Thanks, Liushaofei >-- >2.23.0 >

2 1

[PATCH lkp-tests] stats: add ansible_test stats script
by Zhang Yale 14 Jan '21

14 Jan '21

output: ansible_test.total.nr_ok: 2 ansible_test.total.nr_changed: 0 ansible_test.total.nr_failed: 1 ansible_test.total.nr_skipped: 0 ansible_test.total.nr_rescued: 0 ansible_test.total.nr_ignored: 0 playbook_run_on_fail: 0 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/ansible_test | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100755 stats/ansible_test diff --git a/stats/ansible_test b/stats/ansible_test new file mode 100755 index 00000000..74ac32f1 --- /dev/null +++ b/stats/ansible_test @@ -0,0 +1,16 @@ +#!/usr/bin/env ruby + +while (line = STDIN.gets) + case line.chomp! + when /\: ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ + puts "ansible_test.total.nr_ok: #{$1}" + puts "ansible_test.total.nr_changed: #{$2}" + puts "ansible_test.total.nr_failed: #{$4}" + puts "ansible_test.total.nr_skipped: #{$5}" + puts "ansible_test.total.nr_rescued: #{$6}" + puts "ansible_test.total.nr_ignored: #{$7}" + when /.*playbook_run_on_fail/ + puts line + end +end + -- 2.23.0

1 1

[PATCH compass-ci] docker2rootfs: add unzip vmlinux function
by Wang Chenglong 14 Jan '21

14 Jan '21

[Why] Sometimes, the kernel is in vmlinux format and needs to be unpacked into vmlinuz format. Signeo-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker2rootfs/common | 57 ++++++++++++++++++++++++++++- container/docker2rootfs/run | 1 + container/docker2rootfs/setup-image | 2 +- 3 files changed, 58 insertions(+), 2 deletions(-) diff --git a/container/docker2rootfs/common b/container/docker2rootfs/common index e908079..66aba62 100755 --- a/container/docker2rootfs/common +++ b/container/docker2rootfs/common @@ -133,7 +133,7 @@ get_rootfs_kernel() local vmlinuz_file local vmlinuz kernel - vmlinuz_file=$(find ./boot -name "vmlinuz-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz_file=$(find ./boot -name "vmlinu[z|x]-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel @@ -155,3 +155,58 @@ create_links_vmlinuz_initrd() echo "[INFO] Create links to initrd.lkp and vmlinuz success!" } + +get_vmlinuz_start_postion() { + local vmlinuz=$1 + + local vmlinuz_start_line + local vmlinuz_start_postion + # '1f 8b 08' - https://www.filesignatures.net/index.php?page=search&search=1F8B08&mode=SIG + vmlinuz_start_line=$(od -A d -t x1 ${vmlinuz} | grep "1f 8b 08") + vmlinuz_start_postion=${vmlinuz_start_line%% *} + + [ -z ${vmlinuz_start_postion} ] && { + echo "[WARNING] identify vmlinuz failed." + export VMLINUZ_START_POSTION=-1 + return + } + + [ X${vmlinuz_start_postion} == X0000000 ] || { + local vmlinuz_start_postion_tmp + vmlinuz_start_postion_tmp=$(echo ${vmlinuz_start_line} | awk '{for(i=1;i<=NF;i++)if($i=="1f")print i-1}') + vmlinuz_start_postion=$((${vmlinuz_start_postion} + ${vmlinuz_start_postion_tmp} -1)) + } + + export VMLINUZ_START_POSTION=${vmlinuz_start_postion} +} + +unzip_vmlinuz() { + get_rootfs_kernel + + file ${ROOTFS_VMLINUZ_FILE} | grep -q gzip || return + + get_vmlinuz_start_postion ${ROOTFS_VMLINUZ_FILE} + [ $VMLINUZ_START_POSTION -eq -1 ] && return + + local unzip_str + + if [ $VMLINUZ_START_POSTION -eq 0 ]; then + unzip_str="dd if=$ROOTFS_VMLINUZ bs=1 | zcat > $ROOTFS_VMLINUZ.tmp" + else + unzip_str="dd if=$ROOTFS_VMLINUZ bs=1 skip=$VMLINUZ_START_POSTION | zcat > $ROOTFS_VMLINUZ.tmp" + fi + + echo "Unzipping vmlinuz..." + local cmds + cmds=( + cd $ROOTFS_DIR/boot + $unzip_str + mv $ROOTFS_VMLINUZ.tmp $ROOTFS_VMLINUZ + chmod o+r $ROOTFS_VMLINUZ + ) + if "${cmds[@]}" ; then + echo "[INFO] Unzipping vmlinuz success!" + else + echo "[WARNING] Unzipping vmlinuz failed!" + fi +} diff --git a/container/docker2rootfs/run b/container/docker2rootfs/run index 8a7b315..cdb19bf 100755 --- a/container/docker2rootfs/run +++ b/container/docker2rootfs/run @@ -31,3 +31,4 @@ start=( cp_rootfs $IMAGE_PACK $2 docker rm -f rootfs-docker create_links_vmlinuz_initrd $2 +unzip_vmlinuz diff --git a/container/docker2rootfs/setup-image b/container/docker2rootfs/setup-image index ffe4937..47bfb7e 100755 --- a/container/docker2rootfs/setup-image +++ b/container/docker2rootfs/setup-image @@ -38,7 +38,7 @@ setup_login() pack_cgz() { echo "Packing package. Please wait." - find ./ ! -path "./tmp/${IMAGE_PACK}" | cpio -o -Hnewc | gzip -9 >./tmp/$IMAGE_PACK + find ./ ! -path "./tmp/$IMAGE_PACK" | cpio -o -Hnewc | gzip -9 > "./tmp/$IMAGE_PACK" chmod 644 /tmp/${IMAGE_PACK} } -- 2.23.0

3 4

[PATCH v2 compass-ci] docker2rootfs: add unzip vmlinux function
by Wang Chenglong 14 Jan '21

14 Jan '21

[Why] Sometimes, the kernel is in vmlinux format and needs to be unpacked into vmlinuz format. Signeo-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker2rootfs/common | 57 ++++++++++++++++++++++++++++- container/docker2rootfs/run | 1 + container/docker2rootfs/setup-image | 2 +- 3 files changed, 58 insertions(+), 2 deletions(-) diff --git a/container/docker2rootfs/common b/container/docker2rootfs/common index c26bbc8..91d53e7 100755 --- a/container/docker2rootfs/common +++ b/container/docker2rootfs/common @@ -133,7 +133,7 @@ get_rootfs_kernel() local vmlinuz_file local vmlinuz kernel - vmlinuz_file=$(find ./boot -name "vmlinuz-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz_file=$(find ./boot -name "vmlinu[z|x]-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel @@ -155,3 +155,58 @@ create_links_vmlinuz_initrd() echo "[INFO] Create links to initrd.lkp and vmlinuz success!" } + +get_vmlinuz_start_postion() { + local vmlinuz=$1 + + local vmlinuz_start_line + local vmlinuz_start_postion + # '1f 8b 08' - https://www.filesignatures.net/index.php?page=search&search=1F8B08&mode=SIG + vmlinuz_start_line=$(od -A d -t x1 ${vmlinuz} | grep "1f 8b 08") + vmlinuz_start_postion=${vmlinuz_start_line%% *} + + [ -z ${vmlinuz_start_postion} ] && { + echo "[WARNING] identify vmlinuz failed." + export VMLINUZ_START_POSTION=-1 + return + } + + [ X${vmlinuz_start_postion} == X0000000 ] || { + local vmlinuz_start_postion_tmp + vmlinuz_start_postion_tmp=$(echo ${vmlinuz_start_line} | awk '{for(i=1;i<=NF;i++)if($i=="1f")print i-1}') + vmlinuz_start_postion=$((${vmlinuz_start_postion} + ${vmlinuz_start_postion_tmp} -1)) + } + + export VMLINUZ_START_POSTION=${vmlinuz_start_postion} +} + +unzip_vmlinuz() { + get_rootfs_kernel + + file ${ROOTFS_VMLINUZ_FILE} | grep -q gzip || return + + get_vmlinuz_start_postion ${ROOTFS_VMLINUZ_FILE} + [ $VMLINUZ_START_POSTION -eq -1 ] && return + + local unzip_str + + if [ $VMLINUZ_START_POSTION -eq 0 ]; then + unzip_str="dd if=$ROOTFS_VMLINUZ bs=1 | zcat > $ROOTFS_VMLINUZ.tmp" + else + unzip_str="dd if=$ROOTFS_VMLINUZ bs=1 skip=$VMLINUZ_START_POSTION | zcat > $ROOTFS_VMLINUZ.tmp" + fi + + echo "Unzipping vmlinuz..." + local cmds + cmds=( + cd $ROOTFS_DIR/boot + $unzip_str + mv $ROOTFS_VMLINUZ.tmp $ROOTFS_VMLINUZ + chmod o+r $ROOTFS_VMLINUZ + ) + if "${cmds[@]}" ; then + echo "[INFO] Unzipping vmlinuz success!" + else + echo "[WARNING] Unzipping vmlinuz failed!" + fi +} diff --git a/container/docker2rootfs/run b/container/docker2rootfs/run index 8a7b315..cdb19bf 100755 --- a/container/docker2rootfs/run +++ b/container/docker2rootfs/run @@ -31,3 +31,4 @@ start=( cp_rootfs $IMAGE_PACK $2 docker rm -f rootfs-docker create_links_vmlinuz_initrd $2 +unzip_vmlinuz diff --git a/container/docker2rootfs/setup-image b/container/docker2rootfs/setup-image index ffe4937..47bfb7e 100755 --- a/container/docker2rootfs/setup-image +++ b/container/docker2rootfs/setup-image @@ -38,7 +38,7 @@ setup_login() pack_cgz() { echo "Packing package. Please wait." - find ./ ! -path "./tmp/${IMAGE_PACK}" | cpio -o -Hnewc | gzip -9 >./tmp/$IMAGE_PACK + find ./ ! -path "./tmp/$IMAGE_PACK" | cpio -o -Hnewc | gzip -9 > "./tmp/$IMAGE_PACK" chmod 644 /tmp/${IMAGE_PACK} } -- 2.23.0

2 1

[PATCH lkp-tests] stats: add stats script for openeuler_docker test
by Bai Jing 14 Jan '21

14 Jan '21

[input] ^[[0mwordpress_build_on_fail: 1 [output] wordpress_build_on_fail: 1 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/openeuler_docker | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100755 stats/openeuler_docker diff --git a/stats/openeuler_docker b/stats/openeuler_docker new file mode 100755 index 000000000..5ede8a240 --- /dev/null +++ b/stats/openeuler_docker @@ -0,0 +1,8 @@ +#!/usr/bin/env ruby + +while (line = STDIN.gets) + line.chomp! + next unless line =~ /.*_build_on_fail/ + + puts line +end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] delimiter: use assist-result container client to get job yaml
by Cao Xueliang 14 Jan '21

14 Jan '21

Before: Get the job yaml from es, we need deal some process products of job. After: Get the job yaml from assist-result container without the process products. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/constants.rb | 2 +- src/delimiter/utils.rb | 24 +++++++----------------- 2 files changed, 8 insertions(+), 18 deletions(-) diff --git a/src/delimiter/constants.rb b/src/delimiter/constants.rb index ebf90cc..7b14558 100644 --- a/src/delimiter/constants.rb +++ b/src/delimiter/constants.rb @@ -6,7 +6,7 @@ LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' GIT_MIRROR_HOST = ENV['GIT_MIRROR_HOST'] || '172.17.0.1' MONITOR_HOST = ENV['MONITOR_HOST'] || '172.17.0.1' MONITOR_PORT = ENV['MONITOR_PORT'] || '11310' -DELIMITER_ACCONUT = ENV['DELIMITER_ACCONUT'] || 'compass-ci(a)qq.com' +DELIMITER_ACCONUT = ENV['DELIMITER_ACCONUT'] || 'caoxl78320(a)163.com' TMEP_GIT_BASE = '/c/public_git' DELIMITER_TASK_QUEUE = 'delimiter' BISECT_RUN_SCRIPT = "#{ENV['CCI_SRC']}/src/delimiter/find-commit/bisect_run_script.rb" diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index de5a302..61fe6ab 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -9,6 +9,7 @@ require 'fileutils' require_relative './constants' require_relative '../../lib/sched_client' +require_relative '../../lib/assist_result_client' require_relative '../../lib/compare_error_messages' require_relative "#{ENV['LKP_SRC']}/lib/monitor" @@ -121,9 +122,10 @@ module Utils end def init_job_content(job_id) - es = ESQuery.new - job = es.query_by_id(job_id) - raise "es query job id: #{job_id} failed!" unless job + job_yaml = AssistResult.new.get_job_yaml(job_id) + raise "get job yaml failed, job id: #{job_id} !" unless job_yaml + + job = JSON.parse job_yaml account_info = get_account_info raise "query #{DELIMITER_ACCONUT} account info failed!" unless account_info @@ -135,21 +137,9 @@ module Utils job['my_email'] = account_info['my_email'] job['my_token'] = account_info['my_token'] job['bad_job_id'] = job_id - job['testbox'] = job['config'] ? 'dc-16g' : 'dc-8g' - job['os'] = 'openeuler' - job['os_arch'] = 'aarch64' - job['os_mount'] = 'container' - job['os_version'] = '20.03-pre' - job['docker_image'] = 'openeuler:20.03-pre' - - job.delete('start_time') - job.delete('tbox_group') - job.delete('error_ids') - job.delete('job_state') - job.delete('end_time') - job.delete('loadavg') + + job.delete('subqueue') job.delete('queue') - job.delete('stats') job.delete('id') return job -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib: add assist_result_client for assist-result container
by Cao Xueliang 14 Jan '21

14 Jan '21

[why] Add client for assist-result container, others can call the API by the client. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/assist_result_client.rb | 22 ++++++++++++++++++++++ lib/constants.rb | 3 +++ 2 files changed, 25 insertions(+) create mode 100644 lib/assist_result_client.rb diff --git a/lib/assist_result_client.rb b/lib/assist_result_client.rb new file mode 100644 index 0000000..042b189 --- /dev/null +++ b/lib/assist_result_client.rb @@ -0,0 +1,22 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'rest-client' +require_relative 'constants' + +class AssistResult + def initialize(host = ASSIST_RESULT_HOST, port = ASSIST_RESULT_PORT) + @host = host + @port = port + end + + def get_job_yaml(job_id) + resource = RestClient::Resource.new("http://#{@host}:#{@port}/get_job_yaml/#{job_id}") + response = resource.get() + return nil unless response.code == 200 + + return response.body + end +end diff --git a/lib/constants.rb b/lib/constants.rb index a274adc..ffc4f03 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -15,3 +15,6 @@ SRV_HTTP_HOST = config['SRV_HTTP_HOST'] || ENV['SRV_HTTP_HOST'] || '172.17.0.1' SRV_HTTP_PORT = config['SRV_HTTP_PORT'] || ENV['SRV_HTTP_PORT'] || 11300 SRV_HTTP_DOMAIN = config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.compass-ci.openeuler.org' + +ASSIST_RESULT_HOST = config['ASSIST_RESULT_HOST'] || ENV['ASSIST_RESULT_HOST'] || '172.17.0.1' +ASSIST_RESULT_PORT = config['ASSIST_RESULT_PORT'] || ENV['ASSIST_RESULT_PORT'] || 8102 -- 2.23.0

1 0

[PATCH v2 lkp-tests] jobs: and ansible_test.yaml
by Zhang Yale 14 Jan '21

14 Jan '21

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/ansible_test.yaml | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 jobs/ansible_test.yaml diff --git a/jobs/ansible_test.yaml b/jobs/ansible_test.yaml new file mode 100644 index 00000000..49f31562 --- /dev/null +++ b/jobs/ansible_test.yaml @@ -0,0 +1,6 @@ +suite: ansible_test +category: functional + +ansible_test: + ansible_repo: +# - geerlingguy/ansible-role-jenkins -- 2.23.0

1 0

[PATCH v1 compass-ci 2/2] delimiter: use assist-result container client to get job yaml
by Cao Xueliang 14 Jan '21

14 Jan '21

Before: Get the job yaml from es, we need deal some process products of job. After: Get the job yaml from assist-result container without the process products. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/constants.rb | 2 +- src/delimiter/utils.rb | 24 +++++++----------------- 2 files changed, 8 insertions(+), 18 deletions(-) diff --git a/src/delimiter/constants.rb b/src/delimiter/constants.rb index ebf90cc..7b14558 100644 --- a/src/delimiter/constants.rb +++ b/src/delimiter/constants.rb @@ -6,7 +6,7 @@ LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' GIT_MIRROR_HOST = ENV['GIT_MIRROR_HOST'] || '172.17.0.1' MONITOR_HOST = ENV['MONITOR_HOST'] || '172.17.0.1' MONITOR_PORT = ENV['MONITOR_PORT'] || '11310' -DELIMITER_ACCONUT = ENV['DELIMITER_ACCONUT'] || 'compass-ci(a)qq.com' +DELIMITER_ACCONUT = ENV['DELIMITER_ACCONUT'] || 'caoxl78320(a)163.com' TMEP_GIT_BASE = '/c/public_git' DELIMITER_TASK_QUEUE = 'delimiter' BISECT_RUN_SCRIPT = "#{ENV['CCI_SRC']}/src/delimiter/find-commit/bisect_run_script.rb" diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index de5a302..61fe6ab 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -9,6 +9,7 @@ require 'fileutils' require_relative './constants' require_relative '../../lib/sched_client' +require_relative '../../lib/assist_result_client' require_relative '../../lib/compare_error_messages' require_relative "#{ENV['LKP_SRC']}/lib/monitor" @@ -121,9 +122,10 @@ module Utils end def init_job_content(job_id) - es = ESQuery.new - job = es.query_by_id(job_id) - raise "es query job id: #{job_id} failed!" unless job + job_yaml = AssistResult.new.get_job_yaml(job_id) + raise "get job yaml failed, job id: #{job_id} !" unless job_yaml + + job = JSON.parse job_yaml account_info = get_account_info raise "query #{DELIMITER_ACCONUT} account info failed!" unless account_info @@ -135,21 +137,9 @@ module Utils job['my_email'] = account_info['my_email'] job['my_token'] = account_info['my_token'] job['bad_job_id'] = job_id - job['testbox'] = job['config'] ? 'dc-16g' : 'dc-8g' - job['os'] = 'openeuler' - job['os_arch'] = 'aarch64' - job['os_mount'] = 'container' - job['os_version'] = '20.03-pre' - job['docker_image'] = 'openeuler:20.03-pre' - - job.delete('start_time') - job.delete('tbox_group') - job.delete('error_ids') - job.delete('job_state') - job.delete('end_time') - job.delete('loadavg') + + job.delete('subqueue') job.delete('queue') - job.delete('stats') job.delete('id') return job -- 2.23.0

1 0

[PATCH v1 compass-ci 1/2] lib: add assist_result_client for assist_result container
by Cao Xueliang 14 Jan '21

14 Jan '21

[why] Add client for assist_result container, others can call the assist_result container API by the client. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/assist_result_client.rb | 22 ++++++++++++++++++++++ lib/constants.rb | 3 +++ 2 files changed, 25 insertions(+) create mode 100644 lib/assist_result_client.rb diff --git a/lib/assist_result_client.rb b/lib/assist_result_client.rb new file mode 100644 index 0000000..042b189 --- /dev/null +++ b/lib/assist_result_client.rb @@ -0,0 +1,22 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'rest-client' +require_relative 'constants' + +class AssistResult + def initialize(host = ASSIST_RESULT_HOST, port = ASSIST_RESULT_PORT) + @host = host + @port = port + end + + def get_job_yaml(job_id) + resource = RestClient::Resource.new("http://#{@host}:#{@port}/get_job_yaml/#{job_id}") + response = resource.get() + return nil unless response.code == 200 + + return response.body + end +end diff --git a/lib/constants.rb b/lib/constants.rb index a274adc..ffc4f03 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -15,3 +15,6 @@ SRV_HTTP_HOST = config['SRV_HTTP_HOST'] || ENV['SRV_HTTP_HOST'] || '172.17.0.1' SRV_HTTP_PORT = config['SRV_HTTP_PORT'] || ENV['SRV_HTTP_PORT'] || 11300 SRV_HTTP_DOMAIN = config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.compass-ci.openeuler.org' + +ASSIST_RESULT_HOST = config['ASSIST_RESULT_HOST'] || ENV['ASSIST_RESULT_HOST'] || '172.17.0.1' +ASSIST_RESULT_PORT = config['ASSIST_RESULT_PORT'] || ENV['ASSIST_RESULT_PORT'] || 8102 -- 2.23.0

1 0

[PATCH v3 compass-ci] tool/es-jobs: a new tool for summarizing jobs information
by Wu Zhende 14 Jan '21

14 Jan '21

Use es-jobs to summarizes information about a group of related jobs. [Example] es-jobs submit_id=xxx -f 'job_state,dmesg.timestamp:last' -r '-1' -s 'on_fail' means: 1. query the job whose submit_id is xxx 2. I want to get job_state,dmesg.timestamp:last 3. don't refine jobs 4. -s 'on_fail' to get data in stats which contain 'on_fail' [Output] { "job_state": [ "crystal.605716.failed" ], "dmesg.timestamp:last": [ 657.181408 ], "stats_filter_result": { "crystal.605716.openeuler_docker.\u001b[0mwordpress_build_on_fail": 1 }, "stats.count": { "stats.has_error": 1, "stats.has_error_jobs": [ "crystal.605716" ] } } Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 139 +++++++++++++++++++++++++++++++++++++++++++++++++ sbin/es-jobs | 58 +++++++++++++++++++++ 2 files changed, 197 insertions(+) create mode 100644 lib/es_jobs.rb create mode 100755 sbin/es-jobs diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb new file mode 100644 index 0000000..a90b1ba --- /dev/null +++ b/lib/es_jobs.rb @@ -0,0 +1,139 @@ +# SPDX-License-Identifier: GPL-2.0-only +# frozen_string_literal: true + +LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' + +require "#{LKP_SRC}/lib/stats" +require_relative './es_query' + +# deal jobs search from es +class ESJobs + def initialize(es_query, my_refine = [], fields = [], stats_filter = []) + @es_query = es_query + @es = ESQuery.new(ES_HOST, ES_PORT) + @refine = my_refine + @fields = fields + @stats_filter = stats_filter + @stats_filter_result = {} + @refine_jobs = [] + @jobs = {} + @stats_level = { + 0 => 'stats.success', + 1 => 'stats.unknown', + 2 => 'stats.warning', + 3 => 'stats.has_error' + } + set_defaults + deal_jobs + end + + def set_defaults + query_result = @es.multi_field_query(@es_query) + query_result['hits']['hits'].each do |job| + @jobs[job['_id']] = job['_source'] + end + + @stats = { + 'stats.count' => Hash.new(0), + 'stats.sum' => Hash.new(0), + 'stats.avg' => Hash.new(0) + } + @result = {} + @fields.each do |field| + @result[field] = [] + end + end + + def add_result_fields(job, level) + return unless @refine.include?(level) || @refine.include?(-1) + + @refine_jobs << job['id'] + @fields.each do |field| + value = job[field] + if value + value = job['id'] + '.' + value if field == 'job_state' + @result[field] << value + end + + next unless job['stats'] + + @result[field] << job['stats'][field] if job['stats'][field] + end + end + + def deal_jobs + stats_count = Hash.new(0) + stats_jobs = {} + + @jobs.each do |job_id, job| + level = deal_stats(job) + add_result_fields(job, level) + + stat_key = @stats_level[level] + stat_jobs_key = stat_key + '_jobs' + + stats_count[stat_key] += 1 + stats_jobs[stat_jobs_key] ||= [] + stats_jobs[stat_jobs_key] << job_id + end + + @stats['stats.count'].merge!(stats_count) + @stats['stats.count'].merge!(stats_jobs) + end + + def deal_stats(job, level = 0) + return 1 unless job['stats'] + + job['stats'].each do |key, value| + match_stats_filter(key, value, job['id']) + calculate_stat(key, value) + level = get_stat_level(key, level) + end + return level + end + + def match_stats_filter(key, value, job_id) + @stats_filter.each do |filter| + next unless key.include?(filter) + + key = job_id + '.' + key + @stats_filter_result[key] = value + + break + end + end + + def calculate_stat(key, value) + if function_stat?(key) + return unless @fields.include?('stats.sum') + + @stats['stats.sum'][key] += value + else + return unless @fields.include?('stats.avg') + + @stats['stats.avg'][key] = (@stats['stats.avg'][key] + value) / 2 + end + end + + def get_stat_level(stat, level) + return level if level >= 3 + return 3 if stat.match(/error|fail/i) + return 2 if stat.match(/warn/i) + + return 0 + end + + def output + result = { + 'stats.count' => @stats['stats.count'] + } + + @stats.each do |key, value| + result[key] = value if @fields.include?(key) + end + + @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? + @result.merge!(result) + puts JSON.pretty_generate(@result) + end +end diff --git a/sbin/es-jobs b/sbin/es-jobs new file mode 100755 index 0000000..25cbe9c --- /dev/null +++ b/sbin/es-jobs @@ -0,0 +1,58 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'optparse' +require 'json' + +require_relative '../lib/es_jobs' + +def parse_argv + items = {} + ARGV.each do |item| + key, value = item.split('=') + if key && value + value_list = value.split(',') + items[key] = value_list.length > 1 ? value_list : value + end + end + items +end +opt_refine = [-1] +opt_fields = [] +opt_stats_filter = [] +opt_parser = OptionParser.new do |opts| + opts.banner = 'Usage: es-jobs [options] search_key1=val1[,val2..] ..' + + opts.separator 'search_key can be id, suite, os, etc.' + opts.separator 'how to use -r' + opts.separator 'like es-jobs submit_id=xxx -r "-1,0,1,2,3"' + opts.separator '-1 means not refine' + opts.separator '{ + -1 => null, + 0 => stats.succes, + 1 => stats.unknow, + 2 => stats.warning, + 3 => stats.has_error +}' + + opts.on('-r fields', '--refine fields', 'refine jobs') do |fields| + opt_refine = fields.split(',').map(&:to_i) + end + + opts.on('-f fields', '--fields fields', 'fields you want to see') do |fields| + opt_fields = fields.split(',') + end + + opts.on('-s fields', '--stats-filter fields', 'return data contains fields') do |fields| + opt_stats_filter = fields.split(',') + end +end + +opt_parser.parse!(ARGV) +items = parse_argv +raise 'Please enter a query' if items.empty? + +es_jobs = ESJobs.new(items, opt_refine, opt_fields, opt_stats_filter) +es_jobs.output -- 2.23.0

2 1

[PATCH v2 lkp-tests 2/2] modify reset-tbox's yaml and script
by Zhang Dewan 14 Jan '21

14 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 3 +-- tests/reset-tbox | 7 ++++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml index 9c3f4f01..2ceccdb9 100644 --- a/jobs/reset-tbox.yaml +++ b/jobs/reset-tbox.yaml @@ -1,10 +1,9 @@ suite: reset-tbox -testcase: reset-tbox testbox: dc-8g os_mount: container docker_image: 172.168.131.2:5001/reset-tbox -queue: reset +queue: reset-tbox host_name: reset-tbox: diff --git a/tests/reset-tbox b/tests/reset-tbox index b0f645d7..5b1279b6 100755 --- a/tests/reset-tbox +++ b/tests/reset-tbox @@ -7,11 +7,12 @@ hosts_file=/etc/lab/hosts_file # require $host_name host_ibmc_ip=$(grep -w $host_name $hosts_file | awk '{print $1}') -if [[ "$host_ibmc_ip" =~ ^[0-9]+.* ]]; then +if [[ "$host_ibmc_ip" =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]; then echo ">>> exec ipmitool commamd!" ipmitool -I lanplus -H $host_ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset - [ $? -eq 0 ] && echo "succeed!" + [ $? -eq 0 ] && echo "reboot $host_name succeed!" || \ + "echo reboot $host_name failed!" else - echo "specify host cannot found0!" + echo "specify host cannot found!" exit fi -- 2.23.0

2 2

[PATCH compass-ci 2/2] delimiter: use assist-result container client to get job yaml
by Cao Xueliang 14 Jan '21

14 Jan '21

Before: Get the job yaml from es, we need deal some process products of job. After: Get the job yaml from assist-result container without the process products. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/constants.rb | 2 +- src/delimiter/utils.rb | 24 +++++++----------------- 2 files changed, 8 insertions(+), 18 deletions(-) diff --git a/src/delimiter/constants.rb b/src/delimiter/constants.rb index ebf90cc..7b14558 100644 --- a/src/delimiter/constants.rb +++ b/src/delimiter/constants.rb @@ -6,7 +6,7 @@ LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' GIT_MIRROR_HOST = ENV['GIT_MIRROR_HOST'] || '172.17.0.1' MONITOR_HOST = ENV['MONITOR_HOST'] || '172.17.0.1' MONITOR_PORT = ENV['MONITOR_PORT'] || '11310' -DELIMITER_ACCONUT = ENV['DELIMITER_ACCONUT'] || 'compass-ci(a)qq.com' +DELIMITER_ACCONUT = ENV['DELIMITER_ACCONUT'] || 'caoxl78320(a)163.com' TMEP_GIT_BASE = '/c/public_git' DELIMITER_TASK_QUEUE = 'delimiter' BISECT_RUN_SCRIPT = "#{ENV['CCI_SRC']}/src/delimiter/find-commit/bisect_run_script.rb" diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index de5a302..61fe6ab 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -9,6 +9,7 @@ require 'fileutils' require_relative './constants' require_relative '../../lib/sched_client' +require_relative '../../lib/assist_result_client' require_relative '../../lib/compare_error_messages' require_relative "#{ENV['LKP_SRC']}/lib/monitor" @@ -121,9 +122,10 @@ module Utils end def init_job_content(job_id) - es = ESQuery.new - job = es.query_by_id(job_id) - raise "es query job id: #{job_id} failed!" unless job + job_yaml = AssistResult.new.get_job_yaml(job_id) + raise "get job yaml failed, job id: #{job_id} !" unless job_yaml + + job = JSON.parse job_yaml account_info = get_account_info raise "query #{DELIMITER_ACCONUT} account info failed!" unless account_info @@ -135,21 +137,9 @@ module Utils job['my_email'] = account_info['my_email'] job['my_token'] = account_info['my_token'] job['bad_job_id'] = job_id - job['testbox'] = job['config'] ? 'dc-16g' : 'dc-8g' - job['os'] = 'openeuler' - job['os_arch'] = 'aarch64' - job['os_mount'] = 'container' - job['os_version'] = '20.03-pre' - job['docker_image'] = 'openeuler:20.03-pre' - - job.delete('start_time') - job.delete('tbox_group') - job.delete('error_ids') - job.delete('job_state') - job.delete('end_time') - job.delete('loadavg') + + job.delete('subqueue') job.delete('queue') - job.delete('stats') job.delete('id') return job -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib: add assist_result_client for assist_result container
by Cao Xueliang 14 Jan '21

14 Jan '21

[why] Add client for assist_result container, others can call the assist_result container API by the client. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/assist_result_client.rb | 21 +++++++++++++++++++++ lib/constants.rb | 3 +++ 2 files changed, 24 insertions(+) create mode 100644 lib/assist_result_client.rb diff --git a/lib/assist_result_client.rb b/lib/assist_result_client.rb new file mode 100644 index 0000000..980486d --- /dev/null +++ b/lib/assist_result_client.rb @@ -0,0 +1,21 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2021 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true +require 'rest-client' +require_relative 'constants' + +class AssistResult + def initialize(host = ASSIST_RESULT_HOST, port = ASSIST_RESULT_PORT) + @host = host + @port = port + end + + def get_job_yaml(job_id) + resource = RestClient::Resource.new("http://#{@host}:#{@port}/get_job_yaml/#{job_id}") + response = resource.get() + return nil unless response.code == 200 + + return response.body + end +end diff --git a/lib/constants.rb b/lib/constants.rb index a274adc..ffc4f03 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -15,3 +15,6 @@ SRV_HTTP_HOST = config['SRV_HTTP_HOST'] || ENV['SRV_HTTP_HOST'] || '172.17.0.1' SRV_HTTP_PORT = config['SRV_HTTP_PORT'] || ENV['SRV_HTTP_PORT'] || 11300 SRV_HTTP_DOMAIN = config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.compass-ci.openeuler.org' + +ASSIST_RESULT_HOST = config['ASSIST_RESULT_HOST'] || ENV['ASSIST_RESULT_HOST'] || '172.17.0.1' +ASSIST_RESULT_PORT = config['ASSIST_RESULT_PORT'] || ENV['ASSIST_RESULT_PORT'] || 8102 -- 2.23.0

1 0

Re: [PATCH v2 lkp-tests 2/2] lib: fix 409 conflict
by Wu Fengguang 14 Jan '21

14 Jan '21

On Wed, Jan 13, 2021 at 09:19:47PM +0800, Li Ping wrote: >[why] >find a -type d >a >a/b >a/c >find a -type d -exec curl -sSf -X MKCOL "$http_url/{}" \; >==> curl: (22) The requested URL returned error: 409 Conflict >reason: upload an empty folder causes this error. I suspect 409 happens because server side already have that dir. It's like you run mkdir a => success mkdir a => 409 But will the server side auto create new subdirs if they do not already exist? Probably the below find will fail. Please experiment with deeper dirs, like a/b/c/file Please be very careful and make sure you understand all the error messages, all situations and the root cause before going ahead changing something. >[how] >find a -type f -exec curl -sSf -X MKCOL "$http_url/{}" \; >a/b >a/c >All the files would be found, there is no need to upload empty folder. Thanks, Fengguang

1 0

Re: [PATCH v2 lkp-tests 1/2] tests: modify api-avx2neon
by Bai Jing 14 Jan '21

14 Jan '21

On Wed, Jan 13, 2021 at 09:19:46PM +0800, Li Ping wrote: >Signed-off-by: Li Ping <1477412247(a)qq.com> >--- > distro/depends/api-avx2neon | 4 ++++ > tests/api-avx2neon | 33 ++++++++++++++++++++++----------- > 2 files changed, 26 insertions(+), 11 deletions(-) > create mode 100644 distro/depends/api-avx2neon > >diff --git a/distro/depends/api-avx2neon b/distro/depends/api-avx2neon >new file mode 100644 >index 000000000..d82d2f65e >--- /dev/null >+++ b/distro/depends/api-avx2neon >@@ -0,0 +1,4 @@ >+git >+make >+gcc >+g++ >diff --git a/tests/api-avx2neon b/tests/api-avx2neon >index 9a64f89bf..47421021c 100755 >--- a/tests/api-avx2neon >+++ b/tests/api-avx2neon >@@ -1,18 +1,29 @@ > #!/bin/bash > >+SCRIPT_DIR=$(dirname $(realpath $0)) >+cd "$SCRIPT_DIR" >+: "${git_url:=https://github.com/Siguyi/AvxToNeon}" > >-script=$(dirname $(realpath $0)) >-cd $script >+build() { >+ git clone "$git_url" >+ cd AvxToNeon/tests || exit > >-yum install -y git >-yum install -y make >+ if [ "${arch}" == "aarch64" ]; then >+ make >/dev/null 2>&1 >+ elif [ "${arch}" == "x86_64" ]; then >+ make ARCH=x86 >/dev/null 2>&1 >+ fi >+} > >-git clone https://github.com/Siguyi/AvxToNeon >+run() { >+ build >+ echo "api-avx2neon running" >+ ./test >+ echo "api-avx2neon finished" > >-cd AvxToNeon >-cd tests >-make >+ output=$(find / -name "output") >+ fail=$(grep 'AVX2NEONTest Complete' $output |awk '{print $8}') ^^ need space Thanks, Baijing >+ [ 0 -eq "$fail" ] || exit 1 >+} > >-echo "api-avx2neon running" >-./test >-echo "api-avx2neon finished" >+run >-- >2.23.0 >

1 0

Re: [PATCH v2 lkp-tests 1/2] tests: modify api-avx2neon
by Wang Yong 14 Jan '21

14 Jan '21

On Wed, Jan 13, 2021 at 09:19:46PM +0800, Li Ping wrote: >Signed-off-by: Li Ping <1477412247(a)qq.com> >+ if [ "${arch}" == "aarch64" ]; then >+ make >/dev/null 2>&1 >+ elif [ "${arch}" == "x86_64" ]; then >+ make ARCH=x86 >/dev/null 2>&1 >+ fi avx2neon use cross compile to build x86 version, not only require arch=x86_64 ref: Makefile Thanks, Yong

1 0

Re: [PATCH v2 lkp-tests 1/2] tests: modify api-avx2neon
by Wu Fengguang 14 Jan '21

14 Jan '21

On Wed, Jan 13, 2021 at 09:19:46PM +0800, Li Ping wrote: >Signed-off-by: Li Ping <1477412247(a)qq.com> >--- > distro/depends/api-avx2neon | 4 ++++ > tests/api-avx2neon | 33 ++++++++++++++++++++++----------- > 2 files changed, 26 insertions(+), 11 deletions(-) > create mode 100644 distro/depends/api-avx2neon > >diff --git a/distro/depends/api-avx2neon b/distro/depends/api-avx2neon >new file mode 100644 >index 000000000..d82d2f65e >--- /dev/null >+++ b/distro/depends/api-avx2neon >@@ -0,0 +1,4 @@ >+git >+make >+gcc >+g++ >diff --git a/tests/api-avx2neon b/tests/api-avx2neon >index 9a64f89bf..47421021c 100755 >--- a/tests/api-avx2neon >+++ b/tests/api-avx2neon >@@ -1,18 +1,29 @@ > #!/bin/bash > >+SCRIPT_DIR=$(dirname $(realpath $0)) >+cd "$SCRIPT_DIR" >+: "${git_url:=https://github.com/Siguyi/AvxToNeon}" > >-script=$(dirname $(realpath $0)) >-cd $script >+build() { >+ git clone "$git_url" Better to add "|| exit" here too. >+ cd AvxToNeon/tests || exit > >-yum install -y git >-yum install -y make >+ if [ "${arch}" == "aarch64" ]; then >+ make >/dev/null 2>&1 Why quiet all output and stderr? >+ elif [ "${arch}" == "x86_64" ]; then >+ make ARCH=x86 >/dev/null 2>&1 >+ fi >+} Thanks, Fengguang >-git clone https://github.com/Siguyi/AvxToNeon >+run() { >+ build >+ echo "api-avx2neon running" >+ ./test What if ./test exit 1? >+ echo "api-avx2neon finished" > >-cd AvxToNeon >-cd tests >-make >+ output=$(find / -name "output") output= => output_file= What if find returns nothing? >+ fail=$(grep 'AVX2NEONTest Complete' $output |awk '{print $8}') Please give example line that you want to grep/awk. >+ [ 0 -eq "$fail" ] || exit 1 That logic reads awkward. fail => complete? >+} > >-echo "api-avx2neon running" >-./test >-echo "api-avx2neon finished" >+run >-- >2.23.0 >

1 0

[PATCH v3 lkp-tests] tests/*: add makepkg -s operation in archlinux
by Wang Yong 13 Jan '21

13 Jan '21

use makepkg -s to auto install depends in archlinux ref: https://git.archlinux.org/pacman.git/plain/scripts/makepkg.sh.in Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 7 ++++++- tests/cci-makepkg | 12 +++++++++--- 2 files changed, 15 insertions(+), 4 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 31762384d..554cd56e5 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -92,7 +92,9 @@ create_cgz_name() cgz_path="$PKG_MNT/${pack_to}/${pkgname}" cgz_name="${cgz_path}/${upstream_commit}.cgz" + pkg_args="-A --check --skippgpcheck" + [ "${os}" == "archlinux" ] && pkg_args="${pkg_args} -s --needed --noconfirm" [ -n "$git_tag" ] && { cgz_name="${cgz_path}/${git_tag}-${pkgrel}.cgz" @@ -153,7 +155,10 @@ build_source_pkg() create_cgz_name get_config - PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ + local mark="true" + [ "${os}" == "archlinux" ] && local mark="pacman" + + PACMAN="${mark}" BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf local build_state="$?" diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 7d1e5cb15..ecbf72134 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -133,10 +133,16 @@ update_softlink() build_source_pkg() { - local makepkg="$LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf" - echo "$makepkg" + local mark="true" + local pkg_args="-A --check --skippgpcheck" + [ "${DISTRO}" == "archlinux" ] && { + local mark="pacman" + local pkg_args="${pkg_args} -s --needed --noconfirm" + } + local makepkg="$LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf" + echo "${makepkg}" - PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + PACMAN="${mark}" BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg [ "$?" == 0 ] || exit 1 update_softlink -- 2.23.0

1 0

[PATCH compass-ci 3/3] providers/qemu: check kernel
by Liu Yinsi 13 Jan '21

13 Jan '21

test: /srv/result/host-info/2021-01-13/vm-2p8g/debian-sid-x86_64/z9.351229 /srv/result/host-info/2021-01-13/vm-2p8g/debian-sid-riscv64/z9.351202 /srv/result/host-info/2021-01-13/vm-2p8g/debian-sid-aarch64/z9.351187 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 8979da6..51357a6 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -57,6 +57,9 @@ check_option_value() exit } + # debian has both qemu-system-x86_64 and qemu-system-riscv64 command + [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 + [ -n "$initrds" ] || exit } @@ -104,9 +107,6 @@ set_qemu() do command -v "$qemu" > /dev/null && break done - - # debian has both qemu-system-x86_64 and qemu-system-riscv64 command - [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 } print_message() @@ -195,9 +195,9 @@ check_logfile write_logfile parse_ipxe_script -check_option_value set_options +check_option_value print_message -- 2.23.0

2 2

[PATCH v2 compass-ci 3/3] providers/qemu: check kernel
by Liu Yinsi 13 Jan '21

13 Jan '21

[why] check $kernel includes "riscv64" to set qemu command, this logic is suitable in check_option_value(), not in set_qemu, so move order. test: /srv/result/host-info/2021-01-13/vm-2p8g/debian-sid-x86_64/z9.351229 /srv/result/host-info/2021-01-13/vm-2p8g/debian-sid-riscv64/z9.351202 /srv/result/host-info/2021-01-13/vm-2p8g/debian-sid-aarch64/z9.351187 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 8979da6..51357a6 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -57,6 +57,9 @@ check_option_value() exit } + # debian has both qemu-system-x86_64 and qemu-system-riscv64 command + [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 + [ -n "$initrds" ] || exit } @@ -104,9 +107,6 @@ set_qemu() do command -v "$qemu" > /dev/null && break done - - # debian has both qemu-system-x86_64 and qemu-system-riscv64 command - [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 } print_message() @@ -195,9 +195,9 @@ check_logfile write_logfile parse_ipxe_script -check_option_value set_options +check_option_value print_message -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] providers/qemu: check qemu command exist
by Liu Yinsi 13 Jan '21

13 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 9d15f7d..8979da6 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -102,7 +102,7 @@ set_qemu() for qemu in "${qemus[@]}" do - [ -n "$(command -v ${qemu})" ] && break + command -v "$qemu" > /dev/null && break done # debian has both qemu-system-x86_64 and qemu-system-riscv64 command -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] providers/qemu: set qemu command
by Liu Yinsi 13 Jan '21

13 Jan '21

before: set qemu command by iterate qemus after: set qemu command by arch Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 39a1dc1..9d15f7d 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -96,10 +96,8 @@ set_netdev() set_qemu() { qemus=( - qemu-system-aarch64 + qemu-system-$(arch) qemu-kvm - qemu-system-x86_64 - qemu-system-riscv64 ) for qemu in "${qemus[@]}" -- 2.23.0

1 0

[PATCH compass-ci] container/add-software: redirect redundant logs when clean dirs
by Xu Xijian 13 Jan '21

13 Jan '21

redundant logs as follows: rm: refusing to remove '.' or '..' directory: skipping 'srv/.' rm: refusing to remove '.' or '..' directory: skipping 'srv/..' rm: refusing to remove '.' or '..' directory: skipping 'opt/.' rm: refusing to remove '.' or '..' directory: skipping 'opt/..' rm: refusing to remove '.' or '..' directory: skipping 'mnt/.' rm: refusing to remove '.' or '..' directory: skipping 'mnt/..' rm: refusing to remove '.' or '..' directory: skipping 'media/.' rm: refusing to remove '.' or '..' directory: skipping 'media/..' rm: refusing to remove '.' or '..' directory: skipping 'home/.' rm: refusing to remove '.' or '..' directory: skipping 'home/..' rm: refusing to remove '.' or '..' directory: skipping 'dev/.' rm: refusing to remove '.' or '..' directory: skipping 'dev/..' rm: refusing to remove '.' or '..' directory: skipping 'proc/.' rm: refusing to remove '.' or '..' directory: skipping 'proc/..' rm: refusing to remove '.' or '..' directory: skipping 'run/.' rm: refusing to remove '.' or '..' directory: skipping 'run/..' rm: refusing to remove '.' or '..' directory: skipping 'sys/.' rm: refusing to remove '.' or '..' directory: skipping 'sys/..' Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/add-software/run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/add-software/run b/container/add-software/run index 85f038b..840fb2c 100755 --- a/container/add-software/run +++ b/container/add-software/run @@ -43,7 +43,7 @@ install_cmds=( bash -c " yum install -y --skip-broken --installroot=/mnt/root $packages cd /mnt/root - rm -rf srv/{*,.*} opt/{*,.*} mnt/{*,.*} media/{*,.*} home/{*,.*} lost+found/{*,.*} dev/{*,.*} proc/{*,.*} run/{*,.*} sys/{*,.*} + rm -rf srv/{*,.*} opt/{*,.*} mnt/{*,.*} media/{*,.*} home/{*,.*} lost+found/{*,.*} dev/{*,.*} proc/{*,.*} run/{*,.*} sys/{*,.*} > /dev/null 2>&1 " ) -- 2.23.0

1 0

[PATCH compass-ci 2/3] providers/qemu: check qemu command exist
by Liu Yinsi 13 Jan '21

13 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 9d15f7d..8979da6 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -102,7 +102,7 @@ set_qemu() for qemu in "${qemus[@]}" do - [ -n "$(command -v ${qemu})" ] && break + command -v "$qemu" > /dev/null && break done # debian has both qemu-system-x86_64 and qemu-system-riscv64 command -- 2.23.0

1 0

[PATCH compass-ci 1/3] providers/qemu: set qemu command
by Liu Yinsi 13 Jan '21

13 Jan '21

before: set qemu command by iterate qemus after: set qemu command by arch Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 39a1dc1..9d15f7d 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -96,10 +96,8 @@ set_netdev() set_qemu() { qemus=( - qemu-system-aarch64 + qemu-system-$(arch) qemu-kvm - qemu-system-x86_64 - qemu-system-riscv64 ) for qemu in "${qemus[@]}" -- 2.23.0

1 0

[PATCH compass-ci] provider/qemu/*.sh: add param to configure the RTC time of vm
by Yu Chuan 13 Jan '21

13 Jan '21

[Why] The RTC time of the virtual machine is not consistent with the time of the host physical machine. [Info] Before: [root@localhost ~]# timedatectl Local time: Wed 2021-01-13 15:55:01 CST Universal time: Wed 2021-01-13 07:55:01 UTC RTC time: Wed 2021-01-13 07:55:01 Time zone: Asia/Shanghai (CST, +0800) NTP enabled: yes NTP synchronized: yes RTC in local TZ: no DST active: n/a After: [root@localhost ~]# timedatectl Local time: Wed 2021-01-13 15:55:55 CST Universal time: Wed 2021-01-13 07:55:55 UTC RTC time: Wed 2021-01-13 15:55:55 Time zone: Asia/Shanghai (CST, +0800) NTP enabled: yes NTP synchronized: yes RTC in local TZ: no DST active: n/a Signed-off-by: Yu Chuan <13186087857(a)163.com> --- providers/qemu/kvm.sh | 1 + providers/qemu/pxe.sh | 1 + 2 files changed, 2 insertions(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 39a1dc19f112..064fd711c298 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -130,6 +130,7 @@ public_option() -initrd $initrd -smp $nr_cpu -m $memory + -rtc base=localtime -k en-us -no-reboot -nographic diff --git a/providers/qemu/pxe.sh b/providers/qemu/pxe.sh index 8cf198daa639..fa688331964a 100755 --- a/providers/qemu/pxe.sh +++ b/providers/qemu/pxe.sh @@ -28,6 +28,7 @@ kvm=( -cpu Kunpeng-920 -device virtio-gpu-pci -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd + -rtc base=localtime -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -no-reboot -- 2.23.0

1 0

[PATCH v2 compass-ci] tool/es-jobs: a new tool for summarizing jobs information
by Wu Zhende 13 Jan '21

13 Jan '21

Use es-jobs to summarizes information about a group of related jobs. [Example] es-jobs submit_id=xxx -f 'job_state,dmesg.timestamp:last' -r '-1' -s 'on_fail' means: 1. query the job whose submit_id is xxx 2. I want to get job_state,dmesg.timestamp:last 3. don't refine jobs 4. -s 'on_fail' to get data in stats which contain 'on_fail' [Output] { "job_state": [ "crystal.605716.failed" ], "dmesg.timestamp:last": [ 657.181408 ], "stats_filter_result": { "crystal.605716.openeuler_docker.\u001b[0mwordpress_build_on_fail": 1 }, "stats.count": { "stats.has_error": 1, "stats.has_error_jobs": [ "crystal.605716" ] } } Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 139 +++++++++++++++++++++++++++++++++++++++++++++++++ sbin/es-jobs | 58 +++++++++++++++++++++ 2 files changed, 197 insertions(+) create mode 100644 lib/es_jobs.rb create mode 100755 sbin/es-jobs diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb new file mode 100644 index 0000000..a90b1ba --- /dev/null +++ b/lib/es_jobs.rb @@ -0,0 +1,139 @@ +# SPDX-License-Identifier: GPL-2.0-only +# frozen_string_literal: true + +LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' + +require "#{LKP_SRC}/lib/stats" +require_relative './es_query' + +# deal jobs search from es +class ESJobs + def initialize(es_query, my_refine = [], fields = [], stats_filter = []) + @es_query = es_query + @es = ESQuery.new(ES_HOST, ES_PORT) + @refine = my_refine + @fields = fields + @stats_filter = stats_filter + @stats_filter_result = {} + @refine_jobs = [] + @jobs = {} + @stats_level = { + 0 => 'stats.success', + 1 => 'stats.unknown', + 2 => 'stats.warning', + 3 => 'stats.has_error' + } + set_defaults + deal_jobs + end + + def set_defaults + query_result = @es.multi_field_query(@es_query) + query_result['hits']['hits'].each do |job| + @jobs[job['_id']] = job['_source'] + end + + @stats = { + 'stats.count' => Hash.new(0), + 'stats.sum' => Hash.new(0), + 'stats.avg' => Hash.new(0) + } + @result = {} + @fields.each do |field| + @result[field] = [] + end + end + + def add_result_fields(job, level) + return unless @refine.include?(level) || @refine.include?(-1) + + @refine_jobs << job['id'] + @fields.each do |field| + value = job[field] + if value + value = job['id'] + '.' + value if field == 'job_state' + @result[field] << value + end + + next unless job['stats'] + + @result[field] << job['stats'][field] if job['stats'][field] + end + end + + def deal_jobs + stats_count = Hash.new(0) + stats_jobs = {} + + @jobs.each do |job_id, job| + level = deal_stats(job) + add_result_fields(job, level) + + stat_key = @stats_level[level] + stat_jobs_key = stat_key + '_jobs' + + stats_count[stat_key] += 1 + stats_jobs[stat_jobs_key] ||= [] + stats_jobs[stat_jobs_key] << job_id + end + + @stats['stats.count'].merge!(stats_count) + @stats['stats.count'].merge!(stats_jobs) + end + + def deal_stats(job, level = 0) + return 1 unless job['stats'] + + job['stats'].each do |key, value| + match_stats_filter(key, value, job['id']) + calculate_stat(key, value) + level = get_stat_level(key, level) + end + return level + end + + def match_stats_filter(key, value, job_id) + @stats_filter.each do |filter| + next unless key.include?(filter) + + key = job_id + '.' + key + @stats_filter_result[key] = value + + break + end + end + + def calculate_stat(key, value) + if function_stat?(key) + return unless @fields.include?('stats.sum') + + @stats['stats.sum'][key] += value + else + return unless @fields.include?('stats.avg') + + @stats['stats.avg'][key] = (@stats['stats.avg'][key] + value) / 2 + end + end + + def get_stat_level(stat, level) + return level if level >= 3 + return 3 if stat.match(/error|fail/i) + return 2 if stat.match(/warn/i) + + return 0 + end + + def output + result = { + 'stats.count' => @stats['stats.count'] + } + + @stats.each do |key, value| + result[key] = value if @fields.include?(key) + end + + @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? + @result.merge!(result) + puts JSON.pretty_generate(@result) + end +end diff --git a/sbin/es-jobs b/sbin/es-jobs new file mode 100755 index 0000000..25cbe9c --- /dev/null +++ b/sbin/es-jobs @@ -0,0 +1,58 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'optparse' +require 'json' + +require_relative '../lib/es_jobs' + +def parse_argv + items = {} + ARGV.each do |item| + key, value = item.split('=') + if key && value + value_list = value.split(',') + items[key] = value_list.length > 1 ? value_list : value + end + end + items +end +opt_refine = [-1] +opt_fields = [] +opt_stats_filter = [] +opt_parser = OptionParser.new do |opts| + opts.banner = 'Usage: es-jobs [options] search_key1=val1[,val2..] ..' + + opts.separator 'search_key can be id, suite, os, etc.' + opts.separator 'how to use -r' + opts.separator 'like es-jobs submit_id=xxx -r "-1,0,1,2,3"' + opts.separator '-1 means not refine' + opts.separator '{ + -1 => null, + 0 => stats.succes, + 1 => stats.unknow, + 2 => stats.warning, + 3 => stats.has_error +}' + + opts.on('-r fields', '--refine fields', 'refine jobs') do |fields| + opt_refine = fields.split(',').map(&:to_i) + end + + opts.on('-f fields', '--fields fields', 'fields you want to see') do |fields| + opt_fields = fields.split(',') + end + + opts.on('-s fields', '--stats-filter fields', 'return data contains fields') do |fields| + opt_stats_filter = fields.split(',') + end +end + +opt_parser.parse!(ARGV) +items = parse_argv +raise 'Please enter a query' if items.empty? + +es_jobs = ESJobs.new(items, opt_refine, opt_fields, opt_stats_filter) +es_jobs.output -- 2.23.0

2 1

[PATCH v8 compass-ci] providers/qemu: adapt for x86_64 and riscv64
by Liu Yinsi 13 Jan '21

13 Jan '21

before: support kvm for aarch64 after: support kvm for aarch64, x86_64, riscv64 test: /srv/result/host-info/2021-01-12/vm-2p8g/debian-sid-aarch64/z9.318111 /srv/result/host-info/2021-01-12/vm-2p8g/debian-sid-x86_64/z9.318142 /srv/result/host-info/2021-01-12/vm-2p8g/debian-sid-riscv64/z9.319551 [how] put public options like $kernel, $initrd, $smp together, set individual options like $machine, $bios, $nic according to different qemu command, then merge all options to run qemu. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 121 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 106 insertions(+), 15 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 577bca1..a61acab 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -18,12 +18,6 @@ check_logfile() } } -qemu_command() -{ - qemu=qemu-system-aarch64 - command -v $qemu >/dev/null || qemu=qemu-kvm -} - write_logfile() { ipxe_script=ipxe_script @@ -66,12 +60,57 @@ check_option_value() [ -n "$initrds" ] || exit } -get_initrd() +set_initrd() { initrd=initrd cat $initrds > $initrd } +set_bios() +{ + bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd + [ -f "$bios" ] || bios=/usr/share/ovmf/OVMF.fd +} + +set_helper() +{ + helper=/usr/libexec/qemu-bridge-helper + [ -f "$helper" ] || helper=/usr/lib/qemu/qemu-bridge-helper +} + +set_nic() +{ + nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" +} + +set_device() +{ + device="virtio-net-device,netdev=net0,mac=${mac}" +} + +set_netdev() +{ + netdev="bridge,br=br0,id=net0,helper=${helper}" +} + +set_qemu() +{ + qemus=( + qemu-system-aarch64 + qemu-kvm + qemu-system-x86_64 + qemu-system-riscv64 + ) + + for qemu in "${qemus[@]}" + do + [ -n "$(command -v ${qemu})" ] && break + done + + # debian has both qemu-system-x86_64 and qemu-system-riscv64 command + [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 +} + print_message() { echo $SCHED_PORT @@ -83,36 +122,88 @@ print_message() sleep 5 } -run_qemu() +public_option() { kvm=( $qemu - -machine virt-4.0,accel=kvm,gic-version=3 -kernel $kernel -initrd $initrd -smp $nr_cpu -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -no-reboot -nographic -serial file:${log_file} -monitor null ) +} + +individual_option() +{ + case "$qemu" in + qemu-system-aarch64) + arch_option=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + ;; + qemu-kvm) + [ "$(arch)" == "aarch64" ] && arch_option=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + [ "$(arch)" == "x86_64" ] && arch_option=( + -bios $bios + -nic $nic + ) + ;; + qemu-system-x86_64) + arch_option=( + -bios $bios + -nic $nic + ) + ;; + qemu-system-riscv64) + arch_option=( + -machine virt + -device $device + -netdev $netdev + ) + ;; + esac +} - "${kvm[@]}" --append "${append}" +run_qemu() +{ + "${kvm[@]}" "${arch_option[@]}" --append "${append}" +} + +set_options() +{ + set_initrd + set_bios + set_helper + set_nic + set_device + set_netdev + set_qemu } check_logfile write_logfile -qemu_command parse_ipxe_script check_option_value -get_initrd + +set_options print_message +public_option +individual_option + run_qemu -- 2.23.0

2 2

[PATCH v2 lkp-tests] tests/*: add makepkg -s operation in archlinux
by Wang Yong 13 Jan '21

13 Jan '21

use makepkg -s to auto install depends in archlinux ref: https://git.archlinux.org/pacman.git/plain/scripts/makepkg.sh.in Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 11 +++++++++-- tests/cci-makepkg | 14 +++++++++++--- 2 files changed, 20 insertions(+), 5 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 31762384d..db4007b20 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -92,7 +92,9 @@ create_cgz_name() cgz_path="$PKG_MNT/${pack_to}/${pkgname}" cgz_name="${cgz_path}/${upstream_commit}.cgz" + pkg_args="-A --check --skippgpcheck" + [ "${os}" == "archlinux" ] && pkg_args="${pkg_args} -s --needed --noconfirm" [ -n "$git_tag" ] && { cgz_name="${cgz_path}/${git_tag}-${pkgrel}.cgz" @@ -153,8 +155,13 @@ build_source_pkg() create_cgz_name get_config - PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ - $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf + if [ "${os}" == "archlinux" ]; then + BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ + $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf + else + PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ + $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf + fi local build_state="$?" diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 7d1e5cb15..e4bab41eb 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -133,10 +133,18 @@ update_softlink() build_source_pkg() { - local makepkg="$LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf" - echo "$makepkg" + if [ "${DISTRO}" == "archlinux" ]; then + local makepkg="$LKP_SRC/sbin/makepkg -A -s --needed --noconfirm --skippgpcheck --config $LKP_SRC/etc/makepkg.conf" + echo "$makepkg" - PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + + else + local makepkg="$LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf" + echo "$makepkg" + + PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + fi [ "$?" == 0 ] || exit 1 update_softlink -- 2.23.0

2 2

[PATCH v3 lkp-tests] tests/send_email: refactor send_email
by Wei Jihui 13 Jan '21

13 Jan '21

It is convenient for other tasks reuse to send email [usage] job.yaml test_task: on_fail: send_email: job_failed sleep: 1h This will send email to user when job is failed. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/email.sh | 26 ++++++++++++++++++++++++++ tests/email | 41 ----------------------------------------- tests/send_email | 20 ++++++++++++++++++++ 3 files changed, 46 insertions(+), 41 deletions(-) create mode 100755 lib/email.sh delete mode 100755 tests/email create mode 100755 tests/send_email diff --git a/lib/email.sh b/lib/email.sh new file mode 100755 index 00000000..d7c2c44a --- /dev/null +++ b/lib/email.sh @@ -0,0 +1,26 @@ +#!/bin/sh + +job_failed_content() +{ + email_content="To: $my_email +Subject: [NOTIFY Compass-CI] Test job failed: $id + +Dear $my_name: + + Sorry to inform you that your test job is failed, you can click the following link to view details. + http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT:-11300}$result_root + +Regards +Compass-CI +" +} + +send_email() +{ + local subject=$1 + local email_content + + ${subject}_content + + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$email_content" +} diff --git a/tests/email b/tests/email deleted file mode 100755 index a5f87d42..00000000 --- a/tests/email +++ /dev/null @@ -1,41 +0,0 @@ -#!/bin/sh -# - my_email -# - info - -: ${info:=failure} - -SCRIPT_DIR=$(dirname $(realpath $0)) -PROJECT_DIR=$(dirname $SCRIPT_DIR) - -. $PROJECT_DIR/lib/http.sh -. $PROJECT_DIR/lib/env.sh - -data_failure() -{ - data="To: $my_email -Subject: [NOTIFY Compass-ci] Test job failed - -Dear $my_username: - - Sorry to inform you that your test job is failed. you can click the following link to view details. - http://api.compass-ci.openeuler.org:$SRV_HTTP_PORT/$result_root - -Regards -Compass-Ci -" -} - -compose_email() -{ - [ "$info" == "failure" ] && data_failure -} - -run_email() -{ - [ -n "$my_email" ] || return 0 - my_username=$(echo "$my_email" | awk -F '@' '{print $1}') - compose_email - curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$data" -} - -run_email diff --git a/tests/send_email b/tests/send_email new file mode 100755 index 00000000..0132c3c8 --- /dev/null +++ b/tests/send_email @@ -0,0 +1,20 @@ +#!/bin/sh +# - subject +# my_email + +SCRIPT_DIR=$(dirname $(realpath $0)) +PROJECT_DIR=$(dirname $SCRIPT_DIR) + +. $PROJECT_DIR/lib/email.sh + +[ -n "$my_email" ] || { + echo "Can not get my_email, send email failed!" + exit 0 +} + +[ -n "$subject" ] || { + echo "Can not get email subject, send email failed!" + exit 0 +} + +send_email "$subject" -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] reset-tbox to reboot machine testbox
by Zhang Dewan 13 Jan '21

13 Jan '21

[why] sumbit a job to reboot machine testbox is better. [how] cp ~/lkp-test/jobs/reset-tbox-yaml . submit reset-tbox.yaml host_name="full-name-of-machine-testbox" Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 10 ++++++++++ tests/reset-tbox | 17 +++++++++++++++++ tests/reset-tbox.yaml | 12 ++++++++++++ 3 files changed, 39 insertions(+) create mode 100644 jobs/reset-tbox.yaml create mode 100755 tests/reset-tbox create mode 100644 tests/reset-tbox.yaml diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml new file mode 100644 index 00000000..9c3f4f01 --- /dev/null +++ b/jobs/reset-tbox.yaml @@ -0,0 +1,10 @@ +suite: reset-tbox +testcase: reset-tbox + +testbox: dc-8g +os_mount: container +docker_image: 172.168.131.2:5001/reset-tbox +queue: reset + +host_name: +reset-tbox: diff --git a/tests/reset-tbox b/tests/reset-tbox new file mode 100755 index 00000000..b0f645d7 --- /dev/null +++ b/tests/reset-tbox @@ -0,0 +1,17 @@ +#!/bin/bash +# host_name + +source "/etc/lab/.account.info" +hosts_file=/etc/lab/hosts_file + +# require $host_name +host_ibmc_ip=$(grep -w $host_name $hosts_file | awk '{print $1}') + +if [[ "$host_ibmc_ip" =~ ^[0-9]+.* ]]; then + echo ">>> exec ipmitool commamd!" + ipmitool -I lanplus -H $host_ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset + [ $? -eq 0 ] && echo "succeed!" +else + echo "specify host cannot found0!" + exit +fi diff --git a/tests/reset-tbox.yaml b/tests/reset-tbox.yaml new file mode 100644 index 00000000..f557085f --- /dev/null +++ b/tests/reset-tbox.yaml @@ -0,0 +1,12 @@ +short_description: > + To reboot machine testbox like taishan200-2280-2s64p-256g--aN etc. + +description: | + reset-tbox executed by a special docker which running on ibmc-mgnt, where you can access all server's iBMC-os by ipmitool, and then reboot the host. + +homepage: + +parameters: + host_name: see it on the first line about 'latbox aN' + +results: -- 2.23.0

1 0

[PATCH compass-ci] sparrow/0-package: install libvirt
by Xiao Shenwei 13 Jan '21

13 Jan '21

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sparrow/0-package/common | 1 + sparrow/0-package/os/openEuler | 1 + 2 files changed, 2 insertions(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 4a9da60..2d7c9e3 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -51,3 +51,4 @@ sed '/^Environment=/a\TimeoutSec=0' /usr/lib/systemd/system/docker.service systemctl daemon-reload systemctl restart docker +systemctl start libvirtd diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index a63964f..37c74d0 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -21,6 +21,7 @@ pkgs=( busybox iproute hostname + libvirt ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code: set default values
by Liu Yinsi 13 Jan '21

13 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f2366c2..ca91da5 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -20,6 +20,8 @@ server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} : ${ASSIST_RESULT_PORT:=8102} : ${RESULT_WEBDAV_PORT:=3080} +: ${ES_HOST:=$server_ip} +: ${ES_PORT:=9200} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF @@ -45,6 +47,9 @@ LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS MAILDIR: /srv/cci/Maildir/.compass-ci ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT +lab: nolab +ES_HOST: $ES_HOST +ES_PORT: $ES_PORT EOF mkdir -p /etc/compass-ci/account -- 2.23.0

1 0

[PATCH lkp-tests] tests/send_email: refactor send_email
by Wei Jihui 13 Jan '21

13 Jan '21

It is convenient for other tasks reuse to send email [usage] job.yaml test_task: on_fail: send_email: job_failed sleep: 1h This will send email to user when job is failed. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/email.sh | 26 ++++++++++++++++++++++++++ tests/email | 41 ----------------------------------------- tests/send_email | 21 +++++++++++++++++++++ 3 files changed, 47 insertions(+), 41 deletions(-) create mode 100755 lib/email.sh delete mode 100755 tests/email create mode 100755 tests/send_email diff --git a/lib/email.sh b/lib/email.sh new file mode 100755 index 00000000..a407ad40 --- /dev/null +++ b/lib/email.sh @@ -0,0 +1,26 @@ +#!/bin/sh + +job_failed_content() +{ + email_content="To: $my_email +Subject: [NOTIFY Compass-CI] Test job failed: $id + +Dear $my_name: + + Sorry to inform you that your test job is failed. you can click the following link to view details. + http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT:-11300}$result_root + +Regards +Compass-CI +" +} + +send_email() +{ + local subject=$1 + local email_content + + ${subject}_content + + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$email_content" +} diff --git a/tests/email b/tests/email deleted file mode 100755 index a5f87d42..00000000 --- a/tests/email +++ /dev/null @@ -1,41 +0,0 @@ -#!/bin/sh -# - my_email -# - info - -: ${info:=failure} - -SCRIPT_DIR=$(dirname $(realpath $0)) -PROJECT_DIR=$(dirname $SCRIPT_DIR) - -. $PROJECT_DIR/lib/http.sh -. $PROJECT_DIR/lib/env.sh - -data_failure() -{ - data="To: $my_email -Subject: [NOTIFY Compass-ci] Test job failed - -Dear $my_username: - - Sorry to inform you that your test job is failed. you can click the following link to view details. - http://api.compass-ci.openeuler.org:$SRV_HTTP_PORT/$result_root - -Regards -Compass-Ci -" -} - -compose_email() -{ - [ "$info" == "failure" ] && data_failure -} - -run_email() -{ - [ -n "$my_email" ] || return 0 - my_username=$(echo "$my_email" | awk -F '@' '{print $1}') - compose_email - curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$data" -} - -run_email diff --git a/tests/send_email b/tests/send_email new file mode 100755 index 00000000..2911d3b8 --- /dev/null +++ b/tests/send_email @@ -0,0 +1,21 @@ +#!/bin/sh +# - subject +# my_email + +SCRIPT_DIR=$(dirname $(realpath $0)) +PROJECT_DIR=$(dirname $SCRIPT_DIR) + +. $PROJECT_DIR/lib/email.sh + +[ -n "$my_email" ] || { + echo "Can not get my_email, send email failed!" + exit 0 +} + +[ -n "$subject" ] || { + echo "Can not get email subject, send email failed!" + exit 0 +} + + +send_email $subject -- 2.23.0

3 2

[PATCH v2 lkp-tests 1/3] stats/crystal/memory--stat.cr: create memory--stat.cr with memory--stat
by Lu Kaiyi 13 Jan '21

13 Jan '21

create memory--stat.cr as same content with memory--stat Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/memory--stat.cr | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 stats/crystal/memory--stat.cr diff --git a/stats/crystal/memory--stat.cr b/stats/crystal/memory--stat.cr new file mode 100644 index 000000000..1a39e4075 --- /dev/null +++ b/stats/crystal/memory--stat.cr @@ -0,0 +1,22 @@ +#!/usr/bin/env ruby + +# time: 123456 +# cgroup: pmbench.1 +# cache 0 + +def parse + cg = 'nocg' + $stdin.each_line do |line| + case line + when /^time:/ + puts line + when /^cgroup: (\S+)/ + cg = $1 + when /(\S+)\s*(\S+)/ + k, v, = line.split + puts "#{cg}.#{k}: #{v}" + end + end +end + +parse -- 2.23.0

2 1

[PATCH v8 compass-ci 2/2] lib/es_query.rb: parse es_result to a Hash result
by Lu Kaiyi 13 Jan '21

13 Jan '21

[why] make es result more easy access. [example] input: es_result = [{"key"=>"build-pkg", "doc_count"=>354526, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>5708}, {"key"=>"finished", "doc_count"=>1033}, {"key"=>"incomplete", "doc_count"=>204}, {"key"=>"submit", "doc_count"=>136}, ... output: result_hash = {"build-pkg"=> {"failed"=>5708, "finished"=>1033, "incomplete"=>204, "submit"=>136, "OOM"=>11, "post_run"=>2}, "cci-depends"=> {"finished"=>1785, "failed"=>675, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 46 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 91aa6ab..928bd68 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -128,7 +128,7 @@ class ESQuery result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] return nil if result.empty? - result + parse_fields(result) end end @@ -216,3 +216,47 @@ def build_aggs_from_fields(fields) aggs_hash['aggs']["all_#{field}"].merge!(sub_aggs) if sub_aggs aggs_hash end + +# input: +# es_result = +# [{"key"=>"build-pkg", +# "doc_count"=>354526, +# "all_job_state"=> +# {"doc_count_error_upper_bound"=>0, +# "sum_other_doc_count"=>0, +# "buckets"=> +# [{"key"=>"failed", "doc_count"=>5708}, +# {"key"=>"finished", "doc_count"=>1033}, +# {"key"=>"incomplete", "doc_count"=>204}, +# {"key"=>"submit", "doc_count"=>136}, +# ... +# +# output: +# result_hash = +# {"build-pkg"=> +# {"failed"=>5708, +# "finished"=>1033, +# "incomplete"=>204, +# "submit"=>136, +# "OOM"=>11, +# "post_run"=>2}, +# "cci-depends"=> +# {"finished"=>1785, +# "failed"=>675, +# ... +def parse_fields(es_result) + result_hash = {} + es_result.each do |result| + key = result['key'] + sub_field = result.keys.detect { |field| field.start_with?('all_') } + + if sub_field + all_field = result[sub_field]['buckets'] + result_hash[key] = parse_fields(all_field) + else + result_hash[key] = result['doc_count'] + end + end + + result_hash +end -- 2.23.0

3 3

[PATCH compass-ci] lib: remove file executable permissions
by Lu Kaiyi 13 Jan '21

13 Jan '21

[why] the files in lib dir don't need executable permissions. [how] chmod a-x assign_account_client.rb chmod a-x build_my_info_client.rb chmod a-x redis_client.rb Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/assign_account_client.rb | 0 lib/build_my_info_client.rb | 0 lib/redis_client.rb | 0 3 files changed, 0 insertions(+), 0 deletions(-) mode change 100755 => 100644 lib/assign_account_client.rb mode change 100755 => 100644 lib/build_my_info_client.rb mode change 100755 => 100644 lib/redis_client.rb diff --git a/lib/assign_account_client.rb b/lib/assign_account_client.rb old mode 100755 new mode 100644 diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb old mode 100755 new mode 100644 diff --git a/lib/redis_client.rb b/lib/redis_client.rb old mode 100755 new mode 100644 -- 2.23.0

1 0

[PATCH compass-ci] sbin/compare remove change of failure metric
by Lu Weitao 13 Jan '21

13 Jan '21

[Why] We only display the failure metrics that belong the latest job, aimed to express the latest job exists failure. so the change rate of failure metric is useless like: centos openeuler metric -------------------- ------------------------------ ------------------------------ fails:runs change fails:runs | | | 1:12 -8.3% 0:24 stderr./lkp/lkp/src/lib/env.sh:line#:/proc/sys/kernel/nmi_watchdog:Permission_denied [How] centos openeuler metric -------------------- ------------------------------ ------------------------------ fails:runs fails:runs | | 0:12 20:24 stderr.which:no_time_in(/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/lkp/lkp/src/bin) centos openeuler metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 2.638681e+10 ± 23% -0.1% 2.636343e+10 ± 22% iperf.tcp.receiver.bps 2.638853e+10 ± 23% +0.1% 2.641691e+10 ± 22% iperf.tcp.sender.bps 6.80 ± 141% -50.4% 3.37 ± 223% iperf.time.elapsed_time Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 58 ++++++++--------------------------------- 1 file changed, 11 insertions(+), 47 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index fa578f6..5bb9d26 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -83,15 +83,12 @@ def get_values(value_list, success) end def get_compare_value(base_value_average, value_average, success) - # get compare value(change or reproduction) + # get compare value(change) # - if success - return if base_value_average.zero? + return unless success + return if base_value_average.zero? - (100 * value_average / base_value_average - 100).round(1) - else - (100 * (value_average - base_value_average)).round(1) - end + (100 * value_average / base_value_average - 100).round(1) end def field_changed?(base_values, values, success, field, options) @@ -264,7 +261,7 @@ def get_html_header(matrixes_titles, success) compare_title = CHANGE_STR else common_title = FAILS_RUNS_STR - compare_title = REPRODUCTION_STR + compare_title = '' end header = get_html_index(matrixes_titles) @@ -550,35 +547,6 @@ def format_fails_runs(fails, runs) fails_str + ':' + runs_str end -def get_reproduction_index_str(reproduction, compare_index) - reproduction_str = format('%+.1f%%', reproduction) - reproduction_index = compare_index - reproduction_str.index('.') - if reproduction_index.negative? - reproduction_str = format('%+.1e%%', reproduction).sub('e+0', 'e+').sub('e-0', 'e-') - reproduction_index = compare_index - reproduction_str.index('e') - 1 - end - return reproduction_index, reproduction_str -end - -def get_reprodcution(reproduction, compare_index) - if reproduction - reproduction_index, reproduction_str = get_reproduction_index_str(reproduction, compare_index) - space_str = ' ' * (SUB_SHORT_COLUMN_WIDTH - reproduction_str.length) - reproduction_str = space_str.insert(reproduction_index, reproduction_str) - else - reproduction_str = format("%-#{SUB_SHORT_COLUMN_WIDTH}s", '0') - end - reproduction_str -end - -def format_reproduction(reproduction, theme, compare_index) - color = get_compare_value_color(reproduction, theme) - colorize( - color, - get_reprodcution(reproduction, compare_index) - ) -end - def get_change_index_str(change, compare_index) change_str = format('%+.1f%%', change) change_index = compare_index - change_str.index('.') @@ -693,7 +661,7 @@ def get_title_name(success) compare_title = CHANGE_STR else common_title = FAILS_RUNS_STR - compare_title = REPRODUCTION_STR + compare_title = '' end return common_title, compare_title end @@ -718,14 +686,14 @@ def get_base_matrix_title_symbol(common_index, success) title_symbol end -def get_other_matrixes_title_symbol(_compare_title, matrixes_number, common_index, success) +def get_other_matrixes_title_symbol(compare_title, matrixes_number, common_index, success) title_symbol = ' ' * ( (INTERVAL_WIDTH + COLUMN_WIDTH) * matrixes_number ) start_point = 0 common_symbol = success ? '\\' : '|' - compare_symbol = '|' + compare_symbol = compare_title.empty? ? '' : '|' compare_index = SUB_SHORT_COLUMN_WIDTH / 2 (matrixes_number - 1).times do |_| @@ -786,12 +754,8 @@ def get_success_str(values, index, theme, compare_index) (change_str || '') + stddev_str end -def get_failure_str(values, index, theme, compare_index) - unless index.zero? - reproduction_str = format_reproduction( - values[:reproduction], theme, compare_index - ) - end +def get_failure_str(values, index) + reproduction_str = format("%-#{SUB_SHORT_COLUMN_WIDTH}s", '') unless index.zero? fails_runs_str = format_fails_runs( values[:fails], @@ -810,7 +774,7 @@ def get_values_str(matrixes, success, theme) ) + INTERVAL_BLANK else get_failure_str( - values, index, theme, compare_index + values, index ) + INTERVAL_BLANK end end -- 2.23.0

1 0

[PATCH v8 compass-ci 1/2] lib/es_query.rb: get multi-field aggregation count result
by Lu Kaiyi 13 Jan '21

13 Jan '21

[why] get multi-field aggregation count for overview of report page. [input]: fields, query_items (optional for query_items, default no scope limitation) [example] input: fields = ['suite'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186214}, {"key"=>"cci-depends", "doc_count"=>1230}, {"key"=>"iperf", "doc_count"=>1103}, {"key"=>"spinlock", "doc_count"=>1000}, {"key"=>"cci-makepkg", "doc_count"=>949}, {"key"=>"iperf-walk-os-test", "doc_count"=>474}, {"key"=>"bisect", "doc_count"=>381}, ... input: fields = ['suite', 'job_state'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186227, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3830}, {"key"=>"finished", "doc_count"=>803}, {"key"=>"incomplete", "doc_count"=>196}, {"key"=>"submit", "doc_count"=>136}, ... input: fields = ['os', 'os_version', 'job_state'] query_items = {} output: [{"key"=>"openeuler", "doc_count"=>193287, "all_os_version"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"20.03", "doc_count"=>191789, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3909}, {"key"=>"finished", "doc_count"=>2401}, {"key"=>"incomplete", "doc_count"=>300}, {"key"=>"submit", "doc_count"=>136}, {"key"=>"OOM", "doc_count"=>23}, {"key"=>"post_run", "doc_count"=>1}]}}, {"key"=>"20.03-SP1", "doc_count"=>904, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>641}, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 70 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 88a420f..91aa6ab 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -87,6 +87,49 @@ class ESQuery result end + + # input: + # fields, query_items + # + # fields => fields: Array(keyword) + # eg: + # ['suite', 'job_state'] + # + # query_items => query_items: Hash(keyword, value) + # eg: + # { + # 'os' => 'openeuler', + # ... + # } + # (optional for query_items, default no scope limitation) + # output: + # [{"key"=>"build-pkg", + # "doc_count"=>186175, + # "all_job_state"=> + # {"doc_count_error_upper_bound"=>0, + # "sum_other_doc_count"=>0, + # "buckets"=> + # [{"key"=>"failed", "doc_count"=>3830}, + # {"key"=>"finished", "doc_count"=>803}, + # {"key"=>"incomplete", "doc_count"=>196}, + # ... + def query_fields(fields, query_items = {}) + field1 = fields.first + aggs_hash = build_aggs_from_fields(fields) + query = { + query: { + bool: { + must: build_multi_field_subquery_body(query_items) + } + }, + aggs: aggs_hash['aggs'], + size: 0 + } + result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] + return nil if result.empty? + + result + end end # Range Query Example: @@ -146,3 +189,30 @@ def assign_desc_body(keyword) }] } end + +# input: +# fields = ['os', 'os_version', 'job_state'] +# output: +# aggs_hash = { +# "aggs"=> +# {"all_os"=> +# {"terms"=>{:field=>"os", :size=>1000}, +# "aggs"=> +# {"all_os_version"=> +# {"terms"=>{:field=>"os_version", :size=>1000}, +# "aggs"=> +# {"all_job_state"=>{"terms"=>{:field=>"job_state", :size=>1000}}}}}}} +# } +def build_aggs_from_fields(fields) + aggs_hash = {} + return if fields.empty? + + field = fields.shift + aggs_hash['aggs'] ||= {} + aggs_hash['aggs']["all_#{field}"] = { + 'terms' => { field: field, size: 1000 } + } + sub_aggs = build_aggs_from_fields(fields) + aggs_hash['aggs']["all_#{field}"].merge!(sub_aggs) if sub_aggs + aggs_hash +end -- 2.23.0

1 0

[PATCH v2 lkp-tests 3/3] stats/crystal/memory--stat.cr: fix some errors manually
by Lu Kaiyi 13 Jan '21

13 Jan '21

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/memory--stat.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/stats/crystal/memory--stat.cr b/stats/crystal/memory--stat.cr index da7ac0d86..f45ab5fb1 100644 --- a/stats/crystal/memory--stat.cr +++ b/stats/crystal/memory--stat.cr @@ -6,14 +6,14 @@ def parse cg = "nocg" - $stdin.each_line do |line| + STDIN.each_line do |line| case line when /^time:/ puts line when /^cgroup: (\S+)/ cg = $1 when /(\S+)\s*(\S+)/ - k, v, = line.split + k, v = line.split puts "#{cg}.#{k}: #{v}" end end -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/3] stats/crystal/memory--stat.cr: do first conversion by ruby2crystal tools
by Lu Kaiyi 13 Jan '21

13 Jan '21

use the ruby2crystal tools do first conversion like below: /c/crystal/ruby_crystal_codemod/exe/ruby_crystal_codemod memory--stat.cr /c/lkp-tests/tools/ruby2crystal.sh memory--stat.cr Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/memory--stat.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/stats/crystal/memory--stat.cr b/stats/crystal/memory--stat.cr index 1a39e4075..da7ac0d86 100644 --- a/stats/crystal/memory--stat.cr +++ b/stats/crystal/memory--stat.cr @@ -1,11 +1,11 @@ -#!/usr/bin/env ruby +#!/usr/bin/env crystal # time: 123456 # cgroup: pmbench.1 # cache 0 def parse - cg = 'nocg' + cg = "nocg" $stdin.each_line do |line| case line when /^time:/ -- 2.23.0

1 0

[PATCH compass-ci] doc/send-mail.md: format markdown file
by Wei Jihui 13 Jan '21

13 Jan '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- doc/send-mail.md | 51 +++++++++++++++++++++++------------------------- 1 file changed, 24 insertions(+), 27 deletions(-) diff --git a/doc/send-mail.md b/doc/send-mail.md index f503a08..5af8d17 100644 --- a/doc/send-mail.md +++ b/doc/send-mail.md @@ -1,13 +1,5 @@ - # send mail - - - - - - - ## purpose The service is used to send mails with formatted mail data @@ -23,38 +15,42 @@ The service is used to send mails with formatted mail data ## data format - send_mail_yaml - - data format - data='{ - "subject": "email subject", - "to": "email_to_addr", - "body": "email message" - }' + data='{ + "subject": "email subject", + "to": "email_to_addr", + "body": "email message" + }' or - data=" - subject: email subject - to: email_to_addr - body: email message - " + data=" + subject: email subject + to: email_to_addr + body: email message + " - send_mail_text - - data format - data=" - To: email_to_addr - Subject: email_subject + data=" + To: email_to_addr + Subject: email_subject - mail_msg_line1 - mail_msg_line2 - ... - " + mail_msg_line1 + mail_msg_line2 + ... + " ## usage: - send_mail_yaml + ```shell curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_yaml' -d "$data" + ``` + - send_mail_text + ```shell curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "$data" + ``` ## example + ```ruby data = " To: test_email(a)163.com Subject: test mail 10 @@ -63,3 +59,4 @@ The service is used to send mails with formatted mail data " %x(curl -XPOST 'localhost:49000/send_mail_text' -d "#{data}") + ``` -- 2.23.0

1 0

[PATCH v2 lkp-tests] sbin/submit: modify job_id to job id
by Hu Xuejiao 13 Jan '21

13 Jan '21

[why] We use "es-find (id=xxx)" to search id-information,but when submit, it is (job_id=xxx). so it is recommended that the value be the same as the id=xxx. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbin/submit b/sbin/submit index 5273896b..39cdcf57 100755 --- a/sbin/submit +++ b/sbin/submit @@ -168,10 +168,10 @@ ARGV.each do |jobfile| JSON.parse(messages).each do |msg| if msg['message'].empty? job_ids << msg['job_id'].to_s - puts("submit #{jobfile}, got job_id=#{msg['job_id']}") + puts("submit #{jobfile}, got job id=#{msg['job_id']}") else opt_monitor = false - puts("submit #{jobfile} failed, got job_id=#{msg['job_id']}, error: #{msg['message']}") + puts("submit #{jobfile} failed, got job id=#{msg['job_id']}, error: #{msg['message']}") end end end -- 2.23.0

2 1

[PATCH compass-ci] doc: add a doc about catalogs under /srv/git
by Li Yuanchao 13 Jan '21

13 Jan '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- doc/about_srv_git.md | 182 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 182 insertions(+) create mode 100644 doc/about_srv_git.md diff --git a/doc/about_srv_git.md b/doc/about_srv_git.md new file mode 100644 index 0000000..3374828 --- /dev/null +++ b/doc/about_srv_git.md @@ -0,0 +1,182 @@ + +# ��`/srv/git`�� + +``` +delimiter: + ��git-bisect,��good commit�� + +auto-submit�� + ��job��rabbitmq��git-mirror��job�� + +git-daemon�� + ��git��/srv/git��git clone/git push�� + �� "git clone git://localhost/$project" ��git-daemon�� + +git-mirror�� + ��clone�� + +remote-git�� + ��/srv/git��curl��/srv/git��git�� +``` + + +# /srv/git�� + + +��a-z��git-mirror�� +``` +drwxrwsr-x. 55 lkp committer 4.0K 2020-11-20 03:54 u +drwxrwsr-x. 78 lkp committer 4.0K 2020-12-04 11:24 v +drwxrwsr-x. 75 lkp committer 4.0K 2020-12-04 11:25 i +drwxrwsr-x. 986 lkp committer 36K 2020-12-04 11:37 d +drwxrwsr-x. 1612 lkp committer 60K 2020-12-05 03:50 g +drwxrwsr-x. 827 lkp committer 28K 2020-12-05 04:00 b +drwxrwsr-x. 66 lkp committer 4.0K 2020-12-05 04:05 w +drwxrwsr-x. 32 lkp committer 4.0K 2020-12-05 04:17 y +drwxrwsr-x. 706 lkp committer 28K 2020-12-05 05:49 e +drwxrwsr-x. 837 lkp committer 28K 2020-12-08 22:05 f +drwxrwsr-x. 679 lkp committer 24K 2020-12-10 18:18 k +drwxrwsr-x. 948 lkp committer 36K 2020-12-11 02:31 h +drwxrwsr-x. 384 lkp committer 12K 2020-12-11 03:12 j +drwxrwsr-x. 927 lkp committer 36K 2020-12-15 03:11 n +drwxrwsr-x. 1757 lkp committer 56K 2020-12-15 11:09 l +drwxrwsr-x. 54 lkp committer 4.0K 2020-12-15 14:29 t +drwxrwsr-x. 1018 lkp committer 36K 2020-12-15 15:57 a +drwxrwsr-x. 1344 lkp committer 40K 2020-12-15 22:28 m +drwxrwsr-x. 809 lkp committer 28K 2020-12-16 05:21 o +drwxrwsr-x. 1342 lkp committer 36K 2020-12-21 19:19 c +drwxrwsr-x. 5800 lkp committer 212K 2020-12-22 03:08 p +drwxrwsr-x. 395 lkp committer 12K 2020-12-22 22:16 q +drwxrwsr-x. 1632 lkp committer 64K 2020-12-28 21:25 r +drwxrwsr-x. 445 lkp committer 20K 2020-12-28 23:16 x +drwxrwsr-x. 1796 lkp committer 56K 2021-01-05 09:20 s +drwxrwsr-x. 39 lkp committer 4.0K 2021-01-08 10:48 z +``` + +��lkp:committer��git-mirror�� +``` +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-16 10:57 git.code.sf.net +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-16 11:22 code.reversed.top +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-20 03:56 source.ffmpeg.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-21 02:46 git.kernel.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-22 04:05 gitlab.univ-nantes.fr +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-02 18:14 git.musl-libc.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-05 05:56 gitlab.dkrz.de +drwxrwsr-x. 8 lkp committer 4.0K 2020-12-05 06:17 git.sv.gnu.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-05 06:17 git.ageinghacker.net +drwxrwsr-x. 11 lkp committer 4.0K 2020-12-05 06:39 git.qemu-project.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-05 06:44 git.sigxcpu.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-05 06:56 git.osmocom.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-05 07:38 gitlab.linphone.org +drwxrwsr-x. 8 lkp committer 4.0K 2020-12-05 07:52 gitlab.gnome.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-10 14:40 webrtc.googlesource.com +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-11 03:33 www.kismetwireless.net +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-11 03:34 src.fedoraproject.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-16 05:53 git.tuxfamily.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 17:14 chriso(a)github.com +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 17:55 nedko.arnaudov.name +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 17:59 git.gnome.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 18:01 android.googlesource.com +drwxrwsr-x. 5 lkp committer 4.0K 2020-12-17 18:40 gopkg.in +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 19:53 git.cryptomilk.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 20:05 git.dpkg.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 20:09 kernel.googlesource.com +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-19 14:15 aur(a)archlinux.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-21 05:57 boringssl.googlesource.com +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 04:44 hub.spigotmc.org +drwxrwsr-x. 14 lkp committer 4.0K 2020-12-22 04:49 repo.or.cz +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 04:52 luajit.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 05:02 git.assembla.com +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-22 05:07 git.purrdata.net +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 05:09 brettc(a)github.com +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 05:11 git.teknik.io +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 14:40 code.googlesource.com +drwxrwsr-x. 7 lkp committer 4.0K 2020-12-23 01:50 git.qemu.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-23 01:51 git.merproject.org +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-23 01:52 code.qt.io +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-23 01:57 git.tartarus.org +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-27 22:17 gist.github.com +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-29 00:28 invent.kde.org +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-29 00:55 git.sv.nongnu.org +drwxrwsr-x. 6 lkp committer 4.0K 2020-12-29 02:57 git.savannah.gnu.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-29 02:57 git.xfce.org +drwxrwsr-x. 7 lkp committer 4.0K 2020-12-29 02:58 anongit.freedesktop.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-29 02:59 git.ejiek.com +drwxrwsr-x. 32 lkp committer 4.0K 2020-12-29 03:00 gitlab.com +drwxrwsr-x. 5 lkp committer 4.0K 2020-12-30 22:00 git.torproject.org +drwxrwsr-x. 3 lkp committer 4.0K 2021-01-02 05:58 skia.googlesource.com +drwxrwsr-x. 7 lkp committer 4.0K 2021-01-03 05:57 aur(a)aur4.archlinux.org +drwxrwsr-x. 7 lkp committer 4.0K 2021-01-05 10:56 gitlab.freedesktop.org +drwxrwsr-x. 3 lkp committer 4.0K 2021-01-05 10:59 gitlab.xiph.org +drwxrwsr-x. 3 lkp committer 4.0K 2021-01-05 10:59 git.savannah.nongnu.org +drwxrwsr-x. 4 lkp committer 4.0K 2021-01-05 11:01 git.tukaani.org +drwxrwsr-x. 13 lkp committer 4.0K 2021-01-05 11:04 aur.archlinux.org +drwxrwsr-x. 6 lkp committer 4.0K 2021-01-05 11:07 bitbucket.org +drwxrwsr-x. 1664 lkp committer 40K 2021-01-07 09:08 github.com +drwxrwsr-x. 5 lkp committer 4.0K 2021-01-08 05:03 go.googlesource.com +drwxrwsr-x. 7 lkp committer 4.0K 2021-01-08 05:24 chromium.googlesource.com +drwxrwsr-x. 14 lkp committer 4.0K 2021-01-08 06:00 aur(a)aur.archlinux.org +``` + +��git-mirror�� +``` +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:30 coccinelle +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:45 oec-hardware +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:45 libbpf +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:46 qemu +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:51 smatch +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:51 gtk +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-21 11:02 pixz +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-31 11:26 sparse +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-31 11:27 pahole +drwxrwsr-x. 3 lkp committer 4.0K 2020-09-22 15:10 upstream-repos +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-09 19:31 linux +drwxrwsr-x. 24 lkp committer 4.0K 2020-11-10 16:40 submodules +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-02 10:00 compass-ci +``` + +��pkgbuild��make�� +``` +drwxrwsr-x. 43 lkp team 200K 2020-11-11 15:17 archlinux +drwxrwsr-x. 82 root team 4.0K 2020-12-15 18:12 pkg +drwxrwsr-x. 13 wfg team 4.0K 2020-12-16 12:55 make +``` + +��rpm��spec�� +``` +drwxrwsr-x. 3 root committer 4.0K 2020-12-08 11:15 rpmbuild-pkg +``` + +��clone�� +``` +drwxrwsr-x. 3 zhy committer 4.0K 2020-06-16 11:25 mysql-server +drwxrwsr-x. 3 zhy committer 4.0K 2020-07-10 14:28 atomic +drwxrwsr-x. 3 zhy committer 4.0K 2020-07-10 14:28 spinlock +drwxrwsr-x. 3 zhy committer 4.0K 2020-07-10 14:29 mutex +drwxrwsr-x. 4 zhy committer 4.0K 2020-08-24 11:52 avx2neon +drwxrwsr-x. 5 liping committer 4.0K 2020-08-24 11:52 AvxToNeon +``` + +��compass-ci/lkp-tests/.../upstream-repos�� +``` +drwxrwsr-x. 3 wfg committer 4.0K 2020-09-14 15:26 gitee.com +``` + +�� +``` +drwxrwxrwx. 3 weijihui weijihui 4.0K 2020-07-27 15:26 intrintesttool +``` + +ansible�� +``` +drwxrwsr-x. 17 root committer 4.0K 2020-12-29 10:33 ansible-examples +``` + +�� +``` +drwxrwsr-x. 2 root committer 16K 2020-03-27 10:45 lost+found +drwxrwsr-x. 2 wfg committer 4.0K 2020-07-24 16:58 gzip +drwxrwsr-x. 5 wfg committer 4.0K 2020-08-01 11:15 cci +drwxrwsr-x. 5 wfg committer 4.0K 2020-08-04 16:27 lkp-tests +drwxrwsr-x. 4 wfg committer 4.0K 2020-09-19 14:15 crystal-ci +``` -- 2.23.0

2 1

[PATCH compass-ci] tool/es-jobs: add a new tool
by Wu Zhende 12 Jan '21

12 Jan '21

Use es-jobs to summarizes information about a group of related jobs. [Example] es-jobs submit_id=xxx -f 'id,job_state,dmesg.timestamp:last,stats.sum,stats.avg' -r '0' means: 1.query the job whose submit_id is xxx 2.I want to get id,job_state,dmesg.timestamp:last,stats.sum,stats.avg 3.refine stats.success jobs [Output] { "id": [ "crystal.597090", "crystal.597093" ], "job_state": [ "finished", "finished" ], "dmesg.timestamp:last": [ 75.256233, 48.990905 ], "stats.sum": { "kmsg.timestamp:last": 124.24713799999999, "dmesg.timestamp:last": 124.24713799999999 }, "stats.avg": { "iperf.tcp.receiver.bps": 17008962077.755077, "iperf.tcp.sender.bps": 17036843967.505566, "iperf.udp.bps": 524286.6541701214 }, "stats.count": { "stats.success": 2, "stats.success_jobs": [ "crystal.597090", "crystal.597093" ] } } Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 99 ++++++++++++++++++++++++++++++++++++++++++++++++++ sbin/es-jobs | 53 +++++++++++++++++++++++++++ 2 files changed, 152 insertions(+) create mode 100644 lib/es_jobs.rb create mode 100755 sbin/es-jobs diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb new file mode 100644 index 0000000..78db8f0 --- /dev/null +++ b/lib/es_jobs.rb @@ -0,0 +1,99 @@ +# SPDX-License-Identifier: GPL-2.0-only +# frozen_string_literal: true + +LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' + +require "#{LKP_SRC}/lib/stats" +require_relative './es_query' + +# deal jobs search from es +class ESJobs + def initialize(es_query, my_refine = [-1], fields = []) + @es = ESQuery.new(ES_HOST, ES_PORT) + @refine = my_refine + @fields = fields + @refine_jobs = [] + @query_result = @es.multi_field_query(es_query) + @jobs = {} + @stats_count = Hash.new(0) + @stats_sum = Hash.new(0) + @stats_avg = Hash.new(0) + @stats_level = { + 0 => 'stats.success', + 1 => 'stats.unknow', + 2 => 'stats.warning', + 3 => 'stats.has_error' + } + set_defaults + deal_jobs + end + + def set_defaults + @query_result['hits']['hits'].each do |job| + @jobs[job['_id']] = job['_source'] + end + + @result = {} + @fields.each do |field| + @result[field] = [] + end + end + + def add_result_fields(job, level) + return unless @refine.include?(level) || @refine.include?(-1) + + @refine_jobs << job['id'] + @fields.each do |field| + @result[field] << job[field] if job[field] + + next unless job['stats'] + + @result[field] << job['stats'][field] if job['stats'][field] + end + end + + def deal_jobs + @jobs.each do |job_id, job| + level = deal_stats(job) + add_result_fields(job, level) + + @stats_count[@stats_level[level]] += 1 + stat_key = @stats_level[level] + '_jobs' + @stats_count[stat_key] = [] unless @stats_count.key?(stat_key) + @stats_count[stat_key] << job_id + end + end + + def deal_stats(job, level = 0) + return 1 unless job['stats'] + + job['stats'].each do |key, value| + if function_stat?(key) + @stats_sum[key] += value + else + @stats_avg[key] = (@stats_avg[key] + value) / 2 + end + + level = get_stat_level(key, level) + end + return level + end + + def get_stat_level(stat, level) + return level if level >= 3 + return 3 if stat.match(/error|fail/i) + return 2 if stat.match(/warn/i) + + return 0 + end + + def output + result = { + 'stats.count' => @stats_count + } + result['stats.sum'] = @stats_sum if @fields.include?('stats.sum') + result['stats.avg'] = @stats_avg if @fields.include?('stats.avg') + @result.merge!(result) + puts JSON.pretty_generate(@result) + end +end diff --git a/sbin/es-jobs b/sbin/es-jobs new file mode 100755 index 0000000..f075ef2 --- /dev/null +++ b/sbin/es-jobs @@ -0,0 +1,53 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'optparse' +require 'json' + +require_relative '../lib/es_jobs' + +def parse_argv + items = {} + ARGV.each do |item| + key, value = item.split('=') + if key && value + value_list = value.split(',') + items[key] = value_list.length > 1 ? value_list : value + end + end + items +end +opt_refine = [-1] +opt_fields = [] +opt_parser = OptionParser.new do |opts| + opts.banner = 'Usage: es-jobs [options] search_key1=val1[,val2..] ..' + + opts.separator 'search_key can be id, suite, os, etc.' + opts.separator 'How to use -r' + opts.separator 'Like es-jobs submit_id=xxx -r "-1,0,1,2,3"' + opts.separator '-1 means not refine' + opts.separator '{ + -1 => null, + 0 => stats.succes, + 1 => stats.unknow, + 2 => stats.warning, + 3 => stats.has_error +}' + + opts.on('-r fields', '--refine fields', 'refine jobs') do |fields| + opt_refine = fields.split(',').map(&:to_i) + end + + opts.on('-f fields', '--fields fields', 'fields you want to see') do |fields| + opt_fields = fields.split(',') + end +end + +opt_parser.parse!(ARGV) +items = parse_argv +raise 'Please enter a query' if items.empty? + +es_jobs = ESJobs.new(items, opt_refine, opt_fields) +es_jobs.output -- 2.23.0

3 2

[PATCH v2 lkp-tests] tests/send_email: refactor send_email
by Wei Jihui 12 Jan '21

12 Jan '21

It is convenient for other tasks reuse to send email [usage] job.yaml test_task: on_fail: send_email: job_failed sleep: 1h This will send email to user when job is failed. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/email.sh | 26 ++++++++++++++++++++++++++ tests/email | 41 ----------------------------------------- tests/send_email | 21 +++++++++++++++++++++ 3 files changed, 47 insertions(+), 41 deletions(-) create mode 100755 lib/email.sh delete mode 100755 tests/email create mode 100755 tests/send_email diff --git a/lib/email.sh b/lib/email.sh new file mode 100755 index 00000000..d7c2c44a --- /dev/null +++ b/lib/email.sh @@ -0,0 +1,26 @@ +#!/bin/sh + +job_failed_content() +{ + email_content="To: $my_email +Subject: [NOTIFY Compass-CI] Test job failed: $id + +Dear $my_name: + + Sorry to inform you that your test job is failed, you can click the following link to view details. + http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT:-11300}$result_root + +Regards +Compass-CI +" +} + +send_email() +{ + local subject=$1 + local email_content + + ${subject}_content + + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$email_content" +} diff --git a/tests/email b/tests/email deleted file mode 100755 index a5f87d42..00000000 --- a/tests/email +++ /dev/null @@ -1,41 +0,0 @@ -#!/bin/sh -# - my_email -# - info - -: ${info:=failure} - -SCRIPT_DIR=$(dirname $(realpath $0)) -PROJECT_DIR=$(dirname $SCRIPT_DIR) - -. $PROJECT_DIR/lib/http.sh -. $PROJECT_DIR/lib/env.sh - -data_failure() -{ - data="To: $my_email -Subject: [NOTIFY Compass-ci] Test job failed - -Dear $my_username: - - Sorry to inform you that your test job is failed. you can click the following link to view details. - http://api.compass-ci.openeuler.org:$SRV_HTTP_PORT/$result_root - -Regards -Compass-Ci -" -} - -compose_email() -{ - [ "$info" == "failure" ] && data_failure -} - -run_email() -{ - [ -n "$my_email" ] || return 0 - my_username=$(echo "$my_email" | awk -F '@' '{print $1}') - compose_email - curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$data" -} - -run_email diff --git a/tests/send_email b/tests/send_email new file mode 100755 index 00000000..2911d3b8 --- /dev/null +++ b/tests/send_email @@ -0,0 +1,21 @@ +#!/bin/sh +# - subject +# my_email + +SCRIPT_DIR=$(dirname $(realpath $0)) +PROJECT_DIR=$(dirname $SCRIPT_DIR) + +. $PROJECT_DIR/lib/email.sh + +[ -n "$my_email" ] || { + echo "Can not get my_email, send email failed!" + exit 0 +} + +[ -n "$subject" ] || { + echo "Can not get email subject, send email failed!" + exit 0 +} + + +send_email $subject -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/job.cr: set the upload path of package based on os attribute of top-level package style
by Cao Xueliang 12 Jan '21

12 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 30 ++++++++++++++++++++++++------ 1 file changed, 24 insertions(+), 6 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 218e2e1..6642703 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -249,11 +249,30 @@ class Job set_upload_dirs() end - def get_package_dir - if @hash["cci-makepkg"]? || @hash["cci-depends"]? || @hash["build-pkg"]? - mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup - common_dir = "#{mount_type}/#{os}/#{os_arch}/#{os_version}" + def get_pkg_common_dir + pkg_style = nil + ["cci-makepkg", "cci-depends", "build-pkg"].each do |item| + pkg_style = @hash[item]? + break if pkg_style end + return nil unless pkg_style && pkg_style != nil + + tmp_os = pkg_style["os"]? == nil ? "#{os}" : pkg_style["os"] + tmp_os_arch = pkg_style["os_arch"]? == nil ? "#{os_arch}" : pkg_style["os_arch"] + tmp_os_version = pkg_style["os_version"]? == nil ? "#{os_version}" : pkg_style["os_version"] + + tmp_os_mount = pkg_style["os_mount"]? == nil ? "#{os_mount}" : pkg_style["os_mount"] + mount_type = tmp_os_mount == "cifs" ? "nfs" : tmp_os_mount.dup + + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + + return common_dir + end + + def get_package_dir + package_dir = "" + common_dir = get_pkg_common_dir + return package_dir unless common_dir if @hash["cci-makepkg"]? package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" @@ -265,10 +284,9 @@ class Job else package_name = @hash["pkgbuild_repo"].to_s.split("/")[-1] end + package_dir = ",/initrd/build-pkg/#{common_dir}/#{package_name}" package_dir += ",/cci/build-config" if @hash["config"]? - else - package_dir = "" end return package_dir -- 2.23.0

1 0

[PATCH lkp-tests] tests/*: add makepkg -s operation in archlinux
by Wang Yong 12 Jan '21

12 Jan '21

use makepkg -s to auto install depends in archlinux ref: https://git.archlinux.org/pacman.git/plain/scripts/makepkg.sh.in Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 16 +++++++++++++--- tests/cci-makepkg | 14 +++++++++++--- 2 files changed, 24 insertions(+), 6 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 31762384d..5a473a00c 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -92,7 +92,12 @@ create_cgz_name() cgz_path="$PKG_MNT/${pack_to}/${pkgname}" cgz_name="${cgz_path}/${upstream_commit}.cgz" - pkg_args="-A --check --skippgpcheck" + + if [ "${os}" == "archlinux" ]; then + pkg_args="-A --check --skippgpcheck -s --needed --noconfirm" + else + pkg_args="-A --check --skippgpcheck" + fi [ -n "$git_tag" ] && { cgz_name="${cgz_path}/${git_tag}-${pkgrel}.cgz" @@ -153,8 +158,13 @@ build_source_pkg() create_cgz_name get_config - PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ - $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf + if [ "${os}" == "archlinux" ]; then + BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ + $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf + else + PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ + $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf + fi local build_state="$?" diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 7d1e5cb15..e4bab41eb 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -133,10 +133,18 @@ update_softlink() build_source_pkg() { - local makepkg="$LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf" - echo "$makepkg" + if [ "${DISTRO}" == "archlinux" ]; then + local makepkg="$LKP_SRC/sbin/makepkg -A -s --needed --noconfirm --skippgpcheck --config $LKP_SRC/etc/makepkg.conf" + echo "$makepkg" - PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + + else + local makepkg="$LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf" + echo "$makepkg" + + PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + fi [ "$?" == 0 ] || exit 1 update_softlink -- 2.23.0

1 0

[PATCH v5 compass-ci 2/2] sbin: add email-mapping
by Luan Shengde 12 Jan '21

12 Jan '21

for user to: - add email mapping - delete email mapping - re-map email mapping - search email mapping - search all email mapping from stdin or map file or - delete email mapping queue Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r] email_mapping -f mapping_file [-r] [-d] email_mapping --delete-queue email_mapping --search-all options: -n, --name name|email|tag appoint a name|email|tag to add mapping -e, --email email_address appoint a email to be mapped -f, --file filename appoint a mapping file for name/email -r, --re-map do re-mappings -d, --delete delete email mappings -s, --search search email mapping -h, --help show this message --delete-queue delete mapping queue --search-all search all email mappings the name can be either a name, an email or something like a tag. the file content consists of names and emails, like: name: email_address_d tag: email_address_d email_address_r: email_address_d can map multi name|tag|email_address to a single email address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/email-mapping.rb | 119 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 119 insertions(+) create mode 100755 sbin/email-mapping.rb diff --git a/sbin/email-mapping.rb b/sbin/email-mapping.rb new file mode 100755 index 0000000..365aabf --- /dev/null +++ b/sbin/email-mapping.rb @@ -0,0 +1,119 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require 'optparse' + +REDIS_HOST = 'localhost' +REDIS_PORT = '6379' + +# require_relative "#{ENV['CCI_SRC']}/lib/redis_client" +require '../lib/redis_client' + +r_name = nil +d_email = nil +file_email_mapping = {} +map_conf = { + 're_mapping' => false, + 'del_map' => false, + 'search_map' => false, + 'delete_queue' => false, + 'search_all_map' => false +} + +options = OptionParser.new do |opts| + opts.banner = "Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r]\n" + opts.banner += " email_mapping -f mapping_file [-r] [-d] [-s]\n" + opts.banner += " email_mapping --delete-queue\n" + opts.banner += " email_mapping --search-all\n" + + opts.separator '' + opts.separator 'options:' + + opts.on('-n name|email|tag', '--name name|email|tag', 'appoint a name|email|tag to add mapping') do |name| + r_name = name + end + + opts.on('-e email_address', '--email email_address', 'appoint a email to be mapped') do |email| + d_email = email + end + + opts.on('-f filename', '--file filename', 'appoint a mapping file for name/email') do |filename| + mapping_content = YAML.load_file(filename) || {} + file_email_mapping.update mapping_content + end + + opts.on('-r', '--re-map', 'do re-mappings') do + map_conf['re_mapping'] = true + end + + opts.on('-d', '--delete', 'delete email mappings') do + map_conf['del_map'] = true + end + + opts.on('--delete-queue', 'delete mapping queue') do + map_conf['delete_queue'] = true + end + + opts.on('-s', '--search', 'search email mapping') do + map_conf['search_map'] = true + end + + opts.on('--search-all', 'search all email mappings') do + map_conf['search_all_map'] = true + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +required_opts = ['-n', '-f', '--delete-queue', '--search-all'] + +if ARGV.empty? + ARGV << '-h' +elsif (required_opts - ARGV).eql? required_opts + ARGV.clear + ARGV << '-h' +end + +options.parse!(ARGV) + +def email_mapping(r_name, d_email, map_conf) + email_mapping = RedisClient.new('email_mapping') + if map_conf['re_mapping'] + email_mapping.reset_hash_key(r_name, d_email) + elsif map_conf['del_map'] + email_mapping.delete_hash_key(r_name) + elsif map_conf['search_map'] + mapped_email = email_mapping.search_hash_key(r_name) + if mapped_email.nil? || mapped_email.empty? + puts "#{r_name} has not been mapped an email yet." + else + puts mapped_email + end + elsif map_conf['search_all_map'] + all_mappings = email_mapping.search_all_hash_key + + all_mappings.each do |k, v| + puts "#{k}: #{v}\n" + end + elsif map_conf['delete_queue'] + email_mapping.delete_queue + else + return if email_mapping.add_hash_key(r_name, d_email) + + puts "#{r_name} has already mapped an email." + end +end + +if file_email_mapping.empty? + email_mapping(r_name, d_email, map_conf) +else + file_email_mapping.each do |name, email| + email_mapping(name, email, map_conf) + end +end -- 2.23.0

1 0

[PATCH v5 compass-ci 1/2] lib: add redis_client
by Luan Shengde 12 Jan '21

12 Jan '21

used to: add redis hash key delete redis hash key reset redis hash key search redis hash key search all redis hash keys, values delete redis queue Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/redis_client.rb | 49 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100755 lib/redis_client.rb diff --git a/lib/redis_client.rb b/lib/redis_client.rb new file mode 100755 index 0000000..0c05252 --- /dev/null +++ b/lib/redis_client.rb @@ -0,0 +1,49 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'redis' +require 'json' + +# used to handle the redis queues +class RedisClient + RDS_HOST = ENV['REDIS_HOST'] || REDIS_HOST + RDS_PORT = ENV['REDIS_PORT'] || REDIS_PORT + + def initialize(queue) + @queue = queue + @redis = Redis.new('host' => RDS_HOST, 'port' => RDS_PORT) + end + + def add_hash_key(key, value) + return false if @redis.hexists @queue, key + + @redis.hset @queue, key, value + return true + end + + def delete_hash_key(key) + @redis.hdel @queue, key + end + + def delete_queue + @redis.del @queue + end + + def reset_hash_key(key, value) + @redis.hset @queue, key, value + end + + def search_hash_key(key) + h_value = @redis.hget @queue, key + + return h_value if h_value + end + + def search_all_hash_key + h_values = @redis.hscan_each(@queue).to_h + + return h_values + end +end -- 2.23.0

1 0

[PATCH lkp-tests] setup/pgbench-setup: start postgresql
by Hu Xuejiao 12 Jan '21

12 Jan '21

[why] when after installed postgresql-server, postgres(user) is already created, so initializing the postgresql directly. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- setup/pgbench-setup | 21 +++++---------------- 1 file changed, 5 insertions(+), 16 deletions(-) diff --git a/setup/pgbench-setup b/setup/pgbench-setup index cc0ce37da..ea55b2ebc 100755 --- a/setup/pgbench-setup +++ b/setup/pgbench-setup @@ -25,28 +25,17 @@ postgres_run_log() init_postgresql() { - grep -q postgres /etc/passwd && return - - useradd --home-dir /var/lib/postgresql --no-create-home --no-user-group --gid nogroup --system postgres - data_dir=/var/lib/postgresql/data + data_dir=/var/lib/pgsql/data mkdir -p "$data_dir" - chown postgres "$data_dir" - - bin_dir=/usr/lib/postgresql/*/bin - is_clearlinux && bin_dir=/usr/bin - postgres_run $bin_dir/initdb -D "$data_dir" 2>&1 - postgresql_conf_file="$data_dir"/postgresql.conf - run_dir=/var/run/postgresql - mkdir -p "$run_dir" - chown postgres "$run_dir" - postgres_run $bin_dir/pg_ctl -D "$data_dir" start + postgresql-setup initdb + service postgresql start } -postgresql_conf_file=$(find /etc/postgresql -name postgresql.conf | head -1) - init_postgresql +postgresql_conf_file=$(find "$data_dir" -name postgresql.conf | head -1) + [ -n "$max_wal_size" ] && log_eval "echo 'max_wal_size = $max_wal_size' >> $postgresql_conf_file" -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sbin/auto_submit: only submit one branch job
by Liu Shaofei 12 Jan '21

12 Jan '21

[why] As some repositories have many branches, the git-mirror will submit jobs for each branch. So the large number of jobs are submitted for the same repository. To avoid this problem, just master branch is submitted. If repository does not have master branch, the last branch is selected to submit job. Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 51ecc6a..950d1f0 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -28,16 +28,29 @@ class AutoSubmit return pkgbuild_repo end - def submit(newrefs_info, submit_argv) + # just submit master branch, if not master branch, submit last one branch. + def get_branch_argvs(newrefs_info) + repo = { 'branch' => '', 'commit' => '' } + newrefs_info['new_refs']['heads'].each do |branch, commit_id| - real_argvs = Array.new(submit_argv) - real_argvs.push("upstream_branch=#{branch.delete_prefix('refs/heads/')}") - real_argvs.push("upstream_commit=#{commit_id}") - tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{commit_id}) - real_argvs.push("upstream_tag=#{tag}") unless tag.empty? + repo['branch'] = branch.delete_prefix('refs/heads/') + repo['commit'] = commit_id - system(real_argvs.join(' ')) + break if repo['branch'] == 'master' end + repo + end + + def submit(newrefs_info, submit_argv) + real_argvs = Array.new(submit_argv) + + repo = get_branch_argvs(newrefs_info) + + real_argvs.push("upstream_branch=#{repo['branch']}") + real_argvs.push("upstream_commit=#{repo['commit']}") + tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{repo['commit']}) + real_argvs.push("upstream_tag=#{tag}") unless tag.empty? + system(real_argvs.join(' ')) end def get_argvs(newrefs_info) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] sbin/auto_submit.yaml: update repos for build-pkg jobs
by Liu Shaofei 12 Jan '21

12 Jan '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit.yaml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index 22172eb..c368ad6 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -19,5 +19,7 @@ l/linux/archlinux-kernel: - testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml - testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml archlinux: -- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-pkg.yaml +- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=dc-8g os=openeuler os_version=20.03-pre os_mount=container os_arch=aarch64 docker_image=openeuler:20.03-pre queue=dc-build-pkg build-pkg.yaml +- testbox=dc-8g os=archlinux os_version=2021-01-06-pre os_mount=container os_arch=aarch64 docker_image=archlinux:2021-01-06-pre queue=dc-build-pkg build-pkg.yaml -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests/*: reduce parameters
by Wang Yong 12 Jan '21

12 Jan '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 10 +++++----- tests/cci-depends | 11 +++++++---- tests/cci-makepkg | 11 +++++++---- 3 files changed, 19 insertions(+), 13 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 98341e7ac..31762384d 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -7,12 +7,12 @@ # - config # upstream_url -# We'll use the below global env vars, but not treat them as parameter to avoid +# We'll use the below global env vars, but not treat them as parameters to avoid # duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ -# - os -# - os_arch -# - os_version -# - os_mount +# os +# os_arch +# os_version +# os_mount ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, diff --git a/tests/cci-depends b/tests/cci-depends index 8baf023f3..c89737aca 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -1,9 +1,12 @@ #!/bin/bash # - benchmark -# - os_mount -# - os -# - os_arch -# - os_version + +# We'll use the below global env vars, but not treat them as parameters to avoid +# duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ +# os +# os_arch +# os_version +# os_mount . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/misc-base.sh diff --git a/tests/cci-makepkg b/tests/cci-makepkg index e737955b9..7d1e5cb15 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -1,9 +1,12 @@ #!/bin/bash # - benchmark -# - os_mount -# - os -# - os_arch -# - os_version + +# We'll use the below global env vars, but not treat them as parameters to avoid +# duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ +# os +# os_arch +# os_version +# os_mount ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: change possible matches for webhook
by Li Yuanchao 12 Jan '21

12 Jan '21

As directory structure of upstream-repos changed, the possible git_repo of repository to match also need to change. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index f5c653b..2adaaa1 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -319,16 +319,12 @@ class MirrorMain end # example - # url: https://gitee.com/theprocess/oec-hardware git_repo: oec-hardware/oec-hardware # url: https://github.com/berkeley-abc/abc git_repo: a/abc/abc - # url: https://github.com/Siguyi/AvxToNeon git_repo: AvxToNeon/Siguyi + # url: https://github.com/Siguyi/AvxToNeon git_repo: a/AvxToNeon/Siguyi def get_git_repo(webhook_url) fork_name, project = webhook_url.split('/')[-2, 2] - git_repo = "#{project}/#{project}" - return git_repo if check_git_repo(git_repo, webhook_url) - - git_repo = "#{project}/#{fork_name}" + git_repo = "#{project[0].downcase}/#{project}/#{fork_name}" return git_repo if check_git_repo(git_repo, webhook_url) git_repo = "#{project[0].downcase}/#{project}/#{project}" -- 2.23.0

1 0

[PATCH v4 compass-ci 2/2] sbin: add email-mapping
by Luan Shengde 12 Jan '21

12 Jan '21

for user to: - add email mapping - delete email mapping - re-map email mapping - search email mapping from stdin or map file or - delete email mapping queue Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r] email_mapping -f mapping_file [-r] [-d] email_mapping --delete-queue options: -n, --name name|email|tag appoint a name|email|tag to add mapping -e, --email email_address appoint a email to be mapped -f, --file filename appoint a mapping file for name/email -r, --re-map do re-mappings -d, --delete delete email mappings --delete-queue delete mapping queue -s, --search search email mapping -h, --help show this message the name can be either a name, an email or something like a tag. the file content consists of names and emails, like: name: email_address_d tag: email_address_d email_address_r: email_address_d can map multi name|tag|email_address to a single email address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/email-mapping.rb | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/sbin/email-mapping.rb b/sbin/email-mapping.rb index 9d72ecc..d3a9a62 100755 --- a/sbin/email-mapping.rb +++ b/sbin/email-mapping.rb @@ -9,7 +9,8 @@ require 'optparse' REDIS_HOST = 'localhost' REDIS_PORT = '6379' -require "#{ENV['CCI_SRC']}/lib/email_mapping_client" +# require_relative "#{ENV['CCI_SRC']}/lib/redis_client" +require '../lib/redis_client' r_name = nil d_email = nil @@ -74,17 +75,22 @@ end options.parse!(ARGV) def email_mapping(r_name, d_email, map_conf) - email_mapping = EmailMapping.new(r_name, d_email) + email_mapping = RedisClient.new('email_mapping') if map_conf['re_mapping'] - email_mapping.re_map_email + email_mapping.reset_hash_key(r_name, d_email) elsif map_conf['del_map'] - email_mapping.delete_email_mapping + email_mapping.delete_hash_key(r_name) elsif map_conf['search_map'] - puts email_mapping.search_map + mapped_email = email_mapping.search_hash_key(r_name) + if mapped_email.nil? || mapped_email.empty? + puts "#{r_name} has not been mapped an email yet." + else + puts mapped_email + end elsif map_conf['delete_queue'] - email_mapping.delete_mapping_queue + email_mapping.delete_queue else - return if email_mapping.add_email_mapping + return if email_mapping.add_hash_key(r_name, d_email) puts "#{r_name} has already mapped an email." end -- 2.23.0

1 1

[PATCH v4 compass-ci 1/2] lib: add redis_client
by Luan Shengde 12 Jan '21

12 Jan '21

used to: add redis hash key delete redis hash key reset redis hash key search redis hash key delete redis queue Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/redis_client.rb | 43 +++++++++++++++++++ sbin/email-mapping.rb | 99 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 142 insertions(+) create mode 100755 lib/redis_client.rb create mode 100755 sbin/email-mapping.rb diff --git a/lib/redis_client.rb b/lib/redis_client.rb new file mode 100755 index 0000000..142d7f0 --- /dev/null +++ b/lib/redis_client.rb @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'redis' +require 'json' + +# used to add/delete/re-map/search email mappings +class RedisClient + RDS_HOST = ENV['REDIS_HOST'] || REDIS_HOST + RDS_PORT = ENV['REDIS_PORT'] || REDIS_PORT + + def initialize(queue) + @queue = queue + @redis = Redis.new('host' => RDS_HOST, 'port' => RDS_PORT) + end + + def add_hash_key(key, value) + return false if @redis.hexists @queue, key + + @redis.hset @queue, key, value + return true + end + + def delete_hash_key(key) + @redis.hdel @queue, key + end + + def delete_queue + @redis.del @queue + end + + def reset_hash_key(key, value) + @redis.hset @queue, key, value + end + + def search_hash_key(key) + h_value = @redis.hget @queue, key + + return h_value if h_value + end +end diff --git a/sbin/email-mapping.rb b/sbin/email-mapping.rb new file mode 100755 index 0000000..9d72ecc --- /dev/null +++ b/sbin/email-mapping.rb @@ -0,0 +1,99 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require 'optparse' + +REDIS_HOST = 'localhost' +REDIS_PORT = '6379' + +require "#{ENV['CCI_SRC']}/lib/email_mapping_client" + +r_name = nil +d_email = nil +file_email_mapping = {} +map_conf = { + 're_mapping' => false, + 'del_map' => false, + 'search_map' => false, + 'delete_queue' => false +} + +options = OptionParser.new do |opts| + opts.banner = "Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r]\n" + opts.banner += " email_mapping -f mapping_file [-r] [-d] [-s]\n" + opts.banner += " email_mapping --delete-queue\n" + + opts.separator '' + opts.separator 'options:' + + opts.on('-n name|email|tag', '--name name|email|tag', 'appoint a name|email|tag to add mapping') do |name| + r_name = name + end + + opts.on('-e email_address', '--email email_address', 'appoint a email to be mapped') do |email| + d_email = email + end + + opts.on('-f filename', '--file filename', 'appoint a mapping file for name/email') do |filename| + mapping_content = YAML.load_file(filename) || {} + file_email_mapping.update mapping_content + end + + opts.on('-r', '--re-map', 'do re-mappings') do + map_conf['re_mapping'] = true + end + + opts.on('-d', '--delete', 'delete email mappings') do + map_conf['del_map'] = true + end + + opts.on('--delete-queue', 'delete mapping queue') do + map_conf['delete_queue'] = true + end + + opts.on('-s', '--search', 'search email mapping') do + map_conf['search_map'] = true + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +if ARGV.empty? + ARGV << '-h' +elsif (['-n', '-f', '--delete-queue'] - ARGV).eql? ['-n', '-f', '--delete-queue'] + ARGV.clear + ARGV << '-h' +end + +options.parse!(ARGV) + +def email_mapping(r_name, d_email, map_conf) + email_mapping = EmailMapping.new(r_name, d_email) + if map_conf['re_mapping'] + email_mapping.re_map_email + elsif map_conf['del_map'] + email_mapping.delete_email_mapping + elsif map_conf['search_map'] + puts email_mapping.search_map + elsif map_conf['delete_queue'] + email_mapping.delete_mapping_queue + else + return if email_mapping.add_email_mapping + + puts "#{r_name} has already mapped an email." + end +end + +if file_email_mapping.empty? + email_mapping(r_name, d_email, map_conf) +else + file_email_mapping.each do |name, email| + email_mapping(name, email, map_conf) + end +end -- 2.23.0

1 1

[PATCH v2 compass-ci 3/3] container/auto-submit: add build depends
by Liu Yinsi 12 Jan '21

12 Jan '21

store auto submit info to ES in build script, so add build depends. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/build-depends | 1 + 1 file changed, 1 insertion(+) create mode 100644 container/auto-submit/build-depends diff --git a/container/auto-submit/build-depends b/container/auto-submit/build-depends new file mode 100644 index 0000000..8357fca --- /dev/null +++ b/container/auto-submit/build-depends @@ -0,0 +1 @@ +es -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] container/auto-submit: fix failed to submit job
by Liu Yinsi 12 Jan '21

12 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/container/es# docker exec -it auto-submit sh /c/lkp-tests/sbin $ ./submit iperf.yaml testbox=vm-2p8g submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_email'. Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… [how] for user install compass-ci at their local server, store auto submit info to ES to pass verification when submit jobs in container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/build | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/container/auto-submit/build b/container/auto-submit/build index 96383e3..9530d47 100755 --- a/container/auto-submit/build +++ b/container/auto-submit/build @@ -1,6 +1,14 @@ -#!/bin/sh +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -docker build -t alpine:auto-submit . +require "#{ENV['CCI_SRC']}/lib/build_my_info_client" +config_yaml = '/etc/compass-ci/account/autosubmit.yaml' +option = YAML.load_file(config_yaml) + +build_my_info = BuildMyInfo.new(option['my_email'], option['my_name'], option['lab'], option['my_token']) +build_my_info.store_account_info + +system 'docker build -t alpine:auto-submit .' -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] container/auto-submit: bind mount volume
by Liu Yinsi 12 Jan '21

12 Jan '21

lkp-tests/lib/job.rb: load_self_config self_config_path = "#{ENV['HOME']}/.config/compass-ci" Dir.glob(['/etc/compass-ci/defaults/*.yaml', "#{self_config_path}/defaults/*.yaml"]).each do |file| [why] load_self_config support /etc/compass/defaults/ and $HOME/.config/compass-ci/defaults/, if mount /etc/compass-ci/account, it can't load config, so use /home/lkp/.config/compass-ci/defaults. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/auto-submit/start b/container/auto-submit/start index c25fbdd..e66cf57 100755 --- a/container/auto-submit/start +++ b/container/auto-submit/start @@ -21,7 +21,7 @@ cmd = %W[ -v #{ENV['LKP_SRC']}:#{DOCKER_LKP} -v /etc/localtime:/etc/localtime:ro -v /etc/compass-ci/defaults:/etc/compass-ci/defaults:ro - -v /etc/compass-ci/account:/etc/compass-ci/account:ro + -v /etc/compass-ci/account:/home/lkp/.config/compass-ci/defaults:ro -v /srv/git:/srv/git -w /c/compass-ci/sbin alpine:auto-submit -- 2.23.0

1 0

[PATCH v2 compass-ci] sbin/run_job: update the script
by Hu Xuejiao 12 Jan '21

12 Jan '21

[why] 1.host-info is adapted in physical machine, so replace yaml-file 2.after running multi-docker and multi-qemu, these processes need to be deleted Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/sbin/run_job b/sbin/run_job index 325fa9f..7405ca0 100755 --- a/sbin/run_job +++ b/sbin/run_job @@ -24,7 +24,7 @@ submit_one_job() submit_job() { - yaml_args=("$cci_makepkg_file" "host-info.yaml") + yaml_args=("$cci_makepkg_file" "ebizzy.yaml") for yaml in "${yaml_args[@]}" do submit_one_job & @@ -72,6 +72,18 @@ run_docker() done } +kill_all() +{ + for line in $(cat "$1") + do + [ -z "$ line" ] && break + kill -9 "$line" + done +} + run_qemu run_docker wait + +kill_all pid +kill_all $CCI_SRC/providers/dc.pid -- 2.23.0

2 1

[PATCH compass-ci] fix: the error docker name
by Wang Chenglong 12 Jan '21

12 Jan '21

[Error] + docker cp -a init_docker:/tmp/openeuler-20.09-20210112.cgz /home/wcl/ Error: No such container:path: init_docker:/tmp/openeuler-20.09-20210112.cgz --- container/osimage/openeuler/lib | 2 +- container/osimage/openeuler/run | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/osimage/openeuler/lib b/container/osimage/openeuler/lib index e0b7017..34dd0ec 100755 --- a/container/osimage/openeuler/lib +++ b/container/osimage/openeuler/lib @@ -40,6 +40,6 @@ load_docker_image() cp_image_to_host() { - docker cp -a init_docker:/tmp/${IMAGE_NAME} $HOME/ + docker cp -a initramfs_${image}:/tmp/${IMAGE_NAME} $HOME/ echo "result: $(ls $HOME/${IMAGE_NAME})" } diff --git a/container/osimage/openeuler/run b/container/osimage/openeuler/run index 5d7da72..6ed4069 100755 --- a/container/osimage/openeuler/run +++ b/container/osimage/openeuler/run @@ -24,5 +24,5 @@ cmd=( load_docker_image "${cmd[@]}" cp_image_to_host -docker_rm init_docker &> /dev/null +docker_rm initramfs_${image} &> /dev/null echo "build finished" -- 2.23.0

1 0

[PATCH compass-ci] lib/build_my_info_client: set default value
by Liu Yinsi 12 Jan '21

12 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/build_my_info_client.rb | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb index 162bf9f..c946bec 100755 --- a/lib/build_my_info_client.rb +++ b/lib/build_my_info_client.rb @@ -21,12 +21,13 @@ require_relative '../lib/es_client' # build_my_info = BuildMyInfo.new(email, name, lab) # build_my_info.config_my_info class BuildMyInfo - def initialize(my_email, my_name, lab) + def initialize(my_email, my_name, lab, my_token = nil) @lab = lab || 'nolab' + @my_token = my_token || %x(uuidgen).chomp @my_info = { 'my_email' => my_email, 'my_name' => my_name, - 'my_token' => %x(uuidgen).chomp, + 'my_token' => @my_token, 'my_login_name' => nil, 'my_commit_url' => nil, 'my_ssh_pubkey' => [] -- 2.23.0

1 0

[PATCH compass-ci 3/3] container/auto-submit: add build depends
by Liu Yinsi 12 Jan '21

12 Jan '21

store auto submit info to ES in build script, so add build depends. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/build-depends | 1 + 1 file changed, 1 insertion(+) create mode 100644 container/auto-submit/build-depends diff --git a/container/auto-submit/build-depends b/container/auto-submit/build-depends new file mode 100644 index 0000000..8357fca --- /dev/null +++ b/container/auto-submit/build-depends @@ -0,0 +1 @@ +es -- 2.23.0

1 0

[PATCH compass-ci 2/3] container/auto-submit: fix failed to submit job
by Liu Yinsi 12 Jan '21

12 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/container/es# docker exec -it auto-submit sh /c/lkp-tests/sbin $ ./submit iperf.yaml testbox=vm-2p8g submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_email'. Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… [how] for user install compass-ci at their local server, store auto submit info to ES to pass verification when submit jobs in container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/build | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/container/auto-submit/build b/container/auto-submit/build index 96383e3..8b8cc8c 100755 --- a/container/auto-submit/build +++ b/container/auto-submit/build @@ -1,6 +1,14 @@ -#!/bin/sh +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -docker build -t alpine:auto-submit . +require "#{ENV['CCI_SRC']}/lib/build_my_info_client" +config_yaml = "/etc/compass-ci/account/autosubmit.yaml" +option = YAML.load_file(config_yaml) + +build_my_info = BuildMyInfo.new(option['my_email'], option['my_name'], option['lab'], option['my_token']) +build_my_info.store_account_info + +system 'docker build -t alpine:auto-submit .' -- 2.23.0

1 0

[PATCH compass-ci 1/3] container/auto-submit: bind mount volume
by Liu Yinsi 12 Jan '21

12 Jan '21

lkp-tests/lib/job.rb: load_self_config self_config_path = "#{ENV['HOME']}/.config/compass-ci" Dir.glob(['/etc/compass-ci/defaults/*.yaml', "#{self_config_path}/defaults/*.yaml"]).each do |file| [why] load_self_config support /etc/compass/defaults/ and $HOME/.config/compass-ci/defaults/, if mount /etc/compass-ci/account, it can't load config, so use /home/lkp/.config/compass-ci/defaults. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/auto-submit/start b/container/auto-submit/start index c25fbdd..e66cf57 100755 --- a/container/auto-submit/start +++ b/container/auto-submit/start @@ -21,7 +21,7 @@ cmd = %W[ -v #{ENV['LKP_SRC']}:#{DOCKER_LKP} -v /etc/localtime:/etc/localtime:ro -v /etc/compass-ci/defaults:/etc/compass-ci/defaults:ro - -v /etc/compass-ci/account:/etc/compass-ci/account:ro + -v /etc/compass-ci/account:/home/lkp/.config/compass-ci/defaults:ro -v /srv/git:/srv/git -w /c/compass-ci/sbin alpine:auto-submit -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sbin/auto_submit: only submit one branch job
by Liu Shaofei 12 Jan '21

12 Jan '21

[why] As some repositories have many branches, the git-mirror will submit jobs for each branch. So the large number of jobs are submitted for the same repository. To avoid this problem, just master branch is submitted. If repository does not have master branch, the last branch is selected to submit job. Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 51ecc6a..950d1f0 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -28,16 +28,29 @@ class AutoSubmit return pkgbuild_repo end - def submit(newrefs_info, submit_argv) + # just submit master branch, if not master branch, submit last one branch. + def get_branch_argvs(newrefs_info) + repo = { 'branch' => '', 'commit' => '' } + newrefs_info['new_refs']['heads'].each do |branch, commit_id| - real_argvs = Array.new(submit_argv) - real_argvs.push("upstream_branch=#{branch.delete_prefix('refs/heads/')}") - real_argvs.push("upstream_commit=#{commit_id}") - tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{commit_id}) - real_argvs.push("upstream_tag=#{tag}") unless tag.empty? + repo['branch'] = branch.delete_prefix('refs/heads/') + repo['commit'] = commit_id - system(real_argvs.join(' ')) + break if repo['branch'] == 'master' end + repo + end + + def submit(newrefs_info, submit_argv) + real_argvs = Array.new(submit_argv) + + repo = get_branch_argvs(newrefs_info) + + real_argvs.push("upstream_branch=#{repo['branch']}") + real_argvs.push("upstream_commit=#{repo['commit']}") + tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{repo['commit']}) + real_argvs.push("upstream_tag=#{tag}") unless tag.empty? + system(real_argvs.join(' ')) end def get_argvs(newrefs_info) -- 2.23.0

1 1

[PATCH v2 compass-ci 1/2] bin/auto_submit.yaml: update repos for build-pkg jobs
by Liu Shaofei 12 Jan '21

12 Jan '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit.yaml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index 22172eb..c368ad6 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -19,5 +19,7 @@ l/linux/archlinux-kernel: - testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml - testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml archlinux: -- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-pkg.yaml +- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=dc-8g os=openeuler os_version=20.03-pre os_mount=container os_arch=aarch64 docker_image=openeuler:20.03-pre queue=dc-build-pkg build-pkg.yaml +- testbox=dc-8g os=archlinux os_version=2021-01-06-pre os_mount=container os_arch=aarch64 docker_image=archlinux:2021-01-06-pre queue=dc-build-pkg build-pkg.yaml -- 2.23.0

1 1

[PATCH v2 lkp-tests] tests/*: reduce parameters
by Wang Yong 12 Jan '21

12 Jan '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 8 ++++---- tests/cci-depends | 11 +++++++---- tests/cci-makepkg | 11 +++++++---- 3 files changed, 18 insertions(+), 12 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 98341e7ac..48c738f52 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -9,10 +9,10 @@ # We'll use the below global env vars, but not treat them as parameter to avoid # duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ -# - os -# - os_arch -# - os_version -# - os_mount +# os +# os_arch +# os_version +# os_mount ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, diff --git a/tests/cci-depends b/tests/cci-depends index 8baf023f3..487121454 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -1,9 +1,12 @@ #!/bin/bash # - benchmark -# - os_mount -# - os -# - os_arch -# - os_version + +# We'll use the below global env vars, but not treat them as parameter to avoid +# duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ +# os +# os_arch +# os_version +# os_mount . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/misc-base.sh diff --git a/tests/cci-makepkg b/tests/cci-makepkg index e737955b9..4f5f4c2f2 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -1,9 +1,12 @@ #!/bin/bash # - benchmark -# - os_mount -# - os -# - os_arch -# - os_version + +# We'll use the below global env vars, but not treat them as parameter to avoid +# duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ +# os +# os_arch +# os_version +# os_mount ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, -- 2.23.0

1 0

[PATCH compass-ci] fix: unusable package name
by Wang Chenglong 12 Jan '21

12 Jan '21

[Error] Packing package. Please wait. /usr/bin/setup-image: line 41: ./tmp/$IMAGE_PACK: ambiguous redirect before: IMAGE_PAVK=' openeuler-20.03.cgz' after: IMAGE_PAVK=openeuler-20.03.cgz Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker2rootfs/run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/docker2rootfs/run b/container/docker2rootfs/run index ce51ba3..8a7b315 100755 --- a/container/docker2rootfs/run +++ b/container/docker2rootfs/run @@ -12,7 +12,7 @@ check_passwd_file "$root_pwd_file" load_docker_img "$1" echo $docker_name_tag IMAGE_NAME=${docker_name_tag%%:*} -IMAGE_PACK=${IMAGE_NAME##*/}.cgz +IMAGE_PACK=$(echo ${IMAGE_NAME##*/}.cgz) start=( docker run -- 2.23.0

1 0

[PATCH v2 compass-ci] container/defconfig: optimize
by Luan Shengde 12 Jan '21

12 Jan '21

[why] when submitting jobs, the job will read data from /etc/compass-ci/defaults/*.yaml. the read data is open for user, but there are some sensitive data in the yaml files, like email address, email auth code, my_token .etc. [how] move the sensitive to yaml file in another directory: /etc/compass-ci/account/*.yaml Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/defconfig.rb | 1 + container/defconfig.sh | 7 ++++++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/container/defconfig.rb b/container/defconfig.rb index d1e2af0..fd13312 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -8,6 +8,7 @@ require 'yaml' def cci_defaults hash = {} Dir.glob(['/etc/compass-ci/defaults/*.yaml', + '/etc/compass-ci/accounts/*.yaml', "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| hash.update YAML.load_file(file) || {} end diff --git a/container/defconfig.sh b/container/defconfig.sh index 410ed75..97d924d 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -8,7 +8,12 @@ load_cci_defaults() { shopt -s nullglob - for i in /etc/compass-ci/defaults/*.yaml $HOME/.config/compass-ci/defaults/*.yaml + yaml_file=( + /etc/compass-ci/defaults/*.yaml + /etc/compass-ci/account/*.yaml + "$HOME"/.config/compass-ci/defaults/*.yaml + ) + for i in "${yaml_file[@]}" do create_yaml_variables "$i" done -- 2.23.0

1 0

[PATCH v5 lkp-tests] lib/bootstrap: fix failed to get MAC and IP
by Liu Yinsi 12 Jan '21

12 Jan '21

[error] LKP: HOSTNAME vm-2p8g.2288hv5-2s44p-384g--b5-1573861, MAC , IP , kernel 5.9.0-5-riscv64 1, serial console /dev/ttyS0 [why] [ 93.815004] + network_ok [ 93.874456] ++ cat /sys/class/net/eth0//operstate [ 93.879278] + '[' up = up ']' [ 93.882356] + return 0 [ 93.941910] + export_ip_mac [ 94.018548] + has_cmd ip [ 94.022832] + command -v ip [ 94.029742] ++ awk '{print $5; exit}' [ 94.036137] ++ ip route get 1.2.3.4 [ 94.045622] ++ /usr/bin/mawk '{print $5; exit}' [ 94.056504] RTNETLINK answers: Network is unreachable [ 94.061418] + export PUB_NIC= [ 94.064587] + PUB_NIC= `cat /sys/class/net/eth0/operstate` is up, network is ok, but actually network is unreachable, because routing is not ready, it can't get PUB_NIC and PUB_IP by ip route command, so put export_ip_mac in network_ok to ensure network is ok. test: /srv/result/iperf/2021-01-12/vm-2p8g/debian-sid-riscv64/tcp-30/z9.325217 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/bootstrap.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index b04ea3dd4..2969544a6 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -91,8 +91,8 @@ network_ok() for i in /sys/class/net/*/ do [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 + [ "$(cat $i/operstate)" = 'up' ] && export_ip_mac && return 0 + [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && export_ip_mac && return 0 done is_clearlinux && { @@ -296,6 +296,8 @@ export_ip_mac() export PUB_IP=$(ifconfig $PUB_NIC | awk '/inet / {print $2}') fi + [ -n "$PUB_NIC" ] || return 1 + export PUB_MAC=$(cat /sys/class/net/$PUB_NIC/address) [ -n "$SCHED_HOST" ] && PUB_MAC=$(echo "$PUB_MAC" | tr : -) } @@ -746,7 +748,6 @@ boot_init() setup_network run_ntpdate - export_ip_mac announce_bootup -- 2.23.0

1 0

[PATCH v4 lkp-tests] lib/bootstrap: fix failed to get MAC and IP
by Liu Yinsi 12 Jan '21

12 Jan '21

[error] LKP: HOSTNAME vm-2p8g.2288hv5-2s44p-384g--b5-1573861, MAC , IP , kernel 5.9.0-5-riscv64 1, serial console /dev/ttyS0 [why] [ 93.815004] + network_ok [ 93.874456] ++ cat /sys/class/net/eth0//operstate [ 93.879278] + '[' up = up ']' [ 93.882356] + return 0 [ 93.941910] + export_ip_mac [ 93.956875] + cat /sys/class/net/eth0/operstate [ 93.960236] ++ route -n [ 93.968810] + route='Kernel IP routing table [ 93.988935] Destination GatewayGenmaskFlags Metric Ref Use Iface' [ 94.011625] + echo Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface [ 94.018548] + has_cmd ip [ 94.022832] + command -v ip [ 94.029742] ++ awk '{print $5; exit}' [ 94.036137] ++ ip route get 1.2.3.4 [ 94.045622] ++ /usr/bin/mawk '{print $5; exit}' [ 94.056504] RTNETLINK answers: Network is unreachable [ 94.061418] + export PUB_NIC= [ 94.064587] + PUB_NIC= `cat /sys/class/net/eth0/operstate` is up, but routing is not ready, it can't get PUB_NIC and PUB_IP by ip route command, so check export_ip_mac in network_ok to make ensure network is ok. test: /srv/result/iperf/2021-01-12/vm-2p8g/debian-sid-riscv64/tcp-30/z9.325217 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/bootstrap.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index b04ea3dd4..2969544a6 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -91,8 +91,8 @@ network_ok() for i in /sys/class/net/*/ do [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 + [ "$(cat $i/operstate)" = 'up' ] && export_ip_mac && return 0 + [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && export_ip_mac && return 0 done is_clearlinux && { @@ -296,6 +296,8 @@ export_ip_mac() export PUB_IP=$(ifconfig $PUB_NIC | awk '/inet / {print $2}') fi + [ -n "$PUB_NIC" ] || return 1 + export PUB_MAC=$(cat /sys/class/net/$PUB_NIC/address) [ -n "$SCHED_HOST" ] && PUB_MAC=$(echo "$PUB_MAC" | tr : -) } @@ -746,7 +748,6 @@ boot_init() setup_network run_ntpdate - export_ip_mac announce_bootup -- 2.23.0

1 0