November 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v6 compass-ci 08/12] mail-robot: build fail message
by Luan Shengde 13 Nov '20

13 Nov '20

build fail email message with raised error message - lack commit url - lack pubkey - repo url for the commit url not in upstream - commit not exist or not available - no more available account on jumper server Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account-fail-eamil.rb | 38 ++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100755 lib/assign-account-fail-eamil.rb diff --git a/lib/assign-account-fail-eamil.rb b/lib/assign-account-fail-eamil.rb new file mode 100755 index 0000000..1c2133c --- /dev/null +++ b/lib/assign-account-fail-eamil.rb @@ -0,0 +1,38 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_fail_email(my_info, message) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] apply account failed + + Dear user: + + Your application for account failed with following error: + + #{message} + + In order to successfully apply an account, please pay attention to the following points: + + 1. mail subject + the subject should exactly: apply account + + 2. commit url + when you writing the url, add prefix: my oss commit + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + + attention: + ensure you commit url exist and available to access. + + 3. ssh pubkey + you need to add a pubkey as an attachment with name 'id_rsa.pub' to the email. + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

2 4

[PATCH v6 compass-ci 03/12] mail-robot: check to assign account
by Luan Shengde 13 Nov '20

13 Nov '20

AssignAccount assign account for user required data: mail_content - send_account parse email_content parse commit url parse pubkey apply_my_account generate uuid apply account store account info to es send email send success email send fail email usage: reference mail-robot.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account.rb | 80 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) create mode 100755 lib/assign-account.rb diff --git a/lib/assign-account.rb b/lib/assign-account.rb new file mode 100755 index 0000000..4443d47 --- /dev/null +++ b/lib/assign-account.rb @@ -0,0 +1,80 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative 'es_client' + +# assign uuid/account for user +# send_account +# parse my_commit_url and my_ssh_pubkey +# apply account +# store account info +# send mail +class AssignAccount + def initialize(mail_content) + @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp + @send_mail_port = SEND_MAIL_PORT + @mail_content = mail_content + + @my_info = { + 'my_email' => mail_content.from[0], + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, ''), + 'my_jumper_pubkey' => false + } + end + + def send_account + my_commit_url, my_ssh_pubkey = parse_commit_url_pub_key + acct_info = apply_my_account(my_ssh_pubkey) + + @my_info['my_commit_url'] = my_commit_url + @my_info['my_login_name'] = acct_info['my_login_name'] + @my_info['my_jumper_pubkey'] = acct_info['my_jumper_pubkey'] + + store_account_info + send_mail('') + rescue StandardError => e + puts e.message + puts e.backtrace + + send_mail(e.message) + end + + def parse_commit_url_pub_key + parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) + + my_commit_url = parse_apply_account_email.parse_commit_url + my_ssh_pubkey = parse_apply_account_email.parse_pub_key + + return my_commit_url, my_ssh_pubkey + end + + def apply_my_account(my_ssh_pubkey) + my_uuid = %x(uuidgen).chomp + + @my_info['my_uuid'] = my_uuid + + apply_account = ApplyJumperAccount.new(@my_info, my_ssh_pubkey) + acct_info = apply_account.apply_jumper_account + + return acct_info + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def send_mail(error_message) + email_message = if error_message.empty? + build_apply_account_email(@my_info) + else + build_apply_account_fail_email(@my_info, error_message) + end + + %x(curl -XPOST "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" -d "#{email_message}") + end +end -- 2.23.0

2 3

[PATCH v6 compass-ci 02/12] mail-robot: listen and handle new email
by Luan Shengde 13 Nov '20

13 Nov '20

listen the mailbox for new email handle_new_email handle the email with defined opetions - apply_account email subject: apply account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-robot.rb | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 lib/mail-robot.rb diff --git a/lib/mail-robot.rb b/lib/mail-robot.rb new file mode 100755 index 0000000..0ae80f3 --- /dev/null +++ b/lib/mail-robot.rb @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' + +def monitor_new_email(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + next if added.empty? + + added.each do |mail_file| + handle_new_email(mail_file, mail_drafts) + end + end + listener.start + sleep +end + +def handle_new_email(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + apply_account(mail_content) + + FileUtils.mv(mail_file, mail_drafts) +end + +def apply_account(mail_content) + return unless mail_content.subject == 'apply account' + + assign_uuid = AssignAccount.new(mail_content) + assign_uuid.send_account +end -- 2.23.0

2 4

[PATCH v2 compass-ci] container/srv-http: support display *.log file in browser
by Lu Weitao 13 Nov '20

13 Nov '20

[why] haven't define some file(*.log) behavior of browser in mime.types, so download this file on browser now. [how] 1. add correspondence of file in mime.types 2. set default type in default.conf of nginx, other type file that haven't define in mime.types, will display the file content in browser Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/root/etc/nginx/conf.d/default.conf | 1 + container/srv-http/root/etc/nginx/mime.types | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/container/srv-http/root/etc/nginx/conf.d/default.conf b/container/srv-http/root/etc/nginx/conf.d/default.conf index 9c15d99..91371f1 100644 --- a/container/srv-http/root/etc/nginx/conf.d/default.conf +++ b/container/srv-http/root/etc/nginx/conf.d/default.conf @@ -13,6 +13,7 @@ server { } location / { + default_type text/html; autoindex on; autoindex_localtime on; allow all; diff --git a/container/srv-http/root/etc/nginx/mime.types b/container/srv-http/root/etc/nginx/mime.types index e7f2f1c..a4ee74b 100644 --- a/container/srv-http/root/etc/nginx/mime.types +++ b/container/srv-http/root/etc/nginx/mime.types @@ -4,8 +4,8 @@ types { text/css css; text/xml xml; text/yaml yaml; - text/plain txt time data sh; + text/plain txt time data sh log; application/json json; - application/x-gzip gz tgz; + application/x-gzip gz tgz cgz; } -- 2.23.0

2 2

[PATCH v6 compass-ci 01/12] mail-robot: run mail-robot service
by Luan Shengde 13 Nov '20

13 Nov '20

call mail-robot.rb parse-apply-account-email.rb gitee-commit-url-check.rb assign-account.rb apply-jumper-account.rb assign-account-email.rb assign-account-fail-eamil.rb to run the email-robot service Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/run-mail-robot.rb | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 container/mail-robot/run-mail-robot.rb diff --git a/container/mail-robot/run-mail-robot.rb b/container/mail-robot/run-mail-robot.rb new file mode 100755 index 0000000..2048fda --- /dev/null +++ b/container/mail-robot/run-mail-robot.rb @@ -0,0 +1,19 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative "#{ENV['CCI_SRC']}/lib/assign-account-email" +require_relative "#{ENV['CCI_SRC']}/lib/assign-account-fail-eamil" +require_relative "#{ENV['CCI_SRC']}/lib/apply-jumper-account" +require_relative "#{ENV['CCI_SRC']}/lib/gitee-commit-url-check" +require_relative "#{ENV['CCI_SRC']}/lib/parse-apply-account-email" +require_relative "#{ENV['CCI_SRC']}/lib/assign-account" +require_relative "#{ENV['CCI_SRC']}/lib/mail-robot" + +MAILDIR = ENV['MAILDIR'] +JUMPER_HOST = ENV['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' +JUMPER_PORT = ENV['JUMPER_PORT'] || 29999 +SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 + +monitor_new_email("#{MAILDIR}/new", "#{MAILDIR}/cur") -- 2.23.0

2 6

[PATCH v6 compass-ci 3/3] kernel_version.md: change some detail to make this doc better
by Xu Xijian 13 Nov '20

13 Nov '20

1. add default vmlinuz in Related files example. 2. change the testbox's value and add queue key-value in submit cmd. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/kernel_version.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/job/kernel_version.md b/doc/job/kernel_version.md index f6a5f4d..3cf5e26 100644 --- a/doc/job/kernel_version.md +++ b/doc/job/kernel_version.md @@ -34,6 +34,7 @@ Related files: ├── modules-4.19.90-2003.cgz ├── modules.cgz -> modules-4.19.90-2003.cgz ├── vmlinuz-4.19.90-2003 +└── vmlinuz -> vmlinuz-4.19.90-2003 Usage example: -- submit iperf.yaml testbox=vm-2p8g--$USER os=openeuler os_arch=aarch64 os_version=20.03 runtime=20 kernel_version=4.19.90-2003 +- submit iperf.yaml testbox=vm-2p8g queue=vm-2p8g~$USER os=openeuler os_arch=aarch64 os_version=20.03 runtime=20 kernel_version=4.19.90-2003 -- 2.23.0

2 2

[PATCH v6 compass-ci 1/3] fix: avoid to parse iscsiroot in setup-dracut.sh
by Xu Xijian 13 Nov '20

13 Nov '20

[why] When start with a new kernel, the dracut will parse iscsiroot and halted, which is unnecessary for us. [errmsg] [ 0.976307] dracut: FATAL: iscsiroot requested but kernel/initrd does not support iscsi [ 0.977011] dracut: Refusing to continue ... [ 1.063137] reboot: System halted Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 2e676de..b9a7800 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -14,5 +14,6 @@ rm -rf /var/lib/apt/lists/* # Replace the runtime shell script with a custom shell script cp -a /usr/local/bin/cifs-lib.sh /usr/lib/dracut/modules.d/95cifs/ +sed -i '/inst_hook cmdline 90 "$moddir\/parse-iscsiroot.sh"/d' /usr/lib/dracut/modules.d/95iscsi/module-setup.sh cat overlay-lkp.sh >> /usr/lib/dracut/modules.d/90overlay-root/overlay-mount.sh -- 2.23.0

4 5

[PATCH v3 lkp-tests] test/cci-makepkg: fix a softlink will be established if the package fails
by Sun Yukui 13 Nov '20

13 Nov '20

[why] After packaging fails, a softlink will be established, but this is wrong. If the packaging fails, the softlink should not be established. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/cci-makepkg | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a105f189..c6fce675 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -92,10 +92,6 @@ cgz_name="${pkgver:-0}-${pkgrel:-0}.cgz" setup_proxy -echo $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf -PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf - update_softlink() { [ -e "$sync_dest/$cgz_name" ] || return @@ -109,4 +105,13 @@ update_softlink() rm $sync_dest/$cgz_name } -update_softlink && echo "Makepkg finished successfully" +build_source_pkg() +{ + echo $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf + PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" \ + $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf + + [ "$?" == 0 ] || update_softlink +} + +build_source_pkg -- 2.23.0

1 0

[PATCH lkp-tests] test/build-pkg: fix a softlink will be established if the package fails
by Sun Yukui 13 Nov '20

13 Nov '20

[why] After packaging fails, a softlink will be established, but this is wrong. If the packaging fails, the soft connection should not be established. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/build-pkg | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index ad8d4843..093bbe4d 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -65,6 +65,19 @@ request_pkg() [ -s PKGBUILD ] || die "PKGBUILD is empty" } +create_softlink() +{ + local soft_path="$PKG_MNT/${pack_to}/${pkgname}" + local bm_name=$(basename $(realpath ${cgz_name})) + + if [ -n "$upstream_tag" ]; then + echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" + fi + + echo "create soft link: latest.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/latest.cgz" +} build_source_pkg() { @@ -86,24 +99,11 @@ build_source_pkg() cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 -} -create_softlink() -{ - local soft_path="$PKG_MNT/${pack_to}/${pkgname}" - local bm_name=$(basename $(realpath ${cgz_name})) - - if [ -n "$upstream_tag" ]; then - echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" - fi - - echo "create soft link: latest.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/latest.cgz" + [ "$?" == 0 ] || create_softlink } check_vars mount_dest request_pkg build_source_pkg -create_softlink -- 2.23.0

2 1

[PATCH v4 lkp-tests] tats: add hackbench-git result parsing script
by Zhang Yale 13 Nov '20

13 Nov '20

[input] --------socket thread num=20-------- 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.495 ... 2020-11-10 19:51:38 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.512 [output] socket_thread_num_800: 0.5016 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/hackbench-git | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100755 stats/hackbench-git diff --git a/stats/hackbench-git b/stats/hackbench-git new file mode 100755 index 00000000..187f1c19 --- /dev/null +++ b/stats/hackbench-git @@ -0,0 +1,20 @@ +#!/usr/bin/env ruby + +LKP_SRC = ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require "#{LKP_SRC}/lib/statistics" + +time = [] +while (line = gets) + case line + when /^-+(\w+ \w+ \w+)/ + args = $1.tr(' ', '_') + when /^Running with .* $== (\d+)$ tasks/ + tasks = $1.to_i + when /^Time:/ + seconds = line.split[1] + time << seconds.to_f + end +end + +puts "#{args}_#{tasks}: #{time.sum / time.size}" -- 2.23.0

1 1