- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] container/srv-http: support display *.log file in browser
by Lu Weitao 13 Nov '20

13 Nov '20

[why] haven't define some file(*.log) behavior of browser in mime.types, so download this file on browser now. [how] 1. add correspondence of file in mime.types 2. set default type in default.conf of nginx, other type file that haven't define in mime.types, will display the file content in browser Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/root/etc/nginx/conf.d/default.conf | 1 + container/srv-http/root/etc/nginx/mime.types | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/container/srv-http/root/etc/nginx/conf.d/default.conf b/container/srv-http/root/etc/nginx/conf.d/default.conf index 9c15d99..91371f1 100644 --- a/container/srv-http/root/etc/nginx/conf.d/default.conf +++ b/container/srv-http/root/etc/nginx/conf.d/default.conf @@ -13,6 +13,7 @@ server { } location / { + default_type text/html; autoindex on; autoindex_localtime on; allow all; diff --git a/container/srv-http/root/etc/nginx/mime.types b/container/srv-http/root/etc/nginx/mime.types index e7f2f1c..a4ee74b 100644 --- a/container/srv-http/root/etc/nginx/mime.types +++ b/container/srv-http/root/etc/nginx/mime.types @@ -4,8 +4,8 @@ types { text/css css; text/xml xml; text/yaml yaml; - text/plain txt time data sh; + text/plain txt time data sh log; application/json json; - application/x-gzip gz tgz; + application/x-gzip gz tgz cgz; } -- 2.23.0

2 2

[PATCH v6 compass-ci 01/12] mail-robot: run mail-robot service
by Luan Shengde 13 Nov '20

13 Nov '20

call mail-robot.rb parse-apply-account-email.rb gitee-commit-url-check.rb assign-account.rb apply-jumper-account.rb assign-account-email.rb assign-account-fail-eamil.rb to run the email-robot service Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/run-mail-robot.rb | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 container/mail-robot/run-mail-robot.rb diff --git a/container/mail-robot/run-mail-robot.rb b/container/mail-robot/run-mail-robot.rb new file mode 100755 index 0000000..2048fda --- /dev/null +++ b/container/mail-robot/run-mail-robot.rb @@ -0,0 +1,19 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative "#{ENV['CCI_SRC']}/lib/assign-account-email" +require_relative "#{ENV['CCI_SRC']}/lib/assign-account-fail-eamil" +require_relative "#{ENV['CCI_SRC']}/lib/apply-jumper-account" +require_relative "#{ENV['CCI_SRC']}/lib/gitee-commit-url-check" +require_relative "#{ENV['CCI_SRC']}/lib/parse-apply-account-email" +require_relative "#{ENV['CCI_SRC']}/lib/assign-account" +require_relative "#{ENV['CCI_SRC']}/lib/mail-robot" + +MAILDIR = ENV['MAILDIR'] +JUMPER_HOST = ENV['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' +JUMPER_PORT = ENV['JUMPER_PORT'] || 29999 +SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 + +monitor_new_email("#{MAILDIR}/new", "#{MAILDIR}/cur") -- 2.23.0

2 6

[PATCH v6 compass-ci 3/3] kernel_version.md: change some detail to make this doc better
by Xu Xijian 13 Nov '20

13 Nov '20

1. add default vmlinuz in Related files example. 2. change the testbox's value and add queue key-value in submit cmd. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/kernel_version.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/job/kernel_version.md b/doc/job/kernel_version.md index f6a5f4d..3cf5e26 100644 --- a/doc/job/kernel_version.md +++ b/doc/job/kernel_version.md @@ -34,6 +34,7 @@ Related files: ├── modules-4.19.90-2003.cgz ├── modules.cgz -> modules-4.19.90-2003.cgz ├── vmlinuz-4.19.90-2003 +└── vmlinuz -> vmlinuz-4.19.90-2003 Usage example: -- submit iperf.yaml testbox=vm-2p8g--$USER os=openeuler os_arch=aarch64 os_version=20.03 runtime=20 kernel_version=4.19.90-2003 +- submit iperf.yaml testbox=vm-2p8g queue=vm-2p8g~$USER os=openeuler os_arch=aarch64 os_version=20.03 runtime=20 kernel_version=4.19.90-2003 -- 2.23.0

2 2

[PATCH v6 compass-ci 1/3] fix: avoid to parse iscsiroot in setup-dracut.sh
by Xu Xijian 13 Nov '20

13 Nov '20

[why] When start with a new kernel, the dracut will parse iscsiroot and halted, which is unnecessary for us. [errmsg] [ 0.976307] dracut: FATAL: iscsiroot requested but kernel/initrd does not support iscsi [ 0.977011] dracut: Refusing to continue ... [ 1.063137] reboot: System halted Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 2e676de..b9a7800 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -14,5 +14,6 @@ rm -rf /var/lib/apt/lists/* # Replace the runtime shell script with a custom shell script cp -a /usr/local/bin/cifs-lib.sh /usr/lib/dracut/modules.d/95cifs/ +sed -i '/inst_hook cmdline 90 "$moddir\/parse-iscsiroot.sh"/d' /usr/lib/dracut/modules.d/95iscsi/module-setup.sh cat overlay-lkp.sh >> /usr/lib/dracut/modules.d/90overlay-root/overlay-mount.sh -- 2.23.0

4 5

[PATCH v3 lkp-tests] test/cci-makepkg: fix a softlink will be established if the package fails
by Sun Yukui 13 Nov '20

13 Nov '20

[why] After packaging fails, a softlink will be established, but this is wrong. If the packaging fails, the softlink should not be established. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/cci-makepkg | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a105f189..c6fce675 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -92,10 +92,6 @@ cgz_name="${pkgver:-0}-${pkgrel:-0}.cgz" setup_proxy -echo $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf -PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf - update_softlink() { [ -e "$sync_dest/$cgz_name" ] || return @@ -109,4 +105,13 @@ update_softlink() rm $sync_dest/$cgz_name } -update_softlink && echo "Makepkg finished successfully" +build_source_pkg() +{ + echo $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf + PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" \ + $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf + + [ "$?" == 0 ] || update_softlink +} + +build_source_pkg -- 2.23.0

1 0

[PATCH lkp-tests] test/build-pkg: fix a softlink will be established if the package fails
by Sun Yukui 13 Nov '20

13 Nov '20

[why] After packaging fails, a softlink will be established, but this is wrong. If the packaging fails, the soft connection should not be established. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/build-pkg | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index ad8d4843..093bbe4d 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -65,6 +65,19 @@ request_pkg() [ -s PKGBUILD ] || die "PKGBUILD is empty" } +create_softlink() +{ + local soft_path="$PKG_MNT/${pack_to}/${pkgname}" + local bm_name=$(basename $(realpath ${cgz_name})) + + if [ -n "$upstream_tag" ]; then + echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" + fi + + echo "create soft link: latest.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/latest.cgz" +} build_source_pkg() { @@ -86,24 +99,11 @@ build_source_pkg() cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 -} -create_softlink() -{ - local soft_path="$PKG_MNT/${pack_to}/${pkgname}" - local bm_name=$(basename $(realpath ${cgz_name})) - - if [ -n "$upstream_tag" ]; then - echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" - fi - - echo "create soft link: latest.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/latest.cgz" + [ "$?" == 0 ] || create_softlink } check_vars mount_dest request_pkg build_source_pkg -create_softlink -- 2.23.0

2 1

[PATCH v4 lkp-tests] tats: add hackbench-git result parsing script
by Zhang Yale 13 Nov '20

13 Nov '20

[input] --------socket thread num=20-------- 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.495 ... 2020-11-10 19:51:38 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.512 [output] socket_thread_num_800: 0.5016 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/hackbench-git | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100755 stats/hackbench-git diff --git a/stats/hackbench-git b/stats/hackbench-git new file mode 100755 index 00000000..187f1c19 --- /dev/null +++ b/stats/hackbench-git @@ -0,0 +1,20 @@ +#!/usr/bin/env ruby + +LKP_SRC = ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require "#{LKP_SRC}/lib/statistics" + +time = [] +while (line = gets) + case line + when /^-+(\w+ \w+ \w+)/ + args = $1.tr(' ', '_') + when /^Running with .* $== (\d+)$ tasks/ + tasks = $1.to_i + when /^Time:/ + seconds = line.split[1] + time << seconds.to_f + end +end + +puts "#{args}_#{tasks}: #{time.sum / time.size}" -- 2.23.0

1 1

[PATCH v4 compass-ci 4/4] assign-account: build send account email
by Luan Shengde 13 Nov '20

13 Nov '20

build email message for successfuly assign account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../build-send-account-email.rb | 32 +++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100755 container/assign-account/build-send-account-email.rb diff --git a/container/assign-account/build-send-account-email.rb b/container/assign-account/build-send-account-email.rb new file mode 100755 index 0000000..3c41404 --- /dev/null +++ b/container/assign-account/build-send-account-email.rb @@ -0,0 +1,32 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_message(email, account_info) + message = <<~EMAIL_MESSAGE + To: #{email} + Subject: [compass-ci] jumper account is ready + + Dear user: + + Thank you for joining us. + You can use the following command to login the jumper server: + + Login command: + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} + + Account password: + #{account_info['my_password']} + + You can use your pub_key to login if you have offered one. + + Suggest: + If you use the password to login, change it in time. + + regards + compass-ci + EMAIL_MESSAGE + + return message +end -- 2.23.0

1 0

[PATCH v4 compass-ci 3/4] assign-account: generate ssh keys
by Luan Shengde 13 Nov '20

13 Nov '20

generate private/public key on jumper server and return public key [why] register the public key to es, final register it to testbox enable user one click to login in the testbox [how] generate rsa private/public key for user on jumper server return and register the public key to es Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 41 +++++++++++--------- 1 file changed, 22 insertions(+), 19 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 137f5c6..3cb583b 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -34,12 +34,13 @@ setup_jumper_account_info read_jumper_info config_default_yaml config_authorized_key + generate_ssh_key the returned data for setup_jumper_account_info like: { - "account" => "guest", - "passwd" => "Use pub_key to login", - "jumper_ip" => "10.10.10.10", + "my_login_name" => "test_user", + "password" => "Use pub_key to login", + "jumper_host" => "0.0.0.0", "jumper_port" => "10000" } @@ -94,30 +95,33 @@ class AccountStorage end def setup_jumper_account_info - account_info = read_account_info - jumper_info = read_jumper_info - pub_key = @data['my_ssh_pubkey'] unless @data['my_ssh_pubkey'].nil? + login_name, password = read_account_info + jumper_host, jumper_port = read_jumper_info + pub_key = @data['my_ssh_pubkey'][0] - login_name = account_info[0] - password = if pub_key.nil? - account_info[1] - else - 'Use pub_key to login' - end + ssh_dir = File.join('/home/', login_name, '.ssh') + config_authorized_key(login_name, pub_key, ssh_dir) unless pub_key.nil? + config_default_yaml(login_name) + my_jumper_pubkey = generate_ssh_key(login_name, ssh_dir) if @data['gen_sshkey'].eql? true jumper_account_info = { 'my_login_name' => login_name, 'my_password' => password, - 'jumper_host' => jumper_info[0].chomp, - 'jumper_port' => jumper_info[1].chomp + 'jumper_host' => jumper_host, + 'jumper_port' => jumper_port, + 'my_jumper_pubkey' => my_jumper_pubkey } - config_authorized_key(login_name, pub_key) unless pub_key.nil? - config_default_yaml(login_name) - return jumper_account_info end + def generate_ssh_key(login_name, ssh_dir) + Dir.mkdir ssh_dir, 0o700 unless File.exist? ssh_dir + %x(ssh-keygen -f "#{ssh_dir}/id_rsa" -N '' -C "#{login_name}@account-vm") + %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) + File.read("/home/#{login_name}/.ssh/id_rsa.pub") + end + def config_default_yaml(login_name) default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') FileUtils.mkdir_p default_yaml_dir @@ -132,8 +136,7 @@ class AccountStorage %x(chown -R #{login_name}:#{login_name} "/home/#{login_name}/.config") end - def config_authorized_key(login_name, pub_key) - ssh_dir = File.join('/home/', login_name, '.ssh') + def config_authorized_key(login_name, pub_key, ssh_dir) Dir.mkdir ssh_dir, 0o700 Dir.chdir ssh_dir f = File.new('authorized_keys', 'w') -- 2.23.0

1 0

[PATCH v4 compass-ci 2/4] assign-account: add new option to generate ssh key
by Luan Shengde 13 Nov '20

13 Nov '20

add -g|--gen-sshkey option for answerback-email [why] enable one click login testbox from jumper server [how] add add -g|--gen-sshkey option to enable generate private/public key on jumper server register the public key to es Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 81 ++++++++++---------- 1 file changed, 40 insertions(+), 41 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 686a327..cf1b852 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -7,12 +7,22 @@ # samba mount # ssh logshn (huawei, ) (install pubkey / send password) +# entry point1: external user: email apply => mail robot +# entry point2: ./answerback-email: cli apply +# => params +# => REST api (params different) +# => robot email notify +# user_type: +# apply_from: mail/cli +# allow_login: + require 'json' require 'mail' require 'set' require 'optparse' require_relative '../defconfig' require_relative '../../lib/es_client' +require_relative 'build-send-account-email' names = Set.new %w[ JUMPER_HOST @@ -31,41 +41,49 @@ SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 my_info = { 'my_email' => nil, 'my_name' => nil, + 'my_commit_url' => nil, 'my_uuid' => %x(uuidgen).chomp, - 'my_ssh_pubkey' => nil + 'my_ssh_pubkey' => [], + 'gen_sshkey' => false } -def init_info(email_file, my_info) - mail_content = Mail.read(email_file) +def init_info(mail_content, my_info) my_info['my_email'] = mail_content.from[0] my_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') - my_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' - mail_content.part[1].body.decoded - end + return if mail_content.attachments.empty? + + my_info['my_ssh_pubkey'] << mail_content.attachments[0].body.decoded end options = OptionParser.new do |opts| opts.banner = 'Usage: answerback-mail.rb [-e|--email email] ' - opts.banner += "[-s|--ssh-pubkey pub_key_file] [-f|--raw-email email_file]\n" + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-f|--raw-email email_file] [-g|--gen-sshkey]\n" opts.banner += " -e or -f is required\n" - opts.banner += ' -s is optional when use -e' + opts.banner += " -s is optional when use -e\n" + opts.banner += ' -g is optional, used to generate sshkey for user' opts.separator '' opts.separator 'options:' opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| my_info['my_email'] = email_address - # when apply account with email address, will get no user name + # when apply account with email address, will get no user name and commit url my_info['my_name'] = '' + my_info['my_commit_url'] = '' end opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ 'ssh pub_key file, enable password-less login') do |pub_key_file| - my_info['my_ssh_pubkey'] = File.read(pub_key_file) + my_info['my_ssh_pubkey'] << File.read(pub_key_file).chomp end opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| - init_info(email_file, my_info) + mail_content = Mail.read(email_file) + init_info(mail_content, my_info) + end + + opts.on('-g', '--gen-sshkey', 'generate jumper ras public/private key and return pubkey') do + my_info['gen_sshkey'] = true end opts.on_tail('-h', '--help', 'show this message') do @@ -76,32 +94,6 @@ end options.parse!(ARGV) -def build_message(email, account_info) - message = <<~EMAIL_MESSAGE - To: #{email} - Subject: [compass-ci] jumper account is ready - - Dear user: - - Thank you for joining us. - You can use the following command to login the jumper server: - - Login command: - ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} - - Account password: - #{account_info['my_password']} - - Suggest: - If you use the password to login, change it in time. - - regards - compass-ci - EMAIL_MESSAGE - - return message -end - def apply_account(my_info) account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{my_info.to_json}') JSON.parse account_info_str @@ -114,12 +106,19 @@ def send_account(my_info) raise message if my_info['my_email'].nil? account_info = apply_account(my_info) - # for manually assign account, there will be no my_commit_url - # but the key my_commit_url is required for es - my_info['my_commit_url'] = '' my_info['my_login_name'] = account_info['my_login_name'] - my_info.delete 'my_ssh_pubkey' + + unless account_info['my_jumper_pubkey'].nil? + my_info['my_ssh_pubkey'] << account_info['my_jumper_pubkey'].chomp + end + + my_info.delete 'gen_sshkey' store_account_info(my_info) + + send_mail(my_info, account_info) +end + +def send_mail(my_info, account_info) message = build_message(my_info['my_email'], account_info) %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") -- 2.23.0

1 0