- Compass-ci - mailweb.openeuler.org

[PATCH v8 compass-ci 08/11] mail-robot: Dockerfile
by Luan Shengde 17 Nov '20

17 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..76dc94d --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install fileutils elasticsearch activesupport -- 2.23.0

1 0

[PATCH v8 compass-ci 07/11] mail-robot: build fail message
by Luan Shengde 17 Nov '20

17 Nov '20

build fail email message with raised error message - lack commit url - repo url for the commit url not in upstream - commit not exist or not available - lack pubkey - no more available account on jumper server Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/assign-account-fail-eamil.rb | 38 +++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100755 container/mail-robot/lib/assign-account-fail-eamil.rb diff --git a/container/mail-robot/lib/assign-account-fail-eamil.rb b/container/mail-robot/lib/assign-account-fail-eamil.rb new file mode 100755 index 0000000..769853f --- /dev/null +++ b/container/mail-robot/lib/assign-account-fail-eamil.rb @@ -0,0 +1,38 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_fail_email(my_info, message) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] apply account failed + + Dear user: + + Your application for account failed with following error: + + #{message} + + In order to successfully apply an account, please pay attention to the following points: + + 1. mail subject + The subject should exactly: apply account + + 2. commit url + When you writing the url, add prefix: my oss commit + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + + Attention: + Ensure you commit url exist and available to access. + + 3. ssh pubkey + You need to add a pubkey as an attachment to the email. + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

1 0

[PATCH v8 compass-ci 06/11] mail-robot: build success email
by Luan Shengde 17 Nov '20

17 Nov '20

build email message for successfully assigning account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/assign-account-email.rb | 53 +++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100755 container/mail-robot/lib/assign-account-email.rb diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb new file mode 100755 index 0000000..e882bb4 --- /dev/null +++ b/container/mail-robot/lib/assign-account-email.rb @@ -0,0 +1,53 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_email(my_info) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] Account Ready + + Dear #{my_info['my_name']}, + + Thank you for joining us. + + You can use the following info to submit jobs: + + notice: + the followings are ONE-TIME setup + + 1) setup default config + run the following command to add the below setup to default config file + + mkdir -p ~/.config/compass-ci/defaults/ + cat >> ~/.config/compass-ci/defaults/account.yaml <<-EOF + my_email: #{my_info['my_email']} + my_name: #{my_info['my_name']} + my_uuid: #{my_info['my_uuid']} + EOF + + 2) download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration + + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source ${HOME}/.bashrc && source ${HOME}/.bash_profile + + 3) submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + + reference to 'how to write job yaml' section to write the job yaml + you can also reference to files in compass-ci/jobs as example. + + submit jobs, for example: + + submit -m iperf.yaml testbox=vm-2p8g queue=vm-2p8g~\\$USER + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

1 0

[PATCH v8 compass-ci 05/11] mail-robot: apply new account for user
by Luan Shengde 17 Nov '20

17 Nov '20

ApplyJumperAccount used to apply account for user when successfully parsed my_commit_url and my_ssh_pubkey input: my_info apply_jumper_account apply account with my_info my_info: - my_email - my_name - my_uuid - my_ssh_pubkey Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/apply-jumper-account.rb | 42 +++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100755 container/mail-robot/lib/apply-jumper-account.rb diff --git a/container/mail-robot/lib/apply-jumper-account.rb b/container/mail-robot/lib/apply-jumper-account.rb new file mode 100755 index 0000000..44f412b --- /dev/null +++ b/container/mail-robot/lib/apply-jumper-account.rb @@ -0,0 +1,42 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' + +JUMPER_HOST = ENV['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' +JUMPER_PORT = ENV['JUMPER_PORT'] || 29999 + +# used to apply account +# be called after AssignAccount successfully parsed my_commit_url and my_ssh_pubkey +# apply_jumper_account +# apply jumper account with my_info and my_ssh_pubkey +# account_info_exist +# check account exists +class ApplyJumperAccount + def initialize(my_info) + @jumper_host = JUMPER_HOST + @jumper_port = JUMPER_PORT + @my_info = my_info.clone + end + + def apply_jumper_account + account_info_str = %x(curl -XGET "#{@jumper_host}:#{@jumper_port}/assign_account" \ + -d '#{(a)my_info.to_json}') + account_info = JSON.parse account_info_str + + account_info_exist(account_info) + + return account_info + end + + def account_info_exist(account_info) + return unless account_info['my_login_name'].nil? + + error_message = 'No more available jumper account.' + error_message += 'You may try again later or consulting the manager for a solution.' + + raise error_message + end +end -- 2.23.0

1 0

[PATCH v8 compass-ci 03/11] mail-robot: parse apply account email
by Luan Shengde 17 Nov '20

17 Nov '20

ParseApplyAccountEmail used to parse my_email, my_name, my_commit_url and my_ssh_pubkey when execute check_to_send_account in ApplyAccount input: mail_content output: my_info init my_info parse_commit_url extract_commit_url extract my_commit_url from mail_content base_url_in_upstream_repos check whether the repo's url in upstream-repos commit_url_availability check the commit available - gitee.com call GiteeCommitUrlCheck to check commit available - non-gitee.com execute curl and check feedback for the commit parse_pub_key check attachment for my_ssh_pubkey Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/parse-apply-account-email.rb | 134 ++++++++++++++++++ 1 file changed, 134 insertions(+) create mode 100755 container/mail-robot/lib/parse-apply-account-email.rb diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb new file mode 100755 index 0000000..5467728 --- /dev/null +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -0,0 +1,134 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative 'gitee-commit-url-check' + +# used to parse mail_content for my_commit_url and my_ssh_pubkey +# be called by AssignAccount when it needs to extract required data: +# - my_commit_url +# - my_ssh_pubkey +# +# input: mail_content +# output: my_commit_url, my_ssh_pubkey +# +# parse_commit_url +# parse my_commit_url from the mail_content and return it +# extract my_commit_url +# extract my_coomit_url from the mail_content +# check commit url exists +# base_url_in_upstream_repos +# check whether the base url in upstream-repos +# commit_url_availability +# check commit available +# gitee.com: +# call GiteeCommitUrlCheck to check the commit +# non-gitee.com: +# execute curl to check the commit +# parse_pub_key +# check my_ssh_pubkey exists and return it +class ParseApplyAccountEmail + def initialize(mail_content) + @mail_content = mail_content + + @my_info = { + 'my_email' => mail_content.from[0], + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, ''), + 'my_ssh_pubkey' => [] + } + end + + def build_my_info + @my_info['my_commit_url'] = parse_commit_url + @my_info['my_ssh_pubkey'] << parse_pub_key + + return @my_info + end + + def extract_commit_url + mail_content_body = @mail_content.part[0].part[0].body.decoded || @mail_content.part[0].body.decoded + mail_content_line = mail_content_body.gsub(/\n/, '') + + # the commit url should be headed with a prefix: my oss commit + # the commit url must be in a standart format, example: + # my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + no_commit_url unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) + + mail_content_body.match(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}})[0] + end + + def no_commit_url + error_message = "No matched commit url found.\n" + error_message += 'Ensure that you have add a right commit url, ' + error_message += "and with prefix 'my oss commit:'." + + raise error_message + end + + def parse_commit_url + url = extract_commit_url + base_url = url.gsub(%r{/commit/[\w\d]{40}$}, '') + + base_url_in_upstream_repos('/c/upstream-repos', base_url) + commit_url_availability(url, base_url) + + return url + end + + def base_url_in_upstream_repos(upstream_dir, base_url) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + return unless match_out.empty? + + error_message = "The repo url for your commit is not in our upstream-repo list.\n" + error_message += 'Use a new one, or consulting the manager for available repo list.' + + raise error_message + end + + def commit_url_availability(url, base_url) + hub_name = url.split('/')[2] + + # it requires authentication when execute curl to get the commit information + # clone the repo and then validate the commit for the email address + if hub_name.eql? 'gitee.com' + gitee_commit(url, base_url) + else + non_gitee_commit(url) + end + end + + def gitee_commit(url, base_url) + my_gitee_commit = GiteeCommitUrlCheck.new(@my_info, url, base_url) + my_gitee_commit.gitee_commit_check + end + + def non_gitee_commit(url) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index @my_info['my_email'] + + return unless email_index.nil? + + error_message = "We can not confirm the commit url matches your email.\n" + error_message += 'Make sure that the commit url is right,' + error_message += ' or it is truely submitted with you email.' + + raise error_message + end + + def parse_pub_key + error_message = "No pub_key found.\n" + error_message += 'Please add a pub_key as an attachment to your email.' + + raise error_message if @mail_content.attachments.empty? + raise error_message unless @mail_content.attachments[0].filename =~ /^id_.*\.pub$/ + + pub_key = @mail_content.attachments[0].body.decoded + + return pub_key + end +end -- 2.23.0

1 0

[PATCH v8 compass-ci 02/11] mail-robot: check to apply account
by Luan Shengde 17 Nov '20

17 Nov '20

ApplyAccount check to apply account for user when listened new email with subject 'apply account' input: mail_content check_to_send_account parse_mail_content update my_info apply_my_account generate uuid apply account with my_info my_info: - my_email - my_name - my_uuid - my_ssh_pubkey store account info to es send email send success email send fail email Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 110 ++++++++++++++++++++++ 1 file changed, 110 insertions(+) create mode 100755 container/mail-robot/lib/apply-account.rb diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb new file mode 100755 index 0000000..bf40429 --- /dev/null +++ b/container/mail-robot/lib/apply-account.rb @@ -0,0 +1,110 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative '../../../lib/es_client' +require_relative 'assign-account-email' +require_relative 'assign-account-fail-eamil' +require_relative 'apply-jumper-account' +require_relative 'parse-apply-account-email' + +SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 + +# assign uuid/account for user +# when mail-robot listened new email, and the email's subject +# exactly equal 'apply account', mail-robot will call this class +# entry point: send_account +# input: email's content +# +# send_account +# parse_mail_content +# call ParseApplyAccountEmail to parse: +# - my_commit_url +# - my_ssh_pubkey +# apply_my_account +# call ApplyJumperAccount to apply new account +# required data: +# my_info: +# - my_email +# - my_name +# - my_uuid +# my_ssh_pubkey +# store_account_info +# call ESClient to store my_info +# my_info: +# - my_email +# - my_name +# - my_uuid +# - my_commit_url +# - my_login_name +# - my_ssh_pubkey +# send_mail +# when successfully applied an account +# call build_apply_account_email to send a successful email +# when rescued error message +# call build_apply_account_fail_email to send fail email +class ApplyAccount + def initialize(mail_content) + @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp + @send_mail_port = SEND_MAIL_PORT + @mail_content = mail_content + + @my_info = {} + end + + def check_to_send_account + # in case of failed parsing, parse_mail_content will return none info + # in order to successfully send email for failed parsing + # firstly get my_email before execute parse_mail_content is needed + @my_info['my_email'] = @mail_content.from[0] + parse_mail_content + acct_info = apply_my_account + + @my_info['my_login_name'] = acct_info['my_login_name'] + + store_account_info + send_mail('') + rescue StandardError => e + puts e.message + puts e.backtrace + + send_mail(e.message) + end + + def parse_mail_content + parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) + + parsed_email_info = parse_apply_account_email.build_my_info + + @my_info.update parsed_email_info + end + + def apply_my_account + my_uuid = %x(uuidgen).chomp + + @my_info['my_uuid'] = my_uuid + + apply_account = ApplyJumperAccount.new(@my_info) + acct_info = apply_account.apply_jumper_account + + return acct_info + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def send_mail(error_message) + email_message = if error_message.empty? + build_apply_account_email(@my_info) + else + build_apply_account_fail_email(@my_info, error_message) + end + + %x(curl -XPOST "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" -d "#{email_message}") + end +end -- 2.23.0

1 0

[PATCH v8 compass-ci 01/11] mail-robot: listen and handle new email
by Luan Shengde 17 Nov '20

17 Nov '20

listen the mailbox for new email handle_new_email handle the email with defined opetions - apply_account email subject: apply account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/mail-robot.rb | 41 ++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100755 container/mail-robot/mail-robot.rb diff --git a/container/mail-robot/mail-robot.rb b/container/mail-robot/mail-robot.rb new file mode 100755 index 0000000..a809eae --- /dev/null +++ b/container/mail-robot/mail-robot.rb @@ -0,0 +1,41 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' +require_relative 'lib/apply-account' + +MAILDIR = ENV['MAILDIR'] + +def monitor_new_email(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + next if added.empty? + + added.each do |mail_file| + handle_new_email(mail_file, mail_drafts) + end + end + listener.start + sleep +end + +def handle_new_email(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + apply_account(mail_content) + + FileUtils.mv(mail_file, mail_drafts) +end + +def apply_account(mail_content) + return unless mail_content.subject == 'apply account' + + assign_uuid = ApplyAccount.new(mail_content) + assign_uuid.check_to_send_account +end + +monitor_new_email("#{MAILDIR}/new", "#{MAILDIR}/cur") -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/2] spec/stats/iozone: add test case for iozone
by Lu Kaiyi 17 Nov '20

17 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- spec/stats/iozone/1 | 61 ++++++++++++++++++++++++++++++++++++++++ spec/stats/iozone/1.yaml | 14 +++++++++ 2 files changed, 75 insertions(+) create mode 100644 spec/stats/iozone/1 create mode 100644 spec/stats/iozone/1.yaml diff --git a/spec/stats/iozone/1 b/spec/stats/iozone/1 new file mode 100644 index 00000000..58f2fcb0 --- /dev/null +++ b/spec/stats/iozone/1 @@ -0,0 +1,61 @@ +==> /tmp/stdout <== + +==> /tmp/stderr <== + +==> /tmp/stdout <== +RESULT_ROOT=/result/iozone/taishan200-2280-2s64p-256g--a9/2020-11-17/z9.151579 +job=/lkp/scheduled/job.yaml +result_service=raw_upload, RESULT_MNT=/172.168.131.2/result, RESULT_ROOT=/172.168.131.2/result/iozone/taishan200-2280-2s64p-256g--a9/2020-11-17/z9.151579 +run-job /lkp/scheduled/job.yaml +/usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://172.168.131.2:3000/~lkp/cgi-bin/lkp-jobfile-append-var?job_file=/lkp… -O /dev/null +target ucode: +2020-11-17 14:30:52 dmsetup remove_all +2020-11-17 14:30:52 wipefs -a --force /dev/sda +/dev/sda: 4 bytes were erased at offset 0x00000000 (xfs): 58 46 53 42 +2020-11-17 14:30:52 mkfs -t xfs -f /dev/sda +meta-data=/dev/sda isize=512 agcount=4, agsize=73257062 blks + = sectsz=512 attr=2, projid32bit=1 + = crc=1 finobt=1, sparse=1, rmapbt=0 + = reflink=0 +data = bsize=4096 blocks=293028246, imaxpct=5 + = sunit=0 swidth=0 blks +naming =version 2 bsize=4096 ascii-ci=0, ftype=1 +log =internal log bsize=4096 blocks=143080, version=2 + = sectsz=512 sunit=0 blks, lazy-count=1 +realtime =none extsz=4096 blocks=0, rtextents=0 +2020-11-17 14:30:55 mkdir -p /fs/sda +2020-11-17 14:30:55 modprobe xfs +2020-11-17 14:30:56 mount -t xfs -o inode64 /dev/sda /fs/sda +2020-11-17 14:30:58 iozone -r 256k -s 4g -i 0 -i 1 -i 2 -f /fs/sda/iozonetest & +2020-11-17 14:30:58 wait +perf version 4.19.90-2003.4.0.0036.oe1.aarch64 + Iozone: Performance Test of File I/O + Version $Revision: 3.479 $ + Compiled for 64 bit mode. + Build: linux + + Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins + Al Slater, Scott Rhine, Mike Wisner, Ken Goss + Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, + Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, + Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, + Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, + Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, + Vangel Bojaxhi, Ben England, Vikentsi Lapa, + Alexey Skidanov. + + Run began: Tue Nov 17 14:30:58 2020 + + Record Size 256 kB + File size set to 4194304 kB + Command line used: iozone -r 256k -s 4g -i 0 -i 1 -i 2 -f /fs/sda/iozonetest + Output is in kBytes/sec + Time Resolution = 0.000001 seconds. + Processor cache size set to 1024 kBytes. + Processor cache line size set to 32 bytes. + File stride size set to 17 * record size. + random random bkwd record stride + kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread + 4194304 256 14257802 15088999 8516076 8375138 5221373 5382018 + +iozone test complete. diff --git a/spec/stats/iozone/1.yaml b/spec/stats/iozone/1.yaml new file mode 100644 index 00000000..3ddec7fb --- /dev/null +++ b/spec/stats/iozone/1.yaml @@ -0,0 +1,14 @@ +suite: iozone +category: benchmark +file_size: 4g +test: write, read, rand_rw +block_size: 256k +disk: 1HDD + +fs: +- xfs + +iosched: +- kyber + +iozone: -- 2.23.0

1 0

[PATCH v5 lkp-tests] distro/archlinux: add archlinux distro file
by Wang Yong 17 Nov '20

17 Nov '20

add archlinux pack method Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/archlinux | 111 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100755 distro/archlinux diff --git a/distro/archlinux b/distro/archlinux new file mode 100755 index 000000000..408c5fa23 --- /dev/null +++ b/distro/archlinux @@ -0,0 +1,111 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +shopt -s nullglob + +. $LKP_SRC/lib/install.sh +. $LKP_SRC/distro/common +. $LKP_SRC/lib/debug.sh + +pack_benchmark_deps() +{ + setup_proxy + + for BM_NAME in $benchmark + do + + cleanup_downloaded_pkgs + + check_shared_package $BM_NAME + + packages=$(echo $(get_dependency_packages $DISTRO $BM_NAME)) + + if pacman_download $packages; then + pacman_pack + else + echo "failed to pack-deps $BM_NAME" >&2 + fi + done +} + +cleanup_downloaded_pkgs() +{ + [[ -d /var/cache/pacman/pkg/ ]] && { + find_pacman_package | xargs rm -f + } +} + +find_pacman_package() +{ + find /var/cache/pacman/pkg/ -type f -name "*.pkg.tar*" +} + +pacman_download() +{ + echo "pacman -Sw --noconfirm $*" + pacman -Sw --noconfirm "$@" +} + +pacman_pack() +{ + local target_dir=/opt/pkgs + local date=$(date +"%Y%m%d") + + local downloaded_pkgs=$(find_pacman_package) + + [ "$downloaded_pkgs" ] || return + + mkdir -p $target_dir + + mv $downloaded_pkgs $target_dir + + find $target_dir | cpio --quiet -o -H newc --owner=root.root | gzip -n -9 >$pack_to/${BM_NAME}_$date.cgz + + ln -sf ${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return + chown .lkp $pack_to/${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return + echo package installed to $pack_to/${BM_NAME}.cgz + + ls $target_dir/*.pkg.tar* > $pack_to/.${BM_NAME}.packages + + rm -rf $target_dir +} + +distro_install_depends() +{ + local script + local bm="$1" + local scripts=$(find $LKP_SRC/distro/depends/ -name "$bm" -o -name "${bm}.[0-9]") + + for script in $scripts + do + script=$(basename $script) + packages=$(get_dependency_packages $DISTRO $script) + + [ -z "$packages" ] && continue + + echo install packages for $script: $packages + + for pkg in $packages + do + pacman -Qqs --noconfirm $pkg || pacman -Sy --noconfirm $pkg + done + done +} + +pack_benchmark() +{ + setup_proxy + + distro_install_depends lkp-dev + + # Process each benchmark + for BM_NAME in $benchmark + do + distro_install_depends $BM_NAME-dev || continue + + echo $LKP_SRC/sbin/pack -d $DISTRO -f -c -s $PKG_MNT/$pack_to $BM_NAME + ( + $LKP_SRC/sbin/pack -d $DISTRO -f -c -s $PKG_MNT/$pack_to $BM_NAME + ) + done +} -- 2.23.0

1 0

[PATCH v4 lkp-tests] distro/archlinux: add archlinux distro file
by Wang Yong 17 Nov '20

17 Nov '20

add archlinux pack method Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/archlinux | 111 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100755 distro/archlinux diff --git a/distro/archlinux b/distro/archlinux new file mode 100755 index 000000000..e8e950726 --- /dev/null +++ b/distro/archlinux @@ -0,0 +1,111 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +shopt -s nullglob + +. $LKP_SRC/lib/install.sh +. $LKP_SRC/distro/common +. $LKP_SRC/lib/debug.sh + +distro_install_depends() +{ + local script + local bm="$1" + local scripts=$(find $LKP_SRC/distro/depends/ -name "$bm" -o -name "${bm}.[0-9]") + + for script in $scripts + do + script=$(basename $script) + packages=$(get_dependency_packages $DISTRO $script) + + [ -z "$packages" ] && continue + + echo install packages for $script: $packages + + for pkg in $packages + do + pacman -Qqs --noconfirm $pkg || pacman -Sy --noconfirm $pkg + done + done +} + +pack_benchmark() +{ + setup_proxy + + distro_install_depends lkp-dev + + # Process each benchmark + for BM_NAME in $benchmark + do + distro_install_depends $BM_NAME-dev || continue + + echo $LKP_SRC/sbin/pack -d $DISTRO -f -c -s $PKG_MNT/$pack_to $BM_NAME + ( + $LKP_SRC/sbin/pack -d $DISTRO -f -c -s $PKG_MNT/$pack_to $BM_NAME + ) + done +} + +pacman_download() +{ + echo "pacman -Sw --noconfirm $*" + pacman -Sw --noconfirm "$@" +} + +pacman_pack() +{ + local target_dir=/opt/pkgs + local date=$(date +"%Y%m%d") + + local downloaded_pkgs=$(find_pacman_package) + + [ "$downloaded_pkgs" ] || return + + mkdir -p $target_dir + + mv $downloaded_pkgs $target_dir + + find $target_dir | cpio --quiet -o -H newc --owner=root.root | gzip -n -9 >$pack_to/${BM_NAME}_$date.cgz + + ln -sf ${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return + chown .lkp $pack_to/${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return + echo package installed to $pack_to/${BM_NAME}.cgz + + ls $target_dir/*.pkg.tar* > $pack_to/.${BM_NAME}.packages + + rm -rf $target_dir +} + +cleanup_downloaded_pkgs() +{ + [[ -d /var/cache/pacman/pkg/ ]] && { + find_pacman_package | xargs rm -f + } +} + +pack_benchmark_deps() +{ + setup_proxy + + for BM_NAME in $benchmark + do + + cleanup_downloaded_pkgs + + check_shared_package $BM_NAME + + packages=$(echo $(get_dependency_packages $DISTRO $BM_NAME)) + + if pacman_download $packages; then + pacman_pack + else + echo "failed to pack-deps $BM_NAME" >&2 + fi + done +} + +find_pacman_package() +{ + find /var/cache/pacman/pkg/ -type f -name "*.pkg.tar*" +} -- 2.23.0

2 2

[PATCH v2 lkp-tests 1/2] lib: add multi-args in job yaml
by Bai Jing 17 Nov '20

17 Nov '20

[original style] os: openeuler os_version: 20.03 os_arch: aarch64 os: openeuler os_version: 20 os_arch: aarch64 [now style] os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 [point] Currently, only one key of this style(xx xxx xx) can be contained in YAML files. Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 01719c2e..f3b7a092 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -250,7 +250,7 @@ class Job hash.delete('override') end hash.merge!(@overrides) - @jobs << hash + @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end rescue StandardError => e log_error "#{jobfile}: " + e.message @@ -280,6 +280,30 @@ class Job @jobfile = jobfile end + def multi_args(hash) + jobs_array = [] + hash.each { |key, value| + next unless key =~ /^\w.*\s+.*\w$/ + + next unless value.class == Array + + key_array = key.split + value.each do |v| + v_array = v.split + next unless key_array.size == v_array.size + + hash_job = hash.clone + key_array.size.times do |i| + hash_job[key_array[i]] = v_array[i] + end + hash_job.delete(key) + jobs_array << hash_job + end + } + jobs_array = [hash] if jobs_array.empty? + return jobs_array + end + def delete_keys_from_spec(spec_file) return unless File.exist? spec_file -- 2.23.0

2 3

[PATCH v2 lkp-tests 3/4] test/cci-makepkg: Refactor encapsulate the statement as a function
by Sun Yukui 17 Nov '20

17 Nov '20

[why] Encapsulating statements as functions is important for a better syntax. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/cci-makepkg | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a105f189..a87cf01f 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -92,10 +92,6 @@ cgz_name="${pkgver:-0}-${pkgrel:-0}.cgz" setup_proxy -echo $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf -PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf - update_softlink() { [ -e "$sync_dest/$cgz_name" ] || return @@ -109,4 +105,11 @@ update_softlink() rm $sync_dest/$cgz_name } +build_source_pkg(){ + echo $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf + PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" \ + $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf +} + +build_source_pkg update_softlink && echo "Makepkg finished successfully" -- 2.23.0

2 2

[PATCH lkp-tests] update command parameters
by Hu Xuejiao 17 Nov '20

17 Nov '20

[why] The test date is incorrect and the parameters need to be adjusted Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/sysbench-threads-git | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/sysbench-threads-git b/tests/sysbench-threads-git index df1bb5cf..be83b441 100755 --- a/tests/sysbench-threads-git +++ b/tests/sysbench-threads-git @@ -10,7 +10,7 @@ : "${thread_locks:=2}" args=( - --threads=$nr_threads + --num-threads=$nr_threads --thread-yields=$thread_yields --thread-locks=$thread_locks ) -- 2.23.0

2 1

[PATCH v2 lkp-tests 2/2] job: fix each_jobs to fit multi-args
by Bai Jing 17 Nov '20

17 Nov '20

[why] if you add job field like this : os: openeuler os_version: 20.03 os_arch: aarch64 nr_threads: - 1 - 2 then the @jobs=[hash], hash = {"nr_threads"=>[1,2],"os"=>"openeuler","os_arch"=>"aarch64","os_version"=>"20.03"} if you add job field like this: os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 nr_threads: - 1 - 2 then the @jobs=[h1, h2], h1={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20.03", "os_arch"=>"aarch64"} h2={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20", "os_arch"=>"aarch64"} h1 will split to 2 jobs by function "each_job" Should: job1={"nr_threads"=>1} job2={"nr_threads"=>2} But if "@job2=hash",will get: job1={"nr_threads"=>[1, 2]} job2={"nr_threads"=>[1, 2]} We need to merge the h2 to @job Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index f3b7a092..f8608e94 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -727,7 +727,8 @@ class Job @jobs.each do |hash| @load_hosts_done = false @job = deepcopy(job) - @job2 = hash + @job.merge!(hash) + @job2 = {} load_defaults load_hosts_config each_job_init -- 2.23.0

1 0

[PATCH lkp-tests] job: fix each_jobs to fit multi-args
by Bai Jing 17 Nov '20

17 Nov '20

[why] if you add job field like this : os: openeuler os_version: 20.03 os_arch: aarch64 nr_threads: - 1 - 2 then the @jobs=[hash], hash = {"nr_threads"=>[1,2],"os"=>"openeuler","os_arch"=>"aarch64","os_version"=>"20.03"} if you add job field like this: os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 nr_threads: - 1 - 2 then the @jobs=[h1, h2], h1={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20.03", "os_arch"=>"aarch64"} h2={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20", "os_arch"=>"aarch64"} h1 will split to 2 jobs by function "each_job" Should: job1={"nr_threads"=>1} job2={"nr_threads"=>2} But if "@job2=hash",will get: job1={"nr_threads"=>[1, 2]} job2={"nr_threads"=>[1, 2]} We need to merge the h2 to @job Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index f3b7a092..f8608e94 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -727,7 +727,8 @@ class Job @jobs.each do |hash| @load_hosts_done = false @job = deepcopy(job) - @job2 = hash + @job.merge!(hash) + @job2 = {} load_defaults load_hosts_config each_job_init -- 2.23.0

1 1

[PATCH v2 lkp-tests] update command parameters
by Hu Xuejiao 17 Nov '20

17 Nov '20

[why] The current result: value of "total_time" is the same for different threads Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/sysbench-threads-git | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/sysbench-threads-git b/tests/sysbench-threads-git index df1bb5cf..be83b441 100755 --- a/tests/sysbench-threads-git +++ b/tests/sysbench-threads-git @@ -10,7 +10,7 @@ : "${thread_locks:=2}" args=( - --threads=$nr_threads + --num-threads=$nr_threads --thread-yields=$thread_yields --thread-locks=$thread_locks ) -- 2.23.0

2 1

[PATCH v2 lkp-tests] pkg: add mysql
by Lin Jiaxin 17 Nov '20

17 Nov '20

sysbench-mysql need to test with mysql8.0.17 Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- ...00-mysql-add-fstack-protector-strong.patch | 20 ++++++ pkg/mysql/PKGBUILD | 50 ++++++++++++++ pkg/mysql/my.cnf | 67 +++++++++++++++++++ 3 files changed, 137 insertions(+) create mode 100644 pkg/mysql/0000-mysql-add-fstack-protector-strong.patch create mode 100644 pkg/mysql/PKGBUILD create mode 100644 pkg/mysql/my.cnf diff --git a/pkg/mysql/0000-mysql-add-fstack-protector-strong.patch b/pkg/mysql/0000-mysql-add-fstack-protector-strong.patch new file mode 100644 index 00000000..23c07c87 --- /dev/null +++ b/pkg/mysql/0000-mysql-add-fstack-protector-strong.patch @@ -0,0 +1,20 @@ +diff --git a/cmake/build_configurations/compiler_options.cmake b/cmake/build_configurations/compiler_options.cmake +index fecf2a2..f3e012f 100644 +--- a/cmake/build_configurations/compiler_options.cmake ++++ b/cmake/build_configurations/compiler_options.cmake +@@ -43,6 +43,7 @@ IF(UNIX) + # Default GCC flags + IF(CMAKE_COMPILER_IS_GNUCC) + SET(COMMON_C_FLAGS "-fno-omit-frame-pointer") ++ SET(COMMON_C_FLAGS "-fstack-protector-strong") + # Disable inline optimizations for valgrind testing to avoid false positives + IF(WITH_VALGRIND) + STRING_PREPEND(COMMON_C_FLAGS "-fno-inline ") +@@ -57,6 +58,7 @@ IF(UNIX) + ENDIF() + IF(CMAKE_COMPILER_IS_GNUCXX) + SET(COMMON_CXX_FLAGS "-std=c++14 -fno-omit-frame-pointer") ++ SET(COMMON_CXX_FLAGS "-fstack-protector-strong") + # Disable inline optimizations for valgrind testing to avoid false positives + IF(WITH_VALGRIND) + STRING_PREPEND(COMMON_CXX_FLAGS "-fno-inline ") diff --git a/pkg/mysql/PKGBUILD b/pkg/mysql/PKGBUILD new file mode 100644 index 00000000..2fd32ead --- /dev/null +++ b/pkg/mysql/PKGBUILD @@ -0,0 +1,50 @@ +pkgname=mysql +pkgver=8.0.17 +pkgrel=3 +arch=('aarch64' 'x86_64') +url="http://www.mysql.com" +license=('GPLv2') +source=("https://cdn.mysql.com/archives/mysql-8.0/mysql-boost-$pkgver.tar.gz" + "0000-mysql-add-fstack-protector-strong.patch" + "https://mirrors.huaweicloud.com/kunpeng/archive/openEuler/patch/database/my…" + "my.cnf") +md5sums=('7472a25d91973cbba5d8a8f176a3080b' '6e853850efbd6f887ac3e13b1bf99cc4' 'f13eb11de53616017f7428cc06d4e242' '7bb0372ec4e9281813a9afc67c6cfc20') + +prepare() +{ + cd $srcdir/$pkgname-$pkgver + patch -p1 < ../0000-mysql-add-fstack-protector-strong.patch +} + +build() +{ + cd $srcdir/$pkgname-$pkgver + + CMAKE=cmake + grep -sqF "CentOS Linux release 7" /etc/centos-release && CMAKE=cmake3 + + $CMAKE . -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DMYSQL_DATADIR=/data/mysql/data -DSYSCONFDIR=/etc \ + -DWITH_INNOBASE_STORAGE_ENGINE=1 -DWITH_PARTITION_STORAGE_ENGINE=1 -DWITH_FEDERATED_STORAGE_ENGINE=1 \ + -DWITH_ARCHIVE_STORAGE_ENGINE=1 -DWITH_BLACKHOLE_STORAGE_ENGINE=1 -DWITH_MYISAM_STORAGE_ENGINE=1 \ + -DENABLED_LOCAL_INFILE=1 -DENABLE_DTRACE=0 -DDEFAULT_CHARSET=utf8mb4 -DDEFAULT_COLLATION=utf8mb4_general_ci \ + -DWITH_EMBEDDED_SERVER=1 -DDOWNLOAD_BOOST=1 -DWITH_BOOST=./boost -DFORCE_INSOURCE_BUILD=1 + make -j8 + + cd $srcdir + tar -zxvf getcpu.tgz + + cd getcpu || exit + make +} + +package() +{ + mkdir -p $pkgdir/getcpu + cp getcpu/getcpu-hi.ko $pkgdir/getcpu/ + + mkdir -p $pkgdir/etc + cp my.cnf $pkgdir/etc/ + + cd $srcdir/$pkgname-$pkgver + make DESTDIR=$pkgdir install +} diff --git a/pkg/mysql/my.cnf b/pkg/mysql/my.cnf new file mode 100644 index 00000000..9732af40 --- /dev/null +++ b/pkg/mysql/my.cnf @@ -0,0 +1,67 @@ +[mysqld_safe] +log-error=/data/mysql/log/mysql.log +pid-file=/data/mysql/run/mysqld.pid +[mysqldump] +quick +[mysql] +[client] +default-character-set=utf8mb4 +socket=/data/mysql/run/mysql.sock + +[mysqld] +socket=/data/mysql/run/mysql.sock +tmpdir=/data/mysql/tmp +datadir=/data/mysql/data +default_authentication_plugin=mysql_native_password +port=3309 +user=mysql +memlock=on +bind-address=0.0.0.0 +skip-grant-tables + +lc-messages=en_US +lc-messages-dir=/usr/local/mysql/share + +innodb_page_size=64k +max_connections=4000 +back_log=2048 +performance_schema=OFF +max_prepared_stmt_count=128000 + +#file +innodb_file_per_table +innodb_log_file_size=2048M +innodb_log_files_in_group=32 +innodb_open_files=10000 +table_open_cache_instances=64 + +#buffers +innodb_buffer_pool_size=128G +innodb_buffer_pool_instances=64 +innodb_log_buffer_size=2048M + +#tune +sync_binlog=0 +innodb_flush_log_at_trx_commit=0 +innodb_use_native_aio=1 +innodb_spin_wait_delay=160 +innodb_sync_spin_loops=20 +innodb_flush_method=O_DIRECT +innodb_io_capacity=30000 +innodb_io_capacity_max=40000 +innodb_lru_scan_depth=9000 +innodb_page_cleaners=1 +innodb_spin_wait_pause_multiplier=25 + +#perf special +innodb_flush_neighbors=0 +innodb_write_io_threads=1 +innodb_read_io_threads=1 +innodb_purge_threads=1 + +sql_mode=STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,NO_AUTO_VALUE_ON_ZERO,STRICT_ALL_TABLES + +#skop_log_bin +ssl=0 + +table_open_cache=15000 -- 2.23.0

2 1

[PATCH v6 compass-ci] add the container/initrd-cifs to share /srv/initrd
by Xiao Shenwei 17 Nov '20

17 Nov '20

[why] our os-cifs container will mount three dir, /srv/os, /srv/initrd, /srv/result in order to reduce the coupling, need split them [how] one folder mount by one container [usage] Two ways for the client to mount initrd: 1. initrd mount -t cifs -o guest,port=446 //ip/initrd /tmp/initrd 2. osimage mount -t cifs -o guest,port=446 //ip/osimage /tmp/osimage Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/initrd-cifs/Dockerfile | 24 ++++++++++++++++++++ container/initrd-cifs/build | 5 +++++ container/initrd-cifs/smb.conf | 38 ++++++++++++++++++++++++++++++++ container/initrd-cifs/start | 24 ++++++++++++++++++++ 4 files changed, 91 insertions(+) create mode 100644 container/initrd-cifs/Dockerfile create mode 100755 container/initrd-cifs/build create mode 100644 container/initrd-cifs/smb.conf create mode 100755 container/initrd-cifs/start diff --git a/container/initrd-cifs/Dockerfile b/container/initrd-cifs/Dockerfile new file mode 100644 index 0000000..9590701 --- /dev/null +++ b/container/initrd-cifs/Dockerfile @@ -0,0 +1,24 @@ +# Origin: https://github.com/Stanback/alpine-samba +# Copyright (C) 2016-2020 Eric D. Stanback +# SPDX-License-Identifier: GPL-3.0 + +FROM alpine:edge + +MAINTAINER Xiao Shenwei <xiaoshenwei96(a)163.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN adduser -u 1090 -D lkp + +RUN apk add --update \ + samba-common-tools \ + samba-client \ + samba-server \ + bash && \ + rm -rf /var/cache/apk/* + +COPY ./smb.conf /etc/samba/ + +EXPOSE 446/tcp + +ENTRYPOINT ["smbd", "--foreground", "--no-process-group", "--log-stdout"] diff --git a/container/initrd-cifs/build b/container/initrd-cifs/build new file mode 100755 index 0000000..4b93647 --- /dev/null +++ b/container/initrd-cifs/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t initrd-cifs . diff --git a/container/initrd-cifs/smb.conf b/container/initrd-cifs/smb.conf new file mode 100644 index 0000000..eaddef7 --- /dev/null +++ b/container/initrd-cifs/smb.conf @@ -0,0 +1,38 @@ +# refer to https://lkml.org/lkml/2019/7/16/716 and https://lkml.org/lkml/2019/9/19/586 +[global] + workgroup = MYGROUP + server string = Samba Server + map to guest = Bad User + load printers = no + printing = bsd + printcap name = /dev/null + disable spoolss = yes + disable netbios = yes + server role = standalone + server services = -dns, -nbt + smb ports = 445 + create mode = 0777 + directory mode = 0777 + guest only = yes + guest ok = yes + server min protocol = NT1 + unix extensions = yes + mangled names = no + +[initrd] + path = /srv/initrd/ + comment = initrd + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp + +[osimage] + path = /srv/initrd/ + comment = osimage + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start new file mode 100755 index 0000000..77558a1 --- /dev/null +++ b/container/initrd-cifs/start @@ -0,0 +1,24 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +lsmod | grep -q "^cifs\s" || { + sudo modprobe cifs +} + +docker_rm initrd-cifs + +cmd=( + docker run + -d + -p 446:445 + -v /etc/localtime:/etc/localtime:ro + -v /srv/initrd:/srv/initrd + --name initrd-cifs + --restart=always + initrd-cifs +) + +"${cmd[@]}" -- 2.23.0

4 9

[PATCH v4 lkp-tests 1/2] tests/cci-makepkg: add download PKGBUILD patch function
by Lin Jiaxin 17 Nov '20

17 Nov '20

[Why] Support more version's PKGBUILD. [How] Download the required file through remote-git from /srv/git/pkg/ usage: job yaml add new field: PKGBUILD_TAG About $PKGBUILD_TAG, it is from PKGBUILD file's $pkgver-$pkgrel. example: PKGBUILD_TAG: 1.0-20 You should create a git repo that name same as $benchmark in /srv/git/pkg/ before you add PKGBUILD_TAG to yaml. If not, cci-makepkg will report an error. And everything same as before if no PKGBUILD_TAG. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a105f189..a62b5be0 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -39,12 +39,29 @@ cd $LKP_SRC/pkg/$benchmark || die "pkg is empty" mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } +download_pkgfile() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "$2" +} + +get_pkgfile() +{ + local pkg_tag_file="PKGBUILD-$PKGBUILD_TAG" + + download_pkgfile "pkg/$benchmark" "$pkg_tag_file" + + [ -f "$pkg_tag_file" ] || die "$pkg_tag_file not exist" + [ -s "$pkg_tag_file" ] || die "$pkg_tag_file is empty" +} + get_pkg_info() { var=$1 pkg_dir=${2:-.} ( . $pkg_dir/PKGBUILD + [ -n "$PKGBUILD_TAG" ] && . $pkg_dir/PKGBUILD-$PKGBUILD_TAG eval echo '$'$var ) } @@ -71,6 +88,8 @@ update_shared_pkg() echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } +[ -n "$PKGBUILD_TAG" ] && get_pkgfile + distro_install_depends lkp-dev [ $os = "centos" ] && -- 2.23.0

2 1

[PATCH v3 compass-ci 2/4] assign-account: add new option for ssh key
by Luan Shengde 17 Nov '20

17 Nov '20

add -r|--gen-sshkey option for answerback-email [why] enable one click login testbox from jumper server [how] add -r|--rsa-key option to enable generate private/public key on jumper server register the public key to es Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 77 ++++++++++---------- 1 file changed, 38 insertions(+), 39 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 686a327..39091c7 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -7,12 +7,23 @@ # samba mount # ssh logshn (huawei, ) (install pubkey / send password) +# 1) external user: email apply => mail robot +# 2) ./answerback-email: huawei user: cli apply +# => params +# => REST api (params different) +# => robot email notify + +# user_type: +# apply_from: mail/cli +# allow_login: + require 'json' require 'mail' require 'set' require 'optparse' require_relative '../defconfig' require_relative '../../lib/es_client' +require_relative 'send-account-email' names = Set.new %w[ JUMPER_HOST @@ -31,17 +42,18 @@ SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 my_info = { 'my_email' => nil, 'my_name' => nil, + 'my_commit_url' => nil, 'my_uuid' => %x(uuidgen).chomp, - 'my_ssh_pubkey' => nil + 'my_ssh_pubkey' => [], + 'gen_sshkey' => false } -def init_info(email_file, my_info) - mail_content = Mail.read(email_file) +def init_info(mail_content, my_info) my_info['my_email'] = mail_content.from[0] my_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') - my_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' - mail_content.part[1].body.decoded - end + my_info['my_ssh_pubkey'] << if mail_content.part[1].filename == 'id_rsa.pub' + mail_content.part[1].body.decoded + end end options = OptionParser.new do |opts| @@ -55,17 +67,23 @@ options = OptionParser.new do |opts| opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| my_info['my_email'] = email_address - # when apply account with email address, will get no user name + # when apply account with email address, will get no user name and commit url my_info['my_name'] = '' + my_info['my_commit_url'] = '' end opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ 'ssh pub_key file, enable password-less login') do |pub_key_file| - my_info['my_ssh_pubkey'] = File.read(pub_key_file) + my_info['my_ssh_pubkey'] << File.read(pub_key_file).chomp end opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| - init_info(email_file, my_info) + mail_content = Mail.read(email_file) + init_info(mail_content, my_info) + end + + opts.on('-g', '--gen-sshkey', 'generate jumper ras public/private key and return pubkey') do + my_info['gen_sshkey'] = true end opts.on_tail('-h', '--help', 'show this message') do @@ -76,32 +94,6 @@ end options.parse!(ARGV) -def build_message(email, account_info) - message = <<~EMAIL_MESSAGE - To: #{email} - Subject: [compass-ci] jumper account is ready - - Dear user: - - Thank you for joining us. - You can use the following command to login the jumper server: - - Login command: - ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} - - Account password: - #{account_info['my_password']} - - Suggest: - If you use the password to login, change it in time. - - regards - compass-ci - EMAIL_MESSAGE - - return message -end - def apply_account(my_info) account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{my_info.to_json}') JSON.parse account_info_str @@ -114,12 +106,19 @@ def send_account(my_info) raise message if my_info['my_email'].nil? account_info = apply_account(my_info) - # for manually assign account, there will be no my_commit_url - # but the key my_commit_url is required for es - my_info['my_commit_url'] = '' my_info['my_login_name'] = account_info['my_login_name'] - my_info.delete 'my_ssh_pubkey' + + unless account_info['my_jumper_pubkey'].nil? + my_info['my_ssh_pubkey'] << account_info['my_jumper_pubkey'].chomp + end + + my_info.delete 'gen_sshkey' store_account_info(my_info) + + send_mail(my_info, account_info) +end + +def send_mail(my_info, account_info) message = build_message(my_info['my_email'], account_info) %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") -- 2.23.0

2 12

[PATCH v8 lkp-tests 2/3] submit on_fail to the scheduler
by Hu Xuejiao 17 Nov '20

17 Nov '20

[why] When the task is failed(state), it need in the waiting state [how] When job2sh.rb defined sh_on_state, it will be invoked by the scheduler Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index bc90e2e0..a325e912 100755 --- a/sbin/submit +++ b/sbin/submit @@ -115,12 +115,14 @@ ARGV.each do |jobfile| sh_run_job = job.sh_run_job sh_extract_stats = job.sh_extract_stats sh_define_files = job.sh_define_files + sh_on_fail = job.sh_on_state(state='on_fail') sh_hash = { 'job2sh' => { 'run_job' => sh_run_job, 'extract_stats' => sh_extract_stats, - 'define_files' => sh_define_files + 'define_files' => sh_define_files, + 'on_fail' => sh_on_fail } } -- 2.23.0

1 0

[PATCH v8 lkp-tests 1/3] job2sh.rb: define on_state
by Hu Xuejiao 17 Nov '20

17 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By on_fail(sh_on_state), it will appear in job.sh: on_fail() { . $LKP_SRC/lib/job.sh . $LKP_SRC/lib/env.sh run_test $LKP_SRC/tests/wrapper sleep 300 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 9c8b578d..9d0a7bb2 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -363,14 +363,14 @@ class Job2sh < Job out_line "}\n\n" end - def sh_on_state(job = nil) + def sh_on_state(state = nil, job = nil) @script_lines = [] @programs = available_programs(:workload_elements) job ||= (@jobx || @job).clone - if job.include?('on_fail') + if state == "on_fail" && job.include?('on_fail') @cur_func = :run_job -- 2.23.0

1 0

[PATCH lkp-tests] depends/sysbench-mysql: add depends
by Zhang Yu 17 Nov '20

17 Nov '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- distro/depends/sysbench-mysql | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/sysbench-mysql b/distro/depends/sysbench-mysql index a248fd07..b0d33ffe 100644 --- a/distro/depends/sysbench-mysql +++ b/distro/depends/sysbench-mysql @@ -1,3 +1,4 @@ libnuma1 xfsprogs ethtool +libmariadb3 -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/4] test/build-pkg: fix a softlink will be established if the package fails
by Sun Yukui 17 Nov '20

17 Nov '20

After packaging fails, a softlink will be established, but this is wrong. If the packaging fails, the softlink should not be established. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/build-pkg | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 6426f2da..8d5fc7d4 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -101,10 +101,11 @@ build_source_pkg() cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 + + [ "$?" == 0 ] || create_softlink } check_vars mount_dest request_pkg build_source_pkg -create_softlink -- 2.23.0

3 3

[PATCH v2 lkp-tests 2/3] iso2rootfs: add step: config something of rootfs
by Yu Chuan 17 Nov '20

17 Nov '20

[Why] The initial rootfs have the follow two problems: - no dns resolver; - selinux is opened; Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 2e7bcd47ef4f..87a51acaca5f 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -338,6 +338,27 @@ run_qcow2rootfs() ./run "${qcow2_path}" "${ROOTFS_DES_DIR}" } + +############ config rootfs ############ +config_dns_resolver() +{ + cat <<-EOF > "${ROOTFS_DES_DIR}/etc/resolv.conf" + nameserver 114.114.114.114 + nameserver 8.8.8.8 + EOF +} + +disable_selinux() +{ + sed -i 's/^SELINUX=.*/SELINUX=disabled/g' "${ROOTFS_DES_DIR}/etc/selinux/config" +} + +config_rootfs() +{ + config_dns_resolver + disable_selinux +} + ############ test rootfs ############ get_qemu_efi_fd() { @@ -497,6 +518,8 @@ main() config_qcow2rootfs run_qcow2rootfs + config_rootfs + test_rootfs [ -z ${test_yaml} ] || submit_test_job_yaml -- 2.23.0

3 5

2025

2024

2023

2022

2021

2020

Compass-ci