- Compass-ci - mailweb.openeuler.org

[PATCH v7 lkp-tests] tests: add hackbench-git test script
by Zhang Yale 13 Nov '20

13 Nov '20

[Why] hackbench-git is an old version tool required by the customer. The hackbench-git usage: hackbench [-pipe] <num groups> [process|thread] [loops] So need to add new test script for hackbench-git. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/hackbench-git | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100755 tests/hackbench-git diff --git a/tests/hackbench-git b/tests/hackbench-git new file mode 100755 index 00000000..f2be0c9d --- /dev/null +++ b/tests/hackbench-git @@ -0,0 +1,30 @@ +#!/bin/sh +# - nr_threads +# - mode +# - ipc +# - iterations +# - loops + +## Hackbench-git is both a benchmark and a stress test for the Linux kernel scheduler. +## Its main job is to create a specified number of schedulable entities pairs. +## (either threads or traditional processes)which communicate via either sockets +## or pipes and time how long it takes for each pair to send data back and forth. + +[ "$ipc" = 'pipe' ] && ipc_option='-pipe' +[ -n "$iterations" ] || iterations=10 +[ -n "$loops" ] || loops=100 +[ -n "$nr_threads" ] || nr_threads=1 + +run_hackbench() +{ + # args: groups number, mode [process/thread], loops, ipc [pipe/socket] + hackbench_path="$BENCHMARK_ROOT"/hackbench-git/hackbench/hackbench + [ ! -x "${hackbench_path}" ] && exit 1 + + echo "--------$ipc $mode num=$nr_threads--------" + for i in $(seq 1 $iterations); do + log_cmd $hackbench_path $ipc_option $nr_threads $mode $loops + done +} + +run_hackbench -- 2.23.0

1 1

[PATCH v3 lkp-tests] test/build-pkg: fix a softlink will be established if the package fails
by Sun Yukui 13 Nov '20

13 Nov '20

[why] After packaging fails, a softlink will be established, but this is wrong. If the packaging fails, the softlink should not be established. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/build-pkg | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index ad8d4843..093bbe4d 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -65,6 +65,19 @@ request_pkg() [ -s PKGBUILD ] || die "PKGBUILD is empty" } +create_softlink() +{ + local soft_path="$PKG_MNT/${pack_to}/${pkgname}" + local bm_name=$(basename $(realpath ${cgz_name})) + + if [ -n "$upstream_tag" ]; then + echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" + fi + + echo "create soft link: latest.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/latest.cgz" +} build_source_pkg() { @@ -86,24 +99,11 @@ build_source_pkg() cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 -} -create_softlink() -{ - local soft_path="$PKG_MNT/${pack_to}/${pkgname}" - local bm_name=$(basename $(realpath ${cgz_name})) - - if [ -n "$upstream_tag" ]; then - echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" - fi - - echo "create soft link: latest.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/latest.cgz" + [ "$?" == 0 ] || create_softlink } check_vars mount_dest request_pkg build_source_pkg -create_softlink -- 2.23.0

2 2

[PATCH v3 lkp-tests] stats: add hackbench-git result parsing script.
by Zhang Yale 13 Nov '20

13 Nov '20

[input] --------socket thread num=20-------- 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.495 ... 2020-11-10 19:51:38 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.512 [output] socket_thread_num_800: 0.5016 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/hackbench-git | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100755 stats/hackbench-git diff --git a/stats/hackbench-git b/stats/hackbench-git new file mode 100755 index 00000000..187f1c19 --- /dev/null +++ b/stats/hackbench-git @@ -0,0 +1,20 @@ +#!/usr/bin/env ruby + +LKP_SRC = ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require "#{LKP_SRC}/lib/statistics" + +time = [] +while (line = gets) + case line + when /^-+(\w+ \w+ \w+)/ + args = $1.tr(' ', '_') + when /^Running with .* $== (\d+)$ tasks/ + tasks = $1.to_i + when /^Time:/ + seconds = line.split[1] + time << seconds.to_f + end +end + +puts "#{args}_#{tasks}: #{time.sum / time.size}" -- 2.23.0

2 2

[PATCH v3 lkp-tests] stats: add sysbench-mysql result parsing file
by Zhang Yu 13 Nov '20

13 Nov '20

[input] SQL statistics: queries performed: total: 298163456 transactions: 14908172 (25024.26 per sec.) ... [output] transactions: 25024.26 Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/sysbench-mysql | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100755 stats/sysbench-mysql diff --git a/stats/sysbench-mysql b/stats/sysbench-mysql new file mode 100755 index 00000000..5f4e9fd6 --- /dev/null +++ b/stats/sysbench-mysql @@ -0,0 +1,34 @@ +#!/usr/bin/env ruby + +# SQL statistics: +# queries performed: +# read: 210212086 +# write: 60060591 +# other: 30030296 +# total: 300302973 +# transactions: 15015147 (25024.26 per sec.) +# queries: 300302973 (500485.16 per sec.) +# ignored errors: 2 (0.00 per sec.) +# reconnects:0 (0.00 per sec.) +# +# General statistics: +# total time: 600.0222s +# total number of events: 15015147 +# +# Latency (ms): +# min: 2.74 +# avg: 10.23 +# max: 2238.49 +# 95th percentile: 15.55 +# sum: 153573603.78 +# +# Threads fairness: +# events (avg/stddev): 58652.9180/4292.14 +# execution time (avg/stddev): 599.8969/0.01 + +while (line = STDIN.gets) + case line.chomp! + when /transactions:[^(]*\(([^\s]*)/ + puts "transactions: #{$1}" + end +end -- 2.23.0

2 2

[PATCH lkp-tests] distro/archLinux: add archLinux distro file
by Wang Yong 13 Nov '20

13 Nov '20

add archLinux pack method Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/archLinux | 103 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100755 distro/archLinux diff --git a/distro/archLinux b/distro/archLinux new file mode 100755 index 000000000..152f24ff3 --- /dev/null +++ b/distro/archLinux @@ -0,0 +1,103 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +shopt -s nullglob + +. $LKP_SRC/lib/install.sh +. $LKP_SRC/distro/common +. $LKP_SRC/lib/debug.sh + +distro_install_depends() +{ + local script + local bm="$1" + local scripts=$(find $LKP_SRC/distro/depends/ -name "$bm" -o -name "${bm}.[0-9]") + + for script in $scripts + do + script=$(basename $script) + packages=$(get_dependency_packages $DISTRO $script) + + [ -z "$packages" ] && continue + + echo install packages for $script: $packages + + for pkg in $packages + do + pacman -Qqs $pkg || pacman -Sy $pkg + done + done +} + +pack_benchmark() +{ + setup_proxy + + distro_install_depends lkp-dev + + # Process each benchmark + for BM_NAME in $benchmark + do + distro_install_depends $BM_NAME-dev || continue + echo $LKP_SRC/sbin/pack -d $DISTRO -f -c -s $PKG_MNT/$pack_to $BM_NAME + ( + $LKP_SRC/sbin/pack -d $DISTRO -f -c -s $PKG_MNT/$pack_to $BM_NAME + ) + done +} + +pacman_download() +{ + echo "pacman -Sw --noconfirm $*" + pacman -Sw --noconfirm "$@" +} + +pacman_pack() +{ + local target_dir=/opt/pkgs + local date=$(date +"%Y%m%d") + + local downloaded_pkgs=$(find /var/ -type f -name "*.pkg.tar.xz") + + [ "$downloaded_pkgs" ] || return + + mkdir -p $target_dir + + mv $downloaded_pkgs $target_dir + + find $target_dir | cpio --quiet -o -H newc --owner=root.root | gzip -n -9 >$pack_to/${BM_NAME}_$date.cgz + + ln -sf ${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return + chown .lkp $pack_to/${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return + echo package installed to $pack_to/${BM_NAME}.cgz + + ls $target_dir/*.pkg.tar.gz > $pack_to/.${BM_NAME}.packages + + rm -rf $target_dir +} + +cleanup_downloaded_pkgs() +{ + find /var/ -type f -name "*.pkg.tar.xz" -exec rm -f -- '{}' \; +} + +pack_benchmark_deps() +{ + setup_proxy + + for BM_NAME in $benchmark + do + + cleanup_downloaded_pkgs + + check_shared_package $BM_NAME + + packages=$(echo $(get_dependency_packages $DISTRO $BM_NAME)) + + if pacman_download $packages; then + pacman_pack + else + echo "failed to pack-deps $BM_NAME" >&2 + fi + done +} -- 2.23.0

3 4

[PATCH v6 lkp-tests] stats: add hackbench-git result parsing script
by Zhang Yale 13 Nov '20

13 Nov '20

[input] --------socket thread num=20-------- 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.495 ... 2020-11-10 19:51:38 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.512 [output] socket_thread_num_800: 0.5016 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/hackbench-git | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100755 stats/hackbench-git diff --git a/stats/hackbench-git b/stats/hackbench-git new file mode 100755 index 00000000..4ed6a0bc --- /dev/null +++ b/stats/hackbench-git @@ -0,0 +1,26 @@ +#!/usr/bin/env ruby + +# --------socket thread num=20-------- +# 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 +# Running with 20*40 (== 800) tasks. +# Time: 0.495 +# ... + +LKP_SRC = ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require "#{LKP_SRC}/lib/statistics" + +time = [] +while (line = gets) + case line + when /^-+(\w+ \w+ \w+)/ + args = $1.tr(' ', '_') + when /^Running with .* $== (\d+)$ tasks/ + tasks = $1.to_i + when /^Time:/ + seconds = line.split[1] + time << seconds.to_f + end +end + +puts "#{args}_#{tasks}: #{time.sum / time.size}" -- 2.23.0

1 0

[PATCH v5 lkp-tests] stats: add hackbench-git result parsing script
by Zhang Yale 13 Nov '20

13 Nov '20

[input] --------socket thread num=20-------- 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.495 ... 2020-11-10 19:51:38 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.512 [output] socket_thread_num_800: 0.5016 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/hackbench-git | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100755 stats/hackbench-git diff --git a/stats/hackbench-git b/stats/hackbench-git new file mode 100755 index 00000000..187f1c19 --- /dev/null +++ b/stats/hackbench-git @@ -0,0 +1,20 @@ +#!/usr/bin/env ruby + +LKP_SRC = ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require "#{LKP_SRC}/lib/statistics" + +time = [] +while (line = gets) + case line + when /^-+(\w+ \w+ \w+)/ + args = $1.tr(' ', '_') + when /^Running with .* $== (\d+)$ tasks/ + tasks = $1.to_i + when /^Time:/ + seconds = line.split[1] + time << seconds.to_f + end +end + +puts "#{args}_#{tasks}: #{time.sum / time.size}" -- 2.23.0

2 2

[PATCH v6 compass-ci 08/12] mail-robot: build fail message
by Luan Shengde 13 Nov '20

13 Nov '20

build fail email message with raised error message - lack commit url - lack pubkey - repo url for the commit url not in upstream - commit not exist or not available - no more available account on jumper server Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account-fail-eamil.rb | 38 ++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100755 lib/assign-account-fail-eamil.rb diff --git a/lib/assign-account-fail-eamil.rb b/lib/assign-account-fail-eamil.rb new file mode 100755 index 0000000..1c2133c --- /dev/null +++ b/lib/assign-account-fail-eamil.rb @@ -0,0 +1,38 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_fail_email(my_info, message) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] apply account failed + + Dear user: + + Your application for account failed with following error: + + #{message} + + In order to successfully apply an account, please pay attention to the following points: + + 1. mail subject + the subject should exactly: apply account + + 2. commit url + when you writing the url, add prefix: my oss commit + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + + attention: + ensure you commit url exist and available to access. + + 3. ssh pubkey + you need to add a pubkey as an attachment with name 'id_rsa.pub' to the email. + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

2 4

[PATCH v6 compass-ci 03/12] mail-robot: check to assign account
by Luan Shengde 13 Nov '20

13 Nov '20

AssignAccount assign account for user required data: mail_content - send_account parse email_content parse commit url parse pubkey apply_my_account generate uuid apply account store account info to es send email send success email send fail email usage: reference mail-robot.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account.rb | 80 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) create mode 100755 lib/assign-account.rb diff --git a/lib/assign-account.rb b/lib/assign-account.rb new file mode 100755 index 0000000..4443d47 --- /dev/null +++ b/lib/assign-account.rb @@ -0,0 +1,80 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative 'es_client' + +# assign uuid/account for user +# send_account +# parse my_commit_url and my_ssh_pubkey +# apply account +# store account info +# send mail +class AssignAccount + def initialize(mail_content) + @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp + @send_mail_port = SEND_MAIL_PORT + @mail_content = mail_content + + @my_info = { + 'my_email' => mail_content.from[0], + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, ''), + 'my_jumper_pubkey' => false + } + end + + def send_account + my_commit_url, my_ssh_pubkey = parse_commit_url_pub_key + acct_info = apply_my_account(my_ssh_pubkey) + + @my_info['my_commit_url'] = my_commit_url + @my_info['my_login_name'] = acct_info['my_login_name'] + @my_info['my_jumper_pubkey'] = acct_info['my_jumper_pubkey'] + + store_account_info + send_mail('') + rescue StandardError => e + puts e.message + puts e.backtrace + + send_mail(e.message) + end + + def parse_commit_url_pub_key + parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) + + my_commit_url = parse_apply_account_email.parse_commit_url + my_ssh_pubkey = parse_apply_account_email.parse_pub_key + + return my_commit_url, my_ssh_pubkey + end + + def apply_my_account(my_ssh_pubkey) + my_uuid = %x(uuidgen).chomp + + @my_info['my_uuid'] = my_uuid + + apply_account = ApplyJumperAccount.new(@my_info, my_ssh_pubkey) + acct_info = apply_account.apply_jumper_account + + return acct_info + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def send_mail(error_message) + email_message = if error_message.empty? + build_apply_account_email(@my_info) + else + build_apply_account_fail_email(@my_info, error_message) + end + + %x(curl -XPOST "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" -d "#{email_message}") + end +end -- 2.23.0

2 3

[PATCH v6 compass-ci 02/12] mail-robot: listen and handle new email
by Luan Shengde 13 Nov '20

13 Nov '20

listen the mailbox for new email handle_new_email handle the email with defined opetions - apply_account email subject: apply account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-robot.rb | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 lib/mail-robot.rb diff --git a/lib/mail-robot.rb b/lib/mail-robot.rb new file mode 100755 index 0000000..0ae80f3 --- /dev/null +++ b/lib/mail-robot.rb @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' + +def monitor_new_email(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + next if added.empty? + + added.each do |mail_file| + handle_new_email(mail_file, mail_drafts) + end + end + listener.start + sleep +end + +def handle_new_email(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + apply_account(mail_content) + + FileUtils.mv(mail_file, mail_drafts) +end + +def apply_account(mail_content) + return unless mail_content.subject == 'apply account' + + assign_uuid = AssignAccount.new(mail_content) + assign_uuid.send_account +end -- 2.23.0

2 4