- Compass-ci - mailweb.openeuler.org

[PATCH v3 lkp-tests] stats: add hackbench-git result parsing script.
by Zhang Yale 13 Nov '20

13 Nov '20

[input] --------socket thread num=20-------- 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.495 ... 2020-11-10 19:51:38 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.512 [output] socket_thread_num_800: 0.5016 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/hackbench-git | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100755 stats/hackbench-git diff --git a/stats/hackbench-git b/stats/hackbench-git new file mode 100755 index 00000000..187f1c19 --- /dev/null +++ b/stats/hackbench-git @@ -0,0 +1,20 @@ +#!/usr/bin/env ruby + +LKP_SRC = ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require "#{LKP_SRC}/lib/statistics" + +time = [] +while (line = gets) + case line + when /^-+(\w+ \w+ \w+)/ + args = $1.tr(' ', '_') + when /^Running with .* $== (\d+)$ tasks/ + tasks = $1.to_i + when /^Time:/ + seconds = line.split[1] + time << seconds.to_f + end +end + +puts "#{args}_#{tasks}: #{time.sum / time.size}" -- 2.23.0

2 2

[PATCH v3 lkp-tests] stats: add sysbench-mysql result parsing file
by Zhang Yu 13 Nov '20

13 Nov '20

[input] SQL statistics: queries performed: total: 298163456 transactions: 14908172 (25024.26 per sec.) ... [output] transactions: 25024.26 Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/sysbench-mysql | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100755 stats/sysbench-mysql diff --git a/stats/sysbench-mysql b/stats/sysbench-mysql new file mode 100755 index 00000000..5f4e9fd6 --- /dev/null +++ b/stats/sysbench-mysql @@ -0,0 +1,34 @@ +#!/usr/bin/env ruby + +# SQL statistics: +# queries performed: +# read: 210212086 +# write: 60060591 +# other: 30030296 +# total: 300302973 +# transactions: 15015147 (25024.26 per sec.) +# queries: 300302973 (500485.16 per sec.) +# ignored errors: 2 (0.00 per sec.) +# reconnects:0 (0.00 per sec.) +# +# General statistics: +# total time: 600.0222s +# total number of events: 15015147 +# +# Latency (ms): +# min: 2.74 +# avg: 10.23 +# max: 2238.49 +# 95th percentile: 15.55 +# sum: 153573603.78 +# +# Threads fairness: +# events (avg/stddev): 58652.9180/4292.14 +# execution time (avg/stddev): 599.8969/0.01 + +while (line = STDIN.gets) + case line.chomp! + when /transactions:[^(]*\(([^\s]*)/ + puts "transactions: #{$1}" + end +end -- 2.23.0

2 2

[PATCH lkp-tests] distro/archLinux: add archLinux distro file
by Wang Yong 13 Nov '20

13 Nov '20

add archLinux pack method Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/archLinux | 103 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100755 distro/archLinux diff --git a/distro/archLinux b/distro/archLinux new file mode 100755 index 000000000..152f24ff3 --- /dev/null +++ b/distro/archLinux @@ -0,0 +1,103 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +shopt -s nullglob + +. $LKP_SRC/lib/install.sh +. $LKP_SRC/distro/common +. $LKP_SRC/lib/debug.sh + +distro_install_depends() +{ + local script + local bm="$1" + local scripts=$(find $LKP_SRC/distro/depends/ -name "$bm" -o -name "${bm}.[0-9]") + + for script in $scripts + do + script=$(basename $script) + packages=$(get_dependency_packages $DISTRO $script) + + [ -z "$packages" ] && continue + + echo install packages for $script: $packages + + for pkg in $packages + do + pacman -Qqs $pkg || pacman -Sy $pkg + done + done +} + +pack_benchmark() +{ + setup_proxy + + distro_install_depends lkp-dev + + # Process each benchmark + for BM_NAME in $benchmark + do + distro_install_depends $BM_NAME-dev || continue + echo $LKP_SRC/sbin/pack -d $DISTRO -f -c -s $PKG_MNT/$pack_to $BM_NAME + ( + $LKP_SRC/sbin/pack -d $DISTRO -f -c -s $PKG_MNT/$pack_to $BM_NAME + ) + done +} + +pacman_download() +{ + echo "pacman -Sw --noconfirm $*" + pacman -Sw --noconfirm "$@" +} + +pacman_pack() +{ + local target_dir=/opt/pkgs + local date=$(date +"%Y%m%d") + + local downloaded_pkgs=$(find /var/ -type f -name "*.pkg.tar.xz") + + [ "$downloaded_pkgs" ] || return + + mkdir -p $target_dir + + mv $downloaded_pkgs $target_dir + + find $target_dir | cpio --quiet -o -H newc --owner=root.root | gzip -n -9 >$pack_to/${BM_NAME}_$date.cgz + + ln -sf ${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return + chown .lkp $pack_to/${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return + echo package installed to $pack_to/${BM_NAME}.cgz + + ls $target_dir/*.pkg.tar.gz > $pack_to/.${BM_NAME}.packages + + rm -rf $target_dir +} + +cleanup_downloaded_pkgs() +{ + find /var/ -type f -name "*.pkg.tar.xz" -exec rm -f -- '{}' \; +} + +pack_benchmark_deps() +{ + setup_proxy + + for BM_NAME in $benchmark + do + + cleanup_downloaded_pkgs + + check_shared_package $BM_NAME + + packages=$(echo $(get_dependency_packages $DISTRO $BM_NAME)) + + if pacman_download $packages; then + pacman_pack + else + echo "failed to pack-deps $BM_NAME" >&2 + fi + done +} -- 2.23.0

3 4

[PATCH v6 lkp-tests] stats: add hackbench-git result parsing script
by Zhang Yale 13 Nov '20

13 Nov '20

[input] --------socket thread num=20-------- 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.495 ... 2020-11-10 19:51:38 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.512 [output] socket_thread_num_800: 0.5016 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/hackbench-git | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100755 stats/hackbench-git diff --git a/stats/hackbench-git b/stats/hackbench-git new file mode 100755 index 00000000..4ed6a0bc --- /dev/null +++ b/stats/hackbench-git @@ -0,0 +1,26 @@ +#!/usr/bin/env ruby + +# --------socket thread num=20-------- +# 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 +# Running with 20*40 (== 800) tasks. +# Time: 0.495 +# ... + +LKP_SRC = ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require "#{LKP_SRC}/lib/statistics" + +time = [] +while (line = gets) + case line + when /^-+(\w+ \w+ \w+)/ + args = $1.tr(' ', '_') + when /^Running with .* $== (\d+)$ tasks/ + tasks = $1.to_i + when /^Time:/ + seconds = line.split[1] + time << seconds.to_f + end +end + +puts "#{args}_#{tasks}: #{time.sum / time.size}" -- 2.23.0

1 0

[PATCH v5 lkp-tests] stats: add hackbench-git result parsing script
by Zhang Yale 13 Nov '20

13 Nov '20

[input] --------socket thread num=20-------- 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.495 ... 2020-11-10 19:51:38 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.512 [output] socket_thread_num_800: 0.5016 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/hackbench-git | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100755 stats/hackbench-git diff --git a/stats/hackbench-git b/stats/hackbench-git new file mode 100755 index 00000000..187f1c19 --- /dev/null +++ b/stats/hackbench-git @@ -0,0 +1,20 @@ +#!/usr/bin/env ruby + +LKP_SRC = ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require "#{LKP_SRC}/lib/statistics" + +time = [] +while (line = gets) + case line + when /^-+(\w+ \w+ \w+)/ + args = $1.tr(' ', '_') + when /^Running with .* $== (\d+)$ tasks/ + tasks = $1.to_i + when /^Time:/ + seconds = line.split[1] + time << seconds.to_f + end +end + +puts "#{args}_#{tasks}: #{time.sum / time.size}" -- 2.23.0

2 2

[PATCH v6 compass-ci 08/12] mail-robot: build fail message
by Luan Shengde 13 Nov '20

13 Nov '20

build fail email message with raised error message - lack commit url - lack pubkey - repo url for the commit url not in upstream - commit not exist or not available - no more available account on jumper server Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account-fail-eamil.rb | 38 ++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100755 lib/assign-account-fail-eamil.rb diff --git a/lib/assign-account-fail-eamil.rb b/lib/assign-account-fail-eamil.rb new file mode 100755 index 0000000..1c2133c --- /dev/null +++ b/lib/assign-account-fail-eamil.rb @@ -0,0 +1,38 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_fail_email(my_info, message) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] apply account failed + + Dear user: + + Your application for account failed with following error: + + #{message} + + In order to successfully apply an account, please pay attention to the following points: + + 1. mail subject + the subject should exactly: apply account + + 2. commit url + when you writing the url, add prefix: my oss commit + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + + attention: + ensure you commit url exist and available to access. + + 3. ssh pubkey + you need to add a pubkey as an attachment with name 'id_rsa.pub' to the email. + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

2 4

[PATCH v6 compass-ci 03/12] mail-robot: check to assign account
by Luan Shengde 13 Nov '20

13 Nov '20

AssignAccount assign account for user required data: mail_content - send_account parse email_content parse commit url parse pubkey apply_my_account generate uuid apply account store account info to es send email send success email send fail email usage: reference mail-robot.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account.rb | 80 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) create mode 100755 lib/assign-account.rb diff --git a/lib/assign-account.rb b/lib/assign-account.rb new file mode 100755 index 0000000..4443d47 --- /dev/null +++ b/lib/assign-account.rb @@ -0,0 +1,80 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative 'es_client' + +# assign uuid/account for user +# send_account +# parse my_commit_url and my_ssh_pubkey +# apply account +# store account info +# send mail +class AssignAccount + def initialize(mail_content) + @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp + @send_mail_port = SEND_MAIL_PORT + @mail_content = mail_content + + @my_info = { + 'my_email' => mail_content.from[0], + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, ''), + 'my_jumper_pubkey' => false + } + end + + def send_account + my_commit_url, my_ssh_pubkey = parse_commit_url_pub_key + acct_info = apply_my_account(my_ssh_pubkey) + + @my_info['my_commit_url'] = my_commit_url + @my_info['my_login_name'] = acct_info['my_login_name'] + @my_info['my_jumper_pubkey'] = acct_info['my_jumper_pubkey'] + + store_account_info + send_mail('') + rescue StandardError => e + puts e.message + puts e.backtrace + + send_mail(e.message) + end + + def parse_commit_url_pub_key + parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) + + my_commit_url = parse_apply_account_email.parse_commit_url + my_ssh_pubkey = parse_apply_account_email.parse_pub_key + + return my_commit_url, my_ssh_pubkey + end + + def apply_my_account(my_ssh_pubkey) + my_uuid = %x(uuidgen).chomp + + @my_info['my_uuid'] = my_uuid + + apply_account = ApplyJumperAccount.new(@my_info, my_ssh_pubkey) + acct_info = apply_account.apply_jumper_account + + return acct_info + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def send_mail(error_message) + email_message = if error_message.empty? + build_apply_account_email(@my_info) + else + build_apply_account_fail_email(@my_info, error_message) + end + + %x(curl -XPOST "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" -d "#{email_message}") + end +end -- 2.23.0

2 3

[PATCH v6 compass-ci 02/12] mail-robot: listen and handle new email
by Luan Shengde 13 Nov '20

13 Nov '20

listen the mailbox for new email handle_new_email handle the email with defined opetions - apply_account email subject: apply account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-robot.rb | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 lib/mail-robot.rb diff --git a/lib/mail-robot.rb b/lib/mail-robot.rb new file mode 100755 index 0000000..0ae80f3 --- /dev/null +++ b/lib/mail-robot.rb @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' + +def monitor_new_email(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + next if added.empty? + + added.each do |mail_file| + handle_new_email(mail_file, mail_drafts) + end + end + listener.start + sleep +end + +def handle_new_email(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + apply_account(mail_content) + + FileUtils.mv(mail_file, mail_drafts) +end + +def apply_account(mail_content) + return unless mail_content.subject == 'apply account' + + assign_uuid = AssignAccount.new(mail_content) + assign_uuid.send_account +end -- 2.23.0

2 4

[PATCH v2 compass-ci] container/srv-http: support display *.log file in browser
by Lu Weitao 13 Nov '20

13 Nov '20

[why] haven't define some file(*.log) behavior of browser in mime.types, so download this file on browser now. [how] 1. add correspondence of file in mime.types 2. set default type in default.conf of nginx, other type file that haven't define in mime.types, will display the file content in browser Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/root/etc/nginx/conf.d/default.conf | 1 + container/srv-http/root/etc/nginx/mime.types | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/container/srv-http/root/etc/nginx/conf.d/default.conf b/container/srv-http/root/etc/nginx/conf.d/default.conf index 9c15d99..91371f1 100644 --- a/container/srv-http/root/etc/nginx/conf.d/default.conf +++ b/container/srv-http/root/etc/nginx/conf.d/default.conf @@ -13,6 +13,7 @@ server { } location / { + default_type text/html; autoindex on; autoindex_localtime on; allow all; diff --git a/container/srv-http/root/etc/nginx/mime.types b/container/srv-http/root/etc/nginx/mime.types index e7f2f1c..a4ee74b 100644 --- a/container/srv-http/root/etc/nginx/mime.types +++ b/container/srv-http/root/etc/nginx/mime.types @@ -4,8 +4,8 @@ types { text/css css; text/xml xml; text/yaml yaml; - text/plain txt time data sh; + text/plain txt time data sh log; application/json json; - application/x-gzip gz tgz; + application/x-gzip gz tgz cgz; } -- 2.23.0

2 2

[PATCH v6 compass-ci 01/12] mail-robot: run mail-robot service
by Luan Shengde 13 Nov '20

13 Nov '20

call mail-robot.rb parse-apply-account-email.rb gitee-commit-url-check.rb assign-account.rb apply-jumper-account.rb assign-account-email.rb assign-account-fail-eamil.rb to run the email-robot service Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/run-mail-robot.rb | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 container/mail-robot/run-mail-robot.rb diff --git a/container/mail-robot/run-mail-robot.rb b/container/mail-robot/run-mail-robot.rb new file mode 100755 index 0000000..2048fda --- /dev/null +++ b/container/mail-robot/run-mail-robot.rb @@ -0,0 +1,19 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative "#{ENV['CCI_SRC']}/lib/assign-account-email" +require_relative "#{ENV['CCI_SRC']}/lib/assign-account-fail-eamil" +require_relative "#{ENV['CCI_SRC']}/lib/apply-jumper-account" +require_relative "#{ENV['CCI_SRC']}/lib/gitee-commit-url-check" +require_relative "#{ENV['CCI_SRC']}/lib/parse-apply-account-email" +require_relative "#{ENV['CCI_SRC']}/lib/assign-account" +require_relative "#{ENV['CCI_SRC']}/lib/mail-robot" + +MAILDIR = ENV['MAILDIR'] +JUMPER_HOST = ENV['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' +JUMPER_PORT = ENV['JUMPER_PORT'] || 29999 +SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 + +monitor_new_email("#{MAILDIR}/new", "#{MAILDIR}/cur") -- 2.23.0

2 6