- Compass-ci - mailweb.openeuler.org

[PATCH v3 lkp-tests] distro/archlinux: add archlinux distro file
by Wang Yong 17 Nov '20

17 Nov '20

add archlinux pack method Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/archlinux | 103 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100755 distro/archlinux diff --git a/distro/archlinux b/distro/archlinux new file mode 100755 index 000000000..1e6d0f8a4 --- /dev/null +++ b/distro/archlinux @@ -0,0 +1,103 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +shopt -s nullglob + +. $LKP_SRC/lib/install.sh +. $LKP_SRC/distro/common +. $LKP_SRC/lib/debug.sh + +distro_install_depends() +{ + local script + local bm="$1" + local scripts=$(find $LKP_SRC/distro/depends/ -name "$bm" -o -name "${bm}.[0-9]") + + for script in $scripts + do + script=$(basename $script) + packages=$(get_dependency_packages $DISTRO $script) + + [ -z "$packages" ] && continue + + echo install packages for $script: $packages + + for pkg in $packages + do + pacman -Qqs ----noconfirm $pkg || pacman -Sy --noconfirm $pkg + done + done +} + +pack_benchmark() +{ + setup_proxy + + distro_install_depends lkp-dev + + # Process each benchmark + for BM_NAME in $benchmark + do + distro_install_depends $BM_NAME-dev || continue + echo $LKP_SRC/sbin/pack -d $DISTRO -f -c -s $PKG_MNT/$pack_to $BM_NAME + ( + $LKP_SRC/sbin/pack -d $DISTRO -f -c -s $PKG_MNT/$pack_to $BM_NAME + ) + done +} + +pacman_download() +{ + echo "pacman -Sw --noconfirm $*" + pacman -Sw --noconfirm "$@" +} + +pacman_pack() +{ + local target_dir=/opt/pkgs + local date=$(date +"%Y%m%d") + + local downloaded_pkgs=$(find /var/ -type f -name "*.pkg.tar.xz") + + [ "$downloaded_pkgs" ] || return + + mkdir -p $target_dir + + mv $downloaded_pkgs $target_dir + + find $target_dir | cpio --quiet -o -H newc --owner=root.root | gzip -n -9 >$pack_to/${BM_NAME}_$date.cgz + + ln -sf ${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return + chown .lkp $pack_to/${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return + echo package installed to $pack_to/${BM_NAME}.cgz + + ls $target_dir/*.pkg.tar.gz > $pack_to/.${BM_NAME}.packages + + rm -rf $target_dir +} + +cleanup_downloaded_pkgs() +{ + find /var/ -type f -name "*.pkg.tar.xz" -exec rm -f -- '{}' \; +} + +pack_benchmark_deps() +{ + setup_proxy + + for BM_NAME in $benchmark + do + + cleanup_downloaded_pkgs + + check_shared_package $BM_NAME + + packages=$(echo $(get_dependency_packages $DISTRO $BM_NAME)) + + if pacman_download $packages; then + pacman_pack + else + echo "failed to pack-deps $BM_NAME" >&2 + fi + done +} -- 2.23.0

1 1

[PATCH v4 compass-ci 1/4] git_mirror: pull submodule if repositories have
by Li Yuanchao 17 Nov '20

17 Nov '20

[why] Some repositories have submodules. Without these submodules, they can not be built successfully. Submodules of repositories are repositories too. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 35 ++++++++++++++++++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 9d6a45d..55ec67d 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -56,7 +56,8 @@ class GitMirror def mirror_sync fork_info = @queue.pop - mirror_dir = "/srv/git/#{fork_info['git_repo']}.git" + mirror_dir = "/srv/git/#{fork_info['git_repo']}" + mirror_dir = "#{mirror_dir}.git" unless fork_info['is_submodule'] possible_new_refs = false if File.directory?(mirror_dir) possible_new_refs = git_fetch(mirror_dir) @@ -159,6 +160,8 @@ class MirrorMain feedback_info = @feedback_queue.pop(true) git_repo = feedback_info[:git_repo] + return if check_submodule(git_repo) + update_fork_stat(git_repo, feedback_info[:possible_new_refs]) return unless feedback_info[:possible_new_refs] @@ -219,6 +222,8 @@ class MirrorMain end def get_cur_refs(git_repo) + return if @git_info[git_repo]['is_submodule'] + mirror_dir = "/srv/git/#{git_repo}.git" show_ref_out = %x(git -C #{mirror_dir} show-ref --heads) cur_refs = { heads: {} } @@ -328,4 +333,32 @@ class MirrorMain end end end + + def handle_submodule(submodule, parent_project) + submodule.each_line do |line| + next unless line.include?('url = ') + + url = line.split(' = ')[1].chomp + git_repo = url.split('://')[1] if url.include?('://') + return unless git_repo + + @git_info[git_repo] = { 'url' => url, 'git_repo' => git_repo, 'is_submodule' => true } + fork_stat_init(git_repo) + @priority_queue.push git_repo, @priority + @priority += 1 + end + end + + def check_submodule(git_repo) + if @git_info[git_repo]['is_submodule'] + @fork_stat[git_repo][:queued] = false + return true + end + + mirror_dir = "/srv/git/#{git_repo}.git" + submodule = %x(git -C #{mirror_dir} show HEAD:.gitmodules 2>/dev/null) + return if submodule.empty? + + handle_submodule(submodule, File.basename(git_repo)) + end end -- 2.23.0

2 1

[PATCH v5 compass-ci 2/4] assign-account: add option to generate ssh key
by Luan Shengde 17 Nov '20

17 Nov '20

[why] enable one click login testbox from jumper server [how] add add -g|--gen-sshkey option to enable generate private/public key on jumper server register the public key to es Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 80 ++++++++++---------- 1 file changed, 38 insertions(+), 42 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 686a327..1eb2a10 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -7,12 +7,22 @@ # samba mount # ssh logshn (huawei, ) (install pubkey / send password) +# entry point1: external user: email apply => mail robot +# entry point2: ./answerback-email: cli apply +# => params +# => REST api (params different) +# => robot email notify +# user_type: +# apply_from: mail/cli +# allow_login: + require 'json' require 'mail' require 'set' require 'optparse' require_relative '../defconfig' require_relative '../../lib/es_client' +require_relative 'build-send-account-email' names = Set.new %w[ JUMPER_HOST @@ -31,41 +41,46 @@ SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 my_info = { 'my_email' => nil, 'my_name' => nil, + 'my_commit_url' => nil, 'my_uuid' => %x(uuidgen).chomp, - 'my_ssh_pubkey' => nil + 'my_ssh_pubkey' => [], + 'gen_sshkey' => false } -def init_info(email_file, my_info) - mail_content = Mail.read(email_file) +def init_info(mail_content, my_info) my_info['my_email'] = mail_content.from[0] my_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') - my_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' - mail_content.part[1].body.decoded - end + return if mail_content.attachments.empty? + + my_info['my_ssh_pubkey'] << mail_content.attachments[0].body.decoded end options = OptionParser.new do |opts| opts.banner = 'Usage: answerback-mail.rb [-e|--email email] ' - opts.banner += "[-s|--ssh-pubkey pub_key_file] [-f|--raw-email email_file]\n" + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-f|--raw-email email_file] [-g|--gen-sshkey]\n" opts.banner += " -e or -f is required\n" - opts.banner += ' -s is optional when use -e' + opts.banner += " -s is optional when use -e\n" + opts.banner += ' -g is optional, used to generate sshkey for user' opts.separator '' opts.separator 'options:' opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| my_info['my_email'] = email_address - # when apply account with email address, will get no user name - my_info['my_name'] = '' end opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ 'ssh pub_key file, enable password-less login') do |pub_key_file| - my_info['my_ssh_pubkey'] = File.read(pub_key_file) + my_info['my_ssh_pubkey'] << File.read(pub_key_file).chomp end opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| - init_info(email_file, my_info) + mail_content = Mail.read(email_file) + init_info(mail_content, my_info) + end + + opts.on('-g', '--gen-sshkey', 'generate jumper ras public/private key and return pubkey') do + my_info['gen_sshkey'] = true end opts.on_tail('-h', '--help', 'show this message') do @@ -76,32 +91,6 @@ end options.parse!(ARGV) -def build_message(email, account_info) - message = <<~EMAIL_MESSAGE - To: #{email} - Subject: [compass-ci] jumper account is ready - - Dear user: - - Thank you for joining us. - You can use the following command to login the jumper server: - - Login command: - ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} - - Account password: - #{account_info['my_password']} - - Suggest: - If you use the password to login, change it in time. - - regards - compass-ci - EMAIL_MESSAGE - - return message -end - def apply_account(my_info) account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{my_info.to_json}') JSON.parse account_info_str @@ -114,12 +103,19 @@ def send_account(my_info) raise message if my_info['my_email'].nil? account_info = apply_account(my_info) - # for manually assign account, there will be no my_commit_url - # but the key my_commit_url is required for es - my_info['my_commit_url'] = '' my_info['my_login_name'] = account_info['my_login_name'] - my_info.delete 'my_ssh_pubkey' + + unless account_info['my_jumper_pubkey'].nil? + my_info['my_ssh_pubkey'] << account_info['my_jumper_pubkey'].chomp + end + + my_info.delete 'gen_sshkey' store_account_info(my_info) + + send_mail(my_info, account_info) +end + +def send_mail(my_info, account_info) message = build_message(my_info['my_email'], account_info) %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") -- 2.23.0

3 3

[PATCH v7 compass-ci 06/11] mail-robot: build success email
by Luan Shengde 17 Nov '20

17 Nov '20

build email message for successfully assigning account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/assign-account-email.rb | 53 +++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100755 container/mail-robot/lib/assign-account-email.rb diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb new file mode 100755 index 0000000..1251d1b --- /dev/null +++ b/container/mail-robot/lib/assign-account-email.rb @@ -0,0 +1,53 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_email(my_info) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] Account Ready + + Dear #{my_info['my_name']}, + + Thank you for joining us. + + You can use the following info to submit jobs: + + notice: + the followings are ONE-TIME setup + + 1) setup default config + run the following command to add the below setup to default config file + + mkdir -p ~/.config/compass-ci/defaults/ + cat >> ~/.config/compass-ci/defaults/account.yaml <<-EOF + my_email: #{my_info['my_email']} + my_name: #{my_info['my_name']} + my_uuid: #{my_info['my_uuid']} + EOF + + 2) download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration + + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source ${HOME}/.bashrc && source ${HOME}/.bash_profile + + 3) submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + + reference to 'how to write job yaml' section to write the job yaml + you can also reference to files in compass-ci/jobs as example. + + submit jobs, for example: + + submit jobs/iperf.yaml testbox=vm-2p8g~\\$USER + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

2 3

[PATCH v5 compass-ci 1/2] /lib: keep suite result in top for compare output
by Lu Kaiyi 17 Nov '20

17 Nov '20

[why] Currently, the compare output result default order by alphabet. If there are too many lines in result, it will be difficult for user to find the lines that starts with $suite in metric column. So, need reorder compare output result with keeping $suite item on the top of the table. [how] Divide the lines into two parts before print according to whether the line start with $suite or not, then combine output result into a whole to highlight the important info. [example] command: compare "id=z9.148096" "id=z9.148143" in z9 suites_list:["sysbench-mysql"] [before] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 224.42 -74.1% 58.14 boot-time.boot 20400.95 -73.5% 5397.13 boot-time.idle 11.56 +83.3% 21.19 boot-time.kernel_boot 29424.00 -100.0% 0.00 interrupts.17:GICv3.23.Level.arm-pmu 4100080.00 -99.8% 10196.00 interrupts.226:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 ... 1266042.00 -18.3% 1034206.00 softirqs.RCU 547849.00 -27.4% 397593.00 softirqs.SCHED 159990.00 -80.5% 31181.00 softirqs.TASKLET 1617967.00 -23.0% 1245331.00 softirqs.TIMER 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time.max 214458.00 -100.0% 0.00 sysbench-mysql.time.involuntary_context_switches 2.00 -100.0% 0.00 sysbench-mysql.time.major_page_faults 192832.00 -100.0% 0.00 sysbench-mysql.time.maximum_resident_set_size 90746.00 -100.0% 0.00 sysbench-mysql.time.minor_page_faults 65536.00 -100.0% 0.00 sysbench-mysql.time.page_size 201.00 -100.0% 0.00 sysbench-mysql.time.percent_of_cpu_this_job_got 2796.60 -100.0% 0.00 sysbench-mysql.time.system_time 1535.98 -100.0% 0.00 sysbench-mysql.time.user_time 282616989.00 -100.0% 0.00 sysbench-mysql.time.voluntary_context_switches 2149.23 -100.0% 0.00 time.elapsed_time 2149.23 -100.0% 0.00 time.elapsed_time.max 214458.00 -100.0% 0.00 time.involuntary_context_switches 2.00 -100.0% 0.00 time.major_page_faults 192832.00 -100.0% 0.00 time.maximum_resident_set_size ... [after] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time.max 214458.00 -100.0% 0.00 sysbench-mysql.time.involuntary_context_switches 2.00 -100.0% 0.00 sysbench-mysql.time.major_page_faults 192832.00 -100.0% 0.00 sysbench-mysql.time.maximum_resident_set_size 90746.00 -100.0% 0.00 sysbench-mysql.time.minor_page_faults 65536.00 -100.0% 0.00 sysbench-mysql.time.page_size 201.00 -100.0% 0.00 sysbench-mysql.time.percent_of_cpu_this_job_got 2796.60 -100.0% 0.00 sysbench-mysql.time.system_time 1535.98 -100.0% 0.00 sysbench-mysql.time.user_time 282616989.00 -100.0% 0.00 sysbench-mysql.time.voluntary_context_switches 224.42 -74.1% 58.14 boot-time.boot 20400.95 -73.5% 5397.13 boot-time.idle 11.56 +83.3% 21.19 boot-time.kernel_boot 29424.00 -100.0% 0.00 interrupts.17:GICv3.23.Level.arm-pmu 4100080.00 -99.8% 10196.00 interrupts.226:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 2894304.00 +12.8% 3265730.00 interrupts.4:GICv3.26.Level.arch_timer 11546998.00 -100.0% 0.00 interrupts.746:ITS-MSI.99092481.Edge.hns3-0000:bd:00.1-TxRx-0 7476064.00 -100.0% 0.00 interrupts.747:ITS-MSI.99092482.Edge.hns3-0000:bd:00.1-TxRx-2 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 16 ++++++++++------ lib/compare_matrixes.rb | 41 ++++++++++++++++++++++++++++------------- lib/matrix2.rb | 20 +++++++++++++++----- 3 files changed, 53 insertions(+), 24 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index 645f682..f34e239 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -20,8 +20,8 @@ require 'yaml' def compare_matrices_list(argv, common_conditions, options) condition_list = parse_argv(argv, common_conditions) - matrices_list = create_matrices_list(condition_list) - compare_matrixes(matrices_list, options: options) + matrices_list, suite_list = create_matrices_list(condition_list) + compare_matrixes(matrices_list, suite_list, options: options) end def parse_argv(argv, common_conditions) @@ -37,12 +37,16 @@ end def create_matrices_list(conditions) matrices_list = [] + suite_list = [] es = ESQuery.new(ES_HOST, ES_PORT) conditions.each do |condition| query_results = es.multi_field_query(condition) - matrices_list << combine_query_data(query_results) + matrix, suites = combine_query_data(query_results) + matrices_list << matrix + suite_list.concat(suites) end - matrices_list + + return matrices_list, suite_list end # ------------------------------------------------------------------------------------------- @@ -58,8 +62,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, options) + groups_matrices, suites_list = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_list, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 119d42d..5039b01 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -128,7 +128,7 @@ def set_compare_values(index, values, field, success, options) end def get_values_by_field(matrixes_list, field, matrixes_size, success, options) - # get values by field, values struce example: values[0][:average] + # get values by field, values struct example: values[0][:average] # values = {} matrixes_list.length.times do |index| @@ -193,7 +193,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix @@ -208,6 +208,7 @@ def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, opti result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( matrixes_values, + suite_list, matrixes_titles, matrixes_list.size, options[:theme] @@ -382,7 +383,7 @@ end # compare each matrices_list within pre dimension of group matrices # input: group matrices # output: pre compare result of each group -def compare_group_matrices(group_matrices, options) +def compare_group_matrices(group_matrices, suites_list, options) result_str = '' group_matrices.each do |k, v| matrices_list = [] @@ -392,9 +393,9 @@ def compare_group_matrices(group_matrices, options) matrices_list << matrix end if options[:no_print] - result_str += compare_matrixes(matrices_list, matrices_titles, k, options: options) + result_str += compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) else - print compare_matrixes(matrices_list, matrices_titles, k, options: options) + print compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) end end result_str @@ -807,26 +808,40 @@ def get_theme(matrixes_values, matrixes_titles, theme) return THEMES[:none] end -def get_all_result_str(matrixes_values, matrixes_titles, matrixes_number, theme) +def get_all_result_str(matrixes_values, suite_list, matrixes_titles, matrixes_number, theme) matrixes_titles ||= matrixes_number.times.to_a.map(&:to_s) theme = get_theme(matrixes_values, matrixes_titles, theme) return '' unless theme - failure_str = get_result_str(matrixes_values[false].sort, matrixes_titles, false, theme) - success_str = get_result_str(matrixes_values[true].sort, matrixes_titles, true, theme) + failure_str = get_result_str(matrixes_values[false].sort, suite_list, matrixes_titles, false, theme) + success_str = get_result_str(matrixes_values[true].sort, suite_list, matrixes_titles, true, theme) failure_str + success_str end -def get_result_str(values, matrixes_titles, success, theme) +def get_result_str(values, suite_list, matrixes_titles, success, theme) return '' if values.empty? + suite_set = Set.new(suite_list) result_str = "\n\n\n" common_title, compare_title = get_title_name(success) result_str += get_header(matrixes_titles, success, common_title, compare_title) + ranked_str = get_ranked_str(values, suite_set, success, theme) + result_str += ranked_str + result_str +end + +def get_ranked_str(values, suite_set, success, theme) + suite_str = '' + common_str = '' values.each do |field, matrixes| - result_str += get_values_str(matrixes, success, theme) - result_str += get_field_str(field) - result_str += "\n" + row = get_values_str(matrixes, success, theme) + row += get_field_str(field) + "\n" + field_start_with_suite = suite_set.any? { |suite| field.start_with?(suite) } + if field_start_with_suite + suite_str += row + else + common_str += row + end end - result_str + suite_str + common_str end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 26f07fa..62d174c 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,7 +76,9 @@ end # } def create_matrix(job_list) matrix = {} + suites = [] job_list.each do |job| + suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats @@ -89,7 +91,7 @@ def create_matrix(job_list) matrix.each_value do |value| samples_fill_missing_zeros(value, col_size) end - matrix + return matrix, suites end # input: query results from es_query @@ -110,15 +112,23 @@ end # ... # } def combine_group_query_data(query_data, dims) + suites_list = [] job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + suite_list = [] value.each do |dimension_key, jobs| - groups[group_key][dimension_key] = create_matrix(jobs) + groups[group_key][dimension_key], suites = create_matrix(jobs) + suite_list.concat(suites) end - groups.delete(group_key) if value.size < 2 + if value.size < 2 + groups.delete(group_key) + next + end + suites_list << suite_list end - groups + + return groups, suites_list end # input: @@ -139,7 +149,7 @@ def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) groups.each do |group_key, dims| dims.each do |dim_key, jobs| - groups[group_key][dim_key] = create_matrix(jobs) + groups[group_key][dim_key], = create_matrix(jobs) end end -- 2.23.0

2 2

[PATCH v6 compass-ci] src/lib/web_backend.rb: add date range limit feature
by Zhang Yuhang 17 Nov '20

17 Nov '20

[why] The "start_time" field need limit by date. example: GET /get_jobs?upstream_repo=a&page_size=12&start_date=2020-11-03&end_date=2020-11-03 Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- container/web-backend/web-backend | 57 +++++++++++++++++-------------- src/lib/web_backend.rb | 15 ++++++++ 2 files changed, 46 insertions(+), 26 deletions(-) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index fcc765f..7db82ac 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -1,4 +1,6 @@ #!/usr/bin/ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true require 'sinatra' @@ -12,56 +14,59 @@ set :show_exceptions, false set :environment, :production # GET /compare_candidates -# return: {"query_conditions": {...}, "dimension": [...]} +# Response: +# - {"query_conditions": {...}, "dimension": [...]} get '/compare_candidates' do compare_candidates end # GET /compare?suite=borrow&dimension=tbox_group -# query_conditions: -# > must +# must: +# - query_conditions # - os / os_version / suite / os_arch / tbox_group -# -# dimension: -# > must +# - dimension # - os / os_version / os_arch / suite / tbox_group # -# return: compare result string / "No Data." / "No Difference." +# Response: +# - compare result string / "No Data." / "No Difference." get '/compare' do compare(params) end -# GET /get_jobs?upstream_repo=AvxToNeon/AvxToNeon&page_size=12&page_num=3 -# upstream_repo: -# > optional -# -# page_size: -# > optional +# GET /get_jobs?upstream_repo=a&page_size=12&start_date=2020-11-03&end_date=2020-11-03 +# optional: +# - upstream_repo +# - regex search +# - os +# - os_version +# - os_arch +# - suite +# - category +# - testbox +# - job_state +# - id +# - start_date +# - end_date +# - page_size # - default: 20 -# -# page_num: -# > optional +# - page_num # - default: 1 # -# return: +# Response: # - default: 20 jobs get '/get_jobs' do get_jobs(params) end # GET /get_repos?git_repo=a&page_size=12&page_num=3 -# git_repo: -# > optional -# -# page_size: -# > optional +# optional: +# - git_repo +# - page_size # - default: 20 -# -# page_num: -# > optional +# - page_num # - default: 1 # -# return: +# Response: # - default: 20 repos get '/get_repos' do get_repos(params) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 55c5e4a..b8df6f4 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -1,3 +1,5 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true require 'json' @@ -260,6 +262,17 @@ def get_jobs_result(result) jobs end +def get_job_query_range(condition_fields) + range = { start_time: {} } + start_date = condition_fields[:start_date] + end_date = condition_fields[:end_date] + + range[:start_time][:gte] = "#{start_date} 00:00:00" if start_date + range[:start_time][:lte] = "#{end_date} 23:59:59" if end_date + + { range: range } +end + def search_job(condition_fields, page_size, page_num) must = [] FIELDS.each do |field| @@ -272,6 +285,8 @@ def search_job(condition_fields, page_size, page_num) { term: { field => value } } end end + range = get_job_query_range(condition_fields) + must << range if range[:range][:start_time] result, total = es_search(must, page_size, page_num * page_size) return get_jobs_result(result), total end -- 2.23.0

1 0

[PATCH lkp-tests] update compiling commands
by Hu Xuejiao 17 Nov '20

17 Nov '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- pkg/sysbench-threads-git/PKGBUILD | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pkg/sysbench-threads-git/PKGBUILD b/pkg/sysbench-threads-git/PKGBUILD index 41ed9f46..c79d97b3 100644 --- a/pkg/sysbench-threads-git/PKGBUILD +++ b/pkg/sysbench-threads-git/PKGBUILD @@ -12,7 +12,7 @@ build() { cd "$srcdir/sysbench" ./autogen.sh - ./configure --prefix=/usr --without-gcc-arch --without-mysql + ./configure --build=arm --host=arm --without-mysql make } -- 2.23.0

2 1

[PATCH v7 compass-ci 02/11] mail-robot: check to apply account
by Luan Shengde 17 Nov '20

17 Nov '20

ApplyAccount check to apply account for user when listened new email with subject 'apply account' input: mail_content send_account parse email_content parse my_commit_url parse my_ssh_pubkey apply_my_account generate uuid apply account with my_info and my_ssh_pubkey my_info: - my_email - my_name - my_uuid store account info to es send email send success email send fail email usage: reference mail-robot.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 113 ++++++++++++++++++++++ 1 file changed, 113 insertions(+) create mode 100755 container/mail-robot/lib/apply-account.rb diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb new file mode 100755 index 0000000..1b0f2ad --- /dev/null +++ b/container/mail-robot/lib/apply-account.rb @@ -0,0 +1,113 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative '../../../lib/es_client' +require_relative 'assign-account-email' +require_relative 'assign-account-fail-eamil' +require_relative 'apply-jumper-account' +require_relative 'parse-apply-account-email' + +SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 + +# assign uuid/account for user +# when mail-robot listened new email, and the email's subject +# exactly equal 'apply account', mail-robot will call this class +# entry point: send_account +# input: email's content +# +# send_account +# parse_commit_url_pub_key +# call ParseApplyAccountEmail to parse: +# - my_commit_url +# - my_ssh_pubkey +# apply_my_account +# call ApplyJumperAccount to apply new account +# required data: +# my_info: +# - my_email +# - my_name +# - my_uuid +# my_ssh_pubkey +# store_account_info +# call ESClient to store my_info +# my_info: +# - my_email +# - my_name +# - my_uuid +# - my_commit_url +# - my_login_name +# - my_ssh_pubkey +# send_mail +# when successfully applied an account +# call build_apply_account_email to send a successful email +# when rescued error message +# call build_apply_account_fail_email to send fail email +class ApplyAccount + def initialize(mail_content) + @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp + @send_mail_port = SEND_MAIL_PORT + @mail_content = mail_content + + @my_info = { + 'my_email' => mail_content.from[0], + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, ''), + 'my_ssh_pubkey' => [] + } + end + + def check_to_send_account + my_commit_url, my_ssh_pubkey = parse_commit_url_pub_key + acct_info = apply_my_account(my_ssh_pubkey) + + @my_info['my_commit_url'] = my_commit_url + @my_info['my_login_name'] = acct_info['my_login_name'] + @my_info['my_ssh_pubkey'] << my_ssh_pubkey + + store_account_info + send_mail('') + rescue StandardError => e + puts e.message + puts e.backtrace + + send_mail(e.message) + end + + def parse_commit_url_pub_key + parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) + + my_commit_url = parse_apply_account_email.parse_commit_url + my_ssh_pubkey = parse_apply_account_email.parse_pub_key + + return my_commit_url, my_ssh_pubkey + end + + def apply_my_account(my_ssh_pubkey) + my_uuid = %x(uuidgen).chomp + + @my_info['my_uuid'] = my_uuid + + apply_account = ApplyJumperAccount.new(@my_info, my_ssh_pubkey) + acct_info = apply_account.apply_jumper_account + + return acct_info + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def send_mail(error_message) + email_message = if error_message.empty? + build_apply_account_email(@my_info) + else + build_apply_account_fail_email(@my_info, error_message) + end + + %x(curl -XPOST "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" -d "#{email_message}") + end +end -- 2.23.0

2 2

[PATCH v5 compass-ci 2/2] lib/matrix2.rb: refactor performance optimization
by Lu Kaiyi 17 Nov '20

17 Nov '20

[why] Avoid to execute some unnecessary operations and improve performance. [how] Keep the value.size condition judgement to the front of block. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 62d174c..86cfc93 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -116,15 +116,15 @@ def combine_group_query_data(query_data, dims) job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + if value.size < 2 + groups.delete(group_key) + next + end suite_list = [] value.each do |dimension_key, jobs| groups[group_key][dimension_key], suites = create_matrix(jobs) suite_list.concat(suites) end - if value.size < 2 - groups.delete(group_key) - next - end suites_list << suite_list end -- 2.23.0

1 0

[PATCH compass-ci] git_mirror: fix that README.md in upstream-repos cause a load error
by Li Yuanchao 16 Nov '20

16 Nov '20

[why] As README.md or other docs in upstream-repos, and they are not yaml file. But git-mirror load all files in upstream-repos as yaml file, so there comes out a load error. Now files whose dirname is 'xxx/upstream-repos' will not be loaded. [errmsg] in `load_repo_file': undefined method `[]=' for x:Integer (NoMethodError) Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 9648063..72ca328 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -135,6 +135,8 @@ class MirrorMain traverse_repodir("#{repodir}/#{entry}") end else + return if File.dirname(repodir) == REPO_DIR + project = File.dirname(repodir).delete_prefix("#{REPO_DIR}/") fork_name = File.basename(repodir) load_repo_file(repodir, project, fork_name) -- 2.23.0

1 0

[PATCH compass-ci] container/minio: run minio container
by Cui Yili 16 Nov '20

16 Nov '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/minio/start | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100755 container/minio/start diff --git a/container/minio/start b/container/minio/start new file mode 100755 index 0000000..348bff7 --- /dev/null +++ b/container/minio/start @@ -0,0 +1,18 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ../defconfig.sh +docker_rm minio + +cmd=( + docker run -d + -p 11321:9000 + --name minio + -v /srv/minio/:/srv/minio/ + -e "MINIO_ACCESS_KEY=${MINIO_ACCESS_KEY}" + -e "MINIO_SECRET_KEY=${MINIO_SECRET_KEY}" + jessestuart/minio server /srv/minio/ +) + +"${cmd[@]}" -- 2.23.0

2 1

[PATCH lkp-tests] tests/build-pkg: add download PKGBUILD-$tag function
by Liu Shaofei 16 Nov '20

16 Nov '20

[why] Support more version's PKGBUILD to build different version software. [how] Download the PKGBUILD-$tag file through remote-git server from "/srv/git/pkg/" usage: 1) append upstream_tag field from build-pkg.yaml 2) create git repo that named by software name of $upstream_repo in "/srv/git/pkg/", and add PKGBUILD-$tag file to the directory of "/srv/git/pkg/$upstream_repo". if not PKGBUILD-$tag, the software is built with PKGBUILD. Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 1eed784f..5d2fb792 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -61,6 +61,15 @@ request_pkg() get_pkgfile "${pkgrepo}" "$pkgfile" done + # download PKGBUILD-$tag + benchmark="${upstream_repo##*/}" + benchmark="${benchmark%.git}" + [ -n "$upstream_tag" ] && { + get_pkgfile "pkg/$benchmark" "PKGBUILD-$upstream_tag" + [ -f "PKGBUILD-$upstream_tag" ] || die "PKGBUILD-$upstream_tag not exist" + [ -s "PKGBUILD-$upstream_tag" ] || die "PKGBUILD-$upstream_tag is empty" + } + [ -f PKGBUILD ] || die "PKGBUILD not exist" [ -s PKGBUILD ] || die "PKGBUILD is empty" } @@ -80,7 +89,10 @@ build_source_pkg() fi upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\")" - sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD + sed -i "s|^source=.*|${upstream_source}|g" "PKGBUILD" + [ -n "$upstream_tag" ] && { + sed -i "s|^source=.*|${upstream_source}|g" "PKGBUILD-$upstream_tag" + } } cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" -- 2.23.0

1 0

[PATCH lkp-tests 1/4] test/build-pkg: Refactor move create_softlink function position
by Sun Yukui 16 Nov '20

16 Nov '20

Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/build-pkg | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 1eed784f..2ec45b9c 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -65,6 +65,19 @@ request_pkg() [ -s PKGBUILD ] || die "PKGBUILD is empty" } +create_softlink() +{ + local soft_path="$PKG_MNT/${pack_to}/${pkgname}" + local bm_name=$(basename $(realpath ${cgz_name})) + + if [ -n "$upstream_tag" ]; then + echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" + fi + + echo "create soft link: latest.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/latest.cgz" +} build_source_pkg() { @@ -88,22 +101,6 @@ build_source_pkg() $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 } -create_softlink() -{ - [ -e "$cgz_name" ] || return 0 - - local soft_path="$PKG_MNT/${pack_to}/${pkgname}" - local bm_name=$(basename $(realpath ${cgz_name})) - - if [ -n "$upstream_tag" ]; then - echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" - fi - - echo "create soft link: latest.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/latest.cgz" -} - check_vars mount_dest request_pkg -- 2.23.0

2 2

[PATCH v7 compass-ci] assign-account: add option to generate ssh key
by Luan Shengde 16 Nov '20

16 Nov '20

[why] enable one click login testbox from jumper server [how] add -g|--gen-sshkey option to enable generate private/public key on jumper server register the public key to es Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 71 ++++++++------------ 1 file changed, 29 insertions(+), 42 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 686a327..5ddc451 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -13,6 +13,7 @@ require 'set' require 'optparse' require_relative '../defconfig' require_relative '../../lib/es_client' +require_relative 'build-send-account-email' names = Set.new %w[ JUMPER_HOST @@ -31,41 +32,46 @@ SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 my_info = { 'my_email' => nil, 'my_name' => nil, + 'my_commit_url' => nil, 'my_uuid' => %x(uuidgen).chomp, - 'my_ssh_pubkey' => nil + 'my_ssh_pubkey' => [], + 'gen_sshkey' => false } -def init_info(email_file, my_info) - mail_content = Mail.read(email_file) +def init_info(mail_content, my_info) my_info['my_email'] = mail_content.from[0] my_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') - my_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' - mail_content.part[1].body.decoded - end + return if mail_content.attachments.empty? + + my_info['my_ssh_pubkey'] << mail_content.attachments[0].body.decoded end options = OptionParser.new do |opts| opts.banner = 'Usage: answerback-mail.rb [-e|--email email] ' - opts.banner += "[-s|--ssh-pubkey pub_key_file] [-f|--raw-email email_file]\n" + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-f|--raw-email email_file] [-g|--gen-sshkey]\n" opts.banner += " -e or -f is required\n" - opts.banner += ' -s is optional when use -e' + opts.banner += " -s is optional when use -e\n" + opts.banner += ' -g is optional, used to generate sshkey for user' opts.separator '' opts.separator 'options:' opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| my_info['my_email'] = email_address - # when apply account with email address, will get no user name - my_info['my_name'] = '' end opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ 'ssh pub_key file, enable password-less login') do |pub_key_file| - my_info['my_ssh_pubkey'] = File.read(pub_key_file) + my_info['my_ssh_pubkey'] << File.read(pub_key_file).chomp end opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| - init_info(email_file, my_info) + mail_content = Mail.read(email_file) + init_info(mail_content, my_info) + end + + opts.on('-g', '--gen-sshkey', 'generate jumper ras public/private key and return pubkey') do + my_info['gen_sshkey'] = true end opts.on_tail('-h', '--help', 'show this message') do @@ -76,32 +82,6 @@ end options.parse!(ARGV) -def build_message(email, account_info) - message = <<~EMAIL_MESSAGE - To: #{email} - Subject: [compass-ci] jumper account is ready - - Dear user: - - Thank you for joining us. - You can use the following command to login the jumper server: - - Login command: - ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} - - Account password: - #{account_info['my_password']} - - Suggest: - If you use the password to login, change it in time. - - regards - compass-ci - EMAIL_MESSAGE - - return message -end - def apply_account(my_info) account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{my_info.to_json}') JSON.parse account_info_str @@ -114,12 +94,19 @@ def send_account(my_info) raise message if my_info['my_email'].nil? account_info = apply_account(my_info) - # for manually assign account, there will be no my_commit_url - # but the key my_commit_url is required for es - my_info['my_commit_url'] = '' my_info['my_login_name'] = account_info['my_login_name'] - my_info.delete 'my_ssh_pubkey' + + unless account_info['my_jumper_pubkey'].nil? + my_info['my_ssh_pubkey'] << account_info['my_jumper_pubkey'].chomp + end + + my_info.delete 'gen_sshkey' store_account_info(my_info) + + send_mail(my_info, account_info) +end + +def send_mail(my_info, account_info) message = build_message(my_info['my_email'], account_info) %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") -- 2.23.0

1 0

[PATCH v4 lkp-tests] sbin/makepkg: download submodules code from LKP server
by Liu Shaofei 16 Nov '20

16 Nov '20

Some repositories have submodules. So when build packages, we need download submodules from LKP server, avoid the waste of time in downloading files from the remote end. Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- sbin/makepkg | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sbin/makepkg b/sbin/makepkg index fde82212..6f8b95af 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -623,6 +623,11 @@ extract_git() { cd_safe "${dir##*/}" + # modify submodules address to local server address. + if [[ -f ".gitmodules" ]] && [[ -n ${LKP_SERVER} ]]; then + echo -e "[url \"git://${LKP_SERVER}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig + fi + local ref=origin/HEAD if [[ -n $fragment ]]; then case ${fragment%%=*} in -- 2.23.0

2 2

[PATCH v6 compass-ci 1/2] assign-account: add option to generate ssh key
by Luan Shengde 16 Nov '20

16 Nov '20

[why] enable one click login testbox from jumper server [how] add add -g|--gen-sshkey option to enable generate private/public key on jumper server register the public key to es Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 71 ++++++++------------ 1 file changed, 29 insertions(+), 42 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 686a327..5ddc451 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -13,6 +13,7 @@ require 'set' require 'optparse' require_relative '../defconfig' require_relative '../../lib/es_client' +require_relative 'build-send-account-email' names = Set.new %w[ JUMPER_HOST @@ -31,41 +32,46 @@ SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 my_info = { 'my_email' => nil, 'my_name' => nil, + 'my_commit_url' => nil, 'my_uuid' => %x(uuidgen).chomp, - 'my_ssh_pubkey' => nil + 'my_ssh_pubkey' => [], + 'gen_sshkey' => false } -def init_info(email_file, my_info) - mail_content = Mail.read(email_file) +def init_info(mail_content, my_info) my_info['my_email'] = mail_content.from[0] my_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') - my_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' - mail_content.part[1].body.decoded - end + return if mail_content.attachments.empty? + + my_info['my_ssh_pubkey'] << mail_content.attachments[0].body.decoded end options = OptionParser.new do |opts| opts.banner = 'Usage: answerback-mail.rb [-e|--email email] ' - opts.banner += "[-s|--ssh-pubkey pub_key_file] [-f|--raw-email email_file]\n" + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-f|--raw-email email_file] [-g|--gen-sshkey]\n" opts.banner += " -e or -f is required\n" - opts.banner += ' -s is optional when use -e' + opts.banner += " -s is optional when use -e\n" + opts.banner += ' -g is optional, used to generate sshkey for user' opts.separator '' opts.separator 'options:' opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| my_info['my_email'] = email_address - # when apply account with email address, will get no user name - my_info['my_name'] = '' end opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ 'ssh pub_key file, enable password-less login') do |pub_key_file| - my_info['my_ssh_pubkey'] = File.read(pub_key_file) + my_info['my_ssh_pubkey'] << File.read(pub_key_file).chomp end opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| - init_info(email_file, my_info) + mail_content = Mail.read(email_file) + init_info(mail_content, my_info) + end + + opts.on('-g', '--gen-sshkey', 'generate jumper ras public/private key and return pubkey') do + my_info['gen_sshkey'] = true end opts.on_tail('-h', '--help', 'show this message') do @@ -76,32 +82,6 @@ end options.parse!(ARGV) -def build_message(email, account_info) - message = <<~EMAIL_MESSAGE - To: #{email} - Subject: [compass-ci] jumper account is ready - - Dear user: - - Thank you for joining us. - You can use the following command to login the jumper server: - - Login command: - ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} - - Account password: - #{account_info['my_password']} - - Suggest: - If you use the password to login, change it in time. - - regards - compass-ci - EMAIL_MESSAGE - - return message -end - def apply_account(my_info) account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{my_info.to_json}') JSON.parse account_info_str @@ -114,12 +94,19 @@ def send_account(my_info) raise message if my_info['my_email'].nil? account_info = apply_account(my_info) - # for manually assign account, there will be no my_commit_url - # but the key my_commit_url is required for es - my_info['my_commit_url'] = '' my_info['my_login_name'] = account_info['my_login_name'] - my_info.delete 'my_ssh_pubkey' + + unless account_info['my_jumper_pubkey'].nil? + my_info['my_ssh_pubkey'] << account_info['my_jumper_pubkey'].chomp + end + + my_info.delete 'gen_sshkey' store_account_info(my_info) + + send_mail(my_info, account_info) +end + +def send_mail(my_info, account_info) message = build_message(my_info['my_email'], account_info) %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") -- 2.23.0

2 2

[PATCH v6 compass-ci 2/2] assign-account: build email message for successfuly assign account
by Luan Shengde 16 Nov '20

16 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../build-send-account-email.rb | 32 +++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100755 container/assign-account/build-send-account-email.rb diff --git a/container/assign-account/build-send-account-email.rb b/container/assign-account/build-send-account-email.rb new file mode 100755 index 0000000..3c41404 --- /dev/null +++ b/container/assign-account/build-send-account-email.rb @@ -0,0 +1,32 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_message(email, account_info) + message = <<~EMAIL_MESSAGE + To: #{email} + Subject: [compass-ci] jumper account is ready + + Dear user: + + Thank you for joining us. + You can use the following command to login the jumper server: + + Login command: + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} + + Account password: + #{account_info['my_password']} + + You can use your pub_key to login if you have offered one. + + Suggest: + If you use the password to login, change it in time. + + regards + compass-ci + EMAIL_MESSAGE + + return message +end -- 2.23.0

1 0

[PATCH lkp-tests 3/4] test/cci-makepkg: Refactor
by Sun Yukui 16 Nov '20

16 Nov '20

Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/cci-makepkg | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a105f189..da7050f3 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -92,10 +92,6 @@ cgz_name="${pkgver:-0}-${pkgrel:-0}.cgz" setup_proxy -echo $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf -PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf - update_softlink() { [ -e "$sync_dest/$cgz_name" ] || return @@ -109,4 +105,11 @@ update_softlink() rm $sync_dest/$cgz_name } -update_softlink && echo "Makepkg finished successfully" +build_source_pkg() +{ + echo $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf + PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" \ + $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf +} + +build_source_pkg -- 2.23.0

2 1

[PATCH v2 compass-ci] remove actual ip in compass-ci code
by Xiao Shenwei 16 Nov '20

16 Nov '20

obtain the actual ip address from configuration file, Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/assign-account/get_account_info.rb | 2 +- doc/develop-environment.md | 2 +- src/features/jobs/right_iperf.yaml | 2 +- src/scheduler/README.md | 2 +- user-client/helper/iperf.yaml | 2 +- user-client/jobs/iperf-pxe.yaml | 2 +- user-client/jobs/iperf-vm.yaml | 2 +- 7 files changed, 7 insertions(+), 7 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 137f5c6..ca86e88 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -39,7 +39,7 @@ the returned data for setup_jumper_account_info like: { "account" => "guest", "passwd" => "Use pub_key to login", - "jumper_ip" => "10.10.10.10", + "jumper_ip" => "0.0.0.0", "jumper_port" => "10000" } diff --git a/doc/develop-environment.md b/doc/develop-environment.md index 8760878..ac408a2 100644 --- a/doc/develop-environment.md +++ b/doc/develop-environment.md @@ -4,7 +4,7 @@ cat >> ~/.ssh/config <<-EOF Host crystal - Hostname 124.90.34.227 + Hostname xxx.xxx.xxx.xxx Port 22113 EOF diff --git a/src/features/jobs/right_iperf.yaml b/src/features/jobs/right_iperf.yaml index f81af38..e239ccd 100644 --- a/src/features/jobs/right_iperf.yaml +++ b/src/features/jobs/right_iperf.yaml @@ -30,7 +30,7 @@ proc-vmstat: proc-stat: meminfo: -LKP_SERVER: 172.168.131.113 +LKP_SERVER: LKP_CGI_PORT: 3000 result_root: /result/iperf LKP_DEBUG_PREFIX: bash -x diff --git a/src/scheduler/README.md b/src/scheduler/README.md index fdc7027..8e3a197 100644 --- a/src/scheduler/README.md +++ b/src/scheduler/README.md @@ -6,7 +6,7 @@ - And set the environment variables needed for running (modify "/etc/compass-ci/defaults/crystal.yaml", and run $CCI_SRC/container/defconfig.rb | defconfig.sh) LKP_SRC: ~/code/lkp-tests # path to your lkp-tests repo CCI_SRC: ~/code/compass-ci # path to your compass-ci repo - SCHED_HOST: 172.168.131.131 # hostname or ip of the scheduler + SCHED_HOST: 172.17.0.1 # hostname or ip of the scheduler SCHED_PORT: 3000 # port of the scheduler JOB_INDEX_TYPE: "jobs/_doc" # es job document index and type - '$'{variable} is used at shell command, means the value of variable diff --git a/user-client/helper/iperf.yaml b/user-client/helper/iperf.yaml index 1509470..30e929f 100644 --- a/user-client/helper/iperf.yaml +++ b/user-client/helper/iperf.yaml @@ -30,7 +30,7 @@ proc-vmstat: proc-stat: meminfo: -LKP_SERVER: 172.168.131.113 +LKP_SERVER: LKP_CGI_PORT: 3000 result_root: /result/iperf LKP_DEBUG_PREFIX: bash -x diff --git a/user-client/jobs/iperf-pxe.yaml b/user-client/jobs/iperf-pxe.yaml index 209c5d6..cb5a513 100644 --- a/user-client/jobs/iperf-pxe.yaml +++ b/user-client/jobs/iperf-pxe.yaml @@ -53,7 +53,7 @@ proc-vmstat: proc-stat: meminfo: -LKP_SERVER: 172.168.131.113 +LKP_SERVER: LKP_CGI_PORT: 3000 result_root: /result/iperf LKP_DEBUG_PREFIX: bash -x diff --git a/user-client/jobs/iperf-vm.yaml b/user-client/jobs/iperf-vm.yaml index 434b964..c976bed 100644 --- a/user-client/jobs/iperf-vm.yaml +++ b/user-client/jobs/iperf-vm.yaml @@ -34,7 +34,7 @@ proc-vmstat: proc-stat: meminfo: -LKP_SERVER: 172.168.131.113 +LKP_SERVER: LKP_CGI_PORT: 3000 result_root: /result/iperf LKP_DEBUG_PREFIX: bash -x -- 2.23.0

2 2

[PATCH compass-ci 2/2] remove actual ip in compass-ci code
by Xiao Shenwei 16 Nov '20

16 Nov '20

if need ip: obtain the actual ip address from configuration file Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/assign-account/get_account_info.rb | 2 +- doc/develop-environment.md | 2 +- src/features/jobs/right_iperf.yaml | 2 -- src/scheduler/README.md | 2 +- user-client/helper/iperf.yaml | 1 - 5 files changed, 3 insertions(+), 6 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 137f5c6..ca86e88 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -39,7 +39,7 @@ the returned data for setup_jumper_account_info like: { "account" => "guest", "passwd" => "Use pub_key to login", - "jumper_ip" => "10.10.10.10", + "jumper_ip" => "0.0.0.0", "jumper_port" => "10000" } diff --git a/doc/develop-environment.md b/doc/develop-environment.md index 8760878..ac408a2 100644 --- a/doc/develop-environment.md +++ b/doc/develop-environment.md @@ -4,7 +4,7 @@ cat >> ~/.ssh/config <<-EOF Host crystal - Hostname 124.90.34.227 + Hostname xxx.xxx.xxx.xxx Port 22113 EOF diff --git a/src/features/jobs/right_iperf.yaml b/src/features/jobs/right_iperf.yaml index f81af38..db83832 100644 --- a/src/features/jobs/right_iperf.yaml +++ b/src/features/jobs/right_iperf.yaml @@ -30,8 +30,6 @@ proc-vmstat: proc-stat: meminfo: -LKP_SERVER: 172.168.131.113 LKP_CGI_PORT: 3000 result_root: /result/iperf LKP_DEBUG_PREFIX: bash -x - diff --git a/src/scheduler/README.md b/src/scheduler/README.md index fdc7027..c92c1e8 100644 --- a/src/scheduler/README.md +++ b/src/scheduler/README.md @@ -6,7 +6,7 @@ - And set the environment variables needed for running (modify "/etc/compass-ci/defaults/crystal.yaml", and run $CCI_SRC/container/defconfig.rb | defconfig.sh) LKP_SRC: ~/code/lkp-tests # path to your lkp-tests repo CCI_SRC: ~/code/compass-ci # path to your compass-ci repo - SCHED_HOST: 172.168.131.131 # hostname or ip of the scheduler + SCHED_HOST: 172.17.0.1 # hostname or ip of the scheduler SCHED_PORT: 3000 # port of the scheduler JOB_INDEX_TYPE: "jobs/_doc" # es job document index and type - '$'{variable} is used at shell command, means the value of variable diff --git a/user-client/helper/iperf.yaml b/user-client/helper/iperf.yaml index 1509470..50c9ac6 100644 --- a/user-client/helper/iperf.yaml +++ b/user-client/helper/iperf.yaml @@ -30,7 +30,6 @@ proc-vmstat: proc-stat: meminfo: -LKP_SERVER: 172.168.131.113 LKP_CGI_PORT: 3000 result_root: /result/iperf LKP_DEBUG_PREFIX: bash -x -- 2.23.0

1 0

[PATCH compass-ci 1/2] delete unnecessary yaml files
by Xiao Shenwei 16 Nov '20

16 Nov '20

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- user-client/jobs/iperf-pxe.yaml | 59 --------------------------------- user-client/jobs/iperf-vm.yaml | 40 ---------------------- user-client/jobs/myjobs.yaml | 2 -- 3 files changed, 101 deletions(-) delete mode 100644 user-client/jobs/iperf-pxe.yaml delete mode 100644 user-client/jobs/iperf-vm.yaml delete mode 100644 user-client/jobs/myjobs.yaml diff --git a/user-client/jobs/iperf-pxe.yaml b/user-client/jobs/iperf-pxe.yaml deleted file mode 100644 index 209c5d6..0000000 --- a/user-client/jobs/iperf-pxe.yaml +++ /dev/null @@ -1,59 +0,0 @@ ---- - -#! jobs/iperf.yaml - -#this job yaml file is used to run qemu test by using pxe, in compass-ci/sparrow/6-test/qemu, -#usage like this: 0_addjob iperf-pxe.yaml - -suite: iperf -testcase: iperf -category: benchmark -runtime: 300 -cluster: cs-localhost -if role server: - iperf-server: -if role client: - iperf: - protocol: tcp -job_origin: jobs/iperf.yaml -testbox: vm-pxe-2p8g-1 -arch: x86_64 -node_roles: server client - -#! include/category/benchmark -kmsg: -boot-time: -uptime: -suite: iperf -testcase: iperf -category: benchmark -runtime: 300 -cluster: cs-localhost -if role server: - iperf-server: -if role client: - iperf: - protocol: tcp -job_origin: jobs/iperf.yaml -testbox: vm-pxe-2p8g-1 -arch: x86_64 -node_roles: server client - -#! include/category/benchmark -kmsg: -boot-time: -uptime: -iostat: -heartbeat: -vmstat: -numa-numastat: -numa-vmstat: -numa-meminfo: -proc-vmstat: -proc-stat: -meminfo: - -LKP_SERVER: 172.168.131.113 -LKP_CGI_PORT: 3000 -result_root: /result/iperf -LKP_DEBUG_PREFIX: bash -x diff --git a/user-client/jobs/iperf-vm.yaml b/user-client/jobs/iperf-vm.yaml deleted file mode 100644 index 434b964..0000000 --- a/user-client/jobs/iperf-vm.yaml +++ /dev/null @@ -1,40 +0,0 @@ ---- - -#! jobs/iperf.yaml - -#this job yaml file is used to run qemu test by using kvm, in compass-ci/sparrow/6-test/qemu, -#usage like this: 0_addjob iperf-vm.yaml - -suite: iperf -testcase: iperf -category: benchmark -runtime: 300 -cluster: cs-localhost -if role server: - iperf-server: -if role client: - iperf: - protocol: tcp -job_origin: jobs/iperf.yaml -testbox: vm-2p8g-1 -arch: x86_64 -node_roles: server client - -#! include/category/benchmark -kmsg: -boot-time: -uptime: -iostat: -heartbeat: -vmstat: -numa-numastat: -numa-vmstat: -numa-meminfo: -proc-vmstat: -proc-stat: -meminfo: - -LKP_SERVER: 172.168.131.113 -LKP_CGI_PORT: 3000 -result_root: /result/iperf -LKP_DEBUG_PREFIX: bash -x diff --git a/user-client/jobs/myjobs.yaml b/user-client/jobs/myjobs.yaml deleted file mode 100644 index 73bbb63..0000000 --- a/user-client/jobs/myjobs.yaml +++ /dev/null @@ -1,2 +0,0 @@ -#! jobs/myjobs.yaml -test: -- 2.23.0

1 0

[PATCH compass-ci] container/git-mirror: get the lkp and comitter ID from host
by Cao Xueliang 16 Nov '20

16 Nov '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/git-mirror/start | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/container/git-mirror/start b/container/git-mirror/start index ee79578..6ec0001 100755 --- a/container/git-mirror/start +++ b/container/git-mirror/start @@ -9,11 +9,14 @@ DOCKER_CCI = '/c/compass-ci' DOCKER_REPO = '/c/upstream-repos' docker_rm "git-mirror" +lkp = %x(id -u lkp) +committer = %x(id -u committer) + cmd = %W[ docker run --restart=always --name git-mirror - -u 1090:1999 + -u #{lkp}:#{committer} -d -e CCI_SRC=#{DOCKER_CCI} -e REPO_SRC=#{DOCKER_REPO} -- 2.23.0

1 0

[PATCH lkp-tests] tests: add hackbench-git test script
by Zhang Yale 16 Nov '20

16 Nov '20

[Why] hackbench-git is an old version tool required by the customer. The hackbench-git usage: hackbench [-pipe] <num groups> [process|thread] [loops] So need to add new test script for hackbench-git. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/hackbench-git | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100755 tests/hackbench-git diff --git a/tests/hackbench-git b/tests/hackbench-git new file mode 100755 index 00000000..f2be0c9d --- /dev/null +++ b/tests/hackbench-git @@ -0,0 +1,30 @@ +#!/bin/sh +# - nr_threads +# - mode +# - ipc +# - iterations +# - loops + +## Hackbench-git is both a benchmark and a stress test for the Linux kernel scheduler. +## Its main job is to create a specified number of schedulable entities pairs. +## (either threads or traditional processes)which communicate via either sockets +## or pipes and time how long it takes for each pair to send data back and forth. + +[ "$ipc" = 'pipe' ] && ipc_option='-pipe' +[ -n "$iterations" ] || iterations=10 +[ -n "$loops" ] || loops=100 +[ -n "$nr_threads" ] || nr_threads=1 + +run_hackbench() +{ + # args: groups number, mode [process/thread], loops, ipc [pipe/socket] + hackbench_path="$BENCHMARK_ROOT"/hackbench-git/hackbench/hackbench + [ ! -x "${hackbench_path}" ] && exit 1 + + echo "--------$ipc $mode num=$nr_threads--------" + for i in $(seq 1 $iterations); do + log_cmd $hackbench_path $ipc_option $nr_threads $mode $loops + done +} + +run_hackbench -- 2.23.0

2 2

[PATCH compass-ci 2/2] lib/matrix2.rb: judge the value.size condition first
by Lu Kaiyi 16 Nov '20

16 Nov '20

[why] avoid to execute some unnecessary operations and improve performance. [how] keep the value.size condition judgement to the front of block. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 62d174c..86cfc93 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -116,15 +116,15 @@ def combine_group_query_data(query_data, dims) job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + if value.size < 2 + groups.delete(group_key) + next + end suite_list = [] value.each do |dimension_key, jobs| groups[group_key][dimension_key], suites = create_matrix(jobs) suite_list.concat(suites) end - if value.size < 2 - groups.delete(group_key) - next - end suites_list << suite_list end -- 2.23.0

2 1

[PATCH v3 lkp-tests] sysbench-mysql/PKGBUILD: fix "No such file or directory"
by Zhang Yu 16 Nov '20

16 Nov '20

[why] only openeuler have dir "/usr/local/mysql/lib/" Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- pkg/sysbench-mysql/PKGBUILD | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/pkg/sysbench-mysql/PKGBUILD b/pkg/sysbench-mysql/PKGBUILD index d9f342a5..c214d8a2 100644 --- a/pkg/sysbench-mysql/PKGBUILD +++ b/pkg/sysbench-mysql/PKGBUILD @@ -18,6 +18,8 @@ build() { package() { cd "$pkgname-$pkgver.$pkgrel" make DESTDIR=$pkgdir install - [[ $os == openeuler ]] && mkdir -p $pkgdir/usr/lib64 - cp -rpf /usr/local/mysql/lib/* $pkgdir/usr/lib64/ + [[ $os == openeuler ]] && { + mkdir -p $pkgdir/usr/lib64 + cp -rpf /usr/local/mysql/lib/* $pkgdir/usr/lib64/ + } } -- 2.23.0

1 0